I love starting various side projects, I think this is one of the best ways to learn something new and really worthwhile. And I have one serious flaw - I almost never get things done. This, of course, is not about study projects for which I will be given a grade or tasks set by the employer. I'm talking about my own ideas that I light up with outside of constant work or study. Every time I master some completely new skill that I consider to be in demand and I do not see the prospect of learning something else like that, I forget about the project at all. But this time I decided to improve myself - to start the project, complete it and tell about the path that I went through.
A little about myself
I have a specialized education, bachelor's degree in mathematics, master's degree in IT. Surprisingly, quite a lot of things that I mastered at the university were useful to me, especially the master's degree. In any case, I managed to tune my brain in the right direction in order to productively absorb the necessary information, show myself well in this area, find a job, and advance in my career.
While studying, I was a C ++ fan and a game dev. Then I developed a self-written engine and wrote games on it, tried. I had a diploma in C and was connected with the use of graphics processors in calculations. Then I got interested in Java and Android development. He completed several educational projects in this language, and then, as part of a student project, with the support of an international corporation, wrote a console utility for analyzing program performance. All of this, then, grew into an Android application with the ability to test your phone in terms of performance and compare with others.
The python world was on the horizon. Unexpectedly, at the suggestion of my good friend, I became interested in Data science and python, this was more than four years ago. Coincidentally, I had to change jobs and my skills were assessed as suitable to start working on backends for internal analytic services. So, through data analysis, I came to web development in this language.
By duty, I also had to be a full-stack developer when real front-end developers were extremely busy or when UI quality standards are not the first place.
This is a description of the path that it took to create your own full-fledged web application without full immersion in the technical details. Links to commits from my repository will be dropped along the way. Briefly about the stages of this journey:
- UI
- , , CI
- production-
- production-
- https
- AWS. , .
The choice of the subject of the application is another story, the final choice fell on tracking habits. The home page would have a set of buttons with tracked habits. We made an action - pressed a button - and so every day. The data should be saved and displayed on a separate page in the form of a table, habits in rows, and calendar days in columns, a filled cell indicates that the required action was performed on that day. Like the simplest paper habit tracker.
The technological stack was obvious to me: react, django, postgres, nginx, uwsgi.
Layout initial UI
I decided to start from the user interface and installed nodejs.org/en/download/package-manager , github.com/facebook/create-react-app , then created a project:
npx create-react-app easytrackAnd I started layout. From the very beginning, I couldn't think of anything easier than to hardcode a list of business logic objects directly in the program and display them as a list in the ul tag. These objects are: the topic group, the tracked item, the actual tracking records of a specific item on a specific day.
On the first page, I had thematic groups, and clicking on any of them opened a list of items that can be tracked.
I also made up another page, which contained a simple table with statistics for the last days.
Develop a backend
At this stage, there was a need for a backend. It is needed to save objects to the database, manage users, and differentiate rights. I already had to use Django for my own projects (not brought to its logical conclusion, of course), but there was one difficulty - I had to use Django Rest Framework, which I had never dealt with at all. Thanks to Building Django 2.0 Web-applications [1]. I went through it from cover to cover, except for the last chapter, where they created APIs on DRF, I skimmed through it with my eyes. In the course of the story, I will turn to her more than once.
Nowhere to go, opened the www.django-rest-framework.org documentation and started smoking it, and also turned to the mentioned book.
I installed and activated the virtual environment, installed Django and created a django project in the root of the project:
virtualenv venv
. ./venv/bin/activate
pip install django
django createproject configI will rename the main folder to django, so that by the name everything about its contents is immediately clear. Inside it will be a python module called config, which is also very convenient. Frontend code wrapped in react folder. This is how the general folder structure developed, which has survived to this day ( look at Github ).
At the root of the project, I created the main application:
django createapp coreCreated classes of business logic models in exactly the same format in which I hardcoded them in the front, serializers and views ( link to commit in Github ).
We don't bother with the database and use the default SQLite. Through the admin panel, I uploaded a couple of test data samples that I had hardcoded earlier in the front. We are trying to connect to the frontend. python3 manage.py runserver in one tab, yarn start in another and drove.
The React development server runs on port 3000, and Django runs on port 8000. Nothing is easier than writing a fetch ('http: // localhost: 8000 / ...') request in the front. But it didn't work that way because of cors-origin, a special protection that prevents any site from making automatic requests to any server. Therefore, without thinking twice, I built it into the backenddjango-cors-headers ), configured it - it worked. Only then I guessed to add a proxy section to package.json and point to the backend, then fetch ('/ api / v1 / ...') started working normally and no extra settings were needed anymore.
At first, it was terribly naive, because I made asynchronous requests in the constructor - everything worked for me and okay. Only later did I learn about lifecycle methods, where you can and where you should not do this kind of work. Now the elements were displayed, new elements could be created.
Implement user management and separation of rights
At this stage, the only thing that was missing was the separation of rights to data elements: they were all created on behalf of an anonymous non-logged in user. I needed to somehow integrate into it without breaking the django ecosystem.
To begin with, I hardcoded the login / password on the frontend and formed the Authorization header with the value 'Basic' + base64.encode (username + ":" + password). Then I thought to generate a string in base64 format and save it on the client when entering a username / password. But there were big doubts about this decision in terms of security, I wanted to try something different.
I rummaged on the Internet for a short time and learned about the JWT technology and the django-rest-framework-simplejw modulewhich provided authentication classes for DRF and views for getting a token pair and for updating an access token.
From the frontend side, it was enough to save a couple of tokens in localStorage and pass the Authorization header with the value "Bearer access-token". Finally, I laid out the login pages and closed all access for unauthorized users to the site using permission classes ( link to the Github commit ). On the front, if there was no refresh token, I redirected to the login page ( link to the Github commit ).
Then, I made it possible to register on the site, created a view on the backend, laid out a page, set up sending requests. In the future, my dream was to write account activation by mail.
Refactor, implement missing basic functions, CI
At this moment, a minimally usable application turned out and I wanted to collect feedback. I showed the application to my wife and asked to use it without a single prompt from my side. Registration was quite successful, but then everything did not go quite the way I thought. Then I realized that subject folders should not be at the forefront, but should be an auxiliary tool, and we also need to add hints that we are creating a set of buttons so that we can click them once a day. She expected it to be like a regular paper habit tracker with a spreadsheet where you need to paint over cells. When I realized this, I started refactoring. And besides that, there was something to refactor. I went over the module structure almost from scratch.
At this point, I decided to add beauty, css styles and responsive layout. In this I am not particularly strong and relied on CSS-frameworks. The choice fell on Bulma , based on the number of stars, downloads from npmjs.com , despite the fact that I did not want to take Bootstrap. At the very least, I coped with this task.
In parallel, I improved the backend functions. Made a full CRUD. The dream of confirming registration via mail also came true. I figured out the functions of sending letters, gained the ability to debug everything through a debug mail server.
python -m smtpd -n -c DebuggingServer localhost:1025For staging, I started a junk Google account and managed to set up sending letters via Google mail.
As for the test coverage for the backend, everything worked out quite well for itself, but for the frontend everything is completely sluggish so far. But I do not despair, suddenly then it will turn out to be adjusted.
The next step was to set up CI, I started GithubActions to run the tests, used the configuration constructors to launch it, tweaked it a bit and that's it.
Generate settings for the production environment
After a short time putting the code and small details of the logic in order, I had to start forming the production configuration. For this I was inspired by the same book [1]. I split the django-configuration files, python-dependencies into 3 parts ( link to the Github commit ):
- common - everything you need for any environment
- dev —
- prod —
, .
uwsgi, nginx , dockerfile, .
[1] «1 — 1 », , , . phusion/baseimage, Ubuntu, .
, , postgres, . ( Github).
production-
, [1], AWS, . , - . , , , , . Free Tier, , , ? .
, AWS - , . , . , ECS.
Elastic Container Service
I saw that the Github Actions builder allows you to create a configuration for a continuous container deployment task on AWS ECS. Let's say I started to delve into this service and realized that in the local console I need to create a cluster, create a task definition and describe the container, having previously saved its image in another AWS ECR service, which is Dockerhub by function. The console offered 2 types of clusters: Fargate and EC2. The first technology is completely serverless, meaning that we just start the container and the runtime takes care of everything. Containers in a cluster of the second type run on their own virtual machine instance in the cloud. Not wanting to dive into this for a long time, I created a cluster based on Fargate. But I ran into the fact that I could not pass secret values to the container,because of this, the task was constantly falling.
While I was trying to bring the container to a working state, the cluster had been working for several hours and money was added to my tab with payment for services. The technology itself turned out to be paid and does not apply to Free Tier. I paid off the cluster and decided to deal with the payment a little later. I read in the documentation that you don't need to pay extra for using ECS on top of EC2, except for using EC2 resources, but EC2 is covered by Free Tier and I decided to try this path.
AWS Support
In the end, someone lied to me and they charged me some more money for this configuration. A total of $ 0.32, albeit a little, but still a shame. Then I wrote in support, said that I had difficulties with the setup and all the time thought that I fit into the Free Tier, but nothing worked for me, and the money was withdrawn. I asked for help. In response to my call, they happily responded, gave a voucher for $ 1, which covered all unexpected expenses. Nicely.
As a result, I had to forget about this container service and master EC2 in its purest form.
Elastic Computing Cloud
EC2 — . Free Tier 750 . , , - , .
[1] docker-machine, . , EC2. docker .
, dockre-machine create… EC2. docker-compose up -d . 80 . , «Public DNS» , .
Relational Database Service
AWS RDS. . , ( postgres) , Free Tier, . , , .
Simple Email Service
I also had to face the fact that the letters were not sent. The fact is that all mail services ban IPs that match EC2 instances in order to counter spam. It was imperative to use SES. Another AWS service.
I also configured it, indicated the mailboxes from which I am going to send letters, everything is fine. However, not everything is so rosy so far. New accounts are in sandbox mode by default, which does not allow sending emails to unconfirmed mailboxes. Had to confirm boxes for my beta testers.
Register a domain and configure a certificate for https access
To fully test the application, it was required to set up an https connection, and this turned out to be impossible without registering a domain. I didn't want to put email addresses, passwords, and tokens at risk by transferring them over an unsecured connection. For this case, I searched the entire Internet. Some registrars did not suit, because it is expensive, but somewhere it is very expensive, besides, many have just terrible reviews. While I was looking, the Yandex advertising network understood what I wanted and immediately offered me a company where I could register a domain for 39 rubles. There were more reviews about this service, so I created an account, chose a free domain and registered it for me.
Then, in order to access my EC2 instance using this domain, I created a hosting zone in the AWS Route 53 service, registered the DNS settings, and rewrote the NS records in the personal account of the service where the domain was registered to Amazon ones.
I used the Let's Encrypt service to create the certificate. I stopped at this configuration: the local nginx makes a proxy_pass to the container address, the container spins and is accessible through port 8080. It was not difficult to use the certbot console utility, generate a certificate and automatically configure https. Finally, allow access to the virtual machine on port 443.
The development of the project has come to its logical conclusion
, . , , SES , . , . , , , , , , , , . , , . :
- , , . .
- -
- ,
- , -
— , , . React , AWS, , , .
But the main realization is that one in the field is not a warrior at all. I worked on this project for over two months, many hours in my spare time ... And you know what? I have not sewn up so terribly yet. It becomes noticeable that the older you get, the less the desire to learn something “just like that”, the less desire to work a lot on some things that are “just interesting”. But at the same time you notice that there are more available opportunities to do something for the good, for the benefit. You begin to appreciate focused efforts, teamwork, as well as relaxation, time spent with close people.