👩🏻‍🎓 🤴🏿 👋 How Highload PHP projects are created: transcript 🍼 👨🏿‍🚒 🕙

On July 28, our Instagram account and YouTube hosted a live broadcast with Alexander Vysotsky, a leading PHP developer at Badoo's London office, who works in the anti-spam team. Sasha talked about how Highload projects are created in PHP, his life in London and, of course, about Badoo.

My name is Alexander Vysotsky, I work as a leading PHP developer at Badoo. We're developing dating apps Badoo and Bumble, which have over 500 million users worldwide.

We have several offices in different countries, but most of the developers are located in Moscow and London. In total, the Badoo development team has about 300 people. We have 20 open source projects and many internal tools, which we often talk about in our blog on Habré.

Today I want to tell you what it's like to work as a backend developer at Badoo in the conditions of two releases a day, a real highload and a million lines of code, how to adapt to life and work abroad and keep your family together.

How I got to Badoo

My hometown is Saratov, where I also received a specialized education: I graduated from the specialty and postgraduate studies at the Faculty of Computer Science and Information Technologies of SSU. By the time he graduated from graduate school, he had worked as a backend developer in various fields: from the tourism sector to games.

In mid-2017, the main project at work ended, and the question arose in front of me, what to do next: look for something new in Saratov, move to Moscow or St. Petersburg, or go to foreign companies? At this point, I already knew about Badoo and applied for an open position in the London office. However, I didn't have enough experience and knowledge to get an offer. But at the same time I received two job offers from Germany and the Netherlands. Together with my wife, we decided to move and work for a German company. For a year and a half we lived in Leipzig - one of the largest cities in Germany - where I worked on tourism solutions.

However, the desire to work at Badoo did not disappear, and I applied for an open position again, a year later. After several phone interviews and one face-to-face interview, I got an offer. So at the beginning of 2019 I moved to London.

Both moves - to Leipzig and London - were a serious test. I found myself outside the usual environment: there were no friends, relatives, parents with whom I was used to communicating every day. It was not easy for me, and for my wife, our family as a whole. We were looking for a way out of this state and tried to integrate into the new society as quickly as possible.

In Germany, the obvious barrier was the language: we always learned English, but here we had to tackle German. It took a lot of effort, but after 1.5 years of living in Leipzig, we learned the language well, practicing every day. In London, this problem did not arise, besides, we already had the experience of living in a foreign country. Badoo provided maximum support during the move in matters of finding an apartment, in communicating with the tax authorities. This helped to integrate into local life.

One of the most difficult moments for a Russian when moving is taxes. In England, a progressive scale is used: the tax rate rises depending on the level of income. Another problematic issue is medicine. Someone says that everything is in order with her, someone does not agree. I have had a positive experience.

A separate story was with the search for housing. Good options are taken apart very quickly. In addition, before you get an apartment for rent, you need to go through a check. In my case, there were even calls to Germany to our previous landlords: they were asked how good a tenant I was.

It is also important for me that my wife likes life in a new place. She always wanted to get the profession of a designer and is now preparing for admission. Found courses that she likes; at the same time, she tightened her language and passed the IELTS test to enter the Bachelor's degree in interior design. London has a very wide range of educational institutions, but you need to remember that for foreigners the cost of education is several times higher than for local ones.

Next, I will answer a few questions from the chat.

What helped to adapt each time to a new place?

The main support is always within the family. In our case, my wife and I supported each other, and this helped to overcome all the initial difficulties. In addition, every time I was lucky with the team: I have very cool colleagues, they always support in word and deed, share their experience, at first they can go with you and solve the problem. In general, a good family and good colleagues are worth their weight in gold.

Is it worth leaving freelancing to work in a large company with a lower salary if you have never had such work experience before?

The question is, rather, not in salary, but in the processes associated with the transition to a large company. When you freelance, you set a working day for yourself and build workflows that help you get things done. In a large company, everything is different: tough deadlines, much more communication - both between colleagues in a team and between teams. You need to think about whether this pace and format of work is right for you, and make a decision based on this.

As for the salary, you need to look at the future. If you now abandon a large salary on freelance and move to a smaller one, to a company, then later the salary can grow stronger thanks to experience in the industry.

Small disclaimer: you can visit our website tech.badoo.comwhere we post current vacancies. Maybe you will come across something to your liking, and you leave a request.

What are some of the benefits of Badoo compared to smaller companies?

It is better to read about this on our website : there is more listed there than I can remember. The main points that cannot but rejoice are voluntary medical insurance, fitness compensation, breakfast, lunch, dinner in the company, high-quality working equipment.

Is it true Badoo doesn't hire to work in England? I can't find an explicit answer.

Before the pandemic, we had vacancies in both Moscow and London. In the current environment, of course, relocation and any business trips between offices are temporarily frozen, although recruiting continues. See updates on the website: it always indicates in which office a vacancy is open. The company adheres to the maximum transparency in this matter.

Did you have to work remotely? Has it become more difficult? How did you interact?

Yes, we still work remotely. At first it was difficult due to the fact that my colleagues and I did not always coincide in time: someone else “did not come” to work, someone had already “left”. Synchronization between different teams is required. Communication issues have become more difficult to solve. Instead of asking questions personally, I had to write or call, which takes much longer.

For interaction, we have a large set of chats, video conferencing that we use. Now it has become easier - we are used to it.

How to control effort, perseverance, self-motivation, procrastination?

It's no secret that due to the pandemic, most IT companies have switched to work from home. It was difficult for me to rebuild on other rails, but I singled out a few points for myself.

First, tight control of working hours is needed. You can inform all colleagues that from 9 to 18 you work and are available in all messengers, and outside of this time do not respond to requests. Everyone treats this with understanding. When you work from home, it is difficult to separate work and family, but it is very important to maintain a psychological mood and relationships.

Second, self-motivation and procrastination are important. Many articles on Habré say that procrastination is normal, but bad when there is a lot of it. I use the following technique: if I come across a large problem, I divide it into many small ones. And then the new scary feature no longer seems scary, and it can be released without any problems.

PHP and highload at Badoo

I'll move on to questions about work. I work in the anti-spam team. As the name suggests, it may seem that we are only dealing with anti-spam, but this is far from the case. Our goal is to provide the best possible experience for using our applications. The tasks of protecting users are very important for us, a lot of resources and efforts are allocated for this.

To summarize, we are engaged in the fight against malicious users: those who send spam, engage in fraud and spoil the user experience. We are actively using ML. Of course, I cannot go into details so as not to make life easier for spammers, but here are some examples.

We have a spam / scam detection model. We made a tool for analyzing mobile traffic for a parallel team. Also, our company uses neural networks: for gesture photo verification and when sending obscene photos in the messenger. Recently, our colleagues launched a so-called "dick pic detector" to protect against inappropriate content in private messages: the user can choose whether he wants to see such content.

How does Badoo handle spam? Simple if or ML already?

I've seen a joke somewhere that ML is just a big bunch of if / else. But, of course, this is not at all the case with us.

Badoo has several uses for ML, as it can greatly improve projects. For example, as I already said, we use user photo verification, and ML helps with this: it determines that a person actually took a photo now and that he did what was asked of him. Neurons are cool.

How do you implement ML? PHP, another language, some kind of framework, completely own development?

We have very cool guys in the data team. The blog has a cool report from Alexander Krasheninnikov - unfortunately, he is already my former colleague - which tells what kind of team it is, what problems it solves, how it improves Badoo's work and helps us all. The Data team has created their own ML framework, which is very easy to use and available to all other teams in the company: we can say that they have already done all the work for us. They have a very cool implementation, excellent documentation, a very straightforward approach to working with the framework.

What were the biggest challenges you faced at Badoo?

I cannot single out a specific task or project that would be the most difficult. There are interesting projects, and there are very interesting ones. In my practice, these are all projects related to Machine Learning. When I was in graduate school, I touched on this topic, and this area appeals to me. We did a project for the Performance Marketing team related to traffic analysis - it was very cool, we found a lot of useful insights.

Why are you using PHP?

PHP is a great tool for solving web development problems and quickly developing a scalable project. But the attitude towards this language in the community is ambiguous, and this is due to its reputation. Since the inception of PHP, there has been a perception that it is very easy to write bad code in it. In my opinion, the low entry threshold is not a disadvantage. On the contrary, it allows you to involve a wide range of people in the development. Besides, it really does a good job of developing web applications, and with each new version the language improves.

PHP 7 has taken a huge step forward in terms of productivity and development. We have a great article in our blog on Habré about how the transition to this version allowed us to free up a significant part of the resources.

The popularity of PHP has recently been falling, and this is natural - other tools and languages appear that compete with PHP, and many developers are switching to them. But it is customary for us to select tools for a specific task, and PHP copes with the task.

What do you think about PHP 8, are you planning to migrate?

We are actively monitoring every new PHP release. Of course, we will use all the features of the new version - of course, after we make sure that our code is compatible, and the transition to PHP 8 will give more benefits than the time that we will spend on the transition itself. We'll decide on the transition when PHP 8 comes out.

As I said, when we switched to PHP 7, we freed up a lot of servers, which we sent to other tasks. That is, the transition to a new version can bring great benefits.

Does Badoo use PHP compilation?

No.

Tell us more about Badoo's self-written framework. On what basis is it implemented and what is it more like?

This is done by the platform team - the guys who make "backend for backend" and support the bulk of backend developers. They give us a lot of cool stuff. I've already partly talked about this: for example, they implement the same queuing functionality that is widely used in the company; they also make a cloud service for our needs.

I would not say that the framework is similar to something specific. I've worked with Laravel and Symfony - of course, there are some parts, and we can use modules that are in open source in our project. But I don't think that our git repository is very different in approaches from other modern frameworks. We use package managers to pull in third party dependencies, we use autoloading, we use modules to encapsulate pieces of code.

Why is Badoo using a monolith and not microservices?

This is a rather holy question, the community is split into two camps about this. It's not a secret for anyone that we use a monolithic architecture, and during the existence of the project we have learned to deal with the disadvantages of this approach and use all its advantages. In addition, we have a set of services (in Go, PHP, C ++) that we actively use in our daily work.

If we understand the question as “should we give up everything and use all available resources to rewrite the existing monolith for a microservice architecture”, I will answer that no. We have business challenges that the existing solution successfully handles. If necessary, we are ready for changes, but, as I said, we choose the tool in accordance with the task at hand.

How do different pieces of a monolith scale under load?

Good question to follow up on monolith and microservices. We have a great presentation in our blog about performance and how our backend is built from an architectural point of view - I'll tell you briefly. We have about 600 servers that handle all requests from clients, and they are responsible for our monorepository. With this approach, we have some flexibility in scaling, add new cars, put in the code - and they are ready to use.

How seamless is deployment in a monolithic environment?

The answer to this question can be divided into two parts. The first is the technical implementation of our CICD-pipeline, which was well described by my former colleague Yuri Nasretdinov in his talk at HighLoad (" 5 ways to deploy PHP code in highload conditions "). I recommend watching. In short, we have several hundred servers that serve user requests. During deployment, we only deploy changes to the repository and atomically switch symlink.

The second part is to make sure that deploy doesn't break our production. Before release, any code is checked using a unit, integration and UI test, as well as a static analyzer for obvious problems. We have a large and professional QA department, which allows us to successfully release twice a day.

With such a short release cycle, it is very important for us to maintain the quality of our product at a high level: we do not want to roll out bugs / fatals into production. Therefore, testing of features that are rolling into production comes first. Every back-end developer is interested in the fact that his feature on the back-end is launched without any additional actions from his department and from the front-end and mobile teams. There may be such a situation when you have a ticket for the development of a backend feature, you release it in production, but in reality they start using it only after a while. And then QA engineers come to you and ask why it doesn't work. Therefore, on the backend side, when we release functionality, we cover it with the maximum number of tests, mocks and QAPs to be 100% sure that everythingwhat we roll is 100% working.

One turnip, everyone is pushing to one place?

Yes, there is only one turnip, and all backend engineers are pushing into it. We have an internal rule for naming branches that are somehow related to the task at hand. Of course, you can't push the master directly, the branch is pushed there after a successful code review, after all the checks and unit tests, and after the QA engineer who worked on the task with you said that everything is in order.

Are you using DDD or other architectural patterns?

DDD is Domain-Driven Design. It is not an architectural pattern, but rather a methodology. I would not say that we have one specific approach, we use a combination of several.

About patterns: in the backend, several design patterns are used to solve problems, I would like to highlight this in detail. We use Event bus implementations, we have many queues, we send millions of events that are processed by the corresponding consumers. Also among the actively used patterns are the Module pattern: most of our code is split into separate, connected instances that interact through a limited open API.

Are you using exceptions or trying to avoid?

We use. And we try to avoid.

Is your API a monolith?

Yes.

How do you test the API?

We have unit tests and a whole framework for running a large number of tests in parallel with a minimal amount of time. You can read more about this in the article by Vladimir Yants in our blog , he described this topic well and in detail. In terms of UI tests, we use Calabash and Selenium to test that the UI is working correctly.

Test-driven development, when tests first, then code - don't you practice?

With us, each team can use their own approach when developing, and I know that some colleagues practice it. I know it works, but I don't practice it myself.

How do you feel about DDD?

As I said, we are compiling from multiple approaches. If we are talking about my personal attitude, I support any approach that allows you to effectively solve problems. DDD is worth investing in: it will allow you to make applications at a new quality level.

Tell us, was it such that production could not stand highload? How did you deal with this?

It was not in my memory. We have experienced engineers, our product has been in development for over 15 years, and the company has vast experience in highload development. We aim to maximize the performance of our applications.

PHP and MySQL - what to do to optimize backend performance?

It touches on the stack used in the company and performance, so I'll split the answer into two parts as well.

About the stack: due to the fact that Badoo has a large number of departments and teams, we use the widest possible set of technologies - from PHP, MySQL, Nginx, Go, C ++ to Tarantool, LUA and Scala. Each team chooses a tool to effectively solve the task. Since we work in highload conditions and process tens of thousands of requests per second, the issue of the performance of our backend becomes critical.

Now it's worth mentioning the tools that were created in-house and released in open source. The first tool is Pinba (PHP is Not a Bottleneck Anymore). It is a tool for collecting statistics and monitoring the performance of an application without affecting its performance and for presenting the collected data in a human-friendly way.

The next one is Codeisok: a tool for managing git repositories and conducting code reviews. We actively use our internal work, and before the feature goes to master, we apply the best code review practices (you can also read about them in our blog ) so that the most efficient code reaches production.

Another tool that allows us to track the performance of each separate piece of code is LifeProf: it allows us to automatically profile all requests. All of these tools (and more) can be found in our Github repository.

Are you using ORM or direct storage interaction? Why?

I already mentioned that we have our own framework. We use our own ORM implementation.

How is the interaction of the project modules organized? Class to class, or something more tricky?

Good question. Each module has one entry point (Front controller), which provides a unified and intelligible API outward - for other modules or for other features. We do not disclose the entire internal implementation for the rest of the project, we leave only what we want to give outside.

What DB-specific buns are used using ORM?

I do not fully understand the question, but I will try to answer.

I have already said that our main database is MySQL, most of the data is stored in it. We also use Exasol, Presto, Tarantool, and Aerospike for specific tasks. That is, we have a large set of storages for each task. We do not limit ourselves in choosing a tool: if the use of technology is profitable, we use it. MySQL is central to our application, and we use a variety of replication, shardings to keep the load efficiently.

How does developer testing work? To raise everything locally in docker, or something more difficult, on virtual servers?

Another situation where I cannot answer exactly.

We do not use Docker when developing in PHP (but admins use it), we have a common dev environment. Our platform team is engaged in, among other things, maintaining our dev-environment in working order for development, and there we run all the tests, lay out the features that we will roll to production. That is, we have a pre-configured environment.

I still don't understand what ML is on: PHP, Python, something else?

We used to use Python for the ML framework, but now they switched to Spark: this has greatly increased performance.

How do you balance the load on 600 servers? Do I understand correctly that this is a monorepa on each server, in docker?

Monorepa on each server, we balance it in a fairly standard way - using Nginx.

Do you use code generators? For what tasks?

We use it quite often. When developing, when you need to generate models according to some description in the configs, or if there is a class that must implement functionality according to a specific template.

How does database replication work for MySQL?

I can only answer superficially.

It's no secret that we have data centers in Europe and the USA, and it is necessary to maintain data consistency between them. We use replication between core repositories to keep information up to date at each location. Due to the fact that there can be a large replication lag between data centers, we use various kinds of caches for those tasks where data relevance is not critical.

How do Badoo engineers support the Russian speaking PHP community? Conferences, meetups, blog, informal gatherings?

Badoo actively hosts and participates in a large number of specialized events. This is embedded in the culture of the company and in the culture of our engineers. Developers constantly share best practices and knowledge at internal and external meetups, meetings and conferences.

We have Badoo PHP Meetup: twice a year, in our Moscow office. The last meetings were attended by about 250 participants. My colleague Vladimir Yants, whom I have already mentioned, is developing informal meetings in Moscow - BeerPHP Moscow. He already has followers in St. Petersburg, Saratov and other cities. The format, of course, is borrowed from similar BeerJS meetups, but it's still very cool: in an informal setting, communicate with like-minded people, colleagues and just dudes from the industry.

Badoo engineers regularly serve on the programming committee of the only PHP conference in Russia, PHP Russia. This year its online part became international and free for all participants thanks to our company.

We also have blogs on Habré and Medium , where we share all the developments (not only in PHP).

Do you take Juns or is the minimum middle?

As I said, you need to leave a request. During the interview, it will become clear whether you now have enough knowledge and experience to start working in the company, or if you should get relevant experience and knowledge elsewhere and then return, as I did.

Are you planning an office in the States?

We have an office in the USA, where Bumble is hosted - in Austin, Texas. But there is no engineering team there, and it is not yet known if we will expand.

, soft/hard- ?

I have already said that all open vacancies are on our website tech.badoo.com . I advise anyone not to be ashamed, not afraid, watch the requirements and leave applications.

It is difficult to say what specific expectations a company might have from a candidate. I would say: if, reading the description, there is a feeling that you are not suitable for this vacancy - you need to apply 100%. Relevant experience or knowledge can be learned from the person through several rounds of interviews. In my opinion, you need to make a request in any case.

From hardskills - of course, you need to have experience and understanding of how PHP and MySQL work: these are the main technologies that we use, our stack when it comes to back-end development. Other departments have their own stack.

Softskills is a broad topic. In my opinion, a lot of development is not about writing code, but about communicating with people. It is very important to ask the right questions. If you know how to do this, the problem is 50% solved. There are always problems of misunderstanding, but the fewer there are, the better for you, for the company and the product. That is, the ability to communicate well, work in a team is definitely necessary. It is also important that people know how to take responsibility for bringing the project (feature, part of the functionality) to completion.

How Highload PHP projects are created: transcript