This is a local vendor. To sell, we buy server hardware from large suppliers, often we transport it to different countries from one center. But for test new configurations, we turn to local suppliers, take hardware for tests or one-time projects. One of the hard drives was found to be defective and we returned it back to the supplier under warranty. And notably swearing that they contacted a small company, which delayed our test schedule.
The warranty department fiddles with the disk, and then they call:
- Why did you change the disk?
We are:
- I mean, changed?
- We sold you another. And here the case is the same, but inside it is different. Some traces of a screwdriver.
The game is complete! We started to swear, but then we began to understand. We started looking at the cameras and saw that our employee was very suspicious when he worked with this disc in the rack. As in bad comedies about crooks: constantly looking around, stepping aside, returning. It turned out that he changed the disk. Honestly, I don't know why. Its financial benefit is minimal, most likely, it does not even pay off the time it takes to replace the case.
We had a few more strange situations, and now I will tell you about them.
âForex trader
VDS are often bought for trading on exchanges. Well, on normal exchanges: I mean those where the geographic location of the server is important to minimize latency. But there are also Forex-level exchanges. Rather, they cannot even be called stock exchanges. This is the slang of traders - the kitchen. Let me remind you that many of them were included in the list of companies with signs of illegal activity,who started the Central Bank. I have a deep personal conviction, based on common sense and mathematics, that the system works like a swing for getting money out of not very knowledgeable clients. Perhaps this is not the case, but I can argue and justify my point of view if necessary. But in history, something else is important. My friend is calling, who took a server from us in Switzerland. And so he begins to openly accuse me and all of our employees that we get on his server during the trading process and interfere in his Forex transactions.
According to him, he came up with a great strategy, and unprofitable transactions are taken by the market, while positive ones are not taken on time, they are ignored. And we are to blame for this. More precisely, at first he applied with a task that he was not satisfied with the performance of the server. In his words, it fell radically at the time of the news release. At 4:30 pm, when the world statistics comes out. At this point in time, all traders who work with automated trading systems begin to multiply their activity. If, roughly speaking, he makes ten transactions within a day, then at these 16:30 and one minute he can make one hundred transactions. Naturally, this creates a peak load, and not locally, but on the receiving server. But the trader does not understand this, he thinks that our server is exactly at 16:30, when he needs to place an order or close an order, it slows down. And this coincides with the most necessary time.And I can't believe that this is just a coincidence.
Since this was an acquaintance of mine, at first I spent a week giving him an educational course on how servers work and how exchanges work. Then I selected the admin so that he personally checked his server. Then it began: âYou brought me here, you climb there, steal my trading strategy, you are a thief, you bastard⊠You are lying to me. You come to me, you know that I have a cool trading robot there, you want to steal it. You climb into my server, you shit statistics for me. You tweak the settings so that nothing works for me. " I switched to another hosting. Then like more and more. And only two years later we began to communicate again, he admitted that I was right. But by this time he was already in debt.
The main problem here is working with a friend. If it was just a client, then we would carry out diagnostics and not receive insults.
âGame servers for schoolchildren
Schoolchildren quite often host game servers with us. These are, let's say, difficult clients, because they choose the third priced tariff plan. The first is a promo for 30 rubles per month (at the price of IP), the second is a stripped-down version of standard configurations for 130 rubles, and the third is already a full-fledged server - from 300 rubles per month.
So the kids open the server and start playing. After a while, they begin to be didos with enviable constancy. Not everyone, roughly every tenth server. After the start of didos, our system blocks them. After another five minutes, a child with great ambition and a rich three-story obscenity begins to write in our support about what kind of bad people we are here. They cut him off, which means the game, he has no income there because of this, we got there for millions because of him now. He will come to beat us.
The support politely explains the essence of the problem and offers to activate protection against DDOS attacks for 400 rubles (and the first month is free). According to our practice, then it will be possible to play completely calmly. To which the client begins to accuse us that it is we who did it on purpose so that he would pay us more. Further, the question of refund naturally arises, and usually we return the funds and easily part with such a client.
One of the tickets looked like a similar one, but with a nuance. There was no blockage, there was no didos, there was no three-story mat either. The client simply requested a refund from the server, which was paid for several months. And here I must say that we have a link for payment that can be displayed outside, and anyone can pay for hosting for some client. This is convenient because the accounting department does not have to log into the admin panel. The students used this link for crowdfunding, jumped onto the server and started playing.
The client writes a request for a refund, we send it to the payment gateway. And suddenly the owner deletes the server, the same mate begins and wild cries about what should be returned to his account, and not to the accounts of the payers. That is, conditionally, they had about forty payments of 100 rubles each, and all their payment gateway was going to shove back on the same cards from which they paid. But the main character needed a return on one - his - account. In general, in the course of a dialogue with support, he has already openly admitted that he wanted to receive other people's money. A couple of months later, the same story repeated itself with minimal variations with another student.
âThe shortest detective
It happens that our servers are taken to bypass locks. People deploy VPN and just go through our server to the big Internet, if it is blocked at work. Well, or just take a server in another jurisdiction such as Switzerland and go through it. Why this is important, there is here about the mask show.
And then the client writes that our server is not responding.
We are asking for a trace. The client does, sends. We inform you that RDP is blocking his hotel and wish you a pleasant holiday in Thailand. The client is a little in a panic, but we explain that his Wi-Fi hotspot is named after the hotel. And we give a link on how to get around it. It helped that time.
âRegistration under bots
A very strange situation was with the test periods on our 32-core machines. Someone learned to take American numbers, parse voice, Russian text and numbers. As soon as cryptocurrencies made another jump, auto-registrations began, and then the servers were used (judging by the nature of the load) for cryptocurrency mining. I don't know what it was, but mining bitcoin on VDS without a video card is a frankly so-so idea. The processor goes to the ceiling, for three days they try to do something there, then the period ends. We updated the captchas, but on the next wave of the bitcoin rate jump, auto-registrations started again. I honestly don't understand what the point is, because renting an American number is more expensive than the possible profit from mining on a processor for three days. Now we have finally defeated these waves.
âBlocking by IP
We issue one static IP to a client when renting a server. This is not a carousel of dynamic addresses, not replacements once a month, but a specific IP tied to a client. First, we check its purity and absence in any blacklists, and then we give it to the client.
Further, the client can spoil this IP address. For example, insta bloggers and insta bloggers often deploy Instagram promotion tools with us. Instagram expectedly bans them after a while. Further, in support, appeals begin: "Why was Instagram banned me ?!" - with a bunch of emoticons. Or: âWhy was Avito banned me, change my IP addressâ, but there are almost no emoticons about Avito.
They also wind up reviews and views on Amazon. Then - in support: "Guys, this is not my first account, can you please change my IP address because I was banned?"
It so happens that the admin configures the client to work on the server, but forgets to clarify the question about the licenses. The CEO is calling, who has 25 employees, and they are all sitting on a remote desktop, we have placed accordingly. The whole batch is that the system administrator who set it up was outsourced. He's set up a bunch of virtual desktops. The person paid about 35 thousand. He had 25 employees there, and for 120 days the person did not know any problem with connecting to a remote desktop at all. And the tsimes is that Microsoft gives a trial for hosting this remote desktop server for 120 days exactly. A person has been using it for four months, and then suddenly, in the middle of the fifth month, he discovers that not a single employee can enter his place. Dictates a mistake to us, we understand everything perfectlywhat's going on there. And we offer him two options:
- You either delete this service, which does not allow you to connect at all, or pay for each license.
- In general, guys, I will not pay triple the price from the server.
It is not surprising, because the license costs 91 thousand rubles, and his server costs 36 thousand.
- Guys, it is necessary, in short, to decide. Let's do it this way: I need it for free anyway. What if we make a contract for another person, and thus I have another 120 days? And what if you change a person, I now have an acquaintance, he will register?
- Do you understand for sure that now you are asking the official partner of MS how to deceive them?
- Yes! Guys, what other options are there?
âUnexpected departure of a colleague
Our turnover is minimal, so when a person leaves, it's a whole event. The main thing in this event is that at the exit interview, the employee can already be completely frank, and we often find out about some things that are annoying, but which are usually not talked about. This is the most helpful feedback for recruiting and retaining people.
And here is one of the situations. The employee picked up and did not show up for work. He worked, worked and then disappeared. No exit interview. They began to figure out what had happened, maybe he was offended by something. No internal prerequisites. Later they began to communicate with him remotely, he sent a link to an article on Habré about professional burnout.
This did not explain anything, but we managed to talk further. The employee said that he read everything carefully and did not want this to happen to him. Therefore, she quits.
No conclusions or actions. It's just that he may be happy somewhere else.
âDebosh
At the corporate party, the employee made a terrible brawl. It is difficult to imagine more terrible! First he fought with colleagues, then he began to attack random people around. Somehow they twisted and pressed to the floor. A man came up, started shouting that we were restricting a person's freedom, pushed two people off him. Our hero got up and gave the human rights activist a powerful bream, after which he helped to keep it. We all came to this: ambulance, police, local security. When handcuffed, he managed to escape and smash the policeman's nose.
As a result, he did not receive a criminal offense, the police turned out to be kind. They only left him in the bullpen until the morning and let him go home to his wife and children. Itâs like the end of the story, but itâs not clear what to do next, because in the conversation he explained that there is no guarantee that exactly the same will not happen again. For example, in the office. Or in the server room. Or in negotiations with a client or a data center.
We thought for a long time what to do. It seems wrong to fire. There are no complaints about the work, everything is fine.
But it seems that if a person screwed up on social networks and offended someone, large companies get fired. Here is a similar situation. Unclear. In general, as a result, they were transferred to full remote control even before it became mainstream.
â Destroyed 1C server
We are terribly serious about protecting the financial data of our clients, as quite often our data centers in other jurisdictions are chosen for this. One of the consequences is that if a client issues a direct command from his personal account to delete a car, then it is deleted, and not placed in storage for several months. To confirm this action, you need to enter the password from the LC again.
So, the accountant of one of the clients needs closing documents that the service has been provided. The documents are in your personal account, they are automatically generated. She reports that access has been lost. But she knows the contract number and all the details. OK, we say: send a letter on the letterhead of the organization with the signature of the general and the seal, we will restore your access.
The letter arrives, in it a request to change the main access mail.
We change the mail accordingly.
Then the CEO writes to us and says that the accountant has deleted his 1C server with the database. And tearfully asks us to restore it. By this time, he had already fired the employee.
This is a tragedy in its purest form, because everything is legally correct, but conceptually it is not. We thought for a long time what could be done in this situation, and decided that we would have a separate role within the account for billing (closing documents and payments). Now it is in the release plans.
âGeneral conclusions and how the story with the hard drive ended
We try to figure it out and understand what can be done so that the risk does not repeat itself.
In the case of the replaced hard drive, the employee wrote on his own and quit the same day. We didnât make any claims against him, because we understood that it would be extremely difficult to prove it. And, despite all the irrationality of the situation, we have a rule: if you know the risk, then you must try to minimize it for the future. In this story, it was not very clear what to do. As a result, they conferred and allocated a separate room in the office for assembling equipment, a camera was placed over the table there. In this case, we are probably more paranoid than optimizers, but this is the principle. Perhaps this is because we are from the financial sector, and this is very serious. Here we insure not only against theft, but also against falsely accusing our employee if there is any confusion from the vendor or something like that, because the equipment is quite expensive.
