Attention, I'm starting without overclocking!
Backdoor to the yard
Our duty room on the ground floor had such large windows, from the basement and almost to the ceiling. They went out to the service parking lot, from where in the morning all sorts of meters and other field workers departed. The parking lot was located at a sufficient distance from the front and all service entrances, and even behind two barriers.
One morning, then police cars drive up to the building, policemen get up at all the checkpoints and inspect all those who leave. An alert arrives in the service mailing list: suddenly (really suddenly, not as usual) a check for the licensed software appeared, they will inspect the workstations. Anyone who has a pirate thing on their computers must demolish it!
Of course, everything related to operating systems, office and utility software was mostly licensed. But not everything, not always and not everywhere; and what the employees instructed themselves on office laptops is a completely dark story. I rushed to check the cars in my area of responsibility for piracy, quickly demolishing something ...
... And at this time, engineers, with laptops and system engineers in their arms, begin to enter the duty room with a hurried and nervous step. They enter through the door, and leave, giggling at the absurdity of the situation, through the window: they blocked all the checkpoints, but the demons of law and order did not think of such a backdoor. So, while the accounting department was checking (where everything was exemplary), the employees pulled everything out.
The past is there
If you are interested and have not closed the tab, here is some exposure of what is happening in time, space and persons. I am a beautifully young, green as a sorrel leaf, an IT graduate who got a job as an engineer on duty at the Samara Megafon (which was then also the MSS Volga Region). For me, this was the first real contact with Technics with a capital letter and Technicians with an even bigger one: being the youngest imp in this hellish kitchen, I watched with delight the work of highly experienced
If any of the characters mentioned here suddenly recognizes themselves - hello to you!
Works - don't touch (but touch if it doesn't)
One of the above-mentioned super-techies was Misha Basov. Over the years of work at Mega, I have heard a lot of good and interesting things about him in the spirit that he was almost at the origin and launched a bunch of processes. I didn't manage to communicate with him properly: we literally met in the personnel department, when I brought the documents, and he took them away.
One of the monitoring systems with which we worked was written by Misha. I don't really remember what was monitored there, but I know that Misha wrote a temporary solution, which quickly became permanent. And it's good: a lot of what true techies do for their own needs in a hurry, turns out just fine. That monitoring also suited everyone, working without any support and maintenance, however, no one knew how.
A couple of years after Misha's dismissal, the monitoring started showing a blank page.
I immediately sounded the alarm. The shift leader sounded the alarm. The sector chief sounded the alarm.
The head of the department sounded the alarm. The head of the service sounded the alarm. The head of the department jingled bells. The call was heard by the IT director of the entire Volga region, immediately convening a meeting. There he called the head of the department. The latter barked at the head of the service. He, not understanding the essence of the problem, called the head of the department. This one, without understanding what had happened, called the head of the sector, who called the shift supervisor. Well, he turned the arrow at me.
Having somehow changed from duty, I went to this meeting. Many words were said, the person responsible for monitoring was called (we did not hear anything intelligible), it was recalled that monitoring Basov wrote that monitoring is very important, but that no one understands and does not know how it works ... It all came down to the fact that a non-working and incomprehensible system should be removed, and instead, a proven solution from a trusted vendor should be implemented.
While all this was being said, I asked someone for a laptop and ssh access to that server. It was interesting for me to see what kind of super cool system the legendary Basov wrote.
I go in, first of all out of habit I type:
df -hThe command answers me something like:
Filesystem Size Used Avail Use% Mounted on
/var 10G 10G 0G 100% /I clean the / var / log that has overflowed over the years, update the monitoring - everything works. Repaired!
The meeting stops, crumples, everyone leaves. On the way, the head of the department rejoices and promises me a prize! ..
... Instead of a bonus, I then received a mental stick for the fact that I accidentally broke off a rollback to order a monitoring system from a trusted vendor.
Where do the houses live
One of the duties of the engineers on duty was to control the electronic keys for accessing the computer rooms. The halls themselves were then very impressive to me: rows of racks clogged with server and switching equipment, fiber optic lines and cross-cables (somewhere - perfectly laid, somewhere - turned into an incredible lump of spaghetti), the constant hum of air conditioners and false floors, under which made it so convenient to cool drinks ... The entrances to the halls were sealed with heavy pressurized doors designed to provide automatic blocking in case of fire. Entry and exit were strictly recorded against signature so that it was known who was inside and why.
Most of all in these halls I liked, of course, the server cabinets of the "super houses" - two HP SuperDome 9000, which ensured the work of billing. Two identical nodes, one has always been a combat node and the other a synchronous hot standby. The difference between them was only in the IP addresses, one was xxx45, the other was xxx46. All the engineers knew both of these IPs, because if something happened on the billing system, the first thing to do is to see if the super-houses are visible. Invisibility of super-houses - akhtung.
One morning a similar Achtung happens. Within two seconds, all services disappear on both servers, the billing collapses into nothing. We quickly check the servers - they respond, but there is really nothing on them!
We don't even have time to start the set of measures, when we hear a loud shout " KILL, STUDENT!"; the archiadmin of all servers runs into the duty room, rips the electronic key from the turbine hall off the shelf and runs there.
Very quickly after that, the monitoring returns to normal.
Here's what happened: a new employee of the contractor, who configured a pack of new virtual machines, wrote them sequential static IP addresses with pens, from xxx1 to xxx100. The "student" did not know about the sacred untouchable addresses, and it never occurred to the old-timers that someone could encroach on them like that.
Service "Antispam"
Wow, night shifts! I loved and hated them, because it was 50/50: either planned work on equipment, where you take an active part, helping an engineer with sleepy brains and trembling hands, or silence with peace of mind. Subscribers are asleep, the equipment is working, nothing breaks, the attendant is relaxed.
The watch is going according to plan.
Once such a midnight calm is disturbed by a call to the office phone: hello, they are worried about Sberbank, your SIM card has stopped working with us, with which our notifications are sent.
After all, it was a long time ago, even before the introduction of IP connections to the SMS gateway. Therefore, in order for Sberbank to be able to send SMS from their famous number 900, they took the provided SIM card (most likely, not even one), stuck it into the GSM modem, and so they worked.
Okay, I accepted the problem and started digging. First of all, I check the status of the SIM card in the billing, it is blocked. What the hell - next to the red inscription "DO NOT BLOCK" and a link to the order of the general archdemon. Wow, that's really interesting.
I check the reason for the blockage, make my eyebrows a house and travel to the next office, where a girl from the fraud department is staring at the monitor.
“Lenochka,” I tell her, “why did you block Sberbank?”
She is in confusion: they say a complaint came that there is spam from number 900. Well, I blocked it, in the morning we would have figured it out.
And you say - subscriber complaints are ignored!
Simka was turned on back, of course.
A very scary story
When I first got a job, I and other newcomers were given something of a sightseeing tour. The equipment was shown: servers, conduits, inverters, fire extinguishing. They showed the base station, which stood in one of the test rooms, explaining that although the transmitters were turned on at minimum power, it was better not to enter the shielded door at this time. They explained about the device of the mobile network, about the main and backup power, about fault tolerance and about the fact that the network is designed to work even after an atomic bomb. I do not know if it was said for a catchphrase, or true, but it was deposited in my head.
And indeed: no matter what akhtung sometimes happens locally, the Volga voice network has always worked continuously. I am not a communications operator, but I am aware that the equipment (both base stations and client terminals) is designed for the maximum survival of the "voice". Is the power supply to the BS disconnected? It will reduce the power, switch to diesel generators / batteries, turn off the transmission of packet traffic, but the voice will go. Cut the cable? The base will switch to the radio channel, which is enough for voice. Lost your phone BS? It will increase the power and will feel the ether until it hooks on the tower (or until it drops the battery). Etc.
But one day a light blinked in the office and diesel generators rattled on the street. Everyone rushed to double-check their pieces of hardware: nothing critical happened in the IT part, but a puzzled "shit" came from the BS monitoring. And then: "guys, we have ALL bases laid down, check the connection."
We take out mobile phones - there is no signal.
Trying IP telephony - there is no way out to mobile communication.
There is no network. Generally. Nowhere.
Remembering the words about the atomic bombing, I subconsciously waited for a few seconds for the shock wave to hit us - for some reason, I did not think of another reason for the loss of the network. It was scary and curious at the same time: I somehow understood that I would not have time to do nothing. The rest of the guys were also taken aback, no one could understand anything.
There was no blast wave. After a five-second shock, they rushed to the wired city network phone available for such a case, starting to ring up regional offices. The city network, fortunately, worked, but in the regions it was confirmed: the whole of Samara is "dead", no hardware ping, no dialing.
Five minutes later, one of the power engineers brought the news: there was a bang somewhere at a power plant, at least the whole of Samara was de-energized, and possibly the region. Exhaled; and when there was a switch to reserve power, they even breathed in.
Another scary (but a bit silly) story
The biggest fuck-up in my memory happened during the next straight line with now zeroed. At that time, they were introducing a trick with sending questions by SMS, so they prepared for a surge in the load on the network in advance: everything was rechecked and prepared, and a whole week before day X, any work was prohibited, except for emergency ones. A similar protocol is used in any cases when an increased load is expected, for example, on holidays. And for the engineers on duty, it is like a day off, because when the equipment is not touched, nothing can happen to it, and even if it does, all the specialists, just in case, sit in the office in advance.
In general, we sit, listen to the national leader, and do not worry about anything.
From the side of the commutators, a quiet "F *** t" is heard.
I look at myself - really "f *** s": the campus network fell off.
In a second, everything dies at all (then there was still no meme about Natasha and cats, but it would be useful). The user segment of the network disappears, the technological one disappears. With increasing horror, we try to check what is left in working order, and after checking, we reach for the cabinet behind a hidden bottle of medicinal brandy: there are only voice calls left (as I said, they are tenacious!), Everything else is dead. There is no Internet - no subscriber GPRS, no optics, which is assigned to several sub-providers. SMS are not sent. Ass! We call the regions - they have a network, but they do not see Samara.
Within half an hour, the end of the world became almost materially tangible. Ten million people who suddenly broke everything and who cannot get through to the call center, because the voice terminals in the call center work via VOIP.
And this during the speech of the darkest ruler! Another victory of the State Department and Obama personally!
The techies on duty blew up from a low start and worked very clearly: within an hour, the network came to life.
Such a flight is not a regional, and not even a regional level, this should be reported to Moscow with all the details and the extradition of those responsible. Therefore, those who took part in the investigation were forbidden to tell the truth on pain of dismissal, and for the GO they wrote a report full of water and fog, according to which it somehow turned out that "it is itself, no one is to blame."
What happened in reality: one of the chiefs had the deadlines for implementations and bonuses for them broke off. And the chief of the chief was broken off, and so on; therefore they pressed on one of the new engineers, ordering him to make the required network connections "while everything is quiet." The engineer did not dare to object, or at least demand a written order: this was his first mistake. The second - he made a mistake when remotely configuring the tsiska, reaching record results on fakap in the shortest possible time.
As far as I know, nobody was punished.
The holiday comes to us
Holidays, as I mentioned, have always been special days for us. On such days, the load on the network increases sharply, the number of congratulatory calls and SMS goes off scale. I don’t know how it is now, with the development of Internet communication, but then in New Year alone, opsos shot a very significant foam on greeting calls.
Therefore, on New Year's Eve, engineers of all divisions were on duty in the office (and outside the office there were teams ready to squeeze through the snowdrifts to eliminate the accident at the base station in the village of Malye Drishchi). Billing specialists, hardware administrators, software plumbers, networkers, switches, service technicians, contractors support - each creature has a creature. And if the conditions allowed, then they hung out in our duty room, observing on our monitoring devices the bursts of traffic following the time zones throughout the Volga region.
Three or four times a night we celebrated the New Year, however, this was not so much festivity as nervous expectation: will the equipment withstand the overload, will some link in a complex technical chain break ...
Sasha, who was in charge of billing, was especially nervous. In principle, he always looked as if his whole life was spent on a bare nerve, because he had to rake all the good that was happening with the billing, be responsible for all the jambs, he was woken up more often than others at night; in general, I have no idea how and why he worked where he worked. Maybe he was paid a lot, or the family was being held hostage. But that night, I generally had a feeling that if you click on Sasha with a fingernail, then from the internal tension accumulated in him, he will crumble into dust. We have a broom for such an unpleasant case, but in the meantime, we are working our work, licking our lips at the cognac waiting in line.
Hour after hour, all the bursts of load passed, everyone began to recheck their systems. The switch turns pale: on one of the regional switches, all billing traffic has disappeared. And this is the data on all calls passed through the switch; they are written to a file, which chunks via FTP (kondovo, but reliably) are downloaded to BRT for billing.
The commutator, imagining how much turpentine enema he would be given for the loss of part of the New Year's proceeds in the whole region, already trembled. Turning to Sasha, he turned to the brilliant mister billing specialist in a voice full of exciting hope: “Sasha, look please, maybe BRT managed to deflate the tariffication? Oh, look, please! "
Sasha took a sip of cognac, ate it with a caviar sandwich, chewed slowly and, rolling his eyes with pleasure due to the fact that he didn't have the joint, answered: "I have already checked, there are no files ...".
(My wonderful proofreader asked about what happened to the poor switchboard later. Oh, his fate was terrible: he was sentenced to a week of duty on the first line of call center support, forbidden to swear. Brrrr!)
Throw a stone who is sinless
From these stories, one might get the impression that neither I personally, nor the other attendants on duty did not screw up. Nothing of the kind, fakapili, but somehow without an interesting epic and consequences. The work was considered suitable for yesterday's students without brains and experience, there is nothing to take from such an employee, to kick out for a joint - so new is not a fact that it will be smarter. But the engineers had a separate sport discipline to dump their jambs on the "duty room": they missed, didn't figure it out, they weren't notified in time, so punish them. "Dzhurka" perfectly mastered the science of dodging, it did not always work out, but everyone understood everything. Therefore, it flew in - but, as a rule, without serious consequences.
We analyze the next "flight" at the change shift.
For several years of work there, I can remember three cases when someone was fired from the department.
Once an engineer decided to drink beer on the night shift, and then the technical director came to the duty room. Sometimes he could come in like this in a simple way to say hello (sort of like he started with the attendants). Fired a guy with a can of beer, clicked on the phone, fired. We didn't drink more beer at night.
Another time the switchboard officer on duty missed some very terrible accident. I don't remember the details.
And the third time - at the end of my work there. Working conditions sagged a lot, there was a wild turnover and terrible overwork. Sometimes people worked for a day, then went to sleep for 12 hours and again went on daily watch. I myself worked like that, as long as my health allowed and it was paid; then they practically stopped paying for overtime (as a standard they promised compensation with time off when there was an opportunity - but everyone understood that no one would ever go for a walk), and they were kicked out on duty with almost threats. One engineer could not stand a cuckoo, he got up from his workplace in the middle of the shift and went home forever, on the way looking into the office of the head of the service, and sending him three letters. I remember a mailing list, in which this engineer was branded a fascist and a traitor after the fact, in every line it was read how the authorities were burned by such an act.
With regards to my personal fakups, one case is remembered for its unusualness. Again, night watch, everything is quiet, nothing happens. At the shift change, we check the monitoring: oops, the processing of data from the switches fell at night, the red light has been on for a long time. I looked at this signal all night - and did not perceive it or something. Despite the fact that it was one of the most obvious and visual monitoring, I still do not understand why I did not see it.
There were no excuses to stick on, the joint was clean and one hundred percent, an accident of the fifth category and a very likely dismissal. After twelve hours of night duty until lunchtime, I was forced to write explanatory notes. Since no one would have believed the truth, I had to come up with some kind of babble that because of the injury I had gobbled up the painkiller and fell asleep. The head of the service yelled at me in his office, in general, everything was going to be fired - but it resulted in a reprimand with a dismissal. Awards by that time in Mega had not been seen for several years, so I did not suffer any damage.
Remembering the episode with the arrival of the techdir: one night some redneck got into the duty room and started yelling that we were sitting unlocked (the duty room should not be locked in principle), that we were reindeer here, and that by morning he expects from all of us explanatory about all our jambs. That redneck was the head of the security service, and it hurt. Having screamed, the nachbez dumped into the darkness, and in the morning we asked our boss - they say, what to do? “Fuck him, send him,” he replied, and that was the end of the incident.
How I broke the department
In those days, Bashorg (then still bash.org.ru, and not what is there now) was a cult resource. Quotes there appeared almost a couple a month, and have YOUR OWN! QUOTE !!! ON THE BASE !!! was as cool as, say, your second-level domain in the year 2000. That Bashorg was somehow more IT-anime, although he was funny for everyone.
Every working morning of the youngest engineer (that is, mine) began with reading a bashorg - thirty seconds of laughter before twelve hours of suffering.
One day a colleague asked me what I was giggling about. I showed him what. He sent out the link to the department.
Work stopped for a couple of days: to my surprise, none of my colleagues knew about bash until that moment. There was laughter in the duty room: "Ah-haha-haha, patch KDE, ahaha-haha!" "Yogo-ho-ho, drown crowbars in mercury, runner!" The working day was lost, on the other hand, life was then extended to themselves notably.
Bonus for those who read it
Remember, in bearded times there was such a popular anecdote “I see two C disks in Norton, I think - why do I need two? Well, I erased one! " It is very reminiscent of one of my favorite stories, which is not told by me, but to me. And every time it's funny, like the first one:
18+, but you can't erase words from a song
P.S
These stories are a processed compilation of some posts from my TG channel. Sometimes such game slips there; I am not hinting at anything, but I will still leave a reference .
Have a good fad-free Friday everyone!