This time we decided to combine these favorite things and make a competition for the best sysadmin tales: about work, clients and funny cases from practice.
Mmmmm, sysadmin-tech-pod-bike-beer For
everyone who tells a funny story in the comments, we will give a can of our signature dark ale DukeNukem , and for the winner, whose story gets the most pluses, we will send a whole box of beer .
For inspiration and the right mood, we share our favorite sysadmin stories: there are almost no new stories, but a lot of nostalgia, good humor and even a couple of still unsolved mysteries. Go?
I used to work in the IT department at Samsung. One of the employees needed to reset the password, I changed the password to "Samsung1" and told him about it. They called me back two minutes later and said that the password did not work. I dropped it again but it didn't work again. I thought that the problem was from the user side and decided to go to him.
Imagine my surprise when I saw that he was typing the password incorrectly! He was typing "Semsung1".
“I suggest! This is the company you work for, and its name is written on the monitors you sit in front of. "
Peter J.
This was in early 2007. Our college was building a data center in a converted classroom, with little regard for climate control, and then there was a conventional split air conditioner. Our college was located in the Southwest Mojave Desert, Southern California. It "saved money" every weekend by turning off all air conditioning throughout the campus, including the data center , so our servers chugged in 50-60 degree heat until Monday morning. In March 2007 we reached out to a backup supplier as we had a Quantum LTO1 tape drive (with only one cassette slot) and it was buggy due to the heat in our EMC Clarion CX400 SAN... We needed a reliable drive that would survive in our conditions. But the existing solution could not make copies of open files, so it took us a week to collect the full set of data (due to the slow drive). We asked for money for a backup SAN node for backup insurance in case of a fall, but we were refused.
Instead, we got the money to run a third-party backup company on tight deadlines. The departure of their specialists was planned on July 17, 2007 at 8:45 am so that they could make a full backup for us. In line with Murphy's Law, our SAN went down the night before the planned arrival of the backup specialists, and one of the 900 GB LUNs was lost forever.
Long live planning!
Justin J.
We had a heads of department meeting at our company to discuss how to use the 911 computer system. Instructions were sent out Friday morning to begin training on how to use it on Monday.
Two users, not really delving into the text of the letter, activated the system according to the instructions as soon as they received the letter, pointing out the naughty among the police officers working in 911.
Zach D.
Like most system administrators, I started with work in the support service.
It once took me over 20 minutes to explain to a new worker how to press CTRL-ALT-DEL to log into the computer. She simply could not understand the concept of simultaneously pressing three keys at once
Conrad George .
Somehow, a worker serving communications in the false ceiling, where our network cables were laid, accidentally cut them. When our engineers arrived at the site to repair the cliff, the worker said that everything was in order, and he fixed it himself.
He connected the pieces of twisted pairs with twisted wire caps.
Wallace F.
One day I almost went crazy trying to figure out where the failure was in the power supply through the raised floor. Suddenly we heard a voice from the next room: “O-la-la! You poured my entire bottle of water onto the outlet! ”.
We ran there and saw a socket covered with a puddle of water ...
Jorge B.
About 10-25 years ago I worked in support of a provider, at some point a woman called me, who immediately admitted that she did not know how to use a computer, but she really needed to run one program. She lived only a couple of minutes away from me, and I decided to just go to her house.
When I arrived at the scene, I told her that first she needed to open the Start menu. She asked me: “What is the Start Menu?” ... Gradually I explained everything to her and found the application that she needed. When I was about to leave, she said that she knew another person whom I could help. It turned out that she had a little daughter, and she had heard that it was possible to turn the mouse cursor into a dinosaur. It was easy, I just changed the Windows XP cursor theme, but the girl was happy.
Back at the office, I filled out a ticket: "Helped the user find application 'X' and turned the mouse cursor into a dinosaur."
Andy S.
I remember that I had a salamander at work in an aquarium, which all the time managed to get out of it, although the lid was closed. Once, she got out when no one was there and disappeared.
After a while, my computer refused to start due to problems with the power supply. When I opened the case, I saw a stick stuck in the fan.
Only it wasn't a wand! ..
Rob H.
We had a client who sent us a picture of his cracked screen ...
... taken with Print Screen.
Lim R.
When I worked in tech support, I was paid to help an elderly person. When we figured everything out, I asked him to close all the windows so that we could do it all over again. He said "Okay, I'll be back soon." Ten minutes later he went back to the phone and said: "Well, I closed all the windows, and now what?"
Ryan G.
One day, I walked away from my work computer, leaving it on and unlocked with a password screen saver. This was never required, because the team was less than ten people, everyone worked together for several years and knew each other quite well. When I returned, I saw that almost the entire contact list was blinking in my ICQ, everyone was writing something to me, including those with whom we spoke very rarely and not even every month.
It turned out that for some reason it was on that day that an unknown substance hit one colleague in the head, and he decided to joke, sat down at my CMOS and wrote to my ICQ: “I shook my head and my hair got stuck in the keyboard”, and then sent it out with one button on all my contacts (thanks to ICQ, then it could be done literally in one click).
I went through a few unpleasant minutes explaining that this was a stupid joke from a colleague. But he no longer greeted him ... And the computer was blocked.
Konstantin V.
There was a case when an employee got a job in our office who, how to put it more delicately, neglected personal hygiene. Everyone got sick of his ambergris, but they were shy to say frankly. As a result, we solved the problem in a rather confused way, but “in an admin style”.
On a Macintosh computer, you can reset the root user password using a special option on the installation disc. We booted his Mac from this disk, reset the root password, added another user, and gave him the password we know. Then they logged in as root, opened the file with the password hash of that employee, copied it into a separate file, and inserted the password hash of the new user in its place. This allowed us to log into the smelly employee account using the temporary user password. We changed the background on the desktop, putting a picture with a baby and a large caption: “Even small children know to take a shower every morning and use deodorant!”.
After that, logged in again as root, copied back the hash of his password from the saved file and pasted it back, and deleted the temporary user.
The next morning, we looked forward to his reaction and were rewarded with a loud exclamation of amazement as he booted up his computer. Our efforts were not in vain, the guy began to take a shower and use an antiperspirant!
Konstantin V.
A few years ago, while working at the MIT Artificial Intelligence Laboratory (actually a lot, because a computer from the 60s is described), I was rummaging through the cabinets where the PDP-10 mainframe was located, and I noticed a small switch attached to frame of one cabinet. Obviously it was a joke that one of the local "hackers" added.
In fact, there is a rule: “Do not touch the switch if you do not know what it is doing, because it can cause your computer to crash. But this one was marked in a completely stupid way, it had two positions, and the words "magic" and "more magic" were scrawled on the metal case with a pencil. Then the toggle switch was in a "more magical" position.
I called another employee to see this miracle. But he, too, had never noticed it before. Upon closer examination, it turned out that only one wire was connected to the switch, which was hidden in a maze inside the computer. It makes sense that a switch cannot do anything if there are no wires connected to it on both sides, but this switch only had one wire.
It is clear that this switch could only be someone's stupid joke. After making sure it shouldn't work, we clicked it. The computer broke down instantly ...
Imagine our utter amazement! We chalked it up as a coincidence, but nevertheless returned the switch to a "more magical" position before starting the computer again and did not touch it again.
A year later, I told this story to another hacker, David Moon. He laughed at me and expressed doubts about my sanity, accusing me of believing in the supernatural power of this switch. To prove my point, I showed him this "magic switch", still screwed to the cabinet with only one wire connected to it, left in the "more magic" position. We examined the switch closely and found that the other end of the wire was connected to a ground pin. It clearly looked doubly useless, not only did it not work electrically, but it was also connected to a pin that, in any case, could not affect anything. So we flipped the switch.
The computer crashed again ...
This time we ran after Richard Greenblatt, a seasoned MIT hacker who was working nearby. He had never noticed this switch before, but when he examined it, he too came to the conclusion that it was useless and decisively bit off the wire. Then we restarted the computer, and since then it has worked without problems ...
We still do not know how the switch broke the car. The theory is that the circuit near the ground pin was at its limit, and the flick of the switch changed the capacitance enough to break the circuit when pulses of a millionth of a second passed through it.
This switch is still in my basement. It might be silly, but I keep him in the "more magic" position.
Added in 1994.
Since then, another explanation for this story has been proposed.
The switch body was metal, and the unconnected side of the switch was screwed to the computer cabinet frame (usually the case is connected to a separate ground terminal, but there are exceptions). Thus, the switch body was connected to the computer case, which is presumably also grounded. The circuit ground inside the machine does not necessarily have the same potential as the chassis ground, so a flick of the switch connects the computer circuit ground to chassis ground, causing a voltage drop / surge that reboots the machine. This was probably discovered by someone who, through their own bitter experience, figured out that there was a potential difference between the two, and then attached a switch as a joke.
MIT hacker folklore
In one office where I once worked, there was a friend.
The intercom receiver in that office was hung up at the door, he sat closest to the door, but he was lazy to run to every call.
How did he solve this problem? He connected the contacts of the intercom button that opens the door through the relay to the floppy drive, shared access to the A: drive, brought out the "Open door" labels to everyone.
After that, any employee could open the door over the local network.
Folklore
In one closed military research institute, they were taking an inventory and stumbled upon a description of equipment that no one could identify. In the “Book of Accounts” it was recorded as “50 heh juice”. Since the equipment is registered, it must be found and checked, especially since it was valued at a noticeable amount of “conventional units”. The people racked their brains for a long time until one researcher came up with a simple solution to this puzzle. A very elderly woman, without higher education, did not know any languages, except Russian, was engaged in accounting. Therefore, the CD-ROM drive, 50-speed, she wrote down "as I understood", that is, "50 heh juice".
Konstantin V.
Once, when I was working in a small firm, they began to carry out an inventory in our office and everyone was asked to leave, because it was necessary to stick inventory numbers on all the furniture and equipment, and we interfered with them. We decided to combine business with pleasure and went to dinner.
Returning from lunch, we found that not a single mouse worked. The reason became apparent when I turned mine upside down. Well, where else can you stick the inventory number on the mouse? Of course, across the hole with the ball!
Konstantin V.
One company had a Xerox MFP, and this device had a useful feature - the ability to send scanned documents to email. For this, oddly enough, he needs an account on the mail server, because he sends via SMTP. The mail server was changed, some of the accounts were migrated, and some were simply re-created. In the process of creating a new account for MFPs, we climbed to the old server to clarify login and drew attention to the folder with incoming letters. The hellish aggregate did not take mail from the server, so about fifty megabytes of it accumulated.
It turned out that the people are in active correspondence with the IFI, they write "Thank you!" and "Thank you for the documents!" Especially memorable was the message of one employee: “Thank you, I received a scan of the contract, I'm waiting for Appendix 2 from you”
Folklore
There was a case when students studying artificial intelligence were given the task to write a self-learning bot for a 3D shooter. The students coped with the task and launched bots on the LAN so that they got the hang of killing each other overnight. In the morning we entered the grid and began testing our bots. It turned out that living players were taken out in the very first seconds, and there was not a single chance to survive. When they began to look at the logs, it turned out that the levels in the game were made carelessly, there were gaps between the textures that were not noticeable to the players, but were discovered by bots, and allowed them to hide inside the walls, passing into any room and kill players, preventing them discover yourself.
Folklore
Friend works in a warehouse of computer components.
They received an order from the SNB for 100 RAM modules. Can you imagine a hundred RAMs? They were packed compactly in a box of 60 by 40 centimeters. A representative of the SNB arrives, they have a dialogue with the manager, after which the manager crawls into the chief's office in convulsions, hands him a box with a memory and asks him to leave himself.
The chef walks out onto the porch. There is the aforementioned representative who points to the KAMAZ ... WITH A TRAILER! .. and gives out: “I drove the car here, if that’s not enough, then we WILL MAKE TWO WALKERS! ..”
The chief was not at a loss, hands the representative a box and asks to hold it. He signs the documents himself, stands and smiles. The representative starts to get nervous: "Where is the product, people are waiting for me, it is NECESSARY TO DOWNLOAD!"
Operations: "He's in your hand."
Folklore
My brother lives in a neighboring state and has already received requests to help him with a computer on the principle of “tyzhprogrammer”. Some of the calls to family support were as painful as dental treatment. It seemed to him that viruses and trojans got into his computer, after which he called me. He could keep me on the phone for hours, forcing me to fix his computer.
I once solved this problem once and for all.
- You need to go to Google
- How can I do this?
- Open your browser, type google and press Enter.
- What kind of browser? How can i do this?
- This is where you watch anything on the Internet ...
"Oh ... Did he just open the Word?" Was that how it should have been?
And in this spirit, the conversation continues for more than an hour. Suddenly, a wild idea comes to my mind. Sequentially, I lead him to download and install VNC. Then I force him to visit several computer security sites in order to scare him with the consequences of reckless behavior on the Internet, lecturing on this topic all the time.
Me: The Internet is really a dangerous place. You must be careful. Very careful. There are hackers who want to hack into your computer. If they infect your computer, they will load hacker programs onto it that will do all sorts of unpleasant things. They can steal your passwords or access your bank and credit cards!
At the same time, I "accidentally" open and close windows through VNC, go to different sites.
Me: They can even take over your computer, turning it into a bot that THEY will control. I have heard stories of people being approached by the FBI because their computer was seized by hackers and used to attack the White House or the Pentagon. The poor people were not even aware that they had been hacked ...
At the same time, I open and close more and more windows.
He: Hey! I'm not sure what's going on, but my computer is behaving strangely. He's trying to go online on his own!
Me: Oh-oh. This is bad! Are you sure you're not typing too fast? Run your antivirus just in case. If you cannot do this, then it is very bad.
He tries, but I interfere with the movement of his mouse. Finally, I let him run the program, and as soon as it opened, I immediately closed it.
He: I'm starting to worry. I do not know what's going on! I can't do anything on my computer !!!
And I open the command line and run the commands: IPCONFIG, netstat, tracert whitehouse.gov, etc.
Me: Wow - what did you do ?! Looks very serious. It looks like your computer has been attacked ... Maybe a hacker is trying to get into it. You may need to turn it off. Looks like they are trying to get to your bank or worse ...
Then I start typing: Naturally, I used his real home address.
Access: Pentagon Classification TS5
PASSWORD REQUIRED: ******
FAILURE 1 NOTED: Alert sequence
Delta Alpha Charlie initiated : IP TRACKING LEVEL 1 PRIORITY INITIATED
WARNING - PASSWORD FAILURE 2
ALERT TEAM NOTIFIED: FINAL PASSWORD FAILURE
Security Breach detected. Tracking completed. IP ADDRESS confirmed. Response Team dispatched. Address Confirmed: Illegal security breach confirmed. Trace complete. Team dispatched - 2123 Main street: Perpetrator confirmation - John Smith.
I kept scaring him by typing the above, but he started interrupting me
- Oh man, this is really very bad ... They are breaking into the Pentagon. What should I do. What to do?!
I intend to act too slowly for his convenience.
- Oh, you have to run the antivirus.
- HELP! They know who I am. What should I do?!
- Well, I would advise you to turn off your computer, put it in the box and return it to the store!
Finally, I could no longer hold back my laughter and burst out, and then explained to him what I was doing. For some reason it didn’t strike him as funny as it did to me. But since then I have not been called as "tyzhprogrammer"!
Commentary on the article
Probably the worst story I can remember, besides the usual headache from users, was about how I had to conduct research on the activity of one user on the network. This person has been shown to generate very high traffic in the afternoon and slow down the entire department. After a few minutes of studying the logs, I noticed the name of one of his frequently visited sites. I don't remember the specific address, but it was a site dedicated to BIG women. I passed this information on to my supervisor, who passed it on, and soon I received a response from my boss's boss that they needed specific examples of what types of sites he visits, with screenshots.
The rest of the day I will never forget ... If I had bleach on hand, I would have poured it into my eye as soon as he entered that site. Apparently this "gentleman" spent 80% of his time looking through highly questionable materials in the office.
After collecting screenshots for two hours, I contacted my boss's boss to show him my investigation ...
He later told me that they talked to that employee and he quit.
110! = 220
This was my first year in IT and I had to change the computer that controlled the huge wood dryer. I was not very many years old, but then many computers were running DOS.
I called a furnace company in Germany, found someone who spoke English, got the specifications, assembled the gear, and drove off to a customer 200 miles away. After backing up the previous system (to floppy disks!), I pulled out the huge NIC from the old computer, inserted it into the new one, plugged in the power to the UPS, and pressed the power button.
BOOM
Blue smoke from a burnt-out power supply unit ... With a white face, I called my boss, told him about everything I had done and what it had led to.
“Don't worry, shield hepenes, just go to a computer store half an hour away, buy a new power supply and try again. Yes, and remove that monstrous NIC before you turn on the power again! "
Quickly driving to the store and back, I come back with a new power supply, replace it, remove the network card and press the power button again.
BOOM
More blue smoke, and my face turned even more white. I call my boss back, "There is something wrong with your setup again." After a few minutes, having discussed everything with him on the phone, we came to the conclusion that the computer was assembled correctly, and the problem was something else.
- Where did you plug in the power cord?
- To the UPS
- Hmmm ... And whose production is it? You do not know? Is it written in German?
- Yes
- Then go back to the store, buy a new power supply and put the red switch on the back panel for 220V. Call me back when the oven is working again.
This is how it turned out that in Germany the equipment works at 220V. I think the 19-year-old boy from Canada, which I was then, might not have known this.
From the comments to the article
When I was just starting my career in the IT field, and I was told that one VIP user could not connect to the VPN in any way, they asked for help. Everything was fine on his work computer, so I asked the user to personally show me what he was doing. He took out his RSA token, entered the code from it and logged in.
However, this is not the first time this user has complained about this issue, and I was not the first to try to help him. I asked him to tell you step by step what he does before he cannot log on to the network.
“First, I take out my laptop, plug in the network cable, connect to the Internet from home. Then I launch the VPN client, pull out this piece of code and type it in ... ”
It turned out that he wrote the code from the RSA token before leaving home, and left the token at work, because this thing was quite expensive, and it did not wanted to lose her. I explained that the number changes all the time and he needs to carry it with him at all times. The VPN issue has been resolved.
Commentary on the article
One of the companies we supported in my previous job was the university police. And then one day my colleague got a call from the chief of police and said that something was wrong with the Ethernet connection on his brand new laptop. We drove up quickly to have a look at his laptop. The problem was obvious!
Somehow (I still don’t know how he had the strength) the chief of police inserted an Ethernet cable into the RJ-11 modem connector. And I inserted it so hard that it cracked the case around the connector.
Commentary on the article
Several years ago a user called me and complained that he had problems with printing on the printer in his office. I went into the "Printers" setting of WindowsXP and saw that the printer was not there.
I went over to his desk to check the connection, but found that his workstation was not connected to the printer.
“Why do I need this?”, - he asked, - “It's a LASER printer!”
Commentary on the article
This one was in the mid 90s, in the good old days of DOS. I worked for a very large bank and provided support to users all over the state. Employee training there was frowned upon and not budgeted, so we had buildings filled with people trying to figure out how to make their computers work. One lady worked in the tax department, she was 190 years old, and she only used the computer because she had to (again, without training!). I configured the system for it and made it as user-friendly as possible in DOS. She had a simple menu with three options, one of which launched a tax program.
One day she called me very irritated: "There is nothing on my screen!" With no diagnostic tools at my disposal other than my imagination, I began to use her as my eyes.
- Look at the monitor, the thing that looks like a TV, is the green LED on?
- DK
- Look to the right of the TV, 2 "down and 1" to the right.
- There is a green light.
I could hardly refrain from commenting, and we repeated this process for a computer, with similar difficulties in translating from “IT” language into one that she understands. After figuring out that everything was indeed on and working correctly, I turned my attention back to the monitor. Could he have burned out?
- Okay, I want you to watch TV. Do you see anything that looks like glowing orange-yellow letters somewhere on the glass part of the TV?
- Oh yeah, it says 'C: ->'
I thought to explain to her that she was looking at the DOS prompt, but decided to skip such details.
- Type MENU, hit Enter and tell me what's going on
- Nothing.
- Are there new bright symbols?
- Yes: “Bad command or filename C: ->”.
Several attempts to launch the “Menu” application were unsuccessful, so I thought that the application might be corrupted. After moaning internally about what would happen next, I said:
- Let's try to run the Tax application without a menu. I want you to type: “CD (space) TAX”, hit Enter and tell me what happened.
- Nothing! There is nothing on the screen!
With growing excitement, I ask:
- LOOK 1/2 inch from where your eyes stopped and read the damn glowing symbols!
- “Bad command or file name".
I was at a loss ... Then I thought for a few more minutes, and an idea came to my mind.
- Spell out what you just typed
- “SEEDEESPACETA X"
It was at that point in my career that I realized what the mute button was really for. When I turned off my phone, I hit my head on the table with a scream!
Commentary on the article
I had a lot of things that I don't remember anymore, but there is one case when one girl said: “On the monitor of my computer a duck”.
Tier 1 support remotely scanned her machine and found no malware or applications that could be causing this behavior. They were about to send me a computer for me to physically check, but before giving me her car, she said she figured it out.
I was very curious as to what caused this problem, so I asked what happened
She replied that the fan had pasted a duck sticker on her monitor, and she just didn't notice it.
Commentary on Article
One evening we received a call from a small business about 70 miles away. Before that, we sold them a new printer, all they had to do was install the drivers and connect the device. Nothing special.
But they called me and said that the CD does not fit in the drive. Reading the sticker on the disc, they said they understood the problem. The disc said it was for MAC, and I had to send them a Windows disc, since the MAC CD would not fit into the drive. The matter was urgent and I went to their office.
I realized the problem as soon as I entered the office because there were small rubber feet on the top of the computer. Decision? Flip the computer over and now the CD is great!
Commentary on the article
There was a case, a woman called technical support and said that every day she saves her database on a floppy disk, and the next day the floppy disk is always empty.
After several days we asked her to check the integrity of the data in the backup, and everything was fine, but the next day everything disappeared again, we sent a technician to her.
What did he see?
After saving the data to disk 5 1/2, she attached it to her metal cabinet using an industrial magnet.
Commentary on the article
At one point, I urgently needed to lay four dozen twisted pair wires over a distance of one hundred meters. Most of the distance passed over the suspended ceiling at a height of three meters. Realizing that for my only admin shoulders in the office, the solution of the problem in an acceptable time frame is unbearable, I began to torment the chief for attracting third-party installers, and he reluctantly agreed. I developed the terms of reference, sent it to different offices and waited.
One of the proposals stood out from the crowd. The guys claimed that they would complete the work in just one day, and three times cheaper than any of the competitors! Just in case, I called back and asked if they understood the task correctly. They replied that they understood perfectly well, and the speed and cheapness are explained by the fact that a cable layer is used in the work. I was intrigued. I dealt with cable layers in the army and had no idea how this device could be adapted to laying a twisted pair cable under a false ceiling.
On the appointed day, only two fitters arrived with ladders and tools. Watching the unloading process, I was looking forward to a miracle machine that could automate the process. The unit did not wait so long, but a medium-sized dachshund was unloaded from the minibus last. Before I had time to be surprised at the appearance of the animal, they explained to me that this was the mysterious cable-laying machine.
Included with the dachshund was a special suit with a handle along the back for carrying, a harness on the head with an LED flashlight and a harness. The cable was laid by attaching it to a harness and launching a dachshund in a hard-to-reach place. A well-trained animal rushed briskly at the sound of a voice or the light of a lantern, dragging a wire behind it. The ceiling could withstand the dog, but the power was enough for fifty meters.
The speed of installation in long and hard-to-reach areas was fantastic. Dachshund briskly ran to one side along the top, and back along the corridor, frightening the female part of the staff. Even the emergency in the form of a dropped ceiling panel did not affect her mood: the pulled wire played the role of a safety cable, and the dog smoothly sank to the floor. The guys said that the cable-layer was accustomed to such situations.
Laughter laughter, but the dachshund was in all seriousness among the guys on the balance sheets of the companies. The documents were shown to me. The friendly team really completed the work in less than a day, and without first unwinding the cable and, accordingly, without scraps. True, the work of our office completely stopped: the entire population came running to look at the work of the cable-laying machine.
Folklore
We put beer in the fridge and are waiting for your stories
The winner will be determined in a week, on March 10.
UPD
Pot, don't boil it.
Acceptance of new comments is suspended, but you can vote for the current ones - to help win a box of beer right away
We remind you of the rules:
- Your favorite story, personal, told by a friend over a bottle of beer, read sometime
- A jar of Duke Nukem for every story
- Duke Nukem 's crate of ale for the most popular story in the comments
