After joining the Rostelecom-DPC team, DataLine specialists will also operate this data center. You have probably already heard something about "Udomlya". Today we decided to tell you in detail how everything works there.
Industrial landscapes: 32,000 m² data center and nuclear power plant in the background. Udomlya sample of spring 2019.
More than 40 photos of data center engineering systems with a detailed description were collected under the cut. A pleasant surprise awaits those who make it to the end.
About logistics
The data center is located in the Tver region. From Moscow to Udomlya the road will take about three hours: 1 hour 45 minutes by Sapsan to Vyshny Volochek station, and from there, upon prior request, a shuttle will meet you and take you to the data center. From St. Petersburg to Vyshny Volochok a little longer - 2 hours 20 minutes.By car, you can get from Moscow in 4.5 hours, from St. Petersburg - in 5.
Yes, for the sake of a couple of units, you probably won't want to go here. But if you need a new home for dozens of racks, then it's worth taking a closer look. There is enough space and electricity, even if you want to double that amount at any time. In Moscow, where, in our experience, data centers are booked even at the construction stage, this trick will not always work.
In addition, the location of the data center between Moscow and St. Petersburg can be used for geo-redundancy. If the main capacities are in Moscow or St. Petersburg, then the reserve site will fit well here.
A smart hands team will assist with all standard operations on site. They will receive, unpack and install equipment in racks, connect it to power and network, and provide remote access to equipment. In the event of a malfunction, they will help with diagnostics and replace the failed components.
The first stage of the data center includes 4 computer rooms, or modules, with 205 racks each. On the first floor there are 2 machine rooms and an energy center, on the second there are two more halls and a cold center. Let's go see how everything works here.
Physical security
The data center occupies a dedicated area, which cannot be entered without a pass and an identity document. Those who arrive by car also receive a transport pass and only then can enter the data center. For those who have everything in order with a pass, the data center is open 24x7.
The first 24-hour security post is the entrance to the territory.
We go further and get to the checkpoint directly at the entrance to the data center.
Security officers not only meet customers and issue passes, but also monitor the video wall around the clock, where images of all the internal premises of the data center and adjacent territories are displayed.
Power supply
Electricity starts its way to the data center from a nuclear power plant. The data center is supplied with 10 kV for 6 step-down transformers. Further - already 0.4 kV by two independent routes go to the low voltage switchgear (LVSN). Then, through the DIBP, power is supplied to IT and engineering equipment. Two independent inputs are suitable for the rack, that is, 2N redundancy. We will tell you more about how everything works in terms of power supply in a separate article.
The path of electricity in the data center "Udomlya"
Power buses through which electricity comes from the low-voltage switchgear to the power panels DIBP Rows of low-voltage switchgear
Despite the fact that a nuclear power plant is close by, in any reliable data center the main power supply is considered to be guaranteed. In our data centers, as you know, DGUs are responsible for it, but here they use dynamic UPSs (DIBPs). They also provide uninterrupted power supply. RIBPs are N + 1 redundant.
DIBP brand Euro-Diesel (Kinolt) with a capacity of 2 MW. They roar in such a way that it is better not to go there without earplugs
And this is how it works. DIBP is a combination of three main components: a diesel engine, a synchronous electric machine and a kinetic energy accumulator with a rotor. They are all fixed to the main shaft.
The electric machine can operate in the mode of an electric motor and a generator. When the DIBP is supplied normally from the city, the electric machine is an electric motor that turns the rotor and stores kinetic energy in the battery.
Gray block in the foreground - synchronous machine DIBP
Diesel engine DIBP
If the city power fails , the electric machine switches to generator mode. Thanks to the accumulated kinetic energy, the rotor makes the main shaft of the DIBP rotate, the electric machine continues to operate without city power supply and the output voltage does not disappear. This ensures uninterrupted power supply in the data center. At the same time, the DIBP control system gives a signal to start the diesel engine. The same kinetic energy of the rotor starts the diesel and helps it to reach the operating frequency. The rotor keeps revving up to one minute, and this is enough for the diesel engine to take over. After starting, the diesel engine rotates the main shaft, and through it - and the electric machine (here is a visual videoswitching DIBP from one mode to another).
As a result, the power in the racks is not lost even for a second.
Each diesel generator tank is designed for 3 hours. The data center also has its own fuel storage for 80 tons, which will keep the entire load of the data center for 24 hours. In case of very hypothetical blackouts (a nuclear power plant nearby will not allow this), there are contracts with several contractors who will promptly deliver diesel fuel to the site on a call. In general, everything is as it should be.
Every week, DIBPs self-test and start the diesel. Once a month, tests are carried out with a short-term disconnection of the city network.
Control panel DIBP
Room ShGP and ShbP
"Magistrali" and "decoupling" of power cables
Machine rooms
Each module is located in a containment area, in a special box. These additional walls and roof protect the machine room from dust, water and fire. When accepting a data center, the containment area is traditionally spilled with water to check for leaks.
The roof of the building and its own roof of the turbine hall with drainage pipes
Water entering the roof of the containment area goes through the gutters into the drainage pipeline.
Each hall is ready to receive 205 racks with an average power of 5 kW.
The arrangement of equipment in the hall is organized according to the scheme of cold and hot corridors.
Bolid early fire detection and gas extinguishing systems are installed along the ceiling.
Smoke detectors are also located under the raised floor. It is enough to trigger any two sensors, and the fire alarm siren will sound, but we will talk about this a little later.
Immediately along the rows of air conditioners, there are tape leakage sensors.
Each corridor between the counters is "shot" by video surveillance cameras.
If desired, the racks can be placed behind a special fence (cage) and additional cameras, ACS and motion sensors, volume sensors, etc. can be installed on it.
Cold supply
The Udomlya data center uses an ethylene glycol chiller circuit. There are air conditioners in the machine rooms, chillers on the roof, and on the second floor there is a refrigeration center with pipelines, an automation and control system, pumps, storage tanks, etc.There are 12 air conditioners in each room, half of them with steam humidifiers. N + 1 redundancy scheme.
The cold aisle maintains a temperature of 21-25 ° C and a humidity of 40-60%.
Precision air conditioners Stulz CyberAir 3PRO CW
There are two rings around each computer room: a “cold” line that supplies cooled ethylene glycol to the air conditioner, and a “hot” line that removes heated glycol from the air conditioners to the chillers. If we open the raised floor in the corridor, we will see drops into the machine rooms from the cooling system.
The ethylene glycol route is as follows: from the air conditioner, heated ethylene glycol enters first into the return line around the machine room, and then into the general ring. Then ethylene glycol goes to the pump and then to the chiller, where it is cooled to 10 ° C. After the chiller, ethylene glycol returns to the air conditioner through the common ring supply line, storage tanks and a ring around the module.
Data center
refrigeration scheme This is how a cold center looks like, through which 100 m 3 of ethylene glycol passes.
Gray tanks are expansion tanks. Heated ethylene glycol passes through them on the way to the chiller. In the summer, ethylene glycol expands and requires additional space.
These impressive capacities are storage tanks, 5 m 3 eachevery. They provide uninterrupted cooling of the data center in the event of a chiller failure.
Cooled ethylene glycol from the tanks is supplied to the system, and this allows maintaining the air temperature at the outlet of the air conditioner at 19 ° C for 5 minutes. Even if outside the window is +40 ° C.
Cooling system pumps
Mesh pocket filters and separator tanks for cleaning ethylene glycol from mechanical particles and air.
The thin red line on the floor under the pipes are tape leak sensors. They go along the entire perimeter of the cold center.
If any of the pipes leaks, the ethylene glycol will go through the drainage system and enter a special tank in the water treatment room. There are also two tanks with "spare" ethylene glycol for feeding the refrigeration system in case of large leaks.
And about chillers. There are 5 chillers on the roof with N + 1 redundancy. Every day, the automation determines, depending on the operating time, which chiller to put into reserve.
Chillers of the Stulz CyberCool 2 brand with a capacity of 1096 kW
Chillers support three modes:
- compressor - from 12 ° C;
- mixed - at 0–12 ° C;
- free cooling - from 0 and below. This mode involves cooling the ethylene glycol using the fans, not the compressor.
Fire safety
The data center has two gas fire extinguishing stations. Each has two batteries of 11 cylinders: the first is the main one, the second is the reserve.The fire protection system of the data center is tied to the server of the Kalinin NPP, and, if necessary, the station's own fire service will arrive on site in minutes.
Pictured is a fire alarm system and an emergency exit button in the engine room. The latter is needed if, during a fire alarm, the doors were not unlocked for some reason: it interrupts the power supply of the electric lock.
Telecom
Two Rostelecom highways come to the data center by independent routes. Each of the DWDM systems has a capacity of 8 Terabits.The data center has two telecom inputs, which are located at a distance of more than 25 meters from each other.
Operators Raskom, Telia Carrier Russia, Consist are also present at the site; DataLine will appear in the near future.
From Udomlya you can build a canal to Moscow, St. Petersburg or anywhere in Russia and the world.
Monitoring
Duty engineers are on duty around the clock in the monitoring center.All information on engineering systems is received here: climatic conditions in the hall, condition of inputs, DIBP, etc.
Every two hours, the duty officers make a round of all infrastructure facilities to inspect the condition of engineering and IT equipment.
Supporting infrastructure
An unloading zone is provided for the delivery of equipment to the data center.
Unloading area from the inside.
If your gym is on the second floor, then this hydraulic forklift will take any equipment there.
Lockers for storing client tools, and more.
A little about everyday life
For permanent staff, you can rent equipped work places in the office part. If you come over from time to time, you can stay in a temporary hotel with all the amenities right on the territory of the data center.The office area also has a dining room and a kitchen.
And there is also awesome nature all around with forests, lakes, rivers, fishing and other outdoor activities. Come to visit my place.
As promised, a nice bonus for those who made it to the end. For the first six months, rent of a rack-place in Udomla with a supplied power of 5 kW will be free of charge. Pay only for the electricity actually consumed. Send your application to request@dtln.ru .