DPC "Tushino" is a commercial retail half-megawatt data center for everyone and everything. The client can not only rent the already installed equipment, but also place his own there, including non-standard devices such as servers in conventional cases for desktop PCs, mining farms or artificial intelligence systems. Simply put, these are the most diverse popular tasks that are most in demand by domestic business of varying degrees of magnitude. This is what makes him interesting. In this post you will not find exclusive technical solutions and a flight of engineering thought. We will talk about standard problems and solutions. That is, about what 90% of specialists employ 90% of their working time.
Tier - the more the better?
Fault tolerance of the Tushino data center complies with the Tier II level. In essence, this means that the data center is located in a normal prepared room, redundant power supplies are used and there are excess system resources.
However, contrary to the common misconception, Tier levels do not characterize the "toughness" of the data center at all, but the degree to which it meets current business objectives. And among them there are many for whom high fault tolerance is either insignificant, or not so important as to overpay 20-25 thousand rubles a year for it, which in a crisis can be very painful for the customer.
Where did this amount come from? It is it that makes the difference between the prices for information placement in data centers of Tier II and Tier III levels in terms of one server. The more data you have, the greater the potential savings.
What tasks do you mean? For example, storing backups or mining cryptocurrency. In these cases, a Tier II allowed server downtime is cheaper than Tier III.
Practice shows that in most cases, savings are more important than increased fault tolerance. There are only five Tier III data centers in Moscow. And there are no fully certified Tier IV at all.
How is the power supply system of the Tushino data center arranged?
Requirements for the power supply system of the Tushino data center complies with the conditions of the Tier II level. These are redundancy of power supply lines according to the N + 1 scheme, redundancy of uninterruptible power supplies according to the N + 1 scheme and redundancy of the diesel generator set according to the N.N + 1 scheme in this case means a scheme with one reserve element, which remains unused until the system one of the main elements will fail, and N - a scheme without redundancy, in which a failure in the operation of any element leads to the termination of the entire system
Many energy-related problems are solved by choosing the right location for the data center. The data center "Tushino" is located on the territory of the enterprise, where two 110 kV lines from different city power plants already come. On the equipment of the plant itself, the high voltage is converted to medium voltage, and two independent lines of 10 kV are fed to the input of the data center.
The transformer substation inside the data center building converts the average voltage into consumer voltage 240-400 V. All lines are run in parallel, so the data center equipment is powered from two independent external sources.
Low voltage from transformer substations is fed to automatic transfer switches, which provide switching between city networks. The motor drives installed on the ATS require 1.2 seconds for this operation. All this time, the load falls on the uninterruptible power supplies.
A separate ATS is responsible for automatically turning on the diesel generator in the event that power is lost on both lines. Starting a diesel generator is not a quick process and requires approximately 40 seconds, during which the power supply is completely supplied to the UPS batteries.
When fully charged, the diesel generator ensures the operation of the data center for 8 hours. Taking this into account, the data center entered into two contracts with independent suppliers of diesel fuel, which undertook to deliver a new portion of fuel within 4 hours after the call. The likelihood that both of them will have some kind of force majeure at once is extremely low. Thus, autonomy can last as long as it is necessary for repair teams to restore power from at least one of the city networks.
As you can easily see, there are no engineering refinements here. This is due, among other things, to the fact that when building the engineering infrastructure, ready-made modules were used, the manufacturers of which are guided by a certain "average consumer".
Of course, any IT professional will tell you that averaging is “neither fish nor meat” and will suggest developing a unique set of components for a particular system. However, those wishing to pay them for this pleasure are clearly not lining up. So you have to be realistic. In practice, everything will be exactly like this: the purchase of finished equipment and the assembly of a system that will solve urgent business problems. Those who disagree with this approach will be quickly returned from heaven to earth by the chief financial officer of the enterprise.
Switchboards
At the moment, nine switchboards provide the operation of the input distribution devices and four distribution boards are used directly to connect the load. There were no serious restrictions in place, but there is never a lot of it, so one interesting engineering moment was still present.
As you can easily see, the number of "input" and "load" boards does not coincide - the second is almost two times less. This became possible because the designers of the data center infrastructure decided to use large panels to add three or more incoming lines there. There are approximately 36 branch lines for each input machine, protected by separate machines.
Thus, sometimes the use of larger models allows you to save a scarce space. Simply because fewer shields are required.
Uninterruptible Power Supplies
Eaton 93PM with a capacity of 120 kVA, operating in double conversion mode, is used as an uninterruptible power supply at the Tushino data center.
The Eaton 93PM UPS is available in a variety of designs. Photo: Eaton
The main reasons for choosing this particular device are the following characteristics.
Firstly, the efficiency of this UPS reaches 97% in double conversion mode and 99% in energy saving mode. The device occupies less than 1.5 square meters. m and does not take up the server room space from the main equipment. The result is low operating costs and the savings that businesses need.
Second, with built-in thermal management, the Eaton 93PM UPS can be placed anywhere. Even close to the wall. Even if it is not needed right away, it may be required later. For example, to free up some space that is not enough for an additional rack.
Third, ease of use. Including - Intelligent Power software used for monitoring and control. The metrics sent by SNMP allow you to control consumption and some global failures, which makes it possible to quickly respond to emergency situations.
Fourth, modularity and scalability. This is perhaps the most important quality due to which only one modular UPS is used in the Tushino data center backup system. It includes two working modules and one backup. This provides the N + 1 scheme required for Tier II.
It is much simpler and more reliable than a three-UPS configuration. Therefore, the choice of a device in which the possibility of parallel operation is initially provided is quite logical.
But why didn't the designers choose DRIBP instead of separate UPS and diesel generator? The main reasons for this lie not in engineering, but in finance.
The modular structure is a priori sharpened for upgrade - as the load grows, sources and generators are added to the engineering infrastructure. At the same time, the old ones worked and still work. With DRIBP, the situation is radically different: you need to buy such a device with a large power reserve. In addition, there are few "small combines", and they cost very decently - incomparably more expensive than a separate diesel generator and UPS. DRIBP is also very capricious in transportation and installation. This, in turn, also affects the cost of the entire system.
The existing configuration solves its tasks quite successfully. The Eaton 93PM UPS can support the main equipment of the data center for 15 minutes, more than 15 times the reserve.
Again, the pure sine wave produced by the UPS online saves the owner of the data center from the need to purchase separate stabilizers. And here comes the savings.
Despite the declared simplicity, the Eaton 93PM UPS is a rather complex device. Therefore, its technical maintenance in the Tushino data center is carried out by a third-party company, which has highly qualified specialists in its staff. It is an expensive pleasure to keep a trained employee on your own staff for this purpose.
Results and prospects
This is how the data center was created, which allows providing high-quality services to consumers whose tasks do not require a high level of redundancy and do not imply large economic costs. Such a service will always be in demand.
The already planned construction of the second phase will use the already purchased Eaton UPS to create a backup power supply system. Due to the modular design, its modernization will be reduced to buying an additional module, which is more convenient and cheaper than a complete replacement of the device. This approach will be approved by both the engineer and the financier.