Structure of the Data Center of TIER 3 Level

IT

Beeline, earlier Bee Line GSM is a Russian telecommunications brand by OJSC VimpelCom. OJSC VimpelCom is a Russia’s third-largest wireless and second-largest telecommunications operator. Its headquarters is located in Moscow. Since 2009, OJSC VimpelCom has been a subsidiary of VimpelCom Ltd., which is based in Amsterdam. VimpelCom’s main competitors in Russia are Mobile TeleSystems and MegaFon.

There used to be a field in place of the data center. We dug a huge 100 meter foundation pit. Then, the data center looked like a concrete pad. Using metal constructions, we have built a hexagonal building, which houses six modules of the data center (they’re highlighted in green), the command center to monitor the backbone network across the country, and the office.

Data Center of TIER 3 Level

Warning! A lot of traffic and geekporn: 91 images of the newly launched first module. There are still some works being maintained there, but no more major construction.

Basic Parameters of the Data Center

Structure of the Data Center of TIER 3 Level

Territory

  • 7 ha;
  • 30 thousand sq m of the building area.

Module Approach

  • 6 independent modules of the data center (the first one is built for now).

Fault Tolerance

  • Tier III Certification according to the Uptime Institute classification.

Uninterrupted Power Supply

  • 2 independent entrance inputs of 10 MW;
  • Dynamic Diesel UPS: 2500 kVA;
  • 2N redundancy;
  • 30 000 liters of fuel storage for 12 hours of operation of 6 modules when fully loaded.

Refrigeration Supply

  • N+2;
  • Water cooling on chiller plants;
  • Natural Free Cooling;
  • Adiabatic cooling system;
  • Uninterrupted power supply of the cooling system equipment;
  • The annual average PUE not more than 1.3.

Firefighting

  • Novec Gas, harmless to humans;
  • Early Warning Fire Detection System (VESDA).

Security

  • 24х7х365 Security;
  • Access Control, CCTV.

Environmental Friendliness

  • Uninterrupted power supply system without chemical batteries;
  • Natural Free Cooling up to 90% time of the year;.
  • Eco-friendly interior decoration materials of the office.

In reality, the building is almost as transparent as in the plan, because a lot of polarized glass is used there.

Structure of the Data Center of TIER 3 Level

The main building that houses the data center.

For comparison: this is a small container data center for seven racks. We have built it within three months. The mobile data center and the neighboring blue container with a diesel generator worked here during the commissioning and generally kept a number of basic services until we raised the first turbine hall.

Structure of the Data Center of TIER 3 Level

The Containerized Data Center

Here’s the plan of the «space cruiser»:

Structure of the Data Center of TIER 3 Level

A hexagon in the middle is the main building, which houses the turbine hall, as well as office and service areas of the Enterprise. The containerized data center underneath is small, they will take it away soon. As for the two blocks of the cruiser’s “engines”, they are independent Power modules. Let’s start there.

This is how the building with power modules looks like from the outside:

Structure of the Data Center of TIER 3 Level

The power module that houses the dynamic diesel UPS

The diesel fuel is stored in the blue container. We use the winter diesel fuel in winter, and the summer one in summer. We do not need to pump out, as we produce diesel fuel during monthly operational checks.

Structure of the Data Center of TIER 3 Level

The fuel storage for 30 000 liters.

But let’s get inside the power module. A wonderful device meets us here.

Structure of the Data Center of TIER 3 Level

A motor-generator of the dynamic diesel UPS system.

It’s a complex of devices. Here is the video from the distributor:

In short, the power we get from the two independent substations is reserved by the diesel generator system. There is a kinetic accumulator in the middle of the circuit, between the sources and the payload.

Structure of the Data Center of TIER 3 Level

Diagram of the electric power supply system of the complex.

The data center will consume up to 10 MW. For obvious reasons, it’s much more practical to take the power from the city. But if anything happens, we should have either a huge stock of USP batteries for the time of heating the diesel, or a stock of smaller batteries and a basin with ice water for cooling, or a constantly running power plant (generation) of our own. One of the best solutions here is the dynamic diesel UPS: a kinetic accumulator, plus a quick start diesel in the standby mode. In practice, everything is much more complicated, but the general principle of operation is like this.

Structure of the Data Center of TIER 3 Level

The control panel of the dynamic diesel UPS.

It should be noted that the power we get from the city is not of the highest quality. When it is worsening or missing, we use the energy of the kinetic accumulator (a seven-ton flywheel spun to high speed). The energy of the flywheel’s residual rotation is enough to start the diesel even at the second attempt. Among other advantages is that this system allows to handle most power surges. Roughly speaking, the dynamic diesel UPS in the middle of the circuit can easily smooth an average of about 80% of peaks and sags without engaging the motor-generator. Another advantage of applying the dynamic diesel UPS is no problem with the delay of switching to the backup power system, as the kinetic accumulator is connected into the circuit.

In addition to conventional filtration, the Fuel Purification System also enables the system of separation of impurities, which performs the purification of fuel right before it is fed to the fuel pump.

Structure of the Data Center of TIER 3 Level

Separator-filter of the fuel supply system

Structure of the Data Center of TIER 3 Level

Cylinder heads

Structure of the Data Center of TIER 3 Level

Solenoid operated valve of the fuel supply system

One of the most important tasks is to ensure energy efficiency at low consumption.

Structure of the Data Center of TIER 3 Level

The pipe leads to the second floor, where the systems of silencing and exhaust gas purification are located. In particular, 4.5 meter two-ton silencers.

Take a look at the real size of silencers which is located at a factory of dynamic diesel UPS in Germany.

Structure of the Data Center of TIER 3 Level

Structure of the Data Center of TIER 3 Level

It’s a fuel tank.

Diesel fuel for the diesel generator is stored in large quantities in the blue house outside. This tank stores the necessary reserve for an hour or two of operation (depending on load). The tanks are connected, we can bring fuel infinitely. First, the main storage will be filled, then this tank, and, finally, the diesel generator set. Of course, the second power module is the same, and it has fuel storage of its own.

Structure of the Data Center of TIER 3 Level

Exhaust fan of the ventilation system.

Structure of the Data Center of TIER 3 Level

Pipelines of the diesel engine cooling systems. Refrigerant circulates in pipes. It passes the radiators, installed on the second floor.

To reduce losses, the flywheel rotates in the gaseous helium. As long as it’s in this environment, the cost of maintaining the rotation is minimum even at the highest speed. Thus, it’s enough to give it energy for the spin-up, and it will consume little energy. Energy conversion efficiency is more than 99%. When necessary, the flywheel gives the stored energy to the system.

Structure of the Data Center of TIER 3 Level

Pressure reducer of the helium supply to the kinetic accumulator.

Structure of the Data Center of TIER 3 Level

An emergency stop valve of the diesel engine for the “fire” signal or the red emergency shutdown button. It is used only as a last resort, and actually cuts off the oxygen from the motor.

The fire is suppressed with the Novec 1230 gas. It is not dangerous to humans in the applied concentration. The firefighting system is smart: in case there’s a fire on the diesel, it gives a chance to deal with it. This increases operability of the data center.

Structure of the Data Center of TIER 3 Level

Starter batteries

Structure of the Data Center of TIER 3 Level

Power cables from the generator to the load (the data center)

Structure of the Data Center of TIER 3 Level

Fire alarm siren. As soon as it’s activated, it is necessary to run out of the building.

Structure of the Data Center of TIER 3 Level

The control cabinet of the ventilation system.

Structure of the Data Center of TIER 3 Level

A circuit breaker of the distribution board.

Here is a foolproof. The lock opens critical panels. To press something, we should remove the plastic shutter by turning it 180 degrees. Then it’s enough to strongly press the button. Such security options have been created, so that no one would do anything without fully understanding what is happening.

Structure of the Data Center of TIER 3 Level

The button of the smoke exhaust system is located at the entrance to a room.

That is the unit, to which the test load is connected outside for operational checks of the dynamic diesel UPS:

Structure of the Data Center of TIER 3 Level

Structure of the Data Center of TIER 3 Level

The load is connected up with a special cable that is very flexible for its diameter.

Here’s the load itself. It’s a semi-trailer truck. In the trailer, it has something like a huge calorifer that effectively heats the air to take away more power from power module.

Not that winter became warmer, but we did what we could.

Structure of the Data Center of TIER 3 Level

The semi-trailer truck.

These are the two interesting objects, the Lightning rod and the diverter. More precisely, the high mast structure on the power module is the lightning rod. It is a metal rod that diverts the discharge deep under the ground. On the roof of the data center, you can see the already active unit of similar purpose. It allows to achieve a similar effect, but with much smaller sizes.

Structure of the Data Center of TIER 3 Level

The passive and the active lightning rods.

The second module is symmetrical to the first one. The only thing that distinguishes its surroundings is this additional facility. It’s a separate diesel generator for the office. Requirements to the uninterrupted operation are not so high, so we do not have to expend the power of the dynamic diesel UPS.

Structure of the Data Center of TIER 3 Level

The diesel generator for the office.

Now, let’s go down to the subway line leading to the Kremlin.

Structure of the Data Center of TIER 3 Level

The cable collector

Of course, it’s a cable collector, but if we turn off the light in the right way, we could let beginners into the secrets of the USSR.

Structure of the Data Center of TIER 3 Level

We can also do like this:

Structure of the Data Center of TIER 3 Level

There are cables as thick as a hand of a strong engineer, or as a leg of a pretty girl.

Structure of the Data Center of TIER 3 Level

We had no problems with cables on the site, but it should be definitely checked at acceptance, as cables can be with cuts or damaged insulation, and much more.

Structure of the Data Center of TIER 3 Level

The “glamorous” fire-fighting foam. During the mounting, it expands and covers very tightly everything it can to prevent the spread of fire and smoke through the holes, in which in our case the cable is.

So, we have the power. But the data center lacks cooling and the Internet. And beer, of course, but there’s a brewery just outside the window. Everything’s fine with the backhaul. We have two backbone rooms reserving each other. There are 4 inputs of the backbone optics supplied to the data center. We have provided the ability to install 100Gbit/s transponders.

Let’s take a look at the cooling process.

Here are some main points:

Structure of the Data Center of TIER 3 Level

Scheme of the Natural Free Cooling system

Free cooling works like this. A wind blows into the “face” (mixing chambers) of our data center. The cold air gets into the main duct, gives the cold to the inner circuit, then moves further through the data center and goes outside. Using the heat exchanger, the hot environment of the turbine hall gives the heat to the hangar, which houses the data center modules. Hot updrafts go up through the openings in the roof, or forced through the mixing chamber. It’s simple, right?

Structure of the Data Center of TIER 3 Level

Layout of the main air duct

The first one we meet is a “spider”. It’s a chiller. We call it spider because of the characteristic shape.

Structure of the Data Center of TIER 3 Level

The Chiller

Why do we need a chiller in the data center with the free cooling system? This is because if the outside temperature rises above 75 F, turbine halls will not be cooled with the desired efficiency. Reminding you that we have chosen Yaroslavl, as it’s cooler in summer here, than in Moscow, so we can save on cooling.

Nevertheless, the average temperature rises above the mentioned boundary for up to a month. In this case, we use the second cooling system on chillers that cool the water in the vast network of pipelines. Simply put, they are huge refrigerators that cool the water. They require a lot more power, but there’s nothing we can do about it. Chiller redundancy is N+2, which means that there are three units for the first turbine hall.

Do you recognize these characters?

Structure of the Data Center of TIER 3 Level

Expansion tanks

A total of 180 tons of water in the circuit.

Structure of the Data Center of TIER 3 Level

Main collector

Structure of the Data Center of TIER 3 Level

Drain cock

Structure of the Data Center of TIER 3 Level

That’s another spider, but from the other side ( the side of the control panel)

Structure of the Data Center of TIER 3 Level

Balancing valve

Structure of the Data Center of TIER 3 Level

If we need emergency water drainage, it will flow into special pits.

Structure of the Data Center of TIER 3 Level

Pumps provide the circulation in the circuit. The second pump is redundant.

Structure of the Data Center of TIER 3 Level

Refrigerants lead to the roof. Bend is a refrigerant trap, which acts as a turbulator to entrain oil droplets and carry them efficiently up the elevated discharge line.

We will go to the turbine hall soon. It’s right behind this wall. But! There is an important point. The thing is that the temperature of the ambient air is sufficient to cool turbine halls with the necessary reserve only for 8 months a year. But we run chillers for two to four weeks. What happened with at least three and a half months? Here, look at this device.

Structure of the Data Center of TIER 3 Level

MNFC — a heat exchanger.

This is the heat exchanger chamber. The cold aisle of the turbine hall starts here, and the hot one ends. Thus, we make hot air cold. This is the key point of the cooling system. The heat exchanger works in the mode of cooling by the ambient air, as well as in the water cooling mode.

Structure of the Data Center of TIER 3 Level

The basic mode is as follows. A large amount of outside air is passed through the heat exchanger. If the air is hot, we close the air intake and begin to cool the exchange chamber by the output from the chiller, wasting electricity on the cooling process.

Structure of the Data Center of TIER 3 Level

The heat is exchanged behind this grating.

But when the air is warmer by just 3-4°C (up to 24°C), which happens during the “missing” months, we apply a life hack of the school physics course. We simply saturate the air with moisture (run the air though a wall of water mist behind this unit), which gives the desired delta for an adiabatic process. This techniques is much more energy conserving than the chiller system, and is widely used together with free cooling.

Here, just through the building, goes the exhaust main duct. It is quite a wide hot aisle.

Structure of the Data Center of TIER 3 Level

Structure of the Data Center of TIER 3 Level

Structure of the Data Center of TIER 3 Level

Roof lights. When necessary, the flaps open and hot air flows out.

Okay, okay, let’s go to the turbine hall. As for now, the halls from second to sixth look like this, modules are not assembled yet.

Structure of the Data Center of TIER 3 Level

Here’s the first module launched and being filled. We came when a part of servers was operating, but the hall was half-empty.

Structure of the Data Center of TIER 3 Level

Module of the data center.

Since racks are standard, and the iron is homogeneous (it’s our data center, and we know what will be installed here), we could mount all the inputs-outputs at once.

Structure of the Data Center of TIER 3 Level

Power sockets for connecting racks.

Another advantage of standard racks is the known dimensions of the hot and cold aisles. Here’s the cold air blowing from the false floor. These panels are covered with polyethylene, as the aisle is not used for the moment, and we don’t want the engineers to be blown away. By the way, these cold aisles is the last reminder of why an admin is wearing a sweater. Because it’s very cold and blowing.

Structure of the Data Center of TIER 3 Level

The cold air aisle.

This is our teleportation device, the prototype.

Structure of the Data Center of TIER 3 Level

Jokes aside, this door closes (limits) the cold aisle, so that cooling would be efficient. And that’s how a part of the hall with racks looks like.

Structure of the Data Center of TIER 3 Level

Racks

As for the racks, there’s following equipment:

  • Oracle servers of Hi-End and Mid-Range types;
  • Data storage system, built on HDS, HP, IBM, Brocade;
  • The network core is based on Avaya equipment;
  • HP servers.

Structure of the Data Center of TIER 3 Level

The system of power distribution to racks: busbars with power take off boxes.

Structure of the Data Center of TIER 3 Level

Our system has structured cabling, top-of-rack Panduit

Structure of the Data Center of TIER 3 Level

Top-of-rack cartridges.

Structure of the Data Center of TIER 3 Level

Crossover with switching is not perfect, but it will be soon.

Accuracy in assembling is a kind of our little fad. We know from experience that an unsigned messy mounted cable can cause interruption of service.

Structure of the Data Center of TIER 3 Level

Switching aggregation of network switches

Structure of the Data Center of TIER 3 Level

Active network equipment

10% of the hall is designed for racks of high density – up to 20 kW per unit. The most powerful rack are located in the center of the hall, as it has a reserve of the cold due to peculiarities of air flows of high speed.

These are data storage systems, but we have not connected them yet. The data about customer’s balance will be stored here.

Structure of the Data Center of TIER 3 Level

Tape Library

And these ate fan heaters. Using them, we tested the capability of the hall to remove heat. Exactly, we simply warmed them and watched what would happen.

Structure of the Data Center of TIER 3 Level

Structure of the Data Center of TIER 3 Level

The tap to the in-line busbar.

Structure of the Data Center of TIER 3 Level

Novec, again. The reserve is enough for two runs of the firefighting system.

Structure of the Data Center of TIER 3 Level

That’s how power cables that we’ve seen in the basement go to the turbine hall.

Now, let’s look at the office of admins and not only them.

Structure of the Data Center of TIER 3 Level

The main entrance to the building of the data center.

We go around the data center and see the temporary modular units. They are the dining room and the «office». These are the places where people worked when the office rooms were not ready. Actually, it’s better than working in a “container”.

Structure of the Data Center of TIER 3 Level

Temporary office and dining room.

There’s the layout of the data center at the entrance:

Structure of the Data Center of TIER 3 Level

The cooling scheme is illuminated:

Structure of the Data Center of TIER 3 Level

On the roof, we can see condensing units of the chillers:

Structure of the Data Center of TIER 3 Level

We can also illuminate the power and main inputs:

Structure of the Data Center of TIER 3 Level

Before we get inside, I’ll show you helmets in the locker room. One of the builders turned out to be an artist and painted his helmet, and not only his.

Structure of the Data Center of TIER 3 Level

Structure of the Data Center of TIER 3 Level

Open space

Structure of the Data Center of TIER 3 Level

The wall is made of a school blackboard.

Structure of the Data Center of TIER 3 Level

Room for administrators.

Structure of the Data Center of TIER 3 Level

Coffe Point

Structure of the Data Center of TIER 3 Level

Structure of the Data Center of TIER 3 Level

WC

Let’s get outside! We’ve been everywhere, except for the roof, but there are chillers on it. Here they are:

Structure of the Data Center of TIER 3 Level

The roof is very big and striped. From the satellite, our data center looks like a our logo.

We’re getting close to chiller units. They’re just like those of household air conditioners, but a bit bigger.

Structure of the Data Center of TIER 3 Level

These “windows” or roof lights are places where the hot air naturally leaves the building.

Structure of the Data Center of TIER 3 Level

Roof lights

Structure of the Data Center of TIER 3 Level

Each roof light has its own weather station that sends data to the automation system.

Structure of the Data Center of TIER 3 Level

Here’s the final destination of refrigerants we’ve seen before.

Structure of the Data Center of TIER 3 Level

There are plenty of pipes here, as each unit is dual circuit, and there are two of them per each chiller (that is to say, the chiller is four circuit).

Structure of the Data Center of TIER 3 Level

Elements of general ventilation systems of the building.

This is the mast of the lightning protection:

Structure of the Data Center of TIER 3 Level

That is how our data center looks like. As for now, we have launched the fist module of the flagship data center, designed for 236 racks. We are moving a part of our “combat” systems there. By the way, the Federal Center for monitoring has moved to us, and girls from the United Service Center will come soon as well. So, we’re quite inspired.

Comments

1,128

Ropes — Fast Strings

Most of us work with strings one way or another. There’s no way to avoid them — when writing code, you’re doomed to concatinate strings every day, split them into parts and access certain characters by index. We are used to the fact that strings are fixed-length arrays of characters, which leads to certain limitations when working with them. For instance, we cannot quickly concatenate two strings. To do this, we will at first need to allocate the required amount of memory, and then copy there the data from the concatenated strings.