From nothing to data center with VXLAN / EVPN or how to cook Cumulus Linux. Part 1

In the last six months, we managed to work on a large and interesting project, which included everything: from installing equipment to creating a single VXLAN / EVPN domain in 4 data centers. Because I had a lot of experience and a lot of bumps in the process, I decided that writing a few articles on this topic would be the best solution. I decided to make the first part more general and introductory. The target design of the factory will be revealed in the next section.

Introducing Cumulus Linux. Hardware installation and initial setup

Introductory to the beginning of work were as follows:

1. Equipment purchased
2. Racks rented
3. Lines to old data centers laid

The first piece of hardware that needed to be delivered was 4 x Mellanox SN2410 with Cumulus Linux pre-installed on them. At first, there was still no understanding of how everything would look like (it will develop only at the stage of VXLAN / EVPN implementation), therefore, we decided to raise them as simple L3 switches with CLAG (Analogue of MLAG from Cumulus). Previously, neither I nor my colleagues had much experience with Cumulus, so everything was to some extent new, then just about that.

No license - no ports

By default, when you turn on the device, only 2 ports are available to you - console and eth0 (aka Management port). To unblock 25G / 100G ports you need to add a license. And it immediately becomes clear that Linux in the name of the software is not for nothing, since after installing the license, you need to restart the switchd daemon through “systemctl restart switchd.service” (in fact, the lack of a license just prevents this daemon from starting).



The next thing that will immediately make you remember that this is still Linux, will be updating the device using apt-get upgrade, as in a regular Ubuntu, but it is not always possible to update this way. When switching between releases, for example, from 3.1.1 to 4.1.1, you need to install a new image, which entails resetting the configs to default. But it saves that DHCP is enabled on the Management interface in the default configuration, which allows you to return control.

Commit system

As a person who has worked a lot with Juniper, for me things like rollbacks, commit confirm, etc. were not new, but managed to step on a couple of rakes.



The first thing I ran into was the rollback numbering of cumulus, due to the habit of rollback 1 == the last working configuration. I am driving this command with great confidence to roll back the latest changes. But what was my surprise when the piece of hardware just disappeared in control, and for some time I did not understand what happened. Then, after reading the doc from cumulus, it became clear what had happened: by driving in the “net rollback 1” command instead of rolling back to the last configuration, I rolled back to the FIRST device configuration. (And again, DHCP saved from the fiasco in the default configuration)





The second thing I had to face was the commit confirm algorithm: unlike the usual “commit confirm 10”, where within 10 minutes you need to write “commit” again, Cumulus had its own vision of this feature. Your “commit confirm” is simply pressing Enter after entering a command, which can play a cruel joke on you if connectivity is not lost immediately after commit.

First topology

The next step was to work out the logic of the switches between themselves, at this stage the hardware was only installed and tested, there was no talk of any target schemes yet. But one of the conditions was that servers connected to different MLAG pairs must be in the same L2 domain. I didn't want to make one of the pairs simple L2, and therefore it was decided to raise L3 connectivity over SVI, OSPF was chosen for routing, since it has already been used in older data centers, making it easier to connect the infrastructure in the next step.







This diagram shows the physics diagram + the division of devices into pairs, all links in the diagram work in Trunk mode.







As mentioned, all L3 connectivity is done through SVI, therefore, only 2 devices out of 4 have an IP address in each Vlan, which allows you to make a kind of L3 p2p bundle.

Basic commands for those interested

Conclusion

I hope someone will find this article interesting. I would like to see feedback: what to add, and what is completely unnecessary. In the next article, we will already move on to the most interesting - to the design of the target network and VXLAN / EVPN configuration. And in the future, an article on VXLAN / EVPN automation using Python is possible.