Let's talk about:
- rsync (remote synchronization)
- DRBD (Distributed Replicated Block Device)
- incremental backups for DRBD using LVM
- DRBD + ThinLVM
- ZFS (Zettabyte File System)
rsync . .
rsync (remote synchronization) is not about backups at all, strictly speaking. It is a program that allows you to synchronize files and directories in two locations while minimizing traffic. Synchronization can be performed for both local folders and remote servers.
Rsync is often used for backups. We used this utility when the sites were simpler and there were significantly fewer clients.
Rsync did a pretty good job, but the biggest issue here is speed. The program is very slow, it loads the system a lot. And with the increase in data, it starts to work even longer.
Rsync can be used as a backup technology, but for very small amounts of data.
LVM (logical volume manager) - logical volume manager
Of course, we wanted to make backups faster with less load, so we decided to try LVM. LVM allows snapshots even using ext 4. This way we could make backups using an LVM snapshot.
We didn't use this technology for long. Although the backup was faster than rsync, it was always full. And I only wanted to copy the changes, so we switched to DRBD.
DRBD
DRBD allows you to sync data from one server to another. Moreover, only changes are synchronized, not all data. This speeds up the process considerably!
And on the side of the store, we could use LVM and take snapshots. Such a system has existed for a very long time and now exists on some of the servers that we have not yet had time to transfer to the new system.
However, even with this method, there is still a drawback. DRBD heavily loads the disk subsystem during synchronization ... This means that the server will run slower. As a result, the backup interfered with the work of the main services, that is, user sites. We even tried to make backups at night, but sometimes they simply did not have time to complete overnight. I had to maneuver, alternate backups. For example, today one part of the servers is running, then another. We distributed backups in a checkerboard pattern.
DRBD, moreover, is highly dependent on network speed and affects the performance of the server from which and to which the backup is carried out. A new solution must be sought!
Thin LVM
At this point, the business set a task to make 30-day backups, and we decided to switch to thinLVM. This did not solve the main problem! We didn't even expect that such high performance of the file system would be required to support thin snapshots. This experience was completely unsuccessful, and we gave up in favor of regular thick LVM snapshots.
ThinLVMs were really just not designed for our purposes. Originally intended for small laptops and cameras, but not for hosting.
Continuing the search ...
It was decided to try ZFS.
ZFS
ZFS is a decent filesystem that has a lot of built-in goodies. What is achieved with ext 4 by installing on LVM, connecting a DRBD device, then with ZFS this is the default. The file system itself is very reliable. We should also mention the Copy-on-write function, this technology allows you to handle data very carefully.
ZFS allows you to make snapshots that can be copied to a store, as well as automate backups. No need to invent anything!
The migration to ZFS was very careful. First, we created a stand where we simply tested for several months. In particular, we tried to reproduce problems with equipment, power supply, network, disk fullness. Through thorough testing, we were able to find bottlenecks.
A sore subject for ZFS is disk fullness. We were able to solve this problem by reserving empty space. When the disk is full, measures will be taken to unload the server and clean up space.
After testing, we gradually began to introduce new servers, transfer old servers to ZFS. No more problems with backups! You can make 30- or 60-day backups, even if backups every hour. In any case, the server will not experience excessive loads.
Collected all the data in the tables below to compare backups using various technologies.
What happened next?
There are plans to upgrade ZFS to version 2 of OpenZFS 2.0.0. in 2021. We are preparing a transition using all the chips that were announced with the release in early December.
The Way this is!
This is the path we have chosen for ourselves! Are you solving similar problems? We will be glad if you share your experience in the comments! We hope the article turned out to be useful and, if suddenly you are also faced with the task of making backups using the built-in utilities in Linux, our story will help you find a suitable solution.