Clever Geek Handbook

MinIo for the little ones

MinIO is a great solution when you need to easily and easily organize object storage. Basic customization, multiple platforms and good performance have done their job in the field of people's love. So we had no other way than to declare the compatibility of Veeam Backup & Replication and MinIO a month ago . Including such an important feature as Immutability. In fact, MinIO has a whole section in the documentation dedicated to our integration.



Therefore, today we will talk about how:



  • Setting up MinIO is very fast.
  • Setting up MinIO is a little less fast, but much better.
  • Use it as the Archive Tier for the Veeam SOBR scalable repository.






What are you?



A short introduction for those who have not encountered MinIO. It is an open source object storage compatible with the Amazon S3 API. It is released under the Apache v2 license and adheres to the philosophy of Spartan minimalism.



That is, it does not have a spreading GUI with dashboards, charts and numerous menus. MinIO simply starts its server with one command, where you can simply store data using the full power of the S3 API. But it should be noted that this simplicity can be deceiving when it comes to the resources used. RAM and CPU are absorbed perfectly well, but the reasons will be below. And, by the way, such combines as FreeNAS and TrueNAS use MinIO under the hood.



This introduction can be completed.



Setting up MinIO is very fast



Setting it up is so quick that we'll cover it for Windows and Linux. There are options for docker, and for kubernetis, and even for makosi, but the meaning will be the same everywhere.



So, in the case of Windows, go to the official website https://min.io/download#/windows and download the latest version. In the same place, we observe the launch instructions: 



 minio.exe server F:\Data


And there is also a link to a slightly more detailed Quick start guide . There is no point in not believing the instructions, so we run and get something like this.





That's all! The storage is working and you can start working with it. I was not joking when I said that MinIO is minimalism and just works. If you follow the link suggested when starting up, then the maximum of the functions available there is to create a bucket. And you can start writing data.



For Linux lovers, everything remains no less simple. The simplest instruction: 




wget https://dl.min.io/server/minio/release/linux-amd64/minio
chmod +x minio
./minio server /data


The result will be indistinguishable from what was seen earlier. 



Setting up MinIO is a little more meaningful



As we understand it, the previous paragraph is mischief for testing purposes. And, frankly, it is for testing MinIO that we use very widely, which we are not at all ashamed to admit. Of course, it works, but it's a shame to endure such a thing beyond test benches. Therefore, we take a file in our hands and begin to bring it to mind.



HTTPS



The first mandatory step on the road to production is encryption. There are already a million and a thousand manuals on the network for adding certificates to MiniIO, but their general plan is as follows:



  • Create a certificate
  • In the case of Windows, put it in C: \ Users \% User% \. Minio \ certs
  • In case of Linux in $ {HOME} /. Minio / certs 
  • We restart the server


Banal Let's Encrypt is boring and described everywhere, so our path is the path of a samurai, so in the case of Windows we download Cygwin , and in the case of Linux we just check that we have openssl installed. And doing some console magic:



  • Create keys: openssl ecparam -genkey -name prime256v1 | openssl ec -out private.key
  • Create a certificate by key: openssl req -new -x509 -days 3650 -key private.key -out public.crt
  • Copy private.key and public.crt to the above folder
  • Restart MinIO


If everything went as it should, then about the following lines will appear in the status.





Turn on MinIO Erasure Coding



First, a few words about the subject. In a nutshell: it is software-based data protection against damage and loss. Like a raid, only much more reliable. While classic RAID6 can afford to lose two drives, MinIO is comfortable losing half. The technology is described in more detail in the official guide . But if you take the very essence, then this is the implementation of Reed-Solomon codes: all information is stored in the form of data blocks, to which there are parity blocks. And it seems that all this has already been done many times, only there is an important "but": we can explicitly indicate the ratio of parity blocks to data blocks for stored objects.

Want 1: 1? You are welcome!

Want 5: 2? No problem!



A very important function if you use several nodes at once and want to find your own balance between the maximum data security and the expended resources. Out of the box, MinIO uses the N / 2 formula (where N is the total number of disks), i.e. divides your data between N / 2 data disks and N / 2 parity disks. Translating into a human: you can lose half of the disks and recover data. This ratio is set through the Storage Class , allowing you to independently choose which is more important: reliability or capacity.



The guide provides an example: suppose you have an installation on 16 disks and you need to save a file of 100 MB. If the default settings are used (8 disks for data, 8 for parity blocks), then the file will eventually take up almost double the size, i.e. 200 MB. If the drive ratio is 10/6, then 160 MB is needed. 14/2 - 114 Mb.



Another important difference from raids: in case of dropping disks, MinIO will work at the object level, restoring one by one without stopping the entire system. While a normal raid will be forced to restore the entire volume, which will take an unpredictable amount of time. In the memory of the author, the disk shelf, which, after the loss of two disks, was recalculated for a week and a half. It was very unpleasant.



And an important note: MinIO divides all disks for Erasure Coding into sets from 4 to 16 disks, using the largest possible set size. And in the future, one piece of information will be stored only within one set.



This all sounds great, but how difficult is the setup going to be? Let's get a look. We take the command to start and simply list the disks on which we need to create storage. If everything is done correctly, then in the report we will see the number of used disks. And advice that it is useless to add half of the disks to one host at once, because this is a loss of data. 



c:\minio>minio.exe server F:\ G:\ H:\ I:\ J:\ K:\

Further, to control and customize the server MinIO we need an agent, which you can download all the same with the official website.



In order not to erase your fingers, each time typing the address and access keys (and this is not safe), it is convenient at the first start to immediately create an alias using the formula mc alias set <YOUR-MINIO-ENDPOINT> [YOUR-ACCESS-KEY] [YOUR-SECRET- KEY] 



mc alias set veeamS3 https://172.17.32.52:9000 YOURS3ACCESSKEY YOURSECERTKE


Or you can add your host right away: 



mc config host add minio-veeam https://minio.jorgedelacruz.es YOURS3ACCESSKEY YOURSECERTKEY


And then we will create an immutable bucket with a beautiful team 



mc mb --debug -l veeamS3/immutable 

mc: <DEBUG> PUT /immutable/ HTTP/1.1
Host: 172.17.32.52:9000
User-Agent: MinIO (windows; amd64) minio-go/v7.0.5 mc/2020-08-08T02:33:58Z
Content-Length: 0
Authorization: AWS4-HMAC-SHA256 Credential=minioadmin/20200819/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-bucket-object-lock-enabled;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**
X-Amz-Bucket-Object-Lock-Enabled: true
X-Amz-Content-Sha256: UNSIGNED-PAYLOAD
X-Amz-Date: 20200819T092241Z
Accept-Encoding: gzip
mc: <DEBUG> HTTP/1.1 200 OK
Content-Length: 0
Accept-Ranges: bytes
Content-Security-Policy: block-all-mixed-content
Date: Wed, 19 Aug 2020 09:22:42 GMT
Location: /immutable
Server: MinIO/RELEASE.2020-08-16T18-39-38Z
Vary: Origin
X-Amz-Request-Id: 162CA0F9A3A3AEA0
X-Xss-Protection: 1; mode=block
mc: <DEBUG> Response Time:  253.0017ms


--debug allows you to see not just the final message, but more detailed information. 



-l means --with-lock, which means immutable



If we now return to the web interface, our new bucket will appear there.





That's all for now. We have created a secure storage and are ready to move on to integration with Veeam.



You can also make sure that everything works perfectly: 



c:\minio>mc admin info veeamS3

●  172.17.32.52:9000
   Uptime: 32 minutes
   Version: 2020-08-16T18:39:38Z
   Network: 1/1 OK
   Drives: 6/6 OK
0 B Used, 1 Bucket, 0 Objects
6 drives online, 0 drives offline


MinIO and Veeam



Attention! If for some incredible reason you want to work over HTTP, then at HKEY_LOCAL_MACHINE \ SOFTWARE \ Veeam \ Veeam Backup and Replication \ create a DWORD key SOBRArchiveS3DisableTLS. Set its value to 1 and remember that we strongly disapprove of such behavior and do not recommend it to anyone.



Attention again! If, due to some misunderstanding, you continue to use Windows 2008 R2, then when you try to connect MinIO to Veeam, you will most likely receive an error similar to the following: Failed to establish connection to Amazon S3 endpoint. This is treated with an official patch from Microsoft .



Well, the preparations are over, let's open the VBR interface and go to the Backup Infrastructure tab, where we will call the wizard for adding a new repository.





Of course, we are interested in Object storage, namely S3 Compatible. In the opened wizard, set the name, go through the steps with the address and account. If required, do not forget to specify the gate through which requests to the storage will be proxied.





Then select the bucket, folder and tick the Make recent backups immutable checkbox. Or we don't. But since we have made a repository with support for this function, it will be a sin not to use it.





Next> Finish and enjoy the result.



Now you need to add it to the SOBR repository as a Capacity Tier. To do this, either create a new one, or edit the existing one. We are interested in the Capacity Tier step.





Here we need to choose which scenario we will work with. All the options are described quite well in another article , so I will not repeat myself



And after the wizard completes, tasks for copying or transferring backups will be automatically launched. But if it is not in your plans to immediately issue the load on all systems, then be sure to set the allowable intervals for working with the Window button.





And, of course, you can make separate Backup Copy jobs. Some people think that it is even more convenient this way, since they are somewhat more transparent and predictable for a user who does not want to delve into the details of the operation of a capital shooting gallery. And there are enough details, so once again I recommend the corresponding article from the link above.



And finally - the answer to the insidious question: what will happen if you still take and try to remove the backup from the Immutable store?



Here is the answer:





That's all for today. By true tradition, catch a list of useful topics on the topic:





More articles:


All Articles