At Veeam, we love logs. And since most of our solutions are modular, they write a lot of logs. And since the sphere of our activity is to ensure the safety of your data (i.e., restful sleep), then the logs should not only record every sneeze, but also do it in some detail. This is necessary so that if something happens it is clear how this “what” happened, who is to blame, and what needs to be done next. It's like forensics: you never know what little thing will help you find Laura Palmer's killer.
Therefore, I decided to aim at a series of articles, where I will consistently talk about what we write to the logs, where we store them, how not to go crazy with their structure and what to look for inside them.
Why a series of articles and why not describe everything at once?
Just to list which log is where and what is stored in it is a rather disastrous undertaking. And it's scary to even think about keeping this information up to date. A simple listing of all possible types of logs in Veeam Backup & Replication is a table on several sheets in small print. And it will be relevant only at the time of publication, because when the next patch is released, new logs may appear, the logic of the stored information in the old ones will change, etc. Therefore, it will be much more advantageous to explain their structure and the essence of the information they contain. This will allow you to better navigate in the field than the trivial cramming of names.
Therefore, in order not to rush headlong into the maelstrom of text sheets, let's do some preparatory work in this article. Therefore, today we will not go into the logs themselves, but will go from afar: compile a glossary and discuss a little the structure of Veeam in terms of generating logs.
Glossary and jargon
Here, first of all, it is worth apologizing to the champions of the purity of the Russian language and witnesses of the Ozhegov dictionary. We all really love our native language, but the damned IT industry works in English. Well, we didn’t come up with it, but it happened historically. I'm not guilty, he came himself (c)
In our business, the problem of Anglicisms (and jargon) has its own specifics. When by innocent words like "host" or "guest" the whole world has long understood completely concrete things, then on ⅙ of the land, heroic confusion and wobbling with poking into dictionaries continues. And the strictly obligatory argument "But at our work ...".
Plus there is purely our terminology, which is inherent in Veeam products, although some words and phrases have gone to the people. Therefore, now we will agree on what term means what, and in the future, under the word "guest" I will mean exactly what is written in this chapter, and not what you are used to at work. And yes, this is not my personal whim, these are well-established terms in the industry. Fighting them is somewhat pointless. Although I'm always in favor of cooking in kamenty.
Unfortunately, there are a lot of terms in our work and products, so I won't try to list them all. Only the most basic information about backups and logs necessary for survival in the sea. For those interested, I can also offer a colleague's article about ribbons, where he also gave a list of terms related to that part of the functionality.
Host: In the virtualization world, this is a hypervisor machine. Physical, virtual, cloud - it doesn't matter. If a hypervisor is running on something (ESXi, Hyper-V, KVM etc), then this “something” is called a host. Whether it is a cluster for ten racks or your laptop with a lab for one and a half virtual machines - if you run a hypervisor, you become a host. Because the hypervisor hosts virtual machines. There is even a tale that VMware at one time wanted to achieve a strong association of the word host with ESXi. But she didn’t.
In the modern world, the concept of "host" has practically merged with the concept of "server", which introduces a certain amount of confusion in communication, especially when it comes to Windows infrastructure. So any machine that hosts some interesting service can be safely called a host. For example, in WinSock logs, the word host marks everything. The classic "Host not found" is an example of this. So we proceed from the context, but remember - in the world of virtualization, the host is what hosts the guests (more on this two lines below).
From local jargon (rather, even acronyms, in this case), I recall here that VMware is VI, vSphere is VC, and Hyper-V is HV.
Guest:A virtual machine running on a host. There is even nothing to explain, everything is so logical and simple. However, many are diligently dragging some other meanings here.
What for? I dont know.
Guest OS, respectively, is the operating system of the guest machine. Etc.
Backup / Replication Job (JobA): Purely Wim's jargon, denoting some of the tasks. Backup job == Backup Job. How beautiful to translate it into Russian - no one came up with it, so everyone says "job". With emphasis on the last syllable.
Yes, they just take it and say “job”. And even in letters they write like that, and everything is fine.
Any backup work, Backup Tasks, etc., thanks, but not necessary. Just a job, and you will be understood. The main thing is to put the stress on the last syllable.
Backup (Backup, backup. For hard-oldfagov, backup is allowed): In addition to the obvious (a backup copy of data lying somewhere), it also means the job itself (three lines above, if you have already forgotten), as a result of which the same backup file appears ... Probably, gentlemen, native English speakers are too lazy to say I ran my backup job every time, so they just say I ran my backup, and everyone understands each other perfectly. I propose to support this wonderful undertaking.
Consolidate:A term introduced in ESXi 5.0 An option in the snapshot menu that starts the process of deleting so-called orphaned snapshots. That is, snapshots that are physically available, but dropped out of the displayed logical structure. In theory, this process should not affect the files displayed in the snapshot manager, but anything can happen. The essence of the consolidation process is that data from a snapshot (child disk) is written to the main (parent) disk. The process of merging disks is called merge. If a consolidation command was issued, then the snapshot record can be deleted from the database before the snapshot is frozen and deleted. And if the snapshot could not be deleted for any reason, then these orphaned snapshots appear. VMware has a good KB about working with snapshots . And we, too, somehow about themwrote on Habré .
Datastore (Stora or store): A very broad concept, but in the world of virtualization, it refers to the place where the files of virtual machines are stored. But in any case, here you need to very clearly understand the context and, with the slightest doubt, clarify what exactly your interlocutor had in mind.
Proxy:It is important to immediately understand that Veeam Proxy is not exactly the same thing that we are used to on the fields of the Internet. Within the Veeam products, this is a kind of entity that is engaged in transferring data from one place to another. Without going into details, VBR is the command server, and proxies are its workhorses. That is, a proxy is a machine through which traffic flows and on which VBR components are installed that help steer this traffic. For example, transfer data from one channel to another, or simply attach disks to yourself (HotAdd mode).
Repository: Technically, this is just an entry in the VBR database indicating the location where the backups are stored and how to connect to this location. In fact, it can be just a CIFS ball, or a separate disk, server or bucket in the cloud. Again, we are in context, but understand that the repository is just where your backups are.
Snapshot:Lovers of Oxford grammar prefer to say who is a snapshot or who is a snapshot, however, the illiterate majority benefit from the greater mass. If anyone does not know, this is a technology that allows you to restore the state of a disk to a specific point in time. This is done either by temporarily redirecting I / O operations away from the main disk - then it will be called RoW (Redirect on Write) snapshot - or by moving rewritable blocks from your disk to another - this will be called CoW (Copy on Write) snapshot. It is thanks to the wide possibilities for the use of these functions that Veeam can do its own backup magic. Strictly speaking, not only for them, but this is the business of the next releases.
There is chaos in the ESXi documentation and logs around this term, and in the context of mentioning snapshots, you can find snapshots themselves, and redo log and even delta disk. There is no such irritation in the Veeam documentation, and a snapshot is a snapshot, and a redo log is exactly a REDO file created by an independent non-persistent disk. REDO files are deleted when the virtual machine is turned off, so confusing them with snapshots is a path to failure.
Synthetic: Synthetic backups refer to reverse incremental and forever forward backups. If you suddenly did not come across this term, then this is just one of the mechanisms used to build \ transform the backup chain. However, in the logs, you can also find the concept of Transform, which is used in the creation of synthetic full copies.
Task:This is the process of processing each individual machine within a job. That is: you have a backup job that includes three machines. This means that each car will be processed as a separate task. In total, there will be four logs: the main one for the job and three for the tasks. However, there is an important nuance: over time, the word "task" has become unnecessarily ambiguous. When we talk about general logs, we mean that a task is exactly a VM. But both the proxy and the repository have their own "tasks". There it can mean a virtual disk, a virtual machine, and the entire job. That is, it is important not to lose context.
Veeam% name% Service : For the benefit of successful backups, several services are working at once, the list of which can be found in the standard equipment. Their names quite transparently reflect their essence, but among equals there is the most important one - Veeam Backup Service, without which the rest will not work.
VSS: Technically, VSS should always stand for Microsoft Volume Shadow Copy Service. In fact it is used by many as a synonym for Application-Aware Image Processing. Which, of course, is categorically wrong, but this is a story from the category "Any SUV can be called a jeep, and you will be understood."
Fantastic logs and places where they live
I want to start this chapter by revealing the great secret - what time is displayed in the logs?
Remember:
- ESXi always logs to UTC + 0.
- vCenter keeps a log of the time of its time zone.
- Veeam .
- EVTX . , . , . — . . , , , , IT , . .
Now let's talk about the places where the logs live and how to get them. In the case of VBR, there are two approaches.
The first option is suitable if you are not eager to search for files in a common heap that are related specifically to your problem. To do this, we have a separate wizard, to which you can specify a specific job and a specific period for which you need logs. Then he will go over the daddies himself and put everything he needs into one archive. About where to look for it and how to work with it is written in detail in this KB .
However, the wizard collects logs not of all tasks and, for example, if you need to study the logs of a restaurant, failover or failback, your path lies in the % ProgramData% / Veeam / Backup folder... This is the main VBR logo repository, and% ProgramData% is a hidden folder and that's okay. By the way, the default location can be reassigned using a registry key like REG_SZ: LogDirectory in the HKEY_LOCAL_MACHINE \ SOFTWARE \ Veeam \ Veeam Backup and Replication \ branch.
On Linux machines, the logs of worker agents should be looked for in / var / log / VeeamBackup / if a root or sudo account is used. If you do not have such privileges, then look for the logs in / tmp / VeeamBackup .
For Veeam agent for% OS_name%, logs should be searched in % ProgramData% / Veeam / Endpoint (or % ProgramData% / Veeam / Backup / Endpoint ) and / var / log / veeam, respectively.
If you use Application-Aware Image Processing (and most likely you use it), then the situation becomes somewhat more complicated. You will need the logs of our helper, which are stored inside the virtual machine itself, and the VSS logs. How and where to get this happiness is described in detail in this article . And, of course, there is a separate article on collecting the necessary system logs.
It is convenient to collect Windows events according to this HF . If you are using Hyper-V, things get more complicated, since you will also need all of its logs from the Applications and Service Logs> Microsoft> Windows branch. Although, you can always go the more dodgy route and just take all the objects from% SystemRoot% \ System32 \ winevt \ Logs.
If something breaks during installation / upgrade, then everything you need can be found in the% ProgramData% / Veeam / Setup / Temp folder. Although I will not hide the fact that in OS events you can find more useful information than in these logs. The rest of the interesting lies in% Temp%, but there are mainly installation logs of related software, such as a database, .Net libraries and other things. Please note that Veeam is installed from msi, and all its components are also installed as separate msi packages, even if this was not displayed in the GUI. Therefore, if the installation of one of the components fails, the entire VBR installation will be stopped. Therefore, you need to go to the logs and see what exactly broke and at what moment.
And finally a life hack: if you get an error during installation, do not rush to click OK. First, we take the logs, then click OK. So you get a log that ends at the time of the error, without garbage at the end.
And it so happens that you need to get into the vSphere logs. It's a very thankless job, but having rolled up your sleeves, you have to do something else. In its simplest form, we need the logs with the events of the virtual machine vmware.log, which are located next to its .vmx file. In a more difficult case, open Google and ask where the logs for your version of the host are, since VMware loves to change this place from release to release. For example, here is an article for 7.0 , but for 5.5 . For vCenter logs, repeat the google procedure... But in general, we will be interested in the event logs of the hostd.log host, events of the hosts controlled by vCenter vpxa.log, the vmkernel.log kernel logs and the auth.log authentication logs. Well, in the most advanced cases, the SSO log can be useful, which is located in the SSO folder.
Is it cumbersome? Confused? Scary? But this is not even half of the information that our support works with on a daily basis. So they're really really cool.
Veeam components
And to end this introductory article, let's talk a little about the components of Veeam Backup & Replication. For when looking for the cause of pain, it would be nice to understand how the patient works.
So, as everyone probably knows, Veeam Backup is a so-called SQL-based application. That is, all settings, all information and in general everything that is only necessary for normal functioning - all this is in its database. Rather, in two bases, if we are talking about a bundle of VBR and EM: VeeamBackup and VeeamBackupReporting, respectively. And so it happened: we put one more application - another base appears. In order not to store all eggs in one corine.
But for all this economy to work smoothly, we need a set of services and applications that will connect all the components together. By way of example only, this is how it looks in one of my labs:
Veeam Backup Service is the chief conductor . It is he who is responsible for the exchange of information with the databases. He is also responsible for launching all tasks, orchestrating the allocated resources and works as a kind of communications center for a variety of consoles, agents and everything else. In a word, there is definitely nothing without him, but this does not mean at all that he does everything himself. Veeam Backup Manager
helps him to carry out his plans . This is not a service, but an entity that launches jobs and monitors the process of their execution. The backup service's working hands, with which it connects to hosts, creates snapshots, monitors retention, and so on. But back to the list of services. Veeam Broker Service
... Introduced in v9.5 (and this is not a crypto miner, as some then thought). Collects information about VMware hosts and keeps it up to date. But do not immediately run to write angry comments that we are spying on you and leaking all logins / passwords to the tashmayor. Everything is somewhat simpler. When you start a backup, the first thing you need to do is connect to the host and update all data about its structure. This is a rather slow and cumbersome story. Just remember how long it takes you to log in via the web interface, and remember that only the top layer is counted there. And then you still need to open the entire hierarchy to the right place, by the way. In a word, horror. If you run a dozen backups, then each job needs to do this procedure. If we are talking about large infrastructures, then this process can take ten minutes or more.Therefore, it was decided to allocate a separate service for this, through which it will be possible to receive always relevant information. At startup, it checks and scans the entire added infrastructure, and then tries to work only at the level of incremental changes. So even if you run a hundred backups at the same time, they will all request information from our broker, and will not torment the hosts with their requests. If you are worried about resources, then according to our calculations for 5000 virtual machines you need only about 100 Mb of memory.they will all request information from our broker, and will not torment the hosts with their requests. If you are worried about resources, then according to our calculations for 5000 virtual machines you need only about 100 Mb of memory.they will all request information from our broker, and will not torment the hosts with their requests. If you are worried about resources, then according to our calculations for 5000 virtual machines you need only about 100 Mb of memory.
Next up is Veeam Console . Aka Veeam Remote Console, aka Veeam.Backup.Shell. This is the same GUI that we see in the screenshots. Everything is simple and obvious - the console can be launched from anywhere, as long as it was Windows and there was a connection to the VBR server. The only thing that can be said is that the FLR process will mount the points locally (i.e. on the machine where the console is running). Well, the variegated Veeam Explorers will also run locally, because they are part of the console. But this has already carried me into the jungle ...
The next interesting service is Veeam Backup Catalog Data Service.Known as Veeam Guest Catalog Service in the list of services. He is engaged in indexing file systems on guest machines and fills the VBRCatalog folder with this knowledge. Used only where the indexing checkbox is enabled. And it only makes sense to enable it if you have Enterprise Manager. Therefore, advice from the bottom of my heart: do not turn on indexing just like that if you do not have an EM. Save your nerves and support time.
Also of other important services, it is worth noting the Veeam Installer Service , which is used to deliver and install the necessary components on proxies, repositories, and other gateways. In fact, he takes the necessary .msi packages to the servers and installs them.
Veeam Data Mover- with the help of auxiliary agents launched on proxies (and not only), it is engaged in data transfer. For example, during backup, one agent will read files from the host datastores, and the second will carefully write them to the backup.
Separately, I would like to note an important thing that clients often react to - this is the difference between the versions of services and information in the Programs and Features snap-in. Yes, the list will be the same, but the versions may be a complete mess. It's not very cool from a visual point of view, but it's completely okay if everything works stably. For example, the version number of the Installer service lags far behind its neighbors. Horror and nightmare? No, because it is not completely reinstalled, but its DLL is simply updated. In patch v9.5 U4, there was a technical support nightmare: during the update, all services received new versions, except for the most important. In the U4b patch, the transport service overtook all the others by as much as two versions (judging by the numbers). And that's okay too - a serious bug was found in it, so it got a bonus update over the rest. Therefore, to summarize:version differences MAY be a problem, but if there is a difference and everything is working properly, then most likely it should be. But no one forbids you to clarify this in technical support.
These were the so-called compulsory or Mandatory services. And there is also a whole bunch of auxiliary ones, such as Tape Service, Mount Service, vPowerNFS Service and so on.
For Hyper-V, in general, everything is the same, only there is a specific Veeam Backup Hyper-V Integration Service and its own driver for working with CBT.
And at the end, let's talk about who works on virtual machines during backup. Veeam Guest Helper is used to run pre- and post-freeze scripts, to create a shadow copy, collect metadata, work with SQL transaction logs, and more . And if file systems are indexed, Veeam Guest Indexer . These are temporary services that are deployed during the backup and deleted after it.
In the case of Linux machines, everything is much simpler due to the presence of a large number of built-in libraries and the capabilities of the system itself. For example, indexing is done via mlocate.
That's all for now
I don’t dare to torment you anymore and I consider a short introduction to the Veeam engine compartment over. Yes, we haven't even gotten close to the logs themselves, but believe me, so that the information presented in them does not seem like an incoherent stream of consciousness, such an introduction is absolutely necessary. I plan to go to the logs themselves only in the third article, and the plan for the next one is to explain who generates the logs, what exactly is displayed in them and why exactly this way, and not somehow otherwise.