In this second part of the series we will discuss how you can manually check Change Block Tracking (CBT) via the Managed Object Browser (MOB).
- Backup and Change Rate, Part 1: Change Rate Explained
- Backup and Change Rate, Part 2: Querying Change Rate in vSphere
Using the Managed Object Browser
For the sake of simplicity everything in these examples has been tested against a standalone ESXi host, although this process should be quite similar if you use vCenter. The vSphere API can be difficult to understand at first. However, VMware offers something called the Managed Object Browser, which basically allows you to browse the API and find out the structure of the API without actually doing any programming. In vSphere 6, MOB is disabled by default on an ESXi host. So you need to enable it. You can follow William Lam’s instructions on how to enable it:
Once enabled, you can browse to https://server.domain.local/mob and log in. The first screen should look something like the below.
The first thing you need to lookup is your virtual machine (VM). The easiest way would be to follow the links in this order on an ESXi host:
|content -> ha-folder-root -> ha-datacenter -> ha-folder-vm|
Alternatively you can just browse to https://server.domain.local/mob/?moid=ha-folder-vm
There you should be able to find your VM. Interestingly enough, there is a number before each VM which is known as the Managed Object ID (MOID) or more commonly as the Managed Object Reference ID (MoRef ID). basically the API does not identify the VM via its name, but by and ID that does not change if you rename the VM. However, one important note, a virtual machine’s MoRef is not necessarily the same for vCenter as it is for the ESX host hosting the VM. Every layer keeps its own set of MoRefs. This allows you, for example, to add ESXi hosts to a cluster without conflicting IDs. This does imply that if you, for example, do a vMotion to another host, perform a Storage vMotion, or make a clone of a virtual machine, the ID might change.
As such, if you are connected to an ESXi individual host and you are following this example with a VM in a cluster, you may either wish to set VM host affinity, or follow this process via vCenter instead.
As a side note: For the very same reason, when you configure Veeam Backup & Replication, it is highly recommended that you connect it to vCenter instead of individual hosts that are part of a cluster (unless you have specific reasons for doing so, for example a distaster recovery scenario where vCenter is not available), to ensure virtual machines maintain consistent MoRefs during backup.
In the example above, you will see that the VM “cbttest” has a MoRef ID of ’27’. Record the number as you will need it later. The next thing we do is follow the link to the VM.
Here we can find the information associated with the VM. This portion also allows you to apply methods to the virtual machine. We will come back to this page regularly as it contains most of the methods we need for this example. So if you like you can save this URL somewhere for later use.
First, to find out what disks are attached to the virtual machine, follow:
|config > hardware|
Here you will find all the disks that are attached. It seems that disks have a device number between 2000-3000, but we could not find any information to confirm that. You can check the device type to find the associated disks. If you follow the device information and click on ‘backing’ information, you will see some useful disk-related information, and more importantly you will see that the ‘changeId’ is ‘unset’. The ‘changeId’ is the current timestamp for the disk. This shows something important, as you can only a timestamp for a specific point in time if CBT is enabled, and you have a snapshot on disk. For now, find the disk(s) you wish to monitor and write down the device ID.
Now lets go back to the VM. You can do this by going back in the browser history or just by following the correct link https://server.domain.local/mob/?moid=27 if the virtual machine id is ’27’.
Go again to config. There look for a field called ‘changeTrackingEnabled’. For this test to work, it should say ‘True’. If it does not, you need to enable Change Block Tracking. If you want to do this safely, you can just run a backup job from Veeam Backup & Replication. This should be automatically enable it. If you are doing this in a test environment, you can go and live a little more on the wild side.
To enable CBT manually, make sure you do not have any snapshots on the virtual machine. Then go back to the VM page and find the method ‘ReconfigVM_Task’ and follow it. A new window should appear, remove the complete predefined XML and copy paste the following XML in the input box.
Then click ‘Invoke Method’ to start the task. This should give you a task you can follow. In the info section, you can then check if the state is ‘success’. You can also follow it via the vSphere client as the task should pop up with the name ‘Reconfigure Virtual Machine’. If you can go back to the ‘config’ section of the virtual machine, ‘changeTrackingEnabled’ should now be enabled.
So far we have, the virtual machine ID, the disk ID, and CBT enabled. Now for the real work. Let us get the ‘changeId’ for our disk. The first thing we need is to create a snapshot. This should be fairly easy. Go back to the VM page and look for the method ‘CreateSnapshot_Task’.
This should open a new window. Just fill in a name and description. Set memory and quiesce both to ‘0’, so it does not snapshot the memory nor will it quiesce applications inside the virtual machine via the VMware tools.
Again invoke this method, and you will get a task. In the information section you can check again what the state is, if it says ‘successful’, you should be able to find the snapshot via the vSphere client as well.
Now lets inspect the snapshot. You can do this by following the result link in the info section of the task we created, or just by returning to the virtual machine and following the snapshot field. The ‘currentSnapshot’ should be the latest snapshot, the one we just created. You should record the snapshot MoRef id, in this case ’27-snapshot-219′.
When you follow that reference, you will see a ‘config’ field (VirtualMachineConfigInfo), which represents the virtual machine configuration at the time the snapshot was made. If we now go to the backing info of the disk via the same path:
|config > hardware > device[diskid] > backing|
The ‘changeId’ should now be set. In this case it is ’52 09 79 1c f6 fd 45 a0-1c d2 3c 01 26 f9 4c 74/1′. This is our timestamp for the disk we want to monitor.
Record the changeID with each disk ID you want to query. We are now able to do our first CBT query. However, since we only have a point in time reference, we will be able to query only all the blocks that have been changed.
To query CBT, go back to the virtual machine page and find the ‘QueryChangedDiskAreas’ method. In the method change the XML to reference your snapshot.
For the deviceKey, enter the disk ID. For startOffset, you can use ‘0’ to query the complete disk. Finally for the changeId you can enter ‘*”, which in effect is an unspecified time. Because you supplied the snapshot, vSphere can figure out the current chanageId for the disk via the snapshot. That means you always record the changeId for the next query (or next backup) when the previous snapshot is already erased.
Although, technically, ‘snapshot’ is an optional parameter, but only if the virtual machine is powered down.
If you then select ‘Invoke Method’, it will return all the blocks that have changed. In reality, CBT tries to consolidate. For example, if you have 20 blocks that changed sequentially, it will just tell you the starting block and its length of 20 blocks. In the screenshot above, you will actually see a rare case, where the complete disk has been used, and it just shows you one continuous change.
Now go back to your virtual machine page, go to the snapshot section, find the current snapshot and use the ‘RemoveSnapshot_Task’ to delete the snapshot (method is available via the snapshot page, not the virtual machine page). For the method, you can set ‘removeChildren’ to 1, since you should not have any child snapshots and you want to consolidate the disk (this is default). If you set consolidate to ‘0’, you will actually create an orphaned snapshot. Removing this can be done by using the consolidate option in the user interface, or by using the ‘ConsolidateVMDisks_Task’ on the virtual machine (in older versions you can try creating a new snapshot and using the ‘RemoveAllSnapshots_Task’)
Time to make some changes on the disk. In our case, on an almost empty disk we just added a small file. Since it only contains a ‘hello world’ string, it should only trigger changes for a couple of blocks (the file record with the data and some other metadata).
Now lets make a new snapshot. In this test case, the snapshot MOID is ’27-snapshot-222′. Again, write down the change ID for each disk you wish to query. In this case “”52 71 8d 85 58 1f c7 00-27 0a 36 45 32 04 80 f8/5”.
For the final, incremental query, go back to the virtual machine and use the ‘QueryChangedDiskAreas’ method. Now in the snapshot input field, reference the current snapshot. For the changeId, do not fill in the current changeId, but rather the changeId that was associated with the previous snapshot. vSphere will now calculate and tell you all the blocks that have been changed between the changeId and the snapshot information you supplied for the current disk.
As expected, a couple of blocks have changed, of which the smallest is 64kb.
That concludes this overview on how to manually query and check change block tracking data via the Managed Object Browser. This should give a decent perspective as to how VMware’s Changed Block Tracking works under the hood.
In a next post we will discuss what disk operations can make minor changes, but result in a large amount of CBT changes – a Changed Block Tracking data explosion if you will.