Running vCloud Usage Meter in IBM Cloud

Running vCloud Usage Meter in IBM Cloud

In 2024, Broadcom simplified VMware product pricing and packaging. The VMware Cloud Foundation (VCF) offering now encompasses a wide variety of VMware software and features, with a relatively smaller number of software and features being sold as add-ons. As part of this simplification, Broadcom required all customers and cloud providers to make new commitments and to create new license keys.

Cloud providers are uniquely entitled for on-demand licensing of VMware products beyond their contract commitment. In exchange for this benefit, Broadcom expects that the vCloud Usage Meter product “must be used” to monitor and report usage of VMware products. IBM secured an extension of this requirement so that we could update our automation and develop integration points for our customers. IBM has now released updated VMware license keys and Usage Meter support, and IBM’s customers are expected by Broadcom—and therefore by IBM—to “immediately” install these in order to remain entitled to VMware software. . . . read more at Updates to VMware license keys and the use of vCloud Usage Meter in IBM Cloud

Using vSphere Trust Authority to geofence workloads

IBM and Kyndryl have in the past used Entrust BoundaryControl to accomplish geofencing. This worked using a combination of their CloudControl and KeyControl products. The CloudControl product was used by security administrators to install cryptographically signed tags into known trusted host TPMs, and then to describe policies for virtual machines that required them to run on hosts with particular tags. In addition to CloudControl enforcing virtual machine placement, the KeyControl product further integrated with this configuration to ensure that virtual machines running on unapproved hosts could not be successfully decrypted and run. Customers could devise tagging schemes according to their needs, such as prod/nonprod, tier1/tier2, and US/EU.

You can accomplish a similar kind of exclusion or geofencing capability using VMware’s vSphere Trust Authority. Although vTA is designed primarily as a means of ensuring that workloads run on hosts with known trusted firmware and software levels, it also has the capability to trust hosts individually. Rather than trusting the vendor TPM CA certificate, you can trust individual host TPM certificates. This allows you to vet the hosts one by one in your environment, and mark them as trusted only if they meet your criteria, including their geographic location. vTA will then help to ensure that the virtual machines in your environment cannot be successfully decrypted and run on hosts outside of your trusted set.

Like any security solution, attestation and geofencing solutions like BoundaryControl and vTA require extra effort to configure and to administrate. In exchange for this effort, however, you can create compelling sovereign cloud solutions.

Estimating vDefend firewall usage

Here are a few key things to know about Broadcom’s vDefend firewall offering:

  • vDefend comes in three flavors: firewall, ATP, and firewall+ATP bundle. The ATP license is not designed to be used on its own, but only to stack ATP capability on top of firewall capability in cases where you have not purchased the bundle.
  • If you are using a vSphere 8 VCF solution key to activate NSX, you will need a vDefend “solution key” to activate vDefend features. Otherwise, if you are using a v7 or v8 component key to activate NSX, you will need to use separate vDefend “component keys” to activate the vDefend features on edges and in distributed firewall, respectively. For more information, see KB 318444.
  • Broadcom assesses vDefend for distributed firewall against the vSphere hosts in the same manner as VCF (i.e., with a floor of 16 cores per CPU). For distributed firewall you should order as many cores of vDefend as for VCF. For edge firewall, Broadcom assesses vDefend at a ratio of 4 cores per vCPU (including passive edge VMs). There is not a technical reason for this ratio; it is simply a business decision on Broadcom’s part.
  • If you are running NSX 4.1 or newer, Broadcom has recently published a script that you can use to survey your environment’s firewall configuration to measure how much vDefend licensing you need; see KB 395111. This script correctly takes into account some cases where Broadcom does not assess vDefend usage; for example, if gateway firewall is enabled but only the default permit-all rule is configured, or only stateless rules are configured.

Fixed: Unexpected reboot when rekeying a virtual machine

vSphere 8.0u3e (see release notes) fixes the issue where a rekeyed VM may experience an unexpected reboot:

PR 3477772: Encrypted virtual machines with active Change Block Tracking (CBT) might intermittently power off after a rekey operation

Due to a race condition between the VMX and hostd services for a specific VMcrypt-related reconfiguration, encrypted VMs with active CBT might unexpectedly power off during a rekey operation.

This issue is resolved in this release.

From what I can tell, this issue is not currently fixed in the vSphere 7 stream.

Unexpected reboot when rekeying a virtual machine

In recent builds of vSphere 7 and vSphere 8, my team has experienced unexpected spontaneous reboots of virtual machines while rekeying them. In our case we were rekeying these machines against a new key provider.

EDIT: Broadcom support has now published KB 387897 documenting this issue. The issue is a kind of race condition between the rekey task and some other activity that is touching the changed block tracking (CBT) file for the virtual machine. Under some conditions the latter activity fails to open the CBT file, and vSphere HA reboots the virtual machine.

The reboots seem unpredictable. Although we are using CBT for backup, we had no in-flight backup job running at the time (since you cannot rekey a virtual machine with snapshots). At times as few as 1% of the rekeyed machines were spontaneously rebooted, but at other times as high as 20% were affected.

We understand that Broadcom will fix this race condition in a future release, but in the meantime if you plan to rekey a virtual machine that is using CBT for backup or replication, you should either:

  1. Perform an orderly shutdown of the virtual machine if you cannot tolerate a spontaneous reboot, or
  2. Disable CBT for the duration of the rekey. You need to evaluate whether your BCDR software can tolerate this, or if you need to perform a full backup or replication to recover from the loss of CBT.