Updated VMware Solutions API samples

It’s been awhile since I first posted sample IBM Cloud for VMware Solutions API calls. Since then, our offering has moved from NSX–V to NSX–T, and to vSphere 7.0. This results in some changes to the structure of the API calls you need to make for ordering instances, clusters, and hosts.

I’ve updated the sample ordering calls on Github. This includes the following changes:

  • Migrate some of the utility APIs to version 2
  • Send transaction ids with each request to aid in problem diagnosis
  • Order NSX–T and vSphere 7.0 instead of NSX–V and vSphere 6.7
  • Restructure some of the ordering code to order a concrete example instead of a random one
  • Use the new price check parameter to obtain price estimates
  • Ensure that the add–cluster path leverages an existing cluster’s location and VLANs

Highly available key management in IBM Cloud for VMware Solutions

IBM Cloud’s KMIP for VMware offering provides the foundation for cloud-based key management when using VMware vSphere encryption or vSAN encryption. KMIP for VMware is highly available within a single region:

  • KMIP for VMware and Key Protect are highly available when you configure vCenter connections to both regional endpoints. If any one of the three zones in that region fail entirely, key management continues to be available to your VMware workloads.
  • KMIP for VMware and Hyper Protect Crypto Services (HPCS) are highly available if you deploy two or more crypto units for your HPCS instance. If you do so and any one of the three zones in that region fail entirely, key management continues to be available to your VMware workloads.

If you need to migrate or failover your workloads outside of a region, your plan depends on whether you are using vSAN encryption or vSphere encryption:

When you are using vSAN encryption, each site is protected by its own key provider. If you are using vSAN encryption to protect workloads that you replicate between multiple sites, you must create separate KMIP for VMware instances in each site, that are connected to separate Key Protect or HPCS instances in those sites. You must connect your vCenter Server in each site to the local KMIP for VMware instance as its key provider.

When you are using vSphere encryption, most VMware replication and migration techniques today (for example, cross-vCenter vMotion and vSphere replication) rely on having a common key manager between the two sites. This topology is not supported by KMIP for VMware. Instead, you must create separate KMIP for VMware instances in each site, that is connected to separate Key Protect or HPCS instances in those sites. You must connect your vCenter server in each site to the local KMIP for VMware instance as its key provider, and then use a replication technology that supports the attachment and replication of decrypted disks.

Veeam Backup and Replication supports this replication technique. To implement this technique correctly, see the steps that you must take as indicated in the Veeam documentation.

Note that this technique currently does not support the replication of virtual machines with a vTPM device.

How to bring your team back to the office (3)


I think of my job often as being a translator between executives, managers, architects, developers, testers, customers, writers, etc. My favorite work projects have been those we conducted war-room style or in an open landscape, yet now it is almost two years since I’ve been in the office. We’ve filled in the gap a little bit with some team outings. Today I went in to the office to collect my belongings, before my vaxx-leper status kicks in and my physical access is deactivated. This is such a stark contrast with my experience at church where we worked hard to find some way to meet, at times even with our fussy government’s disapproval. What a joy and encouragement that fellowship was, and what a missed opportunity these two years have been for camaraderie at so many anxious companies and churches!

See also: How to bring your team back to the office (2)

Rekeying all of your VMware objects

We saw previously that we could use PowerCLI to rekey objects to a different key provider. It is much more common that you simply want to rekey objects within the same key provider, perhaps to meet a compliance requirement. We can use the same set of commands without specifying a key provider to perform rekey operations.

The simplest and fastest of the three is a vSAN rekey, which only needs to reissue one root key for each cluster protected by vSAN encryption:

PS C:\Users\Administrator> Invoke-VsanEncryptionRekey -Cluster cluster1 -DeepRekey $false
Executing shallow rekey of vSAN Cluster cluster1
PS C:\Users\Administrator>

This performs a shallow rekey. You can perform a deep rekey by changing $false to $true. This will take much longer to complete.

We can also rekey each of our VMs that is protected by vSphere encryption, as follows:

PS C:\Users\Administrator> foreach($myvm in Get-VM){
>>  if($myvm.KMSserver){
>>   echo $myvm.name
>>   Set-VMEncryptionKey -VM $myvm
>>  }
>> }

Type Value
---- -----
Task task-23093

PS C:\Users\Administrator>

This took a couple minutes to complete for each VM. You can perform a deep rekey—which will take longer to complete—by adding the -Deep parameter to the Set-VMEncryptionKey cmdlet.

Finally, if you wish to rekey the host encryption keys used to protect core dumps, you can run the following:

PS C:\Users\Administrator> foreach($myhost in Get-VMHost){
>>  echo $myhost.name
>>  Set-VMHostCryptoKey -VMHost $myhost
>> }
PS C:\Users\Administrator>

This took a few minutes to complete for each host. There is no notion of deep rekeying for host encryption keys.

Active Directory and SSO integration for VMware Solutions in IBM Cloud

VMware Solutions instances in IBM Cloud are deployed with a built-in Active Directory domain with one or two directory controllers. Recently IBM Cloud changed the domain name requirements to require three qualifiers (e.g., cloud.example.com) rather than two (e.g., example.com). The reason for this is that we want to ensure you can integrate with your existing domain and forest without experiencing conflict. The domain controllers are configured as SSO provider for vCenter and NSX, and also as DNS provider for the infrastructure components. IBM Cloud creates an administrator userid in this domain which it uses for subsequent operations, such as logging into vCenter to add a new host, updating DNS records for that host, and creating utility accounts for add-on services like Veeam.

This Active Directory domain is your responsibility to secure and manage, including backup, patching, group policy, etc.

In order of integration from loosest to tightest coupling:

1. No integration

You are free to leverage your instance domain directly for user management within the instance. You can point additional components to the instance’s domain controllers for SSO; for example, the IBM Cloud automation does this for you when it deploys and configures HyTrust Cloud Control. You can join other devices to the domain and also use this for DNS management beyond the instance infrastructure.

2. Additional SSO provider

This option and all of the following options each entail some kind of integration with your instance and your existing Active Directory forest. You will first need to establish network connectivity between your instance and your existing Active Directory forest. You might accomplish this with either a VPN connection or a direct link between IBM Cloud and your on-premises environment. As always, you should take great care to secure your domain controllers, so you should explore security measures such as the use of read-only directory controllers, session recording, bastion servers, and gateway firewalls.

You can leverage your own Active Directory domain for SSO purposes by configuring your directory controllers as additional SSO providers for vCenter and NSX manager and by granting your users and groups appropriate permissions. You will need to determine how you configure DNS; some customers manually duplicate the DNS records from their instance domain into their existing Active Directory domain, but it is also possible to establish mutual DNS delegation between the two Active Directory domains.

This approach may allow you to limit the cloud connections to your directory controllers so that you are only opening up LDAPS and DNS ports.

3. One-way trust

You can establish one-way trust from your instance’s Active Directory domain controllers to your existing Active Directory domain. This will enable you to expose and authorize your existing users and groups to vCenter and NSX manager without having to add these directly as SSO providers. You may need to make additional provision for DNS updates, either copying them to your existing domain or establishing DNS delegation to the instance’s domain.

4. Two-way trust

This option requires your existing domain to establish mutual trust with your instance’s domain. If you are comfortable doing this, it could simplify your DNS management between the two domains.

5. Forest merge

I am not aware of any IBM Cloud customers who have done this, and I do not recommend it since it is a disruptive and potentially risky operation. The idea here is to merge the instance’s forest with your existing forest and to configure the instance’s domain as a child domain of your existing domain.

6. Rebuild

IBM Cloud’s VMware Solutions Shared offering implements a variation of the forest merge. It deploys VCS instances and builds VMware Cloud Director environments on top of them. This solution leverages an existing internal Active Directory forest and domain. After each new VCS instance is deployed, our process removes the VCS instance from its domain and reconfigures it to point to the existing domain.

A variation of this option is to create a new child domain in your existing forest for your VCS instance, and leverage the controllers for this child domain for use with your VCS instance.

There are a few important points to observe:

  1. You should either deploy your instance with the same domain name that you intend to convert it to, or else you should accept the fact that your infrastructure components will have host names in a different DNS domain from your Active Directory domain. Changing the DNS domain of infrastructure components is not supported by IBM Cloud automation.
  2. You will need to re-create the IBM Cloud automation user in your existing domain as an administrator and ensure that this user has administrative permissions in vCenter and NSX manager. This user may in the future create additional users or DNS entries. After performing the reconfiguration, you should open a support ticket to the VMware Solutions team asking them to update the automation user’s password in the IBM Cloud database for your instance, and provide the updated password.

Because this process is complex it is error prone, and you should consider this option only if the options above do not work for you. Additionally, you should practice this with a non-production or pre-production VCS deployment, including the test of adding a new host to the environment, before you implement it in production.

Complex (2)

I have generally found NoSQL to be a disaster. Like agile processes, it allows you to dispense with certain disciplines, but for use at scale and over time it requires you to engage in substitute disciplines. Too often these are not practiced. From a recent work chat with minor adaptation:

Data hygiene is crucial. I wouldn’t be opposed to broader NoSQL/JSON use if we used JSON schemas wherever appropriate, but at that point it is probably simpler to flatten the data into tables.

A good schema is a species of defensive code; e.g., you can have higher confidence that the value you are reaching for is actually there no matter how old the document.

See also: Complex