Customizing root login for VMware Cloud Director

Your Linux VM running in VMware Cloud Director might be preconfigured with the best-practice configuration to disable root password login. This might prevent you from using the root password that you set with Director’s Guest OS Customization:

#PermitRootLogin prohibit-password

You can override this behavior using a Guest OS Customization script in a couple of ways. The simplest approach is to use your customization script to set the sshd configuration to allow root password logins:

#!/bin/bash
sed -ie "s/#PermitRootLogin prohibit-password/PermitRootLogin yes/" /etc/ssh/sshd_config

Or, if you prefer, you can use the customization script to insert an SSH public key for the root user:

#!/bin/bash
echo "ssh-rsa AAAAB3...DswrcTw==" >> /root/.ssh/authorized_keys
chmod 644 /root/.ssh/authorized_keys

vSAN sizer

I find that there are several commonly overlooked considerations when sizing a vSAN environment:

  • It is not recommended to operate a vSAN environment at over 70% capacity
  • If you use a resilience strategy of FTT=1, you should plan to perform a full evacuation during host maintenance or else you will be at risk of data loss due to drive failure during maintenance. Depending on your configuration and usage, the time required for a full evacuation can easily take 24 hours or more. In addition, a maintenance strategy of full evacuation requires you to leave one host’s worth of capacity empty.
  • Because of these considerations, I recommend a resilience strategy of FTT=2. With this strategy you have the option of performing host maintenance using ensure-accessibility rather than full evacuation, which is much faster but is still resilient to one failure during maintenance.
  • If you size your environment strictly to the minimum number of nodes for your configuration, then you will fail to create virtual machines or snapshots during host maintenance—including any snapshots used for backups or replication—unless you force provisioning of the object contrary to the storage policy. For this reason, you should consider provisioning at least one more host than is strictly required.

Many of these considerations are summarized in this helpful VMware blog post, which includes a helpful table documenting host minimums and RAID ratios: Adaptive RAID-5 Erasure Coding with the Express Storage Architecture in vSAN 8.

I’ve taken these considerations and created a vSAN sizer Excel workbook, to help both with planning and sizing a vSAN environment.

VMware encryption: leaked keys and key inventory

VMware vSphere generally leaks keys when objects are deleted or decrypted. The reason for doing so, I believe, is because VMware supposes that you might have a backup copy of the object and may need the key in the future to restore that object. For example, consider the case of a VM that is removed from inventory but remains on disk. VMware cannot know whether you will permanently delete this VM or add it back to inventory. Therefore VMware allows the key to remain in your key provider.

Over time this results in the growth of unused keys in your key provider. In order to clean up unused keys, you first need to inventory the keys that are in use by active objects. The following PowerCLI script uses the VMware.VMEncryption and VMware.VsanEncryption modules in VMware’s PowerCLI community repository. It will inventory all keys in use by your hosts (for core dumps), in use by vSAN clusters (for vSAN disk encryption), and in use by VMs and disks (for vSphere encryption).

$keydata = @()

# Collect host keys
foreach($myhost in Get-VMHost) {
  if($myhost.CryptoSafe) {
    $hostdata = [PSCustomObject]@{
      type        = "host"
      name        = $myhost.Name
      keyprovider = $myhost.KMSserver
      keyid       = $myhost.ExtensionData.Runtime.CryptoKeyId.KeyId
    }
    $keydata += $hostdata
  }
}

# collect vSAN keys
foreach($mycluster in Get-Cluster) {
  $vsanClusterConfig = Get-VsanView -Id "VsanVcClusterConfigSystem-vsan-cluster-config-system"
  $vsanEncryption    = $vsanClusterConfig.VsanClusterGetConfig($mycluster.ExtensionData.MoRef).DataEncryptionConfig

  if($mycluster.vSanEnabled -and $vsanEncryption.EncryptionEnabled) {
    $clusterdata = [PSCustomObject]@{
      type        = "cluster"
      name        = $mycluster.Name
      keyprovider = $vsanEncryption.kmsProviderId.Id
      keyid       = $vsanEncryption.kekId
    }
    $keydata += $clusterdata
  }
}

# collect VM and disk keys
foreach($myvm in Get-VM) {
  if($myvm.encrypted) {
    $vmdata = [PSCustomObject]@{
      type        = "vm"
      name        = $myvm.Name
      keyprovider = $myvm.KMSserver
      keyid       = $myvm.EncryptionKeyId.KeyId
    }
    $keydata += $vmdata
  }

  foreach($mydisk in Get-HardDisk -vm $myvm) {
    if($mydisk.encrypted) {
      $diskdata = [PSCustomObject]@{
        type        = "harddisk"
        name        = $myvm.Name + " | " + $mydisk.Name
        keyprovider = $mydisk.EncryptionKeyId.ProviderId.Id
        keyid       = $mydisk.EncryptionKeyId.KeyId
      }
      $keydata += $diskdata
    }
  }
}

$keydata | Export-CSV -Path keys.csv -NoTypeInformation 

There are some important caveats to note:

  1. This script is over-zealous; it may report that a key is in use multiple times (e.g., host encryption keys shared by multiple hosts, or VM encryption keys shared by the disks of a VM).
  2. Your vCenter may be connected to multiple key key providers. Before deleting any keys, take care that you identify which keys are in use for each key provider.
  3. You may have multiple vCenters connected to the same key provider. Before deleting any keys, take care to collect inventory across all vCenters and any other clients connected to each key provider.
  4. As noted above, you may have VM backups or other resources that are still dependent on an encryption key, even after that resource has been deleted. Before deleting any keys, take care to ensure you have identified which keys may still be in use for your backups.
  5. This script does not address the case of environments using VMware vSphere Trust Authority (vTA).
  6. Importantly, this script does not address the case of “first-class disks,” or what VMware Cloud Director calls “named disks.”

Authenticating with the SoftLayer API using IBM Cloud IAM

Traditionally you authenticate with the IBM Cloud SoftLayer “classic infrastructure” API using a SoftLayer or “classic infrastructure” API key. However, IBM Cloud has introduced support to authenticate with these APIs using the standardized IAM API keys and identities. At one point IBM implemented a method to exchange IAM credentials for an IMS token, but IBM’s Martin Smolny writes more recently that the classic APIs now “support IAM tokens directly.”

I’ve written a brief script to demonstrate this approach. The script first calls the IAM token API to exchange an API key for an IAM token. Then it constructs a SoftLayer API client object that leverages this token for authentication. Note that for the Python SDK, some paths which create an API client will use the XMLRPC API endpoint by default instead of using the REST API endpoint. The XMLRPC API does not fully support IAM-based authentication. The method used in this script leverages the REST API endpoint and transport, which does support IAM-based authentication.

Updated VMware Solutions API samples

It’s been awhile since I first posted sample IBM Cloud for VMware Solutions API calls. Since then, our offering has moved from NSX–V to NSX–T, and to vSphere 7.0. This results in some changes to the structure of the API calls you need to make for ordering instances, clusters, and hosts.

I’ve updated the sample ordering calls on Github. This includes the following changes:

  • Migrate some of the utility APIs to version 2
  • Send transaction ids with each request to aid in problem diagnosis
  • Order NSX–T and vSphere 7.0 instead of NSX–V and vSphere 6.7
  • Restructure some of the ordering code to order a concrete example instead of a random one
  • Use the new price check parameter to obtain price estimates
  • Ensure that the add–cluster path leverages an existing cluster’s location and VLANs

Provisioning and expanding an IBM Cloud VMware instance via API

IBM Cloud for VMware Solutions recently released a set of public APIs. These APIs allow you to use your IBM Cloud API key to perform operations such as:

  • Get information about your vCenter instance, admin credentials, deployment history, clusters, and hosts
  • Verify parameters for ordering a new vCenter instance, cluster, or hosts
  • Order or remove a vCenter instance, cluster, or hosts

I’ve written some sample code demonstrating how you can authenticate with the IBM Cloud APIs using your API key, and how to interact with the IBM Cloud for VMware APIs. Note that these samples only perform order verification, but you can easily extend them to perform actual orders or removals.

A key use case for these APIs is to expand and contract your VMware instance based on utilization or for workload bursting scenarios. With these APIs, you can now fully automate this process.

Spectrum Protect Plus on IBM Cloud

Spectrum Protect Plus on IBM Cloud

IBM Cloud for VMware Solutions recently made available IBM Spectrum Protect Plus as part of our family of VMware offerings. Spectrum Protect Plus provides powerful and easy to use backup and restore capabilities for your VMware infrastructure and workload. It is now the default backup offering for VMware on IBM Cloud, complementing our existing offering of Veeam Backup & Replication.

At the same time, the IBM Cloud architecture team just published our Spectrum Protect Plus on IBM Cloud reference architecture. Read it and the associated references for information on how we have deployed Spectrum Protect Plus, how you should plan and size your deployment, and how to manage it.