Customizing root login for VMware Cloud Director

Your Linux VM running in VMware Cloud Director might be preconfigured with the best-practice configuration to disable root password login. This might prevent you from using the root password that you set with Director’s Guest OS Customization:

#PermitRootLogin prohibit-password

You can override this behavior using a Guest OS Customization script in a couple of ways. The simplest approach is to use your customization script to set the sshd configuration to allow root password logins:

#!/bin/bash
sed -ie "s/#PermitRootLogin prohibit-password/PermitRootLogin yes/" /etc/ssh/sshd_config

Or, if you prefer, you can use the customization script to insert an SSH public key for the root user:

#!/bin/bash
echo "ssh-rsa AAAAB3...DswrcTw==" >> /root/.ssh/authorized_keys
chmod 644 /root/.ssh/authorized_keys

vSAN sizer

I find that there are several commonly overlooked considerations when sizing a vSAN environment:

  • It is not recommended to operate a vSAN environment at over 70% capacity
  • If you use a resilience strategy of FTT=1, you should plan to perform a full evacuation during host maintenance or else you will be at risk of data loss due to drive failure during maintenance. Depending on your configuration and usage, the time required for a full evacuation can easily take 24 hours or more. In addition, a maintenance strategy of full evacuation requires you to leave one host’s worth of capacity empty.
  • Because of these considerations, I recommend a resilience strategy of FTT=2. With this strategy you have the option of performing host maintenance using ensure-accessibility rather than full evacuation, which is much faster but is still resilient to one failure during maintenance.
  • If you size your environment strictly to the minimum number of nodes for your configuration, then you will fail to create virtual machines or snapshots during host maintenance—including any snapshots used for backups or replication—unless you force provisioning of the object contrary to the storage policy. For this reason, you should consider provisioning at least one more host than is strictly required.

Many of these considerations are summarized in this helpful VMware blog post, which includes a helpful table documenting host minimums and RAID ratios: Adaptive RAID-5 Erasure Coding with the Express Storage Architecture in vSAN 8.

I’ve taken these considerations and created a vSAN sizer Excel workbook, to help both with planning and sizing a vSAN environment.

GitHub printability

Certain complex GitHub Markdown documents don’t render well—or especially, print well—with the whitespace gutters on either side of the screen. This is especially true when there are complex tables in a document.

I use the following userContent.css stylesheet in my browser to override the use of gutters. This renders complex tables much better:

@-moz-document domain(github.com) {
 .container-lg { max-width: none !important; }
 .container-xl { max-width: none !important; }
}

Some tables are so complex that they still spill off the edge of the page even in portrait mode. To counteract this I adjust the print scaling, or print in landscape mode, or both.

Migration from VMware Shared to VMWaaS

IBM Cloud recently announced the end of sale of new VMware Shared environments, with a planned end of support date for all VMware Shared environments of January 2025.

VMware Shared is IBM Cloud’s first-generation IBM-managed VMware Cloud Director offering. It was based on VMware’s legacy NSX-V network virtualization technology. VMware-as-a-Service (VMWaaS) is IBM Cloud’s next-generation IBM-managed VMware Cloud Director offering. It leverages NSX-T network virtualization and provides other advanced capabilities such as the ability to create dedicated environments whose hosts are not shared with other tenants.

Refer to IBM’s announcement for details on how to migrate between these environments and for minor changes in pricing.

vCenter key provider client certificates

When you are configuring a standard key provider in VMware vCenter, you can authenticate vCenter to the key provider using any of the following options:

  • vCenter Root CA Certificate
  • vCenter Certificate
  • Upload [a custom] certificate and private key
  • New Certificate Signing Request (CSR) [to be processed by the key provider]

I commonly choose “vCenter certificate.” This uses a certificate signed by the vCenter CA. The certificate is generated specifically for use with KMIP and it has a 10-year expiration.

Importantly, a new certificate is generated by vCenter for each key provider that you configure.

Furthermore, if you cancel the trust process before completing it, vCenter will generate a new certificate the next time you perform the trust process. I’ve been bitten by this in the past—I generated a certificate, cancelled the dialog, and sent the certificate to my cryptographic administrator. When I received confirmation the certificate had been configured, I re-initiated the trust process, but this time it used a new certificate. This took quite some time to debug. Make sure that you complete the trust process even if you expect there to be a waiting period before the certificate is configured in your key provider!

Updated instructions for multipath iSCSI in IBM Cloud

Several years ago I blogged detailed instructions to configure multipath iSCSI in IBM Cloud’s classic infrastructure using Endurance block storage.

Since then I’ve learned that VMware documents you should not use port binding in this topology. I was skeptical of this since I wasn’t confident that an HBA scan would make attempts on all vmkernel ports. However, I’ve retested my instructions without using port binding and I can confirm that I’m able to achieve MPIO connectivity to the storage without it.

I’ve updated the my instructions to remove the port binding step.

Firefox exits full-screen mode when you press Esc

I’m switching back from Google Chrome to Firefox as my primary web browser. One thing that annoys me with Firefox is that it exits full-screen mode when you press the Escape key.

You can change this behavior with the following steps:

  1. Type about:config in the address bar to access Advanced Preferences
  2. If Firefox displays a warning message, click “Accept the Risk and Continue”
  3. In the search box, type “escape”
  4. Firefox should display a configuration item named browser.fullscreen.exit_on_escape
  5. If this item’s value is false, you are good to go. If it is true, click the toggle button (⇌) on the right hand side to set it to false.
  6. Close the Advanced Preferences tab.

Failure to decrypt VM or disk

You want to be upgrading your vCenter to 7.0u3o anyway because of VMSA-2023-0023.

However, you may also want to upgrade to this version if you are using vSphere encryption. I have found that some earlier versions of vCenter 7.0u3 may at times fail to decrypt VMs or decrypt disks. This seems to occur when moving a VM from one host to another, when starting a stopped VM, and when creating a snapshot. I’m not sure what the cause of this error is; in our case it seemed to happen for recently rekeyed VMs, and I hypothesize that it occurred in cases where the rekey succeeded but where it took the key provider a long time to generate the key.

Initially we were able to recover from this state by attempting to motion VMs to alternate hosts until successful. However, VMware support recommended we upgrade to 7.0u3o, and we haven’t seen the problem since then. There is a relevant release note in 7.0u3o referring to a failure to “apply an encryption storage policy to a virtual machine,” and I believe this is related to the issue we saw.