Category: Tools

From the cutting room floor: creating a VPC boot volume based on an ISO image

Among the VM to VSI migration techniques I considered, one possible option was to boot the destination VSI using a GHOST-like ISO tool. This is difficult to accomplish because IBM Cloud does not currently support booting a VSI from an ISO. What I did instead was to craft a qcow2 disk image that loaded this ISO. Then, any VSI booted using this image would initially load the tool. Afterwards the tool could be used to transfer the actual disk to be used for the VM.

There are some limitations for this approach. Most importantly, the use of the tool as an image template entirely obscures the actual underlying OS for any VSI booted from that image. It seems to me an abuse of the idea of an image template. Furthermore, given the limitations of many small Linux distributions, customizing the image so that it has all of the packages you need is potentially a tedious process. For example, I found that out of the box TinyCore Linux did not have openssh, qemu, nor did it have virtio drivers. Combined with the fact that I also wanted to leverage libguestfs in many cases, this was a challenging limitation.

As a result, I rejected this approach as one of the viable migration paths. However, it was still a fun experiment to build the boot image and boot my VSI using it. Here are the steps I took to create a TinyCore Linux boot image:

  1. Download Core ISO: http://www.tinycorelinux.net/16.x/x86/release/ and rename it to core.iso
  2. Create raw disk image that will be our UEFI boot image:
    qemu-img create -f raw core.img 64M
  3. Create EFI partition:
    parted core.img --script mklabel gpt
    parted core.img --script mkpart ESP fat32 1MiB 100%
    parted core.img --script set 1 boot on
  4. Attach image as a loopback device:
    LOOPDEV=$(losetup --find --show --partscan core.img)
  5. Format the partition:
    mkfs.vfat "${LOOPDEV}p1"
  6. Install grub:
    mkdir -p /mnt/esp
    mount "${LOOPDEV}p1" /mnt/esp
    grub-install --target=x86_64-efi --efi-directory=/mnt/esp --boot-directory=/mnt/esp/boot --removable --recheck --no-nvram
  7. Edit /mnt/esp/boot/grub/grub.cfg:
    menuentry "Boot Core" {
        set isofile="/boot/core.iso"
        loopback loop $isofile
        linux (loop)/boot/vmlinuz quiet
        initrd (loop)/boot/core.gz
    }
  8. Copy ISO:
    cp core.iso /mnt/esp/boot
  9. Cleanup:
    umount /mnt/esp; losetup -d "$LOOPDEV"
  10. Convert to qcow2:
    qemu-img convert -f raw -O qcow2 core.img core.qcow2
  11. Upload this to COS and create an image as in our migration method #1. I chose generic OS.
  12. Deploy a VSI using this image. Configure profile, volumes, and network as needed.
  13. Display VNC console for your VSI. TinyCore booted with the network properly configured using DHCP. At this point if you had a public gateway attached to the VSI subnet, you could even install extensions like SSH: tce-load -wi openssh, or qemu tools: tce-load -wi qemu.

PowerCLI native key management capabilities, continued

I mentioned previously that PowerCLI allows you to rekey VM and VMHost objects natively without needing to use community-supported extensions. As far as I can tell, rekeying vSAN clusters still requires you to work in the UI or to use the community-supported extensions.

Examining the code for these extensions, I was able to put together a brief way to display the current key manager in use by each object. This way you can verify your rekeying is successful! Here is an example:

$vmlist = @()
foreach($vm in Get-VM) {
  $vmlist += [pscustomobject]@{ vm = $vm.name; provider = $vm.ExtensionData.Config.KeyId.ProviderId.Id}
}
$vmlist | Format-Table

$hostlist = @()
foreach($vmhost in Get-VMHost) {
  $vmhostview = Get-View $vmhost
  $hostlist += [pscustomobject]@{ host = $vmhost.name; provider = $vmhostview.Runtime.CryptoKeyId.ProviderId.Id}
}
$hostlist | Format-Table

$clusterlist = @()
$vsanclusterconfig = Get-VsanView -Id "VsanVcClusterConfigSystem-vsan-cluster-config-system"
foreach($cluster in Get-Cluster) {
  $encryption = $vsanclusterconfig.VsanClusterGetConfig($cluster.ExtensionData.MoRef).DataEncryptionConfig
  $clusterlist += [pscustomobject]@{ cluster = $cluster.name; provider = $encryption.KmsProviderId.Id }
}
$clusterlist | Format-Table

Gmail DMARC failure

I use Fastmail as my email provider. Fastmail provides helpful instructions on how to setup both SPF and DKIM for a custom domain. In spite of properly configuring these, I was surprised recently to see that Google mail was marking my emails as having failed a DMARC check:

I spent some time double checking that my email met the normal conditions for DMARC checks. In my case the From addresses were consistent, and they matched a domain for both the SPF and DKIM check. So, nominally, I am compliant with DMARC expectations.

However, my domain did not have a DMARC policy configured. After configuring a DMARC policy, I found that Google began to treat my emails as passing DMARC:

If you’re encountering the same, you should check that your emails meet the normal DMARC conditions, but also consider configuring a basic DMARC policy for your domain.

Deleting all of the keys in your Key Protect or HPCS instance

It’s a common problem that you want to delete an IBM Cloud Key Protect instance but there are still some keys remaining in that instance. For your protection, Key Protect and Hyper Protect Crypto Services require you to take action to delete those keys rather than allowing you to delete them as a side effect of deleting the Key Protect instance itself.

This is challenging if you have a large number of keys. That may be the case if you have a development or test environment that you are cleaning up, or if you have migrated your keys to another key provider.

It’s possible to script this using the Key Protect CLI.

First, login to IBM Cloud and install the key protect plugin if necessary:

$ ibmcloud login --sso
$ ibmcloud plugin install key-protect -r "IBM Cloud"

If your Key Protect instance is private-only you may need to export the KP_PRIVATE_ADDR environment variable to point to the service endpoint or VPE for Key Protect in your region. Next you need to identify the instance id for your Key Protect instance, which you can find in the instance details tab in the IBM Cloud UI, or by using the following command if you know the instance name:

$ ibmcloud resource service-instance smoonenKPmadrid --id
Retrieving service instance smoonenKPmadrid in all resource groups under account Development Account as smoonen@us.ibm.com...
crn:v1:bluemix:public:kms:eu-es:a/3f1b08d9abdc5d98ffeb0d3bdc279c04:1f8011c9-7fd9-4fe9-af5e-2fefcfda8cfc:: 1f8011c9-7fd9-4fe9-af5e-2fefcfda8cfc

You can save typing or pasting in subsequent commands by exporting the instance id:

$ export KP_INSTANCE_ID=1f8011c9-7fd9-4fe9-af5e-2fefcfda8cfc

The following command displays all of the key ids and names in your instance:

$ ibmcloud kp keys

You can adjust this command to display only the key ids:

$ ibmcloud kp keys --output json | jq -r '.[] | .id'

If you are confident that all of these keys can be safely deleted, and you have the appropriate permissions to do so, in your shell session you can loop through these and issue a delete command for each of them:

$ foreach key in $(ibmcloud kp keys --output json | jq -r '.[] | .id')
foreach> do
foreach> ibmcloud kp key delete $key
foreach> done

If any of the keys is known to be in use by a resource, you will receive an error. You may also receive other errors, for example, if you do not have sufficient permission to delete the key. You’ll have to rectify these issues before you can successfully delete the key and the Key Protect instance. For example, the following key was a root key that was in use by a Key Protect KMIP adapter:

Targeting endpoint: https://eu-es.kms.cloud.ibm.com
Deleting key: 'b262754c-f30d-4b5f-984c-f9c21b7ae13a', from instance: '1f8011c9-7fd9-4fe9-af5e-2fefcfda8cfc'...
FAILED
ASSOCIATED_KMIP_ADAPTER_ERR
The key cannot be deleted because it is associated with 1 KMIP adapter(s) in the instance
Correlation-ID:ef7ae793-945f-4b10-aa4b-f24b340bb3e1

PowerCLI native rekey capabilities

In the past I’ve written about using some PowerCLI extensions from the community repository to rekey individual objects, to rekey all objects, and to migrate from one key provider to another. I’ve recently discovered that the native PowerCLI commands support rekeying of VM and host keys, although not vSAN keys.

It is straightforward to rekey a vSAN cluster using the vCenter UI, and this has the side benefit that it will rekey your host encryption keys as well. But if you want to rekey virtual machines or host encryption keys directly, you could use a script like the following without needing to install the community modules:

$kp = Get-KeyProvider new-kmip

foreach($vm in Get-VM) {
  if($vm.ExtensionData.Config.KeyId) {
    Set-VM $vm -KeyProvider $kp -Confirm:$false
  }
}

foreach($vmhost in Get-VMHost) {
  if($vmhost.ExtensionData.Runtime.CryptoState -eq "safe") {
    Set-VMHost $vmhost -KeyProvider $kp
  }
}

Authenticating with VMware Cloud Director using IBM Cloud IAM

If you are automating activities in VMware Cloud Director—for example, if you are using Terraform to manage your edges and deploy your vApps—you will typically create a Cloud Director API token, which your automation can use to create an authenticated login session with Director for subsequent API calls.

There are interesting complex automation use cases where you might want to create an automation pipeline stretching from the IBM Cloud APIs to the Cloud Director APIs. For example, you might want to use the IBM Cloud APIs to provision a virtual data center (VDC) and then use the Cloud Director APIs—perhaps using Terraform—to deploy a vApp in that VDC. In cases like this, you prefer not to interrupt your automation to create your Cloud Director API token; instead, you want to be able to authenticate with Cloud Director by means of your IBM Cloud API key. Fortunately, that is possible because IBM preconfigures your Director organization with OIDC SSO integration with IBM Cloud IAM.

There are two ways to approach this. Most straightforwardly, if you are a REST API user, you can take the IBM Cloud IAM token that you got in exchange for your IBM Cloud API key, and submit this to Director as an OAuth identity provider token to authenticate a new login session and receive a Director bearer token for that session. You can then use this Director bearer token to make Director API calls for the length of that login session. Alternately, you can further use that Director bearer token to make an API call to create a long-lived Director API token, which you can then provide to tooling like Terraform in order to conduct ongoing management of your VDCs and other Director resources.

I’ve created two sample scripts demonstrating how this works. The first script obtains the Director bearer token and then uses this to call a Director API to list all vApps in each Director instance. Here is an example of its use:

smoonen@smoonen vmware-solutions % python3 get-vapps.py
Site: 'https://dirw003.eu-gb.vmware.cloud.ibm.com' / Organization 'd568ebe2-4042-4bc3-82c2-a3a7935cf9b9'
  vApp: vm1-1adc17be-3a7a-4460-82a8-ce821d3f5612
  vApp: vm2-000a9834-0037-4fc7-b6fd-0b2ec0927a28
Site: 'https://dirw082.us-south.vmware.cloud.ibm.com' / Organization '577fbceb-23ce-4361-bd11-1797931cb69b'
  vApp: auto_vapp
  vApp: VM-WIN-1ebfec4b-d754-4f6c-8ef9-e1adab14900b
Site: 'https://dirw003.ca-tor.vmware.cloud.ibm.com' / Organization '44445dba-16f0-488f-842c-a184f8b1d4e2'
  vApp: vm-1-39534998-c323-4484-9246-df57b258216e
  vApp: vm-2-330f574e-868b-45ae-934f-df007f2a30d8
  vApp: vm-3-3855594d-ce3b-4de7-8a81-8f4dcbc87a5b
Site: 'https://dirw003.us-east.vmware.cloud.ibm.com' / Organization '3bb02c20-e9df-4b39-ab76-94d43567add7'
  vApp: test-2de106b7-9107-40b8-9ec1-2287046df186

Interestingly, IBM Cloud service IDs are also represented in the Director OIDC SSO. You can create a service ID, and provided you have assigned the service ID sufficient IAM permissions to your VCF as a Service resources, you can use an IAM token generated from the service ID’s API key to authenticate with Director and call Director APIs.

IBM Cloud trusted profiles do not support the creation of API keys. However, trusted profiles are allowed to login to Cloud Director. In order to authenticate your trusted profile with Cloud Director (and possibly to create a Director API token) you will need to extract your trusted profile IAM token by other means than exchange of an API key. If you login to your trusted profile using the ibmcloud CLI (or by means of the IBM Cloud shell), you can extract your IAM token by this means:

scott_test@cloudshell:~$ ibmcloud iam oauth-tokens | grep IAM | cut -d \: -f 2 | sed 's/^ *//'
Bearer eyJraWQiOi. . .aZoC_fZQ
scott_test@cloudshell:~$

My second script uses the alternate approach of leveraging the Director bearer token to create a long-lived Director API token, in this case for each Director instance to which your user has access. Here is an example of its use:

smoonen@smoonen vmware-solutions % python3 create-director-tokens.py
Site: 'https://dirw003.eu-gb.vmware.cloud.ibm.com' / Organization 'd568ebe2-4042-4bc3-82c2-a3a7935cf9b9'
  token: leTf. . .TIs5
Site: 'https://dirw002.eu-de.vmware.cloud.ibm.com' / Organization 'ba10c5c7-7e15-41b5-aa4c-84bd373dc2b1'
  token: CL9G. . .IJRY
Site: 'https://dirw003.ca-tor.vmware.cloud.ibm.com' / Organization '44445dba-16f0-488f-842c-a184f8b1d4e2'
  token: p9cx. . .LdGt
Site: 'https://dirw082.us-south.vmware.cloud.ibm.com' / Organization '577fbceb-23ce-4361-bd11-1797931cb69b'
  token: ygc7. . .FVjB
Site: 'https://dirw003.us-east.vmware.cloud.ibm.com' / Organization '3bb02c20-e9df-4b39-ab76-94d43567add7'
  token: UCIf. . .aPBE

The Director APIs to create these long-lived tokens are not well documented. But essentially what is happening here is that we are creating an OAuth client ID and obtaining the refresh token for that client.

Retrieving VCFaaS sites and pVDCs

IBM Cloud offers a terraform provider for our managed VMware Cloud Director offering, VCFaaS, to provision virtual data centers (VDCs).

This provider requires that you input the site id for the Director instance of your choice, as well as a pVDC id for the provider VDC (which we sometimes call resource pool) in which you want to create your VDC. The ids for these are not well known.

I have written a brief Python script to invoke the VCFaaS API and list the sites and pVDCs.

At the time of this writing, these are the multi-tenant sites and pVDCs available for use:

Site 'IBM VCFaaS Multitenant - SYD', ID 1a4bb41e-f3ce-4b1f-bdb9-b0a77cf83f50, in region au-syd
  pVDC 'SYD04', ID be95acb1-ba8f-466b-b796-22dcd648fd15, in location syd04 supporting provider types: on_demand, reserved
  pVDC 'SYD05', ID 1c4ea91b-fb36-4cd8-9ef7-6fd098db59c7, in location syd05 supporting provider types: on_demand, reserved
Site 'IBM VCFaaS Multitenant - TOR', ID 6352e951-70f6-497f-a80f-94dff58c0734, in region ca-tor
  pVDC 'TOR04', ID de4be466-dfb9-48d4-9d28-657657fef571, in location tor04 supporting provider types: on_demand, reserved
  pVDC 'TOR05', ID 132f3818-0060-4e3e-90db-8b117416fb27, in location tor05 supporting provider types: on_demand, reserved
Site 'IBM VCFaaS Multitenant - FRA', ID 1fff1209-55b8-4667-b737-0e6de5cf5756, in region eu-de
  pVDC 'FRA02', ID f34e7a9f-afb4-430e-86d8-6b1978aebb9c, in location fra02 supporting provider types: on_demand, reserved
  pVDC 'FRA04', ID c66ac0a4-5e52-4b5e-9387-6bb8de7e42b1, in location fra04 supporting provider types: on_demand, reserved
Site 'IBM VCFaaS Multitenant - MAD', ID 8cef7547-3e44-4288-b849-1cd748e7d954, in region eu-es
  pVDC 'MAD02', ID e917160f-54b0-4a01-ac59-f9e23f44e8bd, in location mad02 supporting provider types: on_demand, reserved
Site 'IBM VCFaaS Multitenant - LON', ID f7074684-de9c-42e9-93a5-0358fcb2bf92, in region eu-gb
  pVDC 'LON04', ID 25e910c2-c790-4550-a316-03efa7c29888, in location lon04 supporting provider types: on_demand, reserved
  pVDC 'LON06', ID a24bbff9-8c79-44ab-8bb1-7699f304a1d1, in location lon06 supporting provider types: on_demand, reserved
Site 'IBM VCFaaS Multitenant - TOK', ID 3dca2ae4-fdea-4791-8092-0c879daa2097, in region jp-tok
  pVDC 'TOK02', ID 029c7ff7-da97-48ea-8ed9-0b86c2918f82, in location tok02 supporting provider types: on_demand, reserved
  pVDC 'TOK04', ID 08b78261-d4c9-41e9-baa9-421b493385f7, in location tok04 supporting provider types: on_demand, reserved
Site 'IBM VCFaaS Multitenant - WDC', ID 25fb5553-72a6-49ca-85b8-f18086cbac0b, in region us-east
  pVDC 'WDC06', ID 595d7f76-b1b2-429f-bb67-e357befa9da7, in location wdc06 supporting provider types: on_demand, reserved
  pVDC 'WDC07', ID efc17eff-fb3e-4b56-8252-44657738c539, in location wdc07 supporting provider types: on_demand, reserved
  pVDC 'WDC04-WDC07', ID 9a46dcce-84d1-4fca-a929-42bc2174ffba, in location wdc04 supporting provider types: on_demand, reserved
Site 'IBM VCFaaS Multitenant - DAL', ID 40e701cd-ef86-4d5e-a847-e7c336f11f27, in region us-south
  pVDC 'DAL10', ID f864f016-2c34-4658-95e7-dd3363408d76, in location dal10 supporting provider types: on_demand, reserved
  pVDC 'DAL13', ID 1df84086-b3af-481e-a535-e3554b809aed, in location dal13 supporting provider types: on_demand, reserved
  pVDC 'DAL12', ID 5b4af31c-383b-422b-b297-0ea6ed8af479, in location dal12 supporting provider types: on_demand, reserved

Estimating vDefend firewall usage

Here are a few key things to know about Broadcom’s vDefend firewall offering:

  • vDefend comes in three flavors: firewall, ATP, and firewall+ATP bundle. The ATP license is not designed to be used on its own, but only to stack ATP capability on top of firewall capability in cases where you have not purchased the bundle.
  • If you are using a vSphere 8 VCF solution key to activate NSX, you will need a vDefend “solution key” to activate vDefend features. Otherwise, if you are using a v7 or v8 component key to activate NSX, you will need to use separate vDefend “component keys” to activate the vDefend features on edges and in distributed firewall, respectively. For more information, see KB 318444.
  • Broadcom assesses vDefend for distributed firewall against the vSphere hosts in the same manner as VCF (i.e., with a floor of 16 cores per CPU). For distributed firewall you should order as many cores of vDefend as for VCF. For edge firewall, Broadcom assesses vDefend at a ratio of 4 cores per vCPU (including passive edge VMs). There is not a technical reason for this ratio; it is simply a business decision on Broadcom’s part.
  • If you are running NSX 4.1 or newer, Broadcom has recently published a script that you can use to survey your environment’s firewall configuration to measure how much vDefend licensing you need; see KB 395111. This script correctly takes into account some cases where Broadcom does not assess vDefend usage; for example, if gateway firewall is enabled but only the default permit-all rule is configured, or only stateless rules are configured.

KMIP test client

I created a simple test client for the KMIP protocol leveraging the pykmip Python package.

My purpose in doing so was to enable some simple performance testing; the script generates a continuous series of connections and requests to a KMS.

In the process I discovered that pykmip has not shipped a new release in several years, and is not compatible with recent versions of Python. I included a simple monkey patch which adapts to recent changes in the SSL implementation.

Customizing root login for VMware Cloud Director

Your Linux VM running in VMware Cloud Director might be preconfigured with the best-practice configuration to disable root password login. This might prevent you from using the root password that you set with Director’s Guest OS Customization:

#PermitRootLogin prohibit-password

You can override this behavior using a Guest OS Customization script in a couple of ways. The simplest approach is to use your customization script to set the sshd configuration to allow root password logins:

#!/bin/bash
sed -ie "s/#PermitRootLogin prohibit-password/PermitRootLogin yes/" /etc/ssh/sshd_config

Or, if you prefer, you can use the customization script to insert an SSH public key for the root user:

#!/bin/bash
echo "ssh-rsa AAAAB3...DswrcTw==" >> /root/.ssh/authorized_keys
chmod 644 /root/.ssh/authorized_keys