Author: Scott Moonen

PowerCLI native key management capabilities, continued

I mentioned previously that PowerCLI allows you to rekey VM and VMHost objects natively without needing to use community-supported extensions. As far as I can tell, rekeying vSAN clusters still requires you to work in the UI or to use the community-supported extensions.

Examining the code for these extensions, I was able to put together a brief way to display the current key manager in use by each object. This way you can verify your rekeying is successful! Here is an example:

$vmlist = @()
foreach($vm in Get-VM) {
  $vmlist += [pscustomobject]@{ vm = $vm.name; provider = $vm.ExtensionData.Config.KeyId.ProviderId.Id}
}
$vmlist | Format-Table

$hostlist = @()
foreach($vmhost in Get-VMHost) {
  $vmhostview = Get-View $vmhost
  $hostlist += [pscustomobject]@{ host = $vmhost.name; provider = $vmhostview.Runtime.CryptoKeyId.ProviderId.Id}
}
$hostlist | Format-Table

$clusterlist = @()
$vsanclusterconfig = Get-VsanView -Id "VsanVcClusterConfigSystem-vsan-cluster-config-system"
foreach($cluster in Get-Cluster) {
  $encryption = $vsanclusterconfig.VsanClusterGetConfig($cluster.ExtensionData.MoRef).DataEncryptionConfig
  $clusterlist += [pscustomobject]@{ cluster = $cluster.name; provider = $encryption.KmsProviderId.Id }
}
$clusterlist | Format-Table

Gmail DMARC failure

I use Fastmail as my email provider. Fastmail provides helpful instructions on how to setup both SPF and DKIM for a custom domain. In spite of properly configuring these, I was surprised recently to see that Google mail was marking my emails as having failed a DMARC check:

I spent some time double checking that my email met the normal conditions for DMARC checks. In my case the From addresses were consistent, and they matched a domain for both the SPF and DKIM check. So, nominally, I am compliant with DMARC expectations.

However, my domain did not have a DMARC policy configured. After configuring a DMARC policy, I found that Google began to treat my emails as passing DMARC:

If you’re encountering the same, you should check that your emails meet the normal DMARC conditions, but also consider configuring a basic DMARC policy for your domain.

Deleting all of the keys in your Key Protect or HPCS instance

It’s a common problem that you want to delete an IBM Cloud Key Protect instance but there are still some keys remaining in that instance. For your protection, Key Protect and Hyper Protect Crypto Services require you to take action to delete those keys rather than allowing you to delete them as a side effect of deleting the Key Protect instance itself.

This is challenging if you have a large number of keys. That may be the case if you have a development or test environment that you are cleaning up, or if you have migrated your keys to another key provider.

It’s possible to script this using the Key Protect CLI.

First, login to IBM Cloud and install the key protect plugin if necessary:

$ ibmcloud login --sso
$ ibmcloud plugin install key-protect -r "IBM Cloud"

If your Key Protect instance is private-only you may need to export the KP_PRIVATE_ADDR environment variable to point to the service endpoint or VPE for Key Protect in your region. Next you need to identify the instance id for your Key Protect instance, which you can find in the instance details tab in the IBM Cloud UI, or by using the following command if you know the instance name:

$ ibmcloud resource service-instance smoonenKPmadrid --id
Retrieving service instance smoonenKPmadrid in all resource groups under account Development Account as smoonen@us.ibm.com...
crn:v1:bluemix:public:kms:eu-es:a/3f1b08d9abdc5d98ffeb0d3bdc279c04:1f8011c9-7fd9-4fe9-af5e-2fefcfda8cfc:: 1f8011c9-7fd9-4fe9-af5e-2fefcfda8cfc

You can save typing or pasting in subsequent commands by exporting the instance id:

$ export KP_INSTANCE_ID=1f8011c9-7fd9-4fe9-af5e-2fefcfda8cfc

The following command displays all of the key ids and names in your instance:

$ ibmcloud kp keys

You can adjust this command to display only the key ids:

$ ibmcloud kp keys --output json | jq -r '.[] | .id'

If you are confident that all of these keys can be safely deleted, and you have the appropriate permissions to do so, in your shell session you can loop through these and issue a delete command for each of them:

$ foreach key in $(ibmcloud kp keys --output json | jq -r '.[] | .id')
foreach> do
foreach> ibmcloud kp key delete $key
foreach> done

If any of the keys is known to be in use by a resource, you will receive an error. You may also receive other errors, for example, if you do not have sufficient permission to delete the key. You’ll have to rectify these issues before you can successfully delete the key and the Key Protect instance. For example, the following key was a root key that was in use by a Key Protect KMIP adapter:

Targeting endpoint: https://eu-es.kms.cloud.ibm.com
Deleting key: 'b262754c-f30d-4b5f-984c-f9c21b7ae13a', from instance: '1f8011c9-7fd9-4fe9-af5e-2fefcfda8cfc'...
FAILED
ASSOCIATED_KMIP_ADAPTER_ERR
The key cannot be deleted because it is associated with 1 KMIP adapter(s) in the instance
Correlation-ID:ef7ae793-945f-4b10-aa4b-f24b340bb3e1

PowerCLI native rekey capabilities

In the past I’ve written about using some PowerCLI extensions from the community repository to rekey individual objects, to rekey all objects, and to migrate from one key provider to another. I’ve recently discovered that the native PowerCLI commands support rekeying of VM and host keys, although not vSAN keys.

It is straightforward to rekey a vSAN cluster using the vCenter UI, and this has the side benefit that it will rekey your host encryption keys as well. But if you want to rekey virtual machines or host encryption keys directly, you could use a script like the following without needing to install the community modules:

$kp = Get-KeyProvider new-kmip

foreach($vm in Get-VM) {
  if($vm.ExtensionData.Config.KeyId) {
    Set-VM $vm -KeyProvider $kp -Confirm:$false
  }
}

foreach($vmhost in Get-VMHost) {
  if($vmhost.ExtensionData.Runtime.CryptoState -eq "safe") {
    Set-VMHost $vmhost -KeyProvider $kp
  }
}

VCFaaS available in São Paulo

Today IBM Cloud announced the release of our VCFaaS managed VMware offering in São Paulo.

I previously posted the list of site and pVDC ids for our multi-tenant offering for use with automation such as Terraform. Here is an updated list reflecting the currently available multi-tenant sites and pVDCs:

Site 'IBM VCFaaS Multitenant - SYD', ID 1a4bb41e-f3ce-4b1f-bdb9-b0a77cf83f50, in region au-syd
  pVDC 'SYD04', ID be95acb1-ba8f-466b-b796-22dcd648fd15, in location syd04 supporting provider types: on_demand, reserved
  pVDC 'SYD05', ID 1c4ea91b-fb36-4cd8-9ef7-6fd098db59c7, in location syd05 supporting provider types: on_demand, reserved
Site 'IBM VCFaaS Multitenant - SAO', ID 01b0b968-1e74-44ef-b014-0aa85eacf188, in region br-sao
  pVDC 'SAO01', ID 515eb4e2-49d6-44c3-8a12-2ea12abda116, in location sao01 supporting provider types: on_demand, reserved
Site 'IBM VCFaaS Multitenant - TOR', ID 6352e951-70f6-497f-a80f-94dff58c0734, in region ca-tor
  pVDC 'TOR04', ID de4be466-dfb9-48d4-9d28-657657fef571, in location tor04 supporting provider types: on_demand, reserved
  pVDC 'TOR05', ID 132f3818-0060-4e3e-90db-8b117416fb27, in location tor05 supporting provider types: on_demand, reserved
Site 'IBM VCFaaS Multitenant - FRA', ID 1fff1209-55b8-4667-b737-0e6de5cf5756, in region eu-de
  pVDC 'FRA02', ID f34e7a9f-afb4-430e-86d8-6b1978aebb9c, in location fra02 supporting provider types: on_demand, reserved
  pVDC 'FRA04', ID c66ac0a4-5e52-4b5e-9387-6bb8de7e42b1, in location fra04 supporting provider types: on_demand, reserved
Site 'IBM VCFaaS Multitenant - MAD', ID 8cef7547-3e44-4288-b849-1cd748e7d954, in region eu-es
  pVDC 'MAD02', ID e917160f-54b0-4a01-ac59-f9e23f44e8bd, in location mad02 supporting provider types: on_demand, reserved
Site 'IBM VCFaaS Multitenant - LON', ID f7074684-de9c-42e9-93a5-0358fcb2bf92, in region eu-gb
  pVDC 'LON04', ID 25e910c2-c790-4550-a316-03efa7c29888, in location lon04 supporting provider types: on_demand, reserved
  pVDC 'LON06', ID a24bbff9-8c79-44ab-8bb1-7699f304a1d1, in location lon06 supporting provider types: on_demand, reserved
Site 'IBM VCFaaS Multitenant - TOK', ID 3dca2ae4-fdea-4791-8092-0c879daa2097, in region jp-tok
  pVDC 'TOK02', ID 029c7ff7-da97-48ea-8ed9-0b86c2918f82, in location tok02 supporting provider types: on_demand, reserved
  pVDC 'TOK04', ID 08b78261-d4c9-41e9-baa9-421b493385f7, in location tok04 supporting provider types: on_demand, reserved
Site 'IBM VCFaaS Multitenant - WDC', ID 25fb5553-72a6-49ca-85b8-f18086cbac0b, in region us-east
  pVDC 'WDC06', ID 595d7f76-b1b2-429f-bb67-e357befa9da7, in location wdc06 supporting provider types: on_demand, reserved
  pVDC 'WDC07', ID efc17eff-fb3e-4b56-8252-44657738c539, in location wdc07 supporting provider types: on_demand, reserved
  pVDC 'WDC04-WDC07', ID 9a46dcce-84d1-4fca-a929-42bc2174ffba, in location wdc04 supporting provider types: on_demand, reserved
Site 'IBM VCFaaS Multitenant - DAL', ID 40e701cd-ef86-4d5e-a847-e7c336f11f27, in region us-south
  pVDC 'DAL10', ID f864f016-2c34-4658-95e7-dd3363408d76, in location dal10 supporting provider types: on_demand, reserved
  pVDC 'DAL13', ID 1df84086-b3af-481e-a535-e3554b809aed, in location dal13 supporting provider types: on_demand, reserved
  pVDC 'DAL12', ID 5b4af31c-383b-422b-b297-0ea6ed8af479, in location dal12 supporting provider types: on_demand, reserved

Using your IBM Cloud API key to act as a trusted profile

I mentioned previously that an IBM Cloud trusted profile could not create an API key to act on its behalf.

However, if your user ID has permission to act as a trusted profile, you can use your own API key to authenticate as yourself, and then exchange this authentication token for a time-limited token belonging to the trusted profile. You can then use the latter token to perform actions as the trusted profile.

IBM Cloud IAM provides an “assume” API for this purpose. Here is an example of its use:

# Exchange my IAM token for a trusted profile token
tp_exchange = { 'grant_type'   : 'urn:ibm:params:oauth:grant-type:assume',
                'access_token' : headers['Authorization'].split(' ')[1],
                'profile_name' : 'scott-test',
                'account'      : '187851. . .d02e02' }
tp_token = requests.post('https://iam.cloud.ibm.com/identity/token', data = tp_exchange).json()
# Make subsequent calls using the trusted profile identity
headers['Authorization'] = f"Bearer {tp_token['access_token']}"

Authenticating with VMware Cloud Director using IBM Cloud IAM

If you are automating activities in VMware Cloud Director—for example, if you are using Terraform to manage your edges and deploy your vApps—you will typically create a Cloud Director API token, which your automation can use to create an authenticated login session with Director for subsequent API calls.

There are interesting complex automation use cases where you might want to create an automation pipeline stretching from the IBM Cloud APIs to the Cloud Director APIs. For example, you might want to use the IBM Cloud APIs to provision a virtual data center (VDC) and then use the Cloud Director APIs—perhaps using Terraform—to deploy a vApp in that VDC. In cases like this, you prefer not to interrupt your automation to create your Cloud Director API token; instead, you want to be able to authenticate with Cloud Director by means of your IBM Cloud API key. Fortunately, that is possible because IBM preconfigures your Director organization with OIDC SSO integration with IBM Cloud IAM.

There are two ways to approach this. Most straightforwardly, if you are a REST API user, you can take the IBM Cloud IAM token that you got in exchange for your IBM Cloud API key, and submit this to Director as an OAuth identity provider token to authenticate a new login session and receive a Director bearer token for that session. You can then use this Director bearer token to make Director API calls for the length of that login session. Alternately, you can further use that Director bearer token to make an API call to create a long-lived Director API token, which you can then provide to tooling like Terraform in order to conduct ongoing management of your VDCs and other Director resources.

I’ve created two sample scripts demonstrating how this works. The first script obtains the Director bearer token and then uses this to call a Director API to list all vApps in each Director instance. Here is an example of its use:

smoonen@smoonen vmware-solutions % python3 get-vapps.py
Site: 'https://dirw003.eu-gb.vmware.cloud.ibm.com' / Organization 'd568ebe2-4042-4bc3-82c2-a3a7935cf9b9'
  vApp: vm1-1adc17be-3a7a-4460-82a8-ce821d3f5612
  vApp: vm2-000a9834-0037-4fc7-b6fd-0b2ec0927a28
Site: 'https://dirw082.us-south.vmware.cloud.ibm.com' / Organization '577fbceb-23ce-4361-bd11-1797931cb69b'
  vApp: auto_vapp
  vApp: VM-WIN-1ebfec4b-d754-4f6c-8ef9-e1adab14900b
Site: 'https://dirw003.ca-tor.vmware.cloud.ibm.com' / Organization '44445dba-16f0-488f-842c-a184f8b1d4e2'
  vApp: vm-1-39534998-c323-4484-9246-df57b258216e
  vApp: vm-2-330f574e-868b-45ae-934f-df007f2a30d8
  vApp: vm-3-3855594d-ce3b-4de7-8a81-8f4dcbc87a5b
Site: 'https://dirw003.us-east.vmware.cloud.ibm.com' / Organization '3bb02c20-e9df-4b39-ab76-94d43567add7'
  vApp: test-2de106b7-9107-40b8-9ec1-2287046df186

Interestingly, IBM Cloud service IDs are also represented in the Director OIDC SSO. You can create a service ID, and provided you have assigned the service ID sufficient IAM permissions to your VCF as a Service resources, you can use an IAM token generated from the service ID’s API key to authenticate with Director and call Director APIs.

IBM Cloud trusted profiles do not support the creation of API keys. However, trusted profiles are allowed to login to Cloud Director. In order to authenticate your trusted profile with Cloud Director (and possibly to create a Director API token) you will need to extract your trusted profile IAM token by other means than exchange of an API key. If you login to your trusted profile using the ibmcloud CLI (or by means of the IBM Cloud shell), you can extract your IAM token by this means:

scott_test@cloudshell:~$ ibmcloud iam oauth-tokens | grep IAM | cut -d \: -f 2 | sed 's/^ *//'
Bearer eyJraWQiOi. . .aZoC_fZQ
scott_test@cloudshell:~$

My second script uses the alternate approach of leveraging the Director bearer token to create a long-lived Director API token, in this case for each Director instance to which your user has access. Here is an example of its use:

smoonen@smoonen vmware-solutions % python3 create-director-tokens.py
Site: 'https://dirw003.eu-gb.vmware.cloud.ibm.com' / Organization 'd568ebe2-4042-4bc3-82c2-a3a7935cf9b9'
  token: leTf. . .TIs5
Site: 'https://dirw002.eu-de.vmware.cloud.ibm.com' / Organization 'ba10c5c7-7e15-41b5-aa4c-84bd373dc2b1'
  token: CL9G. . .IJRY
Site: 'https://dirw003.ca-tor.vmware.cloud.ibm.com' / Organization '44445dba-16f0-488f-842c-a184f8b1d4e2'
  token: p9cx. . .LdGt
Site: 'https://dirw082.us-south.vmware.cloud.ibm.com' / Organization '577fbceb-23ce-4361-bd11-1797931cb69b'
  token: ygc7. . .FVjB
Site: 'https://dirw003.us-east.vmware.cloud.ibm.com' / Organization '3bb02c20-e9df-4b39-ab76-94d43567add7'
  token: UCIf. . .aPBE

The Director APIs to create these long-lived tokens are not well documented. But essentially what is happening here is that we are creating an OAuth client ID and obtaining the refresh token for that client.

Retrieving VCFaaS sites and pVDCs

IBM Cloud offers a terraform provider for our managed VMware Cloud Director offering, VCFaaS, to provision virtual data centers (VDCs).

This provider requires that you input the site id for the Director instance of your choice, as well as a pVDC id for the provider VDC (which we sometimes call resource pool) in which you want to create your VDC. The ids for these are not well known.

I have written a brief Python script to invoke the VCFaaS API and list the sites and pVDCs.

At the time of this writing, these are the multi-tenant sites and pVDCs available for use:

Site 'IBM VCFaaS Multitenant - SYD', ID 1a4bb41e-f3ce-4b1f-bdb9-b0a77cf83f50, in region au-syd
  pVDC 'SYD04', ID be95acb1-ba8f-466b-b796-22dcd648fd15, in location syd04 supporting provider types: on_demand, reserved
  pVDC 'SYD05', ID 1c4ea91b-fb36-4cd8-9ef7-6fd098db59c7, in location syd05 supporting provider types: on_demand, reserved
Site 'IBM VCFaaS Multitenant - TOR', ID 6352e951-70f6-497f-a80f-94dff58c0734, in region ca-tor
  pVDC 'TOR04', ID de4be466-dfb9-48d4-9d28-657657fef571, in location tor04 supporting provider types: on_demand, reserved
  pVDC 'TOR05', ID 132f3818-0060-4e3e-90db-8b117416fb27, in location tor05 supporting provider types: on_demand, reserved
Site 'IBM VCFaaS Multitenant - FRA', ID 1fff1209-55b8-4667-b737-0e6de5cf5756, in region eu-de
  pVDC 'FRA02', ID f34e7a9f-afb4-430e-86d8-6b1978aebb9c, in location fra02 supporting provider types: on_demand, reserved
  pVDC 'FRA04', ID c66ac0a4-5e52-4b5e-9387-6bb8de7e42b1, in location fra04 supporting provider types: on_demand, reserved
Site 'IBM VCFaaS Multitenant - MAD', ID 8cef7547-3e44-4288-b849-1cd748e7d954, in region eu-es
  pVDC 'MAD02', ID e917160f-54b0-4a01-ac59-f9e23f44e8bd, in location mad02 supporting provider types: on_demand, reserved
Site 'IBM VCFaaS Multitenant - LON', ID f7074684-de9c-42e9-93a5-0358fcb2bf92, in region eu-gb
  pVDC 'LON04', ID 25e910c2-c790-4550-a316-03efa7c29888, in location lon04 supporting provider types: on_demand, reserved
  pVDC 'LON06', ID a24bbff9-8c79-44ab-8bb1-7699f304a1d1, in location lon06 supporting provider types: on_demand, reserved
Site 'IBM VCFaaS Multitenant - TOK', ID 3dca2ae4-fdea-4791-8092-0c879daa2097, in region jp-tok
  pVDC 'TOK02', ID 029c7ff7-da97-48ea-8ed9-0b86c2918f82, in location tok02 supporting provider types: on_demand, reserved
  pVDC 'TOK04', ID 08b78261-d4c9-41e9-baa9-421b493385f7, in location tok04 supporting provider types: on_demand, reserved
Site 'IBM VCFaaS Multitenant - WDC', ID 25fb5553-72a6-49ca-85b8-f18086cbac0b, in region us-east
  pVDC 'WDC06', ID 595d7f76-b1b2-429f-bb67-e357befa9da7, in location wdc06 supporting provider types: on_demand, reserved
  pVDC 'WDC07', ID efc17eff-fb3e-4b56-8252-44657738c539, in location wdc07 supporting provider types: on_demand, reserved
  pVDC 'WDC04-WDC07', ID 9a46dcce-84d1-4fca-a929-42bc2174ffba, in location wdc04 supporting provider types: on_demand, reserved
Site 'IBM VCFaaS Multitenant - DAL', ID 40e701cd-ef86-4d5e-a847-e7c336f11f27, in region us-south
  pVDC 'DAL10', ID f864f016-2c34-4658-95e7-dd3363408d76, in location dal10 supporting provider types: on_demand, reserved
  pVDC 'DAL13', ID 1df84086-b3af-481e-a535-e3554b809aed, in location dal13 supporting provider types: on_demand, reserved
  pVDC 'DAL12', ID 5b4af31c-383b-422b-b297-0ea6ed8af479, in location dal12 supporting provider types: on_demand, reserved

Running vCloud Usage Meter in IBM Cloud

Running vCloud Usage Meter in IBM Cloud

In 2024, Broadcom simplified VMware product pricing and packaging. The VMware Cloud Foundation (VCF) offering now encompasses a wide variety of VMware software and features, with a relatively smaller number of software and features being sold as add-ons. As part of this simplification, Broadcom required all customers and cloud providers to make new commitments and to create new license keys.

Cloud providers are uniquely entitled for on-demand licensing of VMware products beyond their contract commitment. In exchange for this benefit, Broadcom expects that the vCloud Usage Meter product “must be used” to monitor and report usage of VMware products. IBM secured an extension of this requirement so that we could update our automation and develop integration points for our customers. IBM has now released updated VMware license keys and Usage Meter support, and IBM’s customers are expected by Broadcom—and therefore by IBM—to “immediately” install these in order to remain entitled to VMware software. . . . read more at Updates to VMware license keys and the use of vCloud Usage Meter in IBM Cloud