Using your IBM Cloud API key to act as a trusted profile

I mentioned previously that an IBM Cloud trusted profile could not create an API key to act on its behalf.

However, if your user ID has permission to act as a trusted profile, you can use your own API key to authenticate as yourself, and then exchange this authentication token for a time-limited token belonging to the trusted profile. You can then use the latter token to perform actions as the trusted profile.

IBM Cloud IAM provides an “assume” API for this purpose. Here is an example of its use:

# Exchange my IAM token for a trusted profile token
tp_exchange = { 'grant_type'   : 'urn:ibm:params:oauth:grant-type:assume',
                'access_token' : headers['Authorization'].split(' ')[1],
                'profile_name' : 'scott-test',
                'account'      : '187851. . .d02e02' }
tp_token = requests.post('https://iam.cloud.ibm.com/identity/token', data = tp_exchange).json()
# Make subsequent calls using the trusted profile identity
headers['Authorization'] = f"Bearer {tp_token['access_token']}"

Authenticating with VMware Cloud Director using IBM Cloud IAM

If you are automating activities in VMware Cloud Director—for example, if you are using Terraform to manage your edges and deploy your vApps—you will typically create a Cloud Director API token, which your automation can use to create an authenticated login session with Director for subsequent API calls.

There are interesting complex automation use cases where you might want to create an automation pipeline stretching from the IBM Cloud APIs to the Cloud Director APIs. For example, you might want to use the IBM Cloud APIs to provision a virtual data center (VDC) and then use the Cloud Director APIs—perhaps using Terraform—to deploy a vApp in that VDC. In cases like this, you prefer not to interrupt your automation to create your Cloud Director API token; instead, you want to be able to authenticate with Cloud Director by means of your IBM Cloud API key. Fortunately, that is possible because IBM preconfigures your Director organization with OIDC SSO integration with IBM Cloud IAM.

There are two ways to approach this. Most straightforwardly, if you are a REST API user, you can take the IBM Cloud IAM token that you got in exchange for your IBM Cloud API key, and submit this to Director as an OAuth identity provider token to authenticate a new login session and receive a Director bearer token for that session. You can then use this Director bearer token to make Director API calls for the length of that login session. Alternately, you can further use that Director bearer token to make an API call to create a long-lived Director API token, which you can then provide to tooling like Terraform in order to conduct ongoing management of your VDCs and other Director resources.

I’ve created two sample scripts demonstrating how this works. The first script obtains the Director bearer token and then uses this to call a Director API to list all vApps in each Director instance. Here is an example of its use:

smoonen@smoonen vmware-solutions % python3 get-vapps.py
Site: 'https://dirw003.eu-gb.vmware.cloud.ibm.com' / Organization 'd568ebe2-4042-4bc3-82c2-a3a7935cf9b9'
  vApp: vm1-1adc17be-3a7a-4460-82a8-ce821d3f5612
  vApp: vm2-000a9834-0037-4fc7-b6fd-0b2ec0927a28
Site: 'https://dirw082.us-south.vmware.cloud.ibm.com' / Organization '577fbceb-23ce-4361-bd11-1797931cb69b'
  vApp: auto_vapp
  vApp: VM-WIN-1ebfec4b-d754-4f6c-8ef9-e1adab14900b
Site: 'https://dirw003.ca-tor.vmware.cloud.ibm.com' / Organization '44445dba-16f0-488f-842c-a184f8b1d4e2'
  vApp: vm-1-39534998-c323-4484-9246-df57b258216e
  vApp: vm-2-330f574e-868b-45ae-934f-df007f2a30d8
  vApp: vm-3-3855594d-ce3b-4de7-8a81-8f4dcbc87a5b
Site: 'https://dirw003.us-east.vmware.cloud.ibm.com' / Organization '3bb02c20-e9df-4b39-ab76-94d43567add7'
  vApp: test-2de106b7-9107-40b8-9ec1-2287046df186

Interestingly, IBM Cloud service IDs are also represented in the Director OIDC SSO. You can create a service ID, and provided you have assigned the service ID sufficient IAM permissions to your VCF as a Service resources, you can use an IAM token generated from the service ID’s API key to authenticate with Director and call Director APIs.

IBM Cloud trusted profiles do not support the creation of API keys. However, trusted profiles are allowed to login to Cloud Director. In order to authenticate your trusted profile with Cloud Director (and possibly to create a Director API token) you will need to extract your trusted profile IAM token by other means than exchange of an API key. If you login to your trusted profile using the ibmcloud CLI (or by means of the IBM Cloud shell), you can extract your IAM token by this means:

scott_test@cloudshell:~$ ibmcloud iam oauth-tokens | grep IAM | cut -d \: -f 2 | sed 's/^ *//'
Bearer eyJraWQiOi. . .aZoC_fZQ
scott_test@cloudshell:~$

My second script uses the alternate approach of leveraging the Director bearer token to create a long-lived Director API token, in this case for each Director instance to which your user has access. Here is an example of its use:

smoonen@smoonen vmware-solutions % python3 create-director-tokens.py
Site: 'https://dirw003.eu-gb.vmware.cloud.ibm.com' / Organization 'd568ebe2-4042-4bc3-82c2-a3a7935cf9b9'
  token: leTf. . .TIs5
Site: 'https://dirw002.eu-de.vmware.cloud.ibm.com' / Organization 'ba10c5c7-7e15-41b5-aa4c-84bd373dc2b1'
  token: CL9G. . .IJRY
Site: 'https://dirw003.ca-tor.vmware.cloud.ibm.com' / Organization '44445dba-16f0-488f-842c-a184f8b1d4e2'
  token: p9cx. . .LdGt
Site: 'https://dirw082.us-south.vmware.cloud.ibm.com' / Organization '577fbceb-23ce-4361-bd11-1797931cb69b'
  token: ygc7. . .FVjB
Site: 'https://dirw003.us-east.vmware.cloud.ibm.com' / Organization '3bb02c20-e9df-4b39-ab76-94d43567add7'
  token: UCIf. . .aPBE

The Director APIs to create these long-lived tokens are not well documented. But essentially what is happening here is that we are creating an OAuth client ID and obtaining the refresh token for that client.

Retrieving VCFaaS sites and pVDCs

IBM Cloud offers a terraform provider for our managed VMware Cloud Director offering, VCFaaS, to provision virtual data centers (VDCs).

This provider requires that you input the site id for the Director instance of your choice, as well as a pVDC id for the provider VDC (which we sometimes call resource pool) in which you want to create your VDC. The ids for these are not well known.

I have written a brief Python script to invoke the VCFaaS API and list the sites and pVDCs.

At the time of this writing, these are the multi-tenant sites and pVDCs available for use:

Site 'IBM VCFaaS Multitenant - SYD', ID 1a4bb41e-f3ce-4b1f-bdb9-b0a77cf83f50, in region au-syd
  pVDC 'SYD04', ID be95acb1-ba8f-466b-b796-22dcd648fd15, in location syd04 supporting provider types: on_demand, reserved
  pVDC 'SYD05', ID 1c4ea91b-fb36-4cd8-9ef7-6fd098db59c7, in location syd05 supporting provider types: on_demand, reserved
Site 'IBM VCFaaS Multitenant - TOR', ID 6352e951-70f6-497f-a80f-94dff58c0734, in region ca-tor
  pVDC 'TOR04', ID de4be466-dfb9-48d4-9d28-657657fef571, in location tor04 supporting provider types: on_demand, reserved
  pVDC 'TOR05', ID 132f3818-0060-4e3e-90db-8b117416fb27, in location tor05 supporting provider types: on_demand, reserved
Site 'IBM VCFaaS Multitenant - FRA', ID 1fff1209-55b8-4667-b737-0e6de5cf5756, in region eu-de
  pVDC 'FRA02', ID f34e7a9f-afb4-430e-86d8-6b1978aebb9c, in location fra02 supporting provider types: on_demand, reserved
  pVDC 'FRA04', ID c66ac0a4-5e52-4b5e-9387-6bb8de7e42b1, in location fra04 supporting provider types: on_demand, reserved
Site 'IBM VCFaaS Multitenant - MAD', ID 8cef7547-3e44-4288-b849-1cd748e7d954, in region eu-es
  pVDC 'MAD02', ID e917160f-54b0-4a01-ac59-f9e23f44e8bd, in location mad02 supporting provider types: on_demand, reserved
Site 'IBM VCFaaS Multitenant - LON', ID f7074684-de9c-42e9-93a5-0358fcb2bf92, in region eu-gb
  pVDC 'LON04', ID 25e910c2-c790-4550-a316-03efa7c29888, in location lon04 supporting provider types: on_demand, reserved
  pVDC 'LON06', ID a24bbff9-8c79-44ab-8bb1-7699f304a1d1, in location lon06 supporting provider types: on_demand, reserved
Site 'IBM VCFaaS Multitenant - TOK', ID 3dca2ae4-fdea-4791-8092-0c879daa2097, in region jp-tok
  pVDC 'TOK02', ID 029c7ff7-da97-48ea-8ed9-0b86c2918f82, in location tok02 supporting provider types: on_demand, reserved
  pVDC 'TOK04', ID 08b78261-d4c9-41e9-baa9-421b493385f7, in location tok04 supporting provider types: on_demand, reserved
Site 'IBM VCFaaS Multitenant - WDC', ID 25fb5553-72a6-49ca-85b8-f18086cbac0b, in region us-east
  pVDC 'WDC06', ID 595d7f76-b1b2-429f-bb67-e357befa9da7, in location wdc06 supporting provider types: on_demand, reserved
  pVDC 'WDC07', ID efc17eff-fb3e-4b56-8252-44657738c539, in location wdc07 supporting provider types: on_demand, reserved
  pVDC 'WDC04-WDC07', ID 9a46dcce-84d1-4fca-a929-42bc2174ffba, in location wdc04 supporting provider types: on_demand, reserved
Site 'IBM VCFaaS Multitenant - DAL', ID 40e701cd-ef86-4d5e-a847-e7c336f11f27, in region us-south
  pVDC 'DAL10', ID f864f016-2c34-4658-95e7-dd3363408d76, in location dal10 supporting provider types: on_demand, reserved
  pVDC 'DAL13', ID 1df84086-b3af-481e-a535-e3554b809aed, in location dal13 supporting provider types: on_demand, reserved
  pVDC 'DAL12', ID 5b4af31c-383b-422b-b297-0ea6ed8af479, in location dal12 supporting provider types: on_demand, reserved

Running vCloud Usage Meter in IBM Cloud

Running vCloud Usage Meter in IBM Cloud

In 2024, Broadcom simplified VMware product pricing and packaging. The VMware Cloud Foundation (VCF) offering now encompasses a wide variety of VMware software and features, with a relatively smaller number of software and features being sold as add-ons. As part of this simplification, Broadcom required all customers and cloud providers to make new commitments and to create new license keys.

Cloud providers are uniquely entitled for on-demand licensing of VMware products beyond their contract commitment. In exchange for this benefit, Broadcom expects that the vCloud Usage Meter product “must be used” to monitor and report usage of VMware products. IBM secured an extension of this requirement so that we could update our automation and develop integration points for our customers. IBM has now released updated VMware license keys and Usage Meter support, and IBM’s customers are expected by Broadcom—and therefore by IBM—to “immediately” install these in order to remain entitled to VMware software. . . . read more at Updates to VMware license keys and the use of vCloud Usage Meter in IBM Cloud

Using vSphere Trust Authority to geofence workloads

IBM and Kyndryl have in the past used Entrust BoundaryControl to accomplish geofencing. This worked using a combination of their CloudControl and KeyControl products. The CloudControl product was used by security administrators to install cryptographically signed tags into known trusted host TPMs, and then to describe policies for virtual machines that required them to run on hosts with particular tags. In addition to CloudControl enforcing virtual machine placement, the KeyControl product further integrated with this configuration to ensure that virtual machines running on unapproved hosts could not be successfully decrypted and run. Customers could devise tagging schemes according to their needs, such as prod/nonprod, tier1/tier2, and US/EU.

You can accomplish a similar kind of exclusion or geofencing capability using VMware’s vSphere Trust Authority. Although vTA is designed primarily as a means of ensuring that workloads run on hosts with known trusted firmware and software levels, it also has the capability to trust hosts individually. Rather than trusting the vendor TPM CA certificate, you can trust individual host TPM certificates. This allows you to vet the hosts one by one in your environment, and mark them as trusted only if they meet your criteria, including their geographic location. vTA will then help to ensure that the virtual machines in your environment cannot be successfully decrypted and run on hosts outside of your trusted set.

Like any security solution, attestation and geofencing solutions like BoundaryControl and vTA require extra effort to configure and to administrate. In exchange for this effort, however, you can create compelling sovereign cloud solutions.

Estimating vDefend firewall usage

Here are a few key things to know about Broadcom’s vDefend firewall offering:

  • vDefend comes in three flavors: firewall, ATP, and firewall+ATP bundle. The ATP license is not designed to be used on its own, but only to stack ATP capability on top of firewall capability in cases where you have not purchased the bundle.
  • If you are using a vSphere 8 VCF solution key to activate NSX, you will need a vDefend “solution key” to activate vDefend features. Otherwise, if you are using a v7 or v8 component key to activate NSX, you will need to use separate vDefend “component keys” to activate the vDefend features on edges and in distributed firewall, respectively. For more information, see KB 318444.
  • Broadcom assesses vDefend for distributed firewall against the vSphere hosts in the same manner as VCF (i.e., with a floor of 16 cores per CPU). For distributed firewall you should order as many cores of vDefend as for VCF. For edge firewall, Broadcom assesses vDefend at a ratio of 4 cores per vCPU (including passive edge VMs). There is not a technical reason for this ratio; it is simply a business decision on Broadcom’s part.
  • If you are running NSX 4.1 or newer, Broadcom has recently published a script that you can use to survey your environment’s firewall configuration to measure how much vDefend licensing you need; see KB 395111. This script correctly takes into account some cases where Broadcom does not assess vDefend usage; for example, if gateway firewall is enabled but only the default permit-all rule is configured, or only stateless rules are configured.

Fixed: Unexpected reboot when rekeying a virtual machine

vSphere 8.0u3e (see release notes) fixes the issue where a rekeyed VM may experience an unexpected reboot:

PR 3477772: Encrypted virtual machines with active Change Block Tracking (CBT) might intermittently power off after a rekey operation

Due to a race condition between the VMX and hostd services for a specific VMcrypt-related reconfiguration, encrypted VMs with active CBT might unexpectedly power off during a rekey operation.

This issue is resolved in this release.

From what I can tell, this issue is not currently fixed in the vSphere 7 stream.