Highly available key management in IBM Cloud for VMware Solutions

IBM Cloud’s KMIP for VMware offering provides the foundation for cloud-based key management when using VMware vSphere encryption or vSAN encryption. KMIP for VMware is highly available within a single region:

KMIP for VMware and Key Protect are highly available when you configure vCenter connections to both regional endpoints. If any one of the three zones in that region fail entirely, key management continues to be available to your VMware workloads.
KMIP for VMware and Hyper Protect Crypto Services (HPCS) are highly available if you deploy two or more crypto units for your HPCS instance. If you do so and any one of the three zones in that region fail entirely, key management continues to be available to your VMware workloads.

If you need to migrate or failover your workloads outside of a region, your plan depends on whether you are using vSAN encryption or vSphere encryption:

When you are using vSAN encryption, each site is protected by its own key provider. If you are using vSAN encryption to protect workloads that you replicate between multiple sites, you must create separate KMIP for VMware instances in each site, that are connected to separate Key Protect or HPCS instances in those sites. You must connect your vCenter Server in each site to the local KMIP for VMware instance as its key provider.

When you are using vSphere encryption, most VMware replication and migration techniques today (for example, cross-vCenter vMotion and vSphere replication) rely on having a common key manager between the two sites. This topology is not supported by KMIP for VMware. Instead, you must create separate KMIP for VMware instances in each site, that is connected to separate Key Protect or HPCS instances in those sites. You must connect your vCenter server in each site to the local KMIP for VMware instance as its key provider, and then use a replication technology that supports the attachment and replication of decrypted disks.

Veeam Backup and Replication supports this replication technique. To implement this technique correctly, see the steps that you must take as indicated in the Veeam documentation.

Note that this technique currently does not support the replication of virtual machines with a vTPM device.

Encryption at rest for VMware on IBM Cloud

One of the key topics we covered as part of our Fast Start education was encryption at rest for VMware on the IBM Cloud. There are many options for encrypting your workloads at rest, including:

VMware vSAN encryption
VMware vSphere encryption
HyTrust Data Control, part of IBM Cloud Secure Virtualization
Any other existing encryption solution you wish to bring to IBM Cloud

The first three offerings are available today directly from IBM Cloud for VMware Solutions, although some assembly is required in each case. There are important tradeoffs between these options that you need to take into consideration, such as ease of use, interoperability with other solutions like workload migration tooling, and the nature of what is encrypted. The following table that I shared at Fast Start summarizes the differences between these solutions:

Comparison	vSAN encryption	vSphere encryption	HyTrust Data Control
Encryption type	Datastore disks encrypted @ hypervisor Secures: disk drives	VM disks encrypted @ hypervisor Secures: VMDK files, disk traffic en route to datastore	Agent-based encryption of disks within VM Secures: VMDK files, disk traffic en route to datastore
Key management	External KMS must be provided (not included) supporting KMIP 1.1 (e.g., IBM KMIP for VMware, IBM SKLM, or HyTrust Key Control)	External KMS must be provided (not included) supporting KMIP 1.1 (e.g., IBM KMIP for VMware, IBM SKLM, or HyTrust Key Control)	HyTrust Key Control (included)
Additional capabilities			Together with HyTrust Cloud Control, provides advanced access control, auditing, approval, and compliance capabilities; and enables Boundary Control for geofencing and hardware trust
Cost	vSAN Enterprise is required (per socket) Key management server	Key management server	HyTrust Data Control (per socket) HyTrust Cloud Control (optional, per socket)
Limitations	Not compatible with other storage types (e.g., IBM Cloud Endurance storage, NetApp ONTAP Select) Does not encrypt storage traffic in flight between hosts	Eliminates benefit of vSAN deduplication and compression	Eliminates benefit of vSAN deduplication and compression
Migration	Compatible with all migration technologies	Compatible with Veeam Compatible with VMware SRM when using array based replication Not currently compatible with VMware HCX Not currently compatible with Zerto Not currently compatible with vSphere replication Not currently compatible with cross-vCenter vMotion	Compatible with all migration technologies provided that HyTrust key management server availability and host compliance (if applicable) are maintained across sites. Some extra recovery steps are required post migration if the workload IP addressing has changed.

File encryption with public-key cryptography

Public-key cryptography is not suitable for encrypting large files. A naive approach to encrypting a large file will return an error if the file is larger than the RSA key:

[smoonen@smoonen encryption]$ dd if=/dev/zero bs=1024 count=1024 | openssl pkeyutl -encrypt -pubin -inkey pubkey.pem
Public Key operation error
140544802154400:error:0406D06E:rsa routines:RSA_padding_add_PKCS1_type_2:data too large for key size:rsa_pk1.c:151:

If you want to accomplish asymmetric encryption of large files, the general approach is to encrypt the file using symmetric cryptography, and encrypt the symmetric key using public-key cryptography. The OpenSSL smime command uses this approach, but it does not support extremely large files.

To support this case, I’ve written some simple file encryption shell scripts which I’ve posted on GitHub. These scripts are as follows:

genkeypair generates a private and public key pair
encrfile encrypts one or more files using AES-256 encryption, encrypts the AES-256 keys using public-key encryption, and saves the encrypted key as part of the encrypted file
decrfile decrypts a single file previously encrypted by encrfile, by extracting the encrypted AES-256 key, decrypting it using public-key encryption, and then decrypting the file itself. The decrypted data is sent to stdout.

Pattern: Password Reset

[Security] Problem: A user has forgotten her password. You need to generate a password reset token to send in an email to confirm her identity before allowing her to establish a new password.

Context: You are developing a web application requiring user password authentication and using a password salt and hashing algorithm to store passwords. You are reluctant to create a random nonce for your password reset token, since this needs to be stored in your database. But this seems inefficient; in most cases, the user object doesn’t need to hold a nonce, but this seems like such a trivial problem to create an entirely new password reset nonce table to associate with the user object.

Solution: You can generate a unique password reset token by hashing the internal state of the user object, including the user’s password salt and password hash. Because you are including the password salt and hash as part of the hash to produce the reset token, the reset token has the following properties:

It can be computed only by the server (the password salt and hashes should not be externalized),
It can be computed at any time by the server (it does not need to be stored in your database),
It is constant until the user changes their password (i.e., an attacker cannot cause it to be invalidated), and
It is guaranteed to change whenever the password is actually changed (since the user object’s internal state change from the password update will cause the reset token hash to change), so that an attacker who later discovers the token cannot exploit it.

If you desire to limit the viability of a password reset token to a certain period of time (e.g., 24 hours), you can include an expiration timestamp in the token and also as input to the hashing operation:

reset_token = timestamp + hash(timestamp + user.password_salt + user.password + ...)

By including the timestamp in the token, you provide an indication to your application when the token expires. By including the timestamp as input to the hash portion of the token, you ensure that it is not possible for an attacker to take a stale token and manufacture a valid token.

full◦valence

connect◦the◦dots

Category: Security

Highly available key management in IBM Cloud for VMware Solutions

Encryption at rest for VMware on IBM Cloud

File encryption with public-key cryptography