Large boot volumes and images for IBM Cloud VPC VSI

IBM Cloud recently released support for VPC virtual server instance (VSI) boot volumes larger than 250GB when using the second-generation sdp storage profile. However, this release did not include any increase in size restrictions for VSI images. This is primarily due to limitations of the general-purpose storage profile which shares the same image namespace.

If you need to create a VSI from an image whose allocated space is larger than 250GB, here is how to accomplish that:

Operating system and licensing

First, determine whether the operating system and licensing for your image is listed among the stock images in the IBM Cloud catalog. If it is, then you can skip this step. If it is not (for example, you need Windows BYOL or you need to label it as a Generic OS), then you must first create a custom image smaller than 250GB.

For this purpose it is sufficient to create a small qcow2 volume with blank data. You can follow the documentation to upload this to Cloud Object Storage (COS) and create an image. Be sure to specify your required OS and licensing.

Create original boot volume

Next we need to create a boot volume of the desired size. For this purpose we will create a throwaway VSI but keep its boot volume. We will call this the “original boot volume”:

  • Select the appropriate OS image (if you created a custom image in the previous step, select this). Note that your VSI will fail to boot if you are using a blank qcow2 image, but that is okay.
  • Make sure that your boot volume uses the sdp storage profile.
  • Select your required boot volume size (e.g., 1000 GB) and performance.
  • Ensure that you set Auto-Delete to Disabled for this boot volume. This will preserve the original boot volume after you delete the VSI.

Delete the VSI once it has started.

Copy image to original boot volume

Now create a new worker VSI that you will use to write your image to the original boot volume. Ensure that the boot volume for this worker VSI is large enough to account for any scratch space you need. For example, if you plan to download a qcow2 file which you will then write to the original boot volume, you should provision enough space to hold the qcow2 file.

After the worker VSI has started, view its details. On the Storage tab, click Actions | Attach. Select the original boot volume and attach it to the worker VSI. Note that after attachment it has has Auto-Delete disabled.

Now probe the devices on the worker VSI to discover where the original boot volume was attached. If your worker is a Linux system, it might typically be attached as /dev/vdb or /dev/vdd, among several other small block devices that are attached for cloud-init purposes. You might discover the original boot volume by querying its size:

for x in /dev/vd*; do echo "$x - $(blockdev --getsize64 $x)"; done

Next, populate this block device with your image content. For example, you might use COS as a means to download your qcow2 file to the worker VSI, and then use qemu-img to convert from qcow2 to raw format:

qemu-img convert -f qcow2 -O raw appliance.qcow2 /dev/vdb

At this point I like to ensure that all buffers are flushed to disk:

blockdev --flushbufs /dev/vdb

Now detach the original boot volume from the worker VSI, then delete the worker VSI.

Deploy your image

At this point you have two options:

Singleton

If you only need to deploy one instance of this image (e.g., it is a migrated virtual machine rather than a repeatable template), then you can create a new VSI and choose the original boot volume as your image. Note well that you need to configure this on the image configuration rather than the boot volume configuration. In the image selection dialog there is a tab containing your “Existing volume” list, where you should find the unattached original boot volume.

Repeatable

If you need to deploy this image repeatedly, you should first create a snapshot of the original boot volume. This snapshot will function just like a repeatable boot image.

Find the original boot volume in the IBM Cloud block storage volumes list and choose Actions | Create snapshot. You can safely delete the original storage volume once the snapshot is complete. Then you can select the snapshot as part of the image configuration for any new VSI.

Miscellany

All of the usual considerations apply for boot images, such as the availability of cloud-init and the need for virtio drivers, as well as boot mode considerations (IBM Cloud currently infers BIOS vs. UEFI based on whether the volume is formatted using MBR or GPT). For some treatment of these considerations you can read the IBM Cloud documentation on custom images or my article on virtual machine migration.

    From VMware to IBM Cloud VPC VSI, part 8: Veeam Backup and Replication

    See all blog posts in this series:

    1. From VMware to IBM Cloud VPC VSI, part 1: Introduction
    2. From VMware to IBM Cloud VPC VSI, part 2: VPC network design
    3. From VMware to IBM Cloud VPC VSI, part 3: Migrating virtual machines
    4. From VMware to IBM Cloud VPC VSI, part 4: Backup and restore
    5. From VMware to IBM Cloud VPC VSI, part 5: VPC object model
    6. From VMware to IBM Cloud VPC VSI, part 6: Disaster recovery
    7. From VMware to IBM Cloud VPC VSI, part 7: Automation
    8. From VMware to IBM Cloud VPC VSI, part 8: Veeam Backup and Replication

    Previously we examined methods for backing up your IBM Cloud VPC VSI workloads using volume snapshots and using file-level recovery.

    It’s also possible to leverage enterprise backup software such as Veeam Backup and Replication to perform volume-level backup and restore of your VSI workloads, but this requires a little more effort. Here are some high-level pointers for using Veeam specifically:

    1. VBR 13 is available in an appliance form factor, including both ISO and OVA. Installation of the OVA in IBM Cloud VSI is possible but tricky. The OVA contains two disks and these must be extracted and transferred separately to a boot volume (or image) and a data volume before deploying them together as part of a single VSI.
    2. You must use the Veeam Agent for Windows and the Veeam Agent for Linux to perform backups of your VSI workloads.
    3. For restore, if your boot volume is not corrupted, you can use the Veeam agent on your VSI to perform restore of files, folders, and data volumes.
    4. If your boot volume is corrupted, you can recover the original boot volume by the following process:
      1. Create a new VSI or use an existing VSI running the Veeam agent
      2. Delete your original VSI being careful to preserve all of the volumes
      3. Attach the volume(s) you wish to restore to the VSI previously selected
      4. Use the Veeam agent to restore the volume from backup
      5. Detach the volume(s) from this VSI
      6. Recreate your original VSI using the original volumes, and boot it

    The recovery process described above is not sufficient to migrate a Veeam backup from a VMware environment to a VSI. This is because the recovery process does not address other aspects of migration including the installation of virtio drivers and cloud-init.

    Identify owners of VSIs

    In IBM Cloud’s virtual server instance (VSI) API, the VSI object does not itself identify the creator of the VSI. However, the creator is known to the cloud resource controller, so you can use the cloud resource API to determine the creator of a VSI.

    The path to do this is to:

    1. Call VSI API to identify VSIs
    2. Call resource API to identify owner
    3. Call (or cache) user API to map owner to email

    Here is some sample code using the ibmcloud CLI to do this:

    #!/bin/zsh

# Identify owners of all VPC VSIs in IBM Cloud account

users=$(ibmcloud account users --output json)

for region in $(ibmcloud regions --output json | jq -r '.[]|.Name')
do
  ibmcloud target -r $region -q > /dev/null
  for instance in $(ibmcloud is instances --output json | jq -r '.[]|.name')
  do
    # VSI names might be reused in multiple regions; filter by region
    user=$(ibmcloud resource service-instance $instance --output json 2> /dev/null | jq -r ".[]|select(.region_id==\"$region\")|.created_by")
    email=$(echo $users | jq -r ".[]|select(.ibmUniqueId==\"$user\")|.email")
    echo "VSI $instance in region $region was deployed by $email"
  done
done

    From VMware to IBM Cloud VPC VSI, part 7: Automation

    See all blog posts in this series:

    1. From VMware to IBM Cloud VPC VSI, part 1: Introduction
    2. From VMware to IBM Cloud VPC VSI, part 2: VPC network design
    3. From VMware to IBM Cloud VPC VSI, part 3: Migrating virtual machines
    4. From VMware to IBM Cloud VPC VSI, part 4: Backup and restore
    5. From VMware to IBM Cloud VPC VSI, part 5: VPC object model
    6. From VMware to IBM Cloud VPC VSI, part 6: Disaster recovery
    7. From VMware to IBM Cloud VPC VSI, part 7: Automation
    8. From VMware to IBM Cloud VPC VSI, part 8: Veeam Backup and Replication

    If you consider the VPC object model, it is clear that to deploy and manage a large-scale environment, you need to consider the use of automation. Your operational challenge is further multiplied if you are planning for disaster recovery and intend to replicate or re-create your VPC environment from one region to another.

    I’ve created a relatively simple set of Terraform modules with the goal of demonstrating how to:

    1. Deploy a simple two-tier autoscaling application to one IBM Cloud region;
    2. Create a skeleton VPC network topology for this application in a second IBM Cloud region, in preparation for disaster recovery;
    3. Replicate application data from the first region to the second region; and
    4. Failover the application to the second region in case of a disaster

    Application design

    My two-tier application is a toy application. It consists of:

    1. An IBM Cloud regional application load balancer; in front of
    2. A set of three or more autoscaling application VSIs spread across the three zones of the region; connected to
    3. A pair of primary-standby database VSIs spread across the first two zones of the region

    The “application” running in the first tier is simply SSH. The VSIs share a common SSH host key much like you would share a certificate among a web application.

    The database is PostgreSQL configured in streaming replication mode. The “application” connection to the database is simply by means of the psql command. The database is configured to allow direct connection from the application VSIs without password.

    Apply complete! Resources: 76 added, 0 changed, 0 destroyed.

Outputs:

region1_lb_hostname = "2e3d3210-us-east.lb.appdomain.cloud"
region2_lb_hostname = "5532cedf-ca-tor.lb.appdomain.cloud"
smoonen@laptop ibmcloud-vpc-automation % ssh root@2e3d3210-us-east.lb.appdomain.cloud
Welcome to Ubuntu 24.04.3 LTS (GNU/Linux 6.8.0-1041-ibm x86_64)
. . .
root@smoonen-tier1-tlha2hqzy6-lrioe:~# psql -h db-primary.example.com testdb appuser
psql (16.11 (Ubuntu 16.11-0ubuntu0.24.04.1))
testdb=> \dt
              List of relations
 Schema |       Name       | Type  |  Owner   
--------+------------------+-------+----------
 public | test_replication | table | postgres
(1 row)

    Failover within the primary region of the primary database server is beyond the scope of this test. You would need to develop your own automation or administrative process to manage the PostgreSQL failover and the DNS reassignment.

    The following diagram illustrates the application topology as well as the storage replication that is established to a secondary region:

    Replication

    As we discussed previously, this automation leverages block storage snapshots and cross-region copies as a simple approach to replication. This imposes some limitations, including a lack of write-order consistency between volumes, and RPO constraints. This simple example has volumes that can be copied at hourly intervals, but a real-world example is likely to have a longer RPO.

    Because of the lack of write-order consistency, in this model you would need to assess which of the two databases had won the race and should be reconstituted as the primary database server. If you were storing and replicating application data (for example, transaction logs stored on IBM Cloud VPC file storage which is also being replicated to the secondary region) you would need to perform a similar analysis of consistency before completing the recovery process.

    In this example, since the application servers are stateless, their storage is not replicated to the secondary region. They can be re-created purely from their definition.

    Failover

    You can see from the diagram above that no running infrastructure other than the load balancer exists in the secondary region during steady-state replication. Upon failover, this example leverages an additional Terraform module to identify the most recent copied storage snapshot and re-create the instances and instance groups for the application and database servers.

    Details

    Refer to the documentation in the GitHub repository for additional instructions and considerations.

    DIY audio transcription on macOS

    Here’s how I transcribe audio on macOS. To prepare:

    1. Install Homebrew
    2. Install required tools: brew install ffmpeg whisper.cpp
    3. Download an appropriate Whisper speech recognition model, for example, from the Hugging Face repository. I have been using the ggml-large-v3.bin model with good results.

    Then, to convert a file:

    # Convert MP3 file to WAV
ffmpeg -i lecture01.mp3 -ar 16000 lecture01.wav

# Run Whisper model to transcribe audio
whisper-cli --language en --max-context 0 --max-len 65 --split-on-word --output-json --model ~/Downloads/ggml-large-v3.bin --file lecture01.wav --output-file lecture01

# Convert JSON output to text (see explanation and sample script below)
python3 json2text.py lecture01.json > lecture01.txt

# Cleanup
rm lecture01.wav lecture01.json

    Some notes on the above:

    1. The tool automatically appends a file extension on the output file name; this is why I have left the extension off in the --output-file specification.
    2. I had difficulty with the model hallucinating about 30 minutes into a 45-minute lecture recording; it generated repetitive text from that point to the end of the recording. I found the use of --max-context recommended to help with this, and it did help in my case. Other people recommend switching to the ggml-large-v2 model, or restarting the transcription near that point in the recording using the --offset-t parameter (note that this takes input in milliseconds).
    3. I like the idea of including timestamps in the output text, and while the tool prints this on stdout, it does not include them in an output text file. Therefore I’ve chosen to output JSON and convert it to text myself. I use the simple script below to do so.
    # json2text.py
import json, sys

j = json.load(open(sys.argv[1]))

for i in j['transcription'] :
  print(f"[{i['timestamps']['from']} --> {i['timestamps']['to']}]  {i['text']}")

    As a test case I transcribed a Biblical theology lecture, and I was pleased to find that the model had no difficulty with names such as Hittites, Ephraim, Manasseh, Melchizedek, and Eber. It also had a decent sense of capitalization of titles and of acronyms. My test case was also a relatively poor-quality recording and this did not seem to pose a problem for the model.

    I found on an M1 MacBook Air this 45-minute lecture took about 20 minutes to transcribe. It successfully leveraged the GPU. This was much faster than using the faster-whisper tool (about 120 minutes, using CPU only). I also attempted the insanely-fast-whisper tool, but this took even longer as well as having difficulty using the GPU. I confess I did no tuning, but in spite of using a batch-size of 4 (as recommended) it failed after many hours with a GPU allocation error. So I am quite pleased with the performance of whisper.cpp!

    By comparison, an M4 MacBook Pro was able to process the same 45-minute lecture in 3 minutes.

    From VMware to IBM Cloud VPC VSI, part 6: Disaster recovery

    See all blog posts in this series:

    1. From VMware to IBM Cloud VPC VSI, part 1: Introduction
    2. From VMware to IBM Cloud VPC VSI, part 2: VPC network design
    3. From VMware to IBM Cloud VPC VSI, part 3: Migrating virtual machines
    4. From VMware to IBM Cloud VPC VSI, part 4: Backup and restore
    5. From VMware to IBM Cloud VPC VSI, part 5: VPC object model
    6. From VMware to IBM Cloud VPC VSI, part 6: Disaster recovery
    7. From VMware to IBM Cloud VPC VSI, part 7: Automation
    8. From VMware to IBM Cloud VPC VSI, part 8: Veeam Backup and Replication

    In this article we will briefly consider the native capabilities of IBM Cloud VPC VSI that you could use to build a disaster recovery solution and compare this with alternative approaches.

    Copied snapshots

    As we saw previously, the IBM Cloud Backup for VPC backup policies allow you not only to schedule the creation of snapshots for your VSI volumes, but you can also schedule the copying of these snapshots to another IBM Cloud region. You could use this approach to perform periodic replication of all of your VSI data to another region for the purpose of disaster recovery. This approach has a number of limitations that you should take into consideration:

    • We saw that for in-region snapshots, you can create policies that will generate consistent snapshots of all of the volumes of individual VSIs, but not of multiple VSIs. By contrast, policy-based cross-region snapshot copies are available only if you snapshot volumes by tag without respect to their associated VSI. These snapshots and copies are not write-order consistent even within a single VSI. If you wanted to work around this limitation, you would need to use the API or CLI to invoke VSI-consistent snapshots and then separately invoke individual copies of each volume to another region.
    • You could combine such automation with automated quiesce activity in your VSIs if you wanted to ensure that you had stable replicas or even to possibly achieve write-order consistency across a fleet of VSIs.
    • Backup for VPC allows you to use crontab style expressions to schedule the snapshot and copy. Note that in principle your snapshots and copies within a given region exist in a space-efficient chain. However, you should note that the size of your volumes will affect the time that it takes to perform the initial full copy from region to region. And furthermore, for performance reasons you will need to backoff your snapshot and copy frequency based on your volume size if you want the cross-region copy to be incremental; see this reference table. Thus, for example, in my testing I had a 250GB boot volume and needed to set my snapshot and copy frequency to 2 hours.
    • The table seems to indicate a minimum expectation. I found in some cases that even with the 2-hour interval, sometimes the copied snapshot size reflected a full copy rather than an incremental copy.
    • In any case, whether the snapshot is being copied incrementally or in full, the copy time results in an effective RPO for this setup that is somewhat longer than the interval of 2 hours. Although these copies are space efficient they are not true replicas, which would typically have a much lower RPO.
    • I recommend that you configure your policy to keep at least 3 copies in the source region and 3 copies in the destination region. This is to ensure that not only do you have a viable copy at all times (minimum of 2 copies) but also that you are not asking the IBM Cloud storage system to calculate an increment off of the most recent snapshot at the same time it is deleting and consolidating an older snapshot into that same snapshot.
    • As we discussed for backups and our review of the VPC object model, you will need to plan to reconstitute all aspects of your environment and not just the storage volumes. Some resources like public IP addresses must change in the new region. You also need to reroute private network connectivity to the new region.
    • When you recreate your VM in the new region, it will be provisioned with a new uuid and this will cause cloud-init to re-run; you should be prepared for its side effects such as resetting your root password and authorized SSH keys.

    Depending on your application and requirements, you may be able to work with these limitations. If not, you will need to devise an alternate approach.

    Moving up the stack

    It is well known that you need to move up the stack—or invest in solutions that stretch across layers of the stack—to achieve more stringent BCDR goals. For example, you may be able to leverage storage array replication for highly efficient replication with low RPOs, but you will need to pair this with a solution that is able to quiesce your file system or your database if you want your replicas to be transactionally consistent rather than merely crash consistent.

    Thus, enterprise architectures often need to leverage agent-based tools or application- and database-specific methods either to perform the replication or at least to orchestrate it. Such approaches are highly dependent on your solution architecture, including your choice of operating systems, application software, and messaging and database software.

    Because of this, you need to investigate and evaluate which tools and techniques are suitable for your solution architecture and your disaster recovery objectives. For example, if you are using DB2, you might consider Q replication or SQL replication to replicate your database between different IBM Cloud regions. Use of OS agents tends to be more common in the backup realm than in the disaster recovery realm, but this may be a viable option for you depending on your RPO objectives. However, for agent-based backups you will need to investigate whether your recovery options are limited due to the current lack of support for booting a VSI from an ISO image.

    Approaches like this typically depend on having active infrastructure running in both your production and DR locations. This complicates some aspects of planning and execution; for example, your replicated infrastructure will likely not have the same IP addressing as your original infrastructure, and you will likely use DNS updates to hide this from your application users. On the other hand, it simplifies other aspects of your planning and execution, because you will have pre-created most of the necessary resources instead of needing to create them during the failover.

    From VMware to IBM Cloud VPC VSI, part 5: VPC object model

    See all blog posts in this series:

    1. From VMware to IBM Cloud VPC VSI, part 1: Introduction
    2. From VMware to IBM Cloud VPC VSI, part 2: VPC network design
    3. From VMware to IBM Cloud VPC VSI, part 3: Migrating virtual machines
    4. From VMware to IBM Cloud VPC VSI, part 4: Backup and restore
    5. From VMware to IBM Cloud VPC VSI, part 5: VPC object model
    6. From VMware to IBM Cloud VPC VSI, part 6: Disaster recovery
    7. From VMware to IBM Cloud VPC VSI, part 7: Automation
    8. From VMware to IBM Cloud VPC VSI, part 8: Veeam Backup and Replication

    We interrupt our normal VPC VSI programming to briefly discuss the VPC object model, which is relevant not only to your VPC design and your automation strategy, but also to your backup and especially your DR scenarios. For BCDR, you need to make plans to reconstitute all of these resources. As we have already discussed, it is not sufficient to backup your VSI volumes; you need to be prepared to reconstitute the VSI itself, including details such as the instance profile, IP address, floating IP, and security groups.

    Here is my rough attempt to diagram the VPC object model to help you think about your VPC design as well as your BCDR design and planning. Afterwards I will list some caveats.

    Some resources I have designated by abbreviation (e.g., VPE, LB, FIP, PAR, VNI). I have attempted to specify some cardinality based on my understanding, but it is likely that I have made some mistakes. I’ve also set aside some related resources to the side as a kind of appendix rather than attempting to include them and all of their possible relations in the main diagram. Because the security group impact is so extensive, I have used blue color coding to highlight the extent of its influence.

    I’ve included some loosely coupled resources (such as DNS service), but not all such resources (for example, you may be leveraging Cloud Object Storage, IBM Cloud Databases, or Backup and Recovery resources connected through a VPE).

    There are other considerations you will need to make such as IBM Cloud IAM permissions; these apply to every single resource. Minimally you need to consider which resource groups each resource is placed in and which users, access groups, and services should have access to them. If you are using the metadata service or allowing your VSIs to access trusted profiles and other resources, you will also need to consider the appropriate IAM configuration for this as well.

    You may also need to consider quota and capacity management.

    From VMware to IBM Cloud VPC VSI, part 4: Backup and restore

    See all blog posts in this series:

    1. From VMware to IBM Cloud VPC VSI, part 1: Introduction
    2. From VMware to IBM Cloud VPC VSI, part 2: VPC network design
    3. From VMware to IBM Cloud VPC VSI, part 3: Migrating virtual machines
    4. From VMware to IBM Cloud VPC VSI, part 4: Backup and restore
    5. From VMware to IBM Cloud VPC VSI, part 5: VPC object model
    6. From VMware to IBM Cloud VPC VSI, part 6: Disaster recovery
    7. From VMware to IBM Cloud VPC VSI, part 7: Automation
    8. From VMware to IBM Cloud VPC VSI, part 8: Veeam Backup and Replication

    IBM Cloud provides two backup offerings that are relevant to the backup of your applications running on IBM Cloud VSI:

    1. For catastrophic VM failures, leverage IBM Cloud Backup for VPC to perform crash-consistent volume backup and recovery for entire virtual machines
    2. For recovery of specific files and folders, leverage IBM Cloud Backup and Recovery‘s agent-based “physical server” backup capabilities to backup part or all of your VSI filesystems, and restore selected files or folders either to the original location or a new location

    Because of these offerings’ complementary focus on volume backup versus file backup, you will need to combine the two of them to cover all failure scenarios. Let’s consider their capabilities and limitations in turn.

    Backup for VPC—volume snapshots

    If you navigate the IBM Cloud “hamburger menu” to Infrastructure | Storage | Backup policies, you can create backup policies. These backup policies allow you to select one or more volumes (block or file) by volume tagging criteria, or to select the volumes for one or more VSIs by VSI tagging criteria. You can define up to four schedules for the snapshots, meaning that you could, for example, have a daily schedule with 7 days of retention plus a weekly schedule with 90 days of retention. IBM Cloud maintains the snapshots in a space-efficient chain. Unlike VMware snapshots, IBM Cloud’s block storage snapshots exist in a separate chain from the VSI volumes and the VSI boot image. IBM Cloud’s snapshots remain intact even if the VSI is deleted. There are some important things to be aware of:

    1. You can also schedule your own automation to create point-in-time snapshots of either individual volumes or consistent snapshots of sets of volumes for individual VSIs. You could use this approach, for example, if you want to quiesce a database or filesystem prior to taking the snapshot.
    2. Snapshots for VSI volumes (either standalone snapshots or snapshots created via backup policy) are write-order-consistent across all of the volumes for the VSI. However, it is not currently possible to ensure write-order consistency across multiple VSIs unless you employ your own out-of-band approach to quiesce database or filesystem writes during the snapshot.
    3. Consistency groups are not currently supported for the second-generation sdp volume profiles. I hope for this to change over time, but for now I recommend against using them.
    4. If you plan to leverage the Backup for VPC service to create backup policies rather than directly invoking snapshots yourself, you need to create a set of four service-to-service authorizations between it and several other resources in your VPC, so that it can perform the snapshot scheduling and execution. It is important to create the specific detailed authorizations to these four resource types; in my experience, creating a blanket authorization for Backup for VPC to manage all of VPC Infrastructure Services resources does not work.
    5. If you require a resilient backup outside of the region where your VSI is running, it is possible to copy individual snapshots (one by one, not as part of their consistency groups) to another region. You can automate this process to ensure that some number of your backups are available outside of your region.
    6. As the number of VSIs and volumes grows, multiplied by your backup policy, you will find that visualizing and managing your consistency groups and snapshots in the IBM Cloud UI becomes unwieldy. You will likely need to build your own automation for visualizing and managing these.
    7. The process described here only backs up your VSI volumes. Additional VSI configuration such as its name, instance profile, VNI and IP address, security group, floating IP, etc. is not backed up by this approach. In order to restore your data, you will need to create a new VSI from the volume(s) or from each consistency group, and you will need to reconstitute all of the additional configuration for the VSI. You will likely need to build your own approach to recording this data and automating the restoration of VSIs if you expect to need to restore VSIs at scale.
    8. There is a concept of fast restore where select snapshots are copied from the regional backup storage to zonal block storage for fast provisioning. However, fast restore is available only for snapshots of individual volumes, not for VSI consistency groups.
    9. Consult the IBM Cloud docs for additional limitations on backup policies.

    Whole-volume and whole-VSI backup and restore is quite heavy-handed for some backup scenarios such as recovery of deleted files. For your convenience, you may wish to complement the Backup for VPC capabilities by also using Backup and Recovery.

    Backup and Recovery of files and folders

    The Backup and Recovery offering leverages agents running on your VSIs (although it calls them “physical servers”) to backup files, folders, and certain databases. IBM publishes a list of currently supported operating system and database versions.

    The steps you will follow to leverage this service are:

    1. Open the IBM Cloud “hamburger menu” and navigate to Backup and Recovery | Backup service instances
    2. Create an instance in the region of your choice. Note that Backup and Recovery currently integrates with HPCS encryption keys, but support for Key Protect encryption keys is forthcoming. If you don’t configure your own key, your backup storage will be encrypted using an IBM-managed key.
    3. Click on the details of your newly created instance and then click on Launch dashboard to login to your instance’s dashboard. Although the dashboard leverages the same credentials you use for IBM Cloud, the dashboard does not have dynamic SSO integration which means that you will need to login a second time.
    4. Navigate to System | Data Source Connections and create a logical connection representing your VPC. If you have multiple VPCs you should create a connection for each one.
    5. Now you need to perform some steps within your IBM Cloud VPC:
      • Prepare your VPC by creating a virtual private endpoint (VPE) gateway for the cloud object storage (COS) service in the same region as your VPC. You may already have such a gateway created if you have provisioned an IBM Cloud kubernetes instance into your VPC.
      • Prepare your VPC by creating a VPE gateway for your new Backup and Recovery instance.
      • Follow the instructions to create one or more VSI connectors within your VPC that will serve as the data movers for the logical connection that you created in the previous step. IBM recommends that you create at least two VSI connectors for high availability, and recommends as a rule of thumb that you create one connector for every 10 VSIs that you will be backing up simultaneously.
      • It’s important to ensure that your security groups allow for these connector VSIs to communicate with the two VPEs and with your workload VSIs. By default this is normally the case; however, in my case I found that the COS VPE which had been previously created by my IBM Cloud kubernetes service did not share a security group with my connector VSIs. This prevented my backups from succeeding until I corrected the problem.
    6. Now you need to prepare the workload VSIs that you will be backing up. You manage this within the Data Protection | Sources view in your Backup and Restore dashboard. From this view, you will download the agents for your VSI, then register each VSI as a “physical server.” In the process of doing this, you will associate it with the logical connection that you created previously. This will trigger your connectors to connect to the agents on your VSI. If you are carefully managing the firewall on your VSI you will want to take note of the ports you need to open between the connectors and your workload VSIs.
    7. Now you can manage your backup schedule in the Data Protection | Policies view, and schedule backup jobs leveraging these policies by creating protection groups in the Data Protection | Protection view. By default the system will backup all files on your VSI, but you can also select lists of folders either to include or to exclude.
    8. Finally, you can recover files and folders using the Data Protection | Recoveries view. You have the option of downloading them through your browser, restoring them to the original system (either in the original location or in a temporary folder), or restoring them to an alternative system that is running the backup agent and is registered in Data Protection | Sources.

    As always, be sure to thoroughly review the documentation to familiarize yourself with other considerations such as alerting.