It’s also possible to leverage enterprise backup software such as Veeam Backup and Replication to perform volume-level backup and restore of your VSI workloads, but this requires a little more effort. Here are some high-level pointers for using Veeam specifically:
VBR 13 is available in an appliance form factor, including both ISO and OVA. Installation of the OVA in IBM Cloud VSI is possible but tricky. The OVA contains two disks and these must be extracted and transferred separately to a boot volume (or image) and a data volume before deploying them together as part of a single VSI.
You must use the Veeam Agent for Windows and the Veeam Agent for Linux to perform backups of your VSI workloads.
For restore, if your boot volume is not corrupted, you can use the Veeam agent on your VSI to perform restore of files, folders, and data volumes.
If your boot volume is corrupted, you can recover the original boot volume by the following process:
Create a new VSI or use an existing VSI running the Veeam agent
Delete your original VSI being careful to preserve all of the volumes
Attach the volume(s) you wish to restore to the VSI previously selected
Use the Veeam agent to restore the volume from backup
Detach the volume(s) from this VSI
Recreate your original VSI using the original volumes, and boot it
The recovery process described above is not sufficient to migrate a Veeam backup from a VMware environment to a VSI. This is because the recovery process does not address other aspects of migration including the installation of virtio drivers and cloud-init.
In IBM Cloud’s virtual server instance (VSI) API, the VSI object does not itself identify the creator of the VSI. However, the creator is known to the cloud resource controller, so you can use the cloud resource API to determine the creator of a VSI.
#!/bin/zsh
# Identify owners of all VPC VSIs in IBM Cloud account
users=$(ibmcloud account users --output json)
for region in $(ibmcloud regions --output json | jq -r '.[]|.Name')
do
ibmcloud target -r $region -q > /dev/null
for instance in $(ibmcloud is instances --output json | jq -r '.[]|.name')
do
# VSI names might be reused in multiple regions; filter by region
user=$(ibmcloud resource service-instance $instance --output json 2> /dev/null | jq -r ".[]|select(.region_id==\"$region\")|.created_by")
email=$(echo $users | jq -r ".[]|select(.ibmUniqueId==\"$user\")|.email")
echo "VSI $instance in region $region was deployed by $email"
done
done
If you consider the VPC object model, it is clear that to deploy and manage a large-scale environment, you need to consider the use of automation. Your operational challenge is further multiplied if you are planning for disaster recovery and intend to replicate or re-create your VPC environment from one region to another.
I’ve created a relatively simple set of Terraform modules with the goal of demonstrating how to:
Deploy a simple two-tier autoscaling application to one IBM Cloud region;
Create a skeleton VPC network topology for this application in a second IBM Cloud region, in preparation for disaster recovery;
Replicate application data from the first region to the second region; and
Failover the application to the second region in case of a disaster
Application design
My two-tier application is a toy application. It consists of:
An IBM Cloud regional application load balancer; in front of
A set of three or more autoscaling application VSIs spread across the three zones of the region; connected to
A pair of primary-standby database VSIs spread across the first two zones of the region
The “application” running in the first tier is simply SSH. The VSIs share a common SSH host key much like you would share a certificate among a web application.
The database is PostgreSQL configured in streaming replication mode. The “application” connection to the database is simply by means of the psql command. The database is configured to allow direct connection from the application VSIs without password.
Failover within the primary region of the primary database server is beyond the scope of this test. You would need to develop your own automation or administrative process to manage the PostgreSQL failover and the DNS reassignment.
The following diagram illustrates the application topology as well as the storage replication that is established to a secondary region:
Replication
As we discussed previously, this automation leverages block storage snapshots and cross-region copies as a simple approach to replication. This imposes some limitations, including a lack of write-order consistency between volumes, and RPO constraints. This simple example has volumes that can be copied at hourly intervals, but a real-world example is likely to have a longer RPO.
Because of the lack of write-order consistency, in this model you would need to assess which of the two databases had won the race and should be reconstituted as the primary database server. If you were storing and replicating application data (for example, transaction logs stored on IBM Cloud VPC file storage which is also being replicated to the secondary region) you would need to perform a similar analysis of consistency before completing the recovery process.
In this example, since the application servers are stateless, their storage is not replicated to the secondary region. They can be re-created purely from their definition.
Failover
You can see from the diagram above that no running infrastructure other than the load balancer exists in the secondary region during steady-state replication. Upon failover, this example leverages an additional Terraform module to identify the most recent copied storage snapshot and re-create the instances and instance groups for the application and database servers.
Download an appropriate Whisper speech recognition model, for example, from the Hugging Face repository. I have been using the ggml-large-v3.bin model with good results.
Then, to convert a file:
# Convert MP3 file to WAV
ffmpeg -i lecture01.mp3 -ar 16000 lecture01.wav
# Run Whisper model to transcribe audio
whisper-cli --language en --max-context 0 --max-len 65 --split-on-word --output-json --model ~/Downloads/ggml-large-v3.bin --file lecture01.wav --output-file lecture01
# Convert JSON output to text (see explanation and sample script below)
python3 json2text.py lecture01.json > lecture01.txt
# Cleanup
rm lecture01.wav lecture01.json
Some notes on the above:
The tool automatically appends a file extension on the output file name; this is why I have left the extension off in the --output-file specification.
I had difficulty with the model hallucinating about 30 minutes into a 45-minute lecture recording; it generated repetitive text from that point to the end of the recording. I found the use of --max-context recommended to help with this, and it did help in my case. Other people recommend switching to the ggml-large-v2 model, or restarting the transcription near that point in the recording using the --offset-t parameter (note that this takes input in milliseconds).
I like the idea of including timestamps in the output text, and while the tool prints this on stdout, it does not include them in an output text file. Therefore I’ve chosen to output JSON and convert it to text myself. I use the simple script below to do so.
# json2text.py
import json, sys
j = json.load(open(sys.argv[1]))
for i in j['transcription'] :
print(f"[{i['timestamps']['from']} --> {i['timestamps']['to']}] {i['text']}")
As a test case I transcribed a Biblical theology lecture, and I was pleased to find that the model had no difficulty with names such as Hittites, Ephraim, Manasseh, Melchizedek, and Eber. It also had a decent sense of capitalization of titles and of acronyms. My test case was also a relatively poor-quality recording and this did not seem to pose a problem for the model.
I found on an M1 MacBook Air this 45-minute lecture took about 20 minutes to transcribe. It successfully leveraged the GPU. This was much faster than using the faster-whisper tool (about 120 minutes, using CPU only). I also attempted the insanely-fast-whisper tool, but this took even longer as well as having difficulty using the GPU. I confess I did no tuning, but in spite of using a batch-size of 4 (as recommended) it failed after many hours with a GPU allocation error. So I am quite pleased with the performance of whisper.cpp!
By comparison, an M4 MacBook Pro was able to process the same 45-minute lecture in 3 minutes.
In this article we will briefly consider the native capabilities of IBM Cloud VPC VSI that you could use to build a disaster recovery solution and compare this with alternative approaches.
Copied snapshots
As we saw previously, the IBM Cloud Backup for VPC backup policies allow you not only to schedule the creation of snapshots for your VSI volumes, but you can also schedule the copying of these snapshots to another IBM Cloud region. You could use this approach to perform periodic replication of all of your VSI data to another region for the purpose of disaster recovery. This approach has a number of limitations that you should take into consideration:
We saw that for in-region snapshots, you can create policies that will generate consistent snapshots of all of the volumes of individual VSIs, but not of multiple VSIs. By contrast, policy-based cross-region snapshot copies are available only if you snapshot volumes by tag without respect to their associated VSI. These snapshots and copies are not write-order consistent even within a single VSI. If you wanted to work around this limitation, you would need to use the API or CLI to invoke VSI-consistent snapshots and then separately invoke individual copies of each volume to another region.
You could combine such automation with automated quiesce activity in your VSIs if you wanted to ensure that you had stable replicas or even to possibly achieve write-order consistency across a fleet of VSIs.
Backup for VPC allows you to use crontab style expressions to schedule the snapshot and copy. Note that in principle your snapshots and copies within a given region exist in a space-efficient chain. However, you should note that the size of your volumes will affect the time that it takes to perform the initial full copy from region to region. And furthermore, for performance reasons you will need to backoff your snapshot and copy frequency based on your volume size if you want the cross-region copy to be incremental; see this reference table. Thus, for example, in my testing I had a 250GB boot volume and needed to set my snapshot and copy frequency to 2 hours.
The table seems to indicate a minimum expectation. I found in some cases that even with the 2-hour interval, sometimes the copied snapshot size reflected a full copy rather than an incremental copy.
In any case, whether the snapshot is being copied incrementally or in full, the copy time results in an effective RPO for this setup that is somewhat longer than the interval of 2 hours. Although these copies are space efficient they are not true replicas, which would typically have a much lower RPO.
I recommend that you configure your policy to keep at least 3 copies in the source region and 3 copies in the destination region. This is to ensure that not only do you have a viable copy at all times (minimum of 2 copies) but also that you are not asking the IBM Cloud storage system to calculate an increment off of the most recent snapshot at the same time it is deleting and consolidating an older snapshot into that same snapshot.
As we discussed for backups and our review of the VPC object model, you will need to plan to reconstitute all aspects of your environment and not just the storage volumes. Some resources like public IP addresses must change in the new region. You also need to reroute private network connectivity to the new region.
When you recreate your VM in the new region, it will be provisioned with a new uuid and this will cause cloud-init to re-run; you should be prepared for its side effects such as resetting your root password and authorized SSH keys.
Depending on your application and requirements, you may be able to work with these limitations. If not, you will need to devise an alternate approach.
Moving up the stack
It is well known that you need to move up the stack—or invest in solutions that stretch across layers of the stack—to achieve more stringent BCDR goals. For example, you may be able to leverage storage array replication for highly efficient replication with low RPOs, but you will need to pair this with a solution that is able to quiesce your file system or your database if you want your replicas to be transactionally consistent rather than merely crash consistent.
Thus, enterprise architectures often need to leverage agent-based tools or application- and database-specific methods either to perform the replication or at least to orchestrate it. Such approaches are highly dependent on your solution architecture, including your choice of operating systems, application software, and messaging and database software.
Because of this, you need to investigate and evaluate which tools and techniques are suitable for your solution architecture and your disaster recovery objectives. For example, if you are using DB2, you might consider Q replication or SQL replication to replicate your database between different IBM Cloud regions. Use of OS agents tends to be more common in the backup realm than in the disaster recovery realm, but this may be a viable option for you depending on your RPO objectives. However, for agent-based backups you will need to investigate whether your recovery options are limited due to the current lack of support for booting a VSI from an ISO image.
Approaches like this typically depend on having active infrastructure running in both your production and DR locations. This complicates some aspects of planning and execution; for example, your replicated infrastructure will likely not have the same IP addressing as your original infrastructure, and you will likely use DNS updates to hide this from your application users. On the other hand, it simplifies other aspects of your planning and execution, because you will have pre-created most of the necessary resources instead of needing to create them during the failover.
We interrupt our normal VPC VSI programming to briefly discuss the VPC object model, which is relevant not only to your VPC design and your automation strategy, but also to your backup and especially your DR scenarios. For BCDR, you need to make plans to reconstitute all of these resources. As we have already discussed, it is not sufficient to backup your VSI volumes; you need to be prepared to reconstitute the VSI itself, including details such as the instance profile, IP address, floating IP, and security groups.
Here is my rough attempt to diagram the VPC object model to help you think about your VPC design as well as your BCDR design and planning. Afterwards I will list some caveats.
Some resources I have designated by abbreviation (e.g., VPE, LB, FIP, PAR, VNI). I have attempted to specify some cardinality based on my understanding, but it is likely that I have made some mistakes. I’ve also set aside some related resources to the side as a kind of appendix rather than attempting to include them and all of their possible relations in the main diagram. Because the security group impact is so extensive, I have used blue color coding to highlight the extent of its influence.
I’ve included some loosely coupled resources (such as DNS service), but not all such resources (for example, you may be leveraging Cloud Object Storage, IBM Cloud Databases, or Backup and Recovery resources connected through a VPE).
There are other considerations you will need to make such as IBM Cloud IAM permissions; these apply to every single resource. Minimally you need to consider which resource groups each resource is placed in and which users, access groups, and services should have access to them. If you are using the metadata service or allowing your VSIs to access trusted profiles and other resources, you will also need to consider the appropriate IAM configuration for this as well.
You may also need to consider quota and capacity management.
IBM Cloud provides two backup offerings that are relevant to the backup of your applications running on IBM Cloud VSI:
For catastrophic VM failures, leverage IBM Cloud Backup for VPC to perform crash-consistent volume backup and recovery for entire virtual machines
For recovery of specific files and folders, leverage IBM Cloud Backup and Recovery‘s agent-based “physical server” backup capabilities to backup part or all of your VSI filesystems, and restore selected files or folders either to the original location or a new location
Because of these offerings’ complementary focus on volume backup versus file backup, you will need to combine the two of them to cover all failure scenarios. Let’s consider their capabilities and limitations in turn.
Backup for VPC—volume snapshots
If you navigate the IBM Cloud “hamburger menu” to Infrastructure | Storage | Backup policies, you can create backup policies. These backup policies allow you to select one or more volumes (block or file) by volume tagging criteria, or to select the volumes for one or more VSIs by VSI tagging criteria. You can define up to four schedules for the snapshots, meaning that you could, for example, have a daily schedule with 7 days of retention plus a weekly schedule with 90 days of retention. IBM Cloud maintains the snapshots in a space-efficient chain. Unlike VMware snapshots, IBM Cloud’s block storage snapshots exist in a separate chain from the VSI volumes and the VSI boot image. IBM Cloud’s snapshots remain intact even if the VSI is deleted. There are some important things to be aware of:
You can also schedule your own automation to create point-in-time snapshots of either individual volumes or consistent snapshots of sets of volumes for individual VSIs. You could use this approach, for example, if you want to quiesce a database or filesystem prior to taking the snapshot.
Snapshots for VSI volumes (either standalone snapshots or snapshots created via backup policy) are write-order-consistent across all of the volumes for the VSI. However, it is not currently possible to ensure write-order consistency across multiple VSIs unless you employ your own out-of-band approach to quiesce database or filesystem writes during the snapshot.
Consistency groups are not currently supported for the second-generation sdp volume profiles. I hope for this to change over time, but for now I recommend against using them.
If you plan to leverage the Backup for VPC service to create backup policies rather than directly invoking snapshots yourself, you need to create a set of four service-to-service authorizations between it and several other resources in your VPC, so that it can perform the snapshot scheduling and execution. It is important to create the specific detailed authorizations to these four resource types; in my experience, creating a blanket authorization for Backup for VPC to manage all of VPC Infrastructure Services resources does not work.
If you require a resilient backup outside of the region where your VSI is running, it is possible to copy individual snapshots (one by one, not as part of their consistency groups) to another region. You can automate this process to ensure that some number of your backups are available outside of your region.
As the number of VSIs and volumes grows, multiplied by your backup policy, you will find that visualizing and managing your consistency groups and snapshots in the IBM Cloud UI becomes unwieldy. You will likely need to build your own automation for visualizing and managing these.
The process described here only backs up your VSI volumes. Additional VSI configuration such as its name, instance profile, VNI and IP address, security group, floating IP, etc. is not backed up by this approach. In order to restore your data, you will need to create a new VSI from the volume(s) or from each consistency group, and you will need to reconstitute all of the additional configuration for the VSI. You will likely need to build your own approach to recording this data and automating the restoration of VSIs if you expect to need to restore VSIs at scale.
There is a concept of fast restore where select snapshots are copied from the regional backup storage to zonal block storage for fast provisioning. However, fast restore is available only for snapshots of individual volumes, not for VSI consistency groups.
Whole-volume and whole-VSI backup and restore is quite heavy-handed for some backup scenarios such as recovery of deleted files. For your convenience, you may wish to complement the Backup for VPC capabilities by also using Backup and Recovery.
Backup and Recovery of files and folders
The Backup and Recovery offering leverages agents running on your VSIs (although it calls them “physical servers”) to backup files, folders, and certain databases. IBM publishes a list of currently supported operating system and database versions.
The steps you will follow to leverage this service are:
Open the IBM Cloud “hamburger menu” and navigate to Backup and Recovery | Backup service instances
Create an instance in the region of your choice. Note that Backup and Recovery currently integrates with HPCS encryption keys, but support for Key Protect encryption keys is forthcoming. If you don’t configure your own key, your backup storage will be encrypted using an IBM-managed key.
Click on the details of your newly created instance and then click on Launch dashboard to login to your instance’s dashboard. Although the dashboard leverages the same credentials you use for IBM Cloud, the dashboard does not have dynamic SSO integration which means that you will need to login a second time.
Navigate to System | Data Source Connections and create a logical connection representing your VPC. If you have multiple VPCs you should create a connection for each one.
Now you need to perform some steps within your IBM Cloud VPC:
Prepare your VPC by creating a virtual private endpoint (VPE) gateway for the cloud object storage (COS) service in the same region as your VPC. You may already have such a gateway created if you have provisioned an IBM Cloud kubernetes instance into your VPC.
Prepare your VPC by creating a VPE gateway for your new Backup and Recovery instance.
Follow the instructions to create one or more VSI connectors within your VPC that will serve as the data movers for the logical connection that you created in the previous step. IBM recommends that you create at least two VSI connectors for high availability, and recommends as a rule of thumb that you create one connector for every 10 VSIs that you will be backing up simultaneously.
It’s important to ensure that your security groups allow for these connector VSIs to communicate with the two VPEs and with your workload VSIs. By default this is normally the case; however, in my case I found that the COS VPE which had been previously created by my IBM Cloud kubernetes service did not share a security group with my connector VSIs. This prevented my backups from succeeding until I corrected the problem.
Now you need to prepare the workload VSIs that you will be backing up. You manage this within the Data Protection | Sources view in your Backup and Restore dashboard. From this view, you will download the agents for your VSI, then register each VSI as a “physical server.” In the process of doing this, you will associate it with the logical connection that you created previously. This will trigger your connectors to connect to the agents on your VSI. If you are carefully managing the firewall on your VSI you will want to take note of the ports you need to open between the connectors and your workload VSIs.
Now you can manage your backup schedule in the Data Protection | Policies view, and schedule backup jobs leveraging these policies by creating protection groups in the Data Protection | Protection view. By default the system will backup all files on your VSI, but you can also select lists of folders either to include or to exclude.
Finally, you can recover files and folders using the Data Protection | Recoveries view. You have the option of downloading them through your browser, restoring them to the original system (either in the original location or in a temporary folder), or restoring them to an alternative system that is running the backup agent and is registered in Data Protection | Sources.
As always, be sure to thoroughly review the documentation to familiarize yourself with other considerations such as alerting.
Among the VM to VSI migration techniques I considered, one possible option was to boot the destination VSI using a GHOST-like ISO tool. This is difficult to accomplish because IBM Cloud does not currently support booting a VSI from an ISO. What I did instead was to craft a qcow2 disk image that loaded this ISO. Then, any VSI booted using this image would initially load the tool. Afterwards the tool could be used to transfer the actual disk to be used for the VM.
There are some limitations for this approach. Most importantly, the use of the tool as an image template entirely obscures the actual underlying OS for any VSI booted from that image. It seems to me an abuse of the idea of an image template. Furthermore, given the limitations of many small Linux distributions, customizing the image so that it has all of the packages you need is potentially a tedious process. For example, I found that out of the box TinyCore Linux did not have openssh, qemu, nor did it have virtio drivers. Combined with the fact that I also wanted to leverage libguestfs in many cases, this was a challenging limitation.
As a result, I rejected this approach as one of the viable migration paths. However, it was still a fun experiment to build the boot image and boot my VSI using it. Here are the steps I took to create a TinyCore Linux boot image:
Edit /mnt/esp/boot/grub/grub.cfg: menuentry "Boot Core" { set isofile="/boot/core.iso" loopback loop $isofile linux (loop)/boot/vmlinuz quiet initrd (loop)/boot/core.gz }
Copy ISO: cp core.iso /mnt/esp/boot
Cleanup: umount /mnt/esp; losetup -d "$LOOPDEV"
Convert to qcow2: qemu-img convert -f raw -O qcow2 core.img core.qcow2
Upload this to COS and create an image as in our migration method #1. I chose generic OS.
Deploy a VSI using this image. Configure profile, volumes, and network as needed.
Display VNC console for your VSI. TinyCore booted with the network properly configured using DHCP. At this point if you had a public gateway attached to the VSI subnet, you could even install extensions like SSH: tce-load -wi openssh, or qemu tools: tce-load -wi qemu.
full◦valence 