Provisioning and expanding an IBM Cloud VMware instance via API

IBM Cloud for VMware Solutions recently released a set of public APIs. These APIs allow you to use your IBM Cloud API key to perform operations such as:

  • Get information about your vCenter instance, admin credentials, deployment history, clusters, and hosts
  • Verify parameters for ordering a new vCenter instance, cluster, or hosts
  • Order or remove a vCenter instance, cluster, or hosts

I’ve written some sample code demonstrating how you can authenticate with the IBM Cloud APIs using your API key, and how to interact with the IBM Cloud for VMware APIs. Note that these samples only perform order verification, but you can easily extend them to perform actual orders or removals.

A key use case for these APIs is to expand and contract your VMware instance based on utilization or for workload bursting scenarios. With these APIs, you can now fully automate this process.

Spectrum Protect Plus on IBM Cloud

Spectrum Protect Plus on IBM Cloud

IBM Cloud for VMware Solutions recently made available IBM Spectrum Protect Plus as part of our family of VMware offerings. Spectrum Protect Plus provides powerful and easy to use backup and restore capabilities for your VMware infrastructure and workload. It is now the default backup offering for VMware on IBM Cloud, complementing our existing offering of Veeam Backup & Replication.

At the same time, the IBM Cloud architecture team just published our Spectrum Protect Plus on IBM Cloud reference architecture. Read it and the associated references for information on how we have deployed Spectrum Protect Plus, how you should plan and size your deployment, and how to manage it.

Connecting to the IBM Cloud VPN

IBM Cloud offers a VPN service for your account which you can use to access your dedicated IBM Cloud network. The VPN access is available from your browser using a Java applet, but is also available using a standalone VPN application for Windows, Linux, or macOS.

Unfortunately, I’ve found that the version 2.0 update of the MotionPro Plus application for macOS has broken my VPN access. Not only has it lost all of the passwords I had previously saved, but when I do enter my password and attempt to connect to the IBM Cloud, it immediately disconnects.

While we await a fix from Array Networks, it is possible to revert to an older version of MotionPro:

  1. View the macOS Launchpad and find the MotionPro+ icon
  2. Click and hold the icon until it begins to jiggle, then release. Note that this will delete your MotionPro configuration.
  3. Click the X to uninstall MotionPro+
  4. From the Array Networks support site, download the latest Mac OS MotionPro client corresponding to AG-OS 9.4.0.x
  5. Open the disk image and run the MotionPro installer package within the image. The installer will also install some command line tools
  6. Recreate your MotionPro configuration

See also: managing SoftLayer VPN subnet access.

Monitoring your IBM Cloud vSphere servers

The IBM Cloud for VMware Solutions architecture specifies that vSphere (ESXi) servers should be attached to the public network, but should be configured not to enable their own public IP address. This ensures that workloads running on the servers can access the public network as necessary (e.g., using an NSX Edge Services Gateway), but that the hosts themselves cannot be reached over the internet.

When IBM Cloud (a.k.a. SoftLayer) provisions a bare metal server, the default monitoring configuration for that server is to ping its public IP address. This means that by default all of your ESXi hosts are reported by the IBM Cloud infrastructure portal to be down:

servers-down

You can correct this by re–configuring the monitor for each server to test the private IP address rather than the public IP address. Since you cannot change the IP address of a monitor, you will have to remove the existing monitor and create a new monitor for the private IP address.

If you have many bare metal servers in this situation, you’ll want to automate the re–configuration. To help with this, I wrote a Python script to reconfigure your bare metal server monitors. You’ll have to fill in your SoftLayer username and API key, and the script will reconfigure the monitors for all servers that are (1) marked down, and (2) have a monitor configured for their public IP but not their private IP. The new monitor for the private IP will retain the same attributes as the existing monitor for the private IP. Voila:

sl-monitor-fixed

Managing SoftLayer VPN subnet access

The IBM SoftLayer VPN only supports connection to 64 of your private subnets. If you have more than 64 private subnets in your SoftLayer account, you need to switch your VPN’s subnet management from Automatic to Manual, and select the specific subnets to which you want to connect.

The process for selecting subnets in the UI is not simple, especially if your account has hundreds of subnets. The subnets are not sorted, the dialog is small, and the pagination is slow.

VPN

However, it is possible to manage your VPN subnets programmatically using the SoftLayer API. I have created a Python script that allows you to manage your SoftLayer VPN subnet access. The script requires your SoftLayer username, API key, and a list of private IP addresses to which you want to connect. The script locates the subnets in your account that match your selected IP addresses, and assigns exactly these subnets to your SoftLayer VPN account.

You should wait a few minutes after running the script for it to take effect.

Ansible Playbook for Wekan

Ansible Playbook for Wekan

My team is experimenting with using open-source tools deployed internally for Kanban cards.

One tool we are exploring is Wekan, formerly known as Libreboard.

I deployed a Wekan instance to a RHEL 7 virtual machine for our testing. For this deployment, I wrote a simple Ansible playbook with a few additional configuration files (nginx config, Node PM2 configuration), in case there is ever a need to re-deploy the instance.

You can find my playbook and associated files on Github: wekan-setup. The files are as follows:

  • purekan.yml—Ansible playbook
  • wekan.yml—Node PM2 configuration
  • wekan.conf—nginx proxy configuration

You’ll need to customize things slightly based on your domain name or if you are using a distribution other than RHEL.

Kindle

I bought a basic Kindle recently and I’m enjoying it. I don’t currently plan to buy many e-books, but rather use the Kindle as a better tool for existing reading compared to my computer and phone. Here’s what I’ve discovered so far:

PDFs

PDF’s aren’t the best format for reading on Kindle. I’ve found two tools for converting PDFs to e-books and uploading them to my Kindle. For simple PDFs (e.g., single column, and not a scanned image), Calibre is great for converting and uploading. However, Calibre does a poor job with PDFs that are scanned copies of books (this applies to many of the books linked above). For these I use a two-step process: first, I run the PDF through the K2PDFOpt tool (at the time of this writing, version 1.63 crashes for me on some books, but 1.51 is stable). This increases the size of the PDF file significantly, but it organizes it in a form that Calibre is much better able to handle. Then I use Calibre to convert these PDFs to e-books, and upload them to my Kindle.

Articles

Until now, I saved longer articles and blog posts for later reading using open tabs in my browser. This quickly grows unwieldy. The Instapaper service allows you to save web pages for later reading, and it integrates with Kindle. Now when I run across a longer article, I click a button to send it to Instapaper, and by the next morning the article is ready to read on my Kindle.

Blogs

The Kindlefeeder service allows you to send blog and news feeds to your Kindle. I’ve selected several of the blogs I read (ones that tend to have longer articles) to be sent to my Kindle, and now I read them there rather than on my computer.

Other

If you have any other tips and tricks I’d appreciate hearing about them.

All of the above should work with e-readers other than Kindle. In the case of Instapaper and Kindlefeeder, you may need to upload a file manually to your reader instead of having it automatically sent there.

SmugMug uploader

[SmugMug]I’ve written a small Python script to upload pictures to a SmugMug gallery. I love SmugMug and use it extensively for family photos. I’m using this script for my personal use because it’s much simpler and much less of a resource hog than a browser-based uploader, and also because it was a fun exercise to try out the SmugMug API. You can run this script as follows to upload one or more files:

python upload.py gallery-name picture-file-name . . .

On Windows I’ve set up a desktop shortcut pointing to the script, and I can drag and drop a pile of picture files onto the icon and it will upload away. I’ve tested it using both Python 2.5 using simplejson, and also using Python 2.6 which has simplejson built in. Earlier versions of Python may require you to change the import of hashlib to md5, and change the hashlib.md5() invocation to a md5.new() invocation. You’ll also need to modify the script to contain your email address and SmugMug password, and obtain a SmugMug API key for your own development use, but this is a very painless process. Here is the script:

#!/usr/bin/python

##########
# Requirements: Python 2.6 or
#               simplejson from http://pypi.python.org/pypi/simplejson
##########

EMAIL='...'
PASSWORD='...'

##########
APIKEY='...'
API_VERSION='1.2.2'
API_URL='https://api.smugmug.com/services/api/json/1.2.2/'
UPLOAD_URL='http://upload.smugmug.com/photos/xmlrawadd.mg'

import sys, re, urllib, urllib2, urlparse, hashlib, traceback, os.path
try    : import json
except : import simplejson as json

if len(sys.argv) < 3 :
  print 'Usage:'
  print '  upload.py  album  picture1  [picture2  [...]]'
  print
  sys.exit(0)

album_name = sys.argv[1]
su_cookie  = None

def safe_geturl(request) :
  global su_cookie

  # Try up to three times
  for x in range(5) :
    try :
      response_obj = urllib2.urlopen(request)
      response = response_obj.read()
      result = json.loads(response)

      # Test for presence of _su cookie and consume it
      meta_info = response_obj.info()
      if meta_info.has_key('set-cookie') :
        match = re.search('(_su=S+);', meta_info['set-cookie'])
        if match and match.group(1) != "_su=deleted" :
          su_cookie = match.group(1)
      if result['stat'] != 'ok' : raise Exception('Bad result code')
      return result
    except :
      if x < 4 :
        print "  ... failed, retrying"
      else :
        print "  ... failed, giving up"
        print "  Request was:"
        print "  " + request.get_full_url()
        try :
          print "  Response was:"
          print response
        except :
          pass
        traceback.print_exc()
        #sys.stdin.readline()
        #sys.exit(1)
        return result

def smugmug_request(method, params) :
  global su_cookie

  paramstrings = [urllib.quote(key)+'='+urllib.quote(params[key]) for key in params]
  paramstrings += ['method=' + method]
  url = urlparse.urljoin(API_URL, '?' + '&'.join(paramstrings))
  request = urllib2.Request(url)
  if su_cookie :
    request.add_header('Cookie', su_cookie)
  return safe_geturl(request)

result = smugmug_request('smugmug.login.withPassword',
                         {'APIKey'       : APIKEY,
                          'EmailAddress' : EMAIL,
                          'Password'     : PASSWORD})
session = result['Login']['Session']['id']

result = smugmug_request('smugmug.albums.get', {'SessionID' : session})
album_id = None
for album in result['Albums'] :
  if album['Title'] == album_name :
    album_id = album['id']
    break
if album_id is None :
  print 'That album does not exist'
  sys.exit(1)

for filename in sys.argv[2:] :
  data = open(filename, 'rb').read()
  print 'Uploading ' + filename
  upload_request = urllib2.Request(UPLOAD_URL,
                                   data,
                                   {'Content-Length'  : len(data),
                                    'Content-MD5'     : hashlib.md5(data).hexdigest(),
                                    'Content-Type'    : 'none',
                                    'X-Smug-SessionID': session,
                                    'X-Smug-Version'  : API_VERSION,
                                    'X-Smug-ResponseType' : 'JSON',
                                    'X-Smug-AlbumID'  : album_id,
                                    'X-Smug-FileName' : os.path.basename(filename) })
  result = safe_geturl(upload_request)
  if result['stat'] == 'ok' :
    print "  ... successful"

print 'Done'
# sys.stdin.readline()

I am donating this script to the public domain. You are welcome to use and modify it as you please without conditions. I’d appreciate hearing about your experience with this script or any changes and improvements you’ve made; please leave a comment. Thanks!

Update 2010-07-20

Since I first posted this, I’ve updated it as follows:

  1. Add a Content-Type header of ‘none’. This is to workaround a bug in the SmugMug API.
  2. Use basename() to send only the file’s basename for X-Smug-FileName.
  3. Rewrite safe_geturl() to loop up to five times if the upload attempt fails. I’ve found that uploading is surprisingly unreliable, and re-attempting the upload generally works fine.
  4. Add a commented call to readline() at the end of the script. In my case, I run my script by dragging files onto an icon on my Windows desktop, which causes it to run in a DOS window and vanish when done. If you uncomment this line, it will wait for you to press Enter when it is done uploading. You’ll be able to see any files that weren’t uploaded successfully.

Update 2010-11-28

SmugMug made a recent change to their API’s login behavior which broke this script. While the new login behavior is not documented in the API docs, the fix is apparently to use a session cookie along with the session ID. While it’s a bit of a kludge, I’ve updated the script above to save this cookie in a global variable and submit it on subsequent requests.

Update 2011-06-24

I’ve fixed a bug in the script causing it to wrongly report a failure for certain requests that don’t send back the session cookie. The fix involves testing whether a set-cookie header was returned before accessing the header.

Update 2013-10-01

Version 1.2.0 of the SmugMug API has stopped working, so I have updated the script to use version 1.2.2 of the API.