How network outages affect PureApplication multi-system deployment

How network outages affect PureApplication multi-system deployment

What happens if network connectivity is lost in your multi-system deployment? Because of the variety of network communications that take place, the answer is “it depends.”

There are four different network endpoints involved in PureApplication System’s multi-system deployment:

  1. The virtual machine data addresses (on NIC en1/eth1)
  2. The virtual machine management addresses (on NIC en0/eth0)
  3. The PureApplication Systems’ management addresses
  4. The iSCSI tiebreaker address

Between these addresses, there are five different network interactions that take place. Connectivity failures in or between these networks result in different outcomes:

  1. Communication between all of the VMs on the deployment over their data addresses, [A to A]
    What happens if this communication is broken depends on the application being deployed. It might be application-to-deployment manager traffic, or application-to-database traffic. Depending on the application, if this communication is broken the application may not be available. For example, if you have deployed GPFS mirrors across two sites, and the data communication is severed, then GPFS will still be available in one site provided that it can still access its GPFS tiebreaker. If you have deployed a WAS cluster using this GPFS mirror, then the WAS custom nodes that can connect to the surviving GPFS mirror will still be able to function provided that they can access their database.
  2. Management communications between the virtual machines [B to B]
    See next.
  3. Management communications between the virtual machines and the system [B to C]
    These communications are used to keep the PureApplication UI up to date with the status of the system. If these communications are broken then the application is not affected, but some of the VMs may have unknown status in the UI. Scaling the deployment will not be possible if [B to C] communications are broken on both racks.
  4. Communication between the systems [C to C]
    1. If neither system can communicate with the iSCSI tiebreaker [C to D] then externally managed deployments on both systems are frozen (no deploys, no deletes, no scaling).
    2. If one system can communicate with the iSCSI tiebreaker [C to D], then external deployments are not frozen on that system but are frozen on the other system.
    3. If both systems can communicate with the iSCSI tiebreaker [C to D], then external deployments are not frozen on one system (unpredictable) but are frozen on the other system.
  5. Communication between the systems and the tiebreaker [C to D]
    If the systems can communicate with each other [C to C] then the tiebreaker communication is just a failsafe mechanism and it is harmless for it to experience an outage. However, if there is a double failure of communication between the systems [C to C] and also to the tiebreaker [C to D] then externally managed deployments on both systems will be frozen (no deploys, no delete, no scaling) as indicated above.

PureApplication System and AIX SMT

If you are running AIX on a PureApplication W3700 POWER8 system, you should pay attention to this APAR: IT14338: PureApplication System: Some AIX virtual machines have 8 SMT threads and others have 4 SMT threads per processor setting

The implications of this are that virtual machines that have been rebooted do not preserve their SMT8 setting and revert to SMT4. The fix for this issue is contained in the IBM AIX OS image beginning with PureApplication, but for any virtual machines deployed at earlier levels you need to take manual action to ensure the SMT8 setting is preserved.

You can preserve the SMT8 setting on your existing LPARs by following the instructions in this dwAnswers post: Why is SMT (simultaneous multi-thread) value set to 4 on my AIX virtual machine after VM reboot on PureApplication System W3700?

IBM Cloud and VMware Cloud Foundation

IBM Cloud and VMware Cloud Foundation

This morning, Pat Gelsinger of VMware announced the availability of VMware Cloud Foundation, a complete solution for provisioning and maintaining VMware virtualization environments. IBM Cloud’s vice president, Robert LeBlanc, joined Pat on stage at VMworld 2016 to announce the availability of VMware Cloud Foundation in IBM Cloud’s global public cloud infrastructure.

Using VMware Cloud Foundation to automate the deployment of vSphere hypervisors, vCenter, NSX, VSAN, and SDDC Manager, IBM and VMware have been able to reduce the time to provision a fully virtualized and validated compute, storage, and networking environment in the public cloud from weeks to hours. The resulting environment, built on the global reach and secure foundation of the IBM Cloud, can be scaled as needed to accommodate additional workload, and uses Cloud Foundation’s capabilities to manage and maintain the currency of the environment.


Today’s announcement builds on the IBM-VMware partnership announced in February, and represents significant effort by both IBM and VMware engineering. The IBM Cloud for VMware Solutions offering of VMware Cloud Foundation is planned to be available in September 2016.

Counting PureApplication System deployments

Counting PureApplication System deployments

You can count how many deployments you have cumulatively made on your PureApplication System since it was first installed using the following CLI script:

virtual_systems = http.get('/resources/virtualSystems/?type=WORKLOAD')
try :
  max_id = max([x['id'] for x in virtual_systems])
  print "The maximum deployment id is %d" % max_id
except :
  print "There are no running deployments"

This script takes advantage of the fact that PureApplication deployments have an internal identifier with a monotonically increasing value. This allows the script to account for older deployments that have been deleted. However, the count assumes that your most recent deployment is still active. If you currently have no deployments, it will not be able to calculate a result; or if you have deleted your most recent deployments, it will account for all deployments up to the most recent remaining deployment.

Hit List

IBM’s Verse email solution has a feature that dynamically displays your most frequently emailed contacts.

I call it my “chain letter hit list.” If I were going to send a chain letter, these are the folks that would receive it:


As you may have guessed from recent posts, my job responsibilities have shifted from a focus on PureApplication to a focus on IBM’s Cloud for VMware solutions. My “hit list” has also shifted to this new group of folks that I’m privileged and excited to work with.

Oh, and I promise not to send anyone a chain letter.