Mirantis Acquires Docker Enterprise Platform Business


An OpenStack guy takes CloudStack for a test drive

Mirantis writing a blog about CloudStack? WTFBBQ? No, it’s not April Fool’s day. As much as we love OpenStack here at Mirantis, I wanted to see for myself what the CloudStack platform offers, things it does better and worse, what works and what doesn’t, how it stacks (pun intended) up against OpenStack, and give CloudStack credit where it’s due.

The idea came to me from a Mirantis Bootcamp OpenStack training session we delivered where a number of attendees asked about CloudStack and how it compares to OpenStack. I spent some time researching and didn’t find many technical reviews, let alone comparisons. I decided the best way to learn more and better prepare for this question was to deploy CloudStack in a test environment and try it myself. The following are my impressions of CloudStack along with comparison notes to OpenStack when applicable.

CloudStack Installation and Configuration

An exercise in prerequisites

If someone told me a week ago that I couldn’t install CloudStack on top of the latest Ubuntu 12.04 LTS release, I would have been shocked and assumed they got their facts mixed up. However, after attempting a basic two-node CloudStack installation, I’m convinced that it’s easier to use the two-year-old version of Ubuntu (listed in their requirements) and install the missing hardware drivers manually than it is to use Ubuntu 12.04 LTS and deal with script hacking and modification to get CloudStack running.

Attempt #1: First, I proceeded to ignore the install guide and used Ubuntu 12.04 LTS instead of Ubuntu 10.04 LTS because I did not want to install my hardware drivers manually. I expected to have minor problems with Ubuntu 12.04 with easy workarounds. Instead, I was met with error after error when running the install scripts. Each time I found a workaround through online forums for one script, I’d run into a new error on the next script. I quickly concluded it was a dead end and gave up.


Attempt #2: Eventually, I gave in and downgraded to Ubuntu 10.04 (almost three years old) and manually installed the drivers for my hardware. I was able to get further in the installation process but ran in to some hiccups when attempting add a host and provision my instances, more details to follow.

I did not try Red Hat or Centos with CloudStack but found similar complaints here: CloudStack OS Versions.

Comparison Note: CloudStack’s insistence of having to use an outdated Operating System that required manual driver installation is cumbersome. OpenStack wins this category since it works with latest operating systems. In addition, it has Linux community support from Ubuntu, Redhat, SUSE etc.


CloudStack setup and configuration wizard

Now, a shining moment for CloudStack. The wizard for setting up the infrastructure is really slick and easy to use. It reminds of the wizards they use for routers. You can click through the screens I captured below to get an idea. They also included mouse over hints on the parameters in case you have trouble filling out the data. These guys really nailed this part of the experience.

Comparison Note: No wizards are available in Openstack Horizon. In addition, things like networking need to be set up with CLI or nova.conf

“Unable to add CloudStack host”

As you may have noted from the slide deck above, it ended with “Add Host”. This is because I was unable to add a host and continue the wizard. There were two reason why the system kept failing with “unable to add host”.

  1. SSH is not installed on Ubuntu 10.04 LTS but it’s required in order for a CloudStack host to connect to Cloud Management Server. While attempting to add a host, the wizard complained with “unable to add host”  which couldn’t be less helpful. I found the root cause after looking at /var/log/cloud/management/management-log and saw an error about not being able to connect to the agent on port 22.
  2. After getting past the first error by installing SSH, I was still “unable to add host” due to missing network configurations. The cause:  Ubuntu has a GUI for configuring networking which is the default way a user would set up the networking but isn’t usable CloudStack. CloudStack requires networking to be configured using /etc/network/interfaces, otherwise it will think it has no network. See here for help Ubuntu Network Configuration

Neither of these requirements are a big deal to resolve; the problem was that they weren’t documented.

Comparison Note: I had fewer problems getting from bare metal to provisioning instances on the OpenStack platform. While OpenStack lacks fancy wizards and setup GUI for configuring infrastructure, the installation process was smooth and basic functions like creating VM instances, retrieving images, and attaching volumes just worked. None of the CloudStack problems I encountered were mentioned in the documentation, which could lead to frustration and installers giving up.


CloudStack Features

Zones, Pods, Clusters and Hosts

CloudStack uses Organizational Units (OU) to group hosts within a CloudStack deployment. Zones, the largest OU, usually corresponds to a datacenter and consists of at least one pod and a ‘Secondary’ storage shared by all pods in a zone. Pods represent a single rack and consist of at least one cluster. Clusters in the same pod use the same subnet.  All hosts in a cluster are homogeneous in that they use the same hardware, hypervisor, subnet, and access the same shared storage (‘Primary’ Storage). Hosts are single nodes running hypervisor software that provide computing resources to run guest VMs.

Comparison Note: While this is a structured and clear approach to grouping/nesting objects, it doesn’t seem customizable and lacks flexibility. What if a deployment calls for five layers for example? Or what if I only have two layers starting with pods, why do I need to fill in information about Zones?


CloudStack Storage

CloudStack offers two types of storage for use by the platform, which they call ‘Primary’ and ‘Secondary’. Primary storage is configured at the cluster level and stores all disk volumes for VMs in a cluster. These volumes are persistent, and will remain after you terminate the VM. Local disk, iSCSI, and NFS can be used for Primary storage.  Secondary storage is configured at the Zone level, and is used to store templates, ISOs, and snapshots. All zones need at least one Secondary storage. The documentation says that either NFS and OpenStack Swift can be used as Secondary storage.

Comparison Note: I would say Primary storage is comparable to OpenStack nova-volume in that it supports multiple disk formats and would be just as scalable. Regarding Secondary storage, it’s not quite as flexible as Glance (OpenStack’s image service). Glance is capable of using any back end, including local disk on the controller node to store images, whereas Cloudstack Secondary Storage uses NFS or OpenStack Swift. Also, CloudStack does not have its own object storage (as in a comparable product to OpenStack Swift or Amazon S3).


CloudStack Wizards and Monitoring (“Eye Candy”)

There’s no denying that the CloudStack GUI is beautiful and easy to use. I was seduced by the pictures. But when I dug  deeper, I found some basic functionality that didn’t work though it appeared to on the surface. For example, after attaching a volume to an instance to use as persistent storage, the UI showed no errors. However, when I logged in to the VM instance, the volume was not there (I checked dmesg and /proc/partitions ). It was only after scouring the internet that I found a suggestion to reboot the VM. No problem, since I remembered seeing the VM reboot functionality in the GUI, but I was again let down. I tried rebooting the instance several times and each time the UI promised that it had succeeded, but the VM never rebooted and showed an uptime of several hours. Fortunately, I was able to ssh in to the VM to reboot manually. This is potentially a big issue for people who don’t have ssh access to the instance and can only reboot via the GUI.

That said, the GUI for provisioning instances is really user-friendly and will impress new users. See video below:




CloudStack also does a good job of using the GUI to monitor resources. See the pictures below; managers will eat this stuff up.



Comparison Note: The setup wizards and resource metrics are CloudStack’s strongest selling point and where it has a big lead over OpenStack’s Horizon dashboard.  While OpenStack Horizon can be more flexible (plug-ins and add-ons to nova-api) and has more features coming on the roadmap, common tasks such as uploading images, managing roles, and creating networks are not available today and require the use of CLI.


CloudStack System Virtual Machines

CloudStack uses two system virtual machines for each zone. Console Proxy VM provides console access to VM instances and SSVM(Secondary Storage VM) manages and downloads images. During my testing, I came across a third system VM, a virtual router that provides networking to the tenants. While this System VM approach may be useful for isolating major services, I think it could be inefficient for resource utilization. These system VMs need their own IPs, CPU, RAM, and disk space that can’t be shared. For example, if SSVM is not doing any work, the reserved CPU and RAM can’t be used for something else. Also, I wonder how well these system VMs will scale and support HA.

Comparison Note: OpenStack uses daemon for major components like nova-network and glance which can share resources on the same host; what’s more, they are HA ready because they are stateless.


CloudStack Templates and ISOs

In CloudStack, a template or ISO is required to provision an instance. CloudStack creates a Secondary Storage VM (SSVM) to manage and download ISOs and templates. If the SSVM can’t connect to the internet after installation, you won’t be able to add a template or ISO. In addition, the default template needs to be downloaded before a VM can be provisioned. To fix the internet access for SSVM, I had to ssh in to the SSVM and add a gateway to /etc/network/interfaces. There’s no information in the documentation on SSVM, how to ssh to it, what it’s used for, or how to troubleshoot it. I found the links below helpful in getting SSVM working.


Comparison Note: I’m not sure I understand CloudStack’s approach for loading templates and ISOs exclusively over an internet connection with a URL path.  Perhaps their vision was to have a giant library of templates and ISOs online. This seems rigid for anyone who wants to use their own templates and ISOs. There is a workaround that I’ll reference below, but it’s more of a hack (running your own webserver to serve your own templates and ISOs). It’s also one of the most popular topics in the CloudStack forums, which means it’s not an isolated issue for just a few people. In OpenStack, images can be loaded from any file system to Glance



The Snapshot functionality did not work for me in CloudStack. After some research, I found that the problem has to do with the older version of Ubuntu.


Some concluding observations

I was a bit let down by how much trouble I had getting CloudStack to a state where it could provision virtual machines. The prime cause of this frustration came from shortcomings in the documentation and lack of support for newer operating systems. Once you are able get over the humps during deployment and configuration, there are some areas like user interface, setup wizards and resource metrics where it shines. Another feature  which I really liked (but didn’t try because I only had two nodes) is Live Migration – that’s something OpenStack can’t do yet via its dashboard.

Based on what I learned from this experience, I sense that the biggest disadvantage with the CloudStack platform is where it lacks flexibility. I also think that the breadth of support in the open source community is shallower, considering the outdated OS and how much I had to dig around for answers to questions – much more digging than I experienced in deploying OpenStack.

Disagree? Had similar experience? Please post your comments below!

How to Use Service Mesh with VMs and Containers