An OpenStack guy takes CloudStack for a test drive

Mirantis writing a blog about CloudStack? WTFBBQ? No, it’s not April Fool’s day. As much as we love OpenStack here at Mirantis, I wanted to see for myself what the CloudStack platform offers, things it does better and worse, what works and what doesn’t, how it stacks (pun intended) up against OpenStack, and give CloudStack credit where it’s due.

The idea came to me from a Mirantis Bootcamp OpenStack training session we delivered where a number of attendees asked about CloudStack and how it compares to OpenStack. I spent some time researching and didn’t find many technical reviews, let alone comparisons. I decided the best way to learn more and better prepare for this question was to deploy CloudStack in a test environment and try it myself. The following are my impressions of CloudStack along with comparison notes to OpenStack when applicable.

CloudStack Installation and Configuration

An exercise in prerequisites

If someone told me a week ago that I couldn’t install CloudStack on top of the latest Ubuntu 12.04 LTS release, I would have been shocked and assumed they got their facts mixed up. However, after attempting a basic two-node CloudStack installation, I’m convinced that it’s easier to use the two-year-old version of Ubuntu (listed in their requirements) and install the missing hardware drivers manually than it is to use Ubuntu 12.04 LTS and deal with script hacking and modification to get CloudStack running.

Attempt #1: First, I proceeded to ignore the install guide and used Ubuntu 12.04 LTS instead of Ubuntu 10.04 LTS because I did not want to install my hardware drivers manually. I expected to have minor problems with Ubuntu 12.04 with easy workarounds. Instead, I was met with error after error when running the install scripts. Each time I found a workaround through online forums for one script, I’d run into a new error on the next script. I quickly concluded it was a dead end and gave up.


Attempt #2: Eventually, I gave in and downgraded to Ubuntu 10.04 (almost three years old) and manually installed the drivers for my hardware. I was able to get further in the installation process but ran in to some hiccups when attempting add a host and provision my instances, more details to follow.

I did not try Red Hat or Centos with CloudStack but found similar complaints here: CloudStack OS Versions.

Comparison Note: CloudStack’s insistence of having to use an outdated Operating System that required manual driver installation is cumbersome. OpenStack wins this category since it works with latest operating systems. In addition, it has Linux community support from Ubuntu, Redhat, SUSE etc.


CloudStack setup and configuration wizard

Now, a shining moment for CloudStack. The wizard for setting up the infrastructure is really slick and easy to use. It reminds of the wizards they use for routers. You can click through the screens I captured below to get an idea. They also included mouse over hints on the parameters in case you have trouble filling out the data. These guys really nailed this part of the experience.

Comparison Note: No wizards are available in Openstack Horizon. In addition, things like networking need to be set up with CLI or nova.conf

“Unable to add CloudStack host”

As you may have noted from the slide deck above, it ended with “Add Host”. This is because I was unable to add a host and continue the wizard. There were two reason why the system kept failing with “unable to add host”.

  1. SSH is not installed on Ubuntu 10.04 LTS but it’s required in order for a CloudStack host to connect to Cloud Management Server. While attempting to add a host, the wizard complained with “unable to add host”  which couldn’t be less helpful. I found the root cause after looking at /var/log/cloud/management/management-log and saw an error about not being able to connect to the agent on port 22.
  2. After getting past the first error by installing SSH, I was still “unable to add host” due to missing network configurations. The cause:  Ubuntu has a GUI for configuring networking which is the default way a user would set up the networking but isn’t usable CloudStack. CloudStack requires networking to be configured using /etc/network/interfaces, otherwise it will think it has no network. See here for help Ubuntu Network Configuration

Neither of these requirements are a big deal to resolve; the problem was that they weren’t documented.

Comparison Note: I had fewer problems getting from bare metal to provisioning instances on the OpenStack platform. While OpenStack lacks fancy wizards and setup GUI for configuring infrastructure, the installation process was smooth and basic functions like creating VM instances, retrieving images, and attaching volumes just worked. None of the CloudStack problems I encountered were mentioned in the documentation, which could lead to frustration and installers giving up.


CloudStack Features

Zones, Pods, Clusters and Hosts

CloudStack uses Organizational Units (OU) to group hosts within a CloudStack deployment. Zones, the largest OU, usually corresponds to a datacenter and consists of at least one pod and a ‘Secondary’ storage shared by all pods in a zone. Pods represent a single rack and consist of at least one cluster. Clusters in the same pod use the same subnet.  All hosts in a cluster are homogeneous in that they use the same hardware, hypervisor, subnet, and access the same shared storage (‘Primary’ Storage). Hosts are single nodes running hypervisor software that provide computing resources to run guest VMs.

Comparison Note: While this is a structured and clear approach to grouping/nesting objects, it doesn’t seem customizable and lacks flexibility. What if a deployment calls for five layers for example? Or what if I only have two layers starting with pods, why do I need to fill in information about Zones?


CloudStack Storage

CloudStack offers two types of storage for use by the platform, which they call ‘Primary’ and ‘Secondary’. Primary storage is configured at the cluster level and stores all disk volumes for VMs in a cluster. These volumes are persistent, and will remain after you terminate the VM. Local disk, iSCSI, and NFS can be used for Primary storage.  Secondary storage is configured at the Zone level, and is used to store templates, ISOs, and snapshots. All zones need at least one Secondary storage. The documentation says that either NFS and OpenStack Swift can be used as Secondary storage.

Comparison Note: I would say Primary storage is comparable to OpenStack nova-volume in that it supports multiple disk formats and would be just as scalable. Regarding Secondary storage, it’s not quite as flexible as Glance (OpenStack’s image service). Glance is capable of using any back end, including local disk on the controller node to store images, whereas Cloudstack Secondary Storage uses NFS or OpenStack Swift. Also, CloudStack does not have its own object storage (as in a comparable product to OpenStack Swift or Amazon S3).


CloudStack Wizards and Monitoring (“Eye Candy”)

There’s no denying that the CloudStack GUI is beautiful and easy to use. I was seduced by the pictures. But when I dug  deeper, I found some basic functionality that didn’t work though it appeared to on the surface. For example, after attaching a volume to an instance to use as persistent storage, the UI showed no errors. However, when I logged in to the VM instance, the volume was not there (I checked dmesg and /proc/partitions ). It was only after scouring the internet that I found a suggestion to reboot the VM. No problem, since I remembered seeing the VM reboot functionality in the GUI, but I was again let down. I tried rebooting the instance several times and each time the UI promised that it had succeeded, but the VM never rebooted and showed an uptime of several hours. Fortunately, I was able to ssh in to the VM to reboot manually. This is potentially a big issue for people who don’t have ssh access to the instance and can only reboot via the GUI.

That said, the GUI for provisioning instances is really user-friendly and will impress new users. See video below:




CloudStack also does a good job of using the GUI to monitor resources. See the pictures below; managers will eat this stuff up.



Comparison Note: The setup wizards and resource metrics are CloudStack’s strongest selling point and where it has a big lead over OpenStack’s Horizon dashboard.  While OpenStack Horizon can be more flexible (plug-ins and add-ons to nova-api) and has more features coming on the roadmap, common tasks such as uploading images, managing roles, and creating networks are not available today and require the use of CLI.


CloudStack System Virtual Machines

CloudStack uses two system virtual machines for each zone. Console Proxy VM provides console access to VM instances and SSVM(Secondary Storage VM) manages and downloads images. During my testing, I came across a third system VM, a virtual router that provides networking to the tenants. While this System VM approach may be useful for isolating major services, I think it could be inefficient for resource utilization. These system VMs need their own IPs, CPU, RAM, and disk space that can’t be shared. For example, if SSVM is not doing any work, the reserved CPU and RAM can’t be used for something else. Also, I wonder how well these system VMs will scale and support HA.

Comparison Note: OpenStack uses daemon for major components like nova-network and glance which can share resources on the same host; what’s more, they are HA ready because they are stateless.


CloudStack Templates and ISOs

In CloudStack, a template or ISO is required to provision an instance. CloudStack creates a Secondary Storage VM (SSVM) to manage and download ISOs and templates. If the SSVM can’t connect to the internet after installation, you won’t be able to add a template or ISO. In addition, the default template needs to be downloaded before a VM can be provisioned. To fix the internet access for SSVM, I had to ssh in to the SSVM and add a gateway to /etc/network/interfaces. There’s no information in the documentation on SSVM, how to ssh to it, what it’s used for, or how to troubleshoot it. I found the links below helpful in getting SSVM working.


Comparison Note: I’m not sure I understand CloudStack’s approach for loading templates and ISOs exclusively over an internet connection with a URL path.  Perhaps their vision was to have a giant library of templates and ISOs online. This seems rigid for anyone who wants to use their own templates and ISOs. There is a workaround that I’ll reference below, but it’s more of a hack (running your own webserver to serve your own templates and ISOs). It’s also one of the most popular topics in the CloudStack forums, which means it’s not an isolated issue for just a few people. In OpenStack, images can be loaded from any file system to Glance



The Snapshot functionality did not work for me in CloudStack. After some research, I found that the problem has to do with the older version of Ubuntu.


Some concluding observations

I was a bit let down by how much trouble I had getting CloudStack to a state where it could provision virtual machines. The prime cause of this frustration came from shortcomings in the documentation and lack of support for newer operating systems. Once you are able get over the humps during deployment and configuration, there are some areas like user interface, setup wizards and resource metrics where it shines. Another feature  which I really liked (but didn’t try because I only had two nodes) is Live Migration – that’s something OpenStack can’t do yet via its dashboard.

Based on what I learned from this experience, I sense that the biggest disadvantage with the CloudStack platform is where it lacks flexibility. I also think that the breadth of support in the open source community is shallower, considering the outdated OS and how much I had to dig around for answers to questions – much more digging than I experienced in deploying OpenStack.

Disagree? Had similar experience? Please post your comments below!

18 responses to “An OpenStack guy takes CloudStack for a test drive

  1. I have never touched cloudstack, but I still know /etc/network/interfaces and sshd are the default in Ubuntu server, and not on Ubuntu desktop. Did you try installing this on Desktop?

    1. Hi,

      Thanks for the reply, I did use a laptop with Ubuntu 10.04 LTS desktop version, but this is supported per CloudStack requirements documentation. Also,ssh by default is included in the Ubuntu 12.04 LTS desktop but not Ubuntu 10.04 LTS.
      Regarding /etc/network/interfaces, it is included as part of the OS, however the default networking set up is via the GUI. Both things were very easy to correct/enable, they just need to be mentioned in the install guide.

  2. Hi, I’m a cloudstack developer. I agree to most major points you’ve mentioned. But the great thing about opensource is that we can all learn from each other. Thanks to reminding us things that a new user would want to have. While the enterprise mostly would use CentOS/RHEL, we’ll be working on supporting more Linux distros in future.

    1. Cloud Hacker,
      Thanks for the reply, it means a lot to me that CloudStackers are reading this post and commenting on its validity. I was not able to try out CloudStack installation on top of CentOS/RHEL due to time constraints but plan to in the future and will either update this blog post or write a new one. Also, I completely agree that opensource projects help us learn from one another and it’s one of the reasons I made this post.

  3. I spent about 9 months working with the CloudStack (2.x version) and the experience left me with the opinion that if faced again with the decision of which oss cloud ecosystem to invest time and effort in, I would pick the OpenStack. Among the issues I found particularly annoying were the homemade ORM (really?), the unelegant rest api, the lack of a proper message queue, the use of waf instead of maven (it’s Java) and surprisingly poor test coverage.

    1. Thank you for sharing your working experience with CloudStack. My blog was a test drive/first impression so it did not cover quite as much. The working experience is good feedback and perhaps someday we can collaborate on a post that compares working experience with CloudStack and OpenStack

  4. This post was spot on. For the last 18 months or so I’ve been looking at OpenStack, but recently due to events outside our control regarding a 3rd party, we were forced to use CloudStack. This was OK as we keep being reminded that CloudStack is a robust installation supporting thousands of instances across hundreds of hosts. We aren’t at that scale so this fits. On installation of CloudStack its predominantly GUI based, this is one of its strong points – but I feel so let down by it. A user journey to find information leads you down many different routes – ones you don’t expect. Troubleshooting CloudStack is such an effort considering that the GUI is touted so much. Under the hood is Java – that in itself isn’t great to troubleshoot. Sure, installation was nice using a GUI – but in this day and age, we’ve moved on from such methods anyway. For a small installation it feels a nice touch. But it doesn’t work all the time. I fell foul to many a time of watching a pulsating dot and not knowing what was going on. Adding a host – a fundamental feature – resulted in the author’s issue. Booting instances where they failed was painful to troubleshoot as its so GUI driven. If the emphasis is on the GUI then give the user the ability to use the GUI as an operator.
    CloudStack, in my opinion, fills a void between VMware and what many are understanding as open cloud infrastructure. It plays heavily on expectations and features that VMware does very well in its ability to manage VMs. Sure, it has an API that allows it to do more advanced things – but if you’re evaluating CloudStack – you’ll quickly encounter the issues mentioned here and not have chance to see what CloudStack has to offer.

    Overall I was just let down. The amount of arguments put forward against OpenStack compared to CloudStack – they make out as if OpenStack has some way to go to catch up to CloudStack. If CloudStack is the yardstick, OpenStack is doing the right thing and not trying to play second fiddle to VMware and AWS.

    I absolutely and fundamentally agree with “Cloud Hacker” and that this is Open Source, so if there’s a problem – there are ways of fixing it. I really do want to emphasise that I was genuinely let down by the experiences shared by using CloudStack. We had a Citrix partner install our environment and after 3 days we only managed a very basic install on 2 hosts that isn’t production ready: that’s not a great introduction to it in itself.

    1. Hi,

      Thanks for the thoughtful response and sharing your experience. Although, I recognize the problems you had and share your problems with setting up CloudStack , I do think that the GUI, Wizards, and resource metrics will impress decision makers when compared with CLI or Horizon Dashboard. While OpenStacks may work better under the hood as you described, sometimes it’s the shiny paint that sells the car.

  5. Hi,
    Thanks for the very detailed review. We are in the process of deciding CloudStack / OpenStack going forward. I think very few good real life comparisons exist out there at the moment. Most reviews are high level and based on data sheet comparisons. The impression I had was that CloudStack deployments were easier to setup and get running with, but the future was in OpenStack.
    For me this review has evened out the playing field in OpenStack’s favour.

    I would be interested to see how version 4 of CloudStack compares since its release and if it fixes any of the major issues. It is meant to fix the Ubuntu 12.04 issue at least.

  6. Great article. I’ve been test driving CloudStack 4.0 on Ubuntu 12.04 and didn’t have the problems you had, but that’s not to say I haven’t had problems. I will say the documentation is a little off but who likes to update documentation anyway 😉 . I have some experience with OpenStack and have used Dells Crowbar to install it. The main issue I have with OpenStack is all the pieces to it and what’s with all the code names? I left the Army several years ago and my codebook was left behind.

    I’m excited about both of these open-source projects and I think we all should be. They both have strong points but the greatest of them all is the fact when we do run into an issue we can dig into the code and find out what the heck is going on. I’ll agree that the OpenStack community is much more evolved than the CloudStack community and they do take the accuracy of documentation much more seriously.

    Your openness to try CloudStack and share your honest assessment is greatly appreciated. I can’t say I agree with all your points but I do think you were fair.

    Now get back to work and build a cloud!


  7. Now that Cloudstack 4 is out, would you be doing an update to this article?

    It would be great to see your view of the current state of affairs.


  8. Hi Lee,
    well a lot of this issues also Openstack has. F.e. we support a lot of OS but we do not support upgrades! Also nearly all Install Howto just do a SingleCompute install. ALL multi node install howto´s have significant and terrible errors cause Ubuntu changed so much packages.

    Did you ever install Essex on 12.04? Nearly all function of Openstack are depreciated all your error logs run full in a view days. Especially 12.04 with Essex should be supported very well cause it is the only LTS which will also get a possibility of upgrade. But the LTS + 12.04 is treated like shit by Canonical and the Openstack developers.

    Just want to remember tho NOVNC problem, the keystone shit with the Grand Privileges or the problems with the FQDN … i can go on for hours!

    I never tried Cloudstack but from what i can read it has less more problems than Openstack.

    You do not believe? Ask Rob from Dell or the guys from Stackops or at Rackspace who try to deliver a Distro they spend so much time on debugging Openstack and i revied hours of Code to compare how they work around this bugs.

    You ever thought about why this distos are all freezed? Cause they are all afraid that someone will enter apt-get upgrade and the whole Openstack cloud will lay in pieces.

    If you do not believe i give you a view examples. Openstack will not install without a Nameserver or adding the controller and notes in /etc/hosts … do you find this in any Howto? Just Brand point this out.

    Another example Grand Privilege without adding localhost and IP Adress to Grand Privileges Keystone will not work properly … Half of the Howtos referred at the Openstack Page miss this point and so the installation will fail.

    Crowbar is very far but they do not support mdadm at install so you have to have servers which have Hardware Raid to use Crowbar in production enviroment.

    So really i would say Openstack is better when is see that i can upgrade from Essex to Flossom with just hitting the update button in Horizon and not running thru pages of code to to a simple upgrade.

    I am a big openstack fan and poweruser and i am sorry to disturb your dreams but face reality!

    Try writing a howto for installing Essex at 12.04 with working Live Migration and HA and you will change your opinion very quick. You will run in much more problems that you write about here for Cloudstack.


Leave a Reply

Your email address will not be published. Required fields are marked *




Mirantis Cloud Platform
Automate Upgrades with Mirantis DriveTrain
Kubernetes & Docker Mini-Bootcamp