Make No Small Plans – OpenStack Silicon Valley
Cross posted from the OpenStack Silicon Valley Blog
Guest Post: By Florian Leibert
I’m often asked if Mesos, with its use of containers, is better than OpenStack and virtual machines (VMs) for cloud computing.
I think it’s the wrong question. You can actually run Mesos on any cloud that is provisioned using OpenStack, and that’s exactly how many of our customers are deploying it today. Of course, Mesos will also run directly on bare metal, and so that’s another choice as well.
What choice you make in how you deploy Mesos depends on what you want to accomplish and the role that VMs play in your architecture. If your datacenter or cloud uses OpenStack as its provisioning manager, then the easiest way to deploy Mesos may very well be on top of a cloud of VMs provisioned using those tools.
But if you’re building a new datacenter, refactoring an old datacenter, starting a greenfield project, or looking to build apps in the same way that Google, Facebook and Twitter do, then you might want to deploy Mesos and use containers directly on bare metal. A bare metal deployment will eliminate “VM sprawl” and you’ll notice an improvement in performance over VMs. It’s really your choice. That’s the power of Mesos: to users of Mesos, the underlying architecture is entirely transparent. Mesos will combine all of the resources in the datacenter or cloud into a single pool of resources, no matter who those resources are provisioned—whether virtual or physical.
It seems clear to me, however, that the tide of history is flowing in the direction of containers (in a Mesos framework) running on bare metal, leaving VMs increasingly stranded on the beach when organizations are looking to run distributed applications at massive scale.
This is Silicon Valley. Who doesn’t aspire to massive scale? Make no small plans is the rule around here. When I was at Twitter we moved to Mesos and containers as our plans got big much faster than we had anticipated or built for. When I moved to AirBnb we just started the greenfield data infrastructure projects with Mesos. To the extent that OpenStack is built around managing workloads in virtual machines, it faces increasing challenges that will hinder adoption for any organization making big plans about its future. For other workloads, it may be the right choice – but not massive scale workloads.
Google arguably runs the most massive computing infrastructure in the world. They also pioneered container technology in Linux. Why does Google’s architecture avoid VMs?
Technically VMs were designed to solve a different problem than scale. They became popular as a way for companies to consolidate more workloads on fewer servers and slash capital spending budgets. Servers, following Moore’s Law, have bulked up massively over the past decade. Virtual machines allowed you to run lots of applications on bigger and bigger servers. Everyone wins except the person in charge of managing all those applications on individual VMs. Scaling up can be a nightmare. Talk about complexity.
The cool thing about Mesos is that it reverses the VM paradigm. Instead of splitting up the applications to run on multiple machines, Mesos pools all your systems and presents them to the application as a single resource – one machine. From a design perspective, it makes running apps on your cloud or datacenter conceptually the same as running them on a single (very big) desktop. This approach brings many of the hardware utilization benefits of VMs (although we’re even more efficient) but without all the complexity.
OpenStack will play a big role in organizations who have needs for virtual machines. Embracing this as a starting point, supporting deployments on OpenStack is important to our customers. But our customers are also expecting us to guide them toward the future, where all applications are distributed applications.
In the future, I see more and more traditional enterprises building applications like Silicon Valley companies do. Silicon Valley companies, increasingly, are building applications from the start as distributed systems designed for scale. They use distributed analytics systems, like Hadoop and Spark, run distributed databases like Cassandra, and use scheduling and orchestration systems like Mesos and Marathon. Traditional enterprises can experience these systems on OpenStack, but I predict that over time we will see a massive shift away from VMs because VMs were designed for something different.
Florian (Flo) Leibert is the CEO and Co-Founder of Mesosphere
