 |
Openstack Networking for Scalability and Multi-tenancy with VlanManager
August 7, 2012
In a previous post I explained the basic mode of network operation for OpenStack, namely FlatManager and its extension, FlatDHCPManager. In this post, I’ll talk about VlanManager. While flat managers are designed for simple, small scale deployments, VlanManager is a good choice for large scale internal clouds and public clouds. As its name implies, VlanManager relies on the use of vlans (“virtual LANs”). The purpose of vlans is to partition a physical network into distinct broadcast domains (so that host groups belonging to different vlans can’t see each other). VlanManager tries to address two main flaws of flat managers, those being:
In this post I will focus on VlanManager using multi-host network mode in OpenStack. Outside the sandbox, this is considered safer than using single-host mode, as multi-host does not suffer from the SPOF generated by running a single instance of a nova-network daemon per an entire openstack cluster. However, using VlanManager in single-host mode is in fact possible. (More about multi-host vs single-host mode can be found here). Difference between “flat” managers and VlanManagerWith flat managers, the typical administrator’s workflow for networking is as follows:
So typically, this is how IPs are allocated their instances in this mode: tenant_1:
tenant_2:
We can see tenant_1 and tenant_2 instances have landed on the same IP network, 10.0.0.0. With VlanManager, the admin workflow changes:
So, compared to FlatDHCPManager, we additionally define two things for the network:
From now on, once a tenant spawns a new vm, it will automatically get the address from his dedicated pool. It will also be put on a dedicated vlan which OpenStack will automatically create and maintain. So if we created two different networks for two tenants, the situation will look like this: tenant_1:
tenant2:
It can be clearly seen that tenants’ instances have landed on different IP pools. But how are vlans supported? How VlanManager configures networkingVlanManager does three things here:
Let’s suppose that the tenant named “t1” spawns its instance t1_vm_1. It lands on one of the compute nodes. This is how the network layout looks:
We can see that a dedicated bridge named “br102” has been created along with a vlan interface “vlan102”. Also a dnsmasq process has been spawned and is listening on address 10.0.2.1. Once instance t1_vm_1 boots up, it receives its address from the dnsmasq based on a static lease (please see this previous post on the details of how dnsmasq is managed by OpenStack). Now, let’s assume now that tenant “t1” spawns another instance named t1_vm_2, and it happens to land on the same compute node as the instance previously created:
Both instances end up being attached to the same bridge, since they belong to the same tenant, and thus they are placed on the same dedicated tenant’s network. They also get their DHCP configuration from the same dnsmasq server. Now let’s say that tenant “t2” spawns his first instance. It also lands on the same compute node as tenant “t1”. Also, for his network, a dedicated bridge, vlan interface and dnsmasq are configured:
So it turns out that depending on the number of tenants, this is a normal situation where you have quite a large number of network bridges and dnsmasq processes, all running within a single compute node. There’s nothing wrong with this, however - OpenStack will manage all of them automatically. Unlike the case of using flat managers, here both tenants’ instances reside on different bridges which are not connected to each other. This ensures traffic separation on L2 level. In case of tenant “t1”, the ARP broadcasts sent over br102 and then through to vlan102 are not visible on br103 and vlan103, and vice versa.
Support for tenant networks across multiple compute nodesSo far, we’ve talked about how this plays out on a single compute node. Most likely, you’ll probably use a lot more than one compute node. Usually we want to have as many of them as possible. Then, likely, tenant “t1″ instances will be scattered among many compute nodes. This means that his dedicated network must also be spanned across many compute nodes. Still it will need to meet two requirements:
Typically, compute nodes are connected to a network switch by a single cable. We want multiple tenants to share this link in a way that they don’t see one another’s traffic. There is a technology that addresses this requirement called Vlan tagging. Technically, it extends each Ethernet frame by adding a 12-bit field called VID (Vlan ID), which bears the vlan number. Frames bearing an identical Vlan tag belong to a single L2 broadcast domain; thus devices whose traffic is tagged with the same Vlan ID can communicate. It should be obvious, then, that one can isolate tenants’ networks by tagging them with different Vlan IDs. Traffic for tenant “t1” leaves the compute node via “vlan102″. Vlan102 is a virtual interface connected to eth0. Its sole purpose is to tag frames with a vlan number “102″, using the 802.1q protocol. Traffic for tenant “t2” leaves the compute node via “vlan103″, which is tagged with vlan tag 103. By bearing different vlan tags, “t1′s” traffic will in no way interfere with “t2’s” traffic. They are unaware of each other, even though they both use the same physical interface eth0 and, afterwards, the switch ports and backplane. Next, we need to tell the switch to pass tagged traffic over its ports. This is done by putting a given switch port into “trunk” mode (as opposed to “access” mode, which is the default). In simple words, trunk allows a switch to pass VLAN-tagged frames; more information on vlan trunks can be found in this article. At this time, configuring the switch is the duty of the system administrator. Openstack will not do this automatically. Not all switches support vlan trunking. It’s something you need to look out for prior to procuring the switch you’ll use. Also – if you happen to use devstack + virtualbox to experiment with VlanManager in a virtual environment, make sure you choose “PCNET – Fast III” as the adapter to connect your vlan network. Having done this, we come to this model of communication:
The thick black line from compute nodes to the switch is a physical link (cable). On top of the same cable, vlan traffic tagged by both 102 and 103 is carried (red and green dashed lines). There is no interference in traffic (the two lines never cross). So how does the traffic look when tenant “t1” wants to send a ping from 10.0.2.2 to 10.0.2.5?
Configuring VlanManagerTo configure VlanManager networking in OpenStack, put the following lines into your nova.conf file:
ConclusionVlanManager is by all means the most sophisticated networking model offered by OpenStack now. L2 scalability and inter-tenant isolation have been addressed. Nevertheless, it still has its limitations. For example, for each tenant network it relates ip pools (L3 layer) to vlans (L2 layer) (remember? – each tenant’s network is identified by a pair of ip pool + vlan). So it is not possible to have two different tenants use the same ip addressing schemes independently in different L2 domains. Also – vlan tag field is only 12 bits long, which tops out at only 4096 vlans. That means you can have no more than 4096 potential tenants, not that many at cloud scale. These limitations are yet to be addressed by emerging technologies such as Quantum, the new network manager for OpenStack and software-defined networking.
In the next post in this continuing series, I will give an explanation of how floating ips work. 19 comments19 Responses
Continuing the Discussion
|
Piotr Siwczak
Hendrik
Ben Grissinger
Realde
padnala balaji
ritesh
Shlomi
klepamar
Ghassen
Hi,
Nice article.
Btw nova-manage create network will fail without the –label parameter.
Do you know if the current driver support VXLAN?
Cheers!
August 7, 2012 14:55