Mirantis | The #1 Pure Play OpenStack Company

OpenStack Networking – FlatManager and FlatDHCPManager

Over time, networking in OpenStack has been evolving from a simple, barely usable model, to one that aims to support full customer isolation. To address different user needs, OpenStack comes with a handful of “network managers”. A network manager defines the network topology for a given OpenStack deployment. As  of the current stable “Essex” release of OpenStack, one can choose from three different types of network managers: FlatManager, FlatDHCPManager, VlanManager. I’ll discuss the first two of them here. 

FlatManager and FlatDHCPManager have lots in common. They both rely on the concept of bridged networking, with a single bridge device. Let’s consider her the example of a multi-host network; we’ll look at a single-host use case in a subsequent post.

For each compute node, there is a single virtual bridge created, the name of which is specified in the Nova configuration file using this option:

flat_network_bridge=br100

All the VMs spawned by OpenStack get attached to this dedicated bridge.

Network bridging on OpenStack compute node

This approach (single bridge per compute node) suffers from a common known limitation of bridged networking: a linux bridge can be attached only to a signle physical interface on the host machine (we could get away with VLAN interfaces here, but this is not supported by FlatDHCPManager and FlatManager). Because of this, there is no L2 isolation between hosts. They all share the same ARP broadcast domain. 

The idea behind FlatManager and FlatDHCPManager is to have one “flat” IP address pool defined throughout the cluster. This address space is shared among all  user instances, regardless of which tenant they belong to. Each tenant is free to grab whatever address is available in the pool.

FlatManager

FlatManager provides the most primitive set of operations. Its role boils down just to attaching the instance to the bridge on the compute node. By default, it does no IP configuration of the instance. This task is left for the systems administrator and can be done using some external DHCP server or other means.

FlatManager network topology

FlatDHCPManager

FlatDHCPManager plugs  a given instance into the bridge, and on top of that provides a DHCP server to boot up from.

On each compute node:

  • the network bridge is given an address from the “flat” IP pool
  • a dnsmasq DHCP server process is spawned and listens on the bridge interface IP
  • the bridge acts as the default gateway for all the instances running on the given compute node

 

FlatDHCPManager – network topology

As for dnsmasq, FlatDHCPManager creates a static lease file per compute node to guarantee the same IP address for the instance over time. The lease file is constructed based on instance data from the Nova database, namely MAC, IP and hostname. The dnsmasq server is supposed to hand out addresses only to instances running locally on the compute node.  To achieve this, instance data to be put into DHCP lease file  are filtered by the ‘host’ field from the ‘instances’ table.  Also, the default gateway option in dnsmasq is set to the bridge’s IP address. On the diagram below you san see that it will be given a different default gateway depending on which compute node the instance lands.

Network gateways for instances running on different compute nodes

 

Below I’ve shown the routing table from vm_1 and for vm_3 – each of them has a different default gateway:

root@vm_1:~# route -n
Kernel IP routing table
Destination    Gateway     Genmask Flags Metric Ref Use Iface
0.0.0.0        10.0.0.1     0.0.0.0 UG     0   0   0 eth0

root@vm_3:~# route -n
Kernel IP routing table
Destination    Gateway     Genmask Flags Metric Ref Use Iface
0.0.0.0        10.0.0.4     0.0.0.0 UG     0   0   0 eth0

 

By default, all the VMs in the “flat” network can see one another regardless of which tenant they belong to. One can enforce instance isolation by applying the following  flag in nova.conf:

allow_same_net_traffic=False

This configures  IPtables policies to prevent any traffic between instances (even inside the same tenant), unless it is unblocked in a security group.

From practical standpoint, “flat” managers seem to be usable for homogenous,  relatively small, internal  corporate clouds where there are no tenants at all, or their number is very limited.  Typically, the usage scenario will be a dynamically scaled web server farm or an HPC cluster. For this purpose it is usually sufficient to have a single IP address space where IP address management is offloaded to some central DHCP server or is managed in a simple way by OpenStack’s dnsmasq. On the other hand, flat networking can struggle with scalability, as all the instances share the same L2 broadcast domain.

These issues (scalability + multitenancy) are in some ways addressed by VlanManager, which will be covered in an upcoming blog posts.

32 comments
Google Plus Mirantis

32 Responses

  1. Alex

    It’ll be nice to see nova.conf files additionally.

    August 3, 2012 06:17
    • Piotr Siwczak

      Hi,
      The relevant nova.conf entries are:

      # network manager to be used
      network_manager=nova.network.manager.FlatDHCPManager

      # bridge to attach vm-s to
      flat_network_bridge=br100

      # the physical interface to which the bridge is attached
      flat_interface=eth0

      # in flat network modes, this setting allows for configuring network inside vm prior to its boot
      # Before boot nova mounts the vm image and “injects” network configuration to /etc/network/interfaces
      # inside the vm
      flat_injected=True

      # This setting is used for iptables rules (NAT + filtering) to be set up
      fixed_range=YOUR_IP_RANGE_FOR_FIXED_IP_NETWORK

      August 9, 2012 00:46
  2. Manish

    Nice post. Your diagrams helped in understanding. Thanks! However, I am having trouble setting up FlatDHCP mode (and Flat too :-( ). In FlatDHCP, VMs within a compute node are able to talk to each other but not outside. So, if we consider your diagram 10.0.0.5 cannot ping/ssh to 10.0.0.2. But 10.0.0.5 can ping/ssh to 10.0.0.6. I’ve added rule for ssh and ping. Could you possibly know what am I doing wrong? My nova.conf also matches what you gave above.

    August 9, 2012 08:31
    • Piotr Siwczak

      Manish,

      The reasons can be numerous. I would suggest taking the following approach to address the problem:
      1. Check if you have IP addresses assigned to all the bridges on compute nodes (the incoming packet somehow needs to find the way to the 10.0.0.0 network – this is done by routing – which in turn is based on br100 having an address from 10.0.0.0)
      2. Check with tcpdump if your pings arrive at the destination vm-s interface (vnetX) and on br100

      Also – you can copy your kernel routing table on compute nodes and paste it here.

      Please, let me know about the results – will try to help further.

      August 9, 2012 10:30
  3. Manish

    Thanks a lot for your response! :)

    1. 10.0.0.x IP is assigned to all compute nodes br100 intf.
    2. After checking tcpdump: ping (icmp packet) on destination compute node br100 does not arrive but ARP request for destination is sent and responded by destination compute node. I didn’t tcpdump inside the vm though.

    Controller node with compute and network routing:
    default via 172.16.96.1 dev eth0 metric 100
    10.0.0.0/24 dev br100 proto kernel scope link src 10.0.0.3
    172.16.96.0/21 dev eth0 proto kernel scope link src 172.16.99.84
    192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1
    192.168.206.0/24 dev br100 proto kernel scope link src 192.168.206.130

    Another compute node routing:
    default via 172.16.96.1 dev eth0 metric 100
    10.0.0.0/24 dev br100 proto kernel scope link src 10.0.0.5
    172.16.96.0/21 dev eth0 proto kernel scope link src 172.16.99.95
    192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1
    192.168.206.0/24 dev br100 proto kernel scope link src 192.168.206.131

    August 9, 2012 11:18
    • Piotr Siwczak

      Manish,

      Sorry for my late response. I hope you managed to resolve the problem. If you can see ARP broadcast and replies, but no IP traffic, then I would suspect either firewall or routing (but routing tables are correct here). Also – be sure that your switch works as expected. I am afraid for now I cannot tell much more without just logging in and looking at this particular setup.

      August 13, 2012 11:27
    • Roberto

      Hello Manish,
      did you eventually manage to solve this problem ?

      November 20, 2013 04:35
  4. Tamale

    Be sure to enable ip_forwarding via /proc/sys/net/ipv4/ip_forward / /etc/sysctl.conf

    August 14, 2012 07:00
  5. hjg

    Manish – were you able to resolve the problem. Looks like allow_same_net_traffic=False setting is critical in making vm instances hosted across two compute nodes communicate. I am also seeing the same behavior, but did not try changing the value of this setting to true.

    August 15, 2012 23:20
  6. Manish

    @Piotre: Thanks for your response. I checked firewall. No problems there.

    @Tamale: Thanks for the suggestion. My system already has ip forwarding enabled. cat /proc/sys/net/ipv4/ip_forward gives 1.

    @hjg: Thanks for your suggestion. I had set allow_same_net_traffic=True before trying it out.

    @all: I am trying to setup the controller and compute node in Virtualbox VMs instead of two physical machines. When I manually launched a VM inside a clean Ubuntu 12.04 (that is a Vbox VM) using virt-manager with qemu as hypervisor in bridged mode, the VM was not able to communicate to any other machine other than its host. So, I am guess there is some problem VM inside VM environment. Now, I’ll be trying OpenStack on physical machines.

    August 16, 2012 13:10
    • pramod

      Hello Manish,
      I am having the same issue on physical machines. Did you eventually manage to solve this problem ?

      April 11, 2014 00:42
  7. Antonio

    I have the same problem
    I use FlatDHCP mode with multi_host option: I have a controller node (which also is a compute node) and a compute node (only nova compute-network-api). I cannot ping a VM on the compute node from a VM on the controller: by using tcpdump on br100, I can see that ARP request are sent but no reply arrives.

    August 30, 2012 02:46
    • S.A.

      @Antonio
      I have the same problem too. I noticed that my second compute node is not assigned with any IP address in flat network (ie 10.0.0.x) and hence no routes in 10.0.0.x subnet. I didn’t have to do anything on my controller and its bridge was assigned 10.0.0.1. I am wondering why the bridge in the second compute node is not assigned with 10.0.0.x ip? Did you resolve your issue?

      Thanks

      October 1, 2012 14:27
      • Piotr Siwczak

        Hi,

        Have you this option set up in nova.conf?

        flat_network_bridge=br100

        -Piotr

        October 2, 2012 01:06
        • S.A.

          @Piotr Thanks for your response!
          Yes. I have the same nova.conf file on both the controller and compute node. On compute node I only changed the values for the following:
          novncproxy_base_url
          vncserver_listen
          vncserver_proxyclient_address
          I have br100 up both on controller and compute node. Please see the following:
          https://answers.launchpad.net/centos/+question/210110

          October 2, 2012 08:55
          • Piotr Siwczak

            Have you put br100 into promiscuous mode? (you should)

            October 2, 2012 13:00
  8. S.A.

    @Piotr
    I am not sure if I should do that! I have not done anything to setup br100 on the controller; it was setup by openstack scripts and works just fine. I believe this is going to be done as part of starting up the openstack scripts with the right conf options. In fact I had this working, before it went south!

    October 2, 2012 17:27
  9. Dustin

    Do you need two NIC cards (one for eth0, one for br100) on both controller node and compute node? Or br100 is a virtual bridge used by VM only?

    October 8, 2012 20:25
    • Piotr Siwczak

      Hi,
      Depending on the HA mode you use:
      You need the cloud controller to have two nics if you run with multi_host=False option. Then the cloud controller typically acts the gateway for all the compute nodes.
      If you run multi_host=True, then each compute node needs to have two network cards, as each compute node is the default gateways for locally running vm-s. In contrary – no need for 2 NICs on the cloud controller then.

      And yes – br100 is an interface to which all the instances connect. It is connected to the interface specified as FlatInterface in nova.conf.

      October 9, 2012 07:38
  10. Stratos

    Can anyone post here a setup for the following scenario: http://docs.openstack.org/essex/openstack-compute/admin/content/figures/FlatNetworkSingleInterface.png

    regarding /etc/network/interfaces?

    May 7, 2013 08:07
  11. fengli

    This doc is a very clear and helpful. One question: Flat network is a L2 network connect all instances(throughout all nodes), as we know, the gateway of a L2 network should be the router’s interface, not each switch’s ip address (here is br100). So, 1)why don’t set instances’ default gateway in every node to a common router interface in the network? 2)If no default gateway in a instance, it can also communicate with each other because of in same L2 network, right? 3)In FlatManager mode, the external DCHP will set an unique default gateway address to all instances because he see every instance are equal, right? Thanks.

    September 17, 2013 19:18
  12. Genc

    Hi,
    I am using openstack only in one vm and therefore the controller and the compute nodes are the same vm. In addition, I have only one NIC, eth0. I tried using FlatDHCPManager but since eth0 is bridged with the veth pairs through br100, a dhcp request on br100 goes to a dhcp server from which eth0 gets the ip, not on the local dnsmasq. So it is either a race condition or the ip is overwritten (i am not sure). Is there any configuration which disables a dhcp request being replied by an external dhcp server?
    Thanks

    September 30, 2013 12:47
  13. Roberto

    Hello,
    I have a 3 nodes setup:
    - controller
    - compute1
    - compute2

    Relevant lines in the nova.conf in compute nodes are:

    multi_host=True
    allow_same_net_traffic=True
    flat_network_bridge=br100
    flat_interface=eth2
    flat_injected=true
    fixed_range=10.0.0.0/24
    public_interface=eth0


    eth2 is an interface with no IP address in both compute1 and compute2
    If I do not specify br100 in /etc/network/interfaces the br100 interface is configured with 10.0.0.1
    in both compute1 and compute2
    (differently from what it appears in this tutorial).
    I configured br100 in /etc/network/interfaces with different IPs:
    10.0.0.131 in compute1
    10.0.0.132 in compute2
    with bridge-ports eth2 for both.
    When I spawn VMs, again br100 gets the same 10.0.0.1 address in both compute1 and compute2.
    However, ip a shows that br100
    keeps the statically configured IP as secondary in both nodes (i.e. 10.0.0.131 in compute1 and 10.0.0.132 in compute2).
    The result of this latter configuration is that spawned VMs can ping the outside world and can ping other VMs hosted by the same compute node.
    However pings do not work across different compute nodes.
    The problem is somewhat similar to what Manish raised last year.
    Tcpdump shows that ARP packets cross the boundaries but ICPM packets do not.
    I would appreciate any hint to fix this problem.
    Thanks

    November 20, 2013 04:51
    • Roberto

      Let me add that putting eth2 and br100 in promisc mode in both compute1 and compute2 did not solve the issue

      November 20, 2013 05:02
      • Roberto

        I finally managed to solve this problem.
        Actually I omitted a relevant piece of information: the three nodes are VMs running in an ESXi server.
        The problem was fixed by enabling promiscuous mode in the vSwitch connecting the eth2 virtual NICs of the two compute nodes.

        November 23, 2013 10:01
  14. Roberto

    I finally managed to fix this problem.
    Actually, I omitted a relevant piece of information: the 3 nodes are actually VMware VMs running in an ESXi server.
    The problem was fixed by enabling Promiscuous mode as “accept” in the ESXi vSwitch that connected the nodes’ eth2 interfaces.
    Roberto

    November 22, 2013 09:55
  15. Prashanth

    Hi,

    I have a two node setup:
    controller
    compute1

    compute1 has two NICs and I want to create flat network for two NICs. openstack documentation provides a neat diagram but there is no enough configuration information (http://docs.openstack.org/admin-guide-cloud/content/ch_introduction-to-openstack-compute.html#section_use-multi-nics).
    I could configure one flat network, using FlatDHCPManager, but facing issues while configuring two flat networks (for two NICs on compute1).
    Can anyone one here help in providing the configuration for configuring the same.

    Thanks
    Prashanth

    December 19, 2013 18:49
  16. jothep

    Hi
    I have a two node setup,controller and compute1.
    flat_interface=eth1
    public_interface=eth1

    I had set controller eth1 for 192.168.0.10,and set compute1 eth0 for 192.168.0.11. how can I set nova.conf on both server?
    I saw installguide said set it for flat_interface=eth1
    public_interface=eth1
    I don’t understand

    Anyone can help me ?
    Thanks

    December 22, 2013 02:06
  17. Paul

    Hi,

    Is it possible to have another bridge interface (eg. br101) and use the interface for another network?
    For example: br101 maps to 172.16.0.1/16 and all VMs are under 172.16.0.0/16. Also if it’s possible for a VM to be on the two networks??
    Configuration examples would be big help. The idea behinds this is we have different network for different departments (QA, OPS, DEV …), we would like to place a QA’s VM directly connected to QA network for example.

    QA’s VM1 eth0 br100 Company’s QA network

    DEV’s VM1 eth0 br101 Company’s DEV network

    We will not be using NAT.

    Thanks!

    February 10, 2014 10:18
  18. harman

    please help me ..how to craete a script of creating network for openstack …

    February 16, 2014 21:50

Some HTML is OK


or, reply to this post via trackback.