Configuring Floating IP addresses for Networking in OpenStack Public and Private Clouds
August 10, 2012
Recently I outlined how VlanManager works and how it ensures network scalability and tenant isolation. Up to this point, however, I’ve only dealt with fixed IP networks of different tenants. While fixed IPs are what instances are given by default, they do not ensure that the instance is immediately reachable from the outside world (or from the rest of the data center). Imagine the following scenario:
You run a small LAMP website with one www server, a database server, and a firewall that handles network address translation (NAT) and traffic filtering. Typically you want the following to apply:
You do the following:
Fixed IPs in OpenStack work the same way as the network range 192.168.0.0/16 in the example above. They ensure only inter-instance connectivity inside a single OpenStack cluster. But OpenStack also introduces another pool of IP addresses, called “floating IPs.” Floating IPs are just publicly routable IPs that you typically buy from an ISP (the one that you put on the firewall in the above example). Users can allocate them to their instances, thus making them reachable from the outside world.
Difference between floating and fixed IPs
Floating IPs are not allocated to instances by default. Cloud users need to explicitly “grab” them from the pool configured by the OpenStack administrator and then attach them to their instances. Once the user has grabbed a floating IP from the pool, he becomes the “owner” of it (i.e., at any time he can detach the IP from a given instance and attach it to another). If an instance dies for some reason, the user does not lose the floating IP—it remains his own resource, ready to be attached to another instance. (Unfortunately, right now it is not possible to share a single floating IP among many instances for load balancing purposes, as can be done with elastic load balancing on Amazon EC2. However, lately Mirantis has open sourced a plugin for OpenStack that aims to provide such a feature.)
On the other hand, fixed IPs are allocated dynamically by the nova-network component when instances boot up. There is no way to tell OpenStack to assign a specific fixed IP to an instance. So you will probably find yourself in a situation in which once you terminate a VM accidentally and restore it from a snapshot, the new instance will most likely boot up with another fixed IP.
Systems administrators can configure multiple floating IP pools. However, unlike fixed IP pools, floating IP pools cannot be mapped to specific tenants. Each user can “grab” a floating IP from whichever floating IP pool he wants. But the main motivation behind multiple floating IP pools is that each of them can be served by a different ISP. This way, we can ensure that we maintain connectivity, even if one of the ISPs faces a breakdown.
Just to summarize, the key features of floating IPs are:
Floating IPs—internal vs. public clouds
The “public visibility” of floating IPs is a relative concept. For public clouds you probably want to define a floating IP pool as a pool of IPs publicly visible from the Internet. Your clients then assign them to instances to log into them via SSH from their home/office computers:
If you run a corporate cloud within in your data center, then a floating IP pool can be any IP range that exposes OpenStack instances to the rest of your data center.
For your data center traffic you might have the following range defined: 10.0.0.0/16.
Inside OpenStack you could have the following fixed IP range: 192.168.0.0/16 split into the tenants subnets.
To make OpenStack instances accessible from the rest of your data center, you could define the floating IP pool as a subnet of 10.0.0.0/8, (i.e., 10.0.0.0/16) and register it with OpenStack for the users to grab from.
Working with floating IPs
As I mentioned earlier, first, the systems administrator registers a floating IP pool in OpenStack:
This way the public pool is made available for tenants.
Now users follow this workflow:
The instance should now be visible from outside the OpenStack cluster under the floating IP.
How floating IPs work
So what happens inside the instance once the floating IP is added? The answer is…nothing. If you log in to it via SSH and display the network configuration, you will see that there is still a single network interface with a fixed IP configured.
All the setup is done on the compute node itself. All the floating IP work is nova-network’s job, which means setting up NAT between the instance’s fixed and floating IPs. An explanation of how NAT works can be found here.
Take a look at the following diagram:
It shows a single compute node configured in multihost networking mode and VlanManager used to configure fixed IP networks. The compute node is equipped with two network interfaces: eth0 is dedicated to fixed IP/VLAN traffic and eth1 is the interface on which the compute node is connected to the outside world and where floating IPs go. (To get to know how VlanManager configures fixed IP networks, refer to this previous post.)
Please note that while on the eth0 (fixed/private) interface we have no address configured, eth1 has an IP assigned, which is also the default gateway for the compute node (126.96.36.199).
When the user assigns a floating IP (188.8.131.52) to the instance VM_1, two things are happening:
Getting back to the diagram. Once the user wants to access the instance on its floating IP from the outside world (e.g., “ping 184.108.40.206″):
If an instance sends a packet to the world (e.g. “ping 220.127.116.11):
Notes about security
When using OpenStack, the systems administrator gives complete control of iptables to nova daemons. The set of rules configured is very complex and easily broken by any external manipulation. Moreover, each time nova-network daemon is restarted, it reapplies all the rules in OpenStack-related iptables chains. If there is a need to modify iptables behavior in any way, it should be done by changing the code in relevant places of linux_net.py (for NAT rules it would be the function floating_forward_rules).
It is also worth mentioning that nova-network does not seem to be monitoring its tables in any way. So if we manually throw away some rules from OpenStack related chains, they will not be fixed until the next nova-network restart.
So a sysadmin could easily open unwanted access to the compute host itself by accident. Remember that nova-network placed the floating IP as the secondary address on eth1 and set DNAT rules that direct the traffic to the instance’s fixed IP:
So all traffic hitting 18.104.22.168 goes effectively to 10.0.0.3.
Now let’s imagine that the sysadmin was fixing some network connectivity issues during the night and flushed all the NAT rules by accident, typing:
The above NAT rule has been thrown away, but eth1 still has a secondary IP 22.214.171.124 on it. So we can still reach 126.96.36.199 from the outside world, but instead of hitting the instance, we now have access to the compute node itself (the destination IP is no longer DNATed as we flushed all the NAT rules). The hole will be open until the next restart of the nova-network process, which will set up the rules again.
Configuring floating IPs
These are flags in nova.conf that influence the behavior of floating IPs:
The floating IP mechanism, besides exposing instances directly to the Internet, gives cloud users some flexibility. Having “grabbed” a floating IP from a pool, they can shuffle them (i.e., detach and attach them to different instances on the fly) thus facilitating new code releases and system upgrades. For sysadmins it poses a potential security risk, as the underlying mechanism (iptables) functions in a complicated way and lacks proper monitoring from the OpenStack side. Proper resource state monitoring in OpenStack is being addressed by this blueprint. Since it is still marked as a “work-in-progress”, it is very important to allow only the OpenStack software to touch firewall policies and not tamper with them by hand.9 comments
Continuing the Discussion