Tag Archives: kvm

Desktop Virtualization and IP addressing issue

My client’s problem

A bank, one of my clients, bought new hardware for their desktops some time ago. They have to run Windows 2000 on the branches because there is an critical application that doesn’t work on new versions. After they received the computers (around 700 units), they found that Windows 2000 isn’t compatible with the hardware. They are really new machines and the vendor doesn’t provide drivers for this old version of the Microsoft operating system. Porting the application to a new version is difficult and specially requires a lot of time.  Old computers are breaking from time to time and provisions for new hardware is urgent.

My client’s solution

The only solution that they found was to run Windows 2000 in the new machine virtualized. They install Linux, KVM and run the end user operating system over there. Hardware abstraction of KVM solves the problem, and Windows never sees the real hardware. This workaround works perfect. This may not be the best solution, but the other ones requires a lot of time.

The IP addressing problem

After finding this solution, they faced a new problem. Addressing. They use /24 subnets in the branches and big ones have more than 100 of desktops running. If they deploy, the virtualized desktop will double the required IPs per branch. One option is to change the IP addressing to support more IPs per branch, but that’s another big modification that requires time (IPs hardcoded in some apps, routers and firewalls configuration, etc, etc). It isn’t an option.

Linux hosts requires IP address because support techies will need to access to fix issues.

The solution to the IP addressing issue

The first measure to fix this issue was to configure every Linux with and IP addresss within the range, a special network range used for local communication between computers in the same network segment. All the branches will use the same subnet for the hosts. Connection between computers of every network branch is solved with this address, but connections from headquarters are impossible. This network isn’t routable.

So, another problem appears… how are support techies able to access the Linux hosts from the headquarters?

KVM uses a bridge to connect virtual machines to the physical network. With ebtables and iptables I’ve found a trick that permits connections to port 22 of the host using the IP address of the virtual machine. Let’s say that the VM uses the address which is a valid address in the bank network. VM also has it’s own MAC address, for example 52:54:00:bf:57:bb. Have a look at this ebtables rule: 

ebtables -t nat -A PREROUTING -p arp --arp-opcode 1 --arp-ip-dst -j arpreply --arpreply-mac 52:54:00:bf:57:bb

 This rule captures all the ARP requests asking for the IP address of Windows, generating a reply with the MAC address of the VM. So, ARP requests will be replied whether the Windows is running or not. This allows the packets to go through Linux always.

Now check this rule: 

iptables -t nat -A PREROUTING -i virbr0 -p tcp -d --dport 22 -j REDIRECT

This is a typical REDIRECT rule, all the packets that have the IP address of the Windows machine and destination port 22 will be redirected to the Linux host.

Looks easy, right? But there are more work that needs to be done. In the default gateway of your network, you have to insert these rules:


iptables -t nat -A POSTROUTING -o eth0 -d -p tcp --dport 22 -j SNAT --to 
iptables -t nat -A POSTROUTING -o ! eth0 -s -o -j SNAT --to

The first one is because the workstation only accepts traffic from local link network and the second one is to allow the machine to communicate with the rest of the world.

Post to Twitter

Confusion using Iptables, nat and bridge

Today chatting via IRC I remembered a problem that I had some years ago with virtualization, iptables, nat and bridge. The situation of the guy asking was pretty similar. He has a one virtual machine (Qemu/KVM) connected to the world using a bridge and its default gateway is the virtualization host. He was trying to apply destination NAT to the VM in the host machine but it didn’t work. The rule was simple:

iptables -t nat -A POSTROUTING -s -o eth0 -j MASQUERADE

It is perfect, there is nothing wrong there but he never saw the packet in the POSTROUTING chain. Why? The quick answer is “packets don’t cross nat table twice”. There is a flag in the Linux bridge to enable filtering with Iptables. Packets go to Iptables in the kernel when they are forwarded by the bridge. This includes the NAT table.

In the bridging process, you don’t know the outgoing interface so the previous rule doesn’t work. He needs the interface because he’s using MASQUERADE. In the routing process, the packets go to iptables but they never cross NAT tables because the packet already crossed the table in the bridging process.

How can we fix this? There are two options I think:

  • echo 0 > /proc/sys/net/bridge/bridge-nf-call-iptables #To disable Iptables in the bridge.
  • Raw table: Some years ago appeared a new tables in Iptables. This table can be used to avoid packets (connection really) to enter the NAT table: iptables -t raw -I PREROUTING -i BRIDGE -s x.x.x.x -j NOTRACK.
If you still don’t understand why this happens, I’ll try to explain one more thing. If you have an scenario with Virtualization and you host is your gateway, the packets follow this steps: [VM]->[bridge]->[virtual interface]->[host]->[physical interface]->[net]. When they cross the host, you have the routing process there.

Post to Twitter

processor.max_cstate=0 saved my life :P

There is a bug in some place of Ubuntu 10.04 that does crazy things with virtual machines using KVM. Symptoms are a inconsistent cron process and time moving backwards. I didn’t know the root of the problem but I suspected that it was related to power saving. Why? Because the problems appeared at night, when the systems are usually idle. Solution: processor.max_cstate=0. That disables CPU power saving. I don’t have an entire explanation, but that has been working for me.

Post to Twitter