Category Archives: Virtualization

Making VM cloning simpler v2

In my last post I wrote about a procedure that I use to help me to locate VM IP address after creation. You can read about the problem here.

The problem with my solution is that you need to take the networking control away from Libvirt, creating the bridge with hand, configuring dnsmasq, etc. So, I looked at the code and I made libvirt do it for me. Here is the commit.

Basically, this adds an option to network config inside libvirt to setup forwarders for dnsmasq. This allows you to use the last one as the nameserver for your host machine (using in your resolv.conf). Every time a VM boots up, it receives an IP address from dnsmasq and gets its FQDN mapped to its IP address (something like a dnsupdate, but this happens internally because the same daemon provides both services).

You have to setup the domain and the forwarders options to get this working.

This new feature is documented and will appear in a future release of libvirt.

Post to Twitter

Making VM cloning simpler with libvirt, fabric and dnsmasq

When I bought my laptop I chose one with a lot of RAM because I use virtualization for development and testing. I use KVM and libvirt tools to manage my VMs. I have different templates of different Linux distributions that I clone frequently to do my tasks. One of the annoying things that I faced is the IP address assignment. Every time I new VM is created, I have log into the VM to see the IP address, to configure hostnames, /etc/hosts, etc.

libvirt uses dnsmasq by default as DHCP/DNS server for the virtualization environment. dnsmasq supports something like DNS update (because the same daemon has both services, it’s a built-in feature), so when an address is provided, the host sends its hostname and the dns is updated. To get this working you need to configure a couple of parameters. One of the problems is that libvirt doesn’t support this (until my patch is accepted). You have to disable the network configuration and configure it manually:

# virsh net-autostart default --disable
# virsh net-destroy default
# cat >> /etc/network/interfaces << EOF
auto virbr0
iface virbr0 inet static
        bridge_ports none
        up iptables -t nat -A POSTROUTING -s -j MASQUERADE
        up /usr/sbin/dnsmasq -C /etc/dnsmasq.d/virt.conf

Then, it’s time to configure dnsmasq:
cat > /etc/dnsmasq.d/virt.conf << EOF
# mkdir -p /var/lib/dnsmasq/
# touch /var/lib/dnsmasq/default.hostsfile /var/lib/dnsmasq/default.addnhosts

And now… up!
# ifup virbr0

Don’t forget to configure as nameserver for your local machine and to set the domain configured in dnsmasq in the search parameter of resolv.conf.

At this point, every time a VM boots up, you will be able to log in using its hostname. But the story hasn’t finished yet. In my machine, when I clone a guest I still have to log into it to change hostname, /etc/hosts, etc. To avoid this I created a script that I added to my toolbox, that uses Fabric to configure this stuff. It has the following requirements:

  • The hostname of the guest that you use as template must match the VM name in libvirt configuration.
  • The root user should be enabled and the ssh pubkey of your PC user must be in the authorized_file of the root user in the template.

Now, when I need to create one or more virtual machines, I just execute:

# templaname virtual1 virtual2 ... virtualN

The VMs are created, and I am able to use the hostname to log into them.

My laptop has Ubuntu 13.04 installed, and I tested this procedure using templates with Ubuntu 12.04 , Ubuntu 13.04 and Debian Wheezy.

Post to Twitter

Desktop Virtualization and IP addressing issue

My client’s problem

A bank, one of my clients, bought new hardware for their desktops some time ago. They have to run Windows 2000 on the branches because there is an critical application that doesn’t work on new versions. After they received the computers (around 700 units), they found that Windows 2000 isn’t compatible with the hardware. They are really new machines and the vendor doesn’t provide drivers for this old version of the Microsoft operating system. Porting the application to a new version is difficult and specially requires a lot of time.  Old computers are breaking from time to time and provisions for new hardware is urgent.

My client’s solution

The only solution that they found was to run Windows 2000 in the new machine virtualized. They install Linux, KVM and run the end user operating system over there. Hardware abstraction of KVM solves the problem, and Windows never sees the real hardware. This workaround works perfect. This may not be the best solution, but the other ones requires a lot of time.

The IP addressing problem

After finding this solution, they faced a new problem. Addressing. They use /24 subnets in the branches and big ones have more than 100 of desktops running. If they deploy, the virtualized desktop will double the required IPs per branch. One option is to change the IP addressing to support more IPs per branch, but that’s another big modification that requires time (IPs hardcoded in some apps, routers and firewalls configuration, etc, etc). It isn’t an option.

Linux hosts requires IP address because support techies will need to access to fix issues.

The solution to the IP addressing issue

The first measure to fix this issue was to configure every Linux with and IP addresss within the range, a special network range used for local communication between computers in the same network segment. All the branches will use the same subnet for the hosts. Connection between computers of every network branch is solved with this address, but connections from headquarters are impossible. This network isn’t routable.

So, another problem appears… how are support techies able to access the Linux hosts from the headquarters?

KVM uses a bridge to connect virtual machines to the physical network. With ebtables and iptables I’ve found a trick that permits connections to port 22 of the host using the IP address of the virtual machine. Let’s say that the VM uses the address which is a valid address in the bank network. VM also has it’s own MAC address, for example 52:54:00:bf:57:bb. Have a look at this ebtables rule: 

ebtables -t nat -A PREROUTING -p arp --arp-opcode 1 --arp-ip-dst -j arpreply --arpreply-mac 52:54:00:bf:57:bb

 This rule captures all the ARP requests asking for the IP address of Windows, generating a reply with the MAC address of the VM. So, ARP requests will be replied whether the Windows is running or not. This allows the packets to go through Linux always.

Now check this rule: 

iptables -t nat -A PREROUTING -i virbr0 -p tcp -d --dport 22 -j REDIRECT

This is a typical REDIRECT rule, all the packets that have the IP address of the Windows machine and destination port 22 will be redirected to the Linux host.

Looks easy, right? But there are more work that needs to be done. In the default gateway of your network, you have to insert these rules:


iptables -t nat -A POSTROUTING -o eth0 -d -p tcp --dport 22 -j SNAT --to 
iptables -t nat -A POSTROUTING -o ! eth0 -s -o -j SNAT --to

The first one is because the workstation only accepts traffic from local link network and the second one is to allow the machine to communicate with the rest of the world.

Post to Twitter

Convert Rackspace Cloud Server images to OVA

I wrote an script to convert Rackspace Cloud Server Images to OVA files. This file can be imported to Vmware and Virtualbox (and may be other hypervisors).

You have to get a copy of .tgz files generated from Cloud Servers snapshots and then provide it as first argument of this script.

The script is here.

In the comments you can see the requirements and how to use it.

Feedback is welcome.

Post to Twitter

Confusion using Iptables, nat and bridge

Today chatting via IRC I remembered a problem that I had some years ago with virtualization, iptables, nat and bridge. The situation of the guy asking was pretty similar. He has a one virtual machine (Qemu/KVM) connected to the world using a bridge and its default gateway is the virtualization host. He was trying to apply destination NAT to the VM in the host machine but it didn’t work. The rule was simple:

iptables -t nat -A POSTROUTING -s -o eth0 -j MASQUERADE

It is perfect, there is nothing wrong there but he never saw the packet in the POSTROUTING chain. Why? The quick answer is “packets don’t cross nat table twice”. There is a flag in the Linux bridge to enable filtering with Iptables. Packets go to Iptables in the kernel when they are forwarded by the bridge. This includes the NAT table.

In the bridging process, you don’t know the outgoing interface so the previous rule doesn’t work. He needs the interface because he’s using MASQUERADE. In the routing process, the packets go to iptables but they never cross NAT tables because the packet already crossed the table in the bridging process.

How can we fix this? There are two options I think:

  • echo 0 > /proc/sys/net/bridge/bridge-nf-call-iptables #To disable Iptables in the bridge.
  • Raw table: Some years ago appeared a new tables in Iptables. This table can be used to avoid packets (connection really) to enter the NAT table: iptables -t raw -I PREROUTING -i BRIDGE -s x.x.x.x -j NOTRACK.
If you still don’t understand why this happens, I’ll try to explain one more thing. If you have an scenario with Virtualization and you host is your gateway, the packets follow this steps: [VM]->[bridge]->[virtual interface]->[host]->[physical interface]->[net]. When they cross the host, you have the routing process there.

Post to Twitter

processor.max_cstate=0 saved my life :P

There is a bug in some place of Ubuntu 10.04 that does crazy things with virtual machines using KVM. Symptoms are a inconsistent cron process and time moving backwards. I didn’t know the root of the problem but I suspected that it was related to power saving. Why? Because the problems appeared at night, when the systems are usually idle. Solution: processor.max_cstate=0. That disables CPU power saving. I don’t have an entire explanation, but that has been working for me.

Post to Twitter