Hacked By Prosox
GreetZ: Shade & Sxtz
In my last post I wrote about a procedure that I use to help me to locate VM IP address after creation. You can read about the problem here.
The problem with my solution is that you need to take the networking control away from Libvirt, creating the bridge with hand, configuring dnsmasq, etc. So, I looked at the code and I made libvirt do it for me. Here is the commit.
Basically, this adds an option to network config inside libvirt to setup forwarders for dnsmasq. This allows you to use the last one as the nameserver for your host machine (using 127.0.0.1 in your resolv.conf). Every time a VM boots up, it receives an IP address from dnsmasq and gets its FQDN mapped to its IP address (something like a dnsupdate, but this happens internally because the same daemon provides both services).
You have to setup the domain and the forwarders options to get this working.
This new feature is documented and will appear in a future release of libvirt.
When I bought my laptop I chose one with a lot of RAM because I use virtualization for development and testing. I use KVM and libvirt tools to manage my VMs. I have different templates of different Linux distributions that I clone frequently to do my tasks. One of the annoying things that I faced is the IP address assignment. Every time I new VM is created, I have log into the VM to see the IP address, to configure hostnames, /etc/hosts, etc.
libvirt uses dnsmasq by default as DHCP/DNS server for the virtualization environment. dnsmasq supports something like DNS update (because the same daemon has both services, it’s a built-in feature), so when an address is provided, the host sends its hostname and the dns is updated. To get this working you need to configure a couple of parameters. One of the problems is that libvirt doesn’t support this (until my patch is accepted). You have to disable the network configuration and configure it manually:
# virsh net-autostart default --disable # virsh net-destroy default # cat >> /etc/network/interfaces << EOF auto virbr0 iface virbr0 inet static bridge_ports none address 192.168.122.1 netmask 255.255.255.0 up iptables -t nat -A POSTROUTING -s 192.168.122.0/24 -j MASQUERADE up /usr/sbin/dnsmasq -C /etc/dnsmasq.d/virt.conf EOF
cat > /etc/dnsmasq.d/virt.conf << EOF strict-order domain-needed domain=woitasen.local expand-hosts local=/woitasen.local/ no-resolv server=220.127.116.11 server=18.104.22.168 pid-file=/var/run/dnsmasq-diegows-laptop.pid bind-dynamic addn-hosts=/var/lib/dnsmasq/default.addnhosts dhcp-option=3,192.168.122.1 dhcp-range=192.168.122.2,192.168.122.254 dhcp-no-override dhcp-leasefile=/var/lib/dnsmasq/default.leases dhcp-lease-max=253 dhcp-hostsfile=/var/lib/dnsmasq/default.hostsfile EOF # mkdir -p /var/lib/dnsmasq/ # touch /var/lib/dnsmasq/default.hostsfile /var/lib/dnsmasq/default.addnhosts
# ifup virbr0
At this point, every time a VM boots up, you will be able to log in using its hostname. But the story hasn’t finished yet. In my machine, when I clone a guest I still have to log into it to change hostname, /etc/hosts, etc. To avoid this I created a script that I added to my toolbox, that uses Fabric to configure this stuff. It has the following requirements:
Now, when I need to create one or more virtual machines, I just execute:
# my-virt-clone.py templaname virtual1 virtual2 ... virtualN
My laptop has Ubuntu 13.04 installed, and I tested this procedure using templates with Ubuntu 12.04 , Ubuntu 13.04 and Debian Wheezy.
My client’s problem
A bank, one of my clients, bought new hardware for their desktops some time ago. They have to run Windows 2000 on the branches because there is an critical application that doesn’t work on new versions. After they received the computers (around 700 units), they found that Windows 2000 isn’t compatible with the hardware. They are really new machines and the vendor doesn’t provide drivers for this old version of the Microsoft operating system. Porting the application to a new version is difficult and specially requires a lot of time. Old computers are breaking from time to time and provisions for new hardware is urgent.
My client’s solution
The only solution that they found was to run Windows 2000 in the new machine virtualized. They install Linux, KVM and run the end user operating system over there. Hardware abstraction of KVM solves the problem, and Windows never sees the real hardware. This workaround works perfect. This may not be the best solution, but the other ones requires a lot of time.
The IP addressing problem
After finding this solution, they faced a new problem. Addressing. They use /24 subnets in the branches and big ones have more than 100 of desktops running. If they deploy, the virtualized desktop will double the required IPs per branch. One option is to change the IP addressing to support more IPs per branch, but that’s another big modification that requires time (IPs hardcoded in some apps, routers and firewalls configuration, etc, etc). It isn’t an option.
Linux hosts requires IP address because support techies will need to access to fix issues.
The solution to the IP addressing issue
The first measure to fix this issue was to configure every Linux with and IP addresss within the range 169.254.0.0/16, a special network range used for local communication between computers in the same network segment. All the branches will use the same subnet for the hosts. Connection between computers of every network branch is solved with this address, but connections from headquarters are impossible. This network isn’t routable.
So, another problem appears… how are support techies able to access the Linux hosts from the headquarters?
KVM uses a bridge to connect virtual machines to the physical network. With ebtables and iptables I’ve found a trick that permits connections to port 22 of the host using the IP address of the virtual machine. Let’s say that the VM uses the address 10.60.130.100 which is a valid address in the bank network. VM also has it’s own MAC address, for example 52:54:00:bf:57:bb. Have a look at this ebtables rule:
ebtables -t nat -A PREROUTING -p arp --arp-opcode 1 --arp-ip-dst 10.60.130.100 -j arpreply --arpreply-mac 52:54:00:bf:57:bb
Now check this rule:
iptables -t nat -A PREROUTING -i virbr0 -p tcp -d 10.60.130.100 --dport 22 -j REDIRECT
Looks easy, right? But there are more work that needs to be done. In the default gateway of your network, you have to insert these rules:
iptables -t nat -A POSTROUTING -o eth0 -d 10.60.130.0/24 -p tcp --dport 22 -j SNAT --to 169.254.0.1 iptables -t nat -A POSTROUTING -o ! eth0 -s 169.254.0.0/24 -o 10.0.0.0/8 -j SNAT --to 10.60.130.254
The first one is because the workstation only accepts traffic from local link network and the second one is to allow the machine to communicate with the rest of the world.
I wrote an script to convert Rackspace Cloud Server Images to OVA files. This file can be imported to Vmware and Virtualbox (and may be other hypervisors).
You have to get a copy of .tgz files generated from Cloud Servers snapshots and then provide it as first argument of this script.
The script is here.
In the comments you can see the requirements and how to use it.
Feedback is welcome.
Today chatting via IRC I remembered a problem that I had some years ago with virtualization, iptables, nat and bridge. The situation of the guy asking was pretty similar. He has a one virtual machine (Qemu/KVM) connected to the world using a bridge and its default gateway is the virtualization host. He was trying to apply destination NAT to the VM in the host machine but it didn’t work. The rule was simple:
iptables -t nat -A POSTROUTING -s 10.0.3.11 -o eth0 -j MASQUERADE
It is perfect, there is nothing wrong there but he never saw the packet in the POSTROUTING chain. Why? The quick answer is “packets don’t cross nat table twice”. There is a flag in the Linux bridge to enable filtering with Iptables. Packets go to Iptables in the kernel when they are forwarded by the bridge. This includes the NAT table.
In the bridging process, you don’t know the outgoing interface so the previous rule doesn’t work. He needs the interface because he’s using MASQUERADE. In the routing process, the packets go to iptables but they never cross NAT tables because the packet already crossed the table in the bridging process.
How can we fix this? There are two options I think:
There is a bug in some place of Ubuntu 10.04 that does crazy things with virtual machines using KVM. Symptoms are a inconsistent cron process and time moving backwards. I didn’t know the root of the problem but I suspected that it was related to power saving. Why? Because the problems appeared at night, when the systems are usually idle. Solution: processor.max_cstate=0. That disables CPU power saving. I don’t have an entire explanation, but that has been working for me.