My thoughts about SaltStack and why I use it

One year ago I started to use SaltStack. I have experience using Puppet for a long time, but a client was using Salt so I had to learn it. After a year using it, this is why it’s my preferred automation tool:

  • Simple syntax, you define the state of your servers, files, pkgs, whatever using YAML files.
  • You can apply your recipes on demand. You run a command and all the changes are applied at the moment. You don’t need to wait for polling.
  • Integrated remote execution. It’s in the core, you need anything external.
  • Cloud support. Salt cloud it’s part of the core right now. You set your credentials, you tell how many instances you want and salt-cloud will launch them, it will connect with the salt master automatically. Again, you don’t need anything from outside cloud.
  • Flexibility, it’s really flexible. You can code custom modules in Python easily to customize everything according to your needs.
  • Orchestration thanks to overstate. Overstate allows you to apply recipes in your  server in a specific order. For example, database servers first, then app servers.
  • The project is young, but it’s growing and growing. The number of bug reports, features requests and the fixes and enhancements on every version are impressive.

There are more reasons that I’m missing for sure.

You can see an example of something done using Salt here. It’s an example of a MongoDB cluster managed by Mongo. Let me know if you have questions about it.

Post to Twitter

Elastic MongoDB

A few months ago, I had spare time and I wanted to combine my experience in the last year with MongoDB and SaltStack. The result is Elastic MongoDB, an example of orchestration/automation to setup a MongoDB cluster with replication and sharding using Salt. The instructions to launch are in the README, it should be running in a few steps. There is only an issue, you have to run the salt \* state.highstate command three of four times to get everything working. That’s because there are some dependency between the boxes, so the setup should be in some order. By default, Salt applies all recipes  at the same time, generating failures. To fix this I have to use a feature of Salt called overstate, but I haven’t had time to implement it. 

I’ll fix when I have some time, but anyway you can try it. Have look here.

Fell free to ask any question!

Post to Twitter

OpenStack: using one VLAN per project

One of the networking options of OpenStack is to use one VLAN per project/tenant. This allows us to separate projects completely. In the documentation, is not very clear how to configure this. The steps about how to modify the configuration files are fine, but the command lines required to setup the network per project not.

Basically, when you have a VLAN per project you have to create a network and then assign a subnet to it.

This is done with the following two commands:

neutron net-create \
 --tenant-id 62bc0d1979cf4bda82f1cde4ac36543a \
 --provider:segmentation_id=12 \

 demo-net is the name of the network that you are creating. You need to be clear with the network_type and the segmentation_id. The first one is self-explanatory and the second one is the VLAN ID that you want to associate to that network. This is very import, if you don’t specify this, packets will be tagged with default VLAN ID (1).


Post to Twitter

Making VM cloning simpler v2

In my last post I wrote about a procedure that I use to help me to locate VM IP address after creation. You can read about the problem here.

The problem with my solution is that you need to take the networking control away from Libvirt, creating the bridge with hand, configuring dnsmasq, etc. So, I looked at the code and I made libvirt do it for me. Here is the commit.

Basically, this adds an option to network config inside libvirt to setup forwarders for dnsmasq. This allows you to use the last one as the nameserver for your host machine (using in your resolv.conf). Every time a VM boots up, it receives an IP address from dnsmasq and gets its FQDN mapped to its IP address (something like a dnsupdate, but this happens internally because the same daemon provides both services).

You have to setup the domain and the forwarders options to get this working.

This new feature is documented and will appear in a future release of libvirt.

Post to Twitter

Making VM cloning simpler with libvirt, fabric and dnsmasq

When I bought my laptop I chose one with a lot of RAM because I use virtualization for development and testing. I use KVM and libvirt tools to manage my VMs. I have different templates of different Linux distributions that I clone frequently to do my tasks. One of the annoying things that I faced is the IP address assignment. Every time I new VM is created, I have log into the VM to see the IP address, to configure hostnames, /etc/hosts, etc.

libvirt uses dnsmasq by default as DHCP/DNS server for the virtualization environment. dnsmasq supports something like DNS update (because the same daemon has both services, it’s a built-in feature), so when an address is provided, the host sends its hostname and the dns is updated. To get this working you need to configure a couple of parameters. One of the problems is that libvirt doesn’t support this (until my patch is accepted). You have to disable the network configuration and configure it manually:

# virsh net-autostart default --disable
# virsh net-destroy default
# cat >> /etc/network/interfaces << EOF
auto virbr0
iface virbr0 inet static
        bridge_ports none
        up iptables -t nat -A POSTROUTING -s -j MASQUERADE
        up /usr/sbin/dnsmasq -C /etc/dnsmasq.d/virt.conf

Then, it’s time to configure dnsmasq:
cat > /etc/dnsmasq.d/virt.conf << EOF
# mkdir -p /var/lib/dnsmasq/
# touch /var/lib/dnsmasq/default.hostsfile /var/lib/dnsmasq/default.addnhosts

And now… up!
# ifup virbr0

Don’t forget to configure as nameserver for your local machine and to set the domain configured in dnsmasq in the search parameter of resolv.conf.

At this point, every time a VM boots up, you will be able to log in using its hostname. But the story hasn’t finished yet. In my machine, when I clone a guest I still have to log into it to change hostname, /etc/hosts, etc. To avoid this I created a script that I added to my toolbox, that uses Fabric to configure this stuff. It has the following requirements:

  • The hostname of the guest that you use as template must match the VM name in libvirt configuration.
  • The root user should be enabled and the ssh pubkey of your PC user must be in the authorized_file of the root user in the template.

Now, when I need to create one or more virtual machines, I just execute:

# templaname virtual1 virtual2 ... virtualN

The VMs are created, and I am able to use the hostname to log into them.

My laptop has Ubuntu 13.04 installed, and I tested this procedure using templates with Ubuntu 12.04 , Ubuntu 13.04 and Debian Wheezy.

Post to Twitter

New page in my wiki knowledge base: 389 DS errors and solutions

In the last two or three years I worked a lot with 389 DS. I setup small installations and really big ones (more than 10.000 objects and 150 replica servers around the world). I decided to open a new page in my wiki to write down the errors and the solutios that I’ve found in this long way with DS. You can have a look here. Right now there is only one error and one solution, but I’ll try to fill it with more information gathered from my experience soon.

Post to Twitter

Another couple of notes about MongoDB

I forgot to mention another couple of thoughts about MongoDB in my last post. 

  1. MongoDB is fast, but there is no magic. When I first implemented it, I didn’t take care of indexes. I thought: “MongoDB is fast, I’ll setup indexes later.” Big mistake! Without indexes the performance was horrible. I have to rollback my deployment in production. 
  2. Padding. When you insert documents in your database, MongoDB adds them next to each other. If your document grows (for example, if you have an array and you add an element), Mongo must move the document to a new place. Document growing implies two tasks: document movement and indexes update (of course). If you add elements to a lot of documents at once, there is a lot of work, locking, reads that have to wait, etc. To avoid this, there is something called Padding. When Mongo detects that documents tend to grow, it adds extra space to them. So, when you insert a new element, movement won’t be needed. This padding is calculated using the padding factor, which is clearly explained in the official documentation. In my case, to be sure that performance will be good, I force padding after my first bulk import. I did it running “db.runCommand ( { compact: ‘collection’, paddingFactor: 1.1 } )“. That command adds a padding of 10% of each document and compacts the collection (eliminating wasted space).

Have a look at the log file, by default MongoDB writes a lot of interesting information there. For example, if you see update log entries with nmoves:1, that means that the update operation required the document to be moved.

Post to Twitter

A couple of notes from my first steps with MongoDB

I’m starting the process of migration from MySQL to MongoDB. I’ve heard and tried a lot of interesting things and it’s worth the migration. Async writes sounds very interesting for my app and schemaless is fantastic (goodbye “alter table” and migration scripts). I faced two problems, the first one is documented, the second not (or I wasn’t able to find it).

  1. Multi-key indexes only supports one array field. Mongo supports array fields, an special type in the document (record in the SQL world) that holds a list of items of any type. You can use them in an index, but only one at a time.
  2. Writing aggressively to indexed fields kills performance. I have a huge collection with more than 10M of documents (rows in the SQL world). In my applications I iterate over the docs every minute to do some processing and then I write the results to a field of the same doc. This field was part of an index (it’s an array).  I have to remove it from the index. After that, processing time was reduced from more than 1 minute to less than 10 seconds. It looks like the index update locks the table, blocking read operations too. I’m waiting for a confirmation from the mailing list about this.

Post to Twitter

Desktop Virtualization and IP addressing issue

My client’s problem

A bank, one of my clients, bought new hardware for their desktops some time ago. They have to run Windows 2000 on the branches because there is an critical application that doesn’t work on new versions. After they received the computers (around 700 units), they found that Windows 2000 isn’t compatible with the hardware. They are really new machines and the vendor doesn’t provide drivers for this old version of the Microsoft operating system. Porting the application to a new version is difficult and specially requires a lot of time.  Old computers are breaking from time to time and provisions for new hardware is urgent.

My client’s solution

The only solution that they found was to run Windows 2000 in the new machine virtualized. They install Linux, KVM and run the end user operating system over there. Hardware abstraction of KVM solves the problem, and Windows never sees the real hardware. This workaround works perfect. This may not be the best solution, but the other ones requires a lot of time.

The IP addressing problem

After finding this solution, they faced a new problem. Addressing. They use /24 subnets in the branches and big ones have more than 100 of desktops running. If they deploy, the virtualized desktop will double the required IPs per branch. One option is to change the IP addressing to support more IPs per branch, but that’s another big modification that requires time (IPs hardcoded in some apps, routers and firewalls configuration, etc, etc). It isn’t an option.

Linux hosts requires IP address because support techies will need to access to fix issues.

The solution to the IP addressing issue

The first measure to fix this issue was to configure every Linux with and IP addresss within the range, a special network range used for local communication between computers in the same network segment. All the branches will use the same subnet for the hosts. Connection between computers of every network branch is solved with this address, but connections from headquarters are impossible. This network isn’t routable.

So, another problem appears… how are support techies able to access the Linux hosts from the headquarters?

KVM uses a bridge to connect virtual machines to the physical network. With ebtables and iptables I’ve found a trick that permits connections to port 22 of the host using the IP address of the virtual machine. Let’s say that the VM uses the address which is a valid address in the bank network. VM also has it’s own MAC address, for example 52:54:00:bf:57:bb. Have a look at this ebtables rule: 

ebtables -t nat -A PREROUTING -p arp --arp-opcode 1 --arp-ip-dst -j arpreply --arpreply-mac 52:54:00:bf:57:bb

 This rule captures all the ARP requests asking for the IP address of Windows, generating a reply with the MAC address of the VM. So, ARP requests will be replied whether the Windows is running or not. This allows the packets to go through Linux always.

Now check this rule: 

iptables -t nat -A PREROUTING -i virbr0 -p tcp -d --dport 22 -j REDIRECT

This is a typical REDIRECT rule, all the packets that have the IP address of the Windows machine and destination port 22 will be redirected to the Linux host.

Looks easy, right? But there are more work that needs to be done. In the default gateway of your network, you have to insert these rules:


iptables -t nat -A POSTROUTING -o eth0 -d -p tcp --dport 22 -j SNAT --to 
iptables -t nat -A POSTROUTING -o ! eth0 -s -o -j SNAT --to

The first one is because the workstation only accepts traffic from local link network and the second one is to allow the machine to communicate with the rest of the world.

Post to Twitter