Category Archives: System Administration

Centos7, Rackspace and kswap0

I’ve just started to test an application from one my clients in CentOS 7. A Python app that runs on Python 2.7 that is the default version in the latest version of CentOS/RHEL. As usual, all the cloud providers don’t configure the swap space by default, so I have to configure Chef to create it when the deploy starts. When I started to test the deployment, I discover that the instance crashed compiling some of the Python modules. Debugging I see a problem that is reported a very frequently if you search in Google: “kswapd0 using all the CPU”.

I switch my tests to Digital Ocean, and there it worked perfectly. After a while, I discovered some differences in /etc/sysctl.conf. Rackspace guys setup vm.swappiness=0,  thing that triggers a bug in kswap0 (there is information about this on the net). I’ve disabled that configuration and the variable took the value of 30, the default in the kernel.

And everything works again…

A little history from the our daily DevOps engineer life .


Post to Twitter

Services that needs /vagrant to start

I have a set of recipes that I use to build the environments from the local developer desktop up to production. The background is provided in my last post.

One of the differences between the Vagrant environment and the other ones, is that HTTP apps have their web server DocumentRoot pointing to the /vagrant directory. This directory is available later at boot time, after the web server is started (at least with Apache in CentOS, I don’t remember in Ubuntu right now). The web server doesn’t start because the directory isn’t available.

To fix this, I use Upstart and its event system. Looks at this file (/etc/init/apache-vagrant.conf):

start on vagrant mounted


service httpd start

This script catches the event vagrant-mount, that is triggered when the /vagrant folder is ready to use. In this way, you execute the command to start Apache.

Let me know if you find a better way to do this.

Post to Twitter

My thoughts about SaltStack and why I use it

One year ago I started to use SaltStack. I have experience using Puppet for a long time, but a client was using Salt so I had to learn it. After a year using it, this is why it’s my preferred automation tool:

  • Simple syntax, you define the state of your servers, files, pkgs, whatever using YAML files.
  • You can apply your recipes on demand. You run a command and all the changes are applied at the moment. You don’t need to wait for polling.
  • Integrated remote execution. It’s in the core, you need anything external.
  • Cloud support. Salt cloud it’s part of the core right now. You set your credentials, you tell how many instances you want and salt-cloud will launch them, it will connect with the salt master automatically. Again, you don’t need anything from outside cloud.
  • Flexibility, it’s really flexible. You can code custom modules in Python easily to customize everything according to your needs.
  • Orchestration thanks to overstate. Overstate allows you to apply recipes in your  server in a specific order. For example, database servers first, then app servers.
  • The project is young, but it’s growing and growing. The number of bug reports, features requests and the fixes and enhancements on every version are impressive.

There are more reasons that I’m missing for sure.

You can see an example of something done using Salt here. It’s an example of a MongoDB cluster managed by Mongo. Let me know if you have questions about it.

Post to Twitter

Elastic MongoDB

A few months ago, I had spare time and I wanted to combine my experience in the last year with MongoDB and SaltStack. The result is Elastic MongoDB, an example of orchestration/automation to setup a MongoDB cluster with replication and sharding using Salt. The instructions to launch are in the README, it should be running in a few steps. There is only an issue, you have to run the salt \* state.highstate command three of four times to get everything working. That’s because there are some dependency between the boxes, so the setup should be in some order. By default, Salt applies all recipes  at the same time, generating failures. To fix this I have to use a feature of Salt called overstate, but I haven’t had time to implement it. 

I’ll fix when I have some time, but anyway you can try it. Have look here.

Fell free to ask any question!

Post to Twitter

New page in my wiki knowledge base: 389 DS errors and solutions

In the last two or three years I worked a lot with 389 DS. I setup small installations and really big ones (more than 10.000 objects and 150 replica servers around the world). I decided to open a new page in my wiki to write down the errors and the solutios that I’ve found in this long way with DS. You can have a look here. Right now there is only one error and one solution, but I’ll try to fill it with more information gathered from my experience soon.

Post to Twitter

Another couple of notes about MongoDB

I forgot to mention another couple of thoughts about MongoDB in my last post. 

  1. MongoDB is fast, but there is no magic. When I first implemented it, I didn’t take care of indexes. I thought: “MongoDB is fast, I’ll setup indexes later.” Big mistake! Without indexes the performance was horrible. I have to rollback my deployment in production. 
  2. Padding. When you insert documents in your database, MongoDB adds them next to each other. If your document grows (for example, if you have an array and you add an element), Mongo must move the document to a new place. Document growing implies two tasks: document movement and indexes update (of course). If you add elements to a lot of documents at once, there is a lot of work, locking, reads that have to wait, etc. To avoid this, there is something called Padding. When Mongo detects that documents tend to grow, it adds extra space to them. So, when you insert a new element, movement won’t be needed. This padding is calculated using the padding factor, which is clearly explained in the official documentation. In my case, to be sure that performance will be good, I force padding after my first bulk import. I did it running “db.runCommand ( { compact: ‘collection’, paddingFactor: 1.1 } )“. That command adds a padding of 10% of each document and compacts the collection (eliminating wasted space).

Have a look at the log file, by default MongoDB writes a lot of interesting information there. For example, if you see update log entries with nmoves:1, that means that the update operation required the document to be moved.

Post to Twitter

A couple of notes from my first steps with MongoDB

I’m starting the process of migration from MySQL to MongoDB. I’ve heard and tried a lot of interesting things and it’s worth the migration. Async writes sounds very interesting for my app and schemaless is fantastic (goodbye “alter table” and migration scripts). I faced two problems, the first one is documented, the second not (or I wasn’t able to find it).

  1. Multi-key indexes only supports one array field. Mongo supports array fields, an special type in the document (record in the SQL world) that holds a list of items of any type. You can use them in an index, but only one at a time.
  2. Writing aggressively to indexed fields kills performance. I have a huge collection with more than 10M of documents (rows in the SQL world). In my applications I iterate over the docs every minute to do some processing and then I write the results to a field of the same doc. This field was part of an index (it’s an array).  I have to remove it from the index. After that, processing time was reduced from more than 1 minute to less than 10 seconds. It looks like the index update locks the table, blocking read operations too. I’m waiting for a confirmation from the mailing list about this.

Post to Twitter

Desktop Virtualization and IP addressing issue

My client’s problem

A bank, one of my clients, bought new hardware for their desktops some time ago. They have to run Windows 2000 on the branches because there is an critical application that doesn’t work on new versions. After they received the computers (around 700 units), they found that Windows 2000 isn’t compatible with the hardware. They are really new machines and the vendor doesn’t provide drivers for this old version of the Microsoft operating system. Porting the application to a new version is difficult and specially requires a lot of time.  Old computers are breaking from time to time and provisions for new hardware is urgent.

My client’s solution

The only solution that they found was to run Windows 2000 in the new machine virtualized. They install Linux, KVM and run the end user operating system over there. Hardware abstraction of KVM solves the problem, and Windows never sees the real hardware. This workaround works perfect. This may not be the best solution, but the other ones requires a lot of time.

The IP addressing problem

After finding this solution, they faced a new problem. Addressing. They use /24 subnets in the branches and big ones have more than 100 of desktops running. If they deploy, the virtualized desktop will double the required IPs per branch. One option is to change the IP addressing to support more IPs per branch, but that’s another big modification that requires time (IPs hardcoded in some apps, routers and firewalls configuration, etc, etc). It isn’t an option.

Linux hosts requires IP address because support techies will need to access to fix issues.

The solution to the IP addressing issue

The first measure to fix this issue was to configure every Linux with and IP addresss within the range, a special network range used for local communication between computers in the same network segment. All the branches will use the same subnet for the hosts. Connection between computers of every network branch is solved with this address, but connections from headquarters are impossible. This network isn’t routable.

So, another problem appears… how are support techies able to access the Linux hosts from the headquarters?

KVM uses a bridge to connect virtual machines to the physical network. With ebtables and iptables I’ve found a trick that permits connections to port 22 of the host using the IP address of the virtual machine. Let’s say that the VM uses the address which is a valid address in the bank network. VM also has it’s own MAC address, for example 52:54:00:bf:57:bb. Have a look at this ebtables rule: 

ebtables -t nat -A PREROUTING -p arp --arp-opcode 1 --arp-ip-dst -j arpreply --arpreply-mac 52:54:00:bf:57:bb

 This rule captures all the ARP requests asking for the IP address of Windows, generating a reply with the MAC address of the VM. So, ARP requests will be replied whether the Windows is running or not. This allows the packets to go through Linux always.

Now check this rule: 

iptables -t nat -A PREROUTING -i virbr0 -p tcp -d --dport 22 -j REDIRECT

This is a typical REDIRECT rule, all the packets that have the IP address of the Windows machine and destination port 22 will be redirected to the Linux host.

Looks easy, right? But there are more work that needs to be done. In the default gateway of your network, you have to insert these rules:


iptables -t nat -A POSTROUTING -o eth0 -d -p tcp --dport 22 -j SNAT --to 
iptables -t nat -A POSTROUTING -o ! eth0 -s -o -j SNAT --to

The first one is because the workstation only accepts traffic from local link network and the second one is to allow the machine to communicate with the rest of the world.

Post to Twitter