cron, /etc/timezone and Debian

Today I had a problem with cron. One of that typical situation where a cron job doesn’t run and you don’t know the reason. If I ran the cron daemon from the command line, the cron job was executed. If I ran the daemon from the init script, nothing. Debugging the problem I executed the init script with “bash -x” where I saw that the script set the variable TZ with the content of /etc/timezone. This file had a different timezone from /etc/locatime link. The problem was that the job was working, but at different time. Cron has been a source of headaches…

Conclusion, if you have a problem with check, check /etc/localtime and /etc/timezone. They should point to the same timezone.

Post to Twitter

Confusion using Iptables, nat and bridge

Today chatting via IRC I remembered a problem that I had some years ago with virtualization, iptables, nat and bridge. The situation of the guy asking was pretty similar. He has a one virtual machine (Qemu/KVM) connected to the world using a bridge and its default gateway is the virtualization host. He was trying to apply destination NAT to the VM in the host machine but it didn’t work. The rule was simple:

iptables -t nat -A POSTROUTING -s 10.0.3.11 -o eth0 -j MASQUERADE

It is perfect, there is nothing wrong there but he never saw the packet in the POSTROUTING chain. Why? The quick answer is “packets don’t cross nat table twice”. There is a flag in the Linux bridge to enable filtering with Iptables. Packets go to Iptables in the kernel when they are forwarded by the bridge. This includes the NAT table.

In the bridging process, you don’t know the outgoing interface so the previous rule doesn’t work. He needs the interface because he’s using MASQUERADE. In the routing process, the packets go to iptables but they never cross NAT tables because the packet already crossed the table in the bridging process.

How can we fix this? There are two options I think:

  • echo 0 > /proc/sys/net/bridge/bridge-nf-call-iptables #To disable Iptables in the bridge.
  • Raw table: Some years ago appeared a new tables in Iptables. This table can be used to avoid packets (connection really) to enter the NAT table: iptables -t raw -I PREROUTING -i BRIDGE -s x.x.x.x -j NOTRACK.
If you still don’t understand why this happens, I’ll try to explain one more thing. If you have an scenario with Virtualization and you host is your gateway, the packets follow this steps: [VM]->[bridge]->[virtual interface]->[host]->[physical interface]->[net]. When they cross the host, you have the routing process there.

Post to Twitter

Synchronizing files with Csync2

When you have to synchronize configuration files between servers you always think in Ssh/Rsync at first. Then you remember that you need to enable login to some account and some security issues appears. You are giving full shell access and you only need copy and execution and nothing else. If you have root login disabled you’ll have problem with permissions too.

I always desired an application that only synchronizes files and executes some action after the copy until I found Csync2. This application is a simple tool to copy configuration files (and other types of files) and do something after that (usually reload the config.). It doesn’t require a shell account because it runs as service. The configuration of Csync2 is simple: a list of hosts, a list of files and a list of actions to execute if it detects a change in a file. Let’s start the configuration.

The following steps must be executed in all servers.

First, you need the configuration file, /etc/csync2.cfg:

group cluster1 {

        host server1;
        host (server2);
        host (server3);

        include /etc/resolv.conf;
        include /etc/apache2/;
        exclude /etc/apach2/ports.conf;

        action {
                pattern /etc/apache2;
                exec "/etc/init.d/apache2 restart";
                logfile "/var/log/csync2/apache-restart.log";
                do-local;
        }
}

Looks easy, right? I’ll explain it anway:

  • host: Each line defines the members of the syncronization group. The hosts with parenthesis are slaves, they only receive files from the member with parenthesis. I usually have all the servers as slaves except one. I prefer to have only one point where you modify the configuration.
  • include: This lines list the files to sync.
  • exclude: If you are synchronizing directories, sometimes you need to exclude some files inside them.
  • action: This section defines a command to execute if a file matching the pattern changes. You can set a file to save the output of the command and do-local tells Csync2 to execute the action in the host where you are going to dispatch the sync.

This file must be in all the servers.

In the second step, you have to create the X.509 certificates to use SSL. Csync2 doesn’t use X.509 authentication, it only requires the certificates to enable a secure communication. This should be in the developer’s TODO list. So, don’t worry about creating good certs, the following commands are enough:
# openssl genrsa -out /etc/csync2_ssl_key.pem 1024 # yes '' | openssl req -new -key /etc/csync2_ssl_key.pem -out /etc/csync2_ssl_cert.csr # openssl x509 -req -days 600 -in /etc/csync2_ssl_cert.csr -signkey /etc/csync2_ssl_key.pem -out /etc/csync2_ssl_cert.pem

The third and last step of configuration is to create the authentication key. This key must be the same in all servers. It’s created with the following command:
csync2 -k /etc/csync2.key
At this point, you are ready to sync files:
csync -xr
You have to execute that command in the server without parenthesis in the csync2.cfg. As I said before, I prefer to have one master server where I make the configuration changes.
If you have problems, Csync2 is not very verbose by default. There are two things that you can do. You can execute “csync2 -xr” with -vvv and you should see something useful. If not, you can execute the service in the failing server in foreground. First, stop inetd (Csync2 runs under Inetd by default in Debian/Ubuntu) and then execute “csync2 -ii -vvv”. Now try the sync. again.

Post to Twitter