Archive for March, 2013


Yesterday, I tried logging in to my Raspberry Pi server from my phone using ConnectBot, an Android SSH client. It was disconnected immediately. Something is wrong. When being back at home, I tried logging in from my desktop, with the same problem. Pre-authentication disconnect.

It appears, when being booted headless, the Raspberry Pi enables a console at its composite port. Being located behind my tv, it’s easy to plug in a cable. systemctl status sshd indicated an error state, and it refused to start again. It would immediately get killed by a sigbus. A sigbus??? wtf?

The logfiles show the following error:

Mar  9 17:55:19 rpi-server sshd[21742]: Inconsistency detected by ld.so: dl-version.c: 224: _dl_check_map_versions: Assertion `needed != ((void *)0)' failed!

I assumed some upgrade I ran earlier upgraded some libraries, conflicting with some in-momory code, therefore I decided to reboot the Pi.

It comes up with the Raspberry logo, saying welcome to ArchLinux ARM, but then it just hangs. Nothing else happens. Nothing.

rpi_noboot

To verify if it’s the Pi, I decided to boot it up using the SD card from my other Pi. Since it boots fine, there is something wrong with the content of the rpi-server’s SD card, which seems to be confirmed by

Mar  9 17:55:14 rpi-server kernel: [3514985.125660] EXT4-fs error (device mmcblk0p2): __ext4_ext_check_block:475: inode #699: comm systemctl: bad header/extent: inval
id extent entries - magic f30a, entries 223, max 340(340), depth 0(0)

I have made an image if the SD card’s current state. I can restore the SD card with an earlier image I’ve made when installing rpi-server. I think I have changed some things since I’ve made that image, but the basic functionality, such as NFS, CUPS and SANE should be on there. (As I’ve mentioned I was going to make this backup in a post mentioning sane.

So, this seems to be a case of file system corruption. The question remains what caused this corrupt file system? Is there something wrong with the Pi? the SD card? Is there a bug in the kernel used by ArchLinuxARM? Was the system possibly hacked?

I ran fsck -y /dev/mmcblk0p2 on the SD card. I have a backup anyways, so the -y won’t hurt. I have mentioned, even though I marked it as “should be checked on boot” in my fstab, it doesn’t appear to be doing so for the rootfs. (It checks the bootfs during boot) But since this system is supposed to be always up, a check during boot shouldn’t make much difference. And in principe, the file system should not get corrupted. So…. why did it become corrupted?

Anyhow, getting multiple claimed blocks. Therefore I say it’s no use to continue, I should just write the old image back and that be it…. thinking about what I’ve installed afterwards… one of the things would be apache I guess…


My laptop’s SD slot stopped working due overheating. The samn /etc/cpufreq-bench.conf was set to governor ondemand. I have set it to powersave a dozen times, but it keeps popping back to ondemand, probably during the installation of updates. I don’t know why, but I need to clock down my laptop to the lowest possible speed or else it will overheat. The SD slot is the first thing to give issues when this happens.


Running on the image I put back, and performed an upgrade, I see these messages in my logs again:

[ 1417.529973] EXT4-fs error (device mmcblk0p2): ext4_ext_check_inode:462: inode #55228: comm pacman: bad header/extent: invalid magic - magic f34a, entries 1, max 4(0), depth 0(0)

I did the upgrade immediately and rebooted, so it is still either the SD card or the current kernel.

Server migration

As I’ve been mentioning before, the content is this server (ks26301.kimsufi.com) will be migrated to a new server (ks3291437.kimsufi.com). These are dedicated servers from http://www.isgenoeg.nl. The ks26301 server, which I have been using since April 2009, is their 2008 model. Back in 2009, their services got introduced in the Netherlands, and the first 1000 subscribers got a year free. I was one of the lucky.

Anyhow, this is a server from 2008, and the price hasn’t changed (apart from the taxes, that is). The point is, for the same price they offer much better specs. So, it makes sense to migrate. Also, over the past years I am hosting services for certain people, which makes the configuration I’ve been using since 2011 less optimal. I never enticipated the fact I would be offering hosting services to third parties, so even more reason to migrate my services to a new server.

At this point, I would like to highlight one of the issues that arrises during such a migration, and provide a solution for it. The problem is the way DNS works. When I change my DNS entries, it takes a while to propagate through the internet. The old IP address might be cached at some DNS server and so on. Therefore, during the migration, requests may arrive at both the old and the new server. So, how to make this situation transparant to the user?

First, let’s have a look at Apache. We’re going to use the mod_proxy for this purpose. I had this module already installed on my system, therefore, in my /etc/httpd/conf/httpd.conf I have

LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_connect_module modules/mod_proxy_connect.so
LoadModule proxy_ftp_module modules/mod_proxy_ftp.so
LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule proxy_scgi_module modules/mod_proxy_scgi.so
LoadModule proxy_ajp_module modules/mod_proxy_ajp.so
LoadModule proxy_balancer_module modules/mod_proxy_balancer.so

I might check later on a different (non-production) server which of these are actually required for this purpose.

Anyhow, in my /etc/httpd/conf/extra/httpd-vhosts.conf, I use the following to proxy the connection to the new server. Please note my old server resolved the domain.tld to the new server. Just to be sure it won’t get caught up in a loop, I might add it to /etc/hosts as well. (I’ve kept the DocumentRoot in there, but it has no real purpose anymore. Merely a fallback in case the mod_proxy isn’t loaded)

<VirtualHost *:80>
  ServerAdmin webmaster@domain.tld
  DocumentRoot "/path/to/documentroot"
  ServerName domain.tld
  ServerAlias www.domain.tld
  ErrorLog "/var/log/httpd/domain.tld-error_log"
  CustomLog "/var/log/httpd/domain.tld-access_log" combined

  <IfModule mod_proxy.c>
    <Proxy *>
      Order deny,allow
      Allow from all
    </Proxy>
    ProxyRequests off
    ProxyPassInterpolateEnv On
    ProxyPass / http://www.domain.tld/ interpolate
  </IfModule>

</VirtualHost>

Next issue, is incoming mail. For now, I start at migrating the websites, and later I will migrate the mail. But as a proof-of-conecpt, I have tested this for one domain which only has a catch-all-forward.

The ks26301 runs exim as smtp server, well, basically, we’re going to tell it to forward mail for the specific domain to the ks3291437 server.

Just below begin routers in the /etc/mail/exim.conf file, we add

send_to_new_server:
  driver = manualroute
  domains = "domain.tld"
  transport = remote_smtp
  route_list = * ks3291437.kimsufi.com

These configurations should make the transition to the new server transparant to the end-user.