Tag Archive: server

Oh my…. I haven’t written at all in this blog this year…
Anyhow…. I will report about a number of technical problems…
They always seem to come at the same time, don’t they?

Let’s see, my raspberry pi server, down again. So… what I did in the past, I moved the root file system to an external hard disk, since I was suffering from file system corruption on the SD card. This week, file system corruption also occured on that external hard disk. Last week, the raspberry pi crashed, and when I rebooted it didn’t come up. It couldn’t find the root file system. Powercycling the hard disk containing the root file system fixed the problem. Seeing the problem I had last week, I blaim faulty hardware for the problems I’ve been experiencing. Now, I come to a point where I suspect the SD card I used before to be faulty as well. I am thinking ways to test that SD card. Perhaps I could create a script that writes random data to random positions, both on the SD card and in an image file, and comparing both in the end.

My dedicated server, which hosts this blog, also encountered a problem. A problem simular to a problem I’ve been experiencing with my previous server. It ran out of memory and killed amavis, virtually disabling the email subsystem. Now, I have configured the OOM to be less likely to kill amavisd-new or clamd, to be less likely to be killed in the event the system runs out of memory. For now, only a single instance. Let’s have a look at the /proc filesystem. In there we see a directory per running process.

Say, the pid of clamd is 12345, then we can set the killing likelyness from -16 (unlikely) to 15 (very likely) like this:

# pidof clamd
# echo -15 > /proc/12345/oom_adj

Well… having that said, the issue still is, the system ran out of memory. Now, I have to say, this machine was pre-installed by OVH using their default ISPConfig configuration. The machine has 3 GB of RAM installed, and was configured with 1 GB swap. Now, the rule of thumb is to have twice as much swap then ram. So, I guess that’s the cause of the problem. Now, I would like to increase the swap, but I cannot repartition a running system. And even if that wasn’t an issue, the system is running a software raid.

# blkid
/dev/sda3: UUID="8199facf-617b-475a-9874-79f017193cce" TYPE="swap" 
/dev/sdb3: UUID="366beea7-8e2f-457b-b184-78c811df7d07" TYPE="swap" 
/dev/sda1: UUID="9cf67138-a39d-3a3f-a4d2-adc226fd5302" TYPE="linux_raid_member" 
/dev/sda2: UUID="f8058e35-2dec-e859-a4d2-adc226fd5302" TYPE="linux_raid_member" 
/dev/sdb1: UUID="9cf67138-a39d-3a3f-a4d2-adc226fd5302" TYPE="linux_raid_member" 
/dev/sdb2: UUID="f8058e35-2dec-e859-a4d2-adc226fd5302" TYPE="linux_raid_member" 
/dev/md2: LABEL="/var" UUID="f8893706-4d3c-45aa-af84-b302221f3bd7" TYPE="ext4" 
/dev/md1: LABEL="/" UUID="4903b0b6-3e50-4fc9-bb0b-1ab3e6c505b7" TYPE="ext4" 

So, I guess I’ll solve it by creating a swap file instead. The system is paritioned to have a small root file system, and a huge /var file system. Therefore I’ll create the swap file in /var

# fallocate -l 6G /var/swapfile
# chmod 600 /var/swapfile 
# mkswap /var/swapfile 
mkswap: /var/swapfile: warning: don't erase bootbits sectors
        on whole disk. Use -f to force.
Setting up swapspace version 1, size = 6291452 KiB
no label, UUID=74989c54-dc14-4b65-8577-13b356ca79c8
# swapon /var/swapfile 

And add the swap file to the fstab.

Checking the swap status:

# swapon -s
Filename				Type		Size	Used	Priority
/dev/sda3                               partition	525308	254816	-1
/dev/sdb3                               partition	525308	16488	-2
/var/swapfile                           file		6291452	0	-3

Well… I guess this solves the memory problem. I might have a closer look at some logs, but so far I haven’t seen any task using suspecious amount of resources. I guess 3 GB of RAM + 1 GB of swap just wasn’t enough.

As I’ve been telling before, due the increased number of sites on my server, I’m hitting its limits. Mainly the fact the server got only 1 GB of RAM is putting limitations. Recently the server was acting up again, and I assumed it was caused by resource limits, therefore I attempted to tune the configuration a little more to use less resources, without any luck.

I started to notice this problem showed different characteristics the problems I’ve been experiencing before. Mainly because it seemed also to influence the mail system rather then just the webserver. The problem seemed to be linked to the mysql server. Restarting the server didn’t work as it should. Defenately something with mysql…. So I decided to start mysql in a console to look and I saw

mysqld: Disk is full writing './mysql-bin.~rec~' (Errcode: 28). Waiting for someone to free space... (Expect up to 60 secs delay for server to continue after freeing disk space)

The problem: the root file system was full: In all the time the server was running, of course I have frequently been upgrading the system. During all this time, packages were downloaded to /var/cache/pacman/pkg and never removed. So… it was eating a couple of gigabytes…. and the root file system is just a small filesystem, just for the OS. The data is elsewhere.

Another mystery solved.