Latest Entries »

When looking at various tutorials on configuring postfix, it’s quite easy to configure a catch all for a certain domain. So any username at the said domain gets forwarded so a said maibox. However, I haven’t been able to find any article describing to do the reverse, catch a said username on any domain.

The reason for this is to catch abuse@, postmaster@, webmaster@ etc. any domain on my servers. When going through the postfix documentation another time I realised it support regular expressions. Using regular expressions this can be fixed

put a file /etc/postfix/regexp-aliases

^webmaster@/i  destination@email.address
^postmaster@/i destination@email.address
^abuse@/i      destination@email.address

Then open /etc/postfix/main.cf and add

regexp:/etc/postfix/regexp-aliases

to virtual_alias_maps = .

Edit 05-03-2016: added the /i to the example. Email user names are supposed to be case insensitive.

For quite some time, I haven’t been able to use wireless communication on my HP EliteBook 8530w. All wireless communications (wifi, wwan, bluetooth) are listed as Hard Blocked by rfkill.

It appears, a kernel bug was introduces with Linux 3.13. This bug is in the hp-wmi kernel module, which controls the hardware buttons, including the wireless switch

My current kernel is 4.3.3, so,we obtain the source file for the hp-wmi for this kernel : https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/plain/drivers/platform/x86/hp-wmi.c?id=refs/tags/v4.3.3
We obtain the patch file https://bugzilla.kernel.org/attachment.cgi?id=122901

We apply the patch to the file

patch -p 4 < hp_wmi.patch

We put a Makefile with this file in it

obj-$(CONFIG_HP_WMI)		+= hp-wmi.o

And we comile it

make -C /lib/modules/`uname -r`/build M=$PWD

Backup the original kernel module

mv /usr/lib/modules/`uname -r`/kernel/drivers/platform/x86/hp-wmi.ko.gz /usr/lib/modules/`uname -r`/kernel/drivers/platform/x86/hp-wmi.ko.gz.bak

And put the patched version in its place.

mv ./hp-wmi.ko.gz /usr/lib/modules/`uname -r`/kernel/drivers/platform/x86/hp-wmi.ko.gz

When we press the wireless button now, rfkill reports Hard Blocked: no and we're able to use WiFi, Bluetooth and 3G again.

As the year is coming to an end it’s time to look back. Usually I would have prepared a post to post today. However, last week has been rather turbulent. So due recent events, I will write my post about this year next year.

I wish…
I wish those who are in despair will find new hope.
I wish those who are depressed will find new happiness.
I wish those who are filled with hate find the courage to let it go, learn to love again.
I wish… I wish… for a world in which everyone can be happy, free and independent.
I wish for a world of peace and understanding, a world where people can be themselves.

I suppose I better write about this now, as I encountered these errors a while ago, and the details already started slipping my mind. So, what’s happening? I am writing some PHP code. As a testing server, I’m using my ODROID U3. An ARMv7 board, running ArchLinuxARM. Therefore, we have a 32 bit machine running PHP 5.6. Now, the (soon-to-be) production server is an x86_64 machine, running Ubuntu 14.04LTS. So, this is a 64 bit machine running PHP 5.5.

Let’s have a look at the mcrypt_create_iv() function. This function produces random data. Now, there is a change introduced in PHP 5.6.0. First, let’s have a look on how random data works on a Linux platform. (Other *NIX platform might differ from this)

/dev/random Gives random data, and blocks until there is enough entropy available.
/dev/urandom Gives random data, but never blocks. This means possible less entropy in the result.

Basically, we’re interested in the most possible entropy when we’re doing crypto, then we use /dev/random. That’s why, for example, generating PGP keys takes so long. It’s just waiting for enough entropy to become available. When we need some randomness in a setting that doesn’t require crypto-grade entropy, /dev/urandom is sufficient.

Now, back to PHP. We have the mcrypt_create_iv() function to generate random data. Now, what happened. Since PHP 5.6.0 the default source for randomness is /dev/urandom. In prior versions it was /dev/random. So…. what happens? Code that runs fine on the development machine hangs on the production server, as it’s waiting for entropy. That’s not what we want to happen on a website. So, when using the mcrypt_create_iv() function, always specify the entropy source as the default differs per PHP version.

Well… the difference between 32 and 64 bit PHP versions, it’s all in the size of the int, or rather, using the PHP_INT_MAX for purposes it wasn’t intended for. So, we’re looking at setting a cookie. The idea is to have a “stay logged in” option, so we set the cookie to a date/time way in the future. Considering the Year 2038 problem, the INT_MAX for 32 bit, results in 19 January 2038. However, this is far enough in the future to be used as a “remember me” function. As systems turn 64 bit, this problem is solved, and why not setting the date even to a later date? It just needs to be way ahead in the future to simulate a “never expire” cookie. Well… let’s have a closer look on how cookies work. In PHP, the setcookie() takes a unix timestamp, so, it’s just an INT value containing the number of seconds since the epoch (1 Jan 1970, 00:00 GMT). So far so good, so, if we give it the maximal int value for a 64 bit int….. we end up 293 billion years in the future. (The English Language uses the Short Scale for Numbers, where Continental European Languages uses the Long Scale, note this is “293 miljard” in Dutch. See “Names of Large Numbers” on Wikipedia) So, we’re setting a cookie to a date probably beyond the end of the universe. So, basically, it will never expire. That’s the intention, so…. what went wrong?

Setting a cookie to a date where it will survive the end of the universe causes issues with the Year 10000 problem. The problem is, the server doesn’t just sent that number to the client, it sends a formatted string, it’s format is Wdy, DD Mon YYYY HH:MM:SS GMT Spot the problem? The year is encoded as YYYY, meaning 9999 is last year it supports. So…. setting a cookie to the end of time is not an option.

I don’t know how many 32 bit clients are still out there, but considering the fact they might suffer from the year 2038 problem, I suppose it’s safer to use 2038 as the the-end-of-time value.

Starting today, I am self employed. My company name is The IT Philosopher. My company website is http://www.philosopher.it. Today is the day I am officially registered at the Kamer van Koophandel (Chamber of Commerce). But of cource, I have been preparing for this event for quite some time. One could say I’ve spend all year in preparation for this event.

As part of such, lately I have been working on my web presence. In an earlier stage of my life, I was eager to ensure everything I’ve ever blogged remained online, and imported all of my blogging elsewhere on the internet into my own server. That is this blog. Ten years of blogging, that’s a lot of Blaat.

Over the years my blogging style has changed, where I shifted into more of a tech-blog. My plans are to export the tech-related posts to a different site (http://www.andrevanschoubroeck.name. I suppose I could put up a redirect here.

I suppose this also means, my hosting activities, I’ve executed as a hobby until now, will be shifted to a professional level. As The IT Philosopher, I’ve got new servers, which are still waiting to be configured. Busy busy.

The past days my server has suffered a DoS attack. The attack consists of flooding HTTP requests, to a single PHP script, part of a WordPress installation.

For now, the problem appears to be under control. However, as the attack is still ongoing, I will not reveal any more details.

This morning, I got an email from one of the people I’m hosting a website for. They’re unable to create a newsletter.
So, their website is running WordPress, and they use a plugin to send newsletters. When trying to create a new newsletter, all it said was “Database error”.

I’ve located the failing code in includes/store.php. The failing code is

  $r = $wpdb->insert($table, $data);
  if ($r === false) {
    $this->logger->fatal($wpdb->last_error);
    die('Database error.');
  }

There was no $wpdb->last_error set, so it’s not quite obvious what was causing the error. To find out what was going on, I had to dive into the WordPress $wpdb class. The insert() method calls the _insert_replace_helper() method. In this method, a call to the process_field() method is made. This call failed. To be specific, the failing code:

  $converted_data = $this->strip_invalid_text( $data );
  if ( $data !== $converted_data ) {
    return false;
  }

The strip_invalid_text() method was added in WordPress 4.2.0. This method strips invalid characters for a specified character set. So… this check was never performed with earlier versions of WordPress…. so… let’s have a look at the cause of the trouble.

The cause of the trouble, the newsletter template, or, more specifically, the template for the plain text version of this template. Rather then the default This email requires a modern e-mail reader. You can view the email online here, I’ve created a plain text output for this template. Part of this template was the following function:

function tse_newsletter_formattext($text){
  $text = strip_tags($text);
  $text = html_entity_decode($text);
  $text = preg_replace('/&#(\d+);/me',"chr(\\1)",$text); #decimal notation
  $text = preg_replace('/&#x([a-f0-9]+);/mei',"chr(0x\\1)",$text);  #hex notation
  $text = str_replace('…','...',$text); 
  $text = wordwrap($text,60,"\r\n",true);
  return $text;
}

And this little function was the cause of the trouble. More specifically, the output of html_entitity_decode(), produced from the input &nbsp;. Character set issues, indeed. When called like this, running on PHP 5.3.3, it returns the decoded entities in ISO-8859-1 encoding. Specifically for &nbsp; this would result in 0xA0. The WordPress Database class however, was expecting UTF-8 encoding, and therefore considering this being invalid data. The decoding of html entities specifying codepoints (preg_replace lines) are probably also causing issues as they assume single-byte encodings.

A friend of mine asked me to install linux on his Asus EeePC 1201HA. So, I am installing Xubuntu 14.04 LTS on his laptop. This laptop has some audio and video issues when running a default Linux installation. In this post, I will discuss the audio and video setup.

This laptop uses an integrated Intel graphics controller. It’s known as GMA500 or Poulsbo.

$ lspci | grep VGA
00:02.0 VGA compatible controller: Intel Corporation System Controller Hub (SCH Poulsbo) Graphics Controller (rev 07)

By default, the display uses the gma500_gfx module. This module has no hardware acceleration, which causes videos to stutter. So, this is not a workable configuration.

Therefore, we will configure the Intel® Embedded Media and Graphics Driver driver. This driver is not available in an official Ubuntu repository and must be installed manually.

sudo add-apt-repository ppa:thopiekar/emgd
sudo apt-get update
sudo apt-get install emgd-drm-dkms xserver-xorg-1.9-video-emgd

Note that we are installing a DKMS package. This means, a kernel module will be compiled. This causes the installation to take quite a while.

Once installed, a little configuration is required. First, we must disable the default kernel module. Create a file /etc/modprobe.d/blacklist-gma500.conf with the following content:

#use the EMGD driver instead
blacklist gma500_gfx

Then we must configure X. Create a file /etc/X11/xorg.conf with the following content:

Section "Device"
    Identifier "Intel EMGD"
    Driver     "emgd"
EndSection

Section "Screen"
	Identifier "Screen0"
	Device "Intel EMGD"
	SubSection "Display"
		Modes "1366x768"
	EndSubSection
EndSection

The video is not the only thing that stuters on this laptop. There are audio problems as well. However the solution for this issue is much simpler then the solution for the video problem. Just changing one line. A solution for this problem is found is described on the Debian wiki for a different model in the EeePC series. Edit the file /etc/pulse/default.pa and add tsched=0 to the line saying load-module module-udev-detect, so in the end, it reads:

load-module module-udev-detect tsched=0

And that’s it… the laptop works as a charm!

I am setting up a web shop using Drupal and Ubercart. As shipping module I am using Ubercart Global Quote. This module allows me to set up shipping zones. For each shipping zone I can configure a shipping rate based on the weight of the products.

Shipping zones for PostNL are, the Netherlands, Europe 1, Europe 2, Europe 3 and World. This means, if a country is not the Netherlands or listed in one of the Europe zones, the World rate applies. The problem is, to create a shipping zone I have to select all the countries in this shipping zone. There is no “unlisted countries” zone.

I have encountered similar problems with other e-commerce software. In this post I will describe what I have done to fix it in Ubercart. Technically spoken this is not a fix. I have not touched any Ubercart code, but I have created a shipping zone with the “unlisted countries” by code. The following code is supposed to be executed after configuring the known shipping zones.

First, we obtain the already configured shipping zones. Shipping zones are stored as a string in the database, in the following format: country_name,country_id, separated by a pipe sign. The country id is the numerical ISO country code.

$country_array = array();
$query1 = "select countries from uc_shipping_zones";
$countriylists = $pdo->query($query1);

foreach ($countriylists as $countrylist) {
  $crt = explode("|",$countrylist['countries']);
  $country_array = array_merge($country_array,$crt);
}

After running this code, we have all countries (in country_name,country_id format) for all configured shipping zones, stored in an array. At this point I would like to elaborate on how countries are stored in Ubercart. Ubercart has a table with country names and their ISO codes, which includes the numerical value.

What we want to obtain is a list of countries not included in the country list we have obtained in the previous step. We will solve this problem is SQL. First, we extract the numerical ids from the array created in previous step. Then we generate a query that selects all countries but the ones we have in our list. For this we generate a subquery that consist of literal selects and unions them. Basically, we generate a table with the country values already listed.

$first = true;
foreach ($country_array as $country) {
  $crt=explode(",",$country);
  if (!$first) {
    $query2_part2 .= " UNION select ";
  } else {
    $first = false;
    $query2_part2 .= "select ";
  }
  // Some country names may include a comma
  // "Moldovoa, Republic of"
  $query2_part2 .= "(" . $crt[sizeof($crt)-1] .")";
}

As some country names in the Ubercart country list contain a comma, we need to select the last value of our array, to make sure we have the number. Then we generate a select and union of all our numbers. Then we include this subquery in a query that will select the country list.

$query2 = "select concat_ws(',',country_name, country_id) from uc_countries where country_id not in ($query2_part2 )";

// for non-ascii country names:
// we get    "c3856c616e642049736c616e64732c323438"
// should be "Åland Islands,248"
// so we should do the concat in php in stead of mariadb

$query2 = "select country_name, country_id from uc_countries where country_id not in ($query2_part2 )";
$othercountries = $pdo->query($query2);

MariaDB can perform string manipulations. We need to obtain a string country_name,country_id, so the way to do this in MariaDB would be concat_ws(',',country_name, country_id. However, this gives incorrect results if a country name contains non-ascii characters. Therefore, this concatenation is done in PHP rather then in MariaDB.

$first = true;
foreach ($othercountries as $othercountry) {
  if (!$first) {
    $otherstring .= "|";
  } else {
    $first = false;
  }
  $otherstring .= $othercountry['country_name'] . "," . $othercountry['country_id'] ;
}

This code takes the result of the previous query, an array of all the country names and id’s unlisted. This list is put together is the desired format, country_name,country_id separated by a pipe sign.

Finally we enter this string into the database

$pq = $pdo->prepare("INSERT INTO uc_shipping_zones (name,countries,regions) VALUES (:name, :countries, :regions )");
$pq->execute(array(":name" => "Other countries", ":countries" => $otherstring, ":regions" => ""));

Now, the shipping rates can be configured to ship to any country.

This might not be the optimal solution, but I suppose this is a way to do it without touching the Ubercart Global Quote code.