Blog

We wanted to use EncFS to be able to store encrypted backups. The requirements for that are: The backup server initiates the backup. That’s where we configure which hours are safe (resource wise) and which files need backing up (etc, home, root, srv, …). And it means the backup server can safely be placed behind a gateway disallowing all incoming connections. The backup server cannot know the passwords of files.

I ran into an obscure Could not link against ! error when configuring EncFS: ~/src$ apt-get source encfs ... ~/src$ cd encfs-1.7.4/ ~/src/encfs-1.7.4$ ./configure ... configure: WARNING: BOOST_CPPFLAGS -I/usr/include checking whether the Boost::Serialization library is available... yes configure: error: Could not link against ! That’s odd. And not immediately obvious how to fix. For starters we need all the dependencies that Debian defines: ~/src/encfs-1.7.4$ sed -e '/^Build-Depends: /!d;s/^[^:]*: //;s/([^)]*)//g;s/,//g' \ debian/control debhelper librlog-dev librlog5 libfuse-dev libssl-dev pkg-config libboost-serialization-dev libboost-filesystem-dev quilt dh-autoreconf ~/src/encfs-1.

Service disruptions on one of the core routers location (CR1) in TCN. 5 Nov 2015 - 01:00 CR1 Supervisor placement Following up the maintenance finished on Monday, we will add an extra Router Supervisor in CR1 for extended redundancy in case of main Supervisor failure. This maintenance will be carried out tonight, at 01:00 on the 5th of November. There is no expected impact. This maintenance is a followup on the hardware replacement of the Supervisor in CR1.

In the night of Thursday (Nov. 5th) to Friday (Nov. 6th) between 1:00 and 3:00 we will perform network maintenance at our TCN co-location. ####Maintenance window 01:00-03:00 on 6th of November 2015 ####Description We will perform changes in the network configuration which will cause a change of mac address of the gateway for each subnet. Impact Depending on the device it may take some time to pick up this change. Generally, busy servers will pick up the changes almost instantly and servers which are mostly idle may take a while.

A few days ago, the Scapy project was brought to my attention. Scapy is an internet packet manipulation library for Python2. It can be used to sniff and decode packets, or to generate your own custom packets. In the most basic form, it runs on raw sockets, sniffing and decoding traffic like tcpdump. See the sniff() examples and the send(IP(dst="1.2.3.4") / ICMP()) example for sending a simple packet. But just as easily, it works on regular datagram sockets — those that you don’t need CAP_NET_RAW powers for.

To syntax check Python code before executing, I use flake8. And when coding in the Vim editor, I use the vim-flake8 plugin that allows me to hit <F7> to quickly check for errors in the file I’m currently working in. But, there are currently two common flavors of Python: python2 and python3. And therefore flake8 comes in two flavors as well — you guessed it — a python2 and a python3 flavor.

While I was writing a Python tool to wrap C Gdb so I could fetch some info out of it automatically, I ran into the issue that it reads the terminal size (lines x columns) to adjust its output. I wanted consistent machine readable output, so I enlarged the terminal size programmatically: now row based output would not get wrapped by Gdb. Later I noticed that it would cease to use the terminal size — in fact, use the default 80 columns — if I also redirected stderr to a non-tty.

As of this writing, Debian testing (stretch) contains Asterisk version 13.1.0. The Debian source as GIT repository is here: https://anonscm.debian.org/git/pkg-voip/asterisk.git (browse) Packaging a newer version is not that hard, if we start out with the debian/ directory kindly supplied by the Debian maintainers. Hints to get things running: Use a local git repository By using a local git repository in your unpacked Asterisk dir, you can quickly restart from scratch any time you mess anything up.

I was wondering how easy it was to encrypt files before rsyncing them away to the backup machine. A quick search turned up the suggestion to use encfs by the user Thor on ServerFault. That looks like a decent solution. Let’s figure out if it meets our needs. The idea is that we do this: # mount read-only encrypted virtual copy of unencrypted local data: encfs --reverse -o ro ~/data/ ~/.

Here, an awesome shell one-liner to find which process uses the most files, relative to its max-open-files soft limit. $ for x in /proc/[0-9]* do fds=0 max=`awk '/^Max open files/ {print $4}' $x/limits 2>/dev/null` && for t in $x/fd/*; do fds=$((fds+1)); done && test "${max:-0}" -gt 0 && echo $((fds*100/max)) ${x##*/} done | sort -rn | while read l do pid=${l##* }; echo "$l`readlink /proc/$pid/exe`"; break; done 57 16674 /usr/lib/dovecot/imap-login So, my imap-login (pid 16674) apparently uses 57% percent of its allowed max open files.

Okay, so we all know that printers are sent from hell, but we still need to use them from time to time. Today, we were trying to print a PDF document with bar codes on it. Amazingly enough, the text on the PDF looked fine, but the bar codes (images) appeared as if they were wrapped at the wrong place. Luckily, convert(1) from ImageMagick came to the rescue: $ convert -density 300 -define pdf:fit-page=A4 input.

As I mentioned the other day, my VM was slow, so I needed a way to figure out which VM guests were causing the heavy load on our Proxmox platform. I hacked up proxtop to enumerate the top resource users: $ ./proxtop -t day proxmox.example.com monitor@pve Password:<enter password> SORTED BY: cpu, avg ... SORTED BY: diskread, avg ------------------ #0: 3.1 MiB/s pve10 (acme-bugs-bunny) #1: 1.3 MiB/s pve07 (customerX-private) #2: 992.3 KiB/s pve10 (acme-road-runner) .

So, my VM was slow, and I needed to know which VM guest was eating all the resources. These VM containers are all managed by Proxmox; which is great, but it doesn’t show which VM guest is eating all the resources. Luckily, Proxmox provides an API to get that info. The docs pointed to two API modules for Python, my language of choice for these kinds of jobs: proxmoxer and pyproxmox.

Today, my choice of Python modules to Interface with Zabbix. They are all pretty similar, so that made it harder to choose. Here the six modules, as mentioned on the Zabbix wiki are, in the order of my preference. Note that second and third came close, but I favor clean documented code and fewer dependencies. The last ones didn’t get tested because of my Python3 requirement. zabbix-client # pip: zabbix-client # pep: 99% # last-update: Aug.

Even after writing plenty of Asterisk PBX dialplan, I occasionally get bitten by the unintuitiveness of the parser. A few rules to avoid mistakes, are: Always use double quotes on no side of the expression, or better yet, on both if there is a chance that the value is empty: $[${HANGUPCAUSE}=17] or $["${value_which_may_be_empty}"="somevalue"] Otherwise try to avoid double quotes (and semi-colons, and backslashes) whenever possible. If you need to escape them, it’s too easy to get it wrong.

A high risk security issue in glibc was disclosed last night. Because of the potential high impact we started our emergency patch procedures for osso managed environments and notify customers with self managed environments. Ghost vulnerability details Qualys discovered a buffer overflow in dns resolve functions in the GNU C library (glibc). They created a proof of concept exploit for exim and dubbed the vulnerability "GHOST". All processes that might do dns lookups are susceptible to attacks when using a vulnerable glibc version.

While we do Python VirtualEnv stuff every day, we rarely do Ruby environments. And after the Ubuntu dist-upgrade, the Ruby dependencies for our GitLab were broken — as was expected. This happens for Python pip installed packages as well. They’re linked against older system libraries, which have been removed by the upgrade. How to fix the Gitlab dependencies? Browse through the upgrade docs to find a bundle install command. # cd /home/git/gitlab # sudo -u git -H bundle install \ --without development test postgres --deployment # for MySQL That did… absolutely nothing — again, as was expected.

Tired of the Fail2ban start and stop e-mails? Especially after a manual fail2ban restart, the [Fail2Ban] vsftpd: stopped on HOSTNAME and [Fail2Ban] vsftpd: started on HOSTNAME mail tuple is too spammy. Quick fix to disable them: Create a new file, named /etc/fail2ban/actions.d/sendmail-no-start-stop.local: diff --git /etc/fail2ban/action.d/sendmail-no-start-stop.local /etc/fail2ban/action.d/sendmail-no-start-stop.local new file mode 100644 index 0000000..cb7ecb9 --- /dev/null +++ /etc/fail2ban/action.d/sendmail-no-start-stop.local @@ -0,0 +1,3 @@ +[Definition] +actionstart = +actionstop = And — you’re using mta = sendmail right?

When trying to keep up with all the TLS/SSL security changes, you need to modify your nginx config every now and then. The good TLS config may look like this: # nginx.conf: http { ssl_certificate /etc/ssl/MY_DOMAIN.pem; ssl_certificate_key /etc/ssl/MY_DOMAIN.key; ssl_protocols TLSv1 TLSv1.1 TLSv1.2; ssl_ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA; ssl_session_cache shared:SSL:5m; ssl_session_timeout 5m; ssl_prefer_server_ciphers on; add_header Strict-Transport-Security "max-age=15768000; includeSubDomains"; And the above config is accompanied by a fairly good A grade from the Qualys SSL Labs Analyzer.

Storing an UUID in MySQL efficiently: DROP FUNCTION IF EXISTS uuidbin; CREATE FUNCTION uuidbin(uuid_val varchar(36)) RETURNS varbinary(16) DETERMINISTIC SQL SECURITY INVOKER RETURN CONCAT(UNHEX(LEFT(uuid_val,8)),UNHEX(MID(uuid_val,10,4)), UNHEX(MID(uuid_val,15,4)),UNHEX(MID(uuid_val,20,4)), UNHEX(RIGHT(uuid_val,12))); DROP FUNCTION IF EXISTS uuidstr; CREATE FUNCTION uuidstr(uuid_val varbinary(16)) RETURNS varchar(36) DETERMINISTIC SQL SECURITY INVOKER RETURN LOWER(CONCAT(HEX(LEFT(uuid_val,4)),'-',HEX(MID(uuid_val,5,2)), '-',HEX(MID(uuid_val,7,2)),'-',HEX(MID(uuid_val,9,2)), '-',HEX(RIGHT(uuid_val,6)))); Now you can create your uuid columns with type binary(16). And conversion is easy: mysql> select uuidstr(uuidbin(uuidstr(uuidbin(uuidstr(uuidbin( 'a89e6d7b-f2ec-11e3-bcfb-5c514fe65f2f')))))) as uuid_back_and_forth; +--------------------------------------+ | uuid_back_and_forth | +--------------------------------------+ | a89e6d7b-f2ec-11e3-bcfb-5c514fe65f2f | +--------------------------------------+

Django makemessages can be quite slow on larger projects. $ time python ../manage.py makemessages -lnl -ddjango processing language nl real 0m8.203s user 0m2.670s sys 0m5.763s Why does it take so long? Well, it’s system call heaven: $ strace -f python ../manage.py makemessages -lnl -ddjango \ >tmp.log 2>&1 $ sed -e 's/(.*//;s/^\[[^]]*\] //;/^ \?</d;/,/d;/^+/d' tmp.log | sort | uniq -c | sort -n | tail -n10 10893 rt_sigaction 16179 stat 16819 fcntl 22875 access 27833 read 32469 open 33650 fstat 40891 mprotect 69181 mmap 1267039 close For every file, a call to xgettext(1) is made.

Sometimes, after a stray copy operation, your filesystem times may reflect the time the files were copied instead of when the file was actually last altered. For example this image folder here: $ ls -l phone2013 total 320856 -rw-rw-r-- 1 walter walter 1524591 nov 17 21:52 2012-10-28 08.54.58.jpg -rw-rw-r-- 1 walter walter 1534840 nov 17 21:52 2012-10-28 08.55.04.jpg -rw-rw-r-- 1 walter walter 1635908 nov 17 21:52 2012-10-28 08.55.09.jpg ... -rw-rw-r-- 1 walter walter 1600504 nov 17 21:52 2013-10-22 11.

So, I was really simply trying to figure out why talking to my OpenSIPS instance over a datagram unix socket failed. If I had bothered to check the server logs, I would immediately have seen that it was a simple stupid permission issue. Instead, I ended up reimplementing recvfrom and sendto in Python using the ctypes library. Which was completely useless, since Python socket.recvfrom and socket.sendto already work properly. To let the time spent on that not go to a complete waste, I give you (and myself) an example of ctypes usage.

Syslog generally works fine as it is, so I don’t need to poke around in it often. That also means that I forget how to tweak it. How did you move those every-5-minutes cron jobs out of /var/log/syslog? The rules (selection + action) look like this in the Debian default config: *.*;auth,authpriv.none -/var/log/syslog #cron.* /var/log/cron.log The manual has this to say about it: You can specify multiple facilities with the same priority pattern in one statement using the comma (,) operator.

One of our datacenter locations (TCN Telehouse) will have major maintenance on its power infrastructure this month. They scheduled 4 maintenance windows of 1,5 hours each during which either the A or B feed will be powerless. In the last weeks we’ve double checked our infrastructure and this week we will finish our last preparations. All equipment that is not equipped with dual power supplies is connected to an ATS (Automatic Transfer Switch) to achieve power redundancy.

While I was battling an obscure Ubuntu shutdown issue — more about that later — I noticed that daemonized jobs started from my X session were not reparented to PID 1 init, but to a custom init --user, owned by me. What? I cannot start daemon that outlives my X session? That’s right, I cannot. Check this out: $ sh -c 'sleep 61 &' $ ps faxu | egrep 'init|sleep 61' root 1 .

Today’s git question: does git reset undo a merge or only parts of it? TL;DR: It undoes the entire merge. If you think about it logically, it must, since an object describes the entire state of the repository. But it can feel awkward and unexpected that older items than the object that we’re resetting to, are removed as well. Let’s just try it. Set up a repository with two branches:

Recently I wrote about cherry picking upgrades. Sometimes you’ll want to do the inverse. For that purpose there exists apt-mark hold (and its counterpart apt-mark unhold). For example, you may to delay the mysql upgrade I mentioned, for now. In that case you do: # apt-mark hold mysql-client-5.5 mysql-common mysql-server-5.5 mysql-server-core-5.5 Now you can apt-get upgrade all the other packages while the mysql packages stay on hold. Note that these are shown in the held list every time you run upgrade, so you won’t forget about them.

SquirrelMail on Debian/Wheezy (2:1.4.23~svn20120406-2) stopped showing (none) for e-mails that lack a subject. Now I cannot open any subject-less mail because there is nothing to click on. The quick fix: --- /usr/share/squirrelmail/functions/mailbox_display.php.orig 2014-08-15 10:37:37.000000000 +0200 +++ /usr/share/squirrelmail/functions/mailbox_display.php 2014-08-15 10:38:27.000000000 +0200 @@ -268,6 +268,9 @@ function printMessageInfo($imapConnectio $title = str_replace('"', "''", $title); $td_str .= " title=\"$title\""; } + if (!$subject) { + $subject = '(none)'; + } $td_str .= ">$flag$subject$flag_end</a>$bold_end"; echo html_tag( 'td', $td_str, 'left', $hlt_color ); break;

Ever needed to restore only one database on a MySQL server and found out you only had one SQL dump containing all databases? Its quite common to dump all databases in one SQL file (mysqldump –all-databases or -A). But when using multiple databases on one MySQL instance you often need to restore just one of them. The minimal effort solution: mysql --one-database desired_db_name < alldatabases.sql fix!

Transcript follows: [him] did I mention I'll be off from work earlier today because I'm having dinner with friends. I'll be off earlier today because I'm having dinner with friends. [me] where did you say you were going? [him] I'll be having dinner at the Grand Cafe Apparently the irony was lost on him. I should’ve used emoticons. But! Instead of emoticons, one may also use the irony punctuation: ⸮

So, doing an apt-get upgrade on a Debian or Ubuntu machine sometimes does more than you want at once. See this upgrade example I encountered just now: # apt-get upgrade ... The following packages will be upgraded: curl dpkg ifupdown iproute libcurl3 libcurl3-gnutls libgnutls26 libmysqlclient18 libsnmp-base libsnmp15 libssl1.0.0 libxml2 linux-firmware linux-generic-lts-quantal mysql-client-5.5 mysql-client-core-5.5 mysql-common mysql-server mysql-server-5.5 mysql-server-core-5.5 openssh-client openssh-server openssl tzdata update-manager-core whoopsie 26 upgraded, 0 newly installed, 0 to remove and 2 not upgraded.

Did you ever wonder what the '<,'> characters mean when you CTRL-V visual block select text in vim? For example: you press CTRL-V and select a bit of text. Then type : (colon). Instead of just the colon, you see: :'<,'>. You append s/^/#/ hit enter. As requested, the selected block is now “commented out”. That’s a nice feature, but why the funny characters? In order to understand that, we remind you of the % (percent sign) that we use to select the entire file.

My .vimrc usually starts out with this. Syntax highlighting is super, and my terminals always have a black background. The modeline option enables me and others to set certain options for certain files only. Like: {# vim: syntax=htmldjango: #} to mark a .html file as using the django html syntax instead of regular html syntax. See also my Inserting vim modelines tip. syn onset bg=darkset modelineSecond, since I develop a lot in Python, I enable the vim-flake8 python source code checker plugin:

I always forget how easy it is to upgrade postgresql on Ubuntu (from 9.1 to 9.3 this time). It seems like a pain to have to manually upgrade the cluster, but when it comes down to it, it’s self-documenting and quick. My shell session basically went like this: $ sudo apt-get install postgresql-9.3 ... The following extra packages will be installed: postgresql-client-9.3 ... $ sudo /etc/init.d/postgresql stop * Stopping PostgreSQL 9.

Recently I tried to open a connection to a remote server over SSH at a new location. The connection opened just fine, but it seemed that a few bytes kept getting buffered. It looked like this first animated gif you see. After a long wait, you realise that the data you’re wating just won’t come. First after pressing a key, you get the data. This isn’t workable… Enumerating the possible culprits, there could really only be the wifi-nat-modem — a Thomson TG789vn, Telia device — doing extra buffering, possibly conflicting with the Nagle algorithm (TCP_NODELAY).

On my recently upgraded Ubuntu Trusty (14.04) machine, git diff started producing colorized output. That’s nice, but it’d be even nicer if it recognised that I’m using a dark background. Put this in your ~/.gitconfig. This colorscheme is the one you’re used to from vim. [color "diff"] meta = green bold frag = yellow bold old = red bold new = cyan bold

Last night an important security vulnerability was made public with corresponding security updates. It risks exposing private keys when vulnerable. OpenSSL was vulnerable starting from their OpenSSL 1.0.1 release on 14th of March 2012 till OpenSSL 1.0.1g released on 7th of April 2014. Two security teams independently reported this issue and it’s safe to assume others did as well. On top of that it’s not possible to trace whether you were successfully exploited.

This morning we experienced packet loss on one of our links. After rerouting the traffic the problem was resolved. Only routes over the GN-IX were affected. timeline 12:15 first notification of the packetloss. 12:36 - 12:40 Hickups in the internet connectivity due to rerouting of traffic. after 12:40 No more issues for our customers. Offhour followup re-enable temporary disabled links and rebalance traffic. Background information The issue occurred because GN-IX hit its capacity limit on their links between Groningen and Amsterdam.

When you’re monitoring security update availability using Zabbix or some other monitoring tool, you’ll need a method to discern regular updates from security updates. I’ve seen my collegues do this: $ /usr/lib/update-notifier/apt-check --human-readable | grep security | awk '{print $1}' But that requires an install of the update-notifier-common package. (Note the -common. The main package has tons of requirements you don’t need.) In the quest for less dependencies — less installed packages — I used aptitude to get the info.

After switching a virtual IP around with keepalived we experienced a locking issue. Some client process on the server had some locks which did not get released but we could not see which query caused it. This broke our MySQL replication in this case, it was waiting on the locks to be released while executing the binlogs. We located the locks with mysql> show engine innodb status\G; The specific transaction did not show any query details but did show us a whole list of locks.

For me, using FreeBSD is still a bit like eating soup with a fork. Everything seems to make perfect sense but when I get to work I feel crippled and get annoyed by small differences with a GNU/Linux environment. A post to reduce the horror. The examples below work for a clean FreeBSD 10 install. This post is mostly useful for Linux users. I welcome additional tips. software management You can now use pkg which is quite friendly.

Today we received a DDoS on our network which caused a service interruption for our customers for about 20 minutes. This blogpost is a short report on the impact and nature of the attack. Impact Impact was network wide and caused degraded service for our customers between 14:45 and 15:08, a little over 20 minutes. The graph shows the impact as seen from our UK monitoring node (off net). Incoming traffic All our uplinks were saturated.

The Python xml.dom.minidom parseString silently skips over unknown entities. The only entities it does know, are &lt;, &gt;, &amp;, &apos; and &quot; and of course the numeric entities &#nn; and &#xhh;. That’s obvious, because those are the only ones defined in the XML 1.0 spec. However, if you’re parsing XHTML documents, it’s not nice that the entity references to special characters silently get dropped. Other people have stubled on the same issue, like in parsing xml containing &entities; with minidom and Problem with minidom and special chars in HTML.

A simple script to convert BSON data to JSON data: bson2json.py (view) Example usage: $ bson2json.py /var/backups/mongodb/all-dbs.mon/graylog2/streams.bson --pretty [ { "_id": "506ed227dc1d710c0700000e", "additional_columns": [], "alarm_active": true, "alarm_callbacks": [ "org.graylog2.emailalarmcallback.callback.EmailAlarmCallback", "org.graylog2.execalarmcallback.callback.ExecAlarmCallback" ], "alarm_limit": 80, "alarm_period": 5, "alarm_timespan": 5, "created_at": "2012-10-05T12:27:19Z", ...

Apparently I’m not the only one who randomly selects text as they read. My colleagues complained about this issue too. If you click Reply in Thunderbird Mail, only the text you recently selected is included in the new message. That’s not what I wanted! Luckily the Mozilla developers realised this too. Go to about:config and flip the switch. mailnews.reply_quoting_selection = false

Today we got a suspiciously good looking e-mail in the inbox. Someone who supposedly got a reminder about an unpaid invoice from us. The mail contained a zip-file with two scans. The first was a PDF, the second was an executable (a virus obviously). So.. where was the Amavis virus/spam scanner in all this? Show headers revealed that something was detected: X-Amavis-Alert: BANNED, message contains .exe,scan2/HP scan scan =?iso-8859-1?Q?HYJKIOPH5600002.=E2=80=AEfdp.exe?= Then why weren’t we informed?

After the upgrade of my desktop to Ubuntu Raring (13.04) my gnome-calculator’s menu bar had become unreachable. I don’t need the menu, except that it went into default BASIC mode. And I need the PROGRAMMING mode. The configuration seemed to be okay (accessible through gconf-editor): $ gconftool /apps/gcalctool --dump | grep -B1 -A4 mode <entry> <key>mode</key> <value> <string>PROGRAMMING</string> </value> </entry> <entry> <key>modetype</key> <value> <string>PROGRAMMING</string> </value> </entry> But that was apparently the old config.

A quick rundown on installing TeamViewer without a gazillion ia32-libs. The problem: if you attempt to install teamviewer_linux_x64.deb on your 64-bit machine, the ia32-libs dependency tries to install more than 200 packages. That not only feels like overkill, it takes a hell of a long time too. The solution: alter the dependency list in the .deb and create a small metapackage that references only the required libs. What follows, is the steps how.

Voilà, a MySQL function to count occurrences of a character (or a string of characters). DROP FUNCTION IF EXISTS OCCURRENCES; delimiter // CREATE FUNCTION OCCURRENCES (`needle` VARCHAR(255), `hackstack` TEXT) RETURNS INT NOT DETERMINISTIC READS SQL DATA SQL SECURITY INVOKER BEGIN DECLARE `result` INT DEFAULT -1; DECLARE `pos` INT DEFAULT 0; DECLARE `skip` INT DEFAULT LENGTH(`needle`); REPEAT SET `pos` = (SELECT LOCATE(`needle`, `hackstack`, `pos` + `skip`)); SET `result` = `result` + 1; UNTIL `pos` = 0 END REPEAT; RETURN `result`; END; // delimiter ; Now you can do things like this: