XtraBackup 1.3b

I have installed the newer version of XtraBackup, a tool for online backup of MySQL databases. The first tests show that a lot of errors were fixed in this, not yet released, version. Version 1.2 failed to make incremental backups on one of my servers, could not restore full backup on another, and segfaulted during full backup on the third one. 1.3b makes copies of all three and is able to restore them flawlessly.

A 233GB database was copied in 2:15'. I highly recommend you upgrade your installation of Xtrabackup to v.1.3b and to test it on production servers.

Download page at Percona.com is here: XtraBackup-1.3-beta. I usually download x86_64 binary files for Linux here.

Gaps in graphs generated by rrdtool

I sometimes see questions like: "Why are there gaps in my Munin graphs?" or "How do I get rid of holes in Nagiosgraph charts?". Some classical answers may be found in Munin FAQ or in 'man munin.conf'.

There is one more option, though. The problem may be caused by buggy plugins. By rounding errors in the plugins, to be precise.

From my experience, the gaps are usually found in the graphs where the data is a sum of some indicators. For example, the CPU load graph. The maximum value is calculated as number-of-cpus*100%. The actual value is calculated as the sum of a number of values: the time spent by the CPU in user mode, in system mode, waiting for I/O, in idle state, etc. Under certain conditions, this sum may exceed 100%. In this case, Rrdtool will omit the measurement from the graph and there will be a hole.

To get rid of the gaps, for example, in Munin's 'cpu' plugin, I replaced the lines

        PERCENT=$(($NCPU * 100))
        MAX=$(($NCPU * 100))

with

        PERCENT=$(($NCPU * 100 + 200))
        MAX=$(($NCPU * 100 + 200))

200 might be an overkill, but it looked a bit better in my Munin :). Of course, another option is to check the actual data for being not greater than the max value.

Using variables in complex configurations of Nginx

There are some situations for which Nginx config file does not provide adequate solutions. Very often, using variables, both built-in and user-defined, might help.

For example, Nginx does not support complex conditions, neither as logical operations nor as nested if's. The solution is to define a new variable. If you want a certain URL rewrite to take place only if the Referer header contains "somedomain.com" OR is empty, you can write:

$dorewrite = 'no';
if ($http_referer ~* somedomain.com) {
       set $dorewrite 'yes';
}
if ($http_referer = '') {
       set $dorewrite 'yes';
}
if ($dorewrite = 'yes') {
       rewrite ^ new-request;
}

If you want to rewrite the URL if both conditions are true, for example, the Referer header contains "somedomain.com" AND there's a GET parameter 'id' equal to '12345', modify the snippet above using the De Morgan's law:

$dorewrite = 'yes';
if ($http_referer !~* somedomain.com) {
       set $dorewrite 'no';
}
if ($arg_ID != '12345') {
       set $dorewrite 'no';
}
if ($dorewrite = 'yes') {
       rewrite ^ new-request;
}

Note the variable $arg_ID. This is a built-in variable. There is one such variable for every GET parameter in the request. Its name is composed of $arg_ and the name of the parameter. If you know all parameters you need to serve the request, these variables will help you to extract and pass them around.

If you use complicated regular expressions which you'd like to simplify, define another variable that would contain a part of the regexp:

if ($arg_ID ~* ^doc-number-([-a-z0-9]*)$) {
        set $newarg $1;
}
if ($dorewrite = 'yes') {
        rewrite ^ /doc/view/$newarg? break;
}

Installation of HP Array Configuration Utility CLI for Linux on Ubuntu 10.04 Lucid Lynx

HP Array Configuration Utility CLI for Linux (hpacucli) is a useful tool that allows you to manage your HP RAID controller from command line (via ssh, that is). It can also be used to monitor the state of the disk subsystem with tools like Nagios or Zabbix. The following controllers are supported:

• Smart Array 5312 Controller
• Smart Array 5302 Controller
• Smart Array 5304 Controller
• Smart Array 532 Controller
• Smart Array 5i Controller
• Smart Array 641 Controller
• Smart Array 642 Controller
• Smart Array 6400 Controller
• Smart Array 6400 EM Controller
• Smart Array 6i Controller
• Smart Array P600 Controller
• Smart Array P400 Controller
• Smart Array P400i Controller
• Smart Array E200 Controller
• Smart Array E200i Controller
• Smart Array P800 Controller
• Smart Array E500 Controller
• Smart Array P700m Contoller
• Smart Array P410i Controller
• Smart Array P411 Controller
• Smart Array P212 Controller
• Smart Array P712m Contoller
• Smart Array B110i SATA RAID
• Smart Array P812 Controller
• MSA500 Controller
• MSA500 G2 Controller
• MSA1000 Controller
• MSA1500 CS Controller
• MSA20 Controller

The tool is supplied on HP Support Pack CDs, but you can download a newer version from the HP site. To install hpacucli, visit this page and download hpacucli-8.50-6.0.noarch.rpm. Now, copy it to the server you'll install it on. Next, we'll unpack the RPM file. You can do it using, for example, rpm2cpio, rpm2tgz, etc. In Ubuntu, there's a utility called alien, that can do the same:

alien --to-tgz hpacucli-8.50-6.0.noarch.rpm

alien will report some errors and warnings, but don't worry. You will now get a new file, called hpacucli-8.50.tgz.

tar -xzf hpacucli-8.50.tgz

Move the unpacked files to corresponding locations:

sudo mv opt/compaq /opt/
sudo mv usr/sbin/* /usr/sbin/

Now, if you run a i386 kernel, you can run hpacucli, which is a 32-bit program. However, if your Ubuntu is a 64-bit system, you will have to allow execution of 32-bit binaries. One of the ways to do so is to install ia32-libs package. After the installation you can run hpacucli.

hpacucli gives you a prompt where you can enter commands or you can give the commands from the shell command line, like:

$ sudo hpacucli help

or

$ sudo hpacucli ctrl all show config

Smart Array P212 in Slot 1                (sn: PACCP9SYJ067  )

   array A (SAS, Unused Space: 0 MB)


      logicaldrive 1 (419.2 GB, RAID 1, OK)

      physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS, 450 GB, OK)
      physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SAS, 450 GB, OK)

   Expander 250 (WWID: 50014380065D7410, Port: 1I, Box: 1)

   Enclosure SEP (Vendor ID HP, Model DL18xG6BP) 248 (WWID: 50014380065D7423, Port: 1I, Box: 1)

   SEP (Vendor ID PMCSIERA, Model  SRC 8x6G) 249 (WWID: 50014380069159CF)

A similar command will give you even more information on the state of your RAID:

$ sudo hpacucli ctrl all show config detail

Now you can use this plugin to monitor the state of the array in Nagios. Besides, the information from hpacucli helped me to identify the causes of the significantly degraded performance of one of my servers (the battery of the write-cache was dead).

logrotate: rotating logs in multiple directories

I've got a server with almost one hundred web-sites. Each of the sites is in its own directory and runs its own logs in /usr/local/www/SITENAME/logs/*.log. When the logs grew up, I decided to set up logrotate. Since there were so many web-sites, my first idea was to create one configuration file for every site in logrotate.d and leave just one line in logrotate.conf:

include /usr/local/etc/logrotate.d

Only one hour later I understood that I can just write a path with multiple meta-characters to include ALL logs in just one line:

/usr/local/www/*/logs/*.log {
    daily
    rotate 60
    compress
    notifempty
    postrotate
          [ ! -f /var/run/nginx.pid ] || kill -USR1 `cat /var/run/nginx.pid`
    endscript

}

PINBA: PHP Is Not a Bottleneck Anymore

Yesterday I installed Pinba on one of my servers. Pinba is a set of tools to monitor performance of PHP scripts. Pinba MySQL database engine runs a list of timers and automatically fills report databases. Pinba PHP extension uses two functions to open and close these timers. Besides, there are default timers, which open when a script is executed and close when it finishes. If you put a timer around some critical piece of code you can get information on how often it runs and how much time it is being executed. Timers can be tagged and the data can be grouped by tags. So, in some pieces of code you can set tags "author" and "task" with corresponding values. Then you'll manage to compare the performance of code written by different developers and identify the most time-consuming parts. The most interesting thing is that when you create database tables following certain rules, these tables become automagically filled with the necessary data. So, if you use the tags "author" and "tags" to group the data, the reports will include all valid combinations of these tags and show summaries on these combinations: how much time Joe's scripts parsed new documents and how often Jackie's front-end scripts were called. Very impressive.

By default, Pinba stores this information for some limited period of time (15 minutes, IIUC), so you need some way to make the data persistent. Since we use Munin to monitor various system indicators, I wrote a couple of plugins (in TCL and Lua, just for fun :)) to display the frequency of execution and the average execution time for each timer. Our developers added a handful of timers in various places of code and here it is:

The last graph looks cluttered and not too informative, so I plan to employ Munin's 'suggest' feature to draw some diagrams using one script. Perhaps, organizing the graphs will be the most difficult part of the deployment. I have to say, though, that the installation was not simple, either. Prerequisites include compiled sources of the installed MySQL (Percona Server 10.2 in my case), Google Protocol Buffers, Judy library, libevent 1.4.1+ (Ubuntu's default one will do) and Hoard memory allocator. And here is the installation process (paths will be different for you, so check carefully):

wget http://pinba.org/files/pinba_engine-0.0.5.tar.gz
tar -xzf pinba_engine-0.0.5.tar.gz
wget http://pinba.org/files/pinba_extension-0.0.5.tgz
tar -xzf pinba_extension-0.0.5.tgz
wget http://protobuf.googlecode.com/files/protobuf-2.3.0.tar.gz
tar -xzf protobuf-2.3.0.tar.gz
wget http://downloads.sourceforge.net/project/judy/judy/\
Judy-1.0.5/Judy-1.0.5.tar.gz?use_mirror=ignum
tar -xzf Judy-1.0.5.tar.gz
wget http://www.cs.umass.edu/%7Eemery/hoard/hoard-3.8/source/hoard-38.tar.gz
tar -xzf hoard-38.tar.gz
sudo aptitude install libevent-1.4-2 libevent-dev
cd protobuf-2.3.0
./configure
make -j
sudo make install
cd ../judy-1.0.5/
./configure
make
sudo make install
cd ../hoard-38/src
make linux-gcc-x86-64
sudo cp libhoard.so /usr/local/lib
sudo cp *.h /usr/local/include
sudo ldconfig
cd pinba_engine-0.0.5/
./configure --with-mysql=/home/minaev/Percona-Server-10.2/ \
--with-judy=/usr/local --with-protobuf=/usr/local \
--with-event=/usr --libdir=/usr/lib/mysql/plugin/ \
--with-hoard=/usr/local
make 
sudo make install
echo "INSTALL PLUGIN pinba SONAME 'libpinba_engine.so'"|mysql
echo "CREATE DATABASE pinba"|mysql
mysql -D pinba <default_tables.sql
cd pinba-0.0.5/
sed -i 's/NOTICE/CHECKING/' config.m4
phpize
./configure --with-pinba=/usr/local
sudo make install

I had to edit config.m4 because my version autoconf was a bit buggy. After this process you'll have to add three lines to your php.ini:

extension=pinba.so
pinba.enabled=1
pinba.server=[MySQL server address]

And here is one of Munin plugins, written in TCL. It collects data on how often certain API parts were called.

#!/usr/bin/tclsh

package require mysqltcl 3.05


proc clean_fieldname arg {
    return [regsub -all {[^A-Za-z]} $arg "_"]
}

set dbuser "pinba"
set db "pinba"

set conn [::mysql::connect -user $dbuser -db $db]

set fields [::mysql::sel $conn 
  "select concat(module_value, '+', action_value) from \
tag_info_module_action" -list]

if {$argc > 0} {
    switch [lindex $argv 0] {
        "config" {
            puts "graph_title PHP Actions per second"
            puts "graph_vlabel reqs per second"
            puts "graph_category Pinba"
            foreach fld $fields {
                set clean [clean_fieldname $fld]
                 puts "$clean.label $fld"
                 puts "$clean.draw LINE3"
            }
        }
        "autoconf" {
            puts "yes"
        }
    }
} else {
    foreach fld $fields {
        set clean [clean_fieldname $fld]
        set data [::mysql::sel $conn 
  "select req_per_sec from tag_info_module_action where \
  concat(module_value, '+', action_value)='$fld'" -list]
        puts "$clean.value $data"
    }
}

::mysql::close $conn

BTW, you may find it interesting that the performance of TCL scripts was almost the same as that of Lua scripts and about 3-4 times higher than for Bash.

Dark sides of Python

While reading about Python and playing around with its objects and classes (OOP being a perversions in itself), I witnessed a slightly weird behaviour. Define a class with a class variable.

class Parent:
  variable = "parent 1"

Then define a descendant class that inherits the class variable:

class Child(Parent):
  pass

(That funny single pass stands for empty definition body) Now, let's have a look at the value of variable in Parent and Child:

print Parent.variable
parent 1
print Child.variable
parent 1

Then, change the value of the variable in the parent class and it should also change in the child class:

Parent.variable = "parent 2"
print Parent.variable
parent 2
print Child.variable
parent 2

Sounds good. The variable must be shared between the two classes. Now, let's change the value of this allegedly shared variable in the child class:

Child.variable = "child 1"
print Parent.variable
parent 2
print Child.variable
child 1

Quite of a sudden, the variable turns out to be two separate variables. We have somehow broke the link that connected them and now, even if we change the value of the variable in Parent, this will not affect the variable in Child anymore:

Parent.variable = "parent 3"
print Parent.variable
parent 3
print Child.variable
child 1

And how a language with such non-trivial idiosyncrasies can be promoted as newbie-friendly, "does-what-you-want" language? My interest to Python evaporates so fast that I will probably never get to the famous included "batteries".