Extending HP SmartArray RAID array and expanding root file system using LVM

Not too ancient Linux distrubutions use LVM even when installed on RAID array (I used to mistake LVM for RAID some time ago). The multitiered construction becomes a bit overcomplicated: you use RAID to merge hard drives into one logical drive, which is split into partitions, which are then merged into logical volumes. What should we do to grow our root file system onto new hard drives when the RAID size is not enough anymore?

Firstly, insert the new drives into the slots and then use hpacucli utility to check that they are recognized now:

hpacucli ctrl all config show

New drives should appear in the 'unassigned' section. Now, merge them into the existing array:

hpacucli ctrl slot=1 array A add drives=allunassigned

Now, wait till RAID finishes the work. It may take some hours. When it's over, attach these drives to an existing logical drive:

hpacucli ctrl slot=1 ld 1 modify size=max

If you try to run fdisk/gdisk now, you will not see the free space, because it was not available to the operating system during the boot time. You'll have to reboot. After reboot, use fdisk or gdisk to create a new partition on the free disk space. The partition must be assigned type 8E00 (Linux LVM). The operating system will not recognize the new partition immediately. You may try using partprobe command, but I preferred to reboot once again.

Next, you have, in sequence, to create LVM physical volume, extend LVM volume group onto the new volume and extend LVM logical volume to use the free space:

pvcreate /dev/cciss/c0d0p3 vgextend VolGroup00 /dev/cciss/c0d0p3 lvextend -l+429214 /dev/mapper/VolGroup00-LogVol01

To determine the sector number you will use in lvextend command, find the information about free physical extents in the output of vgdisplay:

vgdisplay |grep Free Free PE / Size 429214 / 1.64 TiB

And, finally, grow the file system. If you are conservative enough to use the default ext4 system, you can do this using:

resize2fs -p /dev/mapper/VolGroup00-LogVol01

Find files modified on certain date or in a date range

When you want to find files matching certain conditions, this is most probably job for the find command. The problem, though, is that you can't specify a certain date.

Options like -atime, -ctime or -mtime receive an argument that specifies the number of 24-hour periods. This means that when you run the following command:

find /usr/log/mysql -type f -mtime -1 -exec ls -l {} \;

you will get not only the files modified today, but those modified within the last 24 hours. You can change this behaviour by adding the option -daystart, which means that the time periods are calculated from the beginning of the current day:

find /var/log/mysql -type f -daystart -mtime -1 -exec ls -l {} \;

This command will produce the list of files modified today. IMPORTANT! Note that the -daystart option must precede all date-comparing options to have an effect.

To find files modified between two dates you can join two conditions using -a option:

find /var/log/mysql -type f -daystart -mtime -3 -a -mtime +0 -exec ls -l {} \;

The result will include the files modified yesterday or the day before yesterday.

Sometimes, though, you may want to specify the dates as they are, not as relative number of days from today. Traditionally, it was done using an awkward technique that involved creating two empty files with modification dates corresponding to the lower and upper borders of the range (using touch -t filename) and then using these files in options -newer and -older:

touch temp -t 200604141130
touch ntemp -t 200604261630
find /data/ -cnewer temp -and ! -cnewer ntemp

(example taken from here)

New versions of find allow you to do just that using -newerXY. Letters X and Y here stand for some one-letter codes corresponding to various comparison types. The combinations are pretty much incomprehensible, but what we need is -newermt. With this option, life gets simple and sunny!

find /var/log/mysql -type f -newermt 2012-08-21 ! -newermt 2012-08-23 -exec ls -l {} \;

This command produces the list of files modified exactly between the beginning of August 21 and the beginning of August 23.

And a little bonus for those who made it to the end! To sum the sizes of the found files (so you can find out, for example, how many gigabytes were written to binlogs in the last two days) use du -c:

du -c `find /var/log/mysql -type f -newermt 2012-08-21 ! -newermt 2012-08-23`|tail -n1

Chroot, but don't chroot

FTP, SFTP and SCP accounts are often restricted to their home directory, so the users don't mess around with the system. This is done using 'chroot'. You can set up SSH demon or FTP server so the user cannot leave his home directory. But what if you want to give him access to some other directory outside his home directory?

The first thing that comes to mind is links. Soft links don't work, though, because the user cannot see anything outside his directory. Hard links won't work, either, because you usually cannot hardlink directories. They'll do if all you want is access to a single file, though. Also, you can change the user's home directory to the directory you want him to modify. But you may want to grant access to more than one directory. Besides, the implementation of chroot requires that if the user is chrooted to /var/data/lib/img, all directories up the tree (i.e., /var, /var/data and /var/data/lib) must belong to root and nobody else should have write permissions there. This is not always possible.

But the right solution would be to mount a directory to your home directory using bind option. Create the mountpoint inside the home directory and mount:

mkdir /home/remote/img
mount --bind /var/data/lib/img /home/remote/img

Now, the user will be able to work with /var/data/lib/img, but not with any other data on the server.

To make the mount persistent across reboots, add the corresponding entry to /etc/fstab:

/var/data/lib/img /home/remote/img none bind 0 0

More than one 'exec' action in 'find' command

When you run find command, you can pass the names of the found files to an arbitrary command using -exec option:

find /tmp -mtime +3 -exec rm {} \;

The curly braces get replaced by the name of found files and the command is executed for every file. However, if you want to run more than one command on the file or use the filename more than once in one command, you cannot do that:

Only one instance of `{}' is allowed within the command.

To bypass the limitation, you can execute a shell, passing the filename as an argument. In the commands executed by the shell, the argument will be available as $0:

find /tmp -mtime +3 -exec sh -c 'ls -ld "$0"; rm "$0"' {} \;

Getting rid of GUI password prompt in EasyPG

EasyPG is a package used by GNU Emacs to simplify work with files encrypted by GnuPG (GPG). Extremely useful package, it has a bad habit of relying on external tools to prompt for the password. One of such tools is 'gpg-agent'. In Debian/Ubuntu this agent usually replaces 'ssh-agent'. It's a good tool, but it uses a GUI dialog to prompt for the password. This makes your work with EasyPG and Emacs rather uncomfortable.

There's a number of advices on how you can suppress that GUI dialog. I prefer another one, I simply removed 'gpg-agent'. It did not affect the functionality, since it was replaced by 'ssh-agent'. Now, EasyPG prompts for the password in Emacs mini-buffer

moreutils

Discovered 'moreutils' package in Ubuntu. It features a list of nice tools, of which my favorite are 'ts', 'pee' and 'vipe'.

'ts' puts timestamps to every line of text that is piped through it. Like,

$ ping google.com|ts
Dec 14 14:18:45 PING google.com (209.85.173.104) 56(84) bytes of data.
Dec 14 14:18:45 64 bytes from lpp01m01-in-f104.1e100.net (209.85.173.104): icmp_seq=1 ttl=52 time=88.5 ms
Dec 14 14:18:46 64 bytes from lpp01m01-in-f104.1e100.net (209.85.173.104): icmp_seq=2 ttl=52 time=92.5 ms
Dec 14 14:18:47 64 bytes from lpp01m01-in-f104.1e100.net (209.85.173.104): icmp_seq=3 ttl=52 time=90.5 ms

I love timestamps! I put them everywhere I can, it helps so much to find out later what was going on. 'ts' will certainly help me.

Another tool is 'pee'. As 'man pee' puts it, it's like 'tee', but for pipes. 'pee' reads standard input and sends it to a number of commands supplied as arguments to 'pee'. Like,

$ cat file.txt |pee 'wc -c' 'wc -w' 'wc -l'
5870
257
257

And, finally, 'vipe' — it's a way to edit data produced by one program before they are piped into another one. 'sed' is good enough, but when some interactivity is needed, 'vipe' should be handy.

Real conversations in Gmail

Organizing your email in the Gmail way, by conversation, is very handy. What always annoyed me in the Google implementation was that the sent emails did not appear in the default folder. I know it is called Inbox. But sent messages do appear there, too, as soon as the recipient replies to me.

When I used other email clients, like Pine or Thunderbird, I used to set up FCC (folder carbon copy) in the outgoing messages to my Inbox. This gave me a conversation-like structure better than the one found in Gmail. Finally, I have the same structure in Gmail.

To see all your outgoing messages in (so called) Inbox, even when there are no replies in the conversation, create a new filter. Put your address to the From: field. Click "Next Step". Select "Never send it to Spam" and "Create Filter". That's it.