Some Debian and Ubuntu Quirks

EDITOR=nano

I’m not sure why Debian and Ubuntu decided to make the EDITOR default to nano instead of vi. It is annoying whenever I run the visudo command the sudoers file gets opened up by nano instead of vi. Isn’t it why it was called visudo and not nanosudo? :)

Arrow keys in VI

The vi in Debian doesn’t allow the arrow keys in edit mode. Why? An easy fix would be to do :set nocompatible in vi. But still, would it be nice if this would be the default like in other distros (Ubuntu, Redhat, and CentOS)? VI in MacOSX also allows the arrow keys in edit mode.

How to specify an ssh-key file with the Git command

If you want to use an ssh-key file whenever you run the ssh command, one convenient way to do this is to use the -i option.


ssh -i ~/.ssh/thatuserkey.pem thatuser@myserver.com

This is pretty neat. It’s simple, elegant, and highly intuitive. I want to do the same thing with the Git command like this:


git -i ~/.ssh/thatuserkey.pem clone thatuser@myserver.com:/git/repo.git

Unfortunately, there is no such -i option in the git command. Bummer.

I’ve looked around but I can’t find a solution like this. There are two options I can think of: 1) use GIT_SSH and 2) use a wrapper script.

Option 1: Use the GIT_SSH environment variable

The GIT_SSH option will allow you to specify a key file with the Git command like this:


PKEY=~/.ssh/thatuserkey.pem git clone thatuser@myserver.com:/git/repo.git

where ~/.ssh/thatuserkey.pem is the keyfile you want to use.

To make this work, it needs some pre-configuration. The first step is to create a shell script that contains the following.

~/ssh-git.sh


#!/bin/sh
if [ -z "$PKEY" ]; then
# if PKEY is not specified, run ssh using default keyfile
ssh "$@"
else
ssh -i "$PKEY" "$@"
fi

The script needs to be executable so do a chmod +x on it.

Next step is to set the value of the GIT_SSH variable to the path of the script above. The variable will need to be exported to the shell environment.


export GIT_SSH=~/ssh-git.sh

Now every time you run the git command, the keyfile you set to the PKEY variable is passed to the shell script GIT_SSH is pointing to. This will then allow Git to connect using that key file.


PKEY=~/.ssh/thatuserkey.pem git clone thatuser@myserver.com:/git/repo.git

From hereon, every time you run the Git command, you can freely choose any key file you want to use by setting the PKEY variable.[1]

If you run the git command without the PKEY line, the GIT_SSH script will still run since this is exported to the shell environment. The script has a fail safe to avoid using the -i option if there was no keyfile set so that it can still run using the default keyfile.

Be careful when exporting PKEY to the shell environment because GIT_SSH will use whatever value it is set to even if you don’t specify it with the git command. This brings another problem with GIT_SSH exported to the environment since Git will always use this when it runs. So you need to be constantly conscious that you have this set. You can always chain the GIT_SSH setting with the git command to avoid exporting it to the environment, but at the expense making the entire command longer.

The PKEY-line usage works well except that the setting of PKEY together with the git command is somehow unconventional.[2]

If you find this unintuitive, there is another option.

Option 2: Use a wrapper script

The -i option with ssh is pretty neat and elegant. You supply the -i option to choose the key file you want to use. If you don’t use the option, ssh will fall back to use the default ssh-key file.

To use the -i option with the Git command, we need to write a wrapper script. The wrapper script will then allow us to set the usage we like and that is to mimic the -i option in ssh.

The usage will be something like this:


git.sh -i ~/.ssh/thatuserkey.pem clone thatuser@myserver.com:/git/repo.git

where git.sh is the wrapper script.

The only thing you need to do is create that script, put it in your PATH, and you’re all set.

To get the code, you can download it from here or cut-and-paste that below.

git.sh


#!/bin/bash

# The MIT License (MIT)
# Copyright (c) 2013 Alvin Abad

if [ $# -eq 0 ]; then
    echo "Git wrapper script that can specify an ssh-key file
Usage:
    git.sh -i ssh-key-file git-command
    "
    exit 1
fi

# remove temporary file on exit
trap 'rm -f /tmp/.git_ssh.$$' 0

if [ "$1" = "-i" ]; then
    SSH_KEY=$2; shift; shift
    echo "ssh -i $SSH_KEY \$@" > /tmp/.git_ssh.$$
    chmod +x /tmp/.git_ssh.$$
    export GIT_SSH=/tmp/.git_ssh.$$
fi

# in case the git command is repeated
[ "$1" = "git" ] && shift

# Run the git command
git "$@"

The wrapper script can fail gracefully. If you don’t specify the -i option, it will run git using your default key-file.

This wrapper script uses the same principle of the GIT_SSH environment variable. But instead of pre-setting this up manually, the wrapper script sets this up on the fly every time it runs the actual git command.

Other options

There are other ways to use different ssh-keys with the Git command. There is this $HOME/.ssh/config file where you can map different keys to hosts you want to connect to. But this method doesn’t allow you to choose any key file at will when you need to run the git command. The keys need to be pre-defined in the config file.

You can also use ssh-agent to programmatically add the key you want to use. I also wrote a wrapper script that uses ssh-agent to allow the -i option. But it turned out to be more complex than GIT_SSH way. I’ll probably post that to show how it can be done that way.

With all the different methods available, none is necessarily better than the other. It will all depend on the circumstances and of course your personal taste.

Alvin


[1] I prefer this kind of control in my workflow because I use different keys for different servers I use. I have a different key for my servers at work and different keys for my personal servers and public sites (like Github). It works the same with passwords. You don’t use the same password on your Facebook and bank accounts.


[2] Personally, I find this all right since I am used to this usage. I run a lot of scripts and make commands that require environment settings. But I don’t like the idea of exporting all of them to the shell environment and leaking them everywhere so I only specify them with the command.

How to Recover a LUKS Encrypted Disk

This morning, my Ubuntu laptop won’t boot up. My immediate reaction was shivers at the back of my neck and then down the shoulders. I don’t understand why it felt that way. It’s like one of those situations when you suddenly discovered you left your wallet, your laptop, or something valuable in a restaurant and your mind started racing trying to figure out how to get back quickly and at the same time thinking what would happen and what would you do if you can’t locate it anymore… And all those thoughts happening in just two or three seconds.

In this case, my laptop won’t boot up, it uses LUKS full disk encryption, I have no idea how to recover it, I haven’t done it before, I don’t know if it can be recovered, and I’ve got tons of important data in there!

Fortunately, those were just a couple of seconds and after that you go back to your sane state and do what most everyone else does – Google your way out. Thanks again to Google, it was easy to research on how to recover an LUKS-encrypted disk.

I missed the old days when recovering a disk from a failed system was as simple as attaching the disk to another system, mounting it, and start accessing your data back. Life has changed. With most of our valuables in digital format nowadays, you need to apply the same security you do with physical valuables – you lock them up with a key. But instead of locking with a door, you lock the data by encrypting it.

Below are the steps to recover a LUKS encrypted disk. There are tons of information like this online but I thought I just need to add another document among hundreds already out there in case one poor soul can only find this one.

Summary:

  1. Boot the system from a rescue disk or Live CD. If this is not possible, attach the encrypted disk to another Linux system
  2. Identify the device name of the encrypted disk
  3. Identify the encrypted partition in the disk
  4. Unlock the encrypted partition
  5. If the partition is using logical volumes, identify the volume group(s) used
  6. If the partition is using logical volumes, enable the volume group
  7. Mount the logical volume desired. If the partition is not using logical volumes, mount the mapped device directly.
  8. Unmount and lock the encrypted partition when done

Details:

Step 1. Remove the disk from the bad system and attach it to another Linux machine. If the machine is still running fine but the problem was just the OS being corrupted or something, you can boot from a rescue or Live CD.

Step 2. Once the disk is attached to another system, identify the device name of the disk as to how it got attached. You can view this by viewing /proc/partitions. If you are booting from a rescue disk or Live CD, do the same, but you’ll discover that it is easier to identify the disk since you won’t be confusing it with other disks like when it is attached to another system.

Below is a sample output. In this case the device you need to recover is /dev/sda with partitions sda1, sda2, and sda5.

# cat /proc/partitions
major minor #blocks name

7 0 731040 loop0
8 0 58605120 sda
8 1 248832 sda1
8 2 1 sda2
8 5 58353664 sda5

It’s probably easier to use the fdisk command to get a much clearer view of the layout of the partitions.

# fdisk -l /dev/sda

Disk /dev/sda: 60.0 GB, 60011642880 bytes
255 heads, 63 sectors/track, 7296 cylinders, total 117210240 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00089bcb

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048      499711      248832   83  Linux
/dev/sda2          501758   117209087    58353665    5  Extended
/dev/sda5          501760   117209087    58353664   83  Linux

Step 3. Next, find the partition that is encrypted. This is where the actual data resides. Run the cryptsetup command on each partition and find the one that says something like “LUKS header information…”

# cryptsetup -v luksDump /dev/sda1
Device /dev/sda1 is not a valid LUKS device.
Command failed with code 22: Device /dev/sda1 is not a valid LUKS device.

 

# cryptsetup -v luksDump /dev/sda2
Device /dev/sda2 is not a valid LUKS device.
Command failed with code 22: Device /dev/sda2 is not a valid LUKS device.

 

# cryptsetup -v luksDump /dev/sda5
LUKS header information for /dev/sda5

Version:       	1
Cipher name:   	aes
Cipher mode:   	xts-plain64
Hash spec:     	sha1
Payload offset:	4096
MK bits:       	512
MK digest:     	21 84 3e c0 3e 9e 23 1e 34 9b 39 05 8f b9 47 61 89 a6 2a 81
MK salt:       	fc ac 3d 4f 1e 3d d4 ce 66 6b d3 90 ba f4 79 a8
               	d9 c9 38 a0 c2 79 bc 47 71 c6 8f 49 23 46 f1 6b
MK iterations: 	22500
UUID:          	2c8d56ec-749f-4d95-ab39-4ea17edb4c01

Key Slot 0: ENABLED
	Iterations:         	90067
	Salt:               	e4 25 ae 7c 5d 62 81 5e ea 37 95 0f 59 7b c8 7f
	                      	13 4f bc 15 70 4e 82 e1 41 db 1d 4b 65 7a de 5c
	Key material offset:	8
	AF stripes:            	4000
Key Slot 1: DISABLED
Key Slot 2: DISABLED
Key Slot 3: DISABLED
Key Slot 4: DISABLED
Key Slot 5: DISABLED
Key Slot 6: DISABLED
Key Slot 7: DISABLED
Command successful.

From the sample output above, the partition we need to recover is /dev/sda5.

Step 4. Unlock the encrypted partition using the cryptsetup command. You will need the passphrase of the encrypted disk. I forgot to mention, you need this password to recover the data from the disk. If you don’t know the password or you forgot, then you’re out of luck. Unless you can guess the password or do a brute force password guessing, no current technology or witchcraft on earth can help you unlock the encryption.

That’s the whole idea of encrypting a disk so that in case your laptop got stolen, got lost, or got left at a restaurant, no one but you who knows the password can see your data.

# cryptsetup -v luksOpen /dev/sda5 sda5_crypt
Enter passphrase for /dev/sda5:
Key slot 0 unlocked.
Command successful.

Step 5. After unlocking the device, check if the partition is using LVM (Logical Volume Management). If your data resides in a logical volume, you’d need to identify those volumes so that you’d know what to mount. Run the lvdisplay command to see the volumes. Below is a sample output.

# lvdisplay
  --- Logical volume ---
  LV Path                /dev/ubuntu/root
  LV Name                root
  VG Name                ubuntu
  LV UUID                fy5LpG-HPX7-Spe1-C92m-nzPq-B3IB-eTjCFA
  LV Write Access        read/write
  LV Creation host, time ubuntu, XXXX
  LV Status              available
  # open                 0
  LV Size                53.61 GiB
  Current LE             13724
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           252:1

  --- Logical volume ---
  LV Path                /dev/ubuntu/swap_1
  LV Name                swap_1
  VG Name                ubuntu
  LV UUID                ZSwRl4-bdV1-kqJ3-V1lG-ndfZ-cdSh-P5X3kp
  LV Write Access        read/write
  LV Creation host, time ubuntu, XXXX
  LV Status              available
  # open                 0
  LV Size                1.99 GiB
  Current LE             509
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           252:2

Step 6. If the partition has logical volumes, enable the volume group using the vgchange command. From the sample above, the name of the volume group is ubuntu.

# vgchange -a y ubuntu
  2 logical volume(s) in volume group "ubuntu" now active

Step 7a. If the partiton is using logical volumes, mount the logical volume desired. Using the sample above, we want to mount /dev/ubuntu/root.

# mkdir /tmp/disk
# mount /dev/ubuntu/root /tmp/disk

Step 7b. If the partition does not have logical volumes, mount the mapped device directly.

# mkdir /tmp/disk
# mount /dev/mapper/sda5_crypt /tmp/disk

That’s it. You can now view your data in /tmp/disk.

Step 8. When you’re done accessing the data, don’t forget to unmount and close (lock) the encrypted partition.

# umount /tmp/disk
# cryptsetup luksClose /dev/mapper/sda5_crypt

Send a Twitter Message from the Unix Command Line

If you are a fan of Twitter and also a Unix user who spends most of your waking hours on the command line, wouldn’t it be nice if you can send your Twitter update from the Unix command line? Twitter has an API that makes sending from the Unix command line easy. All you need is the curl command (which should be available nowadays on most Unix systems) and you can fire up your Twitter message like this:

    curl -u mytwitterusername:mypassword -d status="the server is up!" \
                  http://twitter.com/statuses/update.json

Note: The command above should be all in one line.

Easy, isn’t it?

But you might say, “Wow, that’s a lot of typing to do. BTW, is that my password in clear text?”

Well, this is where Unix scripting comes to the rescue. You can easily wrap this in a shell script and your usage could be as simple as this:

        twitter.sh "the server is up!"

The rest are inside the script where you can hide your username and password by making sure that only you can read it. For example:

#!/bin/sh
curl -u mytwitterusername:mypassword -d status="$*" \
      http://twitter.com/statuses/update.json

If you are not comfortable putting your password and username in the script, you can always make the script prompt for them. For example:

#!/bin/sh

echo -n "Enter username: "
read USERNAME

echo -n "Enter password: "
stty -echo
read PASSWORD
stty echo
echo

MESSAGE="$*"

curl -u $USERNAME:$PASSWORD -d status="$MESSAGE" \
     http://twitter.com/statuses/update.json
echo

For details about this API, see Twitter documentation: http://apiwiki.twitter.com/REST+API+Documentation

Alvin Abad

Western Digital 1.5 TB My Book World Edition II Hard Drive

I recently purchased a personal network attached storage (NAS). It’s a 1.5TB external hard drive from Western Digital. It’s called – Western Digital 1.5 TB My Book World Edition II Hard Drive. [1] Thanks to my friend, Don, for telling me about this.

There are many good things about this product. Unfortunately, there are some bad things too. Let’s start with the good things.

The good things:

1. Cheap!

I purchased this from Amazon at US$319. [2] At this time, a standalone 500GB hard drive costs around $99. I thought this was impressive. It was the cheapest Linux-based NAS I could find out there.

2. Easy to setup.

It uses a web interface for setup and administration. Setting it up was as easy as setting up a wireless router. By default, it can do Windows-sharing type of access (CIFS/SMB). It also has this proprietary type of file access called MioNet (WDAnywhere), but I didn’t bother using it since I was only interested in using Samba and NFS. I actually disabled this hoping it would help speed up the machine.

3. It can do NFS.

If you’re going to use this with Unix, you would definitely want to have NFS access to this device. However, this is no longer available by default in the new firmware, 2.0.15+. [3] I had to install some nfs files manually (exportfs, rpc.*) to get this running. See my other post for details on how to set this up.

4. It’s a Linux machine.

It’s a small computer in itself and one of the compelling reasons I bought this was that it uses Linux. Therefore, in addition to disk storage, you can use it as a Linux machine with all the goodies like ssh, lighttpd, Perl, etc.

5. Quiet and cool.

Some reviews out there claim that this device is noisy. With two 750GB disks and a Linux machine in itself, I thought this was relatively more quiet than running your own Samba/NFS server on a desktop PC. It uses a small and low-power CPU (ARM926EJ). [6]

6. Plenty of Linux support.

You don’t want to be alone hacking this device. There are plenty of people already hacking this NAS. [4,5]

Now the bad things:

1. It’s slow!

I don’t understand what went wrong here because this NAS has a Gigabit ethernet and according to the specs, the drive speed is 7200RPM. However, I can only get a maximum of 7MB/sec of write speed. Note that this is through local access on the machine itself and not even across the network. I use the Unix command dd to measure write speed.

dd if=/dev/zero of=/tmp/x bs=1024 count=10240

10240+0 records in
10240+0 records out
10485760 bytes (10 MB) copied, 1.6501 seconds, 6.4 MB/s

I suspected that this is due to the CPU or IO of the machine. Therefore, even with a Gigabit ethernet and 7200 RPM drives, I doubt you can get it any faster than 7MB/s if the bottleneck is the CPU or IO.

I tested NFS write speed on a 100Mb/sec network and I can only get a maximum of 5 MB/sec.

2. NFS is disabled.

NFS used to be installed by default. All you need to do to use it was enable it. However, I learned that they removed this in versions v2.00.15 and later. [3] I had to install NFS files manually to get it up and running. I wonder why WD had to remove this. It appears that they don’t want users to be using this NAS with NFS. In addition to not supporting Linux access, WD also makes sure that it’s not there in case you’d want to use it.

3. Certain types of files cannot be shared

The software WDAnywhere restricts certain types of files from being shared. According to WD website, “Due to unverifiable media license authentication, the following file types cannot be shared…” [7] What this means is that you cannot share media files like mp3 or mpeg files even if they are your own personal audio or movie files.

I don’t have this problem with Samba and NFS. This has nothing to do with technology so I don’t want to comment about it. But I thought I find this feature amusing.

Some notes:

It’s probably a good idea to disable access to your network’s Internet gateway. If you only want to access this device on your intranet (LAN), there is no need to have it access your gateway and then the Internet. This makes it more safe in case you accidentally open up your router to your machines on the local network. This also prevents viruses or trojans from running on this device and contacting someone out there on the Internet, although highly unlikely because it uses Linux.

References:

[1] http://www.wdc.com/en/products/Products.asp?DriveID=318

[2] http://www.amazon.com/gp/offer-listing/B000RZ68IG/ref=dp_olp_2

[3] http://mybookworld.wikidot.com/nfs-server

[4] http://mybookworld.wikidot.com/

[5] http://martin.hinner.info/mybook/

[6] http://en.wikipedia.org/wiki/Western_Digital_My_Book#Internals

[7] WD FAQ – What files cannot be shared by WD Anywhere Access?