Some Debian and Ubuntu Quirks
EDITOR=nano
I’m not sure why Debian and Ubuntu decided to make the EDITOR default to nano instead of vi. It is annoying whenever I run the visudo command the sudoers file gets opened up by nano instead of vi. Isn’t it why it was called visudo and not nanosudo? :)
Arrow keys in VI
The vi in Debian doesn’t allow the arrow keys in edit mode. Why? An easy fix would be to do :set nocompatible in vi. But still, would it be nice if this would be the default like in other distros (Ubuntu, Redhat, and CentOS)? VI in MacOSX also allows the arrow keys in edit mode.
How to specify an ssh-key file with the Git command
If you want to use an ssh-key file whenever you run the ssh command, one convenient way to do this is to use the -i option.
ssh -i ~/.ssh/thatuserkey.pem thatuser@myserver.com
This is petty neat. It’s simple, elegant, and highly intuitive. I want to do the same thing with the Git command like this:
git -i ~/.ssh/thatuserkey.pem clone thatuser@myserver.com:/git/repo.git
Unfortunately, there is no such -i option in the git command. Bummer.
I’ve looked around but I can’t find a solution like this. There are two options I can think of: 1) use GIT_SSH and 2) use a wrapper script.
Option 1: Use the GIT_SSH environment variable
The GIT_SSH option will allow you to specify a key file with the Git command like this:
PKEY=~/.ssh/thatuserkey.pem git clone thatuser@myserver.com:/git/repo.git
where ~/.ssh/thatuserkey.pem is the keyfile you want to use.
To make this work, it needs some pre-configuration. The first step is to create a shell script that contains the following.
~/ssh-git.sh
#!/bin/sh
if [ -z "$PKEY" ]; then
# if PKEY is not specified, run ssh using default keyfile
ssh "$@"
else
ssh -i "$PKEY" "$@"
fi
The script needs to be executable so do a chmod +x on it.
Next step is to set the value of the GIT_SSH variable to the path of the script above. The variable will need to be exported to the shell environment.
export GIT_SSH=~/ssh-git.sh
Now every time you run the git command, the keyfile you set to the PKEY variable is passed to the shell script GIT_SSH is pointing to. This will then allow Git to connect using that key file.
PKEY=~/.ssh/thatuserkey.pem git clone thatuser@myserver.com:/git/repo.git
From hereon, every time you run the Git command, you can freely choose any key file you want to use by setting the PKEY variable.[1]
If you run the git command without the PKEY line, the GIT_SSH script will still run since this is exported to the shell environment. The script has a fail safe to avoid using the -i option if there was no keyfile set so that it can still run using the default keyfile.
Be careful when exporting PKEY to the shell environment because GIT_SSH will use whatever value it is set to even if you don’t specify it with the git command. This brings another problem with GIT_SSH exported to the environment since Git will always use this when it runs. So you need to be constantly conscious that you have this set. You can always chain the GIT_SSH setting with the git command to avoid exporting it to the environment, but at the expense making the entire command longer.
The PKEY-line usage works well except that the setting of PKEY together with the git command is somehow unconventional.[2]
If you find this unintuitive, there is another option.
Option 2: Use a wrapper script
The -i option with ssh is pretty neat and elegant. You supply the -i option to choose the key file you want to use. If you don’t use the option, ssh will fall back to use the default ssh-key file.
To use the -i option with the Git command, we need to write a wrapper script. The wrapper script will then allow us to set the usage we like and that is to mimic the -i option in ssh.
The usage will be something like this:
git.sh -i ~/.ssh/thatuserkey.pem clone thatuser@myserver.com:/git/repo.git
where git.sh is the wrapper script.
The only thing you need to do is create that script, put it in your PATH, and you’re all set.
To get the code, you can download it from here or cut-and-paste that below.
git.sh
#!/bin/bash
# The MIT License (MIT)
# Copyright (c) 2013 Alvin Abad
if [ $# -eq 0 ]; then
echo "Git wrapper script that can specify an ssh-key file
Usage:
git.sh -i ssh-key-file git-command"
exit 1
fi
# remove temporary file on exit
trap 'rm -f /tmp/.git_ssh.$$' 0
if [ "$1" = "-i" ]; then
SSH_KEY=$2; shift; shift
echo "ssh -i $SSH_KEY \$@" > /tmp/.git_ssh.$$
chmod +x /tmp/.git_ssh.$$
export GIT_SSH=/tmp/.git_ssh.$$
fi
# in case the git command is repeated
[ "$1" = "git" ] && shift
# Run the git command
git "$@"
The wrapper script can fail gracefully. If you don’t specify the -i option, it will run git using your default key-file.
This wrapper script uses the same principle of the GIT_SSH environment variable. But instead of pre-setting this up manually, the wrapper script sets this up on the fly every time it runs the actual git command.
Other options
There are other ways to use different ssh-keys with the Git command. There is this $HOME/.ssh/config file where you can map different keys to hosts you want to connect to. But this method doesn’t allow you to choose any key file at will when you need to run the git command. The keys need to be pre-defined in the config file.
You can also use ssh-agent to programmatically add the key you want to use. I also wrote a wrapper script that uses ssh-agent to allow the -i option. But it turned out to be more complex than GIT_SSH way. I’ll probably post that to show how it can be done that way.
With all the different methods available, none is necessarily better than the other. It will all depend on the circumstances and of course your personal taste.
Alvin
[1] I prefer this kind of control in my workflow because I use different keys for different servers I use. I have a different key for my servers at work and different keys for my personal servers and public sites (like Github). It works the same with passwords. You don’t use the same password on your Facebook and bank accounts.
[2] Personally, I find this all right since I am used to this usage. I run a lot of scripts and make commands that require environment settings. But I don’t like the idea of exporting all of them to the shell environment and leaking them everywhere so I only specify them with the command.
How to Use LUKS to Encrypt a Disk Partition
You can use LUKS to encrypt a partition of a disk drive or USB. If you store sensitive information in portable drives it’s more compelling than ever to protect them using encryption since they carry a high risk of getting lost or stolen.
LUKS/dm-crypt is a good choice for encrypting Linux devices. It’s usually pre-installed in most Linux distros and if not, it’s easy to install using YUM or APT.
Here are seven easy steps to encrypt a disk partition:
Step 1. Create the disk partition you wish to encrypt. For example, let’s say you have a USB drive and it’s connected to /dev/sdb. The partition you’d want to create would be /dev/sdb1.
# fdisk -l /dev/sdb Disk /dev/sdb: 512 MB, 512483328 bytes 255 heads, 63 sectors/track, 62 cylinders, total 1000944 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000 Device Boot Start End Blocks Id System /dev/sdb1 63 1000943 500440+ 83 Linux
Step 2. Encrypt the partition
# cryptsetup -q -y luksFormat /dev/sdb1 Enter LUKS passphrase: Verify passphrase:
Step 3. Map a logical partition
# cryptsetup luksOpen /dev/sdb1 sdb1_crypt Enter passphrase for /dev/sdb1:
This will create a device mapper:
# ls -al /dev/mapper/sdb1_crypt brw-rw---- 1 root disk 253, 5 Sep 23 11:53 /dev/mapper/sdb1_crypt
Step 5. Format the encrypted partition
# mkfs.ext3 /dev/mapper/sdb1_crypt mke2fs 1.42.5 (29-Jul-2012) Filesystem label= OS type: Linux Block size=1024 (log=0) Fragment size=1024 (log=0) Stride=0 blocks, Stripe width=0 blocks 124928 inodes, 498392 blocks 24919 blocks (5.00%) reserved for the super user First data block=1 Maximum filesystem blocks=67633152 61 block groups 8192 blocks per group, 8192 fragments per group 2048 inodes per group Superblock backups stored on blocks: 8193, 24577, 40961, 57345, 73729, 204801, 221185, 401409 Allocating group tables: done Writing inode tables: done Creating journal (8192 blocks): done Writing superblocks and filesystem accounting information: done
Step 6. Mount the encrypted partition:
# mkdir /mnt/sdb1 # mount /dev/mapper/sdb1_crypt /mnt/sdb1
Step 7. When done unmount the logical partition and close (unlock) the encrypted partition
# unmount /mnt # cryptsetup luksClose sdb1_crypt
How to Recover a LUKS Encrypted Disk
This morning, my Ubuntu laptop won’t boot up. My immediate reaction was shivers at the back of my neck and then down the shoulders. I don’t understand why it felt that way. It’s like one of those situations when you suddenly discovered you left your wallet or something valuable at a restaurant and your mind started racing trying to figure out how to get back quickly and at the same time thinking what would happen and what would you do if you can’t find it anymore… And all those thoughts happening in just two or three seconds.
In this case, my laptop won’t boot up, it uses LUKS full disk encryption, I have no idea how to recover it, I haven’t done it before, I don’t know if it can be recovered, and I’ve got tons of important data in there. Ahhh!
Fortunately, those were just a couple of seconds and after that you go back to your sane state and then do what everyone else does – Google your way out. Thanks again to Google, it was easy to research how to recover an encrypted disk.
I missed the old days when recovering a disk from a failed system was as simple as attaching the disk to another system, mounting it, and start accessing your data back. Life has changed. With most of our valuables in digital format nowadays, you need to apply the same security you do with physical valuables – you lock them up with a key. But instead of locking with a door, you lock the data by encrypting it.
Below are the steps to recover a LUKS encrypted disk. There are tons of information like this online but I thought I just need to add another document among hundreds already out there in case one poor soul can only find this one.
Summary:
- Boot the system from a rescue disk or Live CD. If this is not possible, attach the encrypted disk to another Linux system
- Identify the device name of the encrypted disk
- Identify the encrypted partition in the disk
- Unlock the encrypted partition
- If the partition is using logical volumes, identify the volume group(s) used
- If the partition is using logical volumes, enable the volume group
- Mount the logical volume desired. If the partition is not using logical volumes, mount the mapped device directly.
- Unmount and lock the encrypted partition when done
Details:
Step 1. Remove the disk from the bad system and attach it to another Linux machine. If the machine is still running fine but the problem was just the OS being corrupted or something, you can boot from a rescue or Live CD.
Step 2. Once the disk is attached to another system, identify the device name of the disk as to how it got attached. You can view this by viewing /proc/partitions. If you are booting from a rescue disk or Live CD, do the same, but you’ll discover that it is easier to identify the disk since you won’t be confusing it with other disks like when it is attached to another system.
Below is a sample output. In this case the device you need to recover is /dev/sda with partitions sda1, sda2, and sda5.
# cat /proc/partitions major minor #blocks name 7 0 731040 loop0 8 0 58605120 sda 8 1 248832 sda1 8 2 1 sda2 8 5 58353664 sda5
It’s probably easier to use the fdisk command to get a much clearer view of the layout of the partitions.
# fdisk -l /dev/sda Disk /dev/sda: 60.0 GB, 60011642880 bytes 255 heads, 63 sectors/track, 7296 cylinders, total 117210240 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00089bcb Device Boot Start End Blocks Id System /dev/sda1 * 2048 499711 248832 83 Linux /dev/sda2 501758 117209087 58353665 5 Extended /dev/sda5 501760 117209087 58353664 83 Linux
Step 3. Next, find the partition that is encrypted. This is where the actual data resides. Run the cryptsetup command on each partition and find the one that says something like “LUKS header information…”
# cryptsetup -v luksDump /dev/sda1 Device /dev/sda1 is not a valid LUKS device. Command failed with code 22: Device /dev/sda1 is not a valid LUKS device.
# cryptsetup -v luksDump /dev/sda2 Device /dev/sda2 is not a valid LUKS device. Command failed with code 22: Device /dev/sda2 is not a valid LUKS device.
# cryptsetup -v luksDump /dev/sda5
LUKS header information for /dev/sda5
Version: 1
Cipher name: aes
Cipher mode: xts-plain64
Hash spec: sha1
Payload offset: 4096
MK bits: 512
MK digest: 21 84 3e c0 3e 9e 23 1e 34 9b 39 05 8f b9 47 61 89 a6 2a 81
MK salt: fc ac 3d 4f 1e 3d d4 ce 66 6b d3 90 ba f4 79 a8
d9 c9 38 a0 c2 79 bc 47 71 c6 8f 49 23 46 f1 6b
MK iterations: 22500
UUID: 2c8d56ec-749f-4d95-ab39-4ea17edb4c01
Key Slot 0: ENABLED
Iterations: 90067
Salt: e4 25 ae 7c 5d 62 81 5e ea 37 95 0f 59 7b c8 7f
13 4f bc 15 70 4e 82 e1 41 db 1d 4b 65 7a de 5c
Key material offset: 8
AF stripes: 4000
Key Slot 1: DISABLED
Key Slot 2: DISABLED
Key Slot 3: DISABLED
Key Slot 4: DISABLED
Key Slot 5: DISABLED
Key Slot 6: DISABLED
Key Slot 7: DISABLED
Command successful.
From the sample output above, the partition we need to recover is /dev/sda5.
Step 4. Unlock the encrypted partition using the cryptsetup command. You will need the passphrase of the encrypted disk. I forgot to mention, you need this password to recover the data from the disk. If you don’t know the password or you forgot, then you’re out of luck. Unless you can guess the password or do a brute force password guessing, no current technology or witchcraft on earth can help you unlock the encryption.
That’s the whole idea of encrypting a disk so that in case your laptop got stolen, got lost, or got left at a restaurant, no one but you who knows the password can see your data.
# cryptsetup -v luksOpen /dev/sda5 sda5_crypt Enter passphrase for /dev/sda5: Key slot 0 unlocked. Command successful.
Step 5. After unlocking the device, check if the partition is using LVM (Logical Volume Management). If your data resides in a logical volume, you’d need to identify those volumes so that you’d know what to mount. Run the lvdisplay command to see the volumes. Below is a sample output.
# lvdisplay --- Logical volume --- LV Path /dev/ubuntu/root LV Name root VG Name ubuntu LV UUID fy5LpG-HPX7-Spe1-C92m-nzPq-B3IB-eTjCFA LV Write Access read/write LV Creation host, time ubuntu, XXXX LV Status available # open 0 LV Size 53.61 GiB Current LE 13724 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 252:1 --- Logical volume --- LV Path /dev/ubuntu/swap_1 LV Name swap_1 VG Name ubuntu LV UUID ZSwRl4-bdV1-kqJ3-V1lG-ndfZ-cdSh-P5X3kp LV Write Access read/write LV Creation host, time ubuntu, XXXX LV Status available # open 0 LV Size 1.99 GiB Current LE 509 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 252:2
Step 6. If the partition has logical volumes, enable the volume group using the vgchange command. From the sample above, the name of the volume group is ubuntu.
# vgchange -a y ubuntu 2 logical volume(s) in volume group "ubuntu" now active
Step 7a. If the partiton is using logical volumes, mount the logical volume desired. Using the sample above, we want to mount /dev/ubuntu/root.
# mkdir /tmp/disk # mount /dev/ubuntu/root /tmp/disk
Step 7b. If the partition does not have logical volumes, mount the mapped device directly.
# mkdir /tmp/disk # mount /dev/mapper/sda5_crypt /tmp/disk
That’s it. You can now view your data in /tmp/disk.
Step 8. When you’re done accessing the data, don’t forget to unmount and close (lock) the encrypted partition.
# umount /tmp/disk
# cryptsetup luksClose /dev/mapper/sda5_crypt
Include a JavaScript Source File into Google’s CodeRunner or OpenSocialDevApp
OpenSocialDevApp or Google’s CodeRunner is a great tool for testing code snippets using OpenSocial API. It allows you to easily test code by cutting and pasting your JavaScript code into a textbox. You don’t need to load a gadget XML file into the server to test out something which can be a tedious process.
This works out really well when your code is small. However, once your code gets bigger, cutting and pasting starts to get painful. Wouldn’t it be nice if you can simply save your JavaScript code into a file, say mytestcode.js, and then just cut and paste a neat single line into the textbox like this?
<script type="text/javascript"
src="http://myserver.com/mytestcode.js"></script>
To test changes in mytestcode.js, all you need to do is re-execute CodeRunner every time you make an update to the file. No need for cutting-and-pasting.
However, this OpenSocial application only accepts raw JavaScript code, so the code above won’t work since it’s an HTML code.
To fix this problem, what we need is a way to convert the HTML code above into a JavaScript code. This is where the technique of dynamically including a JavaScript file comes to the rescue.
The code below is the script-src-tag converted into a JavaScript code. If you are not familiar with this technique, see this posting for details.
function includeJavascript(src) {
if (document.createElement && document.getElementsByTagName) {
var head_tag = document.getElementsByTagName('head')[0];
var script_tag = document.createElement('script');
script_tag.setAttribute('type', 'text/javascript');
script_tag.setAttribute('src', src);
head_tag.appendChild(script_tag);
}
}
includeJavascript("http://myserver.com/mytestcode.js");
Instead of cutting-and-pasting the entire mytestcode.js into the CodeRunner’s textbox, just use the code above and make your changes directly to the mytestcode.js file on your server.
Furthermore, you don’t need to put all your code in a single file. You can break them up into multiple files and call out the include function for each.
includeJavascript("http://myserver.com/mytestcode1.js");
includeJavascript("http://myserver.com/mytestcode2.js");
includeJavascript("http://myserver.com/mytestcode3.js");
includeJavascript("http://myserver.com/mytestcode4.js");
With this technique, you can even run your entire OpenSocial application in CoderRunner!
Alvin Abad
When Firebug’s Console Object Cannot be Overridden
In my previous blog post, I’ve shown how to use Firebug’s console object to log debug messages. I’ve also shown how to perform graceful degradation if the console object is not available like when using a non-Firefox browser or if Firebug’s console is not enabled. This way, one can still retain the calls to console.log() without generating any error messages. However, there were still some issues I failed to address.
Production deployment
Normally, when you deploy code into production, you don’t want your debug calls to be included. You don’t want users to be getting those debug messages. There are two ways to fix this problem: 1) Add a step to your deployment process to search and remove all the debug calls in your code; or 2) Add some kind of a switch, say a global variable DEBUG, to decide whether to use Firebug’s console logger or redirect the call to a dummy console.log() that will not do anything. This functionality is similar to loggers like log4j in Java where you can set different settings (info, debug, etc.) to control logging without modifying the code.
Let’s say we wanted #2 above. The code below illustrates how this can be done.
if (!window.DEBUG || typeof(console) == 'undefined') {
console = {};
console.log = function(msg) {
return false;
};
}
If DEBUG is off (false), e.g., production, it will create the console object and implement a no-operation log() method. If the browser is Firefox, this will effectively override its console object, therefore, …
Wait, that is not correct. As of this writing (Firebug-v1.3.3), Firebug’s console object cannot be overridden! If you attempt to do so, you will get an error message of something like, “setting a property that has only a getter.”
Programming to an Interface and not an implementation
To fix this problem, what we should do instead is not use the Firebug console object directly to log messages. This is like the design pattern where one programs to an Interface and not an implementation.
Following this principle, we create our own implementation of the console object, let’s say mylogger. With this object, we can then control if we want to use Firebug’s console.log() or the dummy one depending on our DEBUG setting. Below is the modified version of the HTML page using mylogger.log().
<html>
<head>
<script type="text/javascript">
DEBUG = true;
// by default, set mylogger to do nothing
mylogger = {};
mylogger.log = function(msg) {
return false;
};
// if DEBUG is true and Firebug console is available, use it for logging
if (window.DEBUG && typeof(console) !== 'undefined') {
mylogger = console;
}
function demo() {
// I did lots of things here
mylogger.log("I'm here!");
// and a lot more
}
</script>
</head>
<body>
<h1>Firebug's console.log() Demo</h1>
<button onclick="demo();">Firebug console.log() demo</button>
</body>
</html>
The code above has a DEBUG switch that we set on the server side to true or false depending if we want to log messages or not. Instead of using the console.log() method directly in the demo() function, we use mylogger.log(). Since we cannot override Firebug’s console object, what we can do instead is make the decision whether to use it or not at runtime.
Debugging JavaScript Using Firebug’s Console Logger
Thanks to Firebug, debugging JavaScript has never been easier. Firebug is an add-on to the FireFox browser. It has many wonderful features. One of my favorite is the console logger. This is where you can call the function console.log() in your code to write log messages to the Firebug console. This is definitely much better than using the alert() function. It’s unobtrusive and transparent to the execution of your JavaScript program.
Below is a sample HTML page with JavaScript code demonstrating how to use this function.
<html>
<head>
<script type="text/javascript">
function demo() {
// do something
console.log("I'm here!");
// do something
}
</script>
</head>
<body>
<h1>Firebug's console.log() Demo</h1>
<button onclick="demo();">Firebug console.log() demo</button>
</body>
</html>
The code above has a button element that will print out the message “I’m here!” to the Firebug console.
Graceful Degradation
Of course, console.log() only works with the Firefox browser, with Firebug installed, and the console is enabled. Otherwise, you’ll get an error saying the console object is not defined.
For browsers other than Firefox or if Firebug is not installed, you can still retain the console.log() calls in your code and avoid getting any error messages by redirecting this call to a no-operation. To do this, create a console object and an implementation of the log() method that does not do anything. Below is a sample implementation.
// if console is not defined, e.g., Firebug console is not enabled or Non-Firefox browser
if (typeof console == 'undefined') {
var console = {};
console.log = function(msg) {
return;
};
}
The code above shows that if the console object is not defined, like when using a browser other than FireFox, a console object will be created together with a method named log() that does not do anything. That console object will then be the one that will be used by your log() calls in non-Firefox browser or when Firebug console is not enabled, which will just fail silently.
The console.log() method also supports formatted messages similar to C or Perl’s printf function. For example:
console.log("This is loop %s", loop_counter);
Firebug’s console object is not limited to the log() method. It can do other cool things. You can learn more about them here – http://getfirebug.com/console.html
Many sites have discussed this topic with great examples and illustrations. Here’s a nice one – http://ajax.phpmagazine.net/2006/05/firebug_the_future_of_javascri.html.
Note: The solution presented here is not without problems. See my other post for details.
Alvin Abad
