How to use Git with Subversion

If you know Git, then you know very well its productivity benefits over other source code control like Subversion. If you work on a project that uses Git entirely, then good for you. However, some software projects are still using Subversion. If you end up working on a project that still uses Subversion, you don’t have to use Subversion entirely to do your work. You can use Git for almost all of your source code control tasks. And all those tasks can remain transparent to the Subversion repository and to the other people on the project that uses Subversion. All they’ll ever see is that you are checking out and then committing back to the Subversion repository like from any svn client. But in reality it was from Git. All they will ever find out is how fast you perform merges, rebases, and cherry-picks – because Git makes all that easy compared to Subversion. They will also be amazed by how clean your commits are.

To use Git with Subversion, you use the git-svn-clone command to checkout the Subversion repository. Once checked-out, everything is now in Git and you can work on your code entirely using Git. To commit changes back to Subversion, you use the git-svn-dcommit command.

Below is a sample workflow on how to use Git with SVN.

1. Clone the Subversion repository

mkdir -p ~/work/
cd ~/work/
git svn clone https://mysvnrepo.com/project/trunk project.git

The svn url is the same url you use when you run the svn checkout command. The subversion repository does not need to be the trunk. It could be a branch.

2. Create a branch off the master

cd ~/work/project.git
git checkout -b feature_branch master

By default, the cloned repository will create a master branch. Although you can work on the master branch directly, it is preferable that you create a branch off it and then use this branch to make your changes. Think of it like a feature branch. The reason behind doing this is so that you can maintain the master branch that will be in-sync with the svn repository while you have a separate branch to perform your work. This will allow you to regularly pull updates from the svn repository through the master branch and then merge or pick only some commits into your feature branch. And when you are ready to commit your feature branch to the svn repository, merge or rebase the feature branch to the master branch and commit it to subversion. The master branch serves as a gateway or staging between the Subversion repository and your Git branches.

3. Make code changes to the feature_branch

(edit files)
git add file1 file2 ...
git commit -m "trying this optionA..."

(test application)
(edit more files)
git add file1 file2 file3 ...
git commit -m "optionA didn't work. trying optionB..."
(test application)

While working on the feature-branch, you can perform as many commits as you want. In fact, this is preferable so that you can save your work and push them to a remote repository to serve as a backup. You don’t have to hurry to commit your changes back to subversion if you don’t need to just because you want to save your changes. You can use a remote git repository to serve as another site and save your work.

The commits you made can also be used if you need to revert back to them. Don’t worry about doing many commits even silly ones because when you finally commit your changes back to the svn repository, you will have the opportunity to clean them up before they get committed into subversion. This includes the comments in every commit. You will have the opportunity to modify them later when you are ready to commit to Subversion.

4. Merge changes from Subversion

If you need to get new updates from Subversion into your feature-branch, run the following steps:

# Switch to the master branch
git checkout master

# Pull changes from Subversion
git svn rebase

# Switch back to the feature branch and run rebase off master
git checkout feature-branch
git rebase -i master

The rebase command above will merge the master branch into the feature branch but in such a way that your commits are put back on top of the commits of the updates in the master. This rebase is a kind of merge where it’s like you’ve just created the feature-branch off the repository and then made your commits after it. Even though your commits were made earlier than the new updates in Subversion, a rebase will make it in such a way that your commits were made after. When you run git log to view the history, you will see it like your commits in the feature-branch were done after the updates from Subversion.

The -i option will perform an interactive operation under a VI session. It will show the history of commits in your feature branch. Below is a sample output.

pick eab79aa Made change #1
pick 97bd219 Made change #2
pick 5f7122f Made change #3
pick 33a8b6c Made change #4

# Rebase 59ef334..43910eb onto 59ef334 (4 command(s))
#
# Commands:
# p, pick = use commit
# r, reword = use commit, but edit the commit message
# e, edit = use commit, but stop for amending
# s, squash = use commit, but meld into previous commit
# f, fixup = like "squash", but discard this commit's log message
# x, exec = run command (the rest of the line) using shell
#
# These lines can be re-ordered; they are executed from top to bottom.
#
# If you remove a line here THAT COMMIT WILL BE LOST.
#
# However, if you remove everything, the rebase will be aborted.
#
# Note that empty commits are commented out

You may accept the commits as is or modify the commits by merging them together as fewer commits or one single commit for all.

Replace the word “pick” with “squash” to remove the commit from the history. Only the commit entry will be deleted but its changes will be merged into the previous commit. You can use the “s” letter as a shortcut for squash.

pick eab79aa Made change #1
squash 97bd219 Made change #2
squash 5f7122f Made change #3
squash 33a8b6c Made change #4

Repeat the rebase process every time you need to pull updates from the Subversion repository.

When doing a rebase, you can squash commits now to clean up your git commits or use the actual commit comments you wanted to appear in Subversion. You don’t have to because you can do this later when you are ready to commit to Subversion. I usually reserve this in the final rebase unless the number of commits have become too many to maintain and I see no need to revert to or use any of them.

One thing to be aware about rebase is that it will change the history of your feature-branch. If you are pushing your feature branch to a remote git repository, you will need to use the –force option. This should not be problem if you are the only one using this branch. This becomes an issue when the feature-branch is shared by more than one git use and also performing updates.

5. Merge the feature branch to the master and commit to Subversion

When you are ready to commit your changes back to Subversion, merge your feature branch to master and commit to svn.

git checkout master
git svn rebase
git merge feature_branch
git svn dcommit

6. Continue using the feature branch

You can still continue to use your feature-branch even after committing it to Subversion. Same process applies. You update your master from Subversion and rebase your feature branch off it. This is to ensure that your feature branch is updated with the Subversion repository and that any new commits you make goes on top just like a freshly created branch off the Subversion repository.

# make sure you're on master branch
git checkout master

# Update master branch with latest from Subversion
git svn rebase

# switch to feature-branch and rebase from master
git checkout feature-branch
git rebase -i master

Merging or cherry-picking changes between Subversion branches

Another most commonly used workflow in source code control is merging between branches. This could be a full merge or partial merge like selecting only certain changes (commits) from one branch and putting them to another. This workflow is commonly used when you need to port some fixes from one branch to another without necessarily merging everything. All of this be done easily using Git.

To do this in Git, do a git-svn-clone of each of the branches you need to merge or cherry-pick commits. This will create a Git repository for each Subversion branch. For each git repository, create a branch for each master and then push them to a common Git repository. Once they are in a common repository, you can then query the commits for each and perform cherry-pick merges. Below is a step-by-step example.

1. Clone each Subversion branch

# Clone Subversion branchA
cd ~/work
git svn clone https://mysvnrepo.com/svn/branches/branchA branchA.git

# Clone Subversion branchB
git svn clone https://mysvnrepo.com/svn/branches/branchB branchB.git

# create a Git branch for each
cd ~/work/branchA.git
git checkout -b branchA master

cd ~/work/branchB.git
git checkout -b branchB master

You now have two directories and each is a Git repository for each Subversion branches A and B.

2. Create a common Git repository

cd ~/work
git init common.git

3. Set common.git as a remote repository for branchA.git and branchB.git

cd ~/work/branchA.git
git remote add origin ~/work/common.git

cd ~/work/branchB.git
git remote add origin ~/work/common.git

4. Push branchA and branchB into the common repository

cd ~/work/branchA.git
git push origin branchA

cd ~/work/branchB.git
git push origin branchB

5. Go to the common repository

From the common repository, if you need to merge changes from branchB into BranchA, checkout branchA and merge or cherry-pick commits from branchB.

cd ~/work/common.git
git checkout branchA

# view commits in branchB to find which one you wish to merge
git log branchB
git cherry-pick e065b0230d7b368cfb534fc1818d55adc8911e2c

The cherry-pick command above will merge that commit in branchB into branchA.

6. Commit the merged changes in branchA into Subversion

Go to branchA.git repository and pull the merged changes from common.git

# Go to branchA.git and run git-fetch
cd ~/work/branchA.git
git fetch
git checkout branchA
git pull origin branchA

7. Commit merged changes in branchA into Subversion

To commit back to Subversion, first update the master branch with the latest from the Subversion repository by running git-svn-rebase.

cd ~/work/branchA.git
git checkout master
git svn rebase

Switch to branchA and perform a rebase off the master so that your commits will go on top of the latest SVN commits.

cd ~/work/branchA.git
git checkout branchA
git rebase -i master

Finally, merge branchA into master and commit to svn.

git checkout master
git merge branchA
git svn dcommit

It looks complicated with a lot of steps. But I assure you that once you have understood the concept, you’d realize how elegant and powerful merges and cherry-picks can be done using Git.

You don’t need to clone for each Subversion branch. You can run a single clone for both branches. The git-svn-dcommit will cover for all branches (all or selectable) in the Subversion repository. This should work well too and could be simpler since you don’t have to maintain many repositories. But I prefer separate repositories for each svn branch so that I can maintain isolation between svn branches while working in Git.

How to Remotely Access the Initial Ramdisk of an Encrypted Linux System

If you use cryptsetup to encrypt your Linux root file system, the default setup requires console access to enter the password and boot up the system. If your system is remote or doesn’t have console access, you will need to find a way to get remote access to the console.

If you cannot install a remote console or if your system doesn’t allow one, for example, instances in Amazon Web Services (AWS), you can still obtain remote access to enter the password by installing an ssh server in the initial ramdisk. Once you can login by ssh into the initrd, you can then supply the password to decrypt and boot up the system.

Below is a step-by-step guide on how to install an ssh server in the initial ramdisk, login to it, and enter the password to boot up your encrypted Linux system. The guide is created for Ubuntu and Debian systems and using DHCP to establish network access. The idea can be easily applied to other distributions like CentOS or RHEL.

The guide shows both a manual process and a script to automate the steps. It is recommended to try the manual steps first so that you will have an understanding of the underlying process. This can help in troubleshooting the script in case you encounter problems.

You will need access to GitHub to download the scripts.

Install dropbear

apt-get install dropbear

Get your public ssh-key and copy it to this file on your remote system:

/etc/initramfs-tools/root/.ssh/authorized_keys

If you don’t have an ssh-key pair, you can create one using the ssh-keygen command

ssh-keygen

This command will generate two files:

~/.ssh/id_rsa
~/.ssh/id_rsa.pub

The id_rsa.pub file is the public key where you will need to copy to the remote system as described in #2 above.

The id_rsa file is the private key where you will need to copy to the client that will login by ssh.

If you are using an AWS instance, copy the public-key of your instance into this authorized_keys file.

Update the initial ramdisk image

Run this command to update the initial ramdisk file:

update-initramfs -u

Boot up the system

When the system boots up, connect to the system using ssh and the private key you created above.

ssh -i ~/.ssh/id_rsa root@ip-address

You don’t actually need to supply the -i option for that key-file since that is the default. This is just to illustrate that you need to use the matching key-file to login by ssh.

Enter password remotely

Once you gained access to the system by ssh, execute the following steps in sequence:

  1. Kill the cryptroot process.Sample run:
    # pidof cryptroot
    161
    
    # kill -9 161
    
  2. For Ubuntu systems, wait for the cryptsetup process to terminate. You can monitor this process by running this command:
    # pidof cryptsetup
    

    This could take about a minute to complete.

  3. For Debian systems, wait for the /bin/sh -i process to come up. This is also applicable for Ubuntu.
    Example:

    # ps | grep '/bin/sh -i' | grep -v grep
      214 root      4620 S    /bin/sh -i
    

    This could take about a minute to complete. Take note of the ID of this process when it comes up.

  4. Once cryptsetup has terminated (for Ubuntu only) or the /bin/sh -i process has come up, run the cryptroot command:
    /scripts/local-top/cryptroot
    

    This command will prompt for the password to decrypt the system.

    Sample run:

    # /scripts/local-top/cryptroot
    
    /scripts/local-top/cryptroot: line 1: modprobe: not found
    Unlocking the disk /dev/disk/by-uuid/b1aa9e88-c344-4922-a0a6-d3f7c52f947a (sda5_crypt)
    Enter passphrase:
    
       Reading all physical volumes.  This may take a while...
      Found volume group "ubuntu-vg" using metadata type lvm2
      2 logical volume(s) in volume group "ubuntu-vg" now active
    cryptsetup: sda5_crypt set up successfully
    
  5. After entering the password, kill the process-id of the /bin/sh -i you saw earlier.Important Note: At this point, your terminal is kind of screwed up. You won’t be able to see what you are typing. So carefully, type the kill command using the /bin/sh -i process-id you got earlier and hit the enter key.
    # kill -9 214
    
  6. Type ctrl-d to disconnect from your ssh access. At this point, the system should be on the way up booting the system. If you have access to the console, you can observe how long it takes for your system to boot up. You can use this to estimate the time it takes for your system to fully come up after you enter the password remotely and start accessing the system.

Automating the remote password process

The remote password process above seems to be a lot of work just to enter the password and boot up the system. If this doesn’t sound fun to you, I wrote a couple of scripts to automate the steps.

  1. Download these two scripts from GitHub:
  2. Save the files at this location:
    /root/cryptroot.sh
    /etc/initramfs-tools/hooks/my_initrd_hook
    

    Make them executable:

    chmod +x /root/cryptroot.sh
    chmod +x /etc/initramfs-tools/hooks/my_initrd_hook
    
  3. Update the initial ramdisk image
    update-initramfs -u
    
  4. Boot up the system and login remotely by ssh
  5. At the shell prompt, run this command and enter the password when you get the prompt:
     /root/cryptroot.sh
    

    Sample run:

    # /root/cryptroot.sh
    
    cryptroot has terminated.
    Waiting for /bin/sh -i to start...
      277 root      4620 S    /bin/sh -i
    sh -i has started.
    
    Unlocking the disk /dev/disk/by-uuid/b1aa9e88-c344-4922-a0a6-d3f7c52f947a (sda5_crypt)
    Enter passphrase:   
    
    Reading all physical volumes.  This may take a while...
      Found volume group "ubuntu-vg" using metadata type lvm2
      2 logical volume(s) in volume group "ubuntu-vg" now active
    cryptsetup: sda5_crypt set up successfully
    Terminating sh -i process...
    Press ctrl-d or type exit to disconnect from initrd dropbear.
    
    #
    

Show the History of CVS Commits Similar to Git or SVN

I wrote a simple script that will display the history of CVS commits similar to the way Git or SVN do. You can download the script from Github: https://github.com/alvinabad/cvs-utils/blob/master/cvs-history.py

Basic usage

cvs-history.py [-b BRANCH] module or files

For more options, see: https://github.com/alvinabad/cvs-utils#cvs-historypy

Background

If you use the “cvs history” command to display the history of commits in CVS, you will get an output that is not sorted chronologically.

Here’s a sample run:

$ cvs history -a -c hello_cvs/
M 2015-02-28 23:52 +0000 alice 1.6     hello1.py hello_cvs == ~/main/hello_cvs
M 2015-02-28 23:48 +0000 alice 1.3     hello2.py hello_cvs == ~/main/hello_cvs
M 2015-03-01 17:05 +0000 alice 1.3.2.1 hello2.py hello_cvs == ~/workspace/hello_cvs
A 2015-03-01 00:38 +0000 alice 1.1.2.1 hello3.py hello_cvs == ~/feature_branch/hello_cvs
M 2015-03-01 00:42 +0000 alice 1.1.2.2 hello3.py hello_cvs == ~/workspace/hello_cvs
M 2015-02-28 23:10 +0000 alvin 1.2     hello1.py hello_cvs == ~/src/hello_cvs
M 2015-02-28 23:13 +0000 alvin 1.3     hello1.py hello_cvs == ~/src/hello_cvs
M 2015-02-28 23:18 +0000 alvin 1.4     hello1.py hello_cvs == ~/src/hello_cvs
M 2015-02-28 23:23 +0000 alvin 1.5     hello1.py hello_cvs == ~/src/hello_cvs
M 2015-03-01 00:37 +0000 alvin 1.7     hello1.py hello_cvs == ~/src/hello_cvs
A 2015-02-28 23:20 +0000 alvin 1.1     hello2.py hello_cvs == ~/src/hello_cvs
M 2015-02-28 23:23 +0000 alvin 1.2     hello2.py hello_cvs == ~/src/hello_cvs
M 2015-03-01 00:44 +0000 alvin 1.4     hello2.py hello_cvs == ~/src/hello_cvs

As you can see from the output above, it is hard to figure out the order when the changes happened. The output gets worse if you have many branches. The output displays all the history of commits in all branches and mixed altogether.

How to Use LUKS to Encrypt a Disk Partition

You can use LUKS to encrypt a partition of a disk drive or USB. If you store sensitive information in portable drives it’s more compelling than ever to protect them using encryption since they carry a high risk of getting lost or stolen.

LUKS/dm-crypt is a good choice for encrypting Linux devices. It’s usually pre-installed in most Linux distros and if not, it’s easy to install using YUM or APT.

Here are seven easy steps to encrypt a disk partition:

Step 1. Create the disk partition you wish to encrypt. For example, let’s say you have a USB drive and it’s connected to /dev/sdb. The partition you’d want to create would be /dev/sdb1.

# fdisk -l /dev/sdb
Disk /dev/sdb: 512 MB, 512483328 bytes
255 heads, 63 sectors/track, 62 cylinders, total 1000944 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1              63     1000943      500440+  83  Linux

Step 2. Encrypt the partition

# cryptsetup -q -y luksFormat /dev/sdb1
Enter LUKS passphrase: 
Verify passphrase: 

Step 3. Map a logical partition

# cryptsetup luksOpen /dev/sdb1 sdb1_crypt
Enter passphrase for /dev/sdb1:

This will create a device mapper:

# ls -al /dev/mapper/sdb1_crypt
brw-rw---- 1 root disk 253, 5 Sep 23 11:53 /dev/mapper/sdb1_crypt

Step 5. Format the encrypted partition

# mkfs.ext3 /dev/mapper/sdb1_crypt 
mke2fs 1.42.5 (29-Jul-2012)
Filesystem label=
OS type: Linux
Block size=1024 (log=0)
Fragment size=1024 (log=0)
Stride=0 blocks, Stripe width=0 blocks
124928 inodes, 498392 blocks
24919 blocks (5.00%) reserved for the super user
First data block=1
Maximum filesystem blocks=67633152
61 block groups
8192 blocks per group, 8192 fragments per group
2048 inodes per group
Superblock backups stored on blocks: 
	8193, 24577, 40961, 57345, 73729, 204801, 221185, 401409

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done 

Step 6. Mount the encrypted partition:

# mkdir /mnt/sdb1
# mount /dev/mapper/sdb1_crypt /mnt/sdb1

Step 7. When done unmount the logical partition and close (unlock) the encrypted partition

# unmount /mnt
# cryptsetup luksClose sdb1_crypt