Git Bisect #FTW


07-11-2014


Filed under: git

This blog is about 7 years over due. Around 2007 I worked with Sam Pierson on a project at PivotalLabs. It was a Rails app which provided xml payload to clients. Those were the days when alias_method_chain was used to render either html or xml without having to change most of the code logic. The test coverage was quite high at about 2.5 (2.5 lines of test code for every line of production code). One day after upgrading rails, several hundred tests on xml payload were broken. We spend sometime digging into Rails source code hoping to find out why without much luck. One morning Sam came in and said we could use git bisect to run a test script that will helps us to determine which commit in rails breaks our tests.

git-bisect performs binary search for the change that introduces a bug. Once you start the process with git bisect start, and tagging the current version to be bad with git bisect bad, you may choose a version where the bug is not present with git bisect good v1.1.1. Now git-bisect will check out the middle of the revision and developer should test whether the bug is present or not and tag that revision accordingly. As you can imagine, for a large open-source project like Rails, it may take a long time to perform bisect. Luckily git-bisect also supports a run command that takes in a script, which can be used to run tests to determine whether the bug is present or not. The following script is used in our git-bisect. One thing to note is that this is written for an early version of rails so some of the semantic is no longer applicable. The key thing is to run your test and return one of the 3 codes based on the result of the test run. If test succeeds, return 0; if test fails, return 1; if source code is untestable (such as missing gem), then return 125.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
montcalm:~/workspace/fixrails $ more testit.sh
#!/bin/bash

cd $HOME/workspace/fixrails
rm -rf testapp

output=`(
ruby rails/railties/bin/rails --quiet testapp
cd testapp
ln -s ../../rails vendor/rails

# Setup your test case here
script/generate rspec --quiet
script/generate rspec_scaffold --quiet user name:string
rm -rf spec/views
rsync -a ../shared/ .

rake db:migrate > /dev/null 2>&1
rake spec
) 2>&1`

#echo $output
echo $output | grep '[0-9]+ examples, 0 failures' && echo "it was good" && exit 0
echo $output | grep '[0-9]+ examples, ' && echo "it was bad" && exit 1
echo $output && echo "Source code was untestable" && exit 125

Note
- We used regex to grep test status based on rspec stdout. This output maybe different based on versions of rspec.
- We made an innocent assumption that without the word ‘examples’, rspec fails to run for whatever reason.