This blog is about 7 years over due. Around 2007 I worked with Sam Pierson on a project at PivotalLabs. It was a Rails app which provided xml payload to clients. Those were the days when alias_method_chain was used to render either html or xml without having to change most of the code logic. The test coverage was quite high at about 2.5 (2.5 lines of test code for every line of production code). One day after upgrading rails, several hundred tests on xml payload were broken. We spend sometime digging into Rails source code hoping to find out why without much luck. One morning Sam came in and said we could use git bisect to run a test script that will helps us to determine which commit in rails breaks our tests.
git-bisect performs binary search for the change that introduces a bug. Once you start the process with
git bisect start, and tagging the current version to be bad with
git bisect bad, you may choose a version where the bug is not present with
git bisect good v1.1.1. Now git-bisect will check out the middle of the revision and developer should test whether the bug is present or not and tag that revision accordingly. As you can imagine, for a large open-source project like Rails, it may take a long time to perform bisect. Luckily git-bisect also supports a
run command that takes in a script, which can be used to run tests to determine whether the bug is present or not. The following script is used in our git-bisect. One thing to note is that this is written for an early version of rails so some of the semantic is no longer applicable. The key thing is to run your test and return one of the 3 codes based on the result of the test run. If test succeeds, return 0; if test fails, return 1; if source code is untestable (such as missing gem), then return 125.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 montcalm:~/workspace/fixrails $ more testit.sh #!/bin/bash cd $HOME/workspace/fixrails rm -rf testapp output=`( ruby rails/railties/bin/rails --quiet testapp cd testapp ln -s ../../rails vendor/rails # Setup your test case here script/generate rspec --quiet script/generate rspec_scaffold --quiet user name:string rm -rf spec/views rsync -a ../shared/ . rake db:migrate > /dev/null 2>&1 rake spec ) 2>&1` #echo $output echo $output | grep '[0-9]+ examples, 0 failures' && echo "it was good" && exit 0 echo $output | grep '[0-9]+ examples, ' && echo "it was bad" && exit 1 echo $output && echo "Source code was untestable" && exit 125
- We used regex to grep test status based on rspec stdout. This output maybe different based on versions of rspec.
- We made an innocent assumption that without the word ‘examples’, rspec fails to run for whatever reason.