Unit Testing – A Problem of Perspectives

Programmers typically execute unit test cases to ensure the completeness and correctness of their completed code before submitting it to QA. Why is it that virtually every programmers’ initial code submission to QA contains a significant number of bugs? I’ve sat with programmers while they show what they believe to be completed and tested work, only to watch them stumble embarrassingly through one defect after another. It’s a rare programmer who can thoroughly and successfully unit test his/her own work. As project managers, we cannot implement processes designed around the rare programmer. For processes to be effective, they must be designed to support the most commonly expected outcomes. Does this mean that we must design processes that accommodate poorly executed unit testing? Thankfully it does not. However, to understand why, I need to explain why programmers are so bad at unit testing their own code and what impact this has to QA.

Poor unit testing makes hiring and keeping good QA staff very difficult because it’s the rare tester who can feel any sense of job satisfaction testing and re-testing the same code several times, without any realistic expectation that all the defects reported will be corrected the first time around or the second. Testers are further frustrated by finding and reporting bugs that appear to be quite obvious and should have been found during the most basic unit testing. Poor unit testing is not usually due to poor programming skills, or even a lack of conscientiousness. The problem is due to the programmer’s perspective.

When I was a programmer, I was often guilty of the same crimes that I just mentioned. To make it worse, I often incriminated myself. After completing a critical module, I was often eager to show someone how the module worked so I would call them over and begin to demo it. As soon as I began demoing my work, defects, both visual and functional became glaringly obvious to me. I was often dumbfounded how I didn’t see those defects before I went out of my way to embarrass myself.

As a project manager I occasionally ask programmers to demo their completed work to me before submitting it to QA when their assignment is critical to a project. About 9 months ago, I asked a programmer to demo his completed work to me before going to QA because I wanted to make sure the visual concept was implemented properly. He began demoing his work and immediately found a bug. He apologized and continued the demo, only to run into another bug a few seconds later. Then he noticed a couple of visual issues, one related to alignment and a misspelled word. He apologized again and asked if he could stop to fix the issues and start the demo again later in the day. About an hour later we began again. He fixed the earlier issues, but stumbled into a few more issue later in the demo. I never said a word because my commentary wasn’t needed. The programmer found all these issues without any prompting from me. Something began to nag at me about how easily the programmer found his own bugs while demoing the software to me, but was unable to see the same bugs earlier while unit testing, knowing that he was going to demo the code to me later. Then it occurred to me that something was qualitative different with the programmer while demoing his own code. The difference was that his perspective had changed from the myopic view of a programmer to the broader view of a presenter.

Programming is a myopic job. To write software, a programmer must dive deep into the logic that controls the program. Each minute piece of software requires his/her undivided attention, keeping in mind the many section of supporting code needed to program a function to work properly. Each screen that a user interacts with is made up of hundreds of these little bits of logic, each requiring all the programmers focus for a period of time. When programmers test their own work, they don’t realize how myopic their view is, so they miss obvious defects visible outside their current view. Something changes in every programmers’ perspective they demonstrate their work to others. The act of demoing lifts their focus so they see more broadly. All of a sudden, the bugs that were hidden from view become glaringly obvious.

Earlier I explained the need to design processes that support the most common outcomes. Once it dawned the reason for poor unit testing dawned on me I implemented a new step in our unit testing process. Programmers are now required to video capture each completed unit test case using video capture software such as Jing. Programmers simulate a demonstration by verbally explaining the purpose of each unit test, then describing each step as it’s performed. If they run into a bug or notice a visual defect while videoing the test case, they must stop the video and correct the issue. Once correct, they must start the video over again from the beginning. They cannot consider the code unit tested and ready for QA until they are able to flawlessly video the unit test case from beginning to end without interruption and without finding any bugs or visual defects. Once the video is complete, the video is submitted to QA. If QA finds any defects in the video, an issue is created in our issue tracking system and assigned to the programmer. The turnaround from QA back to the programmer is many times faster reviewing a video then executing test cases. Once the programmer fixes all reported issue(s) he/she is required to create a new demo video with the same requirements as before. They must flawlessly video the entire unit test scenario again from beginning to end, not just the fix, before resubmitting it to QA. Only when QA can validate that the video received is without any defects will they begin to execute all the test cases themselves. This change in our unit testing procedures and QA feedback process has dramatically improved the quality of the code delivered to QA, and reduced the number of bug report/fix cycles between QA and programmers. Programmers also benefit because they are more confident that the code they deliver to QA works correctly.

We named this new unit test procedure, “Video QA”. In the past programmers and QA suffered with what seemed to be an inevitable cycle of bugs reports and fixes. With the implementation of the Video QA process, no one seems to be suffering any longer.