QaTraq lessons learned

One of the essential tools for a software quality team is a test management tool. Test management is simply the process of storing test cases in a database, and organizing them in a way that makes it easy to plan, coordinate, and measure the testing activity.

I've worked with many test management tools in my career in software quality, from home-grown tools to commercial ones. But one of the most useful I've worked with is QaTraq, which I used at my last employer, Haydrian Corporation.

QaTraq was a good fit a Haydrian for several reasons. We had a small team of 3-4 developers and we needed to be coordinate and measure our testing. We didn't want to spend a lot of money on commercial test tools, but we preferred an open source product with support behind it. Our product was an appliance with releases several times a year, and smaller patch releases more frequently.

QaTraq is built around the concept of test plans. Test plans contain test designs which in turn contain test scripts. Each level is required. This hierarchy is deep enough to be configurable for most needs, but the fact that you have to fill in every level of the hierarchy creates extra work. For projects that have named release versions (e.g. version 2.2.2) and which release no more often than once a week, the overhead of copying or creating a test plan for each release is manageable. For projects that release every day, the overhead may be too difficult to manage.

One practice I can recommend is using descriptive and distinct names for test plans and other entities. When your test cases get into the hundreds or thousands, test case names like INSTALL_00001 aren't going to be that meaningful. Names should also be short so they will fit into the QaTraq drop down menus.

Another consideration with QaTraq is the fact that once you execute a test case, you can't delete it because it is tied to a test result. You can only remove it from a script. This makes it necessary to have a strategy for marking test cases as deprecated so they don't get added to future test plans.

One approach is to have a "master" test plan which is a superset of all known good test cases. When deprecating a test case, it should also be removed from the test script in both the currently executing test plan, as well as in the master test plan. The Templates and Sets functionality in QaTraq Pro provides this functionality in a more built-in way.

The reports built into QaTraq are useful but are tightly tied to the concept of a Product and its Component, so it is important to think about this hierarchy as well and not make it too complicated. It may be necessary to write custom queries and reports external to QaTraq to get just the information you want.

Automated test cases cannot be run from QaTraq, but manual instructions for running an automated test case can be stored in QaTraq and the results can be entered manually. If the automated tests are constantly changing, it can take quite a lot of effort to keep QaTraq synchronized, but marking automated tests as pass or fail manually takes very little time.

Overall, I would recommend QaTraq for a software development organization that has defined releases no more than once a week most of the time. For products that do not have defined releases, or which release daily, as many Internet companies do, it is possible to use QaTraq in a "light process" that doesn't use some of the built in features. More on that in a later article.