Tests Passing; Solution Doesn't Work

Tests Passing; Solution Doesn't Work One of the more frustrating project situations is to find that tests are reported as passing but the solution under the test still doesn't work for observers outside the test team. In these cases, you want to identify why the tests do not seem to find the same issues that other users do. Figures 9.159.18 are examples of this case. High Bug Find Rate Frequently you see a high test pass rate but still see a large incoming bug rate (or worse, customers or beta users are reporting lots of bugs that testing seems to be missing). This can occur for several reasons: The tests might be too gentle for this stage of the solution. In early iterations, gentle tests are good, but as the solution matures, tests should exercise broader scenarios and integrations. These tests might be missing. Tests might be stale or be testing the wrong functionality. It might be time to switch test techniques. (See Chapter 7.) Consider Figures 9.15, 9.16, and 9.17. Figure 9.15. On the Quality Indicators chart, the test pass rate is high, but active bugs are also high. Figure 9.16. Correspondingly, on the Bug Rates chart, active bugs are high because find rate stays high. Figure 9.17. Tests aren't finding the bugs. On this report, many of the bugs found have no corresponding test. This might be a sign that testing is looking elsewhere. And, if you are expecting regression testing to prevent their undiscovered recurrence, you will need regression tests that you don't have yet. [View full size image] Tests Are Stale Tests do not necessarily evolve at the same rate as the code under test. This risk is present especially when tests are heavily automated. In this situation, you see high test pass rates with ongoing code churn and diminishing code coverage (see Figure 9.18). Figure 9.18. This Quality Indicators chart shows a high rate of code churn and a low rate of code coverage from testing, yet test pass rates remain high. This suggests that the tests being run are not exercising the new code. Don't be lulled by the high test pass ratethese tests are clearly not testing all the new development work. [View full size image]