Characteristics of an effective test suite
A useful suite of tests must reduce the overall effort in verifying that a product exhibits high structural and functional quality. A primary driver for any development team is that the test suite verifies high quality for each successive build. The test suites must be readily maintainable to prevent future regression failures.
Consider a common approach to testing. During feature development, a team runs a particular test case only a few times. After integrating the feature into a build, the test case is run after each code change to verify functional integrity and compliance aligned to a business rule or use case. Whether maintained by the build team or handed off externally in the future, the feature set will expand gradually with the regression suite expanding accordingly. Maintaining the regression test suite benefits the team, boosting the level of confidence in the build’s short-term and long-term quality.
A test suite defines the behavior expectations of a system as it is put to use in various test cases.
System-level test cases define user stories.
Unit tests define details on business rules.
Integration tests define contracts and integration flows that indicate all major dependencies.
While conventional requirements specification often becomes dated quickly, test cases are more dynamic and correspond more closely with all the aspects of the software design. As the team maintains the tests, they remain current with each build. For this reason alone, a good test suite should readily provide accurate specifications for a software system.
Building faster by testing early
System changes may include functionality that supports new use cases, additional steps in the workflow, and integrations with third-party software. The effort to manually verify software changes depends on scope and complexity. The time spent varies greatly, taking anywhere from a few minutes to hours. It's best to find bugs or problems soon after completing development changes since less effort will be needed to find a remedy earlier in the build.
Commonly, there is a deferral of testing. Instead of a developer, a tester eventually identifies, reproduces, and logs defects. Comparatively, most of the manual effort to identify, isolate, and fix the bug often doubles, at least for both testers and developers.
However, what if it was feasible to get feedback while still working on the code? If a developer could get immediate feedback, the adjustment could be part of the initial coding change effort, with no defect ever arising. The preliminary testing would alleviate much of the context-switching from development to QA. The feedback would be applicable to functional, structural, and business use-case testing.
In addition to the QA effort, the development effort to refactor will significantly increase if quality feedback is deferred to future code review. Accordingly, the test suite needs to offer the ability to provide immediate feedback, leading to more efficient use of resources.
Testing for efficiency and resiliency
Many tests are code or pseudo-code, for which there may be a high ongoing cost to maintain. With many aspects to test maintenance, the coupling of tests to system internals tends to be the most problematic. Tightly coupling a test suite to the internals of the system causes much fragility. Each change to the code base necessitates a change to one or more corresponding tests, even with no externally detectable behavior changes. Enforcing a high degree of test coverage becomes a big ask. Not surprisingly, most teams tend to avoid a high number of tedious test changes.
As complexity increases in a tightly-coupled test environment, many reach a point where they consider deleting extraneous test cases. Say, an incorrect identification of module boundaries needs testing, or such modules are merely leaky abstractions that expose all of the internals through the APIs.
The thinking is this: it takes more time to fix the tests than implement the code changes. It is, of course, quite impossible to completely decouple all tests from the software system. However, the result in the suite of end-to-end tests would not give enough confidence.
Ideally, it is best to pursue a loosely-coupled test suite that exhibits the minimum amount of coupling in critical areas.
The characteristics of an ideal test suite include:
all functionality works according to expectations,
existing functionality works (even when features are extended or new features are added),
a sustainable system architecture,
an accurate specification of the system,
a quick feedback loop,
and minimal maintenance effort.