Page Operations
Browse SpaceSingle-point-of-entry to requirements
Assumption: Requirements are expressed as user stories (US).
We want single point of entry to all requirements and we want a 1-1 mapping to the tests that ensure that they are fulfilled. A wiki makes the USs easily available and it is easy to add mapping (URLs) to different types of tests and test reports.
For the component level cycle we suggest adding links to; the test report for the given US/test group* and a link to the test source code and javadoc.
The same applies to the system level cycle, but the links don't necessarily be to source code, but can be to a description of a manual test procedure or a combination (link to source code + recipe) for semi-automatic tests.
Component level cycle
Figure: US -> a group of tests => component release
White-box tests can be put into groups with the group annotation in TestNG. Each US can then be mapped to one such group. And the Maven release of the module/component ensure the mapping between the tests and these groups.
It should be possible to automate all tests in this life cycle. Enterprise Maven Infrastructure can provide the necessary tool support for Maven-based projects.
System (or subsystem) level cycle
Figure: US -> group of tests -> set of released artifacts + configuration => system release
For black-box, integration tests the picture is a bit more complex. First, these tests require that each component is released. In addition we need to lock down the configuration. In other words, the configuration needs to be version controlled as well. Finally, to have complete traceability the tests themselves also need to be version controlled.
Therefore
 | A set of released artifacts + a configuration with a fixed version + a set of tests with a fixed version = a system release |
As we strive to make released artifacts backward compatible, the US (and therefore the tests) can be said to be valid for a _ version range_ (one range is valid per artifact).
Note! Many consider test data as a separate dimension, and it is a separate dimension. However, we can avoid an extra level here by expression the set of acceptable test data as a user story. The user story will thus have its own matching group of tests. This ensures that any change in the set of acceptable testdata which demand a change in a test will also require a new system release.
In an ideal world all these tests could be automated as well. In the real world, this is seldom possible. Tests depend on different types test frameworks, and both automated, semi-automated and manual tests can (and often must) be used to cover all requirements.
TODO: Which tools are relevant and what can they offer?
There was a debate about this on last XP meetup. I promised I would take some notes, but I was (as expected) a bit too engaged in the debate to note anything down. Some keywords:
- Fishbowls are fun
- The voice of testers
- Testers vs developers
- BDD, RSpec, Fitnesse were the tools
- Not a very technical oriented debate..
- One size doesn't fit all
- You can't test everything
- Bad architecture is hard to test
- We should try harder
- Architectural/technical tests vs requirement tests?
- Acceptance test. Getting the good feeling before a release.
- Let them test as much as they want. Eventually, they'll trust your safety net (or tests) enough to only manually test what is absolutely necessary.
Oh, just have to paste this in:
Posted by Mapping between user requirements and tests
While made more explicit in BDD methodology, the concept of using the name of a method to reveal its purpose is nothing new. In fact, it might be considered a common, good programming practice. Nevertheless, by putting some effort into good naming, both code and tests alike become easier to interpret for both technical and non-technical people.
A mapping between tests and user requirements (for instance in the format of user stories) can be made more explicit if tests are grouped according to which requirement or user story they are relevant for. A test report sorted by group can thus be used as a means of communication between the customer and the developers. It should be possible to define a set of groups that (combined) defines the acceptance criteria for the system. A high degree of involvement and trust between customer of development team is required for this tactic to work, but the technical support already exist. (TestNG has support for test reports sorted by groups.)
Note: The acceptance test report and the inverse of this report should make it possible to identify functionality (and tests) that can safely be removed. And as we all know, less code means less bugs, so this bi effect has potential.
Perhaps BDoc can be used?
It does seem a bit cumbersome to have all user stories as enums though.