Skip to end of metadata
Go to start of metadata

JigZaw Multidimensional Test Categorization

Classic test categorization, inspired by the V-model, often use unit, integration, system and acceptance tests as test groups. The tests are hierarchical and overlapping.

In JigZaw a multidimensional test categorization is used and a test can be given multiple tags. Overlap between tests should be kept at a minimum. The purpose of this test categorization is to support communication when designing, implementing and running tests. These characteristics can then be used to determine who, where and when a given test should be run. See Timeline for the details.

Dimension Description
execution time fast or long-running tests, which restricts when it can be run
data separate data-driven tests from regular functionality tests
distribution in process, out-of process, distributed, in test controlled servlet- or application container
platform run in environment similar/equal to production environment, including operating system, the real database, JMS platform, JNDI, app-server platform, ESB platform et al.
integration integration with external services or availability of sensors or hardware components
Examples
Test description Execution time Data Distribution Platform Integration
"Unit test" (plain business logic) Fast Controlled data In-process, white-box same Java and OS as production independent of external systems
Algorithm correctness, complete verification depends Controlled data In-process   no
Logic which use algorithm, focus in correct behaviour for a specific use case depends Controlled data In-process   no
REST endpoints, verify http responses Medium Controlled data out-of-process, black-box in-mem database independent of external systems
Record and replay (logic AND database) Slow Data-driven In-process, white-box real database (database)
"System test": Multiple applications, queue and db infrastructure, verify functionality end-to-end Slow Controlled data out-of-process (distributed) production-like environment yes, plenty
"System regression test" - nightly/weekly re-run of a set of the most common end-to-end process flows Very slow Controlled load, and result verification IntTest / QA environment PreProd full PreProd platform

Other test categorizations


Back to JigZaw Design Principles and Drivers

Labels:
None
Enter labels to add to this page:
Please wait 
Looking for a label? Just start typing.
  1. Oct 29, 2010

    If you want to test your database implementation you will need a database. If the database is unavailable, all database tests will fail. A developer with in-depth knowledge of the application will probably quickly come to the conclusion that the database is unavailable if a hundred DAO-tests fail, but why let 100 tests fail due to a single failure? Why not communicate that all these test depend on the database being available more directly?

    The solution is to introduce dependencies between tests. E.g., create at testDatabaseConnectionTest and let all DAO-tests depend on it. All DAO-tests can thus be skipped if the testDatabaseConnectionTest fail. This makes the test failure easier to debug.

    See page 103 in Next Generation Java Testing: TestNG and Advanced Concepts for a thorough discussion of the concept.