ICS 121: Test Suites

A test suite is a design document for your testing.
Synthesis: Put together a complete set of test cases that you are able to reliably carry out
Analysis: Have you achieved your testing goals? What is your test coverage?
A test suite organizes dozens or hundreds of test cases
- By quality goal: Functional, Scalability/Performance/Volume, Robustness/Error handling, Usability
- By use case or feature
- By system component
- By priority for each milestone
Test cases
- Describe the purpose of the test
- Describe test inputs
- Steps to carry out the test
- Success criteria: expected results or test oracle
ReadySET test suite template

Test coverage is a metric that helps evaluate how well a test suite assures a quality goal.
Specification-based coverage metrics:
- Specification coverage: Is every requirement tested?
- UI coverage: Is every screen seen? Every option selected? Every button pressed? Every link followed?
- API coverage: Is every public function called? Every parameter exercised?
Implementation-based coverage metrics:
- Statement coverage: Is every source code statement exercised at least once?
- Branch coverage: Is every branch taken both ways? Consider short-circuit boolean operators.
- Path coverage: Is every combination of branches exercised?
- Example from Maven.

Try to cover all possible input values:
- Boolean or enumeration: test all possibilities
- Integer: identify equivalence classes, test one value in each class, or test both sides of each class boundary
- Reals: equivalence classes, plus test minor deltas for accuracy
- String: empty, multiple character classes, short and long, same as other strings or different
- Dates: boundaries, past/present/future, multiple formats
- Complex objects: break them down into smaller values, e.g., consider a Point to be an X and a Y
- Anything: make sure to include both valid and invalid values
- Anything: make sure to include both reasonable and unreasonable values
- Combinations of parameters: equal, unequal, related, unrelated, pointers to the same objects
It can also be useful to try to force the system to generate specific outputs: maximum, minimum, valid, invalid, empty, small, large
Make sure that you know the corrected expected output. A "test oracle" is code that computes the expected result in a known-good way, it may work backwards.
Try to cover all possible statements/branches:
- Look at the condition of each branch
- Decide what variable value would be needed to make it true or false
- Work backwards to find a test input that achieves those variable values

First, decide on the overall goals and plan.
Design unit test suite to match the API of each unit. E.g., define test cases for each public method in your class. Add unit test cases to improve coverage.
Draw functional system test suite from requirements use cases, then add test cases to improve coverage.
Focus on usability review rather than controlled usability testing, unless you have the resources and experience needed. Gather and act on informal feedback from users.
Design performance and scalability test suites by identifying key parameters and scaling them up or down, in isolation or combination. Review design and code to estimate performance in terms of parameters, then test the accuracy of your estimates.