FBI ends faltering effort to overhaul software | CNET News.com

FBI ends faltering effort to overhaul software | CNET News.com

Yet another high profile software engineering project failure. I look forward to the details when they come out.

ACM Queue - Too Darned Big to Test - Testing large systems is a daunting task, but there are steps we can take to ease the pain.

ACM Queue - Too Darned Big to Test - Testing large systems is a daunting task, but there are steps we can take to ease the pain.

I liked this article on testing. It makes a few good points and makes people aware of some useful testing concerns and approaches.

Here is a short list of the things that I think are key to good quality assurance:

• QA is more than testing. Say that 10 times.


• You cannot test-in quality, you need to build-in quality. Say that 10 times.


• Start with high quality requirements. Review requirements. Demand specifics. Apply relevant knowledge of the domain.


• Have an actual QA plan document that sets quality goals, chooses appropriate strategies, and allocates resources.


• Design for quality and testability. The architecture must separate core business objects from the UI. In general, to aid testability, you separate components based on testing technique. Unit tests work well for logic-only business objects. System tests work better for UI interactions.


• Design for test access. It should be possible to construct a business object in any particular valid state, perform one operation on the object, and then verify the correctness of its state. This may require exposing additional methods to the test code that you would not want to expose to other components of the system.


• Build in tests in the form of input validation, assertions, preconditions, loop invariants, class invariants, and postconditions. Even if much of this is never executed, the process of writing these conditions and reviewing them greatly improves quality.


• Review the design and code. This helps catch errors and familiarize everyone on the team with the implementation. Do a little bit at a time. Build up checklists of common errors.


• Static analysis tools help detect common errors. More powerful analysis tools can do proofs.


• Unit tests should be written by developers at the time of writing each unit. Measure code coverage and argue for specification coverage. E.g., show that you cover boundary conditions and equivelence classes. When possible, use simple test oracles rather than hard-coded expected answers. Run unit tests very frequently, e.g., every night.


• Integration testing: this is not my favorate because it requires extra work. If possible, try to minimize this type of testing by simplifying interfaces and raising component quality. Try to combine a few components at a time in a test harness. Cover each configuration parameter and try all pairs of 2 configuration parameters, if practical. Fall back to integration testing when system testing indicates problems.


• System testing: test the entire system end-to-end for a given use case from the user's perspective. This is a good time to think about validation again. UI testing automation can help.


• Staging: test the system in a realistic setting with real customer data on hardware and networks that are identical to the production environment.


• Beta testing with key customers using real data in a real usage setting.


• Field failure reports capture detailed information about failures in deployed production software.


• Defect tracking: enumerate the defects and track them to resolution. Every defect gets tracked.


• SPI / Quality management: look back on defect reports and review meeting notes to find patterns of defects. Add additional checklist items to review checklists, and refine standards for design and implementation. Each defect could be the tip of an iceberg, look for related defects that share the same underlying error in developer understanding or the same symptom. Track testing coverage and continue to raise the expectation of coverage.