Static Testing and Static Analysis
Why, when and what to review?
Any document can be reviewed. For instance, requirement specifications, design specifications, code, test plans, user guides, etc. Ideally review as soon as possible.
Costs – on-going review costs of approx. 15% of development budget. The cost of reviews includes activities such as the review process itself, metrics analysis and process improvement.
Benefits – include areas such as development productivity improvements, reduced development time-scales, testing cost and time reductions, lifetime cost reductions, reduced fault levels, etc.
Types of Reviews
Walkthroughs – scenarios, dry runs, peer group, led by author.
Inspections – led by trained moderator (not author), defined roles, includes metrics, formal process based on rules and checklists with entry and exit criteria.
Informal reviews – undocumented, but useful, cheap, widely-used.
Technical reviews (also known as peer reviews) – documented, defined fault-detection process, includes peers and technical experts, no management participation.
Goals – validation and verification against specifications and standards, (and process improvement). Achieve consensus.
Activities – planning, overview meeting, preparation, review meeting, and follow-up (or similar).
Roles and responsibilities – moderators, authors, reviewers/inspectors and managers (planning activities).
Deliverables – product changes, source document changes, and improvements (both review and development).
Pitfalls – lack of training, lack of documentation, lack of management support (and failure to improve process).
Static Analysis
- compiler-generated information; dataflow analysis; control-flow graphing; complexity analysis
Explain that static analysis involves no dynamic execution and can detect possible faults such as unreachable code, undeclared variables, parameter type mismatches, uncalled functions and procedures, possible array bound violations, etc.
Explain that any faults found by compilers are found by static analysis. Compilers find faults in the syntax. Many compilers also provide information on variable use, which is useful during maintenance.
Explain that data flow analysis considers the use of data on paths through the code, looking for possible anomalies, such as ‘definitions’ with no intervening ‘use’, and ‘use’ of a variable after it is ‘killed’.
Explain use of, and provide example of production of control flow graph for a program.
Introduce complexity metrics, including cyclomatic complexity.
Read MoreTypes of Testing – Part II
Functional System Testing – functional requirements; requirements-based testing; business process-based testing
Functional requirement as per IEEE definition, which is “A requirement that specifies a function that a system or system component must perform”.
Requirements-based testing – where the user requirements specification and the system requirements specification (as used for contracts) may be used to derive test cases.
Business process-based testing – based on expected user profiles (e.g. scenarios, use cases, etc.).
Non-Functional System Testing – non-functional requirements; non-functional test types: load, performance and stress; security; usability; storage; volume; installability; documentation; recovery
Explain that non-functional requirements are as important as functional requirements.
Integration Testing in the Large – testing the integration of systems and packages; testing interfaces to external organisations (e.g. Electronic Data Interchange, Internet)
Integration with other (complete) systems.
Identification of, and risk associated with, interfaces to these other systems.
Incremental/non-incremental approaches to integration.
Integration Testing in the Small – assembling components into sub-systems; sub-systems to systems; stubs and drivers; big-bang, top-down, bottom-up, other strategies
Integration testing tests interfaces and interaction of modules/subsystems.
Role of stubs and drivers.
Incremental strategies, to include: top-down, bottom-up and functional incrementation. Non-incremental approach (“big bang”).
Maintenence Testing – problems of maintenance; testing changes; risks of changes and regression testing
Testing old code – with poor/missing specifications.
Scope of testing with respect to changed code.
Impact analysis is difficult – so higher risk when making changes – and difficult to decide how much regression testing to do.
Read MoreTypes of Testing – Part I
Though there are various types of testing, but we will only talk about those types which are used in Real life or in Real Time Environment. Part I & II will give you only brief introduction about them. Details about each and every type of testing will be covered in the later posts.
1) Regression Testing:
Whenever a fault is detected and fixed then the software should be re-tested to ensure that the original fault has been successfully removed. You should also consider testing for similar and related faults.
Tests should be repeatable, to allow re-testing / regression testing.
Regression testing attempts to verify that modifications have not caused unintended adverse side effects in the unchanged software (regression faults) and that the modified system still meets its requirements. It is performed whenever the software, or its environment, is changed.
Regression test suites are run many times and generally evolve slowly, so regression testing is ideal for automation. If automation is not possible or the regression test suite is very large then it may be necessary to prune the test suite. You may drop repetitive tests, reduce the number of tests on fixed faults, combine test cases, designate some tests for periodic testing, etc. A subset of the regression test suite may also be used to verify
2) Acceptance Testing:
Acceptance testing may be the only form of testing conducted by and visible to a customer when applied to a software package.
User acceptance testing – the final stage of validation. Customer should perform or be closely involved in this. Customers may choose to do any test they wish, normally based on their usual business processes. A common approach is to set up a model office where systems are tested in an environment as close to field use as is achievable.
Contract acceptance testing – A demonstration of the acceptance criteria, which would have been defined in the contract, being met.
Alpha & beta testing – In alpha and beta tests, when the software seems stable, people who represent your market use the product in the same way(s) that they would if they bought the finished version and give you their comments. Alpha tests are performed at the developer’s site, while beta tests are performed at the testers’ sites.
Read MoreTypes of Testing – Part I
Though there are various types of testing, but we will only talk about those types which are used in Real life or in Real Time Environment. Part I & II will give you only brief introduction about them. Details about each and every type of testing will be covered in the later posts.
1) Regression Testing:
Whenever a fault is detected and fixed then the software should be re-tested to ensure that the original fault has been successfully removed. You should also consider testing for similar and related faults.
Tests should be repeatable, to allow re-testing / regression testing.
Regression testing attempts to verify that modifications have not caused unintended adverse side effects in the unchanged software (regression faults) and that the modified system still meets its requirements. It is performed whenever the software, or its environment, is changed.
Regression test suites are run many times and generally evolve slowly, so regression testing is ideal for automation. If automation is not possible or the regression test suite is very large then it may be necessary to prune the test suite. You may drop repetitive tests, reduce the number of tests on fixed faults, combine test cases, designate some tests for periodic testing, etc. A subset of the regression test suite may also be used to verify
2) Acceptance Testing:
Acceptance testing may be the only form of testing conducted by and visible to a customer when applied to a software package.
User acceptance testing – the final stage of validation. Customer should perform or be closely involved in this. Customers may choose to do any test they wish, normally based on their usual business processes. A common approach is to set up a model office where systems are tested in an environment as close to field use as is achievable.
Contract acceptance testing – A demonstration of the acceptance criteria, which would have been defined in the contract, being met.
Alpha & beta testing – In alpha and beta tests, when the software seems stable, people who represent your market use the product in the same way(s) that they would if they bought the finished version and give you their comments. Alpha tests are performed at the developer’s site, while beta tests are performed at the testers’ sites.
Read MoreEconomics of Testing
The cost of faults escalates as we move the product towards field use. If a fault is detected before field use, the cost of rework to correct the fault increases dramatically because more than one previous stage of design, coding and testing may have to be repeated. If the fault occurs during field use, the potential cost of the fault might be catastrophic. If faults present in documentation go un-detected, then development based on that documentation might generate many related faults which multiply the effect of the original one.
Early test design can prevent fault multiplication. Analysis of specifications during test preparation often brings faults in specifications to light.
The cost of testing is generally lower than the cost associated with major faults (such as poor quality product and/or fixing faults), although few organisations have figures to confirm this.
What to consider, based on IEEE 829-1998 Test Plan Outline.
- Test plan identifier
- Introduction
- Test items
- Features to be tested
- Features not to be tested
- Approach
- Item pass/fail criteria
- Suspension criteria and resumption requirements
- Test deliverables
- Testing tasks
- Environmental needs
- Responsibilities
- Staffing and training needs
- Schedule
- Risks and contingencies
- Approvals