PART I - Principles of Testing
MODULE 1 - Why Testing is Necessary
The training course begins with an overview of the reasons for software testing. Testing is necessary because the existence of faults in software is inevitable. Beyond fault-detection, the modern view of testing holds that fault-prevention (e.g. early fault detection/removal from requirements, designs etc. through static tests) is at least as important as detecting faults in software by executing dynamic tests. Fault detection (and removal) is the cornerstone of testing, but there are two other objectives overall: risk measurement/reduction and confidence building. Testers need to know how faults occur because it is our business to detect these faults. Only by understanding our quarry, can we prepare effective strategies to detect them.
It is generally believed that software can never be made perfect. Consequently, we test software to find as many faults as we can to ensure that a high quality product with a minimum of faults is delivered. Given that our job as a tester is to find faults, we must understand how faults occur.
Testing Terminology – A Perenial Problem
Most organisations have their own internal terminology for testing. This makes discussions between practitioners from different organisations problematic. The formal definitions from standards sometimes overlap or conflict and are patchily adopted. The ISEB scheme uses the BS 7925-1 definitions in an attempt to ensure clarity and consistency across all training organisations.
BS EN ISO 8402:1995, Quality management and Quality Assurance - Vocabulary.
BS ISO/IEC 2382-1:1993, Data processing - Vocabulary - Part 1: Fundamental terms.
IEEE Standard Glossary of Software Engineering Terminology, IEEE Std 610.12-1990.
BS 7925-1 Standard Glossary of Testing Terms
The ISEB scheme uses BS 7925-1.
This standard was a by-product from the British Computer Society SIG in Software testing initiative that resulted in the Software Component Test Standard - BS 7925-2. The glossary of terms covers component testing more thoroughly than the other test activities, but is internally-consistent and consistent with the terms defined in the other standards above.
Errors, faults and failures
It may seem pedantic (testers should be pedantic), but it is essential that the difference in definitions between errors, faults and failures is understood.
Misunderstandings between project managers, developers and testers often arise because the concepts of human error, bugs and blame are confused. As has been stated before, blaming peoples for their mistakes, without improving the process, the working environment or project constraints is self-defeating as it will only upset people on the team.
The tester should adopt a consistent set of terms to ensure they communicate with clarity. The tester should approach their role with a dispassionate point of view. They should not 'take a position' on the people or personalities, but they should have an objective and realistic view of the product.
A Failure is……………
A deviation of the software from its expected delivery or service
Software fails when it behaves in a different way that we expect or require. If we use the software properly and enter data correctly into the software but it behaves in an unexpected way, we say it fails.
A failure occurs when software does the 'wrong' thing
We can say that if the software does the wrong thing, then the software has failed. This is a judgment made by the user or tester. You cannot tell whether software fails unless you know how the software is meant to behave. This might be explicitly stated in requirements or you might have a sensible expectation that the software should not 'crash'.
Most of the time software does the right thing
Most software does the right thing most of the time. Unfortunately, even a small number of failures may make the Software unusable or unacceptable.
Software faults cause software failures when the program is executed with a set of inputs that expose the fault.
The faults in the software only become apparent when the software is used in a way that exercises the faulty code. The software faults cause the code to fail and from the failure we can find the fault and fix it. It may be easy or very difficult to diagnose where the fault in the code is from the information we get from the failure.
A Fault is
A manifestation of human error in software
A fault in software is caused by an unintentional action by someone building a deliverable. We normally think of programmers when we talk about software faults and human error. However, faults can be created by managers working on project plans, analysts working on requirements, designers working on designs or testers working on test plans too. Human error causes faults in any project deliverable.
Only faults in software cause software to fail. This is the most familiar situation.
Also known as a defect or bug
In the US, faults are more commonly known as defects. Bugs are a universally accepted name for faults. Bugs and faults are used interchangeably in this training course.
Faults may be caused by requirements, design or coding errors
All software development activities are prone to error. Faults may occur in all software deliverables when they are first being written or when they are being maintained.
Software faults are static - they are characteristics of the code they exist in
When we test software, it is easy to believe that the faults in the software move. Software faults are static. Once injected into the software, they will remain there until exposed by a test and fixed.
They are discovered either by inspecting code or by inferring their existence from software failures.
There are two ways of detecting faults in software. The first is by reading the code itself. The second is by executing the software, when the software fails we can infer the existence of a fault.
An Error is………..
A human action producing an incorrect result
The error is the activity undertaken by an analyst, designer, developer or tester whose outcome is a fault in the deliverable being produced.
When programmers make errors they introduce faults to program code
We usually think of programmers when we mention errors, but any person involved in the development activities can make the error which injects a fault into a deliverable.
Errors are not
Errors are not just accidents or mistakes that can be cured by "being more careful".
Errors are usually the result of an unforeseen event that distracts the developer, overwork (or overload), misunderstandings etc. Many can be traced back to some original problem or event.
Being more careful might help a little, but many errors occur because of circumstances beyond our control.
Errors are not just accidents or mistakes
Errors are not cured by "just being more careful"
Errors are not an act of incompetence
Software developers always try and do good work and are competent, but errors are inevitable because of the process, project constraints or our human fallibility.
Errors are inevitable in a complex activity.
Testers should regard errors as an inevitability in software development. Personal blame is counter-productive as most errors occur in spite of our best efforts to eliminate them.
Reliability is………….
Reliability is another term that is much used, but rarely understood.
The probability that software will not cause the failure of a system for a specified time under specified conditions
It is usually easier to consider reliability from the point of view of a poor product. One could say that an unreliable product fails often and without warning and lets its users down. However, this is an incomplete view. If a product fails regularly, but the users are unaffected, the product may still be deemed reliable. If a product fails only very rarely, but it fails without warning and brings catastrophe, then it might be deemed unreliable.
Software with faults may be reliable, if the faults are in code that is rarely used
If software has faults it might be reliable because the faulty parts of the software are rarely or never used - so it does not fail. A legacy system may have hundreds or thousands of known faults, but these exist in parts of the system of low criticality so the system may still be deemed reliable by its users.
Reliability is in the view of the user
The reliability of a system is, like beauty, in the eye of the beholder. Many software products in regular use fail often, but as experienced users, we work around these problems because the main purpose of the system is not undermined by these failures.
Reliability measures must take account of how the software will be used by its users.
The perceived reliability of a system may vary for different users. For example, a word processor print facility might be regarded as perfectly adequate by an office user who prints perhaps ten times a day but has one failure per day. If the same functionality is used as a component in a document production facility for mailshots, a one in ten failure rate would be totally unacceptable, of course, where the volume of document printed would be very large and failures would escalate.
Imprecise capture of Requirements
Imprecisions in requirements are the most expensive faults we encounter. Imprecision takes the form of incompleteness, inconsistencies, lack of clarity, ambiguity etc. Faults in requirements are inevitable, however, because requirements definition is a labour-intensive and error-prone process.
Users cannot express their requirements unambiguously
When a business analyst interviews a business user, it is common for the user to have difficulty expressing requirements because their business is ambiguous. The normal daily workload of most people rarely fits into a perfectly clear set of situations. Very often, people need to accommodate exceptions to business rules and base decisions on gut feel and precedents which may be long standing (but undocumented) or make a decision 'on the fly'. Many of the rules required are simply not defined, or documented anywhere.
The business is not logical
Although the business analyst wants a logical definition of the business process and precise requirements for the system, the user may struggle to present this information in the form of a logical, sequenced set of needs. All too often, the current business process is not logical, and may depend upon issues completely outside the scope of the system to be built.
Users cannot express their requirements completely
It is unreasonable to expect the business user to be able to identify all requirements. Many of the detailed rules that define what the system must do are not written down. They may vary across departments. Under any circumstance, the user being interviewed may not have experience of all the situations within the scope of the system.
Developers don't fully understand the business.
Few business anaysts, and very few developers have direct experience of the business process that a new system is to support. It is unreasonable to expect the business analyst to have enough skills to question the completeness or correctness of a requirement.
Underpinning all this is the belief that users and analysts talk the same language in the first place, and can communicate.
All in all, the elicitation and validation of requirements is extremely difficult.
Progressive distortion
Although this slide is a cartoon, it reflects the all too common situation where the views of the main protagonists in a development project have different views of what is required, how the product is to be used and how it is to be built.
The problem of progressive distortion occurs when, at each stage of the development process, the differing viewpoints causes a distortion of the product from the requirement of the user.
Also, because software systems are built as a series of transformations from requirements to designs to code, at each stage the faults introduced magnify the divergence from the requirement. Faults in and misinterpretation of requirements are particularly expensive in that these have the greatest impact on the end product.
Short Term Memory
Accounts for many detailed errors
It has been reliably demonstrated that humans have a limited short-term memory. In your daily experience, it is clear that if we try to remember too many things at the same time, some things 'fall off the stack'. It is if course possible to remember things for a longer period. The information we assign to short-term memory may only need to remembered for a few seconds, minutes or maybe an hour or two. We choose not to use our long-term memory for information that won't be needed for very long. The problem is that this memory is limited in its capacity.
What are the consequences? Well, the typical problem is that we forget to do something important in our code, or document. For example, we decide that we need a new variable on line 1352 of a long program. We remember to declare it at line 53, but then forget to initialise it somewhere else. The program compiles, and runs, but might not give the required result under all circumstances. Because we forgot to initialise the variable, we might also forget the test that demonstrates the initialisation works properly. Essentially, the problem of short-term memory is that it 'overflows'. We try and remember just one thing too many. Or the phone rings and we are asked to remember to come home early, or attend a meeting in the afternoon, or perhaps someone walks by and asks a 'simple' question. Any one of these can knock an item 'off the stack'.