| |
| |
See Also
Best Practices for Software Projects - Risk Management.
More...
Beyond Broken Links. More...
Minimizing Software Defects via Inspections.
More...
Testing GUI. More...
Test Methods. More...
Test Reviews. More...
Weekly Status Report
Sample. More...
Stocking and Managing a Test Lab.
More...
Best Practices for Software Projects
- Software Measurements. More...
Improving the Quality process by
doing the
Metrics Calculation
Download
Documentation Tips. More...
|
|
|
| |
More Resources
Some templates that
you will find useful
during the development
of your software product.
More...
When is a test plan
to complicated
by Charles Shelby. More...
Test Development
Life Cycle. More...
Test Plan Outline. More...
The test specification should
explain "how" to implement
the test cases described in the test plan. More...
|
|
|
| |
Try our bug tracking system. We would
also appreciate your feedback and comments. More...
|
|
|
| |
Have Something
to Share
software testing tip, interesting bug, or had an
interview lately?
Send us an Email.
|
|
|
| |
Promote Your
Products and Services
SQAtester.com Storefront Program is the best way to help companies Promote their Products and
Services. More...
|
|
|
| |
TrackStudio allows you to reduce the maintenance cost of managing a large number of projects along with lowering security risks. More...
|
|
|
| |
Learn more material at a Fraction
of Price!
Get the training that will help you face the challenges
and meet the demands of today's competitive market place.
Programs
Offered: Oracle DBA, Cisco Networking,
Software Testing & Programming. More...
|
|
|
| |
Advanced bug
tracking system
FREE Try our [ASP Hosted solution] for free for one month, no obligation to pay.
An extremely practical and cost effective Web Based Bug Tracking Software and help desk software ready to be integrated
into company intranet today. More...
|
|
|
| |
Try our bug tracking system. We would also appreciate your feedback and comments. More...
|
|
|
|
|
Manage your bugs
in minutes!
|
|
| |
Bug Tracking Systems are the single Most Important Way to Improve the
Quality of Your Software. They help
manage software development projects by tracking software bugs, action items, and change requests with problem
reports. More...
|
|
|
|
|
| Positive and Negative Testing |
 |
____________________________ |
|
by Jeff Nyman. GlobalTester, TechQA, Copyright © 2002 All Rights Reserved.
The notion of something like "Integration Testing" or "System Testing" can (and should) be
defined so that everyone knows what is meant by that activity within the same organizatoin, but terms like "negative
test" and "positive test" are more of a concept than a strict activity. In both instances you are
dealing with an input, an action, and an output. The action acts upon the input to derive a certain output. So
a test case (and thus a good test) is just one that deals with those three things. Both test cases can produce
errors and, in fact, some say that the success of a test case is based upon the probability of it finding new errors
in an application.
What I want to do here, however, is state clearly one viewpoint of what the distinction between positive and negative
testing is. Then I want to play Devil's Advocate and try to undermine that viewpoint by presenting an argument
that others have put forth - an alternative viewpoint. The real point of this will be to show that sometimes trying
to adhere too rigidly to conceptual terms like this can lead to a lot of stagnating action. Read this section as
a sort of extended argument that I am having with myself as I come to grips with these terms.
So let us first state a simple hypothetical definition: positive testing is that testing which attempts to show
that a given module of an application does what it is supposed to do. Negative testing is that testing which attempts
to show that the module does not do anything that it is not supposed to do. So, by that logic, and to make a concrete
example, an application delivering an error when it should is actually an example of a positive test. A negative
test would be the program not delivering an error when it should or delivering an error when it should not. But
this sounds like it is more based on what the application does during testing rather than how the tester is actually
going about testing it. Well, sort of. The idea here is that neither test necessarily has to force an error condition,
per se, at least by strict definition. But both concepts (negative and positive) are looking for different types
of error conditions. Consider that one part of negative testing is often considered to be boundary analysis. In
this case, you are not so much "forcing an error" because, of course, the application should handle boundary
problems. But what you are doing is seeing if the boundary problem is not, in fact, handled. So if the program
is supposed to give an error when the person types in "101" on a field that should be between "1"
and "100", then that is valid if an error shows up. If, however, the application does not give an error
when the user typed "101" then you have a problem. So really negative testing and positive testing are
the same kinds of things when you really boil it right down.
Now, some make a distinguishing remark from what I said. I said the following:
Positive testing is that testing which attempts to show that a given module of an application does what it is supposed
to do.
Negative testing is that testing which attempts to show that the module does not do anything that it is not supposed
to do.
Playing the Devil's Advocate, others would change this around and say the following is a better distinction:
Positive testing is that testing which attempts to show that a given module of an application does not do what
it is supposed to do.
Negative testing is that testing which attempts to show that the module does something that it is not supposed
to do.
Let us look at this slightly shifted point of view. By this logic, we would say that most syntax/input validation
tests are positive tests. Even if you give an invalid input, you are expecting a positive result (e.g., an error
message) in the hope of finding a situation where the module either gives the wrong error message or actually allows
the invalid input. A negative test is, by this logic, more trying to get the module to do something differently
than it was designed to do. For example, if you are testing a state transition machine and the state transition
sequence is: State 1 -> State 2 -> State 3 -> State 4, then trying to get the module to go from State
2 to State 4, skipping State 3, is a negative test. So, negative testing, in this case, is about thinking of how
to disrupt the module and, by extension, positive testing is examining how well/badly the module does its task.
Now, in response to this, I would agree that most can see looking at it this way from what the tester hopes to
find. Testing pundits often tell testers to look for error because if you look for success, you will often find
success - even when there is error. By proxy, if you do not find an error and you have reliable test cases (that
latter point is crucial), then a positive test case will show that the application did not, in fact, manifest that
error. However, showing an error when it should have done so is an example of a "positive test" by the
strict definition of that term. So in other words:
Positive Testing = (Not showing error when not supposed to) + (Showing error when supposed to)
So if either of the situations in parentheses happens you have a positive test in terms of its result - not what
the test was hoping to find. The application did what it was supposed to do. By that logic:
Negative Testing = (Showing error when not supposed to) + (Not showing error when supposed to)
(Usually these situations crop up during boundary testing or cause-effect testing.) Here if either of the situations
in parentheses happens you have a negative test in terms of its result - again, not what the test was hoping to
find. The application did what it was not supposed to do.
However, in both cases, these were good results because they showed you what the application was doing and you
were able to determine if it was working correctly or not. So, by my original definitions, the testing is all about
errors and finding them. It is just how you are looking for those errors that makes the distinction. (Granted,
how you are looking will often dictate what you are hoping to find but since that is the case, it hardly makes
sense to make a grand distinction between them.) Now, regarding the point I made above, as a Devil's Advocate:
"A negative test is more trying to get the module to do something differently than it was designed to do."
We have to realize, I think, that what we call "negative testing" is often about exercising boundary
conditions - and those boundaries exist within the context of design. Granted, that can be trying to get a value
in to a field that it should not accept. However, a good application should have, during the requirements stage,
had provisions for invalid input. Thus really what you are testing here is (a) whether the provisions for invalid
input exist and (b) whether they are working correctly. And, again, that is why this distiction, for me (between
positive and negative), is somewhat banal.
Your negative test can turn into a positive test just be shifting the emphasis of what you are looking for. To
get the application to do something it is not designed to do could be looked at as accepting invalid input. However,
if you find that the application does accept invalid input and does not, in fact, give a warning, I would agree
that is a negative test if it was specified in requirements that the application should respond to invalid input.
In this case the application did not, but it was not also specified that it should. So, here, by strict requirements
did the application do what it was supposed to do? Technically, yes. If requirements did not specify differently,
design was not put in place to handle the issue. Thus you are not testing something outside the scope of design.
Rather, you are testing something that was not designed in the first place.
So, going back to one of the previous points, one thing we can probably all agree on: it entirely depends on how
you view a test. But are we saying the result of the test determines whether it was a positive or negative test?
If so, many would disagree with that, indicating that it is the thinking behind the test that should be positive
or negative. In actuality, most experienced testers do not think in terms of positive or negative, they think in
terms of "what can I do to establish the level of risk?" However, to this point, I would argue that if
that is truly how the tester thinks of things then all concepts of positive/negative go right out of the window
(as I think they mostly should anyway). Obviously you could classify the test design in terms of negative or positive,
but to some extent that is irrelevant. However, without getting into that, I am not sure we are saying that the
result of the test determines positivity or negativity. What I said earlier, relative to my example, was that "in
both cases, these were good results because they showed you what the application was doing and you were able to
determine if it was working correctly or not." If the application was behaving correctly or incorrectly, you
still determined what the application was actually doing and, as such, those are good results. Thus the result
tells you about the application and that is good (without recourse to terms like positive and negative). If the
result tells you nothing about how the application is functioning that is, obviously, bad (and, again, this is
without recourse to positive or negative).
We can apply the term "effective" to these types of test cases and we can say that all test cases, positive
or negative, should be effective. But what about the idea of relying on the thinking behind the test? This kind
of concept is just a little too vague for me because people's thinking can be more or less different, even on this
issue, which can often depend on what people have been taught regarding these concepts. As I showed, you can transform
a postive test mentality into a negative test mentality just by thinking about the results of the test differently.
And if negative testing is just about "disrupting a module" (the Devil's Advocate position), even a positive
test can do that if there is a fault. However I am being a little flip because with the notion of the thinking
behind the test, obviously someone here would be talking about intent. The intent is to disrupt the module so as
to cause a fault fault and that would constitute a negative test (by the Devil's Advocate position) while a positive
test would not be trying to disrupt the module - even though disruption might occur (again, by the Devil's Advocate
position). The key differentiator is the intent. I could sort of buy that but, then again, boundary testing is
an attempt to disrupt modules because you are seeing if the system can handle the boundary violation. This can
also happen with results. As I said: "Your negative test can turn into a positive test just be shifting the
emphasis of what you are looking for." That sort of speaks to the intention of what you are hoping to find
but also how you view the problem. If the disruption you tried to cause in the module is, in fact, handled by the
code then you will get a positive test result - an error message of some sort.
Now I want to keep on this point because, again, some people state that negative testing is about exercising boundary
conditions. Some were taught that this is not negative testing; rather that this is testing invalid inputs, which
are positive tests - so it depends how you were taught. And figure that a boundary condition, if not handled by
the code logic, will potentially severely disrupt the module - which is the point of negative testing according
to some views of it. However, that is not the intent here according to some. And yet while that was not the intent,
that might be the result. That is why the distinction, for me, blurs. But here is where the crux of the point is
for me: you can generlaly forget all about intent of test case design for the moment and look at the distinction
of what the result is in terms of a "positive result" (the application showed me an error when it should
have) and a "negative result" (the application did not show me an error when it should have). The latter
is definitely a more negative connotation than the former, regardless of the intent of the tester during design
of the test case and that is important to realize because sometimes our intentions for tests are changed by the
reality of what exists and what happens as a result of running the tests. So, in the case of intent for the situation
of the application not showing an error when it was supposed to, this is simply a matter of writing "negative
test cases" (if we stick with the term for a moment) that will generate conditions that should, in turn, generate
error messages.
But the point is that the intent of the test case is to see if the application does not, in fact, generate that
error message. In other words, you are looking for a negative result. But, then again, we can say: "Okay,
now I will look that the application does generate the error message that it should." Well, in that case,
we are really just running the negative test case! Either way the result is that the error either will or will
not show up and thus the result is, at least to some extent, determining the nature of the test case (in terms
of negative or positive connotation). If the error does not show up, the invalid input might break the module.
So is the breakdown this:
P: Not showing error when not supposed to
N: Not showing error when supposed to
P: Showing error when supposed to
N: Showing error when not supposed to
I think the one thing we have to consider is the viewpoint: hinges on the idea of "negative testing"
being looked at as forcing the module to do something it was not designed to do. However, if the module was never
designed to do the thing you are trying, then your testing is of an interesting sort because, after all, you know
nothing exists to handle it. So the real question should not be: "What happens when I do this?" but rather
"Why have we not designed this to handle this situation?" Let us say that something is designed to handle
the "module disruption" you are proposed to test. In that case, you are actually positively testing the
code that handles that situation. To a strict degree, forcing a module to do something it was not designed to do
suggests that this is something your average user can do. In other words, your average user could potentially use
the application in such a fashion that the negative test case you are putting forth could be emulated by the user.
However, if that is the case, design should be in place to mitigate that problem. And, again, you are then positively
testing.
Now, one can argue, "Well, it is possible that the user can try something that there simply is no way to design
around." Okay. But then I ask: "Like what?" If there is no way you can design around it or even
design something to watch for the event, or have the system account for it, how do you write a valid test case
for that? I mean, you can write a test case that breaks the application by disrupting the module but -- you already
knew that was going to happen. However, this is not as cut and dry as I am sure anyone reading this could point
out. After all, in some cases maybe you are not sure that what you are writing as a test case will be disruptive.
Ah, but that is the rub. We just defined "negative testing" as trying to disrupt the module. Whether
we succeed or not is a different issue (and speaks to the result), but that was the intent. We are trying to do
something that is outside the bounds of design and thus it is not so much a matter of testing for disruption as
it is testing for the effects of that disruption. If the effects could be mitigated, that must be some sort of
design that is mitigating them and then you are positively testing that mitigating influence.
As an example, a good test case for a word processer might be: "Turn off the computer to simulate a power
failure when an unsaved document is present in the application." Now, the idea here is that you might have
some document saving feature that automatically kicks in when the application suddenly terminates, say via a General
Protection Fault (GPF). However, strictly speaking, powering down the computer is different than a GPF. So here
you are testing to see what happens if the application is shut down via a power-off of the PC, which, let us say,
the application was not strictly designed to really handle. So my intent is to disrupt the module. However, in
this case, since I can state the negative condition, I can state a possible design that could account for it. After
all: we already know that the document will not be saved because nothing was designed to account for that. But
the crucial point is that if nothing was designed into the system to account for the power-off of the PC, then
what are you really testing? You are testing that the application does what the application does when a power-off
occurs. But if nothing is designed to happen one way or the other, then testing for disruption really does you
no good. After all, you know it is going to be disrupted. That is not in question. What is (or should be) in question
is how you can handle that disruption and then test how that handling works. So let us take those alternate (Devil's
Advocate) definitions:
Positive testing is that testing which attempts to show that a given module of an application does not do what
it is supposed to do.
In this case of the power-down test case, we are not positive testing because we did not test that the application
did not do what it was supposed to do. The application was not "supposed" to do anything because nothing
was designed to handle the power-down.
Negative testing is that testing which attempts to show that the module does something that it is not supposed
to do.
In the case of the power-down test case, we are also not negative testing by this definition because the application,
in not saving the document or doing anything (since it was not designed to do anything in the first place), is
not doing something that it is not supposed to do. Again, the application is not "supposed" to do anything
since it was not designed to do anything in this situation.
Now consider my quasi-definition/equation for positive testing that I gave earlier:
Positive Testing = (Not showing error when not supposed to) + (Showing error when supposed to)
I would have to loosen my language a little but, basically, the application was not supposed to show an error and,
in fact, did not do so in this case. But what if the application was, in fact, supposed to handle that situation
of a power-down? Let us say the developers hooked into the API so that if a shut-down event was fired off, the
application automatically issues an error/warning and then saves the document in a recovery mode format. Now let
us say I test that and find that the application did not, in fact, save the file. Consider again my quasi-definition/equation
for negative testing:
Negative Testing = (Showing error when not supposed to) + (Not showing error when supposed to)
In this case I have done negative testing because the application was supposed to issue an error/warning but did
not. However, notice, that the test case is the same exact test case. The intent of my testing was simply to test
this aspect of the application. The result of the test relative to the stated design is what determines if the
test was negative or positive by my definitions. Now, because I want to be challenged on this stuff, you could
also say: "Yes, but forget the document in the word processor. What if the application gets corrupted because
of the power-off?" Let us say that the corruption is just part of the Windows environment and there is nothing
that can be done about it. Is this negative testing? By the Devil's Advocate definition, strictly it is not, because
remember by that definition: "Negative testing is that testing which attempts to show that the module does
something that it is not supposed to do." But, in this case, the module did not do something (become corrupted)
that it was not supposed to do. This simply happened as a by-product of a Windows event that cannot be handled.
But we did, after all, try to disrupt the module, right? So is it a negative test or not by the definition of disruption?
Incidentally, by my definition, it is not a negative test either. However, what is common in all of what I have
said is that the power-down test case is an effective test case and this is the case regardless of whether you
choose to connote it with a "positive" or "negative" qualifier. Since that can be the case,
then, for me, the use of the qualifier is irrelevant.
But now let us consider another viewpoint from the Devil's Advocate and one that I think is pretty good. Consider
this example: An application takes mouse clicks as input. The requirement is for one mouse click to be processed
at a time, the user hitting multiple mouse clicks will cause the application to discard anything but the first.
Any tester will do the obvious and design a test to hit multiple mouse clicks. Now the application is designed
to discard anything but the first, so the test could be classified (by my definition) a negative one as the application
is designed not to process multiple mouse clicks. The negative test is to try to force the application to process
more than the first. BUT, I hear you say, this is an input validation test that tests that the application does
discard mutiple mouse clicks, therefore it is a positive test (again, by my definition), and I would then agree,
it is a positive test. However, the tester might also design a test that overflows the input buffer with mouse
clicks - is that a negative test?. Note, this situation is not covered explicitly in the requirements - and that
is crucial to what I would call negative testing, that very often it is the tester's "what if" analysis
that designs negative tests - so, yes, it is a negative test as you are forcing the application into a situation
it may not have been designed and/or coded for - you may not know whether it had or not. The actual result of the
test may be that the application stops accepting any more clicks on its input buffer and causes an error message
or it may crash.
Now, having said all this, it makes me realize how my point starts to coalesce with the Devil's Advocate. One way
that might happen is via the use of the term "error" that has gotten tossed around a lot. My language
seemed too restrictive in the sense that when I used the word "error" (as in "showing error when
not supposed to") I did not make it clear enough that I was not necessarily talking about an error screen
of some sort, but rather an error condition or a failure. With this, my negative testing definition really starts
to coalesce with the Devil's Advocate's definition ("does something that it is not supposed to do").
I had said: "Negative Testing = (Showing error when not supposed to) + (Not showing error when supposed to)"
and broadening my language more, really what I am saying is that the application is either showing (doing) something
it is not supposed to (which matches the Devil's Advocate thought) but I was also saying that the application is
not showing (doing) something that it was supposed to. And to the Devil's Advocate that latter is positive testing.
Let me restate the two viewpoints somewhat:
Positive Testing (Jeff):
Not doing something it was not supposed to do.
Doing something it was supposed to do.
Positive Testing (Devil's Advocate):
Not doing what it is supposed to do.
Negative Testing (Jeff):
Doing something it was not supposed to do.
Not doing something it was supposed to do.
Negative Testing (Devil's Advocate):
Doing something that it is not supposed to do.
I think I was essentially saying the same thing in terms of negative testing as my hypothetical opponent, just
not in terms of positive testing. If you notice, both of our "negative testings" really contain the same
point. On the other hand, relative to the common Devil's Advocate position, I am having a hard time seeing the
major distinction between positive and negative. The Devil's Advocate's original conception:
"Positive testing is that testing which attempts to show that a given module of an application does NOT do
what it is supposed to do. Negative testing is that testing which attempts to show that the module does something
that it is not supposed to do."
To me, "doing something you are not supposed to" (the Devil's Advocate negative test) and "not doing
something you are supposed to" (the Devil's Advocate positive test) are really two sides of the same coin
or maybe just two ways of saying the same thing. So let us say that our requirement is "do not process multiple
mouse clicks". In that case, "not doing something you are supposed to" (Devil's Advocate positive
test) means, in this case, "processing multiple mouse clicks". In other words, the application should
not process multiple mouse clicks. If it does, it is doing something it is not supposed to. Likewise, "doing
something you are not supposed to do" (Devil's Advocate negative test) means, in this case, "processing
multiple mouse clicks". In other words, the application should not process multiple mouse clicks. Either way,
it is saying the same thing. So what we are testing for is "application not processing multiple mouse clicks".
It would seem that if the application does process multiple mouse clicks it is both not doing what it is supposed
to do (not processing them) and doing something it is not supposed to do (processing them). The same statement,
just made different ways. Now, let me see if that works with my definitions.
Again, the lemma is "do not process multiple mouse clicks". If the application does this then it falls
under lemma 1 of my negative test ("Doing something it was not supposed to do.") If the application does
not do this, it falls under lemma 1 of my positive test ("not doing something it was not supposed to do").
Even with the mouse click example we have two aspects:
Application designed not to process multiple mouse clicks
Application designed to process only one mouse click
Saying the same thing, and yet a subtle shift in emphasis if you want to go by the positive and negative distinctions.
The difference, however, is also whether you are dealing with active design or passive design. In other words,
does the application actively make sure that only one mouse click is handled (by closing the buffer) or does it
simply only process one click, but allow the buffer to fill up anyway. I like the idea of tying this whole thing
in with "mitgating design factors". I think that we can encapsulate "intent" and "result"
(both of which are important to test casing) by looking more at the efficient and effective demarcations. We have
to consider result that is part of how you do test case effectiveness metrics as well as proactive defect detection
metrics. If a test case is a tautology test then it is not really efficient or effective - but that is based solely
on the result, not the intent or anything else.
And The Point Is ... ?
The point here was simply to show an extended dialogue that I wish more testers would take it upon themselves to
do. Regardless of whether you agree with some of the conclusions I have put foward here (most notably that "positive"
and "negative" is not necessary a very meaningful distinction), at least you will hopefully see that
it pays to sometimes think out these issues because it forces you to better consider what it is you are actually
doing when you are testing and it also helps you think about how to explain it to others.
 Share your thoughts on Positive and Negative Testing in SQAtester Group.
 E-mail to a Friend.
|
| Categories |
|
________________________________ |
| Community |
|
________________________________ |
| Specials |
|
__________________________________ |
| Millions of titles discounted up to 40-90% off. Great
low prices on your favorite books. More... |
|
Find all of your favorite software.
More... |
|
|
