|
Testing Methodology
The following is an overview of the quality practices
of Software Quality Assurance team:
-The iterative approach to software development
presents a significant challenge for SQA. The
iterative, rapid deployment process is characterized by a lack of strict adherence to a traditional waterfall development
methodology (marketing first specs the feature set, then engineering refines the marketing requests into more detailed
specifications and a schedule, then engineering starts building to specification and SQA starts building tests,
then a formal testing cycle, and finally product release). Here is a variant of development:
-As progress is made toward a release, the first
priority features are done to a significant level of completion before much progress is made on the second priority
features. A similar approach is taken for the hopefullys and third priority features. The first priority feature
list is all that has to be completed before a product is feature complete, even though, there has been time built
into the schedule to complete the second priority, as well.
-Other than the initial OK from the executive team
that they want a particular product built, there is not a rigorous set of phases that each feature must pass.
-Developers (designers, coders, testers, writers,
managers) are expected to interact aggressively and exchange ideas and status.
-By not going heavily into complete specifications,
the impact of a better idea along the way need not invalidate a great deal of work.
-One prototype is worth a pound of specification.
However, this does not mean that large scale changes should not be specified in writing. Often times, the effort
to do paper based design is significantly cheaper than investing in a working prototype. The right balance is sought
here.
Complementing the strategy of iterative software
development, the SQA testing assessment is accomplished through personal interaction between SQA engineers and
Development engineers. Lead SQA engineers meet with the development team to assess the scope of the project, whether
new features for an existing product, or the development of a new product. Feature, function, GUI, and
cross-tool interaction are defined to the level of known attributes. When development documentation is provided,
the understanding of the SQA engineer is greatly enhanced. The lead SQA engineer then meets with the test team,
to scope the level and complexity of testing required. An estimate of test cases and testing time is arrived at
and published, based upon the previous discussions.
-Working with the development team, the SQA team
takes the builds, from the first functioning integration, and works with the features as they mature, to determine
their interaction and the level of testing required to validate the functionality throughout the product.
-The SQA engineers, working with existing test plans
and development notations on new functionality, as well as their notes on how new features function, develop significant
guidelines for actual test cases and strategies to be employed in the testing. The SQA engineers actively seek
the input of the development engineers in definition and review of these tests.
-Testing is composed of intertwined layers of manual
ad hoc and structured testing, supplemented by automated regression testing which is enhanced as the product matures.
Test Plan Components
Test requirements based on new features or functions.
Specific testing based on the features defined as
Development Priority 1. There must be a plan in place for these features and they must be scheduled for testing.
A product release date will be slipped in order to complete adequate testing of the Priority 1 features.
Specific testing based on new features or functions
defined as Development Priority 2. There must be a plan in place for these features and they must be scheduled
for testing. If testing of the Priority 1 features impacts adequate testing of these, they may be dropped from
the product.
Specific testing based on new features or functions
defined as Development Priority 3. Software Quality Assurance will not schedule or plan for these features. However,
Priority 3 completed prior to Functional Freeze
will be added to the SQA Priority 2 for testing and appropriate risk assessment will be taken with respect their
inclusion in the released product.
SQA has its own set of Priority 1, Priority 2, Priority
3, which include not only the Development activities, but also testing required as due diligence for product verification
prior to shipment.
-Priority 1, features include the testing of new
features and functions, but also a defined set of base installations, program and data integrity checks, regression
testing, documentation (printed, HTML and
on-line Help) review and final "confidence" (high level manual or automated tests exercising the most
frequently used features of the product) checks on all media to be released to the public. Products being distributed
over the Web, also have their Web download and installation verified.
-Priority 2, include a greater spectrum of installation
combinations, boundary checking, advanced test creation and more in-depth "creative" ad hoc testing.
-Priority 3, usually reflect attempts to bring greater
organization to the SQA effort in documentation of test scripts, creation of Flashboards for metric tracking, or
expanded load testing.
Testing
One of the test methods SQA team practice is "Black Box" testing. The SQA engineers, like the customers whom
they attempt to emulate, are isolated from the source code and must rely upon their understanding of the application
and its features and functions. SQA engineers work with Development engineers toward development of code which
lends itself to the exercise of automated test tools, thus providing for stable, repeatable, reliable testing of
base features and functions. The deductive, reasoning, creative skills of the SQA engineers are thus freed from
the more repetitive tasks to focus on development of more user centric testing, which expands the scope coverage.
Manual Testing
-GUI - SQA team members upon receipt of the Development
builds, walk through the GUI and either
update existing hard copy of the product Roadmaps, or create new hard copy. This is then passed on to the Tools
engineer to automate for new builds and regression testing. Defects are entered into the bugs tracking database,
for investigation and resolution. Questions about GUI content are communicated to the Development team for clarification
and resolution. The team works to arrive at a GUI appearance and function which is "customer oriented"
and appropriate for the platform, Web, UNIX, Windows, Macintosh. Automated GUI regression tests are run against
the product at Alpha and Beta "Hand off to QA" ,(HQA) to validate that the GUI remains consistent throughout the development process. During
the Alpha and Beta periods, selected customers validate the customer orientation of the GUI.
-Features & Functions - SQA test engineers,
relying on the team definition, exercise the product features and functions accordingly. Defects in feature/function capability are entered into the
defect tracking system and are communicated to the team. Features are expected to perform as expected and their
functionality should be oriented toward ease of use and clarity of objective. Tests are planned around new features
and regression tests are exercised to validate existing features and functions are enabled and performing in a
manner consistent with prior releases. SQA using the exploratory testing method, manually tests and then plans
more exhaustive testing and automation. Regression tests are exercised which consist of using developed test cases
against the product to validate field input, boundary conditions and so on... Automated tests developed for prior
releases are also used for regression testing.
-Installation - Product is installed on each of
the supported operating systems in either default, flat file configuration, or with one of the supported databases.
Every operating system and database, supported by the product, are tested, though not in all possible combinations.
SQA is committed to executing, during the development life cycle, the combinations most frequently used by the
customers. Clean and upgrade installations are the minimum requirements.
-Documentation - All documentation, which is reviewed
by Development prior to Alpha is reviewed by the SQA team prior to Beta. On-line help and context sensitive Help
are considered documentation as well as manuals, HTML documentation and Release Notes. SQA not only verifies technical
accuracy, clarity and completeness, they also provide editorial input on consistency, style and typographical errors.
Automated Testing
-GUI - Automated GUI tests are run against the product
at Alpha and Beta "Hand off to QA" (HQA) to validate that the GUI has remained consistent within the
product throughout the development process. The automated Roadmaps, walk through the client tool windows and functions,
validating that each is there and that it functions.
-Data Driven - Data driven scripts developed using
the automation tools and auto driver scripts are exercised for both UNIX and Windows platforms to provide repeatable,
verifiable actions and results of core functions of the product. Currently these are a subset of all functionality.
These are used to validate new builds prior to extensive manual testing, thus assuring both Development and SQA
of the robustness of the code.
-Future - Utilization of automated tools will increase
as our QA product groups become more proficient at the creation of automated tests. Complete functionality testing
is a goal, which will be implemented feature by feature.
Reporting
The SQA team lead produces a formal test plan which
is submitted to the project team for validation and information. Input is solicited for the understanding of requirements,
so that the plan can accurately reflect the testing which will be required. Because of the variances between Development
teams' style of disseminating feature, function scope and definition, the amount of information available is not
always consistent. The SQA lead attempts to establish a good working relationship with the Development team, so
that questions can be asked and comprehensive, identical understanding of the product will exist. The lead also
pairs a QA engineer with a development counter part to facilitate day to day interaction.
SQA engineers produce status sheets for distribution
to the members of the product teams. The data on the status sheets have included: features, functions, time estimated
to test, time consumed and amount of testing yet to be done, the last has proved to be too subjective. Some test
engineers include the staff assigned to specific testing and the percentage complete. The later is difficult to
estimate due to the iterative process, as feature and function specifics change often and rapidly during the development
cycle, as they are refined. The evolving report, which appears to give the most information, is one which lists
the features, the build in which they were tested or re-tested and the pass or fail status of the test.
SQA provides monitoring of the defects which appear
in the product, through the use of QA designed Flashboards (graphical representations of the aggregate numbers
of defects) and reports. The defects found in the product are recorded in an Bug Tracking database, where the information
is made available to the development group. Information stored in the database then provides statistical and trend
analysis for defect find rates and product areas. This information is compared to that presented by the Development
team and the Product team. Customer support is kept abreast of these defects and influences the priority assigned
to the defect by the team.
Team leads have established directories for their
products in which test plans and weekly status reports have been posted. These are updated weekly by the team lead
and reviewed by the manager and linked or posted to the QA home page on the intranet.
QA managers work with their teams to assess better
forms and methods of information dissemination. These reviewed with the larger engineering and project teams so
that the teams feel they understand the scope of work to be done by SQA and the status of a project currently being
tested.
 Share your thoughts on QA Practices and Procedures in SQAtester Group.
 E-mail to a Friend.
|
