Automation Testing

Successful automation mandates a testing process. Just as a developer needs a system development process, testers need a testing process to successfully use test tools. The testing process provides the steps, guidelines and techniques that will ensure practical, successful automation. To achieve the testing and risk management goals of the project, a solid testing process is essential to focus the test automation effort where it can do the most good.

The difficulty in software testing stems from the complexity of software: we can not completely test a program with moderate complexity. Testing is more than just debugging. The purpose of testing can be quality assurance, verification and validation, or reliability estimation.

Testing can be used as a generic metric as well. Correctness testing and reliability testing are two major areas of testing. Software testing is a trade-off between budget, time and quality.

Unlike most physical systems, most of the defects in software are design errors, not manufacturing defects. Software does not suffer from corrosion, wear-and-tear — generally it will not change until upgrades, or until obsolescence. So once the software is shipped, the design defects — or bugs — will be buried in and remain latent until activation. Regardless of the limitations, testing is an integral part in software development. It is broadly deployed in every phase in the software development cycle. Typically, more than 50% percent of the development time is spent in testing. Testing is usually performed for the following purposes:

• To improve quality
• For Verification & Validation (V&V)
• For reliability estimation

Software testing can be very costly. Automation is a good way to cut down time and cost. Software testing tools and techniques usually suffer from a lack of generic applicability and scalability. The reason is straight-forward. In order to automate the process, we have to have some ways to generate oracles from the specification, and generate test cases to test the target software against the oracles to decide their correctness. Today we still don’t have a full-scale system that has achieved this goal. In general, significant amount of human intervention is still needed in testing. The degree of automation remains at the automated test script level. Testing is potentially endless. We can not test till all the defects are unearthed and removed — it is simply impossible. At some point, we have to stop testing and ship the software. The question is when. Realistically, testing is a trade-off between budget, time and quality. It is driven by profit models. The pessimistic, and unfortunately most often used approach is to stop testing whenever some, or any of the allocated resources — time, budget, or test cases — are exhausted. The optimistic stopping rule is to stop testing when either reliability meets the requirement, or the benefit from continuing testing cannot justify the testing cost. This will usually require the use of reliability models to evaluate and predict reliability of the software under test. Each evaluation requires repeated running of the following cycle: failure data gathering — modeling — prediction. This method does not fit well for ultra-dependable systems, however, because the real field failure data will take too long to accumulate.

Manual Testing

What is Testing?
* An examination of the behavior of a program by executing on sample data sets.
* Testing comprises of set of activities to detect defects in a produced material.
* To unearth & correct defects.
* To detect defects early & to reduce cost of defect fixing.
* To avoid user detecting problems.
* To ensure that product works as users expected it to.

Why Testing?
* To unearth and correct defects.
* To detect defects early and to reduce cost of defect fixing.
* To ensure that product works as user expected it to.
* To avoid user detecting problems.

Testing Techniques

Black Box Testing -Testing of a function without knowing internal structure of the program.
White Box Testing -Testing of a function with knowing internal structure of the program.

Regression Testing -To ensure that the code changes have not had an adverse affect to the other modules or on existing functions.

Functional Testing:
* Study SRS
* Identify Unit Functions
* For each unit function
- Take each input function
- Identify Equivalence class
- Form Test cases
- Form Test cases for boundary values
- From Test cases for Error Guessing
* Form Unit function v/s Test cases, Cross Reference Matrix
* Find the coverage

Unit Testing:
* The most ‘micro’ scale of testing to test particular functions or code modules. Typically done by the programmer and not by testers .
* Unit - smallest testable piece of software.
* A unit can be compiled/ assembled/ linked/ loaded; and put under a test harness.
* Unit testing done to show that the unit does not satisfy the functional specification and/ or its implemented structure does not match the intended design structure.

Integration Testing:
* Integration is a systematic approach to build the complete software structure specified in the design from unit-tested modules. There are two ways integration performed. It is called Pre-test and Pro-test.
* Pre-test: the testing performed in Module development area is called Pre-test. The Pre-test is required only if the development is done in module development area.

Alpha testing:
* Testing of an application when development is nearing completion minor design changes may still be made as a result of such testing. Typically done by end-users or others, not by programmers or testers.

Beta testing:
* Testing when development and testing are essentially completed and final bugs and problems need to be found before final release. Typically done by end-users or others, not by programmers.

System Testing:
* A system is the big component.
* System testing is aimed at revealing bugs that cannot be attributed to a component as such, to inconsistencies between components or planned interactions between components.
* Concern: issues, behaviors that can only be exposed by testing the entire integrated system (e.g., performance, security, recovery).

Volume Testing:
* The purpose of Volume Testing is to find weaknesses in the system with respect to its handling of large amounts of data during short time periods. For example, this kind of testing ensures that the system will process data across physical and logical boundaries such as across servers and across disk partitions on one server.

Stress testing:
* This refers to testing system functionality while the system is under unusually heavy or peak load; it’s similar to the validation testing mentioned previously but is carried out in a “high-stress” environment. This requires that you make some predictions about expected load levels of your Web site.

Usability testing:
* Usability means that systems are easy and fast to learn, efficient to use, easy to remember, cause no operating errors and offer a high degree of satisfaction for the user. Usability means bringing the usage perspective into focus, the side towards the user.

Security testing:
* If your site requires firewalls, encryption, user authentication, financial transactions, or access to databases with sensitive data, you may need to test these and also test your site’s overall protection against unauthorized internal or external access.

Test Life Cycle

* Identify Test Candidates
* Test Plan
* Design Test Cases
* Execute Tests
* Evaluate Results
* Document Test Results
* Post Shipment Review

Test Plan:
* A Test Plan is a detailed project plan for testing, covering the scope of testing, the methodology to be used, the tasks to be performed, resources, schedules, risks, and dependencies. A Test Plan is developed prior to the implementation of a project to provide a well defined and understood project roadmap.

Test Specification:
* A Test Specification defines exactly what tests will be performed and what their scope and objectives will be. A Test Specification is produced as the first step in implementing a Test Plan, prior to the onset of manual testing and/or automated test suite development. It provides a repeatable, comprehensive definition of a testing campaign.

Approach for Test Automation

Description

Approach for Test Automation is used for building a strategy for automation starting from the Requirement phase till the deployment phase. Approach for automation begins with finding out the conditions and business rules given by the client and grouping similar conditions, business rules together as Test cases or Test Scenarios.

Steps involved in Approach for Test Automation are as follows:

· Requirement gathering from the client

· Understanding and Analyzing the Requirement

· Grouping the requirement into Test cases

· Preparing the Design for Automation

· Building Scripts depending on the Design

· Review of Scripts at Offshore

· Delivery to the Client

For example in the Unilever project:

Since it is Project for Automation of SAP, Team was divided into two,

SAP Consultants.

Testing Team.

Requirement gathering from the client included knowledge transfer at onsite, where all the transactions including the customized transactions where explained to the SAP Consultants of satyam, Under standing of requirements was done at the client side, Depending on the flow of transactions, all the transactions where grouped together as different scenarios. While SAP Consultants where onsite for gathering Requirements, Work done at the offshore was to analyze on the appropriate version of the QTP Tool to be used and to get licenses for QTP, user accounts for SAP access, Installation of QTP and SAP.

All the Identified Scenarios which consists of different transactions where explained to Testing Team. Then common functionalities where identified, and grouped together as different Test cases.

After defining different Test cases, design for automation of scripts was done, where Reusability was one of the main aspects, where all the transactions, which are common in the scenarios, are identified as Reusable transactions.

Automation Framework:

For Quality deliverables, important things that should be followed are:

· Following common coding structure for all the scripts

· Consistent coding conventions

· Use of Reusability for making code efficient and minimal

· Efficient use of Object Repository

· Use of Error and Exception handling Functions

· Use of Data table object, Environment variables for using data

For example in the Unilever project:
Coding Structure used is that there will a main Action for all the scripts which is named as corresponding Test case name, This Action intern will call all the other actions (Each Transaction used in the script would be an action) in the script. Importing and Exporting of sheets is also done in this main action. Importing of sheets is done at the beginning before calling other actions and exporting of sheets is done at the last. Sheets corresponding to all the actions in the scripts are imported, so that the data in the fields of the Imported sheets are used as input data to the script, output data from the script is also collected and exported to the corresponding sheet.

There are many transactions which are reused in the scripts, similar transactions (Actions) are made as reusable so that its not required to record those transactions again and again, which ever transaction was common among scripts, were recorded in one script and is reused in all the scripts where the transaction is required.

Error handling in the unilever automation is done by using Recovery Scenario where each Recovery scenario calls a function which handles the recovery by exiting all the other transactions (actions) when recovery fires. Major types of Recovery used in this project where Popup Window, Object state and on Error Recoveries. In Error handling using Recovery, all the common types of exceptions or errors are given common Recovery scenarios using regular expressions, which minimized the number of Recovery scenarios and Functions used.

Design for Structure of Scripts used in Unilever Project:

Main Action, which calls all the other Called Actions

Actions (transactions) in the script

Calls

Ex: S014_001 (Action Name) Ex: ME21N (Action Name

Which is same as the

Transaction name in

SAP)