Study on Risk Analysis

Companies are highly dependent on the reliability of their mainframe systems.

If the mainframe doesn’t run, the company stops. Mainframe workloads also are growing considerably as companies’ businesses grow and as they continually seek to leverage data and applications in new ways. At the same time, these companies are losing their experienced mainframe workforce, largely to retirement. This makes the quality of their mainframe management tools even more important to them. CA piloted risk- based testing as part of our larger effort to ensure the quality of the solutions we deliver.

The pilot consisted of six main activities: Training key stakeholders on isk-based testing Holding a quality risk analysis session Analyzing and refining the quality risk analysis Aligning the testing with the quality risks Guiding the testing based on risks Assessing benefits and lessons This article addresses each of these areas – as well as some of the broader issues associated with risk-based testing.

What is Risk-Based Testing? Generally speaking, risk is the possibility of a negative or undesirable outcome or event.

Testing is concerned with two main types of risks: Product or quality risks, which are problems that can potentially affect the quality of the product itself, such as a defect that could ause a system to crash during normal operation. Project or planning risks, which are problems that can potentially affect overall project success, such as a staffing shortage that could delay completion of a deliverable. Of course, not all risks are equal and there are a number of ways to classify the different levels of risk.

The simplest is to look at two factors: www.

rbcs-us. com Rex Black, Ken Young, and Peter Nash Page 1 of 11 A Case Study in Successful Risk-Based Testing at CA The likelihood of the problem occurring, which depends primarily on technical considerations, such as the rogramming languages used and the constraints of a given computing platform. considerations, such as the financial impact of system downtime or the amount of lost staff productivity. Risk-based testing is guided by the level of risk associated with items identified during analysis.

Although risk can guide testing in various ways, there are three common ones. First, during all test activities, test teams allocate effort to each quality risk item based on the relative level of risk.

Test managers and analysts align the rigor and extensiveness of test techniques with the level of risk. They carry out test activities in risk order, starting with the most important risks. They also work with the project team to prioritize resolution of discovered defects based on the level of risk.

Second, test managers implement control steps for all significant identified project risks. A control step is either a mitigation (something done in advance to reduce the likelihood and/or impact of a risk) or a contingency (something you are prepared to do if the risk becomes an event to reduce the impact of the event).

The higher the level of risk, the more thoroughly that project risk is controlled. These project risks must include risks related to testing itself, since problems during test execution can reduce test scope and thereby result in quality risks.

Third, test managers and test analysts report test results and project status in terms of residual risks. Which tests have been run and which haven’t? Which have passed? Which have failed? Which defects have not yet been fixed or retested? How do the tests and defects relate back to the risks? In other words, with risk-based testing, risk management is an ongoing event. The above three responses to risk occur throughout the project lifecycle.

Quality risks are mitigated by running tests, and project risks are mitigated by controls.

Risks and risk levels are periodically re-evaluated based on new information, and, if necessary, priorities, allocation of effort, and project controls are modified. Why Adopt Risk-Based Testing? All testing faces two serious challenges. First, the set of possible test cases is infinite. So, if test coverage is measured by dividing the number of tests run by the number that could have been run, test coverage is always zero percent (n/m = O).

In fact, testers always select a relatively mall subset of tests from the set of tests that they could possibly run – so they have to be very smart about that selection.

Selection based on risk makes the most sense both in terms of product quality and project success. Second, because projects cannot take an infinite amount of time, all testing is “timeboxed” but the time-box is not a fixed size. Changes in upstream task durations for the project often compress the time-box for subsequent testing. The risk-based prioritization of tests directly addresses this challenge.

By prioritizing tests according both to likelihood and mpact, testers give themselves the best possible chance of discovering the worst possible problems.

Also, at any given time during the test execution period, the tests that have been run are more important than the tests that have not yet been run. This allows test managers to make risk-based test Page 2 of 1 1 A Case Study in Successful Risk-Based Testing at CA “triage” decisions to meet inflexible project deadlines – thereby minimizing the risks associated with any reduction in the scope of testing. An assessment of CA’s testing processes indicated that, like many test organizations, we faced challenges regarding both coverage and ime constraint.

The adoption of risk-based testing therefore seemed a wise choice.

Training Key Stakeholders The first step in our pilot project was a one-day workshop on risk-based testing that covered the following topics: The basic principles of and rationale for risk-based testing Understanding categories of quality risks (functionality, performance, reliability, usability, etc. ) How to perform a quality risk analysis and align testing with risk levels How to document quality risks How to monitor quality risks during test execution and report risk-based test results

In addition to the formal presentation, we had an open discussion and a two-hour hands-on exercise based on a hypothetical project, enabling attendees to explore the real-world issues presented by risk-based testing. The Quality Risk Analysis Session The quality risk analysis session consisted of two sub-sessions. During the first, the participants identified as many quality risk items as possible, using brainstorming techniques. We listed the main quality risk categories on three whiteboards.

The participants wrote each risk item on a sticky note and posted it under the appropriate category.

This sub-session lasted about three hours and identified more than 100 risk items. We also identified 11 project risks (e. g. , “The number and timing of QA bug discovery delays the release date. “) and three miscellaneous issues (e.

g. “Have all previous release fixes been merged into the code base? “). During the second sub-session, participants worked as a team to assess the likelihood and impact of each risk item. We also identified and eliminated risk items that were duplicates of other items. To make it easy for the team to assess the likelihood of all risk items, we used the rating scale shown in Table