Sensitivity Analysis Online Free

What is..

.? series Supported by sanofi-aventis New title Health economics What is sensitivity analysis? Matthew Taylor PhD MSc Senior Consultant, York Health Economics Consortium, University of York G While economic models are a useful tool to aid decision-making in healthcare, there remain several types of uncertainty associated with this method of analysis. G One-way sensitivity analysis allows a reviewer to assess the impact that changes in a certain parameter will have on the model’s conclusions.G Sensitivity analysis can help the reviewer to determine which parameters are the key drivers of a model’s results. G By reporting extensive outputs from sensitivity analysis, modellers are able to consider a wide range of scenarios and, as such, can increase the level of confidence that a reviewer will have in the model. G Probabilistic sensitivity analysis provides a useful technique to quantify the level of confidence that a decision-maker has in the conclusions of an economic evaluation.

For further titles in the series, visit: www. whatisseries. co. ukDate of preparation: April 2009 1 NPR09/1109 What is sensitivity analysis? What is sensitivity analysis? Economic models The vast expansion in the number of available healthcare technologies, coupled with constraints placed on healthcare budgets, has increased the importance of decision-making with regards to new and existing health technologies. One method to determine the value for money of an intervention is to develop a ‘cost-effectiveness model’, predicting the costs and health outcomes that are likely to be associated with various different interventions.Models are a useful tool for representing the detailed and complex ‘real world’ with a more simple and understandable structure.

While models do not claim to necessarily create an exact replica of the real world, they can be useful in demonstrating the relationships and interactions between various different factors. Furthermore, models allow the decision-maker to combine information from various sources and, in some cases, to extrapolate findings beyond the trial period. One-way sensitivity analysisThe simplest form of sensitivity analysis is to simply vary one value in the model by a given amount, and examine the impact that the change has on the model’s results. For example, it might be shown that by changing the effectiveness of an intervention by 10%, the cost-effectiveness ratio falls by, say, 20%. This is known as one-way sensitivity analysis, since only one parameter is changed at one time. The analysis could, of course, be repeated on different parameters at different times.

One-way sensitivity analysis can be undertaken using various different approaches, each of which is useful for different purposes.Suppose that a researcher would like to test which parameters have the greatest influence on a model’s results. In this case, each parameter in the model (or, at least, each of the key parameters) could be changed by a specific amount. Say, for example, that all parameters were to be increased and decreased by 20% of their original value. For each parameter change, the researcher might record the percentage impact on the model’s main outcome, which can be shown graphically in the form of a tornado diagram (Figure 1).

While a tornado diagram is useful in emonstrating the impact that a fixed change in each parameter has on the main outcomes, it is not useful in representing the confidence that a decision-maker might have in the model’s inputs. For example, it might be that the level of confidence in one particular parameter is so low that it is entirely reasonable that the current input may be ‘wrong’ by as much as 100%. This might be seen in cases where no published data exist to support a particular model input. An example of this could be the impact of a drug on a patient’s long-term mortality, when only short-term trial data are available.Conversely, some parameters (such as the price of an intervention) may be reasonably well known and, as such, the user would have a high Uncertainty in modelling While models tend to report single summary outcomes, such as ‘the incremental cost per incremental life-year’, the interpretation of those results will largely depend on the level of confidence or uncertainty in various factors. These might involve the methodology that has been used in constructing the model (that is, the model structure) or could be related to the actual values that have been used to populate the model.

For example, a reviewer of a model might suspect that one particular value (for example, the probability of a treatment being successful) is too high in the model. In this case, the reviewer may wish to know the likely impact of using an alternative value. Such an exercise would involve examining the sensitivity of the model to changes in its inputs. There are several different ways of undertaking sensitivity analysis, each of which is discussed in more detail below. Date of preparation: April 2009 2 NPR09/1109 What is sensitivity analysis? 0% decrease Incidence of disease Probability of treatment success Probability of comparator success Cost of treatment Cost of comparator Quality of life for healthy patients Quality of life for patients with disease Discount rate (costs) Discount rate (benefits) -30% -20% -10% 0% 10% 20% 30% 40% 20% increase Change in incremental cost-effectiveness ratio (%) Figure 1.

Tornado diagram degree of confidence in the value. Therefore, one form of one-way sensitivity analysis is to vary each parameter to the highest and lowest possible values.The definition of ‘possible’ might vary from model to model, but it is usually reasonable to vary the parameters according to the confidence intervals of the data (if these are known). Alternatively, it would be reasonable to vary the parameter across the whole range of values that have been observed in the literature. Finally, a more detailed approach to oneway sensitivity analysis on one specific parameter would be to assess the impact of a range of values on the model’s outputs.

In such an analysis, it would be possible to generate a simple graph, plotting the main model outcome against each possible input value. This would demonstrate the relationship between the input value and the model’s results. This type of analysis can also be used to judge the threshold at which the main conclusions of a model might change. For example, Figure 2 suggests that the costeffectiveness of a particular intervention will remain below a prespecified threshold of ? 20,000 only if the intervention is priced below ? 270. As the price of the intervention increases, it becomes less and less costeffective.Multiway sensitivity analysis While one-way sensitivity analysis is useful in demonstrating the impact of one parameter varying in the model, it may be necessary to examine the relationship of two or more different parameters changing simultaneously.

This approach involves the changing of, say, two key parameters (for example, the effectiveness and cost of an intervention), showing the results for each potential combination of values within a given range. An example of the outputs of a two-way sensitivity analysis is shown in Table 1.It should be noted, however, that the presentation and interpretation of multiway sensitivity analysis becomes increasingly difficult and complex as the number of parameters involved increases. One method that is sometimes used to assess the confidence around all parameters is to undertake extreme sensitivity analysis, by Date of preparation: April 2009 3 NPR09/1109 What is sensitivity analysis? Figure 2. Threshold analysis Incremental cost per quality-adjusted life-year ?60,000 – ?50,000 – ?40,000 – ?30,000 – ?20,000 – Base case ? 10,000 – ?0 – ? 0 ?50 ?100 ?150 ?200 ?250 ?300 ?350 ?400 ?450 ?500 Cost of intervention arying all of the parameters in a model to their ‘best’ and ‘worst’ case.

The best- and worst-case values should be chosen from the perspective of the intervention that is being assessed. For example, in one scenario the most optimistic values would be chosen, while in another, the most conservative figures would be used. Again, the extent to which each parameter should be varied will depend upon the confidence intervals associated with each input. Probabilistic sensitivity analysis In most models, each parameter (for example, the probability of a treatment being successful) is assigned a point estimate value.As an illustration, we might assume that the chance of an intervention being successful is deemed to be 60%, as published in a metaanalysis study.

However, the study may also have published the level of confidence around that estimate. For example, it might Table 1. Two-way sensitivity analysis Price of intervention ? 0 0% Effectiveness of intervention 5% 10% 15% 20% 25% 30% 35% 40% Cost saving Cost saving Cost saving Cost saving Cost saving Cost saving Cost saving Cost saving Cost saving ? 100 ? 4,662 ? 4,200 ? 3,780 ? 3,402 ? 3,062 ? 2,756 ? 2,480 ? 2,232 ? 2,009 ? 00 ? 7,459 ? 6,720 ? 6,048 ? 5,443 ? 4,899 ? 4,409 ? 3,968 ? 3,571 ? 3,214 ? 300 ? 11,935 ? 10,752 ? 9,677 ? 8,709 ? 7,838 ? 7,054 ? 6,349 ? 5,714 ? 5,143 ? 400 ? 19,096 ? 17,203 ? 15,483 ? 13,935 ? 12,541 ? 11,287 ? 10,158 ? 9,142 ? 8,228 ? 500 ? 30,553 ? 27,525 ? 24,773 ? 22,295 ? 20,066 ? 18,059 ? 16,253 ? 14,628 ? 13,165 ? 600 ? 48,885 ? 44,040 ? 39,636 ? 35,673 ? 32,105 ? 28,895 ? 26,005 ? 23,405 ? 21,064 ? 700 ? 78,215 ? 70,464 ? 63,418 ? 57,076 ? 51,368 ? 46,232 ? 41,608 ? 37,448 ? 33,703 Date of preparation: April 2009 4 NPR09/1109 What is sensitivity analysis? e stated that the 95% confidence interval is 42% to 77%. In probabilistic sensitivity analysis, rather than assigning a single value to each parameter, computer software (such as Crystal Ball, TreeAge, WinBUGS or other software) is used to assign a distribution to all parameters in the model. The ranges are determined by: G The average value G The standard deviation G The ‘shape’ of the spread of data. Care is taken to ensure that all parameters remain practical.

For example, probabilities must always remain between zero and one, while costs can never be negative.Each time the model is run, the software will be able to randomly ‘select’ one value for each parameter and record the model’s results. If the model is then run a great number of times (in some cases, more than 100,000 ‘iterations’), the software will record the result each time, and present the variation in results. The results are typically presented using a cost-effectiveness scatter plane, where each iteration is plotted on a chart showing the incremental cost and incremental effectiveness of the intervention in question, as shown in Figures 3a and 3b. Figure 3a shows a model whose inputs are Figures 3a and 3b.

Cost-effectiveness scatter planes Incremental cost-effectiveness plane ? 80,000 ? 60,000 ? 40,000 Incremental cost ?20,000 ? 0 0. 0 0. 2 0. 4 0. 6 0.

8 1. 0 1. 2 1. 4 1. 6 1. 8 2.

0 -2. 0 -1. 8 -1. 6 -1. 4 -1.

2 -1. 0 -0. 8 -0. 6 -0. 4 -0. 2 ?20,000 ? 40,000 ? 60,000 ? 80,000 Incremental effectiveness (QALYs) Incremental cost-effectiveness plane ? 80,000 ? 60,000 ? 40,000 Incremental cost ?20,000 ? 0 0.

0 0. 2 0. 4 0. 6 0. 8 1.

0 1. 2 1. 4 1. 6 1. 8 2. 0 -2.

0 -1. 8 -1. 6 -1. 4 -1. 2 -1. 0 -0.

8 -0. 6 -0. 4 -0. 2 ?20,000 ? 40,000 ? 60,000 ? 80,000 Incremental effectiveness (QALYs) Date of preparation: April 2009 NPR09/1109 What is sensitivity analysis? relatively uncertain and, as such, has a large spread of incremental results. In contrast, Figure 3b shows the same model, but whose inputs are associated with a higher level of confidence, resulting in a ‘tighter’ spread of results.

It should be noted that the average results for each model are identical; only the level of confidence differs in each case. For illustration, a dashed line has been added to show a ? 30,000 cost-effectiveness ‘threshold’. Figure 4 illustrates the range of incremental cost-effectiveness ratios (ICERs) for each of the models.It can be seen that the first model shows a wide range of results, while the second model’s results tend to be grouped more tightly. This figure can, therefore, be used to give an indication of the proportion of iterations in each model that provide results below a prespecified costeffectiveness threshold. In turn, this can allow the model user to quantify the level of uncertainty around a model’s results by showing the percentage likelihood that the cost-effectiveness of the technology will fall below that threshold.

As the threshold (or willingness-to-pay) increases, theFigure 4. Range of costeffectiveness results 250 – Model 1 Model 2 200 – Frequency 150 – 100 – 50 – 0– ? 0 ?10,000 ?20,000 ?30,000 ?40,000 ?50,000 ?60,000 Incremental cost-effectiveness ratio Figure 5. Costeffectiveness acceptability curve 100% – 90% – 80% – Proportion cost-effective 70% – 60% – 50% – 40% – 30% – 20% – 10% – 0% – ? 0 ? 10,000 ? 20,000 ? 30,000 ? 40,000 ? 50,000 ? 60,000 ? 70,000 ? 80,000 ? 90,000 ? 100,000 Willingness to pay per quality-adjusted life-year Model 1 Model 2 Date of preparation: April 2009 6 NPR09/1109 What is sensitivity analysis? roportion of iterations falling below that threshold will increase. This is shown in the cost-effectiveness acceptability curve (CEAC) in Figure 5. In this curve, it can be seen that for a willingness-to-pay value of ? 30,000 per quality-adjusted life-year (QALY), for example, Model 1 indicates that an intervention has only a 67% chance of being cost-effective, while Model 2 indicates that the chance of cost-effectiveness is around 98%.

This demonstrates that, even though the average cost-effectiveness in each model is the same, a decision-maker can have far more confidence in the results of Model 2.Furthermore, the level of confidence can be calculated in a quantifiable manner. Further reading 1. Drummond MF, Sculpher MJ, Torrance GW, O’Brien BJ, Stoddart GL. Methods for the Economic Evaluation of Health Care Programmes.

Oxford: Oxford University Press, 2005. 2. Briggs A, Sculpher MJ, Claxton K. Decision Modelling for Health Economic Evaluation. Oxford: Oxford University Press, 2006.

3. Gold MR, Siegel JE, Russell LB, Weinstein MC. CostEffectiveness in Health and Medicine. New York: Oxford University Press, 1996. Date of preparation: April 2009 7 NPR09/1109 What is..

.? series What is sensitivity analysis?This publication, along with the others in the series, is available on the internet at www. whatisseries. co. uk The data, opinions and statements appearing in the article(s) herein are those of the contributor(s) concerned.

Accordingly, the sponsor and publisher, and their respective employees, officers and agents, accept no liability for the consequences of any such inaccurate or misleading data, opinion or statement. Published by Hayward Medical Communications, a division of Hayward Group Ltd. Copyright © 2009 Hayward Group Ltd. All rights reserved. Supported by sanofi-aventis Date of preparation: April 2009 8 NPR09/1109