44.493201 Issues in Technology and Security

home > unit 3: developing evaluation criteria

Phase 3. Evaluate alternatives and make smart choices

Primarily chance to integrate and compare information. "This is the stage where you compare and contrast them along important dimensions such as risk, cost, and acceptability to key stakeholders."

"Usually one or two of the possible solutions seem better to you than the others. The analysis you conduct at this stage needs to use the full range of data to test these preferences and justify why a particular course of action is preferable.

"The most important limitation of any analysis is the quality and completeness of the data on which it is based. Weak data can't be improved by powerful analytical tools or fancy presentations. Be sure to pay attention to assumptions, estimates, and just plain guesses, and be honest with yourself and others in revealing what part they play in your analysis. Remember, too, that everyone has biases. Try to identify yours and counter balance them with solid analysis and reasonable alternative interpretations of the data."

".. you will make an explicit analysis of the risks associated with various tested alternatives. Notice that you have been identifying and dealing with risks from the outset. Opposing stakeholders represent risks, new technologies bring risk, and so on. Risk analysis can be very formal and quantitative or more qualitative. Scenario building is a technique in which you describe a hypothetical situation in a way that helps you predict the consequences of decisions and actions. It helps you see what could go wrong in different situations. You could also model these situations on paper or in simulations to understand how different actions or decisions might lead to unexpected results. This kind of analysis improves the confidence level of decision makers. These realistic projections of risk will help them understand the possible consequences of different choices."

• Compare costs and expected performance improvements
  Do detailed cost analysis at this point, including all long-term and indirect costs.

• Can get cost estimates from "..historical data such as budgets or spending records, feasibility studies, or from outside consultants or agencies that have attempted similar projects. A cost-benefit analysis can be a simple comparison of costs and projected savings, or it can be a more detailed financial model." (Baseline case studies can be great models for this process).

• Make and explain final choices
  May have a clear best choice by now. If there are still at least 2 viable alternatives, "..selection tools will be useful. These are designed to compare alternative courses of action in a structured and explicit way.
• multi-attribute utility model or MAU model
  Based on specifying weighted evaluation criteria (total cost, convenience to users, time to completion, reliance on contractors, etc.). Give each criterion a weight relative to others (often by distributing 100 points among them). Then you score each alternative solution on all criteria, so each alternative gets a total score. May result in a clear winner, or may require other decision-making processes to make final decision. However, "Occasionally the numerical scores do not 'fit' the intuitive assessments of the evaluation group or decision maker. Often this means that an important criterion is not being made explicit and should be added to the model."
• SWOT (for "strengths, weakness, opportunities, and threats") analysis presents another way to compare alternatives. This and other prioritizing and choice-making tools to consider will be described in the tools installment.

1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11