What is evaluation?
It's not testing! You are not finding faults to fix - they should have been found and fixed well before the system was implemented.
Evaluation measures how well the original ambitions of the new system (i.e. the logical design laid down during the analysis phase) have been achieved. Evaluation doesn't really serve to improve the system being evaluated; it serves to improve the next system you will work on. You learn lessons for next time when you evaluate.
Evaluation criteria
What should be evaluated in the new or changed system to see if it has been a successful project? Of course, it depends on what the aims of the system were when it was designed. Typical evaluation criteria include:
- speed
- accuracy
- quality of output
- reliability
- cost of operation
- attractiveness
- ease of use
- robustness
- capacity
- compatibility with other systems
- functionality (does it have the features it needs?)
- security
- flexibility (can it be upgraded, modified, adapted, tweaked, configured?)
- size, portability
Different critera apply to different systems. You have to ask yourself, "What is most important in judging whether this system is good or not?"
Evaluating a pet might include friendliness, healthiness, cost of upkeep, how dangerous it is. The same critera would not apply to a database! A database might be evaluated on its ease of use, its capacity, its speed, its security, its accuracy.
In short: the criteria used to evaluate something must be relevant to the thing being evaluated. Evaluating a pet's accuracy or a database's healthiness is irrelevant and useless..
Evaluation methods
Once you know what criteria you need to evaluate in a system, you need to devise a way to evaluate it. There are two main ways: objective and subjective. Objective evaluation involves collecting facts, figures and measurements. Subjective evaluation involves finding out opinions. Where possible, get objective measurements. Typical evaluation methods include:
- timing how long an operation takes
- counting errors recorded in an error log
- visually inspecting output for quality, listening to sound reproduction
- counting and assessing incidents of failure or downtime
- observing users to see how quickly they learn to use the new system, how often they need help or make errors
- stress-testing the system or abusing it in ways that would be expected in real-life operation
- using the system to its stated capacity, or beyond (e.g. filling a database or hard disk drive)
- using the system in conjunction with another system to see if data and hardware are compatible
- carrying out functions to see if they exist and work as expected
- attempting to break into the system or get access to secured data
- attempt to configure the system to different users, or add modules, or change the way it works to suit new needs
- weigh it, measure it, carry it around to see if it's comfortable.
When to evaluate?
After implementation. How long after system operation begins?
o If you wait too long, users remember less about the development process and how it might be improved
o If too soon, users have insufficient time to assess system strengths and weaknesses
Six months of operation is desirable, but pressure to finish sooner often exists. Wait until the system is 'bedded down' and users are familiar with it, then evaluate whether it has solved the original problem. Has it achieved the goal specified in the Problem Analysis?
If not, fix the thing! Don't sit around with a new system that is as bad, maybe worse, than the old system.
Ideally, post-implementation evaluation should be performed by people who were not involved in the development process. External auditors often are involved, since they are impartial and don't have a stake in the success or failure of the system.
|