Evaluation Scenario

What are the benefits and limitations of an evaluation?

I see several benefits to evaluating the financial literacy course. First of all, a formal evaluation can provide credibility to the program which in turn could lead to opportunities to market this program on a larger scale, possibly nationwide. The evalation can also verify the value of the program for the students of the Maricopa Community College District, the partners from Cashflow Technologies, Inc., as well as the community in general. A thorough evaluation can also provide information on the effectiveness of the books, games, and videos.

An area of limitation that I see could be that since there hasn't been any seminars of training sessions to this point, it would be very difficult to assess the quality and worth that they provide to the big picture of the program. There may be training session and seminar information that is written and available to preview, but I feel that unless there has been an actual seminar and training session to evaluate, the assessment will be missing some valuable information.

Another limitation would be that even if the evaluation said that the program was of great value and worth, that wouldn't guarantee that educational institutes from across the country would jump on the financial literacy course bandwagon.

What factors ensure that an evaluation will be successful?

It is essential that some key questions are answered from an evaluation to ensure that the process was successful.

The evaluation needs to have objectives and goals in order to ensure success. My goal would be to find out if students learn more about financial literacy through this program than they would if they just tried to learn about it on their own.

The evaluation must have activities that are observable, collect data, and have a strong population /sample, therefore I would have a group of 100 students take a pre-test about financial literacy covering key terms and objectives from the program. From there I would randomly select a group of 50 students (from the 100 that took the pre-test) to take the course by using the books, videos, games, training sessions, and seminars. The other 50 students could try to learn as much as they could about financial literacy in any way other than using the program. At the end, I would give both groups a post-test to see how each group fared in their knowledge on the subject. Using a survey (pre-test and/or post-test) is another factor in helping the evaluation gain valuable information to analyze and provide the finding to the intended audience.

How might one use evaluation results?

Maricopa Community College District could use the evaluation's results so that Cashflow Technologies, Inc. will know whether to continue providing resources to the program "as is" because things are working or they may need to see a change in the program's direction before continuing with the resources because things are not working. A third choice from the evaluation would be to dump providing resources all together because they find that it is not worth the investment.

The MCCD staff needs to receive the evaluation's results to help in pinpointing their focus on any areas that may need improvement or know that what they are doing is working.

The answers to the three questions above apply to the following scenario:

Rich Kids, Poor Kids

The Maricopa Community College District (MCCD) partnered with Cashflow Technologies, Inc., to design and deliver a course in financial literacy. The course is based on the products - books, games, and videos - created and sold by the company. Complements to the course include training sessions for future course instructors and seminars at which attendees play the Cashflow game. MCCD commissioned an evaluation of the program to verify its value to students and the community. In addition, Cashflow Technologies is exploring the possibility of marketing the educational program to educational institutions throughout the nation; an educational evaluation of the program might lend it credibility. Cashflow Technologies has been producing the materials for three years. The course was offered for the first time during the spring 2008 semester. No training sessions or seminars have been performed.

The program will be evaluated by students in Educational Technology at Boise State University. The evaluators have access to sales information for the products, student performance data from the course, and contact information for the students and instructors. The budget for the evaluation is $6,000.