Summative Evaluation Plan

The Teacher's (Multimedia) Toolbox

In order to determine the efficacy, efficiency, and appeal of Web Based Instruction created for The Teacher's (Multimedia) Toolbox it is necessary to create a framework to conduct a summative evaluation of the program. This evaluation will occur well after implementation, once the instruction has been used by a significant number of students, and is considered 'mature' and is no longer undergoing major revisions.

The summative evaluation plan of this WBI will consist of an examination of the components of the instruction, learner attitudes toward the instruciton, and various performance data that when examined in combination, will provide an accurate and detailed picture of how this implementation of WBI is performing.

Preliminary Summative Evaluation Plan
Areas of Consideration Sample Questions
Instructional Goals and Content: Effectiveness
  • Did learners master their objectives?
  • Were the goals of the WBI met?
  • Is there evidence that the project was developed using an instructional design process?
Efficiency
  • Were the goals achieved?
  • How well were goals achieved?
  • Were costs reduced?
  • Were there costly technical support issues related to the WBI?
Appeal
  • What recommendations can be made for future improvements?
  • Did the message design provide clarity?

Learners and Instructors: Effectiveness
  • Was the instruction informative to learners?
  • Were the stakeholders satisfied with the outcomes?
  • Was the instructor satisfied with the outcomes?
  • Did learners drop out of the WBI due to incompatible technology?
  • Did learners drop out of the WBI due to a lack of technical support?
Efficiency
  • Did the instructor find the website easy to use?
  • Did learners find the website easy to access?
  • Were there design features that reduced the efficiency of the WBI?
Appeal
  • Did learners find the WBI interesting?
  • Did learners find the WBI helpful?
  • Did learners find the WBI helpful?

Context: Effectiveness
  • Did the WBI fit the website environment?
  • Is the course on a stable system?
  • Does the site work with the lowest specifications of computer or web browser?
  • Were the participants' computers and server compatible?
Efficiency
  • Was there easy, smooth access to the website?
  • Were time efficiencies realized?
Appeal
  • Did technical support meet participants needs?
  • Are disclaimers and cautions provided?
  • Are the authors clearly identified?
  • Did the photos, videos, etc. enhance the WBI?


The summative evaluation identifies the people, materials, and resources involved in delivering the WBI. In this respect, the summative evaluation plan is very similar to the formative evaluation plan used to guide the construction and implementatio of the WBI. The difference between the two evaluations is really the purpose of the evaluations themselves. The formative evaluations are conducted as means of improving the WBI before and during implementation. The summative evaluation is designed to be used to pass judgement on the future of the WBI

While the questions asked by the summative evaluation plan are similar to the formative evaluation are similar, the answers are no necessarily the same. The definition of stake holders, for example, changes with the purpose of the evaluaiton. Primary stakeholders are no longer students and instructors. Primary stakeholders are the decision makers involved in deciding the effectiveness, efficiency, and appeal of the instruction.

Evaluation Planning Worksheet
Summative Evaluation Plan
Who are the stakeholders?
  • Primary: Director of the Distance Education department, Instructor, Paul Desmarais
  • Secondary: Learners, Distance education support staff.
What is being evaluated?
Professional development course "The Teacher's (Multimedia) Toolbox"
  • Effectiveness of video tutorials
  • Effectiveness of WBI structure (chunking)
  • Quality of applications used in WBI
  • Effectiveness of applications selected for WBI
Who are the evaluators?
The Evaluators for this course will be internal evaluators
  • The instructor
  • Distance Education support staff
What are the evaluation methods and tools?
Evaluation Methods
  1. Qualitative
  2. Quantitative
Evaluation Tools
  • Learner opinion surveys
  • Instructor observations
  • assessment data
  • time on task data
How are data collected and analyzed?
Data will be collected at the begnning of instruction, during instruction, and at the end of instruction.
How are results reported?
Results are reported in the form of a report generated by the instructor.
  • Learner completion data will be reported at a percentage of enrolees
  • Learner satisfaction data will be collected & reported via opinion survey
  • Learner perception data will be collected via opinion survey
  • learner time on task will be collected by observation and reported as an avrage

In order to implement the summative evlauation, it is necessary to create a tmieline that will guide not only the evaluation itself, by the data collection and reporting to be used in the evaluation. By following the timeline, evaluators, can plan and coordinate their data collection and analysis; and so produce their final report in a timely fashion.

Data Collection and Analysis Timeline/ Gantt Chart
Tasks Time wk1 wk2 wk3 wk4 wk5 wk6 wk7 wk8 wk9 wk10
Obtain Consent Est.
Actual
Learners complete baseline survey Est
Actual
Interview learners before WBI begins Est
Actual
Observe learners during WBI Est
Actual
Distribute evaluaton survey to current learners Est
Actual
Gather pre-instructional baseline data Est

Actual

Interview instructor Est
Actual
Interview decision makers

Est

Actual
Analyze Data Est

Actual

Develop report Est
Actual
Develop recommendations Est
Actual
Communicate results to stake holders. Est
Actual

© 2009, Paul F. Desmarais: All Rights Reserved | Boise State University | Department of Extended Studies | Contact