Evaluation Planning

Evaluation Criteria

Evaluation Criteria Explanation Data Sources
Effectiveness:
Mastery of WBI goals.
  • Determine if participants are able to upload and edit digital photographs
  • Participants final sharing of web album
  • Pre-instruction learner survey
  • Post-instruction learner survey
Appeal: Gain and maintain learner attention and interest.
Ease of use.
  • Review content for interest.
  • Review message design for interest.
  • Review applications for ease of use, cost, and intuitiveness.
  • Review site for ease of navigation.
  • Participant’s opinion on course content, activities, message design, and delivery system.
  • Instructor opinion on ease of use in relation to course materials and delivery system.
  • Instructor and participant survey on ease of use.
Efficiency: Instruction delivered in a timely manner
  • Review course site and materials to determine time needed for completion.
  • Create course timeline to document instructor time in preparation, facilitation and feedback.
  • Document participant’s time.
  • Document any costs incurred in constructing or deploying the instruction.

General Evaluation Plan

For evaluating this course both the objective and participant models should be followed. In order to evaluate the WBI created for Basic Digital Photography course it is essential to determine the effectiveness of the instruction and the efficiency of the instruction as applied to learner achievement. Evaluation of the level to which this instruction helped participants achieve the objectives will determine whether participants are able to effectively use their digital camera and editing software. The final project of demonstrating use with a digital camera and editing software followed by sharing those images will determine WBI effectiveness. Furthermore evaluating how well the instruction meets the participant’s needs should be taken into account.

Determining the appeal of a WBI course can be subjective especially if it is not a mandatory course such as presented in the Basic Digital Photography course. Because no grades will be taken, a calculation based on how many participants initially signed up to how many participants completed the course can be used to signify an implicit appeal of the course. Peer and SME review can also determine appeal of course.


Back to Top


Evaluation Matrix

Evaluation Criteria & Categories Specific Questions Data Collection Methods & Tools
Effectiveness    
Goals
  • Is the information accurate?
  • Are goals and objectives clear and achievable?
  • Are the goals and lessons appropriate for web delivery?
  • Instructional Expert
  • SME
  • Assignments
  • End-user survey
Content
  • Do instructional activities promote thoughtful and reflective responses & discussion?
  • Are content, activities and objectives congruent with final web sharing project?
  • Do instructional activities promote learning?
  • Is the content complete?
  • Does the content take into account the most relevant features of basic digital photography and editing software?
  • End-user survey
  • SME
  • Instructional Expert
Technology
  • Do the technology applications function properly?
  • Does the course site effectively promote learning?
  • Were materials easy to access by students?
  • To modify by instructor?
  • Are copyright and intellectual property not violated?
  • Is there access to the instructor or other learners?
  • Instructional Expert
  • End-user survey
  • Observation
Message Design
  • Do supporting graphics support learning without offering distraction?
  • Is an appropriate voice used for expressing content to learners?
  • Was humor used appropriately?
  • Are directions clear?
  • Was the time frame of the course appropriate?
  • Does the text stand alone if the graphics are not available?
  • End-user survey
  • SME review
  • Instructional expert
Appeal    
Goals
  • Are goals relevant to learners?
  • Instructional expert
  • SME
  • End-user survey
Content
  • Is the content interesting?
  • Is the content enjoyable?
  • Instructional expert
  • SME
  • End-user survey
Technology
  • Are there errors in typography, spelling, grammar, punctuation etc?
  • Is navigation easy, intuitive?
  • Are materials easy to access by students?
  • To modify by instructor?
  • End-user survey
  • Extant data (emails, discussions)
  • SME
  • Instructor
Message Design
  • Are the media and messages pleasing to view?
  • Are the vocabulary and level and tone appropriate to the lesson and audience?
  • Are screens uncluttered and with appropriate white space?
  • Are color, typeface, and type style used appropriately?
  • Are site design elements and graphics used appropriately, and in a manner that enhances WBI learning?
  • Are screen layouts appropriate to the digital photography content?
  • Instructional expert
  • End-user survey
  • SME
Efficiency    
Goals
  • Is the purpose concisely stated?
  • End-user survey
  • SME
  • Instructional expert
Content
  • Are instructional goals and content aligned?
  • Is the content clearly, concisely presented?
  • End-user survey
  • SME
  • Instructional expert
Technology
  • Does all of the technology within the course function properly and intuitively?
  • Do the graphic devices function properly?
  • SME
  • End-user survey
  • Instructional expert
  • Extant data (amount of user activity, login time)
Message Design
  • Is the organization and structure of the message coherent?
  • Are titles and subtitles used to organize content?
  • Are there synchronous and asynchronous activities for students and instructor?
  • Is the time frame for the WBI appropriate?
  • SME
  • Instructional Expert
  • End-user Surveys

Back to Top


Stakeholders

Primary Stakeholders

Instructor: Lora Evanouski or any designated instructor, responsible for implementing the course and its content. Assist designer with ensuring that course meets needs of faculty in respective areas of responsibility.

Designer: Lora Evanouski, responsible for the design and development of the course, the development of evaluation plans, and conducting formative evaluation.

Students/learners: All adult learners participating in the course.

Secondary Stakeholders

Dr. K Diane Hall: Ed.D. Instructional Technology, Texas Tech University and Graduate Adjunct Faculty at Boise State University is responsible for ensuring proper instructive techniques in her class of EdTech 512, Online Course Design.

Department of Extended Studies: Department Chair of extended studies program at Boise State University responsible for content of department programs.

Back to Top


What is being Evaluated?

Materials to be Examined
Design Plans
  • Clustering of objectives
  • Objectives
  • Assessment Items
  • Instructional Strategies
  • Motivational Strategies
Prototype and Website
  • Function of Technology
  • Interface
  • Navigation features

Back to Top


Evaluators and Reviewers

Evaluator/Designer: Lora Evanouski is the course designer. She is pursuing her Master of Educational Technology (MET) at Boise State University.

Evaluator/Instructor: Lora Evanouski is set to be the instructor. She has 3 years teaching experience, holds several certificates of expertise in digital photography, has teaching experience with target audience, and is participating in the design and development. She is familiar with the campus and extended studies program.

Expert Reviewer (subject matter): Michael Evanouski will be the subject matter expert and expert reviewer as the WBI is designed and deployed. He is the Director of Web Services/Web Design for 6 auto dealerships throughout the greater Boise area as well as holding an undergraduate degree in fine arts. He has 15 years experience using digital cameras and editing software in personal use as well as for large consumer markets in the auto industry.

Expert Reviewer (instructional design): Dr. K. Diane Hall- Graduate adjunct faculty at Boise State University. Instructor of EDTECH 512, Online Course Design.

End-user Reviewer: Learners from the Basic Digital Photography course will be asked to participate in end-user reviews throughout the formative evaluation. Learners will be selected with differing technological abilities.

When and how should evaluation take place?

Formative evaluation will take collected throughout the design, development, and initial offering of the course.

One-to-One Tryout: At least 3 learners will participate in the one-to-one tryouts. The designer will implement the web design through development servers to ensure information is complete and that the message is appropriate for the participants.

Small-Group Tryout: As development nears completion, a group of 3-5 continuing education members of similar characteristics of target audience will be asked to interact with, and evaluate materials within the course and provide feedback via written instrument.

Field Trail: Because this course offering is for the first time it will be also be a field trial. It will be given at the end of the development phase to the target audience. It can be put into final form based upon field trial results.

Back to Top

What decisions need to be made as the WBI plan and prototype are developed and revised?

With each step in formative evaluation the data must be complied and evaluated. The course designer/evaluator will make changes based upon those evaluations from the instructional expert, subject matter expert, and survey data. Data from written feedback will be analyzed to find patterns. Qualitative data from any feedback (participants and instructor) will be used to determine any patterns or common themes. Extant data such as completed assignments, expert reviews, and practice exercises should be considered. All data gathered will ensure the proper requirements and standards given by the department of extended studies are met. All data can be achieved through Google docs for dissemination.

Preliminary Summative Evaluation Plan

Evaluation Criteria Main Questions Data Sources
Effectiveness
  • Did the course meet all the stated objectives?
  • Do learners believe the WBI was worth attending?
  • Were participants successful in using their digital camera and software?
  • Survey (to be completed at the end of curse)
  • Performance activity
Appeal
  • Will the participants enroll in another course offering using WBI?
  • Did participants finish course (or drop early)?
  • Enrollment numbers offered through BSU Extended Studies Program
  • Survey (end of course)
Efficiency
  • How much time did participants spend completing the course?
  • Survey to participants (end of course)

Back to Top