Formative and Summative Evaluation


The utimate purpose of evaluation is to improve the instruction that is being designed.

Evaluation:


Evaluation, the process of determining a system's effectiveness, is typically split into three categories:
  • formative evaluation: the evaluation of the instruction performed while the instruction is being formed
  • summative evaluation: the evaluation of the instruction performed after the instruction has been implemented
  • learner evaluation: determining the performance change of the learners due to the instruction implemented


Formative Evaluation:


Formative evaluation is used throughout the instructional design process to check and refine the design. It should be implemented early in the process and continue throughout. The evaluation works with the design and development stages to form a recursive system of revision. The changes suggested by evaluation, once put into practice, need to be examined by further evaluation.

Formative Evaluation consists of several stages:
  • design review - checks whether the instruction that is designed meets the analysis
  • expert review - determines if the content is accurate and consistent
  • subjects testing - tests the instruction with various groups (one-on-one, small groups, and field trials)
  • ongoing evaluation - continually examines the design with respect to possible change in content, audience, or size


Specific Evaluation Methods:


To evaluate the Polarity instruction module, several methods will be used:
  • A Subject-Matter Expert (SME) will review the presentation slideshow for content accuracy and clarity.
  • Test subjects can view the slideshow and complete a survey about the slideshow.

One of the problems with typical online slideshow hosting (slideshare ) is that they do not support embedded audio, which in this case is essential to the material. To facilitate easy, online review, the slideshow has been converted to a movie to include the audio examples.

Two versions are below - an embedded YouTube video with fixed timings, and a link to a QuickTime Movie (.mov) file which can be downloaded.



The movie can be downloaded below. To advance the slide, simply click on the screen. QuickTime Player is required.



Survey:

After viewing the instructional slideshow, test subjects can complete the survey below, created via Google docs .




Summative Evaluation:


Summative Evaluation is carried out after an instructional design has been implemented to test the effectiveness of the design. This is often carried out by an outside party and seeks to determine is if the desired changes have been brought out by the instruction that was implemented.

In short, have the goals and objectives been met?

Survey Results:


Students in the Fall 2009 section of MUSC 565 were asked to view the Polarity Slideshow and fill out the survey. These students have already completed one audio course, and are therefore somewhat familiar with the concept presented.

Their responses proved to be quite valuable. The first thing I noticed, however, was a flaw in the structure of the survey itself. When I laid out the survey questions with a rating scale from 1 to 5, I labeled 1 as strongly agree and 5 as strongly disagree. 4 of the 9 respondents misread the question and filled out the survey as if 5 was strongly agree. The survey results shown below have been corrected.

Question
Strongly Agree
Somewhat Agree
Neutral
Somewhat Disagree
Strongly Disagree
The content is presented in a clear manner.
7
2



The slideshow is easy to navigate.
7
2



The slideshow content is arranged logically.
8
1



The slideshow text is easy to read and comprehend.
8

1


The graphics are relevant to the content.
9




The audio examples support the content.
6
1
2


The slideshow is effective.
7
2




Even more valuable were their open ended comment responses, which are included below:

  • Possibly make the volume differences more apparent. Consider using a larger font for the descriptions next to the diagrams. Also maybe use delayed animation for the comments so the information progressively enters to the screen.?.
  • I didn't hear any audio, so I wouldnt say the audio examples could be effective...Still, an effective slideshow
  • Very simple. Easy to understand. Perfect tempo. Gives me time to collect the information, analyze it and understand it before moving on to the next slide.
  • Perhaps the audio demonstration of what two wave forms of the same polarity sound like when combined would better suit following immediately after the explanation of two wave forms that share the same polarity. The same applies for placing the opposite polarity audio demonstration immediately after the opposite polarity explanation. Also, for balance sake, since there was a “different signals – same polarity” slide maybe add an additional slide titled “different signals – opposite polarity” where appropriate.
  • Makes a lot of sense.
  • I actually didn't hear any audio, the video is awfully quiet. I'm not saying music would help (I think that would hurt the video). But that's all I can say about it. It's very clear.
  • The video clearly explains the basics of polarity. It's short and concise with out a lot of jargon or fluff. If someone who knew nothing of soundwaves watched this, by the end they would certainly have a clearer idea about how polarity works. The text could be a little larger during the bits where you show both waves. Something that might also be benificial if this were to stay on the web might be someone reading the text aloud.
  • Very informative and nicely organized.
  • The slideshow if clear and describes the concepts well. The pacing of the slide show does not translate well into the video used on this site.

Continue to References