EdTech 505 Home

Evaluation Scenario

Group Definition Page

Project Idea Page

Evaluation Proposal

Evaluation Plan Review

Evaluation: Response to A Solo Performance

Final Project Documents

Survey Data Report

Task List

Observation Protocol

Opt Out form

Instructional Evaluation Survey

 


Final Project

Evaluation

 

Response to a Solo Performance
Introduction to Theatre Studies (H28.0500) Fall 2008
New York University
by Paul F. Desmarais
A report submitted to Dr. Jeremy Tutty December 2008

(download report in PDF format)

Introduction:

The Paul F. Desmarais Research Group, in response to a request for proposal by Dr. Jeremy Tutty, conducted the following evaluation of the class Introduction to Theatre Studies (H28.0500); Response to Solo Performance section. The general purpose of this course is to familiarize first year Theatre Arts students with the core principles of the theatrical arts.

The goal of this formative evaluation report is two fold. First, to insure that this unit of instruction is meeting the university’s goal of a Theatre curriculum that prepares its students for careers as performers and provides for the possibility of an academic career in the arts as well. For a rigorous academic standard and second, to discover student attitudes about introductory course work. The major steps taken during this evaluation included an interview with the course instructor, a review of courseware, classroom observation, a student survey. Following data collection, an evaluation and compilation of the data was undertaken, followed by the packaging of the data into chart & Graph form.

The product of the evaluation is a report that addresses the major goals of the stake holders, namely, it conducts a comparison between the Theatre major at the Tisch School for the Arts with the Theatre programs at four universities known for having excellent Theatre programs. Second, it evaluates the academic rigor of a the unit of instruction that is the focus of the evaluation. Third, it measures the degree of engagement of the students in their class work during this instruction, and fourth, it measures student attitudes toward the class in general and this unit of instruction in particular as the coursework relates to their goal of a career in Theatre.

Program Description:

The program being evaluated is the Response to Solo Performance section of a freshman theatre course; Introduction to Theatre Studies (H28.0500) at New York University’s Tisch School of the Arts. The class is one of 81 separate course offerings in 9 areas of study and within the Drama Department. The course is taught by Cornelia Cody. Professor Cody is one of 75 instructors in the Drama Departments. She is a poet (she has read her poetry on NPR), an actor (she has performed at Arena Stage, The Shakespeare Theater, Center Stage, among others), and playwright (resident playwright at The Discovery Theater at The Smithsonian Institute). This course is the second class for students undertaking a major in Theatre at NYU. It is a requires, or core course for Theatre majors. All of the students in the class are Theatre majors. The goal of the course is to familiarize drama students with the core principles of the theatrical arts. These principles include close reading, analysis, critical thinking, and argumentation as they pertain to the dramatic arts. This course is the second for students undertaking a major in Theatre Studies at NYU’s Tisch School of the Arts. The major is designed to give “students the artistic and intellectual foundations necessary for a successful professional life in the theatre and allied disciplines”. The class is also a first step in a major that endeavors to “prepare students to pursue other career choices within the field, including graduate study in fields such as theatre history, dramatic literature, and performance studies”. According to the instructor, the course aspires to meld the academic and intellectual aspects of with the elements of performance and criticism.

Students attend lectures, read plays, view films, attend live dramatic performances, perform short pieces in class, and participate in class discussions. The course requires students to read 6 plays, watch four films, and attend 5 dramatic performances. In addition, the students are issued a course packet of required reading materials. There are 34 reading assignments in the course packet as a whole. These materials include scholarly papers on a variety of theatre arts topics, criticism, analysis, essays, as well as articles from theatre trade publications, newspaper and magazine reviews, and other materials that discuss the role of the dramatic arts in society. Students produce 5 written assignments and take a mid-term examination as means of assessing their understanding of the materials.

For the purpose of this evaluation, we will focus on the unit of instruction titled Performance Studies: Response to Solo Performance. This unit of instruction includes 5 readings from the course packet, a viewing of solo performance by Danny Hoch (a New York-based comedian) and Anna Deavre Smith (A Pulitzer prize finalist in drama, actress, and poet known for her solo performances), an in-class performance/ demonstration of solo performance principles, and class discussions following the readings and viewing. Students are graded based on attendance, participation, and a paper turned in at the end of the unit.

Evaluation Methods:

The participants in the evaluation are college freshman at New York University’s Tisch School of the Arts, taught by Professor Cornelia Cody. The sources of data for this evaluation will be the instructor via interview, students via survey and in-class observation, course attendance figures, and the course documents themselves.

Evaluation procedures include a review of course materials to compare the materials in this course to a “best practices” set of materials determined by comparison to the materials used in similar courses at other institutions that offer a Theatre Arts major. Comparison will include how much reading is required of students, how many different types of materials students are required to digest, as well as the range of topics covered by reading materials. The other institutions selected for the comparison are Suffolk University in Boston, MA; Northwestern University, in Chicago, IL; University of California-San Diego in San Diego, CA; and the State University of New York-Plattsburgh in Plattsburgh, NY.

The in-class observation will occur during three class sessions (two visits to one section of the course and a single visit to a second section of the identical course) focused on the topic of Response to a Solo Performance. For the observation, the evaluator will compare the student behavior to the protocol listed on the observation form (see Appendix A) and mark the appropriate box for each of the interactions as they take place. The measurements are intended to measure class engagement, preparedness, and knowledge of the material based on response to questions, participation in the discussion, and ability to perform in an unannounced skit relating to the course reading materials as directed by the instructor.

The class survey was issued to each of the students in the 2 sections (20 in one class, 21 in the other; 41 in all) and collected after the second observed class. Some students elected to mail the surveys. The survey instrument (see Appendix C) was constructed to measure student attitudes about the course and how they felt the class prepared them for their major, how well the course materials supported their studies, if the exams and papers assessed what was actually taught, as well as how closely the class followed the syllabus. The instructor interview focused on how her program content compared with the content in other similar courses at other institutions, how she selected course materials and decided on which assessment method was selected for the unit being evaluated. As the instruction covered by this evaluation will be post mid-term examination, those grades have zero value in assessing the unit of instruction under consideration for this report, and will, therefor, not be used in assessing the instructional content. As this evaluation is due prior to the due date for the student’s paper on this topic, those grades, while of great value, will not be available for evaluation.

This evaluation will measure student performance via observation during in-class discussion, student attitude, by survey, and the course materials by comparison to materials used in similar courses at other institutions. Instruments will include a ten question survey (see Appendix C) that measures student attitudes vís a vis the instruction covered by this evaluation, an optional ‘opt out’ card, in the event that course attendees did not wish to participate in the survey (see Appendix B), an observation protocol measuring interaction during class discussion (see Appendix ‘A’).

The classroom observation protocol was constructed to assign positive values to interactions between the course instructor and the students that indicated engagement and knowledge, while assigning negative values to interactions that indicated a lack of preparedness or off topic interactions that detract from the effectiveness of the instruction. In constructing the protocol and assigning values; the intent was to construct a criteria-based means of evaluating class participation of the students in the two sections. As a means of randomizing the measurement and also protecting the anonymity of the participants; students assumed the identity of the chair in which they sat, 1-23, as opposed to assigning their responses to them as individuals. So each chair in the classroom was assigned a number, 1-23. While there were 20 student in one class, and 21 in the other, in some instances there was no student sitting in a seat. When a student sat in chair #1, that student’s behavior was assigned to ‘student #1’. Three class sessions were observed, an in each instance, which ever student sat in seat #1 became student #1 for the purposes of the observation. The student who sat in seat #2 became student #2; and so forth. The following criteria were used in assigning values to class interactions.

On topic, unsolicited peer to peer/ student to teacher participation in discussion was recorded as a positive three (+3). Voluntary correct responses to teacher prompts (students raising their hands and correctly answering a question) as well as on topic student to teacher inquiries was recorded as a positive two (+2). Involuntary (called on by the teacher) correct responses will be recorded as a positive one (1). Off topic school-related student to teacher inquiries were given a zero (0) value, since the subject was related to their enrollment in the university or major or class, but not directly linked to the current discussion, the rationale was that these questions were important to the student’s education or performance, but not precisely on-topic, the interaction was neither particularly positive or particularly negative. Voluntary incorrect responses were awarded a negative one (-1) value. Involuntary incorrect responses were assigned a value of negative two (-2). Off topic peer to peer discussions were assigned a value of negative three (-3).

The goal in constructing the observation protocol was to create a means of assigning a quantitative value to what is essentially a qualitative measurement so comparisons from one group to another, and one student to another can be made using an established means. While the criteria is, in a sense, arbitrary, it is also applied equally, and anonymously to students based on seat assignment, so no inference can be made of a particular student.

Results:

An evaluation of the program by comparison to programs with similar goals and objectives yielded the following results. The NYU Tisch School of the Arts Program was far more academic in nature than any of the other four programs selected for the comparison, even though these four programs were among the more academic of the approximately two dozen programs reviewed for inclusion in this evaluation. This is in line with the program goals and objectives. According to the program objectives, this major is designed to not only produce actors, directors, and designers, the traditional focus of Dramatic Arts majors; but also teachers, scholars, and theorists. More particular to this specific unit of instruction under evaluation, the detailed nature of the Response to a Solo Performance section of instruction was not duplicated in any of the other five programs examined. This unit of instruction included readings, the viewing of films, the viewing of two solo performances by artists known for and respected within the theatre community as cutting edge solo artists, acting, and a 10 page written discussion of the subject. Only two other programs included a similar unit of instruction in the introductory course, and neither of those required as much academic work as Introduction to Theatre Studies (H28.0500). These institutions offered a study of this subject matter, but in more advanced classes, which, focused on either the acting or writing aspect of the solo performance, and did not appear to include an academic or theoretical approach to the material. For the purpose of this evaluation, this comparison was not material.

Course materials provided to students included a syllabus, course packet of required readings, and a schedule of were of good quality in general, and when compared to materials used in other similar courses, were in general either at or well above the quality of materials used in other, similar courses. The amount of reading required was significantly greater than students in similar courses at other institutions. The mandatory viewings and attendances of live performances were entirely unique to this class.

The student survey (see table 1.01) provided an interesting insight into the attitudes of those students enrolled in the class. The 41 students enrolled were all Theatre majors. The ten question student survey focused on four topics. How closely the assigned materials (syllabus, course packet, and required plays/films) matched what was actually taught and assessed. How well the students felt the courseware, either as class presentation or written assignments; prepared them to advance in their studies. Were they dedicated to their course of study. (For example, did student experience; either good or bad; with this unit of instruction, have an effect on whether they would continues to study Theatre). Finally, did students feel the instructor had an impact on the class? The profile of the student responses was fairly uniform in producing a portrait of a student body that was focused, prepared, and dedicated to their course of study. Based on the survey results, the students felt that the effect of this unit of study within this class was fairly limited. This conclusion is the result of comparing the answer to question #6, in which the students were asked if they planned to continue as Theatre majors. The uniform answer to this question was a 10 for 10 ‘strongly agree’. This answer was in spite of a more ambivalent response to questions indicating their feelings about that classes’ effectiveness in preparing them for the increasingly more difficult work at the next level of the major. Students also felt that their course materials, readings, and viewings were well coordinated with the in-class discussion and subsequent assessment. No student responded with anything less than a ‘no opinion’ response in this area, which consisted of four questions. It was interesting to note that while students felt the class work matched the syllabus, and was related to the subject they were studying, they also were of differing opinions on whether the work prepared them for the next level of study and to what extent the instructor was an important factor in the process.

The class observation protocol produced a number of interesting results. The most intriguing result in the survey was that out 146 verbal in-class interactions there were no (zero) instances in which any of the 41 attending students voluntarily offered an incorrect response to a teacher prompt. Second, the mode for the observation was -2. Six seat recorded a -2 total for the protocol, in which the positive values of interactions were totalled, and the negative values were deducted, yielding a sum for that seat. The most often recorded result was -2, with 6 seats out of 23 providing a -2 sum. As -2 was the low point of the range of data, it was interesting that -2 would also prove to be the mode. The expectation was that the mode would be closer to the mean of 5.9, which, plainly, was not the measured result. (See table 1.02)

The average number of interactions per seat was 5.8; with a range of 2 to 13 interactions (of the 7 described types of interactions measured) occurring in the three class sessions observed. (See table 1.03) Empty seat results were discarded. The mode for the number of interactions was 5. The overwhelming number of interactions occurred on the plus side of the scale; that is, indicated a positive involvement in the coursework, though peer to peer of topic discussions were an issue at times.

Finally, during an interview with the professor on the topic of the instruction, it was clear that she had command of the material in general, and knew without referring to notes or the course documents the nature of the assigned work, the timing of the lessons (when certain subject matter had been covered or would be covered) relative to the syllabus and the student’s current work and achievement in class. Referring to the syllabus by notes, the interviewer asked a series of questions about when material was introduced, how students reacted to it, and how they performed on any evaluations or in class discussions related to the materials. The professor answered the questions without having to refer to her own course packet or syllabus, which seemed to indicate she knew precisely the answers to the interviewer questions.

Discussion:

The purpose of the study was to review the overall quality of the instructional unit Response to a Solo Performance, compare the course materials the course’s stated goals of familiarizing students with the core principals of the dramatic arts. In addition, the instruction was measured against the Theatre Department’s stated goal of preparing Tisch School of the Arts students for careers in all aspects of the Theatrical Arts. That goal is to train Tisch students as performers, directors, and designers; and, additionally; to prepare them for careers as academics, theorists, and teachers. This evaluation examines student attitudes toward the instruction, measures the level of engagement between the students and the materials prepared to teach this subject matter, and examines the instructor’s ability to relate the course material to her students. The evaluation also measured the level of engagement of students during classroom discussions.

The results of the evaluation were interesting in many regards. First, it was clear that students were engaged in their course work at a high level. The overwhelming majority of students indicated by their participation in class discussions that they had attended required performances, done (and in many cases enjoyed) required reading, and were ready to answer questions on the assigned topics. Student attitudes toward the course and the assignments, both academic and vocational, were positive. Attitudes toward the instructor and her role were more ambivalent. The instructor, for her part, seemed well prepared, clearly had mastery of the subject matter, and was both organized and positive in her presentation of the materials. In comparing the Introduction to Theatre Studies (H28:0500) materials to other introductory theatre arts courses, it was almost immediately apparent that the NYU class was unique in its approach. Concordant with its stated goals of demanding a high level of academic rigor, something also mentioned by the instructor during her interview, Introduction to Theatre Studies (H28:0500) included both vocational and academic work. Students were required to view and critique performances, perform skits to demonstrate mastery and discuss the subject matter in class, and to author research papers on the subject that were a significant portion of their final grade. Other introductory theatre classes tended toward the more vocational. They stressed acting, for the most part, and in some cases the entry level course for the Theatre major was an acting class. Finally, a review of the class materials; that is the syllabus, course packet, and assigned performances; determined that the assigned materials are relevant to the subject matter being taught, and the assessments reflect what was taught in class or assigned as reading or viewing.

The material review provided the most concrete data in many respects. The materials were prepared well in advance of the evaluation and the course was well under weigh when the evaluation was conducted. In this respect, the synergy between the course materials and the class room learning was impressive. For a class to be so precisely ‘on schedule’ after twelve to thirteen weeks of a semester is in the experience of the evaluator, not all that common. The examination of the course documents (as previously described) was the first concrete step in performing the evaluation. Procuring copies of the readings, and recordings of the required films and dramatic performances familiarized the researcher with the course, and brought into focus exactly how the materials supported (or did not support) the subject matter being taught. This review formed the foundation of the entire evaluation, and the results are the most significant. The class materials, in all regards, supported the course’s stated goals, matched what was planned in the syllabus, and was provided properly and completely in the course packet. The performances were by artists who were acknowledge experts in the area of instruction (Solo Performance).

The next most important data source was the in-class observation protocol. While the data gathered was qualitative as opposed to quantitative in nature, it provided an unbiased, third party picture of how students interacted with their instructor and one another. It also made plain the level of preparedness of the students in the class. The observation revealed that students had a very good idea of what they knew and did not know. No students incorrectly answered a question that they had voluntarily answered. Equally plain was that the students were well prepared for class. That is, they had done the reading, attended the required performances, and watched the assigned movies. Only on five occurrences out 105 direct teacher inquiries of a non-volunteering student; only twice did she receive an incorrect or unsatisfactory (to her) answer. The observation also provided data on how many interactions took place during the three class room sessions attended (393) and what the quality of those engagements were in regards to staying on topic during discussions and making inquiries. Sixty three, or approximately 16% of those 393 student actions were off topic.

The student survey was important in that it measured student attitudes about the instruction under evaluation as well as how they believed it met their needs, and how fairly they were assessed on it. The survey revealed a universal dedication to their chosen field of study, (10 out of 10 students indicted they would remain Theatre majors) a very high degree of positive feelings for the quality of the instruction, and an equally positive impression of how well the course materials were integrated into the instruction. The students were less positive in their assessment of the instructor’s role, and how well the Solo Performance unit prepared them to advance in their studies.

The instructor interview provided some insights into how the instructor valued the unit when putting together the course materials, and how she selected which performances to use, and how she would assess the students. Her responses were very much in line with the department goals of building a strong performance vocation on top of an equally and no less important academic foundation. The interview revealed that the instructor clearly knew the course materials well (not surprising, as she was teaching two sections of the same course, back to back) The interview also revealed that the instructor had an impressively positive attitude toward her students as potential dramatic professionals as well as students in pursuit of an academic degree.

The information accumulated comparing the program at New York University’s Tisch School of the Arts with four other well-known and respected (within the Theatre community) Theatre major programs provided some superficial data that supported (overwhelmingly) the Tisch School’s mission statement to amalgamate the vocational with the academic in teaching students the dramatic arts. This comparison was done via a materials review, comparing syllabi, course descriptions, instructor qualifications, and program mission statements to those obtained for Introduction to Theatre Studies. (H28:0500). In no case was any of the four comparison program as comprehensive, academic in orientation, or demanding of the students versatility as Introduction to Theatre Studies. (H28:0500). As previously stated, most of the introductory courses were straight acting classes, with little or no theory component, and no writing requirement. This difference may reflect a difference in philosophy, different program objectives, and/or differing beliefs regarding the value of academic instruction in the Dramatic Arts. In some cases, that was made plain in the reviewed materials, in others, it was not. More directly important to this evaluation, is that the solo performance unit of instruction in Introduction to Theatre Studies. (H28:0500) is of very high to exceptional quality in comparison to what is available to other first year Theatre majors. One final note about the data unearthed by this review. The quality of instructor available to Introduction to Theatre Studies. (H28:0500) was plainly superior to those available in the other reviewed programs. No other beginning Theatre program had an instructor with the same breadth and quality of performance experience and academic bona fides as the instructor of Introduction to Theatre Studies. (H28:0500).

This unit of instruction Response to a Solo Performance is a unique and well integrated topic within the overall course work of Introduction to Theatre Studies. (H28:0500). It is also well supported by its course materials. That integration, the unit’s support of the department mission, the course’s stated objectives, and the materials selected supporting the unit’s overall quality is truly impressive. As New York University in general, and the Tisch School of the Arts in particular, are recognized as leading performance arts educational institutions, this integration is not surprising, but it is, none the less, impressive.

Less impressive was the ambivalence students felt toward their instructor. This aspect of the evaluation was surprising given the instructors obvious control of the material, but it is an area of consequence, as student surveys showed a clear neutrality toward their instructor but expressing a majority of ‘no opinion’ when asked if they would take another class with this instructor. The instructor also teaches higher level Theatre courses, so it is an issue.

As a whole, the unit of instruction titled Response to a Solo Performance must be considered of a high quality. The approach to the material is unique in that it uses an approach that studies solo performance from both academic and vocation perspectives. It is also unique in introducing a topic which is generally reserved for higher level courses in other programs in a freshman undergraduate first or second semester class. The supporting materials (reading, viewing, and required performances) are tightly integrated into the program as a whole. In class discussions and the in class performance were also very much ‘on-topic’ further supporting the conclusion that the program is well balanced and well-designed.

There was a need, or at least, the possibility for improvement in three areas; revealed by different evaluation techniques. The in class observation noted a relatively low off-topic student actions. Most of those came in the time when students were either regrouping after performing a skit or when they were organizing themselves to perform. There was a bit of a free-for-all atmosphere during this time, that led to significant time off topic. The skits were not announced before class, something that was deliberate; so the instructor could get an immediate and unstudied idea of how well the students grasped three concepts of performance; being, doing, and showing doing. Concepts introduced into dramatic pedagogy by Richard Schechner. These very short skits took the majority of a class period to organize, but not very long at all to perform. The ad hoc nature of the skits left the majority of the students with nothing to do while the student next to perform was given time to organize themselves and prepare mentally.

It is also a concern that there was a disconnect between the student’s enthusiasm for the subject matter and their instructor. Student surveys revealed that students had an overwhelmingly positive attitude about their future in the Dramatic Arts, and evaluated the integration, course materials, assessments, fairness of the course either very positively or positively. Yet no student expressed ‘very strongly’ that they wanted to take another class with the instructor and 60% expressed that they had ‘no opinion’ which can be interpreted as either neutral or negatively. The best possible spin would be to rate the response as neutral, but even so, given the student’s response to the material as positive, and other aspects of the course as positive, the presentation of those materials warrants further study.

A final, an tangential recommendation would be for the two sections of Introduction to Theatre Studies. (H28:0500) to be arranged so they are not back to back. While this may have some positives in terms of instructor scheduling, there were clearly observed negatives in the form of students from one section discussing, for example, the requirement to perform an unannounced skit with students from the following section as they arrived.