Cornell University Center for Teaching Excellence

• Faculty Seminar Series
• Teaching Evaluation Handbook
• Teaching Materials
• Learning and Teaching Related Resources
• Bibliography

Cornell University Teaching Evaluation Handbook

Third Edition, 1997

Table of Contents

Introduction

Chapter 1 - A Conceptual Overview

Chapter 2 - The Teaching Portfolio: Documenting Teaching and Its Improvement

Chapter 3 - Supporting Data: Collection and Presentation

Chapter 4 - Criteria for Evaluating Data on Teaching

Chapter 5 - Improving Practice: Case Examples

Appendix: Evaluation and Recognition of Teaching - A Report of the Select Committee

References

Bibliography

Chapter II—The Teaching Portfolio: A Model for Documenting Teaching and Its Improvement

This discussion of the documentation of teaching for tenure and promotion is based on the assumption that a tenure file should provide thorough documentation of the process whereby the candidate was evaluated, in terms of both research and teaching. Lack of thorough documentation is a liability to the candidate, department and college, on legal, ethical, intellectual and efficiency grounds. It is recognized that there is much more to the tenure process than what is, or can be, put in a file. However, the file is the primary document used to make the candidate’s argument for tenure and should therefore reflect a degree of thoroughness and detail sufficient to stand on its own as a source of evidence.

Inclusiveness and length are two competing factors that must be considered in documenting teaching and its development. Inclusiveness has to do with whether there is enough data available to all those who must make a decision and whether that data represent the full range of activities and responsibilities associated with the candidate’s teaching. Offsetting inclusiveness is the issue of length: has the available data been reduced so it is in a manageable and digestible form without biasing or distorting the facts? What format the data is in is another important factor. Different data will require different format guidelines, yet all data are related in various ways and should not be presented in isolation from each other. As suggested in Chapter 1, a range of data sources on a candidate’s teaching effectiveness improves the quantitative objectivity by which that candidate is evaluated. If a major criterion by which the candidate is evaluated is the improvement of teaching practice, the candidate is one of the best sources of data to document improvement.

This chapter will focus on a tool that can be used by the tenure candidate for documenting the improvement of teaching, the “Teaching Portfolio” (Edgerton, Hutchings, Quinlan, 1991; Edgerton, 1991; Seldin 1989, 1991; Millis, 1991). According to Seldin, the teaching portfolio “...would enable faculty members to display their teaching accomplishments for examination by others. And, in the process, it would contribute both to sound personnel decisions and to the professional development of individual faculty members....It is a factual description of a professor’s major stengths and teaching achievements. It describes documents and materials which collectively suggest the scope and quality of a professor’s teaching performance..”1 The Teaching Portfolio has been chosen as a model because it connects summative and formative evaluation functions in a single process, it honors teaching as a scholarly activity, it is a practical and efficient way to document teaching and its development over time, and it has been experimented with at several institutions. The construction of a teaching portfolio raises issues and questions that must be considered by the candidate and administrators engaged in the evaluation of teaching. Figure 3 illustrates what a Teaching Portfolio is and can do.

The teaching portfolio should be representative enough that the key dimensions of teaching as a scholarly activity are evident. “When defined as scholarship...teaching both educates and entices future scholars....As a scholarly enterprise, teaching begins with what the teacher knows. Those who teach must, above all, be well informed, and steeped in the knowledge of their fields....Teaching is also a dynamic endeavor involving all the analogies, metaphors, and images that build bridges between the teacher’s understanding and the student’s learning. Pedagogical procedures must be carefully planned, continuously examined, and relate directly to the subject taught....teaching, at its best, means not only transmitting knowledge, but transforming and extending it as well.”2 When so conceived teaching encompasses both a logical dimension— in the selection and representation of the subject matter, and a psychological dimension—in the consideration of the process of translating and retranslating that subject matter into a form which is available to students.

At the same time the portfolio has representative breadth, it is also selective. Criteria for inclusiveness must be established which limit the bulk and form of data to a manageable amount. The selection process  should preserve the criteria of representativeness of primary teaching responsibilities, yet reduce and transform the available data into a manageable form which insures efficiency during the subsequent evaluation process. Selectivity is governed by structuring the portfolio into 2 major components (see Figure 4 below): work samples which consist of the details of what was taught and what its impact was on students, and a reflective commentary which extends the meaning of the work samples selected by providing a context in which to comprehend their design and choice from the teacher’s own point of view.

Work samples (see Figure 5 below) constitute direct evidence of teaching such as facts, objects, and reproductions of events from daily practice. Work samples should be selected which “...highlight what is unique about an individual’s approach to teaching.” 3   Just what samples are selected must be negotiated between the candidate and department. From the candidate’s point of view, selection will most probably be governed by an intimate knowledge of what was done, its effect, and how it changed over time. Department decisions about what constitutes adequate sampling will be governed by a consensus of the key scholarly dimensions of teaching in that field.

According to Edgerton, Hutchings and Quinlan (1991), the work samples are artifacts of teaching performance, while the reflective commentary which accompanies each artifact provides the teacher’s rationale for using that artifact and how it was developed. The reflective component of the portfolio is a kind of annotation to each sample of work. While teaching consists of both behavioral and cognitive aspects, there are also ethical aspects which somehow must be discussed. These can be documented in the reflective part of the portfolio and include discussions of what ethical principles and values guided the candidate’s approach to key decisions made about teaching and its improvement, like how multicultural and gender factors were dealt with in both course design and in classroom performance.

The following is an example of how work samples can be connected to a reflective statement.

Examples of Course Materials 4 Description of Particularly Effective Teaching Strategies Dr. Beverly Cameron Department of Economics, University of Manitoba 1990/91:  Class syllabus [A copy is included in the dossier.] Course goals, evaluation procedures (and dates), test(s), office hours, small group feedback sessions, reading assignments and a schedule of lectures are listed.  The innovative section of the syllabus includes: (a) expectations that students will develop and use effective thinking skills (See Small Group Problem Sets, and Goals of Small Group Work on p.2) ; (b) rationale for small group work on problem sets (see Small Group Problem Sets, Goals of Small Group Work, and Calculation of Problem Set Grades on pp. 2-3); (c) an explanation of effective thinking and problem solving skills (See The Process of Effective Thinking and Problem Solving on pp. 3-4); and (d) purpose of the research project and benefits to students from participation (See The Motivated Strategies for Learning Questionnaires [MSLQ] and the “ordered tree” of concepts on p. 5). 1990/91:  Examples of problem sets and answer keys. Problem sets are designed to give students an opportunity to use their knowledge of economics to solve problems and reach conclusions.  Students are specifically asked to follow explicit problem solving steps (e.g., Guided Design steps) which have been shown to be helpful for novice solvers.  [Examples are included in the dossier.] 1988 to present:  Tests with higher and lower order thinking skill requirements marked to evaluate student learning problems. All tests questions are indicated as HO, for higher order thinking skills required, or LO, for lower order thinking skills required to reach a correct answer.  This technique (a) continues the rationale of naming thinking skills when they are useful and (b) helps students learn how to study in the future.  For instance, if students miss LO questions it may mean that they are not reading the text carefully enough.  If students miss HO questions it may mean that they are memorizing and not learning how to use their knowledge.  If students miss HO questions they are directed to certain areas of the Study Guide and problems which require students to practice using their knowledge.  If students miss LO questions it is suggested they read the text assignments more carefully. [Copies are included in the dossier.]

To document teaching on this level will require departments and colleges to agree upon categories and key dimensions reflecting the scholarship of teaching. This, in turn, may require faculty and administrators to examine the roles they play in the summative evaluation of teaching. The candidate must play a very active role in monitoring his or her teaching, while colleagues must play a collaborative role. The collaborative construction of a teaching portfolio between departmental colleagues connects the summative and formative evaluation functions together in a single process, since the decisions made by the candidate and departmental colleagues in determining what is selected for inclusion and how it is structured in a portfolio (summative functions) are intended to foster the improvement of practice (formative function). These decisions will of necessity require a thoughtful discourse about teaching between the candidate, his or her peers, chairperson and dean. The intention is that the activity of building a teaching portfolio during the first six years of teaching practice encourages peer consultation and review, resulting in a profile of how the candidate’s teaching has developed over that period of time. This can itself lead to a kind of professional inquiry since, after enough candidates have undergone the process, it is likely that a clearer set of standards for what constitutes effective teaching may emerge.

The following is an example of what a Teaching Portfolio might consist of, together with a faculty member's rationale for how it was constructed.

Portfolio Outline 5 Professor Margaret Ackman College of Pharmacy, Dalhousie University A. Statement of Teaching Responsibilities     1. Courses Taught     2. Student Advising         a. individual students         b. student committees     3. Practica: Organized and Supervised B. Statement of My Teaching Philosophy and Goals C. Efforts to Improve Teaching     1. Formal Courses in Education     2. Conferences Attended     3. Workshops Attended     4. Participation in Peer Consultation D. Redevelopment of Existing Courses     1. Addition of tutorials, role-playing, case studies, etc.     2. Incorporation of Writing Skills     3. Incorporation of Oral Presentation Skills     4. Appendix of Representative Course Syllabi and Assignments E. Information from Students     1. Summary of Student Ratings     2. Comments from Student Committees Regarding Advising G. Service to Teaching     1. Evaluating Term Papers, Chair     2. Faculty Evaluation, Co-Chair     3. Curriculum Committee, Member     4. Lectures to Special Interest Groups of the Public H. Information from Colleagues I. Information from Other Sources     1. Guest Lectures to Other Faculties     2. Continuing Education Lectures for Peers     3. Lectures to Special Interest Groups of the Public J. Future Teaching Goals Professor Ackman provided the following explanation of her plan to prepare her teaching dossier. I began my teaching dossier with a statement of my teaching responsibilities. This section would discuss not only the courses which I teach, but also the student advising which I do for both individual students and student committees. Our students are required to complete a variety of practicums during their undergraduate education. Therefore, my organization and supervision of these practicums would also be included. The second section would be my Statement of Teaching Philosophy and Goals. The rest of the dossier will complement and enhance this statement and provide a framework of how I intend to achieve my goals. This is only my third year of teaching. As a result, I am very conscious of my lack of experience and formal education in teaching. I have spent a great deal of time and effort trying to improve my teaching and learn more about teaching and learning.  I feel that this ongoing effort is very important and that my teaching has improved because of it.  Therefore, this would be presented as one of the initial sections of my dossier—Efforts to Improve Teaching.  I would follow this section with the section on Redevelopment of Existing Courses.  For me, this is a very logical progression.  When I began teaching, I was presented with an existing course which I did not feel contributed to the goals and objectives of the College.  However, it was only after attending a number of workshops and conferences that I was able to begin redesigning the course. The section, Information from Students, would follow.  This would indicate improvement in overall ratings since my initial attempts at teaching.  This would relate to Efforts to Improve Teaching and also to Redeveloping of Existing Courses.  A section on Student Achievement would be next.  This would indicate that my students do reasonably well on both regional and national levels in comparison to their peers. Service to Teaching would be the next section.  This also relates to the section on Efforts to Improve Teaching.  As I have become better informed concerning teaching, I have become more involved in the administrative aspects of teaching.  This is reflected by the committees which I currently chair or of which I am an active member.  Information from Colleagues would fit best at this point in my dossier.  My colleagues would be in a position to comment not only on my teaching abilities, but also on my Service to Teaching.  Information From Other Sources is a miscellaneous category.  It is important because it demonstrates my degree of involvement with the profession and my ability to teach at a variety of levels.  However, for evaluative purpose, it should not be given the same weight as student evaluations.  Therefore, it would be included near the end. The final section would be my Future Teaching Goals.  This section may actually be one of the most important in my dossier because I am relatively new to teaching.  I considered placing this section after My Teaching Philosophy and Goals.  However, since it would relate to all of the other material in the dossier, it is only reasonable to conclude with this information.  In order to ensure that this section would be given the appropriate consideration, I would refer to it in My Teaching Philosophy and Goals and in other pertinent sections.

Criteria of Inclusiveness

Three criteria for inclusiveness have been suggested for the teaching portfolio: reflectiveness, usefulness and representativeness, as shown in Figure 6 below.

The simplest and most obvious of these criteria is usefulness: are the data chosen for inclusion—and the way they have been structured and presented—useful for the purposes of evaluation? The framework presented in Figure 1, Chapter 1, suggests three areas of data useful in documenting the position and how effectively the candidate filled it: the job, how it was described and how the candidate’s filling the job subsequently shaped it,  documentation of the process whereby the job was filled by the candidate, and teaching data. These kinds of data are useful in providing an overall profile of the relative weights to be attributed to the candidate’s research and teaching data.

The second criteria for inclusiveness in a portfolio is reflectiveness: to what degree are artifacts, such as course materials and other abstract data such as student evaluation scores, accompanied by reflective comments by the candidate which ground them in a meaningful context? Reflectiveness helps those evaluating the portfolio to understand the activities and strategies undertaken by the candidate to improve teaching practice and the data presented to demonstrate that improvement. Reflectiveness on the process of improvement will include statements about what experience has taught the candidate about teaching, what he or she has worked on changing, what experimental actions were taken to effect change, and what change was accomplished, either intentionally or unintentionally. The following two examples contrast a more descriptive personal statement about teaching (Example 1) with a more reflective statement (Example 2).

EXAMPLE 1 6 Dr. Thomas H. MacRae Department of Biology, Dalhousie University Teaching Advanced Students Eleven honours students, eight of whom were NSERC Undergraduate Scholars, have studied in my laboratory (Appendix l) and I have served on several other honours committees.  Mr. John Apple, who spent two years with me, is a co-author on two papers.  Mr. Bill Brown is co-author on two abstracts and three papers now submitted.  Ms. Ann Christopher will be co-author on one paper.  One other student was co-author of an abstract and another, an NSERC Undergraduate Scholar from McMaster University who worked in my laboratory, is now doing her Ph.D. under my direction.  I have supervised several undergraduate students in research and/or writing-based special topics courses.  I normally support only one undergraduate application per year for an NSERC Summer Scholarship, although in 1988 I supported three students all of whom received awards.  I am very selective of the undergraduates I accept for honours, usually taking only those who can generate sufficient money to allow them to work during the summer.  This approach is necessary as the time required to master technical aspects of the projects in my laboratory is usually great.  The student thus has the time to undertake a meaningful project, is subject to closer supervision during the initial phase of his/her work since I do not lecture in the summer, and develops a truer appreciation of research as the distraction of course work is lessened. As my first two years at Dalhousie were on a term appointment, I was not able to supervise graduate students until my third year.  Since 1982, two MSc students and one Ph.D student have graduated from my laboratory, while two other graduate students are in progress (Appendix II).  Two of the students have held NSERC Graduate Scholarships.

EXAMPLE 2 7 Dr. Graham J. Fishburne Department of Elementary Education, University of Alberta Commitment to the Improvement of University Teaching At the outset of this report, I attempted to articulate my own philosophy of teaching.  I believe that all University Professors are working toward the common goal of educating students in the most effective manner possible.  We all possess skills and knowledge that are needed to achieve this common goal.  By sharing our skills and knowledge, we will learn from each other and improve the effectiveness of the instruction that we offer to our students. Since entering into higher education teaching I have spent a great deal of my time in shared and cooperative ventures aimed at improving the quality of University teaching.  These activities have been above and beyond my own areas of research and the academic discipline for which I was hired....It would have been easy for me to remain within the domain of the areas of research enquiry that I have established and within the academic discipline for which I was hired; however, because of my commitment to University teaching, I have endeavored throughout my career to share and assist in any way possible my knowledge of teaching and learning with colleagues who are also involved in the world if instruction: “As a teaching colleague, Dr. Fishburne...(was not only open to new approaches to instruction but also) contributed many creative and innovative ideas...He is an extraordinarily competent, effective teacher.  He has been actively involved in departmental affairs, and takes seriously his obligation to be of service to the University, the profession, and the public.”  Professor of Mathematics Education, 1985. Since joining the Faculty of Education, I have been involved in the development of integrated courses.  This has led to team teaching experiences where several professors team teach and share teaching experiences.  Over the years, I have been involved as a team leader in such courses.  I have found this experience to be extremely valuable since we must plan and share our various teaching methods as we strive for the most effective learning environment.  We engage in a great deal of “reflective practice” and learn much from each other.  The teams regularly change which has allowed me the opportunity to play a leading role in peer teaching with many of my colleagues.   Due to the shared learning commitment the Faculty enjoys with the Alberts Teachers’ Association, we regularly have seconded staff from various school boards involved in our undergraduate program.  My leadership role has been that of “mentor” to these people and to new members of staff.  Over the years I have team taught with approximately 30-35 University teachers.  Feedback on these mentoring experiences has been very positive as the following peer comments attest: “Dr. Fishburne’s teaching is exemplary.  Over the years we taught on the same team, I had the unique opportunity to observe and evaluate his teaching as well as his contributions in collaborative program development.  His teaching is 'dynamic’ He is enthusiastic about his work and demanding in his expectations of students and colleagues.  I consider him to be a model of professionalism-one to whom students and colleagues look for strength in the teaching role and for guidance in their own development.  His general support and encouragement is always a positive influence on both students and colleagues.”  Professor of Mathematics Education, 1987. Over the past few years I have been consistently involved with the University Committee for the Improvement of Teaching and Learning (CITL).  I have made regular presentations to University colleagues on issues aimed at supporting campus who frequently call for advice, research findings, resource material, and general information on effective teaching. I am also a member of CITL’s Peer Consultation Program.  This involves working on CITL’s Peer Consultation Team with University Professors from Faculties other than my own who have requested some form of help or assistance with their University teaching. I am constantly involved in Peer Teaching Evaluation.  I not only work with colleagues in my own Faculty but I am requested by staff in other Faculties to conduct peer teaching evaluations.  I not only offer a perspective on the colleague’s teaching but frequently work with the colleague to help improve their teaching.  I have performed this voluntary activity for the past 10 years and the feedback from colleagues has suggested that it has been a worthwhile endeavour for improving University teaching.  A typical response to my work in this area appears below: “I have often recommended to professors needing teaching evaluations that they seek out Dr. Fishburne because he is not only quick to render expert judgement but he continues to help professors improve their teaching skills.  He shares information and material willingly with colleagues and provides help whenever he can...”  Chair and Professor, Department of Elementary Education.

Some examples of questions a candidate can provide reflective responses to include:

Discipline and Classroom Approach

• Within your discipline, which area do you regard as your strongest?  Your weakest?
• What is your greatest asset as a classroom teacher?  Your greatest shortcoming?
• Which teaching approach works best for your discipline?  Why?
• Do you change methods to meet new classroom situations? Can you give a recent example?
• What is your primary goal with respect to your students?
• How would you describe the atmosphere in your classroom?  Are you satisfied with it?

Knowledge of Subject Matter

• In what ways have you tried to stay current in the subjects you teach?
• How would you judge your knowledge in the subjects you teach?
• Do you think your colleagues agree with that judgement?
• What have you done or could you do to broaden and deepen your knowledge of te subject matter?

Questions About Teaching

• What is the one thing that you would most like to change about your teaching?
• What have you done about changing it?
• What would you most like your students to remember about you as a teacher ten years from now?
• Overall, how effective do you think you are as a teacher? Would your colleagues agree? Your students?
• Which courses do you teach most effectively?
• In what way has your teaching changed in the last five years? Ten years? Are these changes for the better?  Why or why not?

The Stanford Teacher Assessment Project  (King, 1990) suggests four core tasks of teaching: a) planning and preparation, b) performance, c) evaluating student learning and providing them feedback on their learning, and d) teaching development activities. Together, these constitute the third criteria of inclusiveness of the portfolio: does the information provided adequately represent the range of activities of a candidate’s teaching?

According to Edgerton, Hutchings and Quinlan (1991) course planning and preparation can include such artifacts as course syllabi, a series of assignments given to students, readings and teaching plans. For the reviewer to gain a sense of development in practice, the selection of these planning and preparation artifacts should be governed by how they have changed over time. Thus, all artifacts may not necessarily be exemplary cases but serve to show how the candidate has developed in various areas. The reflective commentary accompanying these artifacts will help the reviewers interpret their relative meaning and value.

Classroom performance can be represented by peer observation reports, videotapes of selected classes taught by the candidate, or student evaluation results. If videotapes are chosen for inclusion, the reflective commentary accompanying them must provide a rationale for why and how those teaching episodes were selected for inclusion. Peer observation of classroom performance is a practice which must be carried out with great care if it is to be useful and just. Peer observers must be qualified in terms of the subject matter being taught and in terms of what to observe and report on. In most cases peer observers must be trained in carrying out their responsibilities (Sell & Chism, 1988). The validity of peer observational reports will be increased if they also serve a formative evaluation function.8 Observational reports must also be accompanied by a reflective commentary by the candidate with reactions to what was written by observers. Peer observation will be discussed in more detail in Chapter IV.

Work samples documenting the candidate’s evaluation of student learning and providing them with feedback on their progress include selecting a student paper as evidence of a high standard. Reflective comments accompanying the paper could address why that standard was appropriate, together with an explication of what was done to help that student achieve such a high standard. Another work sample in this area might be a copy of a student’s exam which reflects misconceptions students often bring to the course, accompanied by comments on what strategies were developed and used to deal with those misconceptions and a rationale for evaluating the effectiveness of those strategies.

The publish or perish tradition of publishing within one's discipline the products of one's own research has discouraged faculty from publishing about their teaching experiences. Yet through the experience of teaching—the trials and errors—much is learned about effective teaching practice. Most academic fields have some journal devoted to publishing faculty reports on their teaching experiences. Some are more empirically rigorous than others, yet keeping up with a professional field's knowledge base of teaching, or contributing to that body of knowledge is a legitimate part of a faculty member’s teaching responsibilities and can be documented in a portfolio. At the very least, the candidate could include a paper heard at a professional meeting or read in the professional press, accompanied by a reflective essay on how it influenced changes in course design or teaching practice.

General guidelines for constructing Teaching Portfolios suggested by Edgerton, et. al., include:

• keep size “lean and clean” through applying the following principle to all data considered for inclusion, “What will this entry add to the description of knowledge, skills and perspective of the candidate?”
• Orient the portfolio away from raw data & move toward judgments of data’s meaning
• Maintain an attitude of flexibility & experimentation while lessons about function and process are being learned.
• think about portfolios and the entire evaluation process as connected to the improvement of practice and toward “developing a more professional discourse about teaching”
• work involved in developing portfolio must be considered an integral part of faculty’s responsibility, both for the candidate developing it and for colleagues evaluating it 9

Through the prudent selection of work samples, a teaching portfolio should be able to clearly reflect a candidate’s teaching and teaching development, and do so in a way which does justice to the range of responsibilities and activities engaged in by the candidate without resulting in a cumbersome, redundant and lengthy document for others to wade through. The key in preserving representativeness while restricting bulk is in the establishment of a clearly articulated set of criteria and categories that should be negotiated between the candidate and department at the point of initial employment. Once these have been established and the data categories selected, there are fundamental principles that have been developed through research and experimentation which should govern the collection and form of presentation of that data. We turn now to a detailed discussion of data collection and its representation.