A&A: Assessment - Seminar Assessment

1. Why Develop an Assessment Strategy? Where Do I Begin?

2. What is Authentic Assessment?

3. What Do Students Want?

4. Where Can I Find More Information?

5. What Does a DHC Seminar Assessment Strategy Look Like?

6. What Are Rubrics? How Do I Use Them?

7. Where Can I Find Sample Rubrics?

8. What Is Alternative Assessment?

 

1.Why Develop an Assessment Strategy? Where Do I Begin?

The most difficult yet important aspect of assessment and grading is deciding what you want your students to accomplish by the end of the quarter. Before students can begin working they need to know what they are responsible for learning. When creating objectives for your course keep in mind that objectives usually begin with verbs. As you create assignments, use the list of verbs in Bloom's Taxonomy to help direct you to the verbs that match the level of thinking you are looking to activate. Assignments and tests should work together to help students master the course objectives. For DHC seminars where some of the content is the "process" it may take some extra creativity to plan a comprehensive assessment strategy. A comprehensive assessment strategy will ensure both you and your students end the quarter with a feeling of satisfaction rather than dissatisfaction.

Before the quarter begins decide how you will assess your seminar students. Keep in mind the differences between formative and summative assessment techniques. Giving a pre-test asking students to define terms related to the topic or problem they will be solving is an example of a formative test, it tells you and the student what they need to learn and what they already know. Formative tests are not usually graded. Summative tests are used to check how well students have mastered the objectives and are usually graded.

 

 Formative Assessment  Summative Assessment
 Not graded/not judged  Graded/Evaluated
 Examples: Pre-test, Muddiest Point, Minute Paper  Examples: Exam, Research Paper, Oral Report
 Tells students/instructors how much they have learned and what they need to know.  Tells students/instructors how well they have mastered the objectives.

 

After deciding the course objectives, each instructor must determine the criteria that will be used to evaluate each area. If an objective is to enhance research skills, take time to think about the research skills an excellent or "A" level researcher should have, what a poor researcher looks like and how will you be able to tell the difference. Then communicate the differences to your students. Your criteria are important to your students because criteria tell them what they need to accomplish during the quarter and level of performance that is expected. Developing a set of criteria for evaluating student behavior or performance is called using a rubric. Sometimes it helps to use a sample rubric to help you get started, then modify it to fit your specific needs. This sample outlines criteria for research skills.

A rubric will be helpful for evaluating student work or performance during seminars because the type of work done in seminars can also be referred to as performance-based learning or authentic learning. Projects students complete are often large, open-ended and require many different skills.

A school district has developed a site that describes step by step how to transform objectives into rubrics. This site uses practical charts and diagrams to guide you through the process.

 

2.What is Authentic Assessment?

The term authentic assessment or authentic learning relates to the DHC because each seminar seeks to model a real world experience. In authentic learning classrooms, activities are centered around real-life activities and assessment techniques are closely related to what the students are doing in class. Authentic assessment is characterized by its focus on the process as well as the product. This is also consistent with the goals of DHC seminars because each seminar instructor is interested in students learning the process of solving open-ended problems as well as solutions the students produce.

A site that provides an excellent description of authentic assessment was put together by Ph.D. students from the University of Missouri - Columbia as part of a graduate education course. Not only is the information valuable, but the instructor's syllabus is also an example an authentic learning course.

Do not be discouraged by the fact that most of the assessment research has been done on the K-12 level, much of it is useful and applicable to higher education. It seems K-12 districts and institutions spend a great deal of time and energy over the past few years producing information for training teachers to accurately assess their students.

 

3. What Do Students Want?

Students want to know where they stand in relation to the course objectives. They want to know if their performance is meeting, exceeding or falling below your expectations of them. Developing a well-rounded assessment plan before the course begins and communicating it is one way to address what the students want. The Southern Illinois site for "What students want in a teaching professor" and our own program evaluations demonstrate the most important aspects of course assessment plans are clarity, fairness and feedback. Students want to meet instructors' expectations and are most motivated when they know where they stand in relation to those expectations. Communicate your objectives and expectations to your students early and give them feedback or opportunities for formative assessment often so they know how close or far they are from mastering the objectives of the course.

 

4. Where Can I Find More Information?

The voluminous information on the Internet on Collaborative/Cooperative Learning is matched by the information on Assessment. Just as with the collaborative/cooperative learning information, understanding the terms is an important first step. The best Web site for definitions and most all of the other information you need on assessment is at the University of Southern Illinois at Edwardsville's assessment site.

This handbook also provides an annotated list of other Assessment sites on the Web Resources Page, but most of everything you need to know is at the Southern Illinois site.

The most mentioned book at every site is Classroom Assessment Techniques by Angelo and Cross, this book is available at the DHC library. This text is popular because it is well organized and written so that each assessment technique is easy to implement. However, many ideas and examples from Angelo and Cross's book have been reproduced on severalWeb sites, so you never have to leave your keyboard to access the information.

If you still need more information, the National Institute for Science Education offers advice for doing grades based on current theory on their Cooperative Learning Web site.

Collaborative Learning in Management Education: Issues, benefits, problems and solutions: A literature review by David Kimber of the Royal Melbourne Institute of Technology University. Scroll down to the section on "Matching Assessment with the Teaching Methodology" for more of his comments on successful methods of assessment.This page offers the pros and cons of different assessment techniques that can be used for groups. This page may help you decide on the best assessment technique for your seminar.

 

5. What Does a DHC Seminar Assessment Strategy Look Like?

First decide the objectives of your seminar. Past DHC faculty have developed the following list of possible areas for seminar objectives:

  • Information Gathering or Research Skills
  • Class Participation
  • Collaborative Skills
  • Individual Attendance
  • Oral Presentations
  • Individual Products - Reports or Written Work, E-mail, Journals etc.
  • Group Products - Reports, Web Pages, Posters etc.
  • Critical Thinking - Problem Solving Abilities, Synthesis Skills, Evaluation Skills etc.

 

Once you have decided what will be evaluated you will need to decide how you will evaluate each of the objectives. Again past DHC faculty developed the following list of questions to think about as you develop your assessment strategy:

  • Will you use Formative and/or Summative methods? When?
  • What will be the format?
  • Will it be quantitative?
  • Will it be qualitative?
  • Do you want to have student participation in determining evaluation criteria/methods?
  • Who evaluates whom and how (Instructor, Peer, and Self Evaluation)?
  • What will be the relative weights of Instructor, Peer and Self-Evaluations?
  • How will the final grade be assigned?
  • How will you deal with discrepancies between Instructor, Peer, Self evaluations?

 

Sample DHC Seminar Syllabi

6. What Are Rubrics? How Do I Use Them?

Rubrics
Definition: scoring guides used to define criteria which will judge student performance. They provide for increased consistency in the rating of performances, products and understandings as well as giving students an established set of expectations about what will be assessed as well as the standards that need to be met. Rubrics also protect students against evaluator's biases.

For another introduction see the brief overview of rubrics created by MiningCo.com.

There are several types of rubrics:

Analytic Trait Scoring
Judges one performance/assignment several different times.
Goal: to identify an anchor that will serve as a guide for each aspect being assessed.
Example: Judging an essay for: grammar and punctuation, author's attention to audience, definition of purpose and focus on the topic. Or judging teams for: group dynamics and group work habits.
 
The Analytical (Diedrich) Scale:
An analytical scale, or checklist, is commonly used to determine whether the student provided specific information. It is most appropriately used to assess answers to short essay questions.
 
Holistic scale:
This scale is used to make quick determinations. A truly holistic scale is typically binary, e.g., "acceptable" vs. "unacceptable." Because of this, holistic scales are particularly well suited to screening students during interviews.
 
Primary trait scale:
This scale is used as a diagnostic tool to identify concepts students have mastered or are still having difficulty with. Like holistic scales, primary trait scales provide an effective way to evaluate a large number of assignments very quickly.

The above information is from: " Assessing Authentic Learning " by Glenn Brown & Michelle Craig.

 

How Do I Use Rubrics?

After you have decided on your course objectives and what you are going to assess you can select the rubrics that best fit your needs. Rubrics can be used in two ways. They can be presented to students when the assignment is introduced, as a set of performance standards or as scales used only by the instructor for grading assignments. When they are used by instructors to grade assignments, instructors assign points to each level of criteria then add up the point totals to calculate a letter grade. English departments have been using this method for many years because it ensures more subjective assignments, such as essays, are judged objectively and ensures the same standard is applied to all students.

If students groups are evaluating themselves or each other, a set of rubrics can improve the effectiveness of the evaluations because it communicates to the students the characteristics they should be looking for in themselves and each other.

If criteria or specific characteristics are not listed in the rubric, break downs in communication can occur. For example, if each Friday you plan to hand out peer evaluation forms to each group member to complete. The form might ask, "Is the work load distributed evenly among group members?" and "Rank your self and each other on a likert scale from 1 to 5, 1= Poor and 5= Excellent." One member who had almost no work to do this week and is very happy about ranks himself and his teammates "5". The member who doesn't trust his lazy team mates, and is doing all the work also ranks himself and teammates "5". You get the forms back and as you are reviewing them that evening and think to yourself," wow isn't that team working effectively, each member seems to be assuming an equal amount of the workload each week." The result? Each student is given an "A" for the week.

There has been a break down in the communication somewhere. The group did not assume equal responsibility. It was the worst case scenario, one member was doing all the work and the others were doing nothing, exactly the way you did not want the teams to perform. The behaviors the team exhibited should have earned the team an "F" for the week not an "A".

If the groups had been given a set of criteria for each of the areas they were asked to evaluate and sample behaviors were discussed, the peer and self-evaluations would have been more accurate.

To avoid misunderstandings when using a rubric that asks students to rate each other form 1-5 based on a series of questions try doing the following Activity with them first. This process can be even more refined if you use "fictional" student scenarios and have the team evaluate the "fictional" team using the criteria. Or have students try to evaluate the "fictional team without the criteria. I can guarantee there will be heated discussions within the teams when it come to ranking the "fictional" students based on their observations of the scenarios. A couple of the most valuable aspects of their discussions will be the articulation of assumptions about student behavior and the difficulty agreeing on what behaviors deserve a "5".

As you pull the groups together after the activity you can point out the importance of agreeing upon a set of criteria. You may also want to discuss the assumptions students made about the "fictional" characters based on the scenarios. People often assume the worst about people they don't know. If a team member is always late, a stranger might assume that person does not care about the group's work. However, if the team spends some time getting to know one another they would know that the late person has a class on the other side of campus before the team meetings. If this was the case teammates might be less likely to give that person a poor rating when asked about their commitment to the team.

7. Where Can I Find Sample Rubrics?

Critical Thinking

Holistic Critical Thinking Scoring Rubric - This list provides a starting point for identifying characteristics associated with critical thinking.

Helping Students Assess Their Thinking - A list of characteristics that will help you build assessment rubrics from the Critical Thinking Center at Sonoma State.

Reports and Research Papers

Eighth & Tenth Grade Mechanics Scale -This is useful for any type of written work, but mainly looks at grammar.

Overall Content Quality of the Explanation - This can be used for grading any type of essay question or written/oral explanation of a problem or question.

Evaluating the Writing Process -This site gives an example of a rubric used for evaluating the writing process. A students' writing grade is based on how often they practice the steps in the writing process, not just a sum of scores students receive for the final products produced.

Project Definition - This short rubric might be useful for evaluating group action plans.

Short Answer Task - 4 Point Scale - useful for assigning points to definitions.

Problem Solving

Problem Solving Task - Holistic 5-Point Scale - This will be very useful for students to use or instructors as they look at the solutions to their problems or look at what makes a good solution.

Article on using rubrics for mathematical problem solving with sample rubrics - This is a rubric developed for assessing mathematical problem solving, but many elements are applicable to any form of problem solving. It is in the middle of the article, so you will need to scroll down to find it. You will find a table, called "Table 2, An Instrument to Asses Individual Problem-solving Processes," this table can provide you with examples of criteria used to evaluate problem-solving processes and an example of how to relate criteria to a numerical value to help quantify student performance. You may notice when you look at this table that there is very little emphasis on the correct answer, which may be a useful example because many seminar problems have no right or wrong answer.

Chemical Problem Solving Methods Analysis- This site give both a feedback chart and a detailed description of problem solving criteria. Some are chemistry specific and other examples such as a critical evaluation of a problem solving plan are not chemistry specific.

 

Web Sites on How to Create Rubrics

Student Generated Rubrics - This article from Interactive Classroom gives ideas for involving students in a process of creating rubrics called Negotiable Contracting. It also gives samples of some very simple rubrics and takes you step by step through the process of constructing a rubric.

Questions and answers about assessment - This section will also provide very detailed information about assessment. The Problem Based Learning Program at McMasters University is well-developed and has three forms of handbooks.

From the home page address listed, go to the section on Questions and Answers about assessment and scroll down to the following sections:

  • 5.1 Why is assessment important? Because students learn what is assessed.
  • 5.8 On the role of evidence
  • 5.9 Peer assessment? How to get students to assess each other instead of patting each other on the back or putting all the others down so that they appear to be good?

On the same web page you will find discussion of group processing skills and how they teach and evaluate these skills in the section entitled:

  • 3. What about processing skills used in PBL?

 

Research Skills

Criteria for Evaluating Information Gathering Skills: This site lists a rubric developed for evaluating students research skills.

Research Skills: This is the best example of a research skills rubric, because it is part of a larger research assignment based on the Big6 research skills. If you scroll to the bottom of the page and click on return to lesson you will find information on cooperative learning. The rubric is linked to the original research assignment so students can identify skills they will be expected to demonstrate. It makes for a very effective assignment.

Collaborative Skills:

Evaluates Listening, Cooperating, Decision Making Skills: This rubric is simple and easy to use. There are other rubrics on this site if you scroll to the bottom of the page and return to the Tide pool Study main page.

Field Tested Learning Assessment Guide: This is a simple rubric with three areas for evaluation from the Molecular Science Project.

Freshman Design Project Review: These are more skill specific rubrics designed for giving students feedback in a number of areas, including analytic skills and communication skills.

Peer Evaluations - Project Contributions: This is an evaluation form rather than a rubric. It asks students to rate themselves and group members according to their contribution in areas such as managerial and communication. Before using a form like this one with a group it would be useful to complete a similar likert scale for fictitious students so there is consensus on students who might score a ten and students who score a five. This would be an excellent activity to do in small groups; ask each group to come to a consensus for each candidate. This activity can also work well as a value line, where students physically stand next to a number posted around the room after each candidate profile is read. The most interesting part of this exercise is asking students to defend their rating choices.

Peer Evaluation-Sample Form: <Add link to35_ peereval_1.html>

It important to discuss performance standards with students before using this kind of form. As a group activity have student groups develop rubrics for each of the categories.

Peer Evaluation Sample Form - This form requires students to divide a certain number of points among team members. This type of system eliminates the temptation team members may have to rate all members equally.

 

8. What is Alternative Assessment?

The following is a list of Web sites with ideas for methods of assessment that do not fall under the traditional definition of assessment. Many of these may be useful for your seminar because traditional assessment techniques like objective tests are not appropriate in the seminar context.

Weekly Reports: Weekly Reports as a tool for teaching and feedback. This article gives a detailed description of how an instructor used students weekly reports to keep track of where students were struggling in a lower division Physics class. The instructor asked students to list what they had learned that week, what still remained unclear and what questions they expected the instructor to ask on the exam. The instructor found student feedback a valuable asset for improving how he taught the class.

Peer Evaluation Strategy by Professor Al McLeod, Sociology Department, Californian State University-Fresno. This brief article outlines the instructor's method and comments on peer evaluations of chapter summaries in a sociology course. He provides insight into the dynamics behind the group work and special considerations that need to be made when embarking on this strategy.

Course Structure that Enhances Learning: By Tom Creed of Saint John's University a very brief description of the aspects setting up a course structured increase learning. One of the recommendations includes frequent formative feedback.

Concept Mapping: Concept mapping is a widely used review technique. Generally concept mapping involves students creating "webs" or "maps" using circles and lines to connect ideas. These maps help many students through the process of relating ideas that may seem disconnected and provides a visual picture of the relationships. The process of sharing Maps is useful because it requires students to articulate their thought processes and exposes them to different ways of looking at the same information. Penn State also has a detailed explanation of how to use and score Concept Maps created for faculty.

Evaluation of a Project-Based Class: This is a brief description of how the evaluation practices a faculty handbook from a high school course co-sponsored by Los Alamos National Laboratory. These courses appear similar to DHC seminars because they combine project evaluation and team evaluation aspects in student assessment. The page also lists several examples of rubrics.

 

References:

Mc Neal, A. and D'Avanzo, C. 1997. Student Active Science: Models of Innovation in College Science Teachings. Harcourt Brace and Company.



Main | Seminars | Teams | Thinking | Assessment | Map | Resources