OIRAA

Measures and Criteria Instructions

This section describes the measures that will be used to determine the extent to which a learning outcome is achieved, and defines the level of performance that must be met in order for the learning outcome to be considered achieved.

This section should include:

1. Measures
   
   Measures are used to evaluate student learning, and can be direct or indirect, quantitative, or qualitative.
   Include a description of each measure that will be used to assess achievement of a learning outcome.
   
   Examples of direct and indirect measures:

   Direct Measures	Indirect Measures
   Specific course assignments	Surveys (student satisfaction, employer, alumni, exit, etc.)
   Oral presentations	Interviews
   Embedded test items	Focus groups
   Capstone projects	Case Studies
   Portfolios
   Pre/Post testing
   Research projects or papers
   Manuscript submissions
   Comprehensive exams
   Thesis/dissertation
   Licensure/Certification exams
   National/standardized exams

   
   Avoid the temptation…course grades are poor measures of learning outcomes. Course grades do not reflect the students’ strengths and weaknesses in specific areas.  For instance, all students may have received an A in a course, but none may be able to write a statistical null hypothesis.  Furthermore, many factors contribute to course grades that are unrelated to student learning outcomes, such as class participation, attendance, and various instructor-specific grading policies (e.g. grading on the curve, different cut-off points).
   
   Guidelines:
   1. Each program must use multiple measures, one of which must be a direct measure.
   2. Use at least 1 measure per learning outcome.
   3. Multiple measures are desirable for triangulation of results.
   4. Quality of assessment measures is more important than using numerous measures that are not very meaningful.
   5. To inform improvement efforts, select measures that will identify relative strengths and weaknesses among students’ (aggregate) achievement of the learning outcome. For example, by using oral presentations as a measure of students’ communication skills, faculty may learn that collectively, students’ skills are weaker in the areas of delivery and organization, and stronger in content and adaptation to audience. It would be much more difficult to identifying such strengths and weaknesses when using classroom discussion as a measure.
   6. Consider using rubrics to score subjective assessments. Rubrics provide those doing the assessment with detailed descriptions of what is being learned and what is not, students’ collective strengths and their weaknesses.
   7. A given measure may be able to assess multiple learning outcomes. For instance, a thesis defense could assess students’ ability to demonstrate an understanding of content knowledge and to demonstrate oral communication skills. If using a single measure (e.g. thesis defense) to assess multiple learning outcomes, it is best to use a rubric that will allow these knowledge areas/skills to be evaluated independently.


2. Criteria
   
   Acceptable levels of performance need to be established for aggregate performance for each measure (not for individual student performance, but for students as a group). Examples: 80% pass with a score of 8 or higher on a 10-point rubric, 85% satisfied or very satisfied.
   
   Guidelines:
   1. Programs must have an assessment criterion for each assessment measure.
   2. Set the criteria so that it is ambitious, but attainable. It is okay if all criteria are not met; the point of assessment is to grow and improve the program.
      
      Examples:
      
      Criterion for a national exam:
      The average thermodynamics score of all of our students who take the National Fundamentals of Engineering (FE) examination, administered twice every year, will equal or exceed the national average for the thermodynamics section.
      
      Criterion for an employer survey:
      Based on an employer survey, which is administered once every three years, at least 75% of the employers will be satisfied with the ethical conduct and the knowledge of ethical standards of our students.
      
      Criterion for a course evaluation:
      At least 90% of our students who complete the GEOG 432 course evaluation will report that the course was “beneficial” or “very beneficial” in enhancing understanding of self and others and the ability to work with others.
      
      Criterion for a comprehensive exam:
      At least 90% of students should earn a 3 or better (on a 5-point scale) on each ofthe following dimensions of the comprehensive exam: basic knowledge of general linguistics; basic knowledge of phonology; basic knowledge of syntax; ability to apply knowledge to a given problem.
      
      
      Rubrics
      
      Rubrics are helpful in assessing qualitative student work. A rubric is a guide that describes the criteria that will be used to score or grade an assignment. A rubric identifies the traits that are important and describes the levels of performance (e.g., unacceptable to excellent) within each of the traits.
      
      Rubrics can:
      1. assist faculty in determining which (student) skills/knowledge areas are well-developed and which skills/knowledge areas require improvement. That is, rubrics help communicate students’ strengths and weaknesses.
      2. reduce bias and improve consistency in scoring.
      3. clarify for students the expectations for an assignment.
      
      
      Examples:
      
      Rubistar:
      Rubistar is a free tool intended to help faculty design rubrics. Rubistar allows users to create new rubrics based on templates. After users select a general content area (e.g., research, writing, oral projects) and choose rating dimensions, Rubistar generates descriptions for levels of performance. Users can add or modify the rubric text as desired. Rubrics can be downloaded, saved, and printed. http://rubistar.4teachers.org