What's Missing in Online Course Quality? Tuesday, April 14 7:45 PM

Attending ITC eLearning 2009 Conference earlier this week, I attended multiple sessions on course quality; the sessions were offered by a diverse group using different strategies and evaluation tools: a Quality Matters lead pre-conference workshop; Lake Superior Community College's QM-based in house rubric; Yavapia College's experience with online standards, and Burlington Community College's use of course development, grade distribution and student course evaluations. One thing was missing through all of the presentations, but I think that may be due to a gap in the literature or education sector rather than it being a failing of any one institution.

There are many variations of course development evaluation: an evaluation template that assesses the quality of the development of online courses.  These evaluations address the question, "What features exist in the course content at the outset of the course?" Quality Matters, Illinois' Online QOCI, Cal State Chico's ROI, Michigan Virtual Univeristy (MIVU) Standards and other rubrics like them consider whether the syllabus is readily available, the grading policy is clearly stated, or the learning activities are aligned to the course objectives.  Those are all things that can be evaluated in the absence of learners or faculty. 

There are also numerous variations of the end of course student evaluations of course materials: you know - the institutionalized, (allegedly) more systematically researched versions of RateMyProfessor.  These go back decades and originate from pen and paper scantron versions issued to on campus classes: a college's version of the customer feedback form.  Unfortunately, these are summative evaluations rather than formative ones; they are always collected at the end of a course, never in the middle.  Even if they are collected in the middle, the results aren't typically available immediately (at least in my experience).  And, are they really ever included in or used to drive continuous improvement?  Both of those evaluations have meaning and utility, but that meaning and utility has a specific, narrow scope in my opinion.

Course Development evaluation rubrics and end of course student evaluations will rarely, if ever in my opinion, provide insight regarding the quality of online teaching that takes place in a given course.  Course development is a different skill than teaching an online course; if I don't do a good job teaching a course, I can easily negate the value of the very proficiently developed course materials.  Good course design only carries you so far.  Similarly, good online teaching can take place, although with a tremendous degree of difficulty and time to be invested during the course of the semester, with little to no course development.  Further, student opinions of course quality may be based on wildly different criteria than what an experienced and recognized, 20 year master teacher might use to evaluate a course.

What's missing in online course quality? Every institution with which I've ever been associated - as a student or employee - has used in-class observations of teaching methods to evaluate the quality of teaching and learning experiences offered to students by faculty.  Everyone's familiar with evaluation day; that's the day the faculty at the front of the room (whether us or someone teaching us) has all of their ducks in a row when the department chair or Dean quietly slides into a seat at the back of the room to see what's going on.  The evaluation may occur every semester for adjunct faculty or once a year or every other year for full time faculty.  However, if - as I'm assuming from my own experience - every institution uses this evaluation method for on campus classes, why is there no mention of an analogous, empirically based in-class observation and evaluation for faculty teaching online?  I know the accrediting bodies around the country have a strong preference for online courses having the same rigor and requirements as the on campus courses.  Is that really happening though if even our evaluation of faculty teaching online classes isn't as rigorous as the evaluation process for faculty teaching on campus? Are you familiar with any empirically validated in-class observation and evaluation rubrics designed specifically to evaluate the quality of online teaching?  I'm still investigating ONE (Quality Standards Index) that was suggested by new friends from Quality Matters, and I'd definitely appreciate your help in identifying others.