A Long and Winding Road Through Community College (Voice of OC)
By JENNY CAIN

IVC Foundation event

     So far, by all accounts, the word is that the IVC Foundation event on Saturday was a big success. Several who attended told me that it was the best such event they had been to in years.
     More later. Rebel Girl attended, and she'll have a fuller report.

Systematic rational failures and SLOs, part III: why don’t we do something?

     Outcomes assessment is an odd business. It is not to the credit of higher education that we have tolerated this external assault on our work. Its origins are suspect, its justifications abjure the science we would ordinarily require, it demands enormous efforts for very little payoff, it renounces wisdom, it requires yielding to misunderstandings, and it displaces and distracts us from more urgent tasks, like the teaching and learning it would allegedly help.

—John W. Powell

     The core alleged insight of the “outcomes” philosophy goes like this. Traditionally, our schools and their functioning have been viewed in relation to “inputs”: money, equipment, training, increased instruction hours, and so on. We invest all of these things in our educational system, and then we expect it to produce educated students.
     That’s all wrong, say the proponents of “outcomes.” We need to focus instead on the desired output—for instance, the kinds of skills and knowledge we want our kids to have at the end of their schooling—and then work backwards from there. It’s just logical! How do we get from where we are, point A, to where we want to end up, point B? That’s the $64,000 question. So our emphasis should be first on “outcomes,” which should be measurable, and second on how to achieve them.
     (For a sympathetic overview of the "outcomes" philosophy, see here. For a very different overview that emphasizes the politics, see here.)

     WAIT A MINUTE. Observe, however, that it is not obvious that this approach is workable. (In fact, states and nations who have embraced OBE have tended to find it unworkable and have tended to abandoned it.) Can one step back from, say, my “Introduction to Philosophy” course and capture “that which I intend to teach my students” in a set of sentences of the form, “upon taking this course, a student will be able to...”? Among thoughtful and intelligent people, the notion will arise that this task will be difficult. Perhaps impossible.
     When I step back from my Intro course and ask about the intended “takeaways” for students, I gravitate toward such things as:

• Rational confidence: I want students to begin to reject widely-held skepticism about the possibility of profitably thinking about and discussing/debating difficult (philosophical and other) issues. They need to begin to see that some ideas are more defensible than others and that a community of good thinkers can make progress reasoning about and discussing issues (rejection of the notion that any idea is as good as another). 

• Epistemological moderation: I want students to begin to settle into a middle area between skepticism (the notion that there is no point in thinking/arguing for the sake of maximally justifiable positions, for none exist) and dogmatism (the notion that the truth is had and that, therefore, it should not be questioned or challenged). 

• The particularity of our conceptions/forms of life: that our conceptions and traditions are properly understood in relation to history and that there exist and have existed a multiplicity of intellectual/moral traditions in the world of which ours is one example (please note that this is not necessarily an expression of “relativism”*). 

• Etc.

     When I consider these and other intended “takeaways” for my students, I am doubtful of the prospect of successfully reducing or capturing them in a set of “measurable learning outcomes” that can be listed on the syllabus. Happily, I am inclined to think that students’ exposure to my course and various other courses, assuming some effort at balance, will tend to produce the above “outcomes,” among others, unmeasurable (through, at any rate, SLOs) though they might be.

     HOW TO IMPOVERISH TEACHING/LEARNING. It is true, of course, that, to some extent, takeaways from my Intro course are amenable to the “measurable outcomes” approach. But why would anyone assume that that subset represents what is most important in the course? Mightn’t one worry that that subset represents what is least important and that a focus on securing that subset would dumb down the course or rid it of intellectual depth or gravity?
     And further: why should it be assumed that the ability to reduce the “takeaways” of a course (or a course of study, including a degree) to a set of discrete and “measurable” learning outcomes is necessary? That this reduction (or essence-capture) is available would, I suppose, be desirable. But that such a thing is desirable does not in any way imply that it is available or necessary.
     It is no secret that we—our society, the Academy, et al.—have long debated the question, “What is a college education for?”. But does anyone suppose that the unavailability of some received view, some firmly held consensus, means that our college students generally fail to be educated? (Well, obviously, in fact, some of them do fail. But many do not.)

* * *

     FACTOIDS. Let’s be sensible. Toward that end, here are some factoids to consider:

     1. Near as I can tell, there is no scientific evidence (i.e., studies that do the appropriate comparisons between systems that embrace OBE and systems that do not) supporting the efficacy or unique efficacy of OBE. (Please see John W. Powell’s Outcomes Assessment: Conceptual and Other Problems.) To be sure, proponents of OBE are able to produce enormous piles of studies and the like in "support" of OBE. But these tend to consist of, not evidence that OBE (compared to anything else) works, but that someone somewhere has in fact implemented OBE. Such muddled thinking should not be surprising among "experts" who have never heard of the need for replicating study results.

     2. Given the recent emergence of such tests as the PISA, we are living in an era of improved data concerning the efficacy of education approaches. It is more possible than ever to compare nations with regard to the success of their educational efforts. Guess what? The most successful countries do not embrace OBE. They embrace more traditional approaches.

     3. It is no secret that our nation’s efforts at reform (in the nation’s system of elementary/secondary education) have been dismal. In fact, it has been characterized by (1) a gravitation to meretricious shiny objects (better equipment, smaller class size, educationist trends [whole language, self-esteem]) and (2) being repeatedly hijacked or undermined by political movements and interest groups. Despite spending enormous sums of money, our efforts have plainly failed. (And, no, the data do not suggest that poverty or diversity are insurmountable problems in establishing an effective school system. See what countries like Poland and Singapore have achieved, despite poverty or diversity.)

     4. Thanks to muddled national and state politics, accreditors have been pressured to adopt OBE, that once shiny object, even at the level of higher education. As a result, the accrediting agencies, including our own ACCJC, have indeed adopted OBE, and, consequently, colleges and universities across California continually dedicate vast monies and energies to writing SLOs (for programs, courses, etc.). In general, the “SLO” mandate is poorly received, especially by faculty. Consequently, institutions confront a choice between two options: (1) trying to make a bad system work, to the extent that that is possible (and it is typically judged to be not very possible), or (2) bad faith implementation—i.e., simply giving our benighted accreditors what they want and then, to the extent possible, ignoring SLOs. In fact, the most common response to the SLOs mandate is a mix of (1) and (2). (Some will dispute this. Some believe that the Earth is flat.) If so, our system is wasting vast amounts of energy and money on an approach that is very substantially inferior to previous approaches. Nice going, America.

     This is a fiasco, a disaster. Under the circumstances, it is very odd indeed that there isn’t more push-back.

* * *

     THE STATE SENATE. Community college “Academic Senates” represent faculty with regard to academic matters and are key players in so-called “shared governance” or “collegial consultation.” The Academic Senates of California’s community colleges long ago (1970) organized into the Academic Senate for California Community Colleges (ASCCC), commonly referred to as the “State Senate.”
     As you know (see recent post), thirteen years ago, the State Senate tried to push back against the accreditors (ACCJC), challenging them to justify abandonment of old standards and embrace of the new "measurable outcomes" standards. It was a rare moment of good sense and spunk.
     That got them nowhere.
     The State Senate had its big Spring meeting last week, from Thursday until Saturday.
     Saturday was reserved for “resolutions.” One of the resolutions that (as I understand it) was voted upon yesterday was #2.04 S15: “Justification of SLO Use.” Here it is:

 

Whereas, In the last 15 years, new attempts to track the success of school systems around the world (e.g., Program International Student Assessment) have achieved impressive bodies of data useful in measuring the effectiveness of education approaches; 

Whereas, These data indicate that the more successful countries do not embrace the notion of “measurable student learning outcomes” that are central to the Accrediting Commission for Community and Junior Colleges’ (ACCJC) existing standards for evaluating and reviewing institutions and the philosophy that emphasizes that tool; and 

Whereas, It continues to be the case that research fails clearly to establish that continuous monitoring of course-level student learning outcomes (SLOs) results in measurable improvements in student success at a given institution but does engender frustration that continues to characterize community colleges’ attempts to implement the SLO approach; 

Resolved, That the Academic Senate for California Community Colleges request no later than July 1, 2015 that ACCJC justify its continued implementation of SLOs and explain why it does not opt for approaches more consistent with the approaches of successful countries in educating their students.

     I don’t yet know the outcome of the vote. I'll get back atcha when I do.

P.S.: yesterday (4/13) I contacted someone who attended the Plenary and asked her what happened to our/my resolution. She wrote back, explaining that

     I’m still trying to figure it out (in terms of bigger picture): In Area D it was accepted, someone from another Area “improved” it by adding a due date; Area D left it on the consent calendar. People from Area D & other regions came up to me in the hall to praise it.
     And then, somehow, it failed at the vote. I didn’t understand the opposition. The arguments against it seemed to be small detail stuff....

     Pretty disappointing.

*That there are multiple "traditions" does not in itself imply that they are necessarily equal or incommensurable. Such notions, of course, are matters of controversy.

Systematic rational failures and SLOs, part II: the ACCJC gets dogmatic & dictatorial all over Cal community colleges (2002)

     You're just gonna LOVE this. What follows is a series of ASCCC (aka “state senate”)* resolutions in response to the accrediting agency’s (the ACCJC's) June 2002 imposition of new standards that focus on “student learning outcomes” (SLOs).
     (See especially #02.04, below.)
     Essentially, the State Senate demanded that the accreditors explain the evidential basis for their assumption that the embrace of SLOs will improve education at the colleges.
     In response, the accreditors provided nothing. Evidently, that body ruthlessly and unapologetically imposed the new standards, despite howls of protest and requests for "dialogue."
     Nope. No dialogue. No evidence. Just the hammer coming down.
     At the end of this post, I've added a more recent assessment of the "evidence" in favor of the SLO regime.
     There's still no evidence that this sh*t works.

• Use of Current Measures Absent "Clear Showing of Inadequacy" (Fall 2002, # 02.01)

Whereas, The American Association of University Professors (AAUP) 1991 Statement on "Mandated Assessment of Educational Outcomes" notes that the justification for developing any assessment plan must be "accompanied by a clear showing that existing methods of assessing learning are inadequate for accomplishing the intended purposes of a supplementary plan"; and

Whereas, Given that the new Accrediting Commission for Community and Junior Colleges (ACCJC) Standards require that faculty develop student learning outcome measures at the course, program, certificate, and degree level, even though the ACCJC has provided no evidence that these would document the inadequacy of current methods for assessing learning;

Resolved, That the Academic Senate urge local senates to assert the right and responsibility of faculty to determine appropriate measures of student learning and achievement (such as grades, certificates, and degrees), and that absent "clear showing" of the inadequacy of current measures faculty need not develop additional outcome measures simply to satisfy the Accrediting Commission for Community and Junior Colleges (ACCJC) requirements for continuous documentation and improvement of student learning outcomes.

[The Senate published in December 2002 guidelines for the field covering these points. These guidelines are available on the Senate website.]

• Insistence on Academic and Institutional Excellence in the Self-study Process (Fall 2002, #02.02)

Whereas, The American Association of University Professors (AAUP) statement on the "Role of the Faculty in the Accrediting of Colleges and Universities" addresses the centrality of faculty in the accrediting process and contains recommended standards for institutions of higher education;

Whereas, The Accrediting Commission for Community and Junior Colleges (ACCJC) Standards adopted in June 2002 have generally retreated from a commitment to such commonly accepted standards for excellence in institutions of higher education and to the baseline resources that define a quality educational institution;

Whereas, Quality higher education institutions support the needs of students in the learning process by providing them with qualified full-time faculty who have appropriate control over the assessment of students and over the content and teaching of their courses and programs; and

Whereas, The protection of academic freedom and processes of collegial governance are critical to the sound operation of a college and are essential components of providing students the opportunity to learn and explore in educational environments that are free of coercion, encourage open inquiry, and promote the development of critical thinking and multiple perspectives;

Resolved, That the Academic Senate urge local academic senates to insist that their local accreditation self-studies continue to include attention to generally accepted standards for institutions of higher education including:

• The provision of qualified full-time faculty sufficient to conduct programs of academic excellence and to meet the learning and support needs of our students;
• Appropriate faculty control over the assessment of students And over the content and teaching of their courses and programs;
• The right of faculty to determine that grades and other current indicators of student achievement (such as degree and certificate attainment, transfer, and subsequent occupational success) are appropriate to the measurement of student learning;
• The protection of academic freedom, due process, and tenure;
• A substantive role for faculty in college and district governance and support for the processes of collegial governance; and
• The provision and allocation of sufficient resources to support high quality educational programs, student services, and libraries.

• Documentation of Cost of Implementing New Standards (Fall 2002, #02.03)

• Lack of Evidence for Restructuring Accreditation Standards (Fall 2002, #02.04)

Whereas, The Accrediting Commission for Community and Junior Colleges (ACCJC) adopted new Standards for accreditation over the objections of the Academic Senate for California Community Colleges as well as those of the American Association of University Professors [AAUP] and the Community College Council of the California Federation of Teachers among others;

Whereas, The ACCJC has not responded to repeated requests to provide the Academic Senate the background materials and research upon which it based its decision to restructure the Standards around the continuous monitoring of student learning outcomes; and

Whereas, The ACCJC has not responded to repeated requests to provide the Academic Senate with evidence or research to support the contention that such an approach in fact leads to improvements in the quality of undergraduate education or enhances student achievement;

Resolved, That the Academic Senate continue to request the background evidence and supporting research that would justify recent radical restructuring of the Accrediting Standards by the Accrediting Commission for Community and Junior Colleges (ACCJC); and

Resolved, That the Academic Senate publicize in relevant educational and public venues its concerns regarding the secrecy and lack of substantive evidence provided by ACCJC to support these costly new accreditation requirements.

[Working group formed in 2003, chaired by the Accrediting Liaison, failed to function as chaired [sic] called no meetings. Since then it has become obvious that the ACCJC has no evidence and was only complying with federal demands (also not supported by evidence) to shift to outcomes based assessments.]

• Continued Use of Current Standards and Redirection of Professional Development Resources (Fall 2002, #02.06)

Whereas, The new Accrediting Commission for Community and Junior Colleges (ACCJC) Standards do not actually take effect until Fall 2004, and colleges undergoing accreditation prior to that date will have a choice between the new and the old Standards;

Whereas, Recent reports regarding national and congressional debate over the processes and content of accreditation portend a likely period of instability and possibly even restructuring of accreditation;

Whereas, Local academic senates are experiencing enormous pressure to conform immediately to the new Standards, even if they are not yet the ones by which their college will be next accredited; and

Whereas, Faculty are being offered generous support for attending conferences on learning outcomes and the new Standards, but have had funding cuts for all other professional development programs slashed;

Resolved, That the Academic Senate urge local senates in those colleges undergoing their next accreditation visit prior to Fall 2004 to insist that the current Standards continue to be used and resist the premature imposition of the new Standards;

Resolved, That the Academic Senate urge local senates to recommend directing scarce college resources toward professional development and away from promotion of the new Accrediting Commission for Community and Junior Colleges (ACCJC) Standards; and

Resolved, That the Academic Senate work with local senates to develop materials and strategies for resisting the new Standards in ways most relevant to their local situation and place in the accreditation cycle.

• Continued Coalition of Faculty Organizations to Address Imposition of Standards (Fall 2002, #02.07)

Whereas, The response to the Accrediting Commission for Community and Junior Colleges' (ACCJC's) new Standards will require a coordinated faculty response and multiple, multidimensional strategies at the local, state, and national levels;

Resolved, That the Academic Senate continue to work in alliance with other faculty organizations to develop a coherent set of strategies to oppose the new Accrediting Commission for Community and Junior Colleges Standards, consider and coordinate legal or legislative challenges, and educate legislators and other state and federal policy-makers regarding its concerns with the new Standards.

[Recommend that an effective working group be assembled to pursue these issues. Working group formed in 2003, chaired by the Accrediting Liaison, failed to function as chaired [sic] called no meetings.]

• Opposition to Standardized Objectives (Fall 2002, #02.09)

Whereas, The new Accrediting Commission for Community and Junior Colleges (ACCJC) Standards require that departments develop standardized course content and objectives;

Whereas, The Academic Senate for California Community Colleges as well as the Board of Governors and the Chancellor's Office encourage a range of approaches and course content which are culturally sensitive and reflect the latest disciplinary developments; and

Whereas, The pressure toward curricular standardization in the new ACCJC Standards will likely discourage innovation in course content and objectives and potentially disempower faculty who may belong to traditionally marginalized groups, thus fostering a retreat from a commitment to diversity in both content and teaching and learning approaches;

Resolved, That the Academic Senate urge local senates and the Board of Governors to protect diversity and multiculturalism by respecting academic freedom and not imposing standardized objectives and outcomes at the course, degree and certificate levels; and

Resolved, That The Academic Senate vigorously oppose the Accrediting Commission for Community and Junior Colleges' and legislative pressure to standardize with the resultant exclusion of alternative viewpoints.

[Issues set forth in this resolution have been repeatedly addressed in ASCCC papers, letters, workshops, articles, and institutes, and ongoing resistance to any hints of standardization remain essential.]

• Resisting Imposition of Learner Outcomes Assessment (Fall 2002, #02.10)

Whereas, The Accrediting Commission for Community and Junior Colleges (ACCJC) has adopted a new set of Accreditation Standards which will radically alter the way institutions of higher education are evaluated in the State of California while refusing to produce quantifiable evidence indicating that the imposition of learning outcomes assessment will contribute to any meaningful improvement in the delivery of instruction at the community colleges;

Whereas, Faculty bodies at the state level, the American Association of University Professors (AAUP), the Academic Senate for California Community Colleges, and the Community College Council of the California Federation of Teachers (CCC/CFT), have recommended that faculty not participate in activities involving the development of learning outcomes as a part of accreditation self-studies;

Whereas, The new Accreditation Standards may be interpreted as interfering with the collective bargaining process, may be inconsistent with the shared-governance provisions of Title 5 53200, and may lead to infringement of academic freedom; and

Whereas, The above groups have raised concerns that the level of emphasis on learning outcomes and the associated amount of record keeping mandated by the new Standards will be costly and time-consuming, diverting scarce resources away from classrooms, libraries, and counseling;

Resolved, That the Academic Senate resist and recommend that local senates resist any attempt to have the measurement of learning outcomes imposed upon faculty by the Standards of the Accrediting Commission for Community and Junior Colleges (ACCJC); and

Resolved, That the Academic Senate vigorously reject the notion that faculty evaluation be based on the measurement of student learning outcomes as defined by the Accrediting Commission's new Standards.

[Recommend that an effective working group be assembled to pursue these issues. Working group formed in 2003, chaired by the Accrediting Liaison, failed to function as chaired [sic] called no meetings.]

• Request the Commission to Solicit Input from Pilot Institutions (Fall 2002, #02.11)

Whereas, Several institutions, with the implicit promise of being able to positively impact adjustments to the new Accreditation Standards, agreed to pilot those Standards; and

Whereas, There has been no effort by the Accrediting Commission for California and Junior Colleges to solicit feedback from the pilot schools currently undergoing self-study;

Resolved, That the Academic Senate formally request of the Accrediting Commission for California and Junior College that it actually solicit input and in corporate consensus suggestions from the colleges piloting the new Standards.

• Conflict of Interest (Fall 2002, #02.12)

Whereas, The proposed Accreditation Standards have been developed in accord with consultants who may have a conflict of interest;

Whereas, The implementation of the proposed Accreditation Standards involve high cost to the taxpayers;

Resolved, That the Academic Senate of California Community Colleges formally request investigation by a statewide body, such as the Joint Legislative Audit Committee or a commission appointed by the Legislature, of the potential conflicts of interests and the cost of implementing the proposed Accrediting Commission for Community and Junior Colleges' accreditation Standards.

[Recommend that an effective working group be assembled to pursue these issues. Working group formed in 2003, chaired by the Accrediting Liaison, failed to function as chaired [sic] called no meetings. Recommend that this be directed to the ACCJC Liaison.]

• Make Available Background Research on Accreditation Requirements (Spring 2002, #02.02)

Whereas, Only four of the thirty-four institutions that have undergone the accrediting process in the last several years have satisfactorily met the Commission's expectations with regard to institutional effectiveness and planning, but nevertheless these expectations will become a central focus of accreditation decisions in the new standards;

Whereas, This dramatic shift in emphasis to documentation of student learning outcomes and systematic cycles of data analysis will require all colleges to make new, significant, sustained, and targeted investments in professional researchers, data analysis and computing capability, professional development, and faculty and staff time;

Whereas, This new emphasis will, by necessity, shift resources from those places most likely to produce enhanced student achievement, the classrooms, counseling offices, and libraries, where faculty and students interact, and will coincide with a time of economic downturn and lowered state support for the majority of institutions accredited by Accrediting Commission for Community and Junior Colleges (ACCJC); and

Whereas, Accreditation should evolve gradually and reflect evolving consensus regarding essential standards in the higher education community rather than abrupt, and possibly faddish, changes and trends;

Resolved, That the Academic Senate urge the Accrediting Commission for Community and Junior Colleges, prior to adopting new standards, to make available to the public and the educational community the background research materials that formed the basis for its recommendations, provide the public with data supporting the efficacy of this approach in improving education for students, and provide a more detailed analysis of the projected costs, impact, and implications of this shift in standards for the colleges.

• Excessive and Intrusive Documentation (Spring 2002, #02.03)

Whereas, Draft B of the new accrediting standards will require all colleges to specify student learning outcomes at the course, program, degree, and certificate levels, and to measure, document, and improve the attainment of these outcomes by students;

Whereas, Draft B will require that colleges specify and measure the competencies expected of students at the course, program, degree, and certificate levels;

Whereas, Draft B will require that faculty validate department or program examinations; and

Whereas, Draft B will require demonstration that student services, including counseling and library and learning resources, measure, document, and improve their effectiveness in contributing to the achievement of student learning outcomes;

Resolved, That the Academic Senate continue to engage the Accrediting Commission for Community and Junior Colleges in a dialogue about the need for extensive documentation of student outcomes and competencies (at the course, program, degree, and certificate levels) as well as for documentation of the contribution to these student learning outcomes made by student development and support services and library and learning resource services; and

Resolved, That the Academic Senate work to change the standards to avoid such a prescriptive and intrusive set of requirements.

• Work with Accrediting Commission to Buffer Colleges from Political Pressures and Agendas

• Continue Use of Current Accrediting Standards and Suspend Pilots (Spring 2002, #02.09)

Whereas, The concerns about the new accreditation standards are widespread and still unResolved; and

Whereas, Even members of the Accrediting Commission for Community and Junior Colleges agree that the assessment of student learning outcomes is a new and evolving approach;

Resolved, That the Academic Senate urge the Accrediting Commission for Community and Junior Colleges to continue to use the current accreditation standards while engaging in more extended dialogue about any proposed new standards, and

Resolved, That the Academic Senate urge the Accrediting Commission for Community and Junior Colleges and local districts to refrain from piloting the proposed accreditation standards until the direction and content of any new standards are satisfactorily Resolved.

--This last item is an essay that appeared in the October 2002 ASCCC publication:

• The Accountability Game ... MSLOs in CC's (October, 2002; Leon F. Marzillier; Academic Senate for California Community Colleges, October, 2002; Rostrum)

     …In June 2002, the Accrediting Commission for Community and Junior Colleges (ACCJC) passed radical new standards by which to accredit community colleges, incorporating the idea of "continuous improvement" of "measurable student learning outcomes" (MSLOs) throughout. The ACCJC passed these new standards over the vociferous objections of respected faculty organizations. Nationally, the American Association of University Professors (AAUP) has come out against modifying accreditation standards this way, and in California, the Academic Senate for California Community Colleges along with the Community College Council of the California Federation of Teachers (CCC/CFT) have condemned this radical change by ACCJC. Why?

     The whole concept of MSLOs as the latest fad in education is somewhat akin to the now discredited fad of the '90's, Total Quality Management, or TQM. Essentially, the ACCJC adopted MSLOs as the overarching basis for accrediting community colleges based on their faith in the theoretical treatises of a movement, just as advocates for the use of TQM in education (often called continuous quality improvement or CQI in educational circles) were part of an ideological movement. After repeated requests for research showing that such use of MSLOs is effective, none has been forthcoming from the ACCJC. Prior to large scale imposition of such a requirement at all institutions, research should be provided to establish that continuous monitoring of MSLOs has resulted in measurable improvements in student success at a given institution. No such research is forthcoming because there is none. If the "learning paradigm" is so superior as to justify its widespread adoption, then the research should clearly be compelling.

. . .

     Often, objections to MSLOs are met with, "But, you faculty will define what the outcomes to be measured are." This assumes that what faculty currently measure, via exams and grades are not adequate, and that faculty should spend their time generating new and much more specific skill based measures. However, no evidence has been presented establishing that the outcomes of our pedagogical efforts are not adequately measured by our current approaches, or that new measures would lead to greater student success.

     In addition, much that is most beneficial in higher education is often difficult or impossible to measure—but certainly is not measurable at the course level. A business department might feel the most important outcome is that their students use what they learn in the classroom successfully in a career in the business world. But this is not a learning outcome that can be documented at the course or program level.

. . .

     Furthermore, the MSLO movement utilizes a scorecard approach, in which you assess in percentage terms where your students are now in terms of a defined learning outcome, and how you would like to increase the percentage in 5 years, say. Then, you set as your goal the percentage improvement you want to make each semester! This requirement, that there be continuous improvement of learning outcomes, assumes that student achievement can be increasingly rationalized like a production process.

     This push to document and improve student learning outcomes essentially creates pressure to focus one's course objectives on discrete, skill-based and hence most easily measured variables. Quantitative variables are more easily tracked than qualitative ones. This over time will yield to a "dumbing down" of the curriculum, as broad capacities and more long-term, qualitative changes in student behavior and perception will be relatively de-emphasized in the push to measure.

. . .

     In the teaching and learning process, there is a two-way interaction, and there has to be cooperation and interest on both ends. Whether a student succeeds in a class is a function of not one but many factors. Some of these are: the intellectual level of the student's household, the quality of the preparation the student received in educational institutions attended before reaching ours, the priority that the student places on the class, the amount of effort a student is willing to apply outside of class, resistance to distractions from friends, family, and jobs. Many of these are beyond the instructor's or college's control. Yes, we can find new and better ways to present the material, and we can use tutors and workshops to help motivate students and to help them succeed, but those efforts alone might go for naught for some students.

     Perhaps what irritates us most about the ACCJC's action, besides the fact that they chose to ignore the best advice of the practitioners in the field (the faculty), is that tying accreditation to MSLOs means that the faculty as a whole would have to spend precious time and effort to engage in measuring everything that moves on the campus, diverting our energy and efforts from interacting with students. Will our colleges receive additional funding for these efforts? We seriously doubt it! So, we are being asked to engage in what virtually amounts to a huge unfunded mandate.

. . .

     Another argument that is advanced is that the ACCJC approved these new standards unanimously; the three faculty members sitting on the commission also voted for these new standards, allowing the commission staff to claim that there are faculty supporting the MSLOs movement. Of course it is possible to find individual faculty members to support almost any position that one can think of, but these faculty members were not appointed by any faculty organization, and their votes represented nobody but themselves. The Academic Senate plenary sessions passed resolution after resolution condemning the use of MSLOs as the basis of accrediting decisions, many being passed unanimously. These were votes of faculty members representing faculty at all 108 California community colleges, and represent the collective wisdom of California community college faculty. The faculty members sitting on the ACCJC are or have been active in local or state senates. It is unfortunate that they did not heed that collective wisdom and vote against the implementation of these new standards. Every fad that comes along will find a few adherents among the faculty, but when the opposition among our faculty is as strong as it is, it's clear that the faculty is not split on this issue.

     Accountability is fine, but don't give us an untested, obviously bogus scheme with which to hold us accountable when there is not one institution in the country where it has been shown to be effective. Faculty leaders were not brought into the discussion to construct these new standards. Don't tell us, "Oh, but you establish the outcomes to measure," when you haven't asked us whether or not we want to even establish such outcomes in the first place. Let us have an open, frank discussion with representatives of all constituencies, about how to judge the effectiveness of a community college for the purposes of accreditation. In California, accreditation processes are an academic and professional matter: number 7 of the 10+1 items that require input from the academic senate is "Faculty roles and involvement in accreditation processes, including self-study and annual reports." It is a violation of California law that the Academic Senate was not brought into the discussion in the formation of these new standards.

     Based on the resolutions passed overwhelmingly at its plenary sessions, the Academic Senate is studying ways to combat the institutionalization of MSLOs. The Academic Senate is working with both the AAUP and the CCC/CFT to consider our next steps, whether it is possible to delay implementation of these radical changes in the accreditation standards, as well as to explore alternatives to the ACCJC.

     Please, community college faculty members, give us your ideas on how to resist this reactionary movement so akin to what is going on in health care. Do you really want community colleges to become the HMOs of higher education? If not, spread the word that we do not have to put up with this, and together we can nip this impending disaster in the bud.

SEE also: Systematic rational failures and SLOs: part I (March 28)

*Academic Senate for California Community Colleges

ADDENDUM:

A relatively recent assessment:

• From “Outcomes Assessment: Conceptual and Other Problems” by John W. Powell, in the AAUP Journal of Academic Freedom, Vol 2 (2011)

Where Is the Research?

     What are the measurable effects of outcomes assessment [OA] programs?
     Outcomes assessment is undeniably succeeding on its own terms, in that it is becoming widely implemented and is generating data. Also undeniably, outcomes-assessment-based education lacks evidence that it is an improvement over traditional education based on any other terms of evaluation. That is, when OA-based programs are compared with other programs lacking OA, one would expect to see some kind of results. Instead, the main results one can find are question-begging [see] measures of whether OA has indeed been implemented. One wonders whether we are seeing another educational fad go nova, and what structures will remain among the clouds of gases. Advocates have succeeded in their efforts without the benefits of controlled studies. Those advocates include U.S. secretaries of education, accreditation agencies, and agencies that could have funded scientific research on these issues, and they make their case without appeal to hypothesis testing, without research projects, without publications of successes and failures, without comparison studies of any kind except those comparing before and after implementations, where indicators of success at implementation are assumed to be evidence for OA. No one has assessed the outcomes of implementing programs of outcomes assessment by comparing the results of such programs with the results of other forms of education.
     Some to whom I have made this claim find it implausible. Perhaps we can count on rebuttals after this is published. Part of the puzzlement about how advocates of outcomes assessment could have succeeded without research can be resolved by realizing that many advocates think doing outcomes assessment is doing research because it results in data. But searches of the Internet, the archives of the Chronicle of Higher Education, recent books on outcomes assessment, and their bibliographies yield a clear picture of the state of research, which is that it is made up of substitutes for research. One is directed to manifestos claiming that outcomes assessment makes sense, is needed, is succeeding, and is the wave of the future. Judging from the number of accounts of universities, colleges, school systems, and accreditation agencies in the throes of implementing outcomes assessment, one might be excused for thinking all these claims are true. Among some of these, the job of implementing a program of stipulating outcomes and assessing them is taken up by entities with names like Office of Research and Analytic Studies, again suggesting that implementing outcomes assessment is somehow doing research. A review of literature also turns up journalistic commentary on the manifestos, usually written as though simply chronicling waves of the future. There are a few people dragging their feet and worrying about implications of implementing outcomes assessment, among whom I would place myself with this essay. Somewhere lower, that is, less common, on the list of what one finds in the OA literature are inquiries wondering, “Where the hell is the research?” Occasionally there are reports of educational organizations, such as the school system in New South Wales, Australia, withdrawing from or questioning their commitment to outcomes assessment initiatives.
     Readers may still find it implausible that no controlled studies have addressed whether outcomes assessment improves education. There are journals and metastasizing bookshelves of literature, much of it originating in that part of education in which one can expect insecurities to result in scientific rigor. I finally wrote to the chair of the only PhD program in assessment and measurement in the country, at James Madison University, asking if she could direct me to any controlled studies. The first sentence of her reply was, “Your search for the holy grail and disappointment in finding it is fairly widespread.” She told me of the careful work there to do outcomes assessment testing on students at orientation and as sophomores and juniors at an annual Spring Assessment Day, and said, “We believe that the best comparison is with yourself over time, and we can document progress in a variety of ways.” She pointed out that finding institutions which do not do outcomes assessment will now be very difficult, since all regional accreditors are now mandating OA. Although she did not put it this way, in effect, any possible control group is now vanished. “The pressure from the feds on all accreditors is now unrelenting and growing in force” (Sundre 2011).
     It is striking how the question about lack of research bewilders advocates of OA—the question and the advocates seem ships passing in the night. By now we would expect reports comparing results of doing outcomes assessment with programs in which there is no outcomes assessment. But outcomes assessment is seldom mentioned now except in contexts in which the speakers push to implement it. Advocates, convinced that the results of implementation will produce clear data, present before and after snapshots as evidence that implementation in fact increases an institution’s ability to meet its declared outcomes. The easy availability of that data is evidence that the advocates do not yet understand this question. Declaring outcomes and success at implementing outcomes is, after all, already a part of programs of outcomes assessment. What is needed is testing whether doing all that is educating better. The concern is partly whether the faith in OA is built on wishful thinking. And articulating outcomes is, after all, requires that those who do the declarations be wise about what they do. Where is the research showing that this step leads to better education of students? If you are convinced going in that better education is education which is meeting measurable objectives, then we’ve a no- brainer: of course OA-shaped education will be better (provided it produces evidence that it really is OA-shaped). On any robust conception of evidence, however, in which we insist on evidence that is not part of circular reasoning, the evidence is alarmingly absent. Indeed, when I have asked administrators what got them to endorse OA, all of them have cited their own individual anecdotes, mostly recognitions of shortcomings which came from reexamining their purposes in teaching, with no acknowledgement that they are overgeneralizing or reaching hasty conclusions based not on outcomes assessment or methodical checking but on the confounding variable of having reexamined how they teach—a much more broad and flexible process than articulating measurable student outcomes, testing for them, and accumulating reams of documentation.
     A review of those alarmingly large sections of library bookshelves dealing with outcomes assessment also reveals large overlaps with corporate calls for increased productivity in education. For that large camp of education reformers who think that higher education should be more businesslike, these overlaps seem to be part of the appeal. But calls for increased productivity should be informed by knowledge of what is being produced—what, in other words, education is for. I return to this below in the sections “Neglected (Because Unknown) Outcomes” and “Appropriate Actions for Us to Take Now.” ….

     Trustee Jim Wright: very old school, stubborn. Doesn’t seem to recognize IVC’s existence. Invariably speaks of “our” team or “our” event when referring to anything at Saddleback. Listens hard to nurses, that vast demographic, that tail that wags the dog.

     Trustee Bill Jay: well, he’s dead. Soon to be replaced. God help us!

     Trustee Tim Jemal: the newest and brightest member. Not incorrigible, like the others. Tends to focus on college-private partnerships and the like. He’s the board’s future. Help him if you can.

     Trustee PJ Prendergast: High school teacher and unionist & funny man. He's been drinking the board Kool-ade, it seems, from Day 1, so he's not the working class hero he coulda, shoulda been.

     Trustee Nancy Padberg: very right-wing but has generally learned to play nice with others. Since her accident, seems to be firing not on all cylinders; that situation can’t last.

     Trustee Dave Lang: to paraphrase Lou Reed, “the night has seen your mind, and, inside, you’re twisted and unkind.” A smart and decent guy made ugly & bitter by a fateful decision to kiss Tom Fuentes’ sulfurous red ass (at a crossroads, at midnight, for the secret handshake). There’s no coming back from that, it seems; morally, he’s the walking dead.

     Trustee Marcia Milchiker: wacky but mostly reliable; the buffer between faculty and Trustee Raghu Mathur. Hence, she’s great.