Resize Text: 
In recent years, an acrimonious debate has broken out in higher-education circles about institutional accountability and performance. Efforts to alter federal policy in particular have been flashpoints for often heated discourse about the ways that colleges and universities could—and should—demonstrate their effectiveness to skeptical outsiders. Much of the talk has focused on deliberations in Washington, including those of Secretary of Education Margaret Spellings’ Commission on the Future of Higher Education, since these deliberations have the potential to affect virtually every institution of higher learning across the country.
In quiet counterpoint to the maelstrom over federal policy, though, a more measured approach has been at work on the campuses of a consortium of colleges and universities. Over the past four years, the 33 liberal arts colleges and universities that constitute the Council of Independent Colleges’ Collegiate Learning Assessment Consortium (CIC/CLA Consortium) have been using the Collegiate Learning Assessment (CLA) to gauge how the college experience helps students develop such “higher-order” cognitive skills as critical thinking, analytic reasoning, problem solving, and effective writing. Through tests administered to both first-year students and seniors, it has been possible to evaluate the institution’s impact on students’ command of these skills and to associate the gains—or “value added”—with the institutional effects of the collegiate experience.
The Council of Independent Colleges’ Role
The Council of Independent Colleges' (CIC’s) chief concern in the debates about accountability has been the prospect of a government-controlled testing regimen that would run roughshod over institutional autonomy and individual privacy. However, unlike some organizations that cite these principles when resisting accountability measures, CIC has embraced what it has viewed as the best available nongovernmental approaches to assessing educational effectiveness. In 2001, well before the Spellings Commission called for greater accountability in higher education, CIC became the first major institutional membership association in higher education to urge its members to use the National Survey of Student Engagement (NSSE). This appeal by CIC was followed by one to use the CLA.
CIC took this stance for several reasons. First, we believed that anecdotal ways of describing the effectiveness of small colleges were no longer persuasive in an era when empirical evidence has become the standard way of proving a point. Second, we believed that small colleges would, on the whole, compare favorably with other types of four-year institutions in any data-driven approach to measuring institutional effectiveness and student learning. And third, we believed that we could persuade the federal government to drop its consideration of government-imposed testing if we could show that voluntary approaches were working well.
CIC’s focus on the CLA began in 2002, when it was approached by the Council for Aid to Education to assist in identifying smaller private colleges to test the prototype of the CLA. The following year, CIC recruited a larger group of 12 member colleges and universities to use the CLA. In 2005, with generous support from the Teagle Foundation, CIC expanded this initial group of institutions to include the 33 colleges and universities that comprise the current CIC/CLA Consortium, now in the final year of a three-year commitment. In March 2008, again with Teagle Foundation support, the Consortium was expanded from 33 to 47 institutions for another three-year effort.
Consortium members administer the assessment to cross-sectional samples of first-year students in the fall and seniors in the spring, analyzing the results to determine their areas of strength or weakness in generating student learning and developing appropriate strategies to address the weaknesses. A team of faculty members and administrators from each member institution participates in an annual summer meeting of the Consortium, during which ideas and strategies are shared—a form of mutual support that is continued throughout the year via web conferences, listservs, and email. Team members also serve as interpreters of the CLA on their campuses and as resource persons in assessing student learning more generally.
Using the CLA for Institutional Purposes
The advent of moderately priced, reliable, standardized assessment instruments is equipping college leaders with new tools by which to gather evidence about student learning. Until the development of the National Survey of Student Engagement (NSSE) and the CLA, most colleges’ approaches to assessing student learning had been anecdotal and attitudinal. For example, Terrence Grimes, vice president for academic affairs at Barton College in North Carolina, recalls a time when writing portfolios were expected of students there but were never collected or analyzed, and when some courses that ostensibly emphasized writing “really didn’t.” In short, he says, there was very little concrete evidence of students’ competence in writing.
The potential benefits of using the CLA go well beyond its use as an assessment instrument, especially given the right timing and fit with institutional culture and concerns. At Barton, for instance, the CLA was introduced in 2005 in conjunction with a review of the general education curriculum. At that time, according to Grimes, there was also pressure on Barton from its regional accreditor to develop quality-improvement programs. After some difficulties in the early administration of the instrument, the CLA was integrated into the first-year program as well as administered to seniors.
The CLA showed that a Barton education added value to students’ skills, but the faculty was nevertheless disappointed in the absolute quality of the writing, even for seniors. When, in the spring of 2007, members of the board of trustees reviewed the CLA results, they came away with the same impression. As associate professor of biology Kevin Pennington noted, although the faculty were initially “leery about the effectiveness” of using the CLA, the results “allowed us to talk meaningfully, beyond anecdotes, about how to improve student writing.” The changes in the general-education program, Grimes reports, are now being implemented, and the college is relying on “CLA-style” exercises and measurements through the first three years of study.
Cabrini College in Pennsylvania also used the CLA in conjunction with a reform of general education. One additional result at Cabrini has been especially important—Charlie McCormick, dean for academic affairs, believes that the “value-added” results also gave Cabrini a new way to talk about itself in the competitive market of colleges in the Philadelphia area, as well as a renewed sense of institutional identity.
Presidents of other institutions that participate in the CIC/CLA Consortium have also affirmed its usefulness. Alaska Pacific University uses CLA-based exercises in its faculty development programs. And Richard Cook, president of Allegheny College in Pennsylvania, says, “I think it’s vitally important both for demonstrating value-added and for improving what we do.” Edwin H. Welch, president of the University of Charleston in West Virginia, agrees. As a board member of a regional health facility, he had seen how outcomes-based quality-improvement programs were an effective mechanism for institutionalizing improved practice in health care. He noted that assessing, documenting, and improving quality at regular intervals were accepted as requirements in healthcare and immediately saw that similar practices could be established in higher education. Welch’s observations parallel those of Antul Gawande in his “Bell Curve” article in The New Yorker (December 6, 2004).
Lessons Learned
Several key lessons from the CIC/CLA Consortium have thus far emerged:
1) Involve faculty members. First, as other people have noted since the beginning of the assessment movement, engaging faculty members in assessment is essential to improving student learning. Linda DeMeritt, dean of the college at Allegheny College, says, “I think that once you get faculty members to sit down and look at what the CLA is testing, they agree generally that this is a valuable test.” It’s not that faculty members are leery of assessment, she says, but rather that they are wary of any standardized test they are afraid they’ll end up “teaching to.” Allegheny’s experience, she says, has been that “when faculty members actually see a test like this, which is not your standard multiple choice test, they begin to see its value.” In fact, DeMeritt says that as the Allegheny community reviews syllabi and teaching effectiveness, it is asking whether it can “incorporate any of these types of performance-based tests into our own pedagogy.” In that sense, she says, the CLA is having an impact by prompting Allegheny’s faculty “to think more in terms of learning outcomes than teaching objectives.” The Cabrini experience also illustrates that once faculty members become comfortable with the idea of measuring outcomes, they will become eager to find appropriate assessments and to link them to ongoing efforts to improve the teaching of their subjects.
Ursinus College in Pennsylvania attributes its interest in the CLA to the increased role of faculty members in assessment. The college’s Committee on Outcomes Assessment allows faculty members to draw insights from CLA results that can strengthen students’ learning. They were especially interested in documenting the impact of the first-year liberal studies seminar, the Common Intellectual Experience (CIE), on students’ intellectual development. This seminar and its assessment results are being monitored for applicability to a planned “CIE for Seniors” seminar.
Faculty involvement doesn’t happen by itself. Both Grimes at Barton and McCormick at Cabrini spent a great deal of time meeting with faculty members to articulate the rationale for using the CLA. Perhaps equally important, from the beginning they had the support of a few faculty members who were willing to experiment with the instrument and who then became advocates for it.
2) Don’t rely on a single measure. Much of the resistance to any standardized test has targeted the danger of relying on a single measure of institutional outcomes (see Richard J. Shavelson, "Assessing Student Learning Responsibly," Change, January/February 2007). But pairing standardized test results with other assessment measures, such as the NSSE or portfolio analyses, and linking test scores with other relevant assessment data provides more robust diagnostic information for efforts to improve teaching and student learning.
Cabrini College, for example, compared its NSSE “Level of Academic Challenge” scores with its CLA results to inform its revision of the general education curriculum. Stonehill College in Massachusetts had a similar experience: Initial results from both the CLA and NSSE—in combination with increasingly selective admissions practices—led the college to question whether it was adequately challenging its students and eventually to a proposed modification of its course-credit model. At Alaska Pacific University, the CLA complemented a recent faculty effort to integrate rubrics detailing student-learning outcomes—and ways to assess them—into course syllabi.
Good performance-based models of testing such as the CLA can also inform the development and improvement of other assessments. At Seton Hill University in Pennsylvania, Mary Ann Gawelek, vice president for academic affairs and dean of the faculty, says faculty members and administrators are now asking themselves, “How do we ... push ourselves to use more creative, more applied assessment techniques?”
3) Share your results. A willingness to share CLA results widely opens up important conversations. “We report both NSSE and CLA data at our opening fall workshop, so everybody hears it, not just the faculty—including, by the way, our student leaders,” Seton Hill’s Gawelek says. The university has also shared test results with the educational policy committee of its board. Trustees, Gawelek says, have asked penetrating questions about how the university uses the test results. Cabrini College’s leaders made a point of discussing the assessments with the president’s cabinet, the enrollment management committee, staff members in financial aid, and others. Talking about the test results, Dean McCormick says, opened a way to engage the whole campus, including campus staff other than faculty members, in conversations about what they could do in their areas to help improve student performance. Barton College, which has reported CLA results even to prospective students and their parents, is relying on the hard evidence the test provides to continue pressing for improvements in student writing and critical-thinking skills.
Widespread reporting of results can affect campus culture. Through such activities as strengthened program review, better institutional effectiveness procedures, participation in the NSSE survey, and hard work by faculty committees, Texas Lutheran University is creating what provost and executive vice president John Masterson calls “a real culture of assessment and continuous improvement” that is more clearly focused on improving the quality and structure of student educational experiences. “Assessment is not just something you do for the accrediting bodies,” says Masterson. At Barton too, the CLA has helped create a “culture of evidence” that goes well beyond the particulars of the instrument.
Practical Challenges
Learning how to use CLA results to influence other aspects of educational effectiveness is, frankly, still a work in progress. The institutions that are wrestling with this challenge face many obstacles—some minor, some significant. At Allegheny College, for example, initial CLA results showed that student performance was not at the level expected, nor was it in alignment with more positive results from other measures. After satisfying themselves that the test itself was not flawed, Allegheny’s leaders looked more closely at the logistics of test administration. They suspected that the CLA results might be skewed by the progressively stronger academic performance of entering first-year students. “Our academic profile has gone up in the last three or four years,” President Cook says, “and so we haven’t had an apples-to-apples comparison. We think that puts uncertainty into the system.” But even if it’s “not a perfect measure,” Cook says, “we think it’s worth pursuing.”
Timing can also skew outcomes. When Bethel University in Minnesota gave first-year students the assessment well into their first semester, they got results that school administrators sensed were too good. The university suspected that students did better on the test with a month of classes under their belts than they might have done before classes started, when they were unaffected by the college experience.
Another challenge some institutions have faced is the requirement to provide SAT or ACT scores for students tested, since these measures are used to control for students’ initial academic ability. Alaska Pacific University and Heritage University in Washington state, both with large numbers of students who come from groups that are historically underrepresented among college-goers, are two universities that do not require the SAT or ACT. CLA administrators worked to find an alternative test to use as a control and settled on the Scholastic Level Exam (SLE), a short-form measure of cognitive ability developed by Wonderlic (an Illinois company with more than 40 years experience in educational testing services), which has turned out to be a reliable substitute. The SLE has proved valuable for other institutions that wanted to administer the CLA to nontraditional adult learners and is now being widely used in community colleges.
But by far the biggest challenge has been getting students to take and do well on the CLA. Because it is not a high-stakes exam like the GRE or MCAT, it has been difficult to recruit students and to be sure that they have given the test a good effort. The Council for Aid to Education learned early that the standard three-hour version of the assessment was unwieldy. The solution was to create a matrix sampling design, so that every student takes a 90-minute version of the test—essentially half the instrument.
However, shortening the testing time proved to be only half the challenge. In the initial round of consortium testing, only six of the 12 institutions met minimum sample sizes needed for valid test results. So the current group of 33 institutions had intensive discussions of recruitment strategies, which did yield improved response rates. Nonetheless, after three years of concerted efforts and refining approaches, getting students to take the CLA remains a big challenge—especially among seniors, who are far less responsive to appeals to volunteer than first-year students.
The least successful strategy among consortium members has been a campus-wide call for volunteers, even with prizes of gift certificates, raffles for iPods, or cash offered as inducements. In fact, many of the consortium’s member colleges balked at the initial recommendation to pay students $25 an hour to take the CLA, in part due to the added cost and in part because faculty members believed that such an approach ran counter to campus values.
The most successful approach to administering the CLA has been to incorporate testing into the academic program—giving it during new-student orientation or a campus-wide assessment day, or embedding it in first-year seminar and senior capstone courses. Westminster College in Missouri, for example, now gives the test to all first-year students during their first week on campus in the fall, then tests seniors as part of an annual spring assessment day, when regular classes are suspended to allow time for testing. The expectation is simply that seniors will take the CLA that day, and last year 90 percent complied. It is worth noting, though, how reluctant faculties have been to impose a new requirement, coming to the conclusion that it is appropriate to do so only after becoming more familiar with the CLA and the importance of outcomes assessment.
Bethel University’s progression may be typical. In its first year of participation in the consortium (2005–2006), Bethel struggled to find a time during the fall semester when first-year students could take the test. Administrators solicited the participation of some instructors, invited students to participate, and offered $5 gift certificates as incentives. But in the end Bethel suffered from small samples, according to Richard Sherry, the university’s dean for faculty growth and assessment. So in Bethel’s second and third years, the university decided to make the test part of assessment activities during its new-student “welcome week.” Out of pools of students whose ACT/SAT scores were representative of the entire class, some students were assigned at random to take the CLA, while the remaining students were given a critical-thinking assessment that Bethel has used for some years. “That worked extremely well, and we got our sample in roughly two and a half hours,” Sherry reports.
To get seniors’ participation, Bethel follows a strategy that many members of the consortium employ—working directly with faculty members and students in senior capstone courses, in this case ones that have intensive critical thinking and writing components. Faculty members motivate students to take the test, some by offering bonus points or the equivalent as a “carrot” and others by using the CLA as a course requirement. Similarly, at Seton Hill University, which has built the CLA into the curriculum of senior capstone courses, one seminar requires students to take the test as part of studying why personal assessment is critical to lifelong learning.
A Work in Progress
In 2007, with substantial new funding from the Teagle Foundation, CIC issued a call for proposals to extend and expand the consortium’s work for another three years, through the spring of 2011, to 47 of the institutions that applied. In this new phase, the emphasis will be on developing more comprehensive campus-assessment plans by incorporating additional measures such as student portfolios, campus-based instruments, and other standardized instruments such as the NSSE. Experience has shown that while the CLA provides a reliable measure of overall institutional contribution to student learning, it is most beneficial when used in conjunction with other efforts to improve student learning. Accordingly, institutions in the consortium are now expected to develop multifaceted assessment programs that best suit campus needs.
The consortium’s work has other new features as well. Some institutions will also conduct in-depth sampling, testing additional students who have characteristics of interest to the institution—division or major, gender, or race/ethnicity, for example. Also, continuing consortium members will mentor institutions that are using the CLA for the first time.
Finally, in this new phase the consortium will help institutions involve increased numbers of faculty members in the assessment of student learning and introduce them to the comparative perspectives that the consortium provides. More use of results to improve pedagogy, redesign curricula, and heighten awareness of assessment throughout the campus is likely to occur.
The Council for Aid to Education has developed a new “CLA in the Classroom” program, in which faculty members can use a mock version of the instrument in individual classes as a means of gauging and improving students’ skills and can learn how to develop their own CLA-like performance tasks. In many ways, this represents the next logical phase in the development of the instrument. Created and tested with input from members of the CIC/CLA Consortium, the project provides faculty members with a set of techniques—such as scoring rubrics and test questions patterned after the CLA prompts—that apply some of the principles of performance-based assessment directly to the improvement of student learning. Marc Chun, director of CLA in the Classroom, says that the program will enable faculty members to “have a better conversation with their students about where their performance could be improved relative to higher-order skills, as well as to link institution-wide assessment with classroom-level teaching and learning.” The new instructional tools will provide faculty members with the practical means to do diagnostic work with students—and become more personally invested in an assessment-based commitment to improving student learning.
The varied experiences of the members of the CIC/CLA Consortium suggest that assessment of learning outcomes can be applied to many aspects of educational improvement. “Using the best available tests for measuring student learning and combining those with self-created instruments,” says University of Charleston’s President Edwin Welch, “is providing evidence to students and parents about the process and the results of a University of Charleston education. Improving student learning is not just good for the institution; it is precisely what students and parents deserve.”
*************************************************************
The Collegiate Learning Assessment
The Council for Aid to Education (CAE) developed the Collegiate Learning Assessment (CLA) as a way to measure institutional contributions to gains in student learning. The assessment provides a comprehensive measure of some of the “higher-order” skills that often overlap with the general-education goals of the undergraduate curriculum: critical thinking, analytic reasoning, problem solving, and written communication.
Unlike most tests of student learning that use multiple choice, true-false, or short-answer questions, the CLA uses written, student-constructed responses to open-ended assignments. In contrast to subject-domain instruments that test students’ knowledge of particular disciplinary content, the CLA poses real-world problems that students must address by evaluating evidence, synthesizing information, drawing conclusions, and constructing their own arguments for or against a particular position.
The CLA uses three key measures to assess student abilities:
• Make an Argument. The ability to take and justify a position on an issue.
• Critique an Argument. The ability to evaluate an argument for how well reasoned it is.
• Performance Task. The longest section of the test asks the student to complete a real-world task, such as preparing a briefing report using a set of provided materials.
For additional information, visit www.cae.org/cla.
*************************************************************
Members of The CIC/CLA Consortium (2005-2008)
Alaska Pacific University (AK)
Allegheny College (PA)
Aurora University (IL)
Averett University (VA)
Barton College (NC)
Bethel University (MN)
Cabrini College (PA)
Centenary College (NJ)
Charleston Southern University (SC)
College of Saint Benedict/Saint John’s University (MN)
Franklin Pierce University (NH)
Heritage University (WA)
Indiana Wesleyan University (IN)
Loyola University New Orleans (LA)
Lynchburg College (VA)
Marian College (WI)
Pace University (NY)
Pacific University (OR)
Seton Hill University (PA)
Southwestern University (TX)
Stonehill College (MA)
Texas Lutheran University (TX)
University of Charleston (WV)
University of Evansville (IN)
University of Great Falls (MT)
Ursinus College (PA)
Ursuline College (OH)
Wagner College (NY)
Wartburg College (IA)
Wesley College (DE)
Westminster College (MO)
Westminster College (UT)
William Woods University (MO)
Richard Ekman is president of the Council of Independent Colleges. He previously served as vice president for programs of Atlantic Philanthropies; secretary of the Andrew W. Mellon Foundation; and at the National Endowment for the Humanities, where he was director first of the Division of Education Programs and subsequently of the Division of Research Programs. Previously he was vice president and dean, as well as a tenured member of the history faculty, at Hiram College and assistant to the provost at the University of Massachusetts at Boston. Stephen Pelletier provides writing, editing, and editorial-consultant services for associations, foundations, nonprofit organizations, and corporations. Pelletier previously served as vice president for communications at the Council of Independent Colleges, associate director of communications at the Howard Hughes Medical Institute, and senior director of publications at the National Association of International Educators.