November-December 2010

by Alexander C. McCormick

Accountability figures prominently in the higher education policy discourse these days. Critics of higher education (for instance, members of the Spellings Commission) argue that colleges and universities need to be more “transparent” to justify support by taxpayers, payers of tuition, and other funders. To better inform prospective students and other stakeholders, we are being asked to go public about our programs, educational processes, and results.

The presumption is that better-informed college enrollment decisions will then reward strong institutions, while poor performers will face the stern discipline of the market and either improve or fail. Although this logic assumes students to be rational value-maximizers and neglects some fundamental realities of college choice for the vast majority of students—whose enrollment decisions are largely determined by geography and cost (Ewell, 2009)—it is nevertheless compelling to many external stakeholders.

But transparency can be about more than consumer information. It can provide an opportunity for a college or university to proclaim its successes while acknowledging that it needs to improve in some areas. Information about results can also document progress toward important goals. For internal audiences, this kind of information focuses attention and signals priorities for improvement, while for external observers it offers evidence that the academy takes its educational mission seriously and practices what it preaches regarding the use of evidence to support assertions, interpretations, conclusions, and prescriptions for action.

Such openness is risky for several reasons, though. Revealing shortcomings invites negative consequences, whether from a legislature that may be seeking ways to cut budgets or to demonstrate a hard-nosed commitment to quality, from competitors seeking to exploit vulnerabilities, from alumni or other constituents on the lookout for evidence of a decline in standards, and so on.

As a result, “transparency” is sometimes a euphemism for what might be more accurately described as strategic communication or image management, in which information is carefully selected and presented so as to portray a successful and effective institution. This should come as no surprise. A persistent, paradoxical, and problematic characteristic of accountability systems in general is that they create a powerful incentive for those being held accountable “to look as good as possible, regardless of the underlying performance” (Ewell, 2009, p. 7).

In this article, I examine various forms of public reporting of student engagement information. Some are controlled by third parties and others by the institutions themselves. I look at four third-party efforts: the Community College Survey of Student Engagement (CCSSE), the Voluntary System of Accountability (VSA), Transparency by Design (TbD), and the institutionally authorized online publication of results from the National Survey of Student Engagement (NSSE) by the daily newspaper USA TODAY (USAT). (See Resources for links to these Web sites.) The Center for Community College Student Engagement (which houses CCSSE) recently introduced a new survey focused on the early experience of entering community college students; while this review focuses on CCSSE, it applies equally to the Survey of Entering Student Engagement (SENSE), which uses a comparable reporting framework.

It is worth noting a significant difference between CCSSE and NSSE. NSSE was established first, and in its early days it became clear that most four-year colleges and universities would not participate if their results were to be made public. As a result, NSSE's institutional participation agreement specifies that institutionally identifiable results will not be reported without prior permission of the institution. This is why the USAT initiative is limited to institutions that authorize the release of results to the newspaper. When CCSSE was launched in 2001, its funders, project leadership, and the community college presidents advising the project opted for different ground rules: institutional results would be publicly available.

The four initiatives examined in this article represent a significant advance in the movement for transparency by providing a common framework for the display of comparable information about the student experience for all participating institutions. The institutions involved deserve credit for electing to participate when that decision means ceding control over which institutional results will be displayed and the form that display will take, as well as the danger that the complex information presented will be used by others in simplistic and reductionist rankings enterprises.

In the final section, I advance some propositions regarding the promise, limitations, and potential dangers of transparency. In short, as George Kuh suggested in a 2007 Change article, transparency is “risky business.” The voluntary third-party transparency efforts reviewed here illustrate many of the complications and compromises associated with going public: who elects to participate, which measures are included, and how they are distilled and represented.

I argue that transparency efforts need to move beyond institution-level summary measures to reveal how the student experience varies within campuses, and also that we need to move the discussion beyond comparing institutions on the results of a given assessment to what institutions actually do with assessment results—what improvement plans are put in motion and whether they achieve the desired effects.

Third-Party Efforts

There are several important factors to consider in ensuring that student engagement information is usefully provided. Nine attributes are particularly important (see Table 1). While my intent is not to grade or rank the four projects named above, it is instructive to examine how they respond to the key questions set forth in the table.

Table 1. Important Considerations for Third-Party Reporting

Selection of Participants

In all four initiatives, institutional participation is voluntary, which is either self-evident or made plain in descriptions that are readily available. It is worth noting that “voluntary” may be, to some degree, in the eye of the beholder. For instance, some representatives of NSSE-participating institutions, especially public ones, claim that it is politically difficult for them to opt out of the USAT effort. However, fewer than half of eligible institutions (defined as those that have administered NSSE in the last three years) currently participate in the USAT project, indicating that its voluntary nature is widely understood. I will return to the issue of self-selection below.

Coverage

Two of the four initiatives provide either approximate (VSA) or exact (CCSSE) counts of the number of institutions included (the exact VSA total is given on a different Web site from the one displaying institutional results). Exact totals for TbD and USAT can be tallied from institution lists. TbD is the newest of the four projects examined. At this writing it numbers only 17 institutions, while CCSSE, USAT, and VSA include 658, 455, and 332 institutions, respectively. (The CCSSE-affiliated SENSE project reports results for 120 institutions.)

Knowing the number of participating institutions is most meaningful when the number eligible is also known, to put the count in perspective. Only one of the four, VSA, makes this information readily available in the form of a rough count of the number of eligible institutions (“more than 500”). TbD describes itself as “a consortium of regionally-accredited, adult-serving distance higher education institutions.” Participating institutions run the gamut of control (private for-profit, private not-for-profit, and public) and degree level (two-year, four-year, and exclusively graduate). A precise definition of TbD's eligibility pool is neither provided nor readily inferred.

Clarity

CCSSE and USAT provide the most thorough information about the measures, what they represent, and why they matter. TbD and VSA provide some of this information, but they do not explain why the particular measures were chosen from the range of possibilities, and it is not always clear how the multiple response options for a given survey question (e.g., a scale ranging from one to seven or from “very little” to “very much”) were converted to a percentage. The absence of that information limits the user's understanding of what a score signifies, and indeed it creates a risk of misinterpretation. (I elaborate on this point in the discussion of measurement sensitivity below.)

TbD also takes some liberties in the representation of survey questions. Questions about frequency of behavior (e.g., “In your experience at your institution during the current school year, about how often have you…”) are recast as an assessment of the institution's contribution to students' ability to perform that behavior or as a “rating” of the institution. This misrepresents the information being reported.

Multidimensionality

Student engagement and college quality are not unidimensional attributes. Rather, they are multifaceted. None of the four initiatives distills engagement information to a single number, and in this sense they all respect and convey to users the multidimensional nature of student engagement. However, some do so more comprehensively than others. TbD and VSA report student responses to a small subset of questions from the NSSE survey—15 for TbD and 23 for VSA out of roughly 85 questions about the student experience. They also report results for seniors only, thereby excluding potentially important information about the first-year experience at the participating institutions.

CCSSE and USAT report benchmark scores that use a group of related survey questions to produce an overall score that bears on a broad theme. For example, NSSE's five Benchmarks of Effective Educational Practice (level of academic challenge, active & collaborative learning, student-faculty interaction, enriching educational experiences, and supportive campus environment) draw on 42 questions from the survey. In this respect they provide more comprehensive information that is also more desirable from a measurement perspective, because composite measures are less vulnerable to measurement error than individual items.

A shortcoming of this approach is that the measurement scale for such composite measures is less concrete than the percentages used to describe responses to individual survey items. Benchmark scores do not signify tangible, easily recognized quantities, and as a result they only have meaning relative to a comparison of some kind—a group of comparable institutions, for example, or the same institution's score from a prior administration.

CCSSE provides the greatest transparency, in that it reports not only benchmark scores for each institution but also means and frequency distributions for the individual questions that make up each benchmark. USAT reports NSSE benchmark information for both first-year and senior students, providing more nuanced and complete information about the student experience than the two projects that rely exclusively on seniors' responses to selected survey questions. It seems likely that prospective students would find particular value in learning about the first-year experience (The CCSSE-affiliated SENSE project, as noted above, focuses on entering community college students.)

Contextualization

Seasoned observers of higher education emphasize the importance of interpreting institutional results and performance differences among institutions in the context of the many ways in which US colleges and universities vary, including by mission, resources, selectivity, population served, residential character, and so on (Ewell, 2009; Kuh, 2007). Indeed, my former colleagues and I at the Carnegie Foundation invested a lot of effort in the redesign of the foundation's classification system to improve its sensitivity to those many dimensions of difference (with the predictable result being a more complex classification system).

It is thus important to provide an appropriate reference group that limits comparisons to roughly similar institutions. This reduces the likelihood that users of engagement information will set the bar too high or too low for a given institution, and it increases the likelihood that institutions that achieve positive results relative to their reference group will be recognized (see Kuh et al., 2005).

Given the importance of contextualization, it is somewhat surprising that only USAT provides explicit contextual reference points, in the form of the average benchmark scores for students attending institutions of the same basic Carnegie type as the focal institution. USAT also provides guidance for interpreting the size of differences between an institution's results and those of its reference group. The only shortcoming of the USAT presentation with respect to contextualization is its graphical representations: Bar graphs are presented in a very condensed vertical space, making differences between an institution and its reference group—generally small to begin with—difficult to discern.

CCSSE reports key institutional characteristics such as organization type, size, and urbanicity in a college's profile, and VSA reports each institution's Carnegie classification. But neither provides reference-group scores in its public database (CCSSE does provide such comparisons in its reporting to institutions). TbD does not provide any reference-group information, but it does report institution size and characteristics of enrolled students.

Sensitivity to Variation Among Institutions

Many questions on NSSE or CCSSE ask students to choose a response from an ordered set that uses subjective levels (e.g., never, sometimes, often, very often; or very little, some, quite a bit, very much). Reporting the distribution of responses to such items conveys full information, but it consumes a lot of space, it can be overwhelming to users, and proper interpretation can be complicated.

A common solution is to aggregate responses into a single readily understood percentage (e.g., reporting the percentage who ever perform a certain behavior by summing the percentages for “sometimes,” “often,” and “very often;” or combining the percentages for “often” and “very often” as the percentage who frequently perform the behavior). However, converting multiple response options to a percentage involves a loss of potentially meaningful information.

For example, summing the percentage of students responding “sometimes,” “often,” and “very often” loses all distinctions among these levels of frequency: An institution where half of the students answered “sometimes,” a quarter “often,” and 5 percent “very often” would have the same combined percentage as one where only 5 percent answered “sometimes,” 5 percent “often,” and 70 percent “very often” (see Kelly and Aldeman, 2010).

By contrast, the benchmark approach employed by CCSSE and USAT preserves measurement sensitivity, because the benchmark calculations take the full response set into account. The summaries reported in TbD and VSA risk obscuring the potentially meaningful differences in how students use the available response options.

However, as noted above, the percentage format is more concrete, easily recognizable, and comprehensible to a wide range of users than the more abstract benchmark scores. Indeed, because the benchmarks are reported on a 100-point scale, CCSSE and USAT (and NSSE) take pains to convey to users that these scores are not percentages and should not be interpreted as such. This is a good illustration of the tradeoff between preserving desirable measurement qualities and reporting quantities that are accessible and easily to interpret.

When the percentage summary approach is selected, a reasonable expectation would be to make the basis for collapsing responses readily available to users, so that they are informed and knowledgeable about what the numbers represent. At present neither TbD nor VSA provide this information.

Sensitivity to Variation Within Institutions

Despite widespread and powerful institutional narratives of distinctiveness, rankings implying that small differences meaningfully differentiate institutions, and a policy climate focused on comparing institutions, the simple truth is that student engagement, like most other measures of student experience and outcomes (Pascarella & Terenzini, 1991), is more variable within institutions than among them (Kuh, 2007; NSSE, 2008). Attributing great significance to the experience of the statistically convenient fiction of the “average” student, then, is more than a little problematic. Even institutions with strong average scores can have many students who are relatively disengaged.

But our tools for representing internal variation in a simple, straightforward, and accessible form are limited. Perhaps understandably, none of the transparency initiatives reviewed here do a good job of acknowledging and representing internal variation. This shortcoming is hardly specific to these projects. It is a consequence of deeply held beliefs that educational quality is an institutional attribute rather than something that can vary by school, department, and classroom—and among students within these settings.

Comparison

While the foregoing discussion should make it plain that comparing results for the “average” student at two different institutions has limited utility, institutions do account for a portion of the total variation in student engagement (Kuh, 2007; National Survey of Student Engagement, 2008). So we should not dismiss the role that institutional culture and programmatic interventions can have in shaping student engagement. Moreover, the desire of some policymakers for institutional comparisons, and the public appetite for those comparisons, will not go away anytime soon.

Three of the four initiatives reviewed here (CCSSE, USAT, and VSA) permit institutional comparisons by virtue of the fact that they report institutional results. But such comparisons require manually looking up results for institutions of interest. It is difficult to imagine many prospective students taking the time to do this and then understanding how to draw meaningful conclusions from the comparisons. Only TbD has a rudimentary comparison feature that produces a side-by-side display of results for a specified pair of institutions.

None of the four includes search tools that would permit the user to query the database to generate a list of institutions that meet a threshold level of performance. Easy comparisons do risk facilitating the development of reductionist rankings, and both CCSSE and NSSE (which set limits on USAT's use of NSSE benchmark data) have adopted explicit positions on the appropriate uses of student engagement data and oppose their use to rank institutions. The voluntary nature of these projects virtually ensures that a feature enabling selection or ranking would severely undermine participation. I will return to this important consideration below.

Interests Served

Do these initiatives serve the public interest? In my view they most surely do. They make important information about educational quality readily accessible without corrupting the assessment projects on which they rely (and that's no mean feat). They help focus the discourse about educational quality on matters of teaching and learning instead of reputation and resources. And they offer models of transparency at a time when college and university leaders are skittish about making performance information public in a form that can be used to make comparisons.

Do they serve the interests of participating institutions? Evidently they do, given their voluntary nature. That institutions elected to participate means that on each campus, a decision-maker reached a determination that the benefits of participation would exceed the cost (both financial and nonfinancial).

Do they serve the interests of higher education? While it is foolhardy to ascribe unitary interests and motives to all of higher education, the answer to that question is most straightforward in the case of VSA, which was created by the two largest national organizations of public universities, the American Association of State Colleges and Universities and the Association of Public and Land-grant Universities. The leaders of these organizations decided it was preferable to design their own accountability system than to risk having one designed for them, and they took the bold step of insisting that their system would include direct evidence of learning.

A similar logic applies to TbD, albeit for a far smaller and more narrowly defined group of institutions. While TbD might be dismissed for its small size and hodgepodge of institutional types, it represents an aggressive move by a group of institutions that struggle for legitimacy against mainstream higher education (and also against the largest institution that serves their self-proclaimed market niche—the University of Phoenix—which is notably absent from the group). It seems clear that they are using the initiative to make a strong statement about their members' commitment to transparency—one that might invite questions about the information that mainstream institutions make available.

CCSSE was not created by an institutional interest group, although the American Association of Community Colleges and the League for Innovation in the Community College are both represented on its advisory board. CCSSE's institutional participation agreement makes it clear that institutionally identifiable results will be publicly available on the CCSSE Web site (in marked contrast to the NSSE agreement, which assures institutional control over disclosure). Because public disclosure of CCSSE results is a condition of participation, we can conclude that leaders at a very large number of community colleges—755 to date—reached the cost-benefit decision described above.

Institutional Displays

At NSSE we have undertaken to review institutions' Web site displays of NSSE results, as well as results from two companion surveys, the Faculty Survey of Student Engagement and the Beginning College Survey of Student Engagement. Of 309 institutional Web sites examined to date, we have identified 111 with NSSE results online. Of these, 46 post some or all of their NSSE reports with little or no modification.

One of NSSE's reports, for instance, shows how an institution's students answer the questions in NSSE's “Pocket Guide to Choosing a College: Questions to Ask on your College Visits.” “The Student Experience in Brief” is designed to facilitate the inclusion of campus logos and images, and many participating institutions make this report available online. We have even found institutions that post students' open-ended comments—a true commitment to transparency, because these comments are not always complimentary. The remaining 65 institutions have developed and posted customized displays of NSSE results.

More sophisticated institutional reports present locally produced analyses of NSSE data, complete with charts and graphs. Some provide excellent illustrations of how campuses can make effective use of their results to inform local initiatives. In some cases, institutions post time-series results for specific survey questions, benchmarks, or scales to monitor and document change over time. Such customized displays show careful planning and analysis and that the institutions are committing time and effort to making sense of their results.

Customized displays also raise the possibility of cherry-picking the most favorable results, which is hardly surprising given the competition for students and pressures on institutions to show they deserve support by legislators, students, and parents. While we have found many examples of selective reporting, we have found as well examples of institutions that report both positive findings and areas where improvement is needed (see below).

Examples of Institutional Transparency Regarding NSSE Results

These examples illustrate a range of approaches to making an institution's NSSE results public, from posting comprehensive reports in PDF format to sophisticated and visually appealing Web pages. What they have in common is customization of information from NSSE reports and data files for the institution's audiences.

Adelphi University's Office of Research, Assessment and Planning posts reports for each year of NSSE participation. Recent reports relate NSSE results to the university's strategic plan, while others compare results from multiple administrations or examine a single year's administration in depth. administration.adelphi.edu/orap/research.php

Eckerd College provides an overview of NSSE and a history of Eckerd's participation in the survey. The site provides charts displaying NSSE benchmark scores relative to peers and response summaries for a number of survey items. www.eckerd.edu/nsse/

The Evergreen State College provides a number of comprehensive reports, including time-series comparisons of NSSE results. For each year of participation, both the detailed benchmark reports and short summaries are provided. The site also provides a comprehensive report on results from the Faculty Survey of Student Engagement (FSSE). www.evergreen.edu/institutionalresearch/nsse.htm

Mercer University's Office of Institutional Effectiveness Web site provides access to results from multiple surveys, including NSSE, FSSE, BCSSE, and other national and internal surveys. A dedicated NSSE site provides summary information on benchmarks and component survey items. www.mercer.edu/oie/assessment/testing.htm

Ohio University's Office of Institutional Research displays descriptive analyses, including tabulations by college and results over time, and contextualizes and interprets the results. www.ohio.edu/instres/involve/

Winthrop University provides a description of NSSE, information about students who participated in the survey, and information about benchmark-related and selected other survey items.

www.winthrop.edu/effectiveness/nsse/

Wheaton College (MA) displays its benchmark results compared to selected peers and the entire NSSE cohort, accompanied by a list of survey items that make up each benchmark.

www.wheatoncollege.edu/nsse/

Not surprisingly, we have also found examples that make improper inferences or inaccurate statements about what the results mean. A fairly common error is to infer growth or value added by comparing first-year and senior results. On the surface this may seem reasonable, but there are several reasons why such a comparison is unwarranted.

• NSSE results are cross-sectional: They do not track the same individuals over time. Even if an analyst were to create a synthetic cohort by comparing first-year students and seniors from, say, spring 2007 and 2010 (not necessarily the same individuals but separated by an appropriate time lag), significant difficulties remain. The first-year cohort includes students who may transfer to another institution or drop out, as well as others who will not graduate in four years due to programmatic or curricular delays, stopping out, or part-time enrollment. Similarly, the senior cohort includes students whose first year was at a different institution and others with a prolonged time-to-degree.

• NSSE results describe educational processes and activities, as well as subjective assessments of the student experience. To treat this information as the equivalent of a test score with its expected stability characteristics misrepresents the nature of the information and possible reasons why results might change.

• The first-year and senior experiences are very different. For example, consider the effect that smaller class sizes in the senior year can have on the likelihood that students will participate in class discussions, make class presentations, or work on group projects with other students. To infer that these changes reflect changes in students is to ignore fundamental differences between the first-year and senior experience.

• Another common error is to declare that an institution is a “top 50 percent” or “top 10 percent” performer based on distribution information given in NSSE's institutional reports. What seems like a reasonable inference may in fact be incorrect due to technical adjustments that NSSE makes to reduce measurement error in defining the top-performing groups. (When high average scores are based on a small number of respondents, we may remove an institution from the top-performing groups due to large sampling errors and the resulting uncertainty associated with those scores.)

But despite these and other problems, online displays of NSSE results in the interest of transparency represent a significant step forward in enabling institutions to both proclaim their successes and in some cases to confront uncomfortable realities about the student experience.

So What Does It All Mean?

All of the transparency efforts reviewed here are voluntary efforts by the institutions involved. One likely consequence is a “Lake Wobegon” effect, in which the institutions that volunteer are those with the most positive results. This does appear to come into play in the USAT effort, but it's complicated by the fact that USAT displays scores on all 10 NSSE benchmarks (five for first-year students and five for seniors).

Recall that USAT presents institutions' benchmark scores alongside the mean for students attending institutions of the same basic Carnegie type. In 2009, 37 percent of eligible institutions opted in (up from about 25 percent when the project debuted in fall 2007). The proportion that had above-average scores on a given benchmark compared to other institutions in their basic Carnegie classification ranged from 54 to 71 percent. So a majority were indeed above average.

But only 15 percent were above the mean on all 10 benchmarks, meaning that in this version of Lake Wobegon, 85 percent are below average on at least one measure. Interestingly, 3 percent were below their Carnegie mean on all 10 benchmarks. (It's important to note that some differences are so slight as to be inconsequential.)

So this tells us that the institutional leaders who elect to participate are willing to go public with data that are not uniformly flattering. As noted earlier, feedback suggests that some may participate under duress, and it is certainly true that the annual invitation to participate generates a small number of complaints—some of which confuse USA TODAY with the U.S. News and World Report, the king of the domestic rankings.

Several of the most elite and selective public research universities have shunned participation in the VSA. Kelly and Aldeman (2010) speculate that these institutions opt out due to concerns about institutional autonomy to set standards for student learning (that is, they oppose ceding authority for choosing assessment tools that implicitly define educational goals and standards).

An alternative but not incompatible interpretation involves concern over an “emperor's-new-clothes” effect. Institutions at the top of the prestige ladder have little to gain by assessing and publicly reporting learning outcomes, and possibly much to lose. Similar reasons have been proposed to account for the absence of Ivy League institutions from the roster of NSSE participants (despite the important difference that NSSE does not publicly report institutional results).

A related concern involves the selection and representation of measures in transparency initiatives. The voluntary nature of initiatives such as VSA and TbD may affect the issue of sensitivity to variation among institutions discussed above. By eliding differences that may exist in an ordered response set (e.g., combining the percentages for “sometimes,” “often,” and “very often”), the numbers go up for all institutions while minimizing differences in degree between institutions. This may boost participation, but it risks making transparency take place through rose-colored glasses.

This is a time of great sensitivity about inter-institutional comparisons that rises in some cases to comparison phobia. This is understandable, given the many headaches, public-relations crises, and perverse behavioral responses inspired not just by the U.S. News rankings but also by the Princeton Review's party-school rankings, the new global rankings, and even the Carnegie classifications. It is therefore interesting to observe the relatively high degree of acceptance of comparison (but not explicit ranking) on student engagement results and even (in the case of VSA) learning outcomes.

CCSSE presents an interesting illustration of probable cultural differences between community colleges and bachelor's-granting institutions. Given the de facto public release of results that accompanies CCSSE participation—something that proved to be a deal-breaker for prospective NSSE participants when the project was getting started—it appears that community colleges are more comfortable with or accustomed to public scrutiny, even when it may reveal unflattering truths. A significant factor may be that community colleges do not compete in regional or national markets for students in the same way that four-year institutions do. Be that as it may, community colleges have taken the lead in demonstrating transparency and candor regarding what they do well and where improvement may be needed.

Some advocates of market accountability argue for compulsory public reporting of assessment information, with the ability to select, rank, and sort institutions according to performance measures: Free the data, they believe, and students will vote with their feet. Indeed, in 2007, the National Center for Education Statistics proposed to require the reporting of scores from national assessment projects such as CCSSE, NSSE, and selected learning outcomes assessments as part of its higher education data collection system, but the controversial plan was abandoned. In my analysis, the scenarios in which compulsory reporting would produce unintended and perverse consequences far outnumber the one that results in a world of rational actors armed with near-perfect information to guide their educational investments.

So far, NSSE, CCSSE, the Collegiate Learning Assessment, ACT's Collegiate Assessment of Academic Progress, and ETS's Proficiency Profile are largely voluntary projects that will remain so in the absence of major state or federal programs to both mandate and fund participation—which is not likely at a statehouse near you anytime soon. Now recall my earlier references to cost-benefit analysis. Under a regime of compulsory public reporting (but continued voluntary participation), institutional leaders will weigh the costs and benefits of participation, and those who expect low scores—the institutions most in need of diagnosis and improvement—will likely opt out to avoid the negative consequences of poor performance. Those with mid-range or better scores may continue until attrition from the previous cycle shifts their relative standing downward. And so it goes.

But suppose the mass attrition does not take place. That leads to another scenario, in which NSSE and CCSSE become high-stakes measures that make it easy to produce numeric rankings. Before long, the word will get out that better engagement results may mean more funding or perhaps a higher market value for one's degree. Overnight, the one absolute requirement for useful data from a survey—honest responses—is rendered suspect, if not completely unachievable, and a pair of useful assessment tools that many institutions had used to guide improvement efforts will wither away.

Two natural experiments involving this worrisome possibility are currently underway. CCSSE's public results have been used to rank community colleges, over the strong objections of project leadership. Meanwhile in Canada, Macleans magazine has received NSSE results under public-records laws and published rankings of Canadian universities on NSSE benchmarks. The impact of these uses on institutions and their students remains to be seen.

Now, let me propose two important ways that the transparency movement needs to be refocused. A significant challenge that lies ahead for transparency efforts, and indeed for the entire accountability movement, is the exclusive and potentially misleading focus on institution-level measures of central tendency.

The sooner we come to grips with the fact that variation in the student experience is far greater within institutions than between them and devise ways to represent and talk about this internal variability, the sooner we will focus on the real quality questions. In light of unacceptably low graduation rates and a new national focus on improving college completion, we need to shift much of the energy presently focused on comparing institutions to finding ways to improve the engagement and success of the least engaged students at every institution.

And finally, another imperative for accountability and transparency is to move from performance snapshots—point-estimates of student engagement and learning outcomes—to plans for improvement and results. Policymakers and the news media should be less concerned with where an institution falls in the performance distribution than with what the results signify and what is to be done about them. Rather than pressuring colleges and universities to disclose assessment scores, the emphasis should be on transparency regarding a different part of the assessment cycle: action plans and interventions, followed by careful evaluation of whether those interventions achieve the desired results.

Resources

1. Burke, J. C. (ed) (2004) Achieving accountability in higher education: Balancing public, academic, and market demands, Jossey-Bass, San Francisco, CA.

2. College Choices for Adults [Transparency by Design]. www.collegechoicesforadults.org

3. College Portraits [Voluntary System of Accountability]. www.collegeportraits.org

4. Community College Survey of Student Engagement. www.ccsse.org

5. Ewell, P. T. (2009) Assessment, accountability, and improvement: Revisiting the tension, Occasional paper #1. National Institute for Learning Outcomes Assessment. Retrieved from www.learningoutcomesassessment.org

6. Kelly, A. P. and Aldeman, C. (2010) False fronts? Behind higher education's voluntary accountability systems, American Enterprise Institute and Education Sector, Washington, DC.

7. Kuh, G. D. (2007) Risky business: Promises and pitfalls of institutional transparency. Change 39:5, pp. 30-35.

8. Kuh, G. D., Kinzie, J., Schuh, J. H. and Whitt, E. J. (2005) Student success in college: Creating conditions that matter, Jossey-Bass, San Francisco, CA.

9. National Commission on the Future of Higher Education (2006) A test of leadership: Charting the future of higher education, U.S. Department of Education, Washington, DC.

10. National Survey of Student Engagement (2008) Promoting engagement for all students: The imperative to look within, Author, Bloomington, IN. 2008 results.

11. National Survey of Student Engagement. (N.D.) A pocket guide to choosing a college: Questions to ask on your college visits, Retrieved from nsse.iub.edu/html/pocket_guide_intro.cfm

12. Pascarella, E. T. and Terenzini., P. T. (1991) How college affects students, Jossey-Bass, San Francisco, CA.

13. Survey of Entering Student Engagement. www.enteringstudent.org

14. USA TODAY database of NSSE benchmark scores: www.usatoday.com/news/education/nsse.htm

15. CCSSE and NSSE statements on public reporting and responsible use of student engagement data: www.ccsse.org/datapopup.html www.nsse.iub.edu/links/positions_policies

16. For some examples of institutional Web sites that display NSSE results: nsse.iub.edu/links/website_displays.

Alexander C. McCormick is an associate professor of education at Indiana University Bloomington, where he directs the National Survey of Student Engagement (NSSE). He also serves on the advisory board for the Community College Survey of Student Engagement (CCSSE). Previously, McCormick was a senior scholar at The Carnegie Foundation for the Advancement of Teaching, where he led the revision of Carnegie's widely used classification of US colleges and universities.

Read Comments  |   Submit Your Comment

Become a Subscriber   |   Access for Current Subscribers