Resize Text: 
For an enterprise dedicated to truth, American higher education harbors a lot of myths. Frequently advanced as unexamined propositions about “the way things work,” these deeply held articles of faith within the academy unwittingly mark its core values and priorities and shape public perceptions about what is important in higher education. Meanwhile, they tend to obscure things that really are important and that we continue to ignore at our peril.
Many of the cause-and-effect relationships that we and the public believe in but are not true, as well as some that that we tend to forget about but that actually are, have to do with quality and effectiveness.
Resources and Outcomes
One of the most pervasive and unassailable myths about higher education has to do with the relationship between outcomes and money. Doesn’t it stand to reason that if you want to improve performance, you need to spend more? Well, there is a good deal of evidence that variations in spending have little to no effect on institutional or state higher education performance.
We at the (NCHEMS) got interested in the resources/outcomes relationship through a project funded by the Pew Charitable Trusts a couple of years ago (see Kelly and Jones in Resources). Our motivation to do this work was that policy discussions about higher education funding in most states don’t ever seem to address the overall adequacy of public institutional funding. Of course, it takes a certain level of resources to produce any outcomes at all. But the standard institutional answer to the question of “how much funding is really needed?” is typically “more” or “as much as our peers.” This dodges the question of how much funding is enough to underwrite the kinds and levels of performance that a state wants from its institutions.
Resources relative to performance. We looked at the funding that public institutions get in the form of state and local appropriations plus tuition and fees in relation to various measures of institutional effectiveness that might interest a state. Among the latter were undergraduate enrollments relative to the population of young high school graduates in the state (a measure of service volume), undergraduate degree productivity (the ratio between enrollments and degrees granted each year), graduate degree productivity, six-year graduation rates, and success in acquiring competitive research grants. We examined the relationship between different combinations of these effectiveness measures and funding levels for different sectors of public higher education, including research universities, four-year baccalaureate and master’s institutions, and community colleges.
What we found surprised us. Across both states and institutions, performance relative to funding varied substantially. At equivalent levels of funding, institutions or sectors frequently performed very differently. And given equivalent levels of performance, institutions or sectors frequently spent quite different amounts to get them.
For example, total revenues and expenditures in Colorado’s public research university sector in the year 2000 were about 60 percent of revenues and expenditures for similar institutions in North Carolina. Yet the overall performance of the two systems with respect to graduation and degree production rates—as well as their comparative ability to garner externally funded research dollars—were virtually indistinguishable.
Certainly there were differences in the ways research universities operate in the two states that partially explain these findings. For example, research universities in North Carolina tend to offer instruction in fields that are inherently more expensive than those in Colorado, and they spend more on student services and instructional support. But these operational distinctions by no means account for the enormous disparity in performance relative to funding in the two states. Besides, more spending on student support ought to have yielded better undergraduate degree completion rates in North Carolina.
Try this exercise for yourself by accessing the graduation-rate reporting tool at the College Results Online web page maintained by the Education Trust (www.collegeresults.org). This site contains baccalaureate degree completion rates for most of the nation’s four-year institutions, drawn from the Graduation Rate Survey (GRS) conducted by the National Center for Education Statistics (NCES) each year (computed for first-time full-time students after six years of enrollment). Select a group of roughly similar colleges and universities that spend about the same amount per student on instruction-related activities, and look at the differences in performance.
I did just this for public colleges and universities spending between $10,000 and $11,000 per student, which returned thirty institutions. Four of them had six-year graduation rates of over 70 percent for the class beginning in 1997. But four had rates of less than 35 percent. Again, scanning the lists reveals some expected differences between the two groups—principally with respect to urban/rural location and the racial/ethnic composition of their student bodies. But these differences don’t seem sufficient to explain why one group did twice as well as the other on an important performance measure at the same level of per-student expenditure.
At a different level of analysis, Carol Twigg’s work through the National Center for Academic Transformation (NCAT) makes the same point (see www.thencat.org). The Center has now worked with scores of institutions across many disciplines to redesign large-enrollment freshman courses—typically large lecture courses in general education subjects such as psychology, American government, and art history, or multi-section courses such as English composition or college algebra. The redesigns have been quite varied, but they usually involve reducing faculty contact time by replacing some of it with technology and peer mentorships and allowing students varied opportunities to actively practice their skills.
In the vast majority of cases, the redesigns have yielded better student learning and higher course completion rates at reduced costs. But because these results run against the grain of conventional wisdom, they have frequently met with skepticism or downright disbelief. Even many faculty and administrators who were directly involved in these redesign efforts at first insisted on seeing the project only as a thinly disguised way to save money at the expense of learning. The article of faith that cutting costs invariably yields diminished quality was too firmly entrenched to be overturned, even when the evidence was in.
Spending wisely. The key to making sense of these examples is that it is not how much is invested in a given collegiate experience that is important; it is instead how intentionally these resources are directed. This point was underlined most forcefully when George Kuh of the National Survey of Student Engagement (NSSE) asked NCHEMS to look at whether the spending levels and patterns of the 20 unusually effective institutions he selected for study in Indiana University’s Documenting Effective Educational Practices (DEEP) project—an initiative that resulted in a widely acclaimed book, Student Success in College (see Resources)—were any different from those at similar institutions across the country. The DEEP institutions were originally selected for study by Kuh and his team because they significantly outperformed their peers with respect to both retention and levels of student engagement as measured by the NSSE.
For each of the 20 institutions—which ranged from small private liberal arts colleges to large public research universities—we constructed a peer group of similar institutions using such factors as total enrollment, the proportion of graduate and part-time enrollment, degree mix, Carnegie classification, research expenditures, and admissions selectivity. We then compared instruction-related expenditures for each DEEP institution with those of its peers, as documented by the Department of Education’s Integrated Postsecondary Education Data System (IPEDS).
We found that DEEP institutions did not spend more money per student than their peers, but they did spend it differently. Most intriguingly, they spent a noticeably higher proportion of their available dollars on “academic support,” a category in IPEDS under which most institutions report resources dedicated to such things as faculty development, teaching and learning centers, and academic support staff such as tutors and counselors. These were precisely the activities that the DEEP research team noted as key contributors to a campus ethos devoted to student success when they visited these institutions. A year or so later, another research team replicated our study using a somewhat more statistically sophisticated methodology and found exactly the same thing (see Gansemer-Topf, Saunders, Schuh, and Shelley in Resources).
Selectivity and Learning
Another enduring myth for both academicians and the public is the presumed connection between how difficult a college is to get into and how much learning goes on inside it. Presumably parents and prospective students assume this connection when they battle for admission to one of “America’s Best Colleges” as ranked on such factors as selectivity, resources, and reputation by U.S. News and World Report. But faculty believe in it too, as is revealed by their demand, on virtually every college campus, to “give us better students.”
In this case, the presumed relationship is partially true: Higher levels of admissions selectivity are in fact associated with important college outcomes. More selective institutions tend to have higher graduation rates, and their graduates end up richer than those of their less selective counterparts. But these outcomes are overwhelmingly due to the fact that their entering students are more able and well-off in the first place. In contrast, when it comes to educational good practice or learning gain, there is very little relation.
George Kuh and Ernest Pascarella made this case convincingly in a Change article several years ago (September/October 2004). Pascarella used data from the National Study of Student Learning (NSSL), a federally funded longitudinal study involving eighteen diverse institutions and about 3300 students. Participants completed a variety of surveys about academic behaviors and also took the ACT Collegiate Assessment of Academic Proficiency (CAAP) examination, a test of general intellectual skills. Kuh used information from more than 76,000 freshman and seniors who completed the NSSE at 271 four-year institutions in the spring of 2002. The NSSE asks students to report on their educational experiences, such as active and collaborative learning or student-faculty contact, that previous research has demonstrated to be related to learning.
Both datasets revealed extraordinarily little relationship between institutional selectivity and educational good practices. Selectivity did appear to be modestly related to a few good practices, such as high expectations for student work. But after controlling for a range of other factors, it accounted for only about two percent of the variance in those few effective educational practices where a significant relationship emerged. And Kuh and Pascarella found that selectivity was negatively associated with other good practices, including the number of essay examinations given, instructor feedback to students, and having a supportive campus environment. As Pascarella put it, “If a selective institution makes some of those [educationally effective] practices more likely, it does so in rather minimal ways... . The student body selectivity of an institution may bear too great a burden as a signal for the impact or quality of the undergraduate education actually received” (p. 279).
As might be expected, given the high correlation between selectivity and per-student expenditures, similar findings emerge when the latter is used as a sorting criterion (see Gasemer-Topf, Saunders, Schuh, and Shelley in Resources). And the same is true of the relationship between good practices and institutional rankings by U.S. News and World Report (see Pike in Resources). There is very little likelihood that an institution placed in the U.S. News top 100 is doing more of the things that research has shown to be educationally effective than one ranked toward the middle.
Given these findings, it is not surprising that relationships between institutional characteristics such as overall spending and selectivity and how much students learn are also not very strong. In the NSSL, for example, Pascarella and his colleagues found an anticipated relationship between incoming student ability as measured by SAT and ACT and performance on the ACT CAAP. But these relationships almost completely disappeared after statistical controls on entering student ability were applied. Indeed, students at open-access community colleges frequently experienced more “value added” in terms of cognitive gain than students at far more prestigious places.
One of the reasons for these results is the fact that the experiences of individual students at any given institution vary so much. Within both the NSSE and the NSSL datasets, and indeed throughout the literature on college student learning and engagement, “within-institution” variation far exceeds the variation in experiences that occurs “between institutions.” This suggests a parallel conclusion to the studies about the relationship between spending and good practice noted earlier. Just as what institutions choose to invest in matters more for outcomes than how much they invest, what students choose to do with the learning opportunities provided to them in a given campus environment matters far more than differences between campus environments themselves. These are the relationships that really matter for improved learning. But they remain largely obscured by the prior beliefs that most people hold about collegiate quality.
Some Other Relationships That Do Matter
Balancing the myths about cause-and-effect relationships that are widely believed but do not turn out to be true are important additional cause-and-effect relationships about higher education’s effectiveness that are true but are not widely publicized. For example, much has been made recently about the shortfalls in baccalaureate degree production in the U.S. compared to other nations. In 2005, the U.S. ranked tenth among OECD countries in the proportion of young adults (aged 25-34) with a college degree, down from seventh only a year earlier and first in the world a decade ago. As some have rightly observed, this was not the result of a decline in U.S. performance but was instead a product of a rapid increase in college credentialing on the part of these other countries while U.S. rates remained flat.
Lying behind these statistics, however, are some disturbing findings about why the U.S. has not made more relative progress. Overall, U.S. baccalaureate attainment levels in 2005 were 86 percent of Norway’s—the best-performing OECD country. But U.S. attainment rates were only 40 percent and 60 percent of this “best-practice” rate for
African-American men and women and only 30 percent and 40 percent for Hispanic men and women. Attainment rates among white men, on the other hand, were nearly as high as Norway’s and for white women they were identical. These differences are massive and chilling. Although we all subliminally know that this is happening, the size of the gaps is mind-boggling, and public policy outcry about it is surprisingly rare.
Similar eye-opening numbers document the relationship between family income and access to college. Statistics quoted in Access Denied—a 2001 report of the Advisory Committee on Student Financial Assistance—show that 78 percent of students in the lowest family income quartile and the highest achievement quartile as measured on college admissions examinations had enrolled in postsecondary education within two years of high school graduation (p.13). An essentially equivalent 77 percent of students drawn from the highest income quartile but the lowest achievement quartile had done the same, while only 36 percent of their low-achievement counterparts in the lowest income quartile enrolled in college. Once again, one would think that the sheer size of these discrepancies would demand public and policy attention.
To be sure, some of these substantial cause-and-effect relationships have been explicitly cited and considered. The latter, for instance, provided prominent support for the major increases in need-based student financial assistance called for by the Spellings Commission in 2006 and partly enacted by Congress last year. And the majority of student financial assistance allocated by states and the federal government continues to be awarded on the basis of need. But a study by Don Heller at Penn State indicated that the amount of institutionally awarded merit-based aid increased 212 percent between 1995-96 and 2003-04 compared to an increase of only 47 percent for need-based aid. [Editor’s note: see Will Doyle’s “Playing the Numbers” feature in the September/October 2008 Change for a similar analysis.]
Although there are recent signals from elite institutions that opinions are beginning to change, most institutional leaders continue to pursue a mythical vision of quality that rests on a presumed relationship between outcomes and selectivity plus money. Prospective students and their parents share this vision and behave accordingly. But there is no such correlation.
Resources
Gansemer-Topf, Ann; Saunders, Kevin; Schuh, John; and Shelley, Mack (2004). A Study of Resource Expenditures and Allocation at DEEP Colleges. Ames, IA: Educational Leadership and Policy Studies, Iowa State University.
Kelly, Patrick J. and Jones, Dennis P. (2006). A new look at the institutional component of higher education finance: A guide for evaluating performance relative to financial resources. NCHEMS Newsletter, Volume 23, January 2006, pp. 2-7.
Kuh, George D.; Kinzie, Jillian; Schuh, John H.; Whitt, Elizabeth J.; and Associates (2005). Student Success in College: Creating Conditions that Matter. San Francisco: Jossey-Bass.
Kuh, George D. and Pascarella, Ernest T. (2004). What does institutional selectivity tell us about educational quality? Change, September/October 2004, pp. 52-58.
NCHEMS (2003). Do DEEP Institutions Spend More or Differently than Their Peers? Boulder, CO: National Center for Higher Education Management Systems (NCHEMS).
Pascarella, Ernest T.; Cruce, Ty; Umbach, Paul D.; Wolniak, Gregory C.; Kuh, George D.; Carini, Robert M.; Hayek, John C.; Gonyea, Robert M.; and Zhao, Chen-Mei (2006). Institutional selectivity and good practices: How strong is the link? Journal of Higher Education, March/April 2006, pp. 251-285.
Pascarella, Ernest T. and Terenzini, Patrick T. (2007). How College Affects Students (Volume 2): A Third Decade of Research. San Francisco: Jossey-Bass.
Pike, Gary (2003, May). Measuring quality: A comparison of U.S. News rankings and NSSE benchmarks. Paper presented at the annual meeting of the Association for Institutional Research, Tampa, FL.
Peter Ewell is vice president of the National Center for Higher Education Management Systems (NCHEMS), a research and development center founded to improve the management effectiveness of colleges and universities. He has consulted with over 375 colleges and universities and 24 state systems of higher education on issues that include assessment, program review, enrollment management, and student retention. He led the design team for the National Survey of Student Engagement (NSSE) and currently chairs its technical advisory panel. He has authored six books and numerous articles and papers on the topic of improving undergraduate instruction through the assessment of student learning outcomes.