Resize Text: 
The VSA’s chief product is a common online reporting template, called the College Portrait, which provides “consumer information” (the price of attendance, degree offerings, living arrangements, student characteristics, graduation rates, transfer rates, and post-graduate plans), student experiences and perceptions, and learning outcomes for those colleges and universities that volunteer to participate in the project. As of June 2008, 250 public universities had become VSA participants—about 45 percent of the unduplicated AASCU and NASULGC membership—and the number of participants continues to grow.
Although the VSA was under development before the Spellings Commission on the Future of Higher Education wrote its final report, to many observers it represents the most forthright and constructive response that the academic community has had to the commission’s criticism of colleges and universities for failing to provide the public and policymakers with transparent and comparable information on everything from college costs to learning outcomes.
For more information about the VSA and a look at the College Portrait reporting template, go to http://www.voluntarysystem.org/index.cfm.
—Margaret Miller
Miller: Let’s begin by having you describe what you say when you give the “elevator speech” about the VSA.
Mehaffy: “This is a wonderful opportunity for American higher education to control its own destiny and to talk about the critical parts of our work that the public doesn’t understand.” What I try very hard not to say is, “Well, they made us do it” or “We’re trying to keep the government out of our affairs.”
Many in the public are concerned that American higher education is too expensive or that it’s indifferent to students and their success. We want to show that many of the things we care about are in fact congruent with their concerns: cost, access, success, learning outcomes—a whole set of issues that I think the public doesn’t know that we care about.
Miller: So how did the VSA get started?
Shulenberger: In February 2008, about 80 people—university presidents, provosts, and so on—began to meet in organized taskforces to wrestle with the question of what the public wants to know about what we do. We reviewed findings of American Council on Education-conducted focus groups of prospective students and their parents and looked at the literature. This was the first real conversation many of us had had about the meaningful data we might be willing to share with the public.
What Parents and Students Want to Know
Miller: You’ve said that students and parents are your primary audience for this. When you had those focus groups, what did they say to you about what they wanted to know?
Shulenberger: They wanted to know how much college was going to cost them. This was the only area in which we found lots of frustration; it was not so much over the level of costs but over the uncertainty of how much the cost would be. They felt that the literature most of us gave them was deceptive—that the bills never resembled what they thought they were going to have to pay. They were also concerned about how they or their children were going to fit into this institution; they wanted to know something about the other students.
Miller: That’s why the students’ characteristics section was developed, I take it.
Shulenberger: Yes. Parents and students wanted to know whether the students were going to be successful.
Miller: What did they mean by “successful”?
Shulenberger: Well, am I or is my child going to get a degree, or is the school just going to take our money? The success and progress-rate information grew out of an effort to focus on the experience the student might have of success or failure.
Miller: Did they talk at all about what happens after graduation—that is, employment or further study?
Shulenberger: Yes, they really wanted to know about that. That’s one disappointment we have. We just don’t have any good and consistent data on post-graduation activity. Individual schools have some data from alumni surveys, but we can’t assume that a representative sample of graduates would be willing to share in an unbiased manner how much money they make or what they do. We did ask institutions to ask their seniors, “What are you going to do the year after graduation?” But we can’t verify that they actually do what they plan.
What Employers Want to Know
Miller: Did you talk at all to employers about what they’re looking for or are concerned about with regard to the college graduates they hire?
Shulenberger: We didn’t; instead we relied on the many existing employer surveys. Certainly the cognitive skills that the learning-outcomes section of the report focuses on—critical thinking (including analytic reasoning) and written communication—are among the skills that employers say they want. But we didn’t find a satisfactory instrument with which to assess the soft skills that are important to them—such as the ability to work in teams.
Mehaffy: Recently, the press reported that at the annual meeting of the AAC&U [American Association of Colleges and Universities], business leaders had expressed unhappiness with single-measure tests, echoing some of the concerns of faculty and assessment officers in our institutions. And yet I suspect that most of those leaders, if they actually looked at something like the CLA [the Collegiate Learning Assessment, which, along with the College Assessment of Academic Proficiency or the Measure of Academic Proficiency and Progress, is one of the three tests approved for use in the learning-outcomes section of the College Portrait], would say, “Well, that makes a lot of sense.” I think they’re responding to an old image of what tests are and what they do.
Miller: I’ve found that among policymakers, the tendency is to equate “standardized” with “multiple choice.” They don’t know about new developments in task-based testing. Speaking of policymakers, are they another audience for this work?
What Policymakers Want to Know
Mehaffy: One audience we had in mind was certainly federal and state governments, which are concerned with issues of accountability and transparency. The VSA is clearly a response to that concern. It’s far better for us as a higher-education community to decide what our outcomes should be and to report on them than to leave that task to a government entity, sometimes one with a particular political agenda. The VSA is also a better response than our sometime knee-jerk reaction to governmental pressure, insisting that we don’t want government involved in what we do—that we need to be free and independent agents.
Miller: Does the government have a right to know what the outcomes are?
Mehaffy: We are public institutions with a public obligation to be open and transparent about both our work and our outcomes.
What Institutions Want to Know
Mehaffy: The third—and in some respects the most important—audience is institutions themselves. This is a chance for us to think carefully about our work—about what we do and represent. We are in fact more than just an aggregation of bright people who occupy the same buildings.
Miller: ...and share a concern over parking spaces.
Mehaffy: We have to think of ourselves collectively as institutions with outcomes that are either intentional or unintentional.
Miller: So this is an opportunity to step back from the atomization that’s characterized the academy over the past several decades.
Mehaffy: A huge step back from that and from an exclusive focus on courses and programs. The VSA is not a substitute for but a complement to that attention. The narrow focus on courses and programs obscures our view. We can delude ourselves by saying, “Hey, my students are doing just fine.” And that may be true—or at least it may be true that my successful students are doing just fine. But we also have to decide whether we’re satisfied, as a community of professionals, with significant numbers of our students not graduating in a timely fashion or not learning very much in the years they spend with us.
Miller: Right. And with learning outcomes, the question is, how do you know when you should be satisfied? So can you talk about the role that comparability plays in addressing that issue?
Shulenberger: Because of the palpable fear of No Child Left Behind, our members didn’t all want to be measured by the same test. At the same time, the taskforces decided they wanted some comparability so participating universities could get some idea of how they were doing compared to how well they could be doing. So the developers of the three tests selected have each agreed to express their results not just as raw scores but in terms of whether the scores are above, below, or at the level that would be expected, given the academic preparedness of the students who enter a given institution. So we will have comparability within very broad ranges, but it’s not information one could use to rank schools.
Miller: That reminds me of Sir John Daniels’ comment about elite education: “‘Good little piggies coming in makes good bacon coming out’ is a counsel of despair for educators.”
Mehaffy: I think there’s a huge tension in this work between assessment that measures how far students have come and assessment that measures the absolute level they’ve reached. I can hear some institutions sputter, “After four years of college, they’re only performing at the thirteenth-grade level?!” I understand that very legitimate concern. On the other hand, I can hear others saying, “Is there some societal good in moving them from the ninth- to the thirteenth-grade level?” And my answer is, “Absolutely.” So you can be satisfied with how far you’ve brought your students but not with the absolute level of their performance, or vice versa.
Miller: Information about absolute achievement may well make a sobering point: the little we know about what American college graduates know and can do from the National Assessment of Adult Literacy [NAAL] is that the level of their learning is nowhere near where we’d like it to be or where it needs to be.
Shulenberger: And the questions of whether the scores are good enough and what it means to have a college education haven’t been debated in the American academy. You can’t set standards until you come to a consensus on those questions. Then you could set them, level by level and site by site.
Miller: One of the things that’s going to be interesting is to see whether the elite institutions do as well as some of the others on the value-added criteria. They may not show as much in the way of learning gains, although their absolute standard of performance might be higher.
Mehaffy: The value-added question illustrates my earlier point about relative versus absolute gains. In the University of Texas system, for example, the institution that adds the greatest value for its students is UT-Permian Basin. But that measure only gauges how much higher they scored than expected and doesn’t compare the performance of those students with that of students at other institutions.
Fortunately, the VSA allows two kinds of analysis, both of which are critical for institutions that are serious about learning outcomes. The instruments that the VSA offers to measure learning outcomes each provide both gain scores and average institutional scores, allowing institutions to compare how well they do against predictions of performance and against student scores on other campuses. The result is a rich set of materials for institutional analysis.
Miller: It might be interesting to see what happens over time as this information accumulates.
Mehaffy: Absolutely. Because if over time you start asking those hard questions, and institutions get serious about improving learning outcomes for all students, then a lot of changes might occur as we investigate how best to create optimal conditions for learning.
Shulenberger: I do have to add that the CLA staff tells us that there is as much value added by elite institutions on average as by non-elite ones. But the elite institutions are very, very fearful and skeptical about that assertion. That’s one of the reasons we went to the four-year trial period before institutions must report their scores.
Reporting Learning Outcomes
Miller: Do you think that giving institutions four years to report on the learning outcomes, and then allowing them to administer the tests only once every three years, represents a loss of courage about the hard part of this initiative—the learning piece?
Shulenberger: Well, there’s considerable debate within the academy as to whether these tests really measure the right outcomes. There’s also disagreement as to whether it’s appropriate to measure the core outcomes across all students, as opposed to within a single discipline. Then there are those who argue that you can’t use the results of any of these tests to change the curriculum.
We need experience with the tests to address those concerns. If at the end of four years, participating universities have widely used these tests and conclude that they’re producing random noise or that they are unfair or of no value in improving the curriculum, then the VSA will need to reevaluate its use of them. That said, I think that some institutions or systems are going to publish their results immediately: the Cal State and Texas schools, for instance.
Miller: Given the fact that you’ve not just delayed the publication of learning results but made it voluntary, is there a potential for a Lake Woebegon effect? You wait to see what your results are and if they’re “below expected,” you just don’t report the data?
Shulenberger: That could happen. But if parents and students value the results of these learning measures, then they’re going to prefer schools that produce the best outcomes, and schools that don’t publish the results will be at a disadvantage. In a sense, the VSA sets up a market test, and schools will have to pay attention to the market.
Miller: Congress seems inclined, in the reauthorization of the Higher Education Act, to let colleges set their own learning goals and standards. Will this remove the pressure for transparency?
Mehaffy: Well, Congressional actions this year suggest that there’s not going to be less scrutiny under a Democratically controlled Congress or a Democratic administration. That’s partly because of growing international pressure; we live in a world in which two very large entities, the European Union and China, are moving rapidly in higher education.
Miller: It’s true that if you compare American college graduates’ performance on the NAAL to that of college graduates’ from other OECD countries, we don’t stack up so well—we rank 11th.
Shulenberger: I don’t think that federal pressure was a huge motivator to begin with, frankly. Both AASCU and NASULGC have been out there working as vigorously as we could to encourage Congress to insert into the Higher Education Act reauthorization language that leaves outcomes testing entirely in colleges’ and universities’ hands. As George said at the beginning, we want to keep the assessment of learning within our own control. We believe that the diversity of higher education would be threatened if there were some centralized measurement required by law or administrative fiat. Again, if students and parents value the data, they’re going to insist that institutions release it. And if the faculty and universities use the data to improve what they do, they’re going to want it too.
Mehaffy: Look at healthcare. The federal government recently published mortality and morbidity rates for cardiac care at hospitals across the country. And I’ve heard that Johns Hopkins was surprised to learn that they were in the middle of the pack, instead of leading it (where they assumed they were). Prior to that point, they didn’t have good comparative ways to measure how well they were doing.
Miller: Right. Well, professions are expected to be self-regulating—but, increasingly, transparently self-regulating.
Other Elements of the College Portrait
Miller: What other kinds of information does the College Portrait provide besides learning outcomes?
Mehaffy: One, it has a cost calculator that’s been developed by the University of Texas System, which provides much more accurate information to students and their parents about the likely real cost for families according to their specific circumstances. We also have a new and much better graduation-rate metric that tracks the students who are coming in and who leave, rather than just the native students.
Miller: We’re not doing so well on college completion as a nation. The proportion of the adult population with collegiate degrees has increased in the first world and emerging nations, so that today ten countries have degree attainment rates close to or above ours.
Mehaffy: As we’ve mentioned, we’re also going to ask students what they intend to do after graduation. Right now we know where some students are headed, particularly from the elite institutions, because we can track their progress into graduate school. But we don’t know what happens to the vast majority of students. Finally, the College Portrait will provide measures of engagement, again from a choice of instruments: the National Survey of Student Engagement [NSSE], the College Student Experiences Questionnaire [CSEQ], the College Senior Survey [CSS], or University of California Undergraduate Experiences Survey [UCUES], which have never been widely reported before.
Miller: How does the VSA address the issue of how colleges and universities contribute to the public good?
Mehaffy: The California State University (CSU) system has already included a page 6 addition (the College Portrait has five pages) which describes contributions to the public good—such things as information about degrees granted as a percentage of the CSU system and the state, bachelor’s degrees by ethnicity and in high demand areas, access and completion information such as the number of Pell Grant recipients enrolled and graduated, actual average net tuition paid, and average student debt load (for an example of a CSU campus page 6, see http://daf.csulb.edu/offices/univ_svcs/institutionalresearch/college_portrait/).
The Future
Miller: If at the end of ten years you looked at the VSA, what would make you feel that it’s been a success—that in this part of your careers you were really contributing substantially to the progress of American higher education?
Shulenberger: What I’d focus on would be the learning outcomes area. At the end of ten years, I’d hope that the academy would have reached some consensus about what we ought to be measuring, how we ought to be measuring it, and what set of tests are legitimate for measuring it. If we’ve done that, then the VSA should be widely used within public universities—and some of its elements in private universities as well. At the end of the day, if this information has value for those who produce it, the universities and their faculty, and for those who consume it, it’ll survive and grow and morph.
Miller: And you, George?
Mehaffy: Like Dave, I would be principally focused on learning outcomes. I would be pleased if in ten years assessing and reporting on learning outcomes at the institutional level were a matter of common practice. I would love to see that we had come a long way in developing new and innovative ways to measure learning outcomes. I’d also love to see enormous interest and focus on new research and substantial energy devoted to the teaching strategies and conditions that yield the greatest learning.
But most of all, I hope that in ten years we’ll see stronger, more successful institutions supporting a stronger, more successful nation. That would be the greatest thanks we could give to the people who labored so hard to create this system. Many creative and imaginative leaders stepped forward when we put out a call for participation in this voluntary system. The challenge posed by this proposed system was daunting: Why would higher-education leaders create a process to display information that might reflect badly on their own institutions? Yet over and over we heard the same response from these thoughtful and courageous participants: It’s the right thing to do. We won’t get better until we have a process for describing ourselves publicly, as well as for fairly comparing ourselves to others.
Miller: And that’s the way we protect the integrity of our profession.
Mehaffy: I keep going back to Robert Barr and John Tagg’s Change article from 1995 [“From Teaching to Learning: A New Paradigm for Undergraduate Education,” Change, November/December 1995].
Miller: Say something about that.
Mehaffy: The central thesis of that article is that for too long institutions have been organized for teaching and not for learning—that our chief concern has been how we create operations that are convenient for us rather than ones that produce outcomes for students. And it’s expressed at the most granular level by the faculty member who says, “I taught them, and if they didn’t learn, it’s their problem.” I will think we have made huge advances if at the end of ten years people aren’t saying, “If only I had good students,” because, as I’ve said over and over again, “You have the students you have.”
Miller: Or as Howard Fuller has said, “Parents don’t send us their worst children. They send us the only children they have.”
Mehaffy: So if the VSA helps advance the notion of “learning institutions,” then it will have made a huge contribution.
Shulenberger: The other thing that I hope is that the VSA changes the conversation about success in college—that the “success and progress” rates cause parents to realize that their sons or daughters have a 70 or 80 percent chance of getting a degree and help them worry less about whether that will happen within four years. Higher education is a system, as far as students are concerned. They go into it and swim around until most earn the degree.
Miller: You’ve said, George, that people choose to go to college for all kinds of reasons and that what they’re going to learn is probably 15th on the list. But I think it’s also true that some students coming out of high school are tormented by the thought that if they don’t go to one of the elite institutions, they’re doomed. One of the things this project could demonstrate to them is that they can learn what they need to know just about anywhere, as well as that some institutions (and not necessarily the elite ones) are better organized to help them do that than others.
Mehaffy: Yes. All of us in higher education become the neighborhood experts when our friends and neighbors start to think about colleges for their children. What I say to them is this: “Look, here’s the dirty little secret. Your child can get a great education at a lot of institutions, and what contributes to that isn’t simply the name on the gates. It includes many variables, both institutional and personal: fit and comfort, whether the institution provides for your child’s learning needs, and a variety of other factors, including personal maturity and motivation.”
Miller: And the experiences that they’ll have there.
Mehaffy: Right. The other thing I hope the VSA illuminates is cost. I have a friend, well-educated and sophisticated, who had a special fund for college for his two sons, almost $500,000. And he said to me, “I just don’t know if it’s possible to educate my boys with only $500,000.” I said to him, “Have you lost your mind? You couldn’t spend $500,000 if you tried, unless they went to one of the truly prestigious and elite places that neither one of your kids is eligible for anyway”—which he didn’t appreciate. But this exaggerated perception of the price of college is something that the VSA could change as well.
Miller: And with that, I think we’ll end. Thank you, Dave and George, for all your thoughtful work and for speaking with me about it today.