We seem to be beginning a new wave of technology development in higher education. Freeing student work from paper and making it organized, searchable, and transportable opens enormous possibilities. … In short, ePortfolios might be the biggest thing in technology innovation on campus. Electronic portfolios have a greater potential to alter higher education at its very core than any other technology application we've known thus far.
Trent Batson, 2002
We are inundated with technology on our campuses and in our lives. Our students are increasingly technology savvy, expecting faculty and administrators to function comfortably within the digital world. And we have responded by using technology more and more in teaching and learning.
The following article focuses on one such use—student electronic portfolios, or e-portfolios—as a rapidly emerging, powerful, iterative mode for capturing student work and enabling faculty to assess student learning. Long before the advent of e-portfolios, collections of student work were a means by which students in the arts and architecture could demonstrate their learning and accomplishments (Zubizarreta, 2009). But technology has provided the means to do this more easily, in multiple modes, and portably. So the use of portfolios in electronic form has rapidly spread to other fields (most notably, teacher education) and has been taken up for other purposes (e.g., faculty keeping portfolios of their work for purposes of development and evaluation or institutions for accreditation). Forty percent of campuses of all types – large and small, public and private, research and liberal arts, and community colleges—recently reported using student e-portfolios.
This development has benefits that extend beyond the campus. In 2008, a survey of employers conducted by Peter S. Hart Research Associates asked about the evidence of student learning they would like to have when hiring college graduates. Thirty-five percent indicated that they would like to see the students' work in e-portfolios. Younger respondents were the most enthusiastic, suggesting that this type of evidence will be more in demand in the future.
Part of the current surge of interest in and use of e-portfolios flows from the decades-long requirement of regional and professional accrediting organizations for demonstrations that students are learning what faculty expect them to learn. Meanwhile the legislative and policy attention focused on measuring student learning that has existed since the mid-1980s has intensified in the wake of the Spellings Commission report in 2006. Portfolios have been one way to respond to both of those pressures.
An alternative response to the calls for measures of student learning has been to use existing national standardized tests of general intellectual skills. These tests focus on three primary learning outcomes: written communication, problem solving/analysis, and critical thinking. They are administered to samples of freshmen and seniors, typically in a timed environment. The results are summarized and available at an institutional level, but they are often not reliable at the individual student level and so are not reported to individual students. A key weakness of this kind of assessment strategy is that the tests are not high stakes, so there is neither an incentive for students to perform well nor a penalty for their not taking the exam seriously.
Moreover, faculty members express a need for demonstrations of a much broader array of learning outcomes than the existing tests address—e.g., personal and social responsibility, teamwork, intercultural knowledge and competence, and integrative learning—and for measures that will help them improve pedagogy and the classroom experience. Finally, already overburdened by the demands of increased enrollments and reduced resources, faculty don't see why what their students do in their classes and assignments is not sufficient for assessment and accountability purposes.
E-portfolios provide a means for collecting assigned work, as well as students' accomplishments in non-classroom settings, so that faculty, internship supervisors, and others can assess it and aggregate or disaggregate the results, depending on the purposes of the assessment. There is a need for some means of linking assessments of work done in individual classes with those done by other faculty and evaluators when they are for purposes of programmatic or institutional evaluation; this creates a concomitant need for shared expectations for student performance on a set of specified learning outcomes.
Rubrics for Learning and Assessment
From those needs, the Association of American Colleges and Universities' (AAC&U) Valid Assessment of Learning in Undergraduate Education (VALUE) project emerged. Over the course of the project, over 100 faculty and academic professionals from all types of higher education institutions across the country developed rubrics for the fifteen liberal learning outcomes previously identified in AAC&U's work. The rubrics went through three rounds of drafting, testing on over 100 campuses, and redrafting. The final versions were released in fall 2009.
Intellectual and Practical Skills
Personal and Social Responsibility
- Inquiry and analysis
- Critical thinking
- Creative thinking
- Written communication
- Oral communication
- Quantitative literacy
- Information literacy
- Problem solving
Integrative and Applied Learning
- Civic engagement
- Intercultural knowledge and competence
- Ethical reasoning
- Lifelong learning
- Integrative and applied learning
The VALUE rubrics were based on ones already developed by campus colleagues as they articulated their expectations for learning. For some outcomes there existed few rubrics or articulated expectations, and for others there were many. Although the specific language differed and the emphasis among criteria changed from college to college, the development teams found clearly shared criteria for learning at progressively more sophisticated levels of performance in each of the outcome areas.
In some instances the teams identified missing aspects that needed to be incorporated into the rubrics to make them more complete. For example, criteria tended to be text bound and did not reflect how students could demonstrate learning through visual, graphical, digital, or artistic modes of communication.
E-portfolios as a Medium for Learning
Technology is expanding the modes through which students learn and can demonstrate their learning. No longer are they restricted solely to the printed word but can do so in multiple modes: video, audio, in-the-field community projects, and graphics, for example. These multi-modal approaches are the ways students will need to work in the future: They may need to construct a variety of graphical data displays that effectively communicate with a range of audiences, or capture work in videos, or work in groups using social networking.
As Helen Chen has observed,
In contrast to transcripts, e-portfolios (with their added value of the “e” or electronic nature) allow students to gather in one place a range of digital artifacts that can be used to demonstrate presentation skills (e.g., a video of a presentation and accompanying slides), inquiry and analysis (e.g., a paper that includes instructor feedback and is annotated by the student to highlight key points), or intercultural knowledge (e.g., reflections on a term spent abroad illustrated with photos and reflections). The representations of learning in an e-portfolio reflect the individual student's view of the breadth of his or her education – including what was learned both inside and outside the classroom and as the learning was experienced by the student and not just as it was delivered or packaged by the college or professor. (Chen, June 3, 2010
George Kuh and his colleagues at the National Survey of Student Engagement (NSSE) have found that there are patterns of practice on our campuses associated with enhanced student learning, which he has named high-impact practices, or HIPs. When students engage in two or more of these practices, there is a significant positive impact on their grades and retention, especially for those who are least advantaged when they come to higher education, such as first-generation and minority students (Kuh, 2008). Although e-portfolios are not one of the HIPs included in the survey, there is emerging evidence that e-portfolios may be associated with some of them.
For example LaGuardia Community College, a member of the Inter/National Coalition for Electronic Portfolio Learning, has been comparing students in e-portfolios classes to those in non-e-portfolio sections of the same classes with respect to their engagement in HIPs. La Guardia serves many first-generation, limited-income, non-native-speaking students. They have found, from the Community College Survey of Student Engagement (CCSSE), that e-portfolio students have more of these learning experiences and higher pass rates than do their counterparts in courses that do not have e-portfolios as an integral part of the course (Eynon, 2007).
E-portfolios also require students to reflect on their learning, which is in itself a learning exercise. As Kathleen Blake Yancey has argued:
Collectively, Dewey, Vygotsky, and Polanyi define reflection as a process by which we think: reviewing, as we think about the products we create and the ends we produce, but also about the means we use to get to those ends; and projecting, as we plan for the learning we want to control and accordingly, manage, contextualize, understand. We learn to reflect as we learn to talk: in the company of others. To reflect, as to learn (since reflection is a kind of learning), we set a problem for ourselves, we try to conceptualize that problem from diverse perspectives – the scientific and the spontaneous – for it is in seeing something from divergent perspectives that we see it fully. Along the way, we check and confirm, as we seek to reach goals that we have set for ourselves. Reflection becomes a habit, one that transforms. (Yancey, 1998, pp. 11–12)
Reflection has not only become an essential way for students to speak in their own voices—it has also become a way for them to both learn and provide evidence of their capacity for critical thinking, analytic reasoning, and integrative learning. Reflection at strategic points in the development of the e-portfolio creates a venue for the iterative and formative examination and demonstration of learning and can play a summative role at key points in the assessment of student progress and achievement.
Institution: Clemson University
Purpose: Primarily for the assessment of general education. All students begin e-portfolios their first semester and add evidence each subsequent semester.
Who: The graduating class of 2010 successfully completed e-portfolios, and all students received feedback from mentors and faculty. Faculty evaluated students' e-portfolios at summer assessment workshops.
Results: Faculty members have begun to rethink their syllabi and course assignments based on the work that students are including in their e-portfolios. Substantial changes have been made to the general education program: The number of competencies has been reduced from 22 to eight, and communication has been embedded in all competencies. There are now data spanning four years of a student's education to share with accrediting agencies.
Institutions that have enjoyed success with e-portfolios—for example, Bowling Green State University, St. Olaf College, Kapi'olani Community College, Portland State University, and Westminster College—consider the students to be their owners. Faculty and the institution can designate software platforms for the e-portfolio and determine its structure and required content; however, students determine how it is used beyond the course, program, and institutional assessment needs of the faculty and the college. The ability of students to exercise their voices in presenting and representing their learning has been found to be a significant motivator for students to do their best work (Cambridge, Cambridge and Yancey, 2009).
Validity and Reliability
One of the arguments for not using e-portfolios or rubrics is that they lack the reliability and validity of the standardized tests and therefore cannot be used to compare institutions, courses, programs, etc. It is true that rubrics and e-portfolios do not rely upon standard psychometric calibration. Nor do they require such underpinnings. The VALUE rubrics arose through the efforts of experts across the country (and abroad in some instances) to define core elements of learning for each of the outcomes. The result was standards without standardization. The rubrics exhibit content validity because they have been broadly articulated and accepted by the faculty and academic professionals in each of the areas of learning.
Institution: Queensborough Community College
Purpose: To guide faculty and students in course teams through the various layers of integrative and reflective learning, beginning with students' own experience and moving through disciplinary knowledge to social networking, knowledge production, and reflection in the e-portfolio presentation.
Who: In 2007, the e-portfolio project collaboration brought together courses in three disciplines within the college. Twenty-three faculty from 11 disciplines participated in the pilot in eight teams of three. Four hundred fifty entering students are participating in the fall 2010 launch of the full project.
Assessment: Students in the freshman composition course write personal essays. Students in basic skills and content discipline courses read and offer perspectives on the essays through their research on the Internet. These students post web objects to the comp student's wiki, with a paragraph explaining the connection between the essay and the object. ESL students working on their language skills offer another cultural perspective on the essay, while students from the content discipline course add knowledge through a disciplinary lens. Faculty and student responses to the pilot were sufficiently positive to have QCC commit resources to offer the program to all students.
The process of developing the rubrics and testing them with student work on a variety of campuses provided multiple opportunities to discover whether the rubrics indeed reflected the core elements of learning that academics expected to see. This iterative process refined the rubrics, sharpened the language and the concepts, and engaged experts and non-experts in confirming that the content was representative of the learning sought.
The VALUE rubrics are criterion-referenced, and all students can be judged against the criteria. Yet the primary usefulness of the rubrics lies in their ability to communicate faculty expectations for student performance to others and to engage with the students in gauging their progress during a single program or along an entire educational pathway. Further, the adoption of the rubrics and their adaptations to specific campus missions and outcomes attests to their usability and meaningfulness.
One multi-year research project at the University of Cincinnati compared the Collegiate Learning Assessment (CLA), a national standardized test, with the VALUE rubrics and e-portfolios and found that the two methods had different advantages. As the Cincinnati project leaders concluded, “When we talk about standardized tests, we always need to investigate how realistic the results are, how they allow for drill-down. … The CLA provides scores at the institutional level. It doesn't give me a picture of how I can affect those specific students' learning. So that's where rubric assessment comes in—you can use it to look at data that's compiled over time” (Donnelly-Smith, 2010). In addition, several of the leading e-portfolio companies—TaskStream, Live Text, Epsilen, eLumen, Digication, and Sakai—have adopted the VALUE rubrics in their products because their users were either already using the rubrics to assess student e-portfolio work or wished to do so.
Institution: University of Michigan Medical School
Purpose: Resident trainees in the Department of Internal Medicine participate in an experiential patient safety learning program in order to engage students in the complex and critical issues involved in addressing adverse events that occur in patient-care settings.
Who: Resident physicians record, reflect upon, and analyze critical incidents that include adverse events or near misses, applying the conceptual models introduced in coursework to these cases. Residents receive feedback on their entries and repeat the cycle in structured exercises during their years of training. Learners have recorded over 400 critical incidents spanning the entire range of categories of adverse events.
Results: The program has resulted in an increased commitment to patient safety in both learners and the faculty and enhanced skills in the analysis of adverse events, and has had a measurable impact on specific patient-care processes stemming from the team-based implementation projects. A valid, reliable method of abstracting important information from the entries has been developed and has revealed evidence of improved reflective skills as a result of curricular and/or clinical experience.
Institution: Spelman College
Purpose: The SpelFolio (e-portfolio) in the first- and second-year seminar program and expanded in the senior capstone creates an authentic assessment not tied to one course that encourages interdisciplinarity and writing across the curriculum. Students must pass to graduate.
Who: All entering students. The scoring rubric was developed by the SpelFolio steering committee. Sample essays were submitted to each instructor and each was evaluated and discussed. A norming process created an objective, reliable assessment: External faculty reviewers assessed essays students had already submitted to class without knowing the students personally.
Results: Students, who are used to working in an online environment, like the SpelFolio system. Their submissions include pictures and music and express their personalities and creativity. Students sometimes have trouble understanding the purpose of the portfolio and multiple writing assignments judged through the application of a rubric that articulates progressively more sophisticated writing over time to demonstrate growth. Most students pass initially, but some resubmit. On average, four or five out of 25 resubmit the first essay. Students know what they need to do to improve from the responses to the rubric.
Near the end of the VALUE project, a national review panel of leading assessment experts, faculty, academic professionals, community representatives, teachers, and employers was convened for two and half days to review the draft rubrics and use them with actual student e-portfolios from a variety of campuses. None of the national review panel members had been involved with the VALUE project prior to that time.
Just as inter-rater reliability training was done for each of the campuses that tested the rubrics during their development, the review panel was similarly trained. They were then able to use the rubrics, apply them to e-portfolios from students at very different types of institutions, and do assessments of student performance and achievement with remarkable inter-rater reliability.
The experience of the national review panel and the process used for developing the rubrics has established two important points about the VALUE rubrics and e-portfolios. First, the rubrics reflect nationally shared expectations for student learning across fifteen essential learning outcomes deemed critical for student success by both higher education faculty and employers. Second, e-portfolios of student work and the rubrics by which they are assessed can be used by faculty and others external to the academy for assessing student work with high levels of reliability and validity.
The largest remaining challenge for external validity is developing effective ways to communicate the results of e-portfolio assessments to multiple audiences both on and off the campus, including parents, policy-makers, employers, and legislators. Washington State University has developed one promising method for doing this that has been used on the campus and with employer groups. WSU electronically records the rubric scores from reviewers (faculty, students, and employers) and then imports the results into an Excel radar or spider diagram that displays the scores on each criterion for each outcome, revealing areas of strength and weakness. The shape of the web or radar map provides a clear visual indication of the learning in each outcome area (at the level of the individual student, the class, the program, or the institution), as well as aspects of learning that may need more attention and areas of student mastery (see Harvesting Gradebook at Washington State University's Office of Assessment and Innovation, at https://my.wsu.edu/portal/page?_pageid=177,1&_dad=portal&_schema=PORTAL).
E-portfolios, Rubrics and the Future
It is clear that demands for assessment of student learning will not abate. Regional and professional accreditors will continue to insist that campuses share evidence of their students' knowledge, skills, and abilities. Meanwhile, employers will continue to call for higher levels of performance on a broader range of outcomes from our graduates, and they will want to see examples of how students apply their knowledge and use it to address complex, unscripted problems. Policy makers will continue to ask us to demonstrate that our graduates can compete globally and contribute to the nation's economic, social, and civic health.
The argument here is not that e-portfolios are the only adequate response to these many forces and challenges, but rather that e-portfolios and the shared expectations for learning embodied in the VALUE rubrics can provide direct, robust, and nuanced evidence of students' abilities to function in and shape the future in the many complex ways demanded by society. The capability to provide multi-dimensional evidence though e-portfolios and the desire of students to integrate their lives beyond the academy with their academic experiences are converging to transform how we measure and conceive of student success.
The President, Congress, legislative bodies, policy makers, and economic, civic, and educational organizations continue to call for better-educated college graduates. We need a citizenry prepared to communicate well, to understand science and technology, to think about the rapid changes propelling society in anticipated and unanticipated directions, to function successfully in a global environment with clashing values and interests, and to integrate and reinvent knowledge and skills frequently in the wake of constant change.
It is no longer the world of our parents and grandparents. We can no longer expect the completion metrics of going to and graduating from college to be adequate evidence that students will be prepared for their lives and careers. We can no longer provide the same curriculum taught by the same pedagogies or assess students using the same strategies designed for a privileged segment of the population a century or more ago.
1. Association of American Colleges and Universities (2008) College learning for the new global century, Washington, DC.
2. Barrett, H. Cambridge, B. (ed) (2001) Electronic portfolios = multimedia development + portfolio development: The electronic portfolio development process.. Electronic portfolios, pp. 110-116. American Association for Higher Education., Washington, DC.
3. Batson, T. (Nov. 26 2002) The electronic portfolio boom: What's it all about.. Campus Technology, Retrieved from http://campustechnology.com/articles/2002/11/the-electronic-portfolio-boom-whats-it-all-about.aspx?sc_lang=en
4. Cambridge, B. (ed) (2001) Electronic portfolios: Emerging practice in student, faculty, and institutional learning., American Association for Higher Education., Washington, DC.
5. Cambridge, D., Cambridge, B. and Yancey, K. (2009) Electronic portfolios 2.0: Emergent research on implementation and impact., Stylus Publishing., Sterling, VA.
6. Chen, H. (2010) The promise of e-portfolios to improve learning and help students, faculty, and employers.. Liberal Education blog, June 3, 2010, Association of American Colleges and Universities., Washington, DC.
7. Chen, H. and Light, T. P. (2010) Electronic portfolios and student success: Effectiveness, efficiency, and learning., Association of American Colleges and Universities., Washington, DC.
8. Commission on the Future of Higher Education (2006) A test of leadership: Charting the future of U.S. higher education., U.S. Department of Education., Washington, DC.
9. Donnelly-Smith, L. (April 2010) Assessing learning outcomes at the University of Cincinnati : Comparing rubric assessments to standardized tests.. AAC&U News, Association of American Colleges and Universities., Washington, DC.
10. Eynon, B. (2007) Report from LaGuardia Community College, Inter/National Coalition for Electronic Portfolio Research.Retrieved from http://ncepr.org/cohort1.htmlCohort I
11. Kuh, George (2008) High-impact educational practices: What they are, who has access to them, and why they matter, Association of American Colleges and Universities., Washington, DC.
12. National Survey of Student Engagement (2008) Promoting engagement for all students: The imperative to look within, Indiana Center for Postsecondary Research., Bloomington, IN.
13. Peter D. Hart Research Associates, Inc (2008) How should colleges assess and improve student learning?: Employers' views on the accountability challenge, Association of American Colleges and Universities., Washington, DC.
14. Peter Hart D. Research Associates, Inc (2009) Trends and emerging practices in general education, Association of American Colleges and Universities., Washington, DC.
15. Rhodes, T. (2010) Assessing outcomes and improving achievement: Tips and tools for using rubrics., Association of American Colleges and Universities., Washington, DC.
16. Yancey, K. B. (1998) Reflection in the writing classroom., Utah State University Press., Logan, UT.
17. Zubizaretta, J. (2009) The learning portfolio: Reflective practice for improving student learning, 2nd ed., Jossey-Bass., San Francisco, CA.
Terrel L. Rhodes is vice president for quality, curriculum, and assessment at the Association of American Colleges and Universities (AAC&U) and the director of the VALUE project (Valid Assessment of Learning in Undergraduate Education), which is focused on the quality of student learning and its assessment through rubrics and e-portfolios.