Innovative Approaches to Adapting VALUE Rubrics to Foster Learner Feedback in High Impact Practices

Introduction

Providing clear feedback is a known contributor to positively impacting student achievement. Yet, finding the right instruments and processes to give quality feedback in real-time can be daunting. Also, aligning this work to the mission, vision and institutional goals of a college or university is equally important as it enables assessment for learning to be an essential part of continuous improvement efforts as well as accreditation reporting. A launching point for consideration, are the American Association of Colleges and Universities (AAC&U) VALUE Rubrics. This case study provides insight into how two institutions adapted VALUE Rubrics as a way to delineate relevant criteria for use in assessing high impact practices (HIPs) and reflective processes in multiple disciplines and across platforms.

What are the AAC&U VALUE Rubrics?

As far back as the 1930s, John Dewey’s research established that reflection is connected to student education (Dewey, J., 1938). As such, reflective practice has been a mainstay in classrooms across the country; embedded into coursework and experiential activities for learners of all ages and disciplines. The process of reflecting on one’s own work, one’s own experience transforms learning, and can be linked to a number of general education and subject-specific outcomes and skills for lifelong learning (Weber, K., & Myrick, K., 2018).

In 2009, AAC&U developed and led an assessment approach called VALUE (Valid Assessment of Learning in Undergraduate Education). This approach included a bank of 16 rubrics to enable institutions and students themselves to assess work products/assignments across various fields of study and diverse pathways within postsecondary education. VALUE rubrics enable raters to determine whether and how well students are meeting learning outcomes for program completion and graduation linked to skills that both employers and faculty consider essential. Institutions across the country have been using VALUE rubrics or adapted versions of these 16 rubrics for various assessment initiatives, including co-curricular activities. According to AAC&U, as of 2015, 5,895 unique institutions, including more than 2,188 colleges and universities employ the rubrics (AACU.org/value)

The VALUE rubrics include Inquiry and Analysis, Critical Thinking, Creative Thinking, Written Communication, Oral Communication, Quantitative Literacy, Information Literacy, Reading, Teamwork, Problem Solving, Civic Knowledge and Engagement—Local and Global, Intercultural Knowledge and Competence, Ethical Reasoning and Action, Global Learning, Foundations and Skills for Lifelong Learning, and Integrative Learning.

High Impact Practices

High Impact Practices (HIPs) was coined by Dr. George Kuh in the National Survey of Student Engagement (NSSE) annual report 15 years ago as a way of organizing common experiential activities such as internships, capstone courses/projects, and undergraduate research (Zilvinskis, 2017; Kuh, O’Donnell, & Geary- Schneider, 2017). Since then, researchers have conducted multiple studies into HIPs, highlighting their positive impact on student learning. Suchland C.E. & Carbonaro, S. (2021) made the connection of HIPs with Social Cognitive Theory (SCT) as a methodology that included undergraduate embedded high impact assignments, reflective practices, and a pre/post self-efficacy survey, to provide learners with a more comprehensive practice which they could take with them into future coursework and employment.

Lincoln Land Community College (LLCC)

Lincoln Land Community College (LLCC) is located in Springfield, IL, and enrolls more than 5,000 full-time learners annually. LLCC serves its stakeholders by providing a wide range of degree programs and certificates. Its Honors Program, which is scholarship-based and well-sought by area high school students, offers a rigorous general education curriculum oriented toward transfer students. In alignment with Marzano (1998) research on instruction which recommends giving students clear targets and strategies to meet expectations of knowledge and skills required of them during learning processes, LLCC has employed HIPs in its Honors Program linked to a set of HIPs outcomes that are inspired by AAC&U VALUE Rubrics using AEFIS assessment management platform.

Thomas Jefferson University (TJU)

Thomas Jefferson University is a comprehensive professional university located in Philadelphia, PA, enrolling more than 8,000 students in 120 undergraduate through postdoctoral programs. The Design Essentials and Integrative Design Process(IDP) courses have looked across a variety of AAC&U VALUE rubrics to develop comprehensive analytic rubrics to create performance task assessments for qualitative practices that are both norm-referenced and criterion-based. Using carefully crafted language designed to apply to objectives and competencies, these rubrics align easily across program outcomes to Institutional Learning Goals. The Design Essentials rubric focuses on a balance between critical thinking and problem solving, creativity, and craftsmanship, encouraging students to take risks and push boundaries while the IDP rubric centers around the development of collaboration and integrative thinking skills.

Embedding VALUE into Reflection

Helping students understand for themselves who they are learning for is a critical component of self-reflection and growth. Motivating students to engage in deep reflection to identify strengths and areas for growth can be achieved by embedding self-reflection right where the learning is happening—inside the classroom as ancillary assignments. Giving students time and space to reflect in real-time can help students become advocates for their own learning, and to better ascertain their strengths and areas for growth.

With soft skills being reclassified as “Essential Skills” which employers are looking for in their employees, enabling students to practice metacognition as part of their preparatory course and extending into co-curricular experiences is just good teaching and learning assessment. Navigating ambiguity, practicing problem-solving, identifying the best solutions to problems through practice, and working collaboratively with others are key skills that need to be unpacked within subject-specific coursework as well as the general education curriculum.

LLCC Approach

Students enrolled in the Honors Program at Lincoln Land Community College work as a cohort through an honors curriculum that is guided by select faculty advisors to produce a rigorous undergraduate general education experience that includes community service and leadership training. Enrollment in the program is competitive and limited and honors students receive a scholarship as well as additional academic and professional advising from the LLCC honors faculty and staff. Courses within the honors curriculum reinforce the goals of the Honors Program by providing smaller class sizes, greater student engagement and participation, and opportunities to engage with the community at large. Thus courses within the program are an ideal site for instructors to deploy and assess High Impact Practices (HIPs). With this curricular foundation students enrolled in an honors section of Introduction to Sociology (SOC 101) were able to complete a community service project as their course final in Spring Semester 2020 (even in the face of the COVID-19 pandemic). This pilot project necessitated the development of a unique assignment and assessment process along with related assessment tools, including an assessment rubric for High Impact Practices (Suchland, C.E. & Carbonaro, S., 2021).

The LLCC General Education Program – of which SOC 101 is a part – makes use of several VALUE Rubrics for course and program assessment: Critical Thinking, Cultural and Global Awareness, Written Communication, etc. Therefore creating a VALUE-styled rubric for HIPs was as simple as identifying the relevant performance indicators (PIs) and assigning levels of performance in a manner similar to the VALUE Rubrics already used by the LLCC faculty. The chosen HIPs PIs were: Professional Learning, Civic Engagement, Collaborative Skills, Communication, Cultural and Social Awareness, and Academic Research (Figure 1). Levels of performance start with Benchmark, which is understood to be the expected level of a first-year college student, and proceed through two Milestones ending with the Capstone level. Thus students are scored on a 1-4 point scale, with a score of 0 reserved for missing data/evidence.

The community service project selected for the SOC 101 honors students was to develop a partnership with a non-profit or public institution in the greater Springfield community. Students first profiled their chosen institutions, then they identified the types of communication and specific messages being communicated, and finally, the students worked with their partners to facilitate a media/outreach campaign of some kind. Among the chosen institutions in the final project were churches, libraries, schools, and museums. Students developed a host of outreach from social media campaigns to roadside placards to pamphlets and mailers.

Assessment of the final project and delivery of feedback to students was conducted using the Canvas learning management system linked to the AEFIS assessment platform. Students submitted artifacts into Canvas that were then exported to AEFIS and scored with the High Impact Practices VALUE Rubric (Figure 1). The students also completed a pre-post survey related to their self-efficacy in a number of criteria, including Essential Skills and Sociological Thinking.

The ultimate goal for the SOC 101 project was to develop a scalable and portable tool for assessing HIPs. With honors courses resuming in Spring Semester 2022 – after COVID-19 forced a hiatus – a second iteration of the service project will be possible. Once proven, the VALUE rubric for HIPs will be available for use in other courses within the LLCC Honors Program curriculum. Ultimately this rubric can be used anywhere that HIPs are deployed.

TJU Approach

Faculty across Jefferson have looked to AAC&U VALUE rubrics to create meaningful assessments that effectively evaluate learning for both the students and faculty. For the Design Essential Course, in the School of Design and Engineering, a single rubric (Figure 2) is used for evaluation of all major projects. The rubric was not created based on a specific VALUE rubric, but instead, the structure and function of multiple VALUE rubrics were analyzed to develop a highly functional and flexible evaluation tool. The criteria of the rubric connect directly to the course learning outcomes, aligning each project with overall learning in the course. The rubric is also used by students for self and peer evaluations. This reinforced the connection between students’ own level of learning, in individual projects, to the overall outcomes of the course. These outcomes link directly to a variety of programs, and up to the Institutional Learning Goals which are linked in the curriculum mapping solution within AEFIS assessment platform.

The projects are all open-ended but have very specific parameters. 80% of the rubric is balanced between Project Goals (30%) (problem-solving and critical thinking), Creativity (25%), and Craftsmanship (25%). This encourages multiple iterations and allows the students to step out of their comfort zone and take risks (Creativity), while still solving an open-ended problem (Project Goals/Parameters) in a professional manner (Craftsmanship).

The remainder of the rubric evaluates Research (10%), supporting both Project Goals and Creativity, as well as Productivity (10%), supporting Craftsmanship, time management, and studio culture. An additional line of the rubric is devoted to Growth. This criterion is evaluated at the same levels as the other criteria, but is not calculated into the overall “score”. It adds a metacognitive element, prompting the students to evaluate their own, overall learning, or growth, on the project. Students who are very familiar with a concept or technique may produce excellent work but may score fairly low in Growth if they are simply repeating previous experience-based knowledge. Students who are unfamiliar with a technique or concept may have struggled with the project, but will likely score high in Growth. This pushes students to go beyond their comfort zone, and take risks, as well as reinforces the concept of learning from “failures”. The broad variety of AAC&U VALUE rubrics offer resources for multiple aspects of evaluation; from specific criteria, or language in descriptors, to the general structure and methods of evaluation.

Practices for Engaging Students in Metacognitive Learning

Including simple metacognitive elements, such as minute papers, quick-listing (what I know/don’t know), short reflections to projects and/or evaluations, and SMART Goal worksheets/inventories, reinforces student learning and supplements evaluation. As mentioned above, regarding the Growth criterion of the Design Essentials rubric, students are asked to reflect on their learning as a way to encourage them to push themselves further and to understand that developing new knowledge often comes by “learning from mistakes.” The self-evaluation rubric is accompanied by short reflection questions that vary depending on the project. These can be as simple as, “What did you learn?” to “If you could go back and do this again, what would you change about your process?” These are framed to help students understand their process of learning as a mechanism to improve it. They are also used to supplement the evaluation of their projects and overall course assessment. This becomes particularly valuable when courses are moved to an online format, or even when considering “homework”. The understanding of how students feel about their learning process adds an extra, indirect assessment that can clarify and support assignment rubrics, and overall course assessment.

This process is equally as valuable when considering team-based learning. The use of self-reflection exercises throughout the collaborative process promotes metacognition, reinforces learning, and can improve the overall experience of working in a group (Blythe, H., Sweet, C., & Carpenter, R., 2016). All teams go through a transitional process before effectively working together, and cannot be expected to immediately jump into collaboration. This is commonly referred to as Form, Storm, Norm, Perform, (Tuckman 1965). Carol Hobaugh (1997) describes the latter three steps of the teamwork process as Competition, Connection, and Collaboration. Integrating self-evaluation and self-reflection into these latter steps will assist teams as they maneuver through potentially difficult situations.

Adding self-reflective prompts to group-based evaluations, such as the Teamwork VALUE rubric, or as seen in the example below, helps students to contextualize the experience. For the student, this practice helps them move from more emotion-based responses to a more holistic view of team performance. For the instructor, these reflections can reinforce what they are seeing in the classroom and can be particularly clarifying for work done outside of class or virtually. Intentionally including these opportunities to practice the skill of metacognition is critical for lifelong learning and continuous improvement (Hickey, D., 2021).

Initial reflective questions to help contextualize the group experience, followed by an example of the evaluation form (same for self-eval and peer-eval):

1. Briefly answer the following (answers are confidential).
   a. In your experience so far, please state:
   b. Something that you learned from the group
   c. Something the other group members learned from you
   d. Your greatest contribution to the process
   e. One thing that you would change to improve performance
   
2. On the following pages, accurately evaluate yourself, and each of your team members. Think about each of the 7 criteria, and honestly rate both your own teamwork and that of your team members. The ratings, and any comments that you may provide, are confidential.

A Work in Progress

As both LLCC and Jefferson continue their work to help improve students’ reflective practices as well as their discipline-specific skills, they looking forward to moving this work into AEFIS to make the process more accessible to students and faculty, reportable for continuous improvement plans and accreditation and shareable with other stakeholders including employers.

AEFIS is an Education Technology (EdTech) company that partners with Higher Education to improve learner success and empower authentic lifelong learning. AEFIS is committed to empowering institutions to demonstrate educational impact through a unified platform for assessment management, learner success, and continuous improvement to help activate authentic lifelong learning.

Dr. Colin Suchland is leading efforts to embed VALUE rubrics beyond General Education requirements into HIPs across honors courses at LLCC. Dana Scott is working with assessment partners across Jefferson, using VALUE rubrics as a key resource for meaningful and effective assessment practice.

References

American Association of Colleges & Universities (AAC&U) What is value? Retrieved from: ​​https://www.aacu.org/value 

Blythe, H., Sweet, C., & Carpenter, R. It Works for Me, Metacognitively: Shared Tips for Effective Teaching, 2016, pp.106–110. New Forums Press, Inc., Stillwater, OK, 2016,

Carbonaro, S. (2020). Comprehensive learner record: Exploring a new transcript for lifelong learning. IMS Global Learning Consortium edTech rEvolutions leaders Retrieved from: https://www.imsglobal.org/article/edtech-leaders/clr

Dennen, V. P. & Burner, K. J.  (2007).  Instructional strategies and models associated with cognitive apprenticeship. 

Handbook of Research on Educational Communications and Technology, (pp. 425-439).  Retrieved from: https://pdfs.semanticscholar.org/ecc1/b2df2d37f995739986739397829f7e7ff4d9.pdf 

Hickey, D. (2021) Call for Curriculum Reform: Metacognition is Needed! AEFIS Academy Community Blog, Retrieved from: https://www.aefisacademy.org/call-for-curriculum-reform-metacognition-is-needed/

Hobaugh, C. F. (n.d.). Interactive Strategies for Collaborative Learning. 13th Annual Conference on Distance Teaching and Learning. Retrieved from http://files.eric.ed.gov/fulltext/ED413870.pdf#page=120

Kuh, G., O’Donnell, K., & Geary-Schneider, C. (2017). HIPs at ten. Change: TheMagazine of Higher Learning, 49:5, 8-16,

Marzano, R. (1998) A theory-based meta-analysis of research on instruction. The Office of Education Research and Improvement, Department of Education.

Suchland, C.E., & Carbonaro, S. (2021).  Connecting High-Impact Practices & Student Self-Efficacy: Social cognitive theory as a window into student growth.  Intersection: A Journal at the Intersection of Assessment And Learning, 2(2).  Retrieved from https://aalhe.scholasticahq.com/article/21271-connecting-high-impact-practices-hips-and-student-self-efficacy-social-cognitive-theory-as-a-window-into-student-growth

Tuckman, B. W. (1965). Developmental sequence in small groups. Psychological Bulletin, 63(6), 384-399.

Weber, K., & Myrick, K. (2018). Reflecting on Reflecting: Summer Undergraduate Research Students’ Experiences in Developing Electronic Portfolios, a Meta-High Impact Practice. International Journal of EPortfolio, 8(1), 13–25.

Zilvinskis, J. (2019). Measuring quality in high-impact practices. Higher Education: The International Journal of Higher Education Research, 78(4), 687–709.