During the pandemic, assessment of student learning became even more important than usual as instructors sought tools to gauge student success in a remote, unexpected (and, for many instructors, new) educational environment. Prompted by Georgia Tech’s emergency shift to remote instruction, our team at the Center for 21st Century Universities (C21U) piloted a key performance indicator (KPI) tool designed to provide instructors with expanded insight into student learning and success in remote courses.
The KPI tool as a formative feedback tool
In 2020, we set out to build a tool to help instructors obtain spontaneous feedback from students that would allow educators to quickly adjust their teaching methods. We felt that the most direct way to provide this resource would be through a tool already in use by most of our faculty and staff: the campus learning management system (LMS), Canvas. We also envisaged a tool that could be quickly and effectively scaled to all courses at Georgia Tech without extensive effort by faculty.
- The potential of artificial intelligence in assessment feedback
- Online learning is still challenging for students – they need our support
- A step-by-step guide to implementing an agile curriculum
The KPI Canvas tool is available to all instructors at Georgia Tech who are teaching courses that are fully residential or part of online degree programmes. This new tool enables C21U researchers to connect to a database containing Qualtrics survey data and easily retrieve the data from the KPI learning tools interoperability (LTI) application. Ease of user access is key to the design of a new academic tool, and our KPI tool provides instructors with easily accessed and continuous weekly feedback from students. It also tracks how students are performing on average in their course over time.
When installed, the KPI tool appears in the navigation of each course in Canvas. When a student clicks the tool, they are directed to a brief anonymous survey (see Figure 1) in which they self-rate their learning experiences in five dimensions, including the pace of course modules, self-mastery of learning goals, support for learning, clarity of course materials, and class engagement. Instructors can then view a dashboard that displays a weekly snapshot of students’ assessment of how they’re performing in their course (see Figure 2).
The KPI tool provides instructors with a colour-coded heat map that shows outcome metrics. This feature allows instructors to view weekly averages at a glance and quickly identify areas for improvement. Another innovative feature gives instructors course-specific snapshots of student performance in near real time in their KPI dashboard as student responses are submitted. This feature is designed to help instructors adjust their teaching strategies in a timely manner as the student data is refreshed.
Tips for improving instruction using the KPI tool
Looking to the future and the post-pandemic era, our project can help address common challenges associated with teaching and learning across different modes (such as remote, hybrid and fully residential). For example, the KPI Canvas tool was designed to minimise effort from instructors, who often deal with constrained time and resources. Additionally, the tool can engage students in the reflective activity of self-assessing their effort and performance. This will help enhance their ability to self-regulate and take ownership of their learning.
We recommend that students fill out the KPI survey at the start of the week’s first class, as it can help them reflect on their experience from the week prior. This approach allows the instructor to make adjustments for the next class. We’ve also seen benefits from in-class discussion of the feedback. This use of the data can make students feel heard and motivated to perform well in the class. Likewise, instructors can use the discussion to enhance course design.
Tips for improving assessment
Standard measurement tools (for example, quizzes, multiple-choice tests or essays) can assess student performance and provide information about knowledge and skill level but often do not pinpoint where they struggle during the learning process. The KPI tool plays a critical role in illuminating how and where students are challenged in a class. This delivers valuable insight to instructors so they can revisit and redesign course features according to students’ needs.
Our KPI tool has been piloted with more than 45 courses since its launch in summer 2020. We collect feedback from participating instructors through an online survey and semi-structured interviews. Our initial research into this tool revealed that instructors generally perceive it to be easy to use and find the results on the KPI dashboard easy to comprehend. Instructors reported that they primarily used students’ responses to adjust their instruction methods and monitor students’ learning progress. Feedback from Georgia Tech faculty suggests that this KPI tool can benefit those instructors who need to collect feedback from students in large classes or those who wish to introduce new course materials or elicit continuous feedback from students as they navigate the course.
With students unexpectedly located far from the traditional classroom and faculty redesigning their instruction for emergency remote teaching, the pandemic left many instructors with few avenues for the type of feedback that might occur verbally or organically in a campus classroom. Yet the need for assessment – of both learning and instruction – remains as critical as ever. Our hope is that a small resource such as the KPI tool can empower better learning and teaching experiences.
Jonna Lee is director of research in education innovation and Meryem Yilmaz Soylu is a research scientist, both at the Center for 21st Century Universities at the Georgia Institute of Technology.
If you found this interesting and want advice and insight from academics and university staff delivered directly to your inbox each week, sign up for the THE Campus newsletter.
comment