Logo

IN: Learning analytics project

Learning Analytics Project

CTLI’s learning analytics project encourages theory- and data-informed decision making about course design that supports universal design for learning and learner achievement.

Overview

Learning analytics is a fast-growing field in post-secondary education. Learning management systems collect vast amounts of data on student behaviour in learning. This data can be analyzed to improve learning experiences and student success.

The Centre for Teaching, Learning and Innovation (CTLI) has access to Canvas Data, which can be leveraged to analyze how learning environment design impacts learner achievement and persistence. This initial project focuses on the impact on learners of implementing universal design for learning (UDL).

Timeline

Read through the timeline by clicking on each point.

Fall 2021
Winter 2022
Spring 2022
Fall 2022 Pilot
Prototyping: Project team experiments and prototypes possible Canvas Data outputs.
Pilot: Project team recruits faculty members to pilot the templates in their courses. In addition, the project team explores further learning analytics queries as identified by faculty members.
Submission of a report and recommendations for Canvas Data usage in the future, including a larger scale implementation plan and faculty professional development plan.
Piloting digital dashboards and consultation service model with a larger faculty group.

Deliverables

  • Evaluation of the return on investment of the Canvas Data product
  • Easy-to-use templates for faculty members to note observations about the impact of their course design choices on learner achievement and persistence
  • Defined processes for faculty members with course-design queries that can be answered using learning analytics

Alignments

CTLI Strategic Innovation Model

Support Services Operational Review:

Opportunity 5—KPI Tracking and Data Analysis

Opportunity 6—Process Improvement

Glossary

  • Learning analytics: “The measurement, collection, analytics and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs” (Slade, 2016, p. 2).
  • Canvas Data: A Canvas service that provides administrators with optimized access to their data for reporting and queries.
  • Universal design for learning (UDL): An approach to learning that gives all students equal opportunity to succeed. The learning environment is designed for flexibility and includes options for multiple modes of engagement, representation, and action and expression.
  • Learning environment: An inclusive space (physical or virtual) that mitigates barriers to learning and encourages engagement in learning.
  • Learning management system (LMS): Virtual learning space to house, deliver, and track learning.
  • Canvas: The learning management system that Lethbridge College uses.
  • Badges: Displayed as a digital images, these files also contain detailed, verifiable information about the badge award, including information about the issuer and the learner, criteria for earning the badge, and more. Learners collect these badges and can share them via social media, websites, links, and printed certificates.

Engagement

We believe that meaningful stakeholder engagement is essential to any good initiative. Below is a list of engaged stakeholders. (Classification based on CTLI engagement framework and IAP2 Public Participation Spectrum)

Inform

Our promise: We will keep you informed about the project and decisions that are made.

  • Deans and associate deans

Consult

Our promise: We will consult with you to ensure your viewpoint is heard and considered when making decisions. We will communicate how this input and feedback influenced the decisions made.

  • Institutional Compliance (policy consultation)
  • IPARS (data management and analysis consultation): Darryl Godwin

Involve

Our promise: We will work directly with you to ensure your viewpoints and concerns are reflected in the decisions made.

MLT (Management Leadership Team)

CLC (College Leadership Council)

Joyce Shigehiro (Academic Upgrading)

Scott Lehbauer (Liberal Arts and Life Sciences)

Kelly Thomson (Business)

Natalie Barfuss (Business)

Leanne Jones (Academic Upgrading)

Brenda Bryson (Educational Assistant)

Julie Deimert (General Arts and Science)

Rena Walker (Therapeutic Recreation)

Collaborate

Our promise: We will partner directly with you throughout the project and decision making process. We will create joint solutions or recommendations.

Learning Experience Designers: Donna McLaughlin and Trevor Gellrich

Digital Learning specialists: Rebecca Helmer and Lorne Deimert

Accessibility Services: Cayla Clemens and Ashley Burke

Institutional Planning and Reporting: Bandana Singh (junior analyst)

Empower

Our promise: We will abide by the decisions you make.

  • Project Sponsors: Dean, Centre for Teaching, Learning and Innovation — Jaclyn Doherty

Project Lead: Christie Robertson

Educational Leadership - Scholarly Teaching

Progress Update

Fall 2022

Achieved

Goal: Learn about the ethical use of Canvas Data and create prototypes that might be used with faculty

  • Piloted dashboards to convey learning analytics for 24 chosen courses with 5 faculty members
  • Gathered feedback from faculty on what was useful for them and how they would like to see the dashboards evolve
  • Explored potential insights into data findings from the dashboards and considered the course designs to compile summaries and recommendations for faculty to consider in terms of course design practices
  • Eliminated data sets where students’ identity might be revealed due to small amounts of data
  • Defined the purpose of the data provided, which is to be used for observational purposes and not for isolation for auditing, or other high-stakes analysis involving examining single users or small samples

Not Achieved

Goal: Integrate canvas data into in the evaluation and monitoring of the integrated supports service, ed tech framework, and evaluation of learning café tools/service

  • Exploration of what the tool can do and how it can best serve faculty was more complex than expected. It was determined more time was needed to explore and expand the service to faculty first.

Lessons Learned

  1. Recommendations were to adapt our dashboards to compare learning analytics across multiple offerings of a course to be able to identify possible trends and more sound insights in terms of recommendations for course design practices
  2. The dashboards might benefit from additional features to be more useful
    • Grade correlations
    • More details about the module engagement
    • Clear learner pathways would be more informative than last access timestamps
    • Filters to select what data to view (for comparison and contrasting)
    • Adapt visualization of learning community engagement—particularly for discussion forums
  3. Exploring learning analytics of a single offering of a course is more suitable to consider while teaching for potential just-in-time adjustments to curriculum and teaching
  4. Draw connections between the possible thematic insights (shared in a summary with recommendations) from the learning analytics to our Teaching Excellence Framework to help draw connections as to the importance of the potential insights
  5. Dashboards should continue to be improved over time and likely will be ever-evolving since this is an emerging field

Data Examples

Module Item Clicks

Bar chart showing number of clicks by students on items within the course.

Purpose:

This graph displays a count all the clicks of all active students on every item in each module, including items such as links, assignments, files and Canvas pages and includes all repeat visits to the module items.

This data could help inform instructors on:

  • Which course elements students are engaging with the most
  • If there are course elements students are not engaging with
  • If monitored regularly throughout the courses, instructors can observe if students are engaging with content as intended in the course schedule, working ahead, or falling behind.

 

Limitations:

  • Modules with no views were removed from chart. This issue should be resolved for the next iteration.

Module Items Snapshot

Bar chart showing items that students viewed within the course.

Purpose:

This chart displays whether students viewed or did not view each item in the Modules section through a linear chart. Numbers are based on the active student count in the course.

This data could help inform instructors on:

  • If there is content that a majority of students are engaging with
  • If there is content that a majority of students are not engaging with
  • If there is important content that the majority of students are not engaging with this may prompt an instructor to explore why or who is not engaged and/or direct communication to learners about its importance.

 

Limitations:

  • Items with no views were removed from chart. This issue should be resolved for the next iteration.

Discussion Summary

Bar chart showing activity of students on discussion boards.

Purpose:

This data displays the total amount of posts, replies, and replies to replies for each discussion board.

This data could help inform instructors on:

  • Which discussions posts students are most/least engaged with
  • Posts with more replies to replies generally indicates a deeper, more engaged discussion has happened (note: this is just an indicator that needs to be confirmed by reviewing the actual discussion)

 

Limitations:

  • Data cannot be used for to create student sociogram due to the complexity of data tables

Announcements

Table with data showing student visits to announcements within the course.

Purpose:

This table includes all course announcements made throughout term. % Active Students represents unique visit students to the announcement in Canvas. Users must click on the announcement for the data to be collected. Student Total represents a count of all student visits to each announcement.

Also included in this section is Notification preferences. Students can set specific notification preferences to determine how frequently they will receive email notifications from their courses.

This data could help inform instructors on: 

  • How many students have set their announcements, so they are notified when one is sent
  • Which announcements students engage with most/least
  • How students engaged with announcements throughout the semester (e.g., did it ebb and flow or was it consistent?)

Next Steps

Immediate

  • Refine dashboards and the learning analytics service and process
    • Continue to collaborate with volunteer faculty
    • Pilot incorporating some forms and reflective practice tools to the process to standardize for consistency and faculty growth in the course design dimension of the Teaching Excellence Framework
    • Formalize a service process with consultation and self-directed pathways to increase accessibility across faculty
    • Update project website and add point to existing resources about Canvas analytics that are readily available to faculty

Long-term goals

  • Implement Student dashboards
    • CLTI’s Digital Learning Team is looking into potential Canvas plugins/options that might have an automation for this
  • Shift to live, automated dashboards for faculty to make the service accessible to all
  • Finalize a learning analytics service model that works for faculty and CTLI
  • Develop a PD and communication plan to launch learning analytics to all Lethbridge College faculty by Fall 2023
more2

Want to learn more?

Located on the traditional lands of the Blackfoot Confederacy, Lethbridge College is committed to honouring the land from a place of knowing. We honour the Siksikaitsitapi as both the traditional and current Land Keepers of this area, and we welcome all First Nations, Métis, Inuit, and non-Indigenous peoples who call Blackfoot territory their home.

© Centre for Teaching, Learning & Innovation | Lethbridge College   •   2020  •   3000 College Drive South, Lethbridge, AB T1K 1L6   •   Visit lethbridgecollege.ca  •   Handcrafted by AislinThemes & CTLI
Read about our privacy policy and cookies here.