IN: Learning Analytics Project

Learning Analytics Project

CTLI’s learning analytics project encourages theory- and data-informed decision making about course design that supports universal design for learning and learner achievement.

Overview

Learning analytics is a fast-growing field in post-secondary education. Learning management systems collect vast amounts of data on student behaviour in learning. This data can be analyzed to improve learning experiences and student success.

The Centre for Teaching, Learning and Innovation (CTLI) has access to Canvas Data, which can be leveraged to analyze how learning environment design impacts learner achievement and persistence. This initial project focuses on the impact on learners of implementing universal design for learning (UDL).

Timeline

Read through the timeline by clicking on each point.

Fall 2021
Winter 2022
Spring 2022
And Beyond
Prototyping: Project team experiments and prototypes possible Canvas Data outputs.
Pilot: Project team recruits faculty members to pilot the templates in their courses. In addition, the project team explores further learning analytics queries as identified by faculty members.
Submission of a report and recommendations for Canvas Data usage in the future, including a larger scale implementation plan and faculty professional development plan.
Incorporation of Learning Analytics into course design and support services in CTLI.

Deliverables

  • Evaluation of the return on investment of the Canvas Data product
  • Easy-to-use templates for faculty members to note observations about the impact of their course design choices on learner achievement and persistence
  • Defined processes for faculty members with course-design queries that can be answered using learning analytics

Alignments

CTLI Strategic Innovation Model

Support Services Operational Review:

Opportunity 5—KPI Tracking and Data Analysis

Opportunity 6—Process Improvement

Glossary

  • Learning analytics: “The measurement, collection, analytics and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs” (Slade, 2016, p. 2).
  • Canvas Data: A Canvas service that provides administrators with optimized access to their data for reporting and queries.
  • Universal design for learning (UDL): An approach to learning that gives all students equal opportunity to succeed. The learning environment is designed for flexibility and includes options for multiple modes of engagement, representation, and action and expression.
  • Learning environment: An inclusive space (physical or virtual) that mitigates barriers to learning and encourages engagement in learning.
  • Learning management system (LMS): Virtual learning space to house, deliver, and track learning.
  • Canvas: The learning management system that Lethbridge College uses.
  • Badges: Displayed as a digital images, these files also contain detailed, verifiable information about the badge award, including information about the issuer and the learner, criteria for earning the badge, and more. Learners collect these badges and can share them via social media, websites, links, and printed certificates.

Engagement

We believe that meaningful stakeholder engagement is essential to any good initiative. Below is a list of engaged stakeholders. (Classification based on CTLI engagement framework and IAP2 Public Participation Spectrum)

Our promise: We will keep you informed about the project and decisions that are made.

  • Deans and associate deans

Our promise: We will consult with you to ensure your viewpoint is heard and considered when making decisions. We will communicate how this input and feedback influenced the decisions made.

  • Institutional Compliance (policy consultation)
  • IPARS (data management and analysis consultation): Darryl Godwin

Our promise: We will work directly with you to ensure your viewpoints and concerns are reflected in the decisions made.

MLT (Management Leadership Team)

CLC (College Leadership Council)

Joyce Shigehiro (Academic Upgrading)

Scott Lehbauer (Liberal Arts and Life Sciences)

Kelly Thomson (Business)

Natalie Barfuss (Business)

Leanne Jones (Academic Upgrading)

Our promise: We will partner directly with you throughout the project and decision making process. We will create joint solutions or recommendations.

Learning Experience Designers: Donna McLaughlin and Trevor Gellrich

Digital Learning specialists: Rebecca Helmer and Lorne Deimert

Accessibility Services: Cayla Clemens and Ashley Burke

Institutional Planning and Reporting: Bandana Singh (junior analyst)

Our promise: We will abide by the decisions you make.

  • Project Sponsors: Dean, Centre for Teaching, Learning and Innovation — Jaclyn Doherty

Project Lead: Christie Robertson

Progress Update

Fall 2021

Goal:

Learn about the ethical use of Canvas Data and create prototypes that might be used with faculty

Achieved:

  • Completed a literature review of ethical data usage
  • Reviewed institutional policies on data usage
  • Drafted communication that can be shared with learners to inform them about what Canvas Data is and how it’s being used
  • Created prototype visualizations for Canvas Discussions and HyFlex Learning Design

Lessons learned:

  • Canvas Data is useful for making observations, but we cannot draw conclusions or correlations from the data
  • By pulling multiple data sets on discussion posts, we were able to make better observation of learner engagement (which posts had the most replies, when the most replies happened, which students replied the most)
  • We need to consider the data within other contexts (e.g., scholarly research on teaching and learning) and to observe similar data repeatedly before making conclusions about why patterns are occurring

Data Examples

Purpose:

This graph is a count of students’ initial posts, replies to posts, and replies to replies. This data can inform instructors on

  • which discussions resulted in the most posts
  • which discussions went beyond an initial response and mandatory response to another student

 

Limitations:

  • Doesn’t show individual student responses (e.g., replies to replies could potentially be one student only)
  • Can only observe the response rate and type. Not enough data is provided to infer the reason for the response pattern.

Purpose:

This graph is a count of individual students’ initial posts, replies to posts, and replies to replies. This data can inform instructors on

  • which students posted the most/least responses
  • which students went beyond an initial response and mandatory response to another student

 

Limitations:

  • Doesn’t show which responses are for which discussion (e.g., a student may have made multiple posts, replies, and replies to replies in just one discussion and none of the others)
  • Can only observe the response rate and type. Not enough data is provided to infer the reason for the response pattern.

Purpose:

This graph is a count of both individual students’ initial posts, replies to posts, and replies to replies in each discussion. Note: student numbers were removed for confidentiality purposes, but would appear on this graph if requested by an instructor.

 

This data can inform instructors on

  • which students posted the most/least responses
  • which students went beyond an initial response and mandatory response to another student
  • which discussions had more replies to replies (indicates a more authentic discussion than just mandatory post-and-reply behaviour)
  • if students maintained consistent response patterns throughout each discussion or if this was limited to certain discussions

 

Limitations:

  • Can only observe the response rate and type. Not enough data is provided to infer the reason for the response pattern.

Next Steps

  • Pilot the use of templates with faculty members who use Discussions and/or HyFlex course design elements—elicit their feedback on usefulness and the request-making process. If time allows, create other templates using faculty queries
  • Use Canvas Data with the following current CTLI projects to analyze engagement with design choices in each Canvas Resource:
    • Embedded Supports
    • Learning Café Canvas Resources
    • Student and Teacher Hive Analytics
more2

Want to learn more?