Are you a teacher interested in using learning analytics to improve the teaching and learning in your course? Than this toolkit is for you! Here you will find information on the what and why of learning analytics and how to get started with learning analytics at TU/e. The toolkit includes examples of teacher and student facing learning analytics and what type of questions could be answered. Furthermore, you’ll find examples of learning analytics in practice at TU/e and regulations concerning privacy and ethics for learning analytics. Still want to know more? The last section provides links to more resources.

Learning Analytics

What?

Learning Analytics is the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs.

Why?

Historically, some of the most common uses of learning analytics is prediction of student academic success, and more specifically, the identification of students who are at risk of failing a course or dropping out of their studies. While it is reasonable that these two problems attracted a lot of attention, learning analytics are far more powerful. The evidence from research and practice shows that there are far more productive and potent ways of using analytics for supporting teaching and learning. Some of the most popular goals of learning analytics include:

Supporting student development of lifelong learning skills and strategies
Providing personalized and timely feedback to students regarding their learning
Supporting development of important skills such as collaboration, critical thinking, communication and creativity
Developing student awareness by supporting self-reflection
Supporting quality learning and teaching by providing empirical evidence on the success of pedagogical innovations

Educators and instructional designers can gain new insights into students’ learning processes. Students can be offered more timely, precise, actionable feedback on their learning process and outcomes.
For a brief introduction to Learning Analytics, please check “Learning analytics in a nutshell” (3 minutes).
For a more extensive explanation of this topic, you can watch the video “What is learning analytics?” (21 minutes).
Information retrieved from https://www.solaresearch.org/ and https://steinhardt.nyu.edu/learn

How?

How to get started with learning analytics at TUe.pptx

1. Teacher and Student Facing Learning Analytics

2. Learning Analytics at TU/e

Learning analytics at TU/e is currently in an explorative state. This section presents examples of the use of learning analytics at our university for your inspiration.

Automated Prognostic Student Progress Monitoring System

Automated Prognostic Student Progress Monitoring System
Summary
This project covers the development of the so-called Automated Prognostic Student Progress Monitoring System (APSPMS). The AI-powered system provides students with a constantly updated prognosis concerning the likelihood of them passing the final exam, including suggested actions for improvement if necessary.
Introduction
The course Electromagnetics at the department of Electrical Engineering is a notoriously difficult course for students. This course attracts about 200-300 students per year. To increase the low passing rates, the teachers aimed to increase active student participation from start to finish of the course. From experience, the teachers found that providing students with immediate feedback contributed to their active course participation. They conceived the idea to provide students with a constantly updated prognosis on the likelihood of them passing the course. This prognosis would also include suggestions for action to raise students’ chances of passing the course. To minimize increases in teacher workload, the feedback for students was to be generated by an automated system.
Aim
The aim of this project was to develop the so-called Automated Prognostic Student Progress Monitoring System (APSPMS). This system can automatically and reliably generate a prognosis of the final outcome of a student's performance during a course based on the student's current comprehension of the course content in line with the learning objectives and schedule of the course. Additionally, the system can suggest corrective recommendations for both the student and the educator in order to prevent unsatisfactory results, improve the prognosis, and increase the quality of learning. In other words, the system supports the personal learning path of students.

Some of the key tasks/features envisioned for the APSPMS are:
- Continuous monitoring of individual student performance from the start of the course.
- Capturing of hard indicators (e.g., exam scores) that represent student learning through quizzes, exercises and other tangible assessments.
- Capturing of soft indicators such as time spent by a student for reading a chapter, time spent to answer a question, amount of participation in discussions of a specific topic, etc.
- Predicting the student's expected performance.
- Providing opportunity to analyze and investigate the possibility to use the soft indicators to improve the prediction of the final course grade.
- Providing opportunity for students to individually set their learning curve for that course.
- Providing concrete recommendations for students
- Identifying cases where the predicted performance of a student significantly varies from the final assessment.
Results
A pilot version of the APSMS is up-and-running in the Electromagnetics course. Students receive five prognosis emails during the run of the course. The current prediction rate of the system is 68%. This rate is expected to increase by including more variables in the analysis. Additionally, the prediction rate of the AI-powered system will increase with increasing data input from new runs of the course.

Students can currently choose whether or not they want to use the APSPMS or not. About 60% of the students use the system. These students have reported positive experiences. Moreover, the teachers’ evaluations have shown that students that use the APSPSM achieve significantly better results compared to their peers that don’t use the system.
Next steps
- Increasing the prediction rate to 98% by enriching the system with more variables and data
- Providing more personalized feedback to students
- Moving from an email-based feedback system to another system to be able to track if the provided feedback triggered immediate action
- Upgrading the pilot system into a running business application
Want to know more?
This learning analytics pilot is initiated and lead by two teachers from the department of Electrical Engineering, Mark Bentum or Ramiro Serra. They have published two papers on the APSPMS:

Martinez, C., Booij, T.B.N., Serra, R., Vertegaal, C.J.C., Bentum, M.J., Sundaramoorthy, P., Bounik, Z. (2023). Early Warning System with Formative Feedback to Support Engineering Students in Electromagnetism. 2023 32nd Annual Conference of the European Association for Education in Electrical and Information Engineering (EAEEIE), Eindhoven, Netherlands, pp. 1—6, DOI: 10.23919/EAEEIE55804.2023.10181601.
Link: https://ieeexplore.ieee.org/document/10181601

Martinez, C., Serra, R., Sundaramoorthy, P., Booij, T., Vertegaal, C., Bounik, Z., van Hastenberg, K., Bentum, M. (2023). Content-Focused Formative Feedback Combining Achievement, Qualitative and Learning Analytics Data. Education Sciences, 13(10), 1014. https://doi.org/10.3390/educsci13101014
Link: https://www.mdpi.com/2227-7102/13/10/1014

For more information, contact Mark Bentum (m.j.bentum@tue.nl) or Ramiro Serra (r.serra@tue.nl).

Also see: https://boost.tue.nl/projects/automatic-prognostic-student-progress-monitoring-system/
Everyone’s a winner?

Everyone's a winner?

Summary
This project covers the use of learning analytics to identify students’ effective learning paths in blended courses. It resulted in four scales to measure students' self-regulated learning in detail. The scale scores were substantially associated with final course grades. Measurements can take place unobtrusively and in an automated way.
Introduction
Blended learning is becoming ubiquitous in education at TU/e. This leads to a new diversity of offline and online learning paths that students can potentially follow. However, it is unknown which learning paths (groups of) students pursue, and if and how these chosen learning paths contribute to their study success. Learning analytics of students’ activities in the online learning environment (Canvas) can be used to identify how learning paths correlate with study success. This knowledge can offer insights into whether and how students get engaged with the course content in different phases of their learning.
Aim
The aim of the project is to provide lecturers and students with insights into the learning paths of students and how they relate to their course success. These insights enable teachers to identify students in need of more support and can provide guidelines for adjustments in the course design.
Results
This project resulted in four scales that measure four different dimensions of self-regulated learning in an unobtrusive and reliable way. These scales measure students’ (1) task definition, (2) goals setting, (3) learning tactics, and (4) learning adaption. Moreover, it was found that these scales correlate substantially with students’ final course grades. It was also found that, within some limitations, these scales are transferable to other blended learning courses, meaning that they have value beyond the context of this pilot. This constitutes an important step toward the identification of students in need of additional educational support.

Another important finding is that a well-planned design of the Canvas course environment, including embeddedness of materials such as videos, articles, quizzes et cetera, is necessary to collect useful and sufficient data for effective learning analytics. Furthermore, an adequate overall course design that stimulates student engagement and activation from the start to finish of the course is essential.
Next steps
The BOOST! project is formally finished. However, the teachers continuously apply and refine the scales. If other teachers (or researchers) would like to use the scales, they are happy to provide support.
Want to know more?
This learning analytics pilot is initiated and led by teachers from the Department of Industrial Engineering and Innovation Sciences, Uwe Matzat, Ad Kleingeld, and Chris Snijders. They have published a paper on their work:

Cristea, T., Snijders, C., Matzat, U., & Kleingeld, A. (2023). Unobtrusive measurement of self-regulated learning: A clickstream-based multi-dimensional scale. Education and Information Technologies.
Link: https://doi.org/10.1007/s10639-023-12372-6

For more information, contact Uwe Matzat (u.matzat@tue.nl).

Also see: https://boost.tue.nl/projects/everyones-a-winner/

3. Privacy and Ethics for learning analytics

4. More resources to kickstart your LA thinking and practice

There is vast amount of educational research available on learning analytics. In this section we present some publications on learning analytics to stimulate your learning analytics thinking and practice.

Learning analytics in education design

Learning analytics allows you to collect data from a student’s every mouse click in an online environment, the videos they watch and other digital traces they leave behind that might tell you something about their learning behavior. This report by SURF provides teachers and education designers support and inspiration in applying learning analytics when designing online (parts of) education.

Learning Analytics in Education Design – a guide
Learning Analytics Implementation Design

This text reflects on the state of the art of learning analytics implementation design. The principles of Coordination, Comparison, and Customization provide a lens to examine the different dimensions of design choice that can affect how analytics feedback is taken up and acted on in particular educational contexts. Models are proposed to examine and adjust course-wide learning designs as well as to investigate and respond to individual student activity patterns.

Wise, A.F., & Vytasek, J. (2017). Learning Analytics Implementation Design. In Charles Lang et al., Handbook of Learning Analytics. Retrieved from https://www.solaresearch.org/publications/hla-17/hla17-chapter13/
A practitioner’s guide to measurement in learning analytics – decisions, opportunities, and challenges

Understanding why and what is measured, and the implications of measurement choices for inferring explanations from learning analytics models is important. This paper explores the challenges and opportunities of using different measurement methods, including surveys and trace data from online education environments. It highlights the need for careful consideration of the learning constructs of interest and the instructional and learner contexts when designing data collection instruments and interpreting results.

Gray, G., & Bergner, Y. (2022). A practitioner’s guide to measurement in learning analytics – decisions, opportunities, and challenges. In Charles Lang et al., Handbook of Learning Analytics – Second Edition. Retrieved from https://www.solaresearch.org/publications/hla-22/hla22-chapter20/
An introduction to Fairness, Absence of Bias, and Equity in Learning Analytics

Equity is an important value in education and learning analytics can play a role in achieving this. This text highlights disparities in educational experiences due to factors like race, gender, and economic status. It warns against the uncritical use of learning analytics, which can lead to inequality and discrimination. The need for a critical, political, and nuanced approach to equity is emphasized, and the notions of algorithmic fairness and absence of bias in learning analytics are discussed.

Uttamanchandani, S., & Quick, J. (2017). An introduction to Fairness, Absence of Bias, and Equity in Learning Analytics. In Charles Lang et al., Handbook of Learning Analytics – Second Edition. Retrieved from https://www.solaresearch.org/publications/hla-22/hla22-chapter20/
Let’s not forget: Learning analytics are about learning

Learning analytics creates the potential to understand and optimize learning. To realize this, learning analytics developments should draw on and advance educational research and practice, and ground data collection, measurement, analysis, reporting, and interpretation processes within existing research on learning. This paper stresses that learning analytics are about learning.

Gasevic, D., Dawson, S., & Siemens, G. (2015). Let’s not forget: Learning analytics are about students. TechTrends, 59(1). DOI: https://doi.org/10.1007/s11528-014-0822-x
Theory and learning analytics

The challenge of understanding how theory and analytics relate is to move “from clicks to constructs” in a principled way. It is argued that the use of a learning analytics tool is always aligned with assessment regimes, which are in turn grounded in epistemological assumptions and pedagogical practices. Consequently, deploying a given learning analytics tool expresses a commitment to a particular educational worldview, designed to nurture particular kinds of learners. This text outlines some key provocations in the development of learning analytic techniques, key questions to draw out the purpose and assumptions built into learning analytics.

Knight, S., & Shum, S.B. (2017). Theory and Learning Analytics. In Charles Lang et al., Handbook of Learning Analytics. Retrieved from https://www.solaresearch.org/publications/hla-17/hla17-chapter1/
Looking for more resources?

The Society for Learning Analytics research has published two Handbooks of Learning Analytics. These can be found here: https://www.solaresearch.org/publications/handbook-of-learning-analytics/

The Learning Analytics Research Network also provides valuable resources, including a Learning Analytics 101 online course. These can be found here: https://steinhardt.nyu.edu/learn

Learning Analytics

1. Teacher and Student Facing Learning Analytics

2. Learning Analytics at TU/e

Automated Prognostic Student Progress Monitoring System

Everyone's a winner?

3. Privacy and Ethics for learning analytics

4. More resources to kickstart your LA thinking and practice

5. Upcoming events