Responsible Institutional Analytics: Interpreting Bias with AI Support
Organizations: Pompeu Fabra University, Barcelona, Catalonia
Abstract
Institutional Analytics (IA) dashboards inform decision-making in higher education, yet data limitations, constraints in analytical techniques, and missing contextual information often affect their interpretation. To support more responsible interpretation of IA, we introduce FACTRIA, a framework that organizes potential biasing factors across four areas: the analytics pipeline, institutional context, course-level characteristics, and demographics. We used the FACTRIA framework as input to a generative-AI chatbot designed to prompt users to reflect on these factors while analyzing IA. A qualitative study with stakeholders, drawing on four authentic IA cases, and a transition network analysis showed that the chatbot prompted participants to recognize how overlooked factors influenced their initial interpretation. Findings indicated that combining a structured framework with AI-based guidance can enhance context-aware, responsible interpretation of institutional data.
Figures & tables
| Participant | Role and Experience | Use of Institutional Data |
|---|---|---|
| P1 (Case 3 & 4) | LMS support specialist with 10+ years of experience. | Frequently reviews LMS data. |
| P2 (Case 1 & Case 4 ) | Education specialist in teaching innovation and training with 10+ years of support experience. | Checks institutional data occasionally. |
| P3 (Case 1 & 2) | Teaching quality data support specialist with 20+ years of experience. | Frequently reviews institutional data. |
| P4 (Case 1 & 4) | Education specialist in teaching innovation with 16+ years of support experience. | Frequently reviews institutional data. |
| P5 (Case 2 & 3) | LMS and pedagogy support manager with 2+ years of experience. | Frequently reviews institutional data. |
| P6 (Case 2 & Case 3 ) | Academic leader in teaching quality and innovation, with 15+ years teaching and 4+ years management experience. | Frequently reviews institutional data. |
| Case X + Case Y | 1 o Cases | 2 o Cases | ||||
|---|---|---|---|---|---|---|
| Without | With | Without | With | Without | With | |
| Average | 1.41 | 3.45 | 1.73 | 3.55 | 1.09 | 3.36 |
| Standard deviation | 0.80 | 0.86 | 0.65 | 0.82 | 0.83 | 0.92 |
| Variance | 0.63 | 0.74 | 0.42 | 0.67 | 0.69 | 0.85 |
| Median | 1.50 | 4.00 | 2.00 | 4.00 | 1.00 | 4.00 |
| Reference | Description |
|---|---|
| Radchenko (2020) | Described the aspects that may influence student satisfaction and the effects that result in biases. The author argues that various factors, such as gender and course level, influence data collection on student satisfaction. |
| Marques et al. (2024) | Described the importance of considering reliability to analyze the data collection. Also investigated gender biases by studying different factors such as age, level of the course, and academic centers. |
| Clayson (2009) | Examined previous research on the connection between learning and student satisfaction. The findings indicated that, on average, there is only a weak relationship between the two variables, depending on factors such as context, discipline, course type, and instructor. |
| Ortiz-Beltrán et al. (2022) | Evaluated how changes in teaching modalities influenced student satisfaction scores before, during, and after COVID-19. |
| Rosen (2018) | Investigated gender and the impact of teaching evaluations across various academic departments and disciplines. |
| Aragón et al. (2023) | Investigated the impact of department gender composition on student satisfaction scores. |
| Factor | Description of the factor | Sub-factor | Description of the sub-factor | Examples |
| Analytics Pipeline Factors | Process related to the institutional analytics approach applied in a determined problem. | Data quality | Processes related to gathering, preparing, and structuring data for analysis (pre-processing). | Data collection, feature extraction, sample, data reliability, level of data aggregation. |
| Types of data analysis | Approaches and assumptions used to interpret and extract insights from data. | Selection of analysis techniques, visualization, analytical context, assumptions of the data analyses. | ||
| Institutional Factors | Factors related to the institutional level in Higher Education Institutions. | Academic centers | Structural and demographic characteristics of academic departments and centers. | Academic centers (e.g., Economics, Engineering, Humanities); department gender composition (distribution of professors by gender in each department). |
| Educational framework | Guidelines for teachers in higher education. | Teaching methodologies framework. | ||
| Modality of the university | Refers to the different ways a university operates. | Brick-and-mortar university or online university. | ||
| Course Factors | Factors related to the course level in Higher Education Institutions. | Course level, class size | Aspects regarding the course. | Class size, level of the course. |
| No. | Item |
|---|---|
| 1 | Did the system help you to get a more responsible interpretation? |
| 2 | I think that I would like to use this system frequently |
| 3 | I found the system unnecessarily complex |
| 4 | I thought the system was easy to use |
| 5 | I think that I would need the support of a technical person to be able to use this system |
| 6 | I found the various functions in this system were well integrated |
| Code | SubCode | Definition and Examples |
|---|---|---|
| Agreement | Confirmation | A consensus or affirmation regarding a specific statement or piece of information. User: “Yes, of course, the score could not be representative if the questionnaire is filled by a few students.” Assistant: “Yes, the ‘Bottom Courses’ are those with the lowest student enrollment.” |
| Validation | Acknowledgment that the provided information is accurate. Assistant: “That is an interesting observation about how student and professor gender might relate to satisfaction scores.”; “Your interpretation effectively explores how the gender of both students and professors might influence satisfaction results across the faculties.” | |
| Disagreement | When there is a disagreement between the user and the assistant regarding a statement or piece of information. User: “I don’t think the academic unit influences, but the course factors.”; “there is no information on the course level.” | |
| Express Doubts | When there is ambiguity/doubts about the meaning of a statement or piece of information for either side. User: “I don’t understand what you mind”; “The visualization is comparing the shannon index by teaching methodologies, but for me it is not clear the objective of the visualization.” | |
| Provide Information (PI) | Clarification | When either side clarifies a statement or piece of information. Assistant: “Data Quality examines whether the aggregated scores accurately represent the full range of individual satisfaction responses.” ; User: “just faculty types.” |
| Explanation | When there is an explanation from either side about a statement or piece of information. Assistant: “The way a methodology is implemented certainly affects the relationship between workload and student performance.”; User: “On the other hand for the explorative purposes it is indeed helpful such presentation.” |
| Participant | Assigned case (max number of sub-factors) | Without | With |
|---|---|---|---|
| P1 | CASE 3 (5 sub-factors) | 1 | 3 |
| CASE 4 (4 sub-factors) | 0 | 4 | |
| P2 | CASE 1 (4 sub-factors) | 2 | 4 |
| CASE 4 (4 sub-factors) | 1 | 4 | |
| P3 | CASE 1 (4 sub-factors) | 1 | 2 |
| CASE 2 (5 sub-factors) | 0 | 3 |