cs.HCOct 5, 2026
SaveResponsible Institutional Analytics: Interpreting Bias with AI Support
Organizations: Pompeu Fabra University, Barcelona, Catalonia
Abstract
Institutional Analytics (IA) dashboards inform decision-making in higher education, yet data limitations, constraints in analytical techniques, and missing contextual information often affect their interpretation. To support more responsible interpretation of IA, we introduce FACTRIA, a framework that organizes potential biasing factors across four areas: the analytics pipeline, institutional context, course-level characteristics, and demographics. We used the FACTRIA framework as input to a generative-AI chatbot designed to prompt users to reflect on these factors while analyzing IA. A qualitative study with stakeholders, drawing on four authentic IA cases, and a transition network analysis showed that the chatbot prompted participants to recognize how overlooked factors influenced their initial interpretation. Findings indicated that combining a structured framework with AI-based guidance can enhance context-aware, responsible interpretation of institutional data.
Figures & tables
Figure 1: FACTRIA: A framework of Factors in the Analytical process and the data Context for Responsible IA.
Figure 2: Study Design
Figure 3: Case 1: Distribution of student satisfaction scores with teaching, aggregated by professor gender, student gender, and faculty. Each dot represents a professor’s mean satisfaction score ( Marques et al., 2024 ) .
Figure 4: Case 2: Distribution of the performance (Weighted Grades) by five of the most used Active Teaching Methodologies: Lectures (LECT), Project-based learning (PJBL), Problem-based learning (PRBL), Flipped classroom (FLIP), and Case studies (CASE). Courses are ranked by workload satisfaction, and the top 20% and bottom 20% are highlighted based on their ratings. ( Marques et al., )
Figure 5: Case 3: Comparison of three similar high-interaction courses (A, B, C) against the institutional average on a common 0–1 scale covering 4 student outcomes (students’ performance and 3 satisfaction indicators), 7 LMS usage variables at course and Task—Activity or Resource (A/R)—level, 4 planned task types, 3 social designs, and 2 task modalities. ( Ortiz-Beltrán et al., )
Figure 6: Case 4: Diversity of LMS course configurations using the Shannon Diversity Index (0–2.5), where higher values indicate a wider mix of activity types. Results are grouped by teaching methodology (LECT, PJBL, PRBL, FLIP, CASE) and compare courses with high versus low workload satisfaction (“Top Courses” vs. “Bottom Courses”). ( Marques et al., )
| Participant | Role and Experience | Use of Institutional Data |
|---|---|---|
| P1 (Case 3 & 4) | LMS support specialist with 10+ years of experience. | Frequently reviews LMS data. |
| P2 (Case 1 & Case 4 ) | Education specialist in teaching innovation and training with 10+ years of support experience. | Checks institutional data occasionally. |
| P3 (Case 1 & 2) | Teaching quality data support specialist with 20+ years of experience. | Frequently reviews institutional data. |
| P4 (Case 1 & 4) | Education specialist in teaching innovation with 16+ years of support experience. | Frequently reviews institutional data. |
| P5 (Case 2 & 3) | LMS and pedagogy support manager with 2+ years of experience. | Frequently reviews institutional data. |
| P6 (Case 2 & Case 3 ) | Academic leader in teaching quality and innovation, with 15+ years teaching and 4+ years management experience. | Frequently reviews institutional data. |
Table I: Participant Profiles. In bold are the participants and selected narratives.
Figure 7: User interface of the chatbot
Figure 8: Flowchart describing the conversational flow between user and the chatbot
Figure 9: Transition Network Analysis considering all cases. Node size reflects the prominence of coded actions in the overall network, and edge labels represent relative transition probabilities between coded states.
| Case X + Case Y | 1 o Cases | 2 o Cases | ||||
|---|---|---|---|---|---|---|
| Without | With | Without | With | Without | With | |
| Average | 1.41 | 3.45 | 1.73 | 3.55 | 1.09 | 3.36 |
| Standard deviation | 0.80 | 0.86 | 0.65 | 0.82 | 0.83 | 0.92 |
| Variance | 0.63 | 0.74 | 0.42 | 0.67 | 0.69 | 0.85 |
| Median | 1.50 | 4.00 | 2.00 | 4.00 | 1.00 | 4.00 |
Table II: Descriptive statistics for the number of sub-factors addressed without and with the FACTRIA-aware chatbot, computed across all cases (Case X + Case Y), using only participants’ first assigned case (1 o Cases) and using only participants’ second assigned case (2 o Cases).
| Reference | Description |
|---|---|
| Radchenko (2020) | Described the aspects that may influence student satisfaction and the effects that result in biases. The author argues that various factors, such as gender and course level, influence data collection on student satisfaction. |
| Marques et al. (2024) | Described the importance of considering reliability to analyze the data collection. Also investigated gender biases by studying different factors such as age, level of the course, and academic centers. |
| Clayson (2009) | Examined previous research on the connection between learning and student satisfaction. The findings indicated that, on average, there is only a weak relationship between the two variables, depending on factors such as context, discipline, course type, and instructor. |
| Ortiz-Beltrán et al. (2022) | Evaluated how changes in teaching modalities influenced student satisfaction scores before, during, and after COVID-19. |
| Rosen (2018) | Investigated gender and the impact of teaching evaluations across various academic departments and disciplines. |
| Aragón et al. (2023) | Investigated the impact of department gender composition on student satisfaction scores. |
Table III: Summary of studies that influenced the development of FACTRIA framework
| Factor | Description of the factor | Sub-factor | Description of the sub-factor | Examples |
| Analytics Pipeline Factors | Process related to the institutional analytics approach applied in a determined problem. | Data quality | Processes related to gathering, preparing, and structuring data for analysis (pre-processing). | Data collection, feature extraction, sample, data reliability, level of data aggregation. |
| Types of data analysis | Approaches and assumptions used to interpret and extract insights from data. | Selection of analysis techniques, visualization, analytical context, assumptions of the data analyses. | ||
| Institutional Factors | Factors related to the institutional level in Higher Education Institutions. | Academic centers | Structural and demographic characteristics of academic departments and centers. | Academic centers (e.g., Economics, Engineering, Humanities); department gender composition (distribution of professors by gender in each department). |
| Educational framework | Guidelines for teachers in higher education. | Teaching methodologies framework. | ||
| Modality of the university | Refers to the different ways a university operates. | Brick-and-mortar university or online university. | ||
| Course Factors | Factors related to the course level in Higher Education Institutions. | Course level, class size | Aspects regarding the course. | Class size, level of the course. |
Table IV: Institutional Analytics Framework
| No. | Item |
|---|---|
| 1 | Did the system help you to get a more responsible interpretation? |
| 2 | I think that I would like to use this system frequently |
| 3 | I found the system unnecessarily complex |
| 4 | I thought the system was easy to use |
| 5 | I think that I would need the support of a technical person to be able to use this system |
| 6 | I found the various functions in this system were well integrated |
Table V: System Usability Survey
| Code | SubCode | Definition and Examples |
|---|---|---|
| Agreement | Confirmation | A consensus or affirmation regarding a specific statement or piece of information. User: “Yes, of course, the score could not be representative if the questionnaire is filled by a few students.” Assistant: “Yes, the ‘Bottom Courses’ are those with the lowest student enrollment.” |
| Validation | Acknowledgment that the provided information is accurate. Assistant: “That is an interesting observation about how student and professor gender might relate to satisfaction scores.”; “Your interpretation effectively explores how the gender of both students and professors might influence satisfaction results across the faculties.” | |
| Disagreement | When there is a disagreement between the user and the assistant regarding a statement or piece of information. User: “I don’t think the academic unit influences, but the course factors.”; “there is no information on the course level.” | |
| Express Doubts | When there is ambiguity/doubts about the meaning of a statement or piece of information for either side. User: “I don’t understand what you mind”; “The visualization is comparing the shannon index by teaching methodologies, but for me it is not clear the objective of the visualization.” | |
| Provide Information (PI) | Clarification | When either side clarifies a statement or piece of information. Assistant: “Data Quality examines whether the aggregated scores accurately represent the full range of individual satisfaction responses.” ; User: “just faculty types.” |
| Explanation | When there is an explanation from either side about a statement or piece of information. Assistant: “The way a methodology is implemented certainly affects the relationship between workload and student performance.”; User: “On the other hand for the explorative purposes it is indeed helpful such presentation.” |
Table VI: Coding Scheme for user interactions.
| Participant | Assigned case (max number of sub-factors) | Without | With |
|---|---|---|---|
| P1 | CASE 3 (5 sub-factors) | 1 | 3 |
| CASE 4 (4 sub-factors) | 0 | 4 | |
| P2 | CASE 1 (4 sub-factors) | 2 | 4 |
| CASE 4 (4 sub-factors) | 1 | 4 | |
| P3 | CASE 1 (4 sub-factors) | 1 | 2 |
| CASE 2 (5 sub-factors) | 0 | 3 |
Table VII: Number of sub-factors addressed without and with the use of the FACTRIA-aware chatbot.
Figure 10: Transition Network Analysis Case 1.
Figure 11: Transition Network Analysis Case 2.
Figure 12: Transition Network Analysis Case 3.
Figure 13: Transition Network Analysis Case 4.