Canada's 2026 national AI strategy, AI for All, commits to free AI training for everyone and to expanding business AI adoption, yet it is difficult to anticipate whether these policies will reach adults with uneven skills and circumstances. We ask how adults with diverse skills might rate these policies on support, effectiveness, and personal benefit, how their skills and circumstances shape these evaluations, and what implementation conditions they raise. We built 3,200 generative agents from a weighted sample of the Programme for the International Assessment of Adult Competencies (PIAAC) Cycle 2 respondents, collected their ratings and explanations for 20 policies, and analyzed dialogues between agents with contrasting ratings. The agents supported nearly every policy but were far less confident that they would improve adult AI literacy or benefit them personally. This gap widened as digital technology use declined: agents modeled on adults who had never used a digital device still endorsed the policies but rarely expected to benefit. Agents traced the gap to participation costs, hard-to-exercise rights, and limited community control, and their dialogues specified these conditions rather than resolving disagreement. We describe this pattern as conditional endorsement and argue that AI literacy policy should be designed and evaluated for adults with the fewest digital skills.
Figures & tables
Figure 1. An overview of the research design. Personas built from PIAAC responses evaluated the 20 policies identified in AI for All , and agents with contrasting ratings on the same policy, selected by the rating distance D , then discussed it in dyads. We analyzed the rating distributions, induced concepts from responses and transcripts with LLooM, and used an XLM-RoBERTa model trained on single-agent responses to estimate rating changes within each dialogue.
Figure 2. Joint distribution of support with literacy effectiveness (left) and direct benefit (right) among the 63,875 responses with all three ratings. Cell values are percentages of these responses (values below 0.05% omitted), and outlined cells mark upper-half support combined with a lower-half rating.
Characteristic
Level
Support (%)
Effect. (%)
Benefit (%)
Digital use at home
Never used a device
90.4
13.2
12.3
Lower
99.4
31.0
47.0
Typical
99.7
37.5
57.7
Higher
99.8
39.1
63.4
Employment status
Out of the labour force
98.7
32.4
45.3
Unemployed
99.3
37.0
52.6
Table 1. Share of valid ratings in the upper half of the scale (4–6) across all 20 policies, by persona characteristic. Levels of digital technology use at home are relative to the weighted sample.
Figure 3. Acknowledgment followed by specification in three dyads. Each column runs from the lower-rating speaker’s opening concern to mechanisms the two speakers added in later turns.
Figure 4. Conditional endorsement. Agents endorsed a policy’s public aim but qualified whether it would reach adults, or the persona, through three delivery conditions. The personas’ skills and circumstances moderated expected reach and benefit, and dyadic dialogue mainly specified the delivery conditions.
Appendix figures & tables29 assets
Supplementary material from the paper’s appendix.
Appendix
Step
n remaining
Excluded
All records
11,697
Not a doorstep interview
11,607
90
Completed all three cognitive domains
11,607
0
Administered the Effort module
10,455
1,152
Answered the self-assessment question
10,455
0
Analytic sample
10,455
1,242
Appendix
Table 2. Derivation of the self-efficacy analytic sample.
Figure 5. Weighted distribution of self-assessed performance bands, with 95% confidence intervals.
Figure 6. Pooled GCPI proficiency by self-assessed performance band, with 95% confidence intervals (points) and the 25th–75th percentile range (shaded).
Term
Estimate
SE
p -value
GCPI
1.383
0.057
< 0.001
Appendix
Table 3. Model 0 regression output, an ordered logit of self-assessment on GCPI alone (log-odds per SD).
Term
Estimate
SE
p -value
GCPI
1.278
0.062
< 0.001
ICT use at home
0.057
0.041
0.164
Conscientiousness
0.055
0.038
0.145
Openness
0.085
0.039
0.028
Emotional stability
0.064
0.041
0.116
Extraversion
0.125
0.041
0.003
Appendix
Table 4. Model 1 regression output with all ordered-logit coefficients, in log-odds per SD for continuous predictors and versus the omitted reference for factors (age 35–44, upper secondary education, non-immigrant, male, excellent health).
Persona
Estimate
SE
All adults
10.2
0.4
Men 25–34, bachelor’s degree
14.6
1.3
Men 55 and over, bachelor’s degree
14.5
1.3
Men 24 or less, upper secondary
14.5
1.7
Women 25–34, bachelor’s degree
9.7
0.8
Women 24 or less, upper secondary
9.6
1.2
Appendix
Table 5. Predicted share over-confident relative to measured skill for persona profiles (%), averaged over the observed distribution of every other characteristic.
Step
n remaining
Excluded
All records
10,455
ICT use at home reported
10,356
99
Conscientiousness scale derived
10,322
34
Openness scale derived
10,222
100
Emotional stability scale derived
10,210
12
Extraversion scale derived
10,196
14
Appendix
Table 6. Model 1 exclusion ledger, with covariate completeness layered on the analytic sample.
Step
n remaining
Excluded
All records
9,883
ICT use at work reported
7,458
2,425
Occupation in a scored ISCO major group
6,840
618
Task discretion reported
6,838
2
Learning at work reported
6,825
13
Analytic sample
6,825
3,058
Appendix
Table 7. Model 2 exclusion ledger, layered on Model 1’s retained sample so each row shows only what the new rule drops.
Term
Estimate
SE
p -value
GCPI
1.352
0.074
< 0.001
ICT use at home
0.028
0.054
0.602
Conscientiousness
0.032
0.053
0.546
Openness
0.030
0.051
0.551
Emotional stability
0.108
0.059
0.064
Extraversion
0.139
0.052
0.007
Appendix
Table 8. Model 2 regression output, adding the work-context terms to Model 1 on the employed subsample (log-odds per SD). Factor references as in Model 1.
GCPI quintile
Estimate
SE
Quintile 1
10.6
1.4
Quintile 2
10.7
1.3
Quintile 3
9.8
0.8
Quintile 4
10.1
0.5
Quintile 5
9.9
0.3
Appendix
Table 9. Share of adults over-confident relative to their measured skill, by GCPI quintile (%).
Characteristic
Group
Estimate
SE
Gender
Men
12.1
0.6
Women
8.3
0.4
Age
24 or less
9.9
0.9
25–34
9.3
0.7
35–44
10.7
0.7
45–54
10.3
1.2
Appendix
Table 10. Share of adults over-confident relative to their measured skill, by characteristic (%).
Term
Estimate
SE
p -value
GCPI
− 0.224
0.052
< 0.001
ICT use at home
0.026
0.041
0.522
Conscientiousness
0.060
0.038
0.113
Openness
0.003
0.034
0.933
Emotional stability
0.005
0.047
0.909
Extraversion
0.047
0.040
0.249
Appendix
Table 11. Persona model, a fractional logit of the over-confidence probability on measured proficiency and the Model 1 covariates, in log-odds per SD for continuous predictors and versus the Model 1 reference levels for factors.
Proficiency benchmark
Estimate
SE
Composite (GCPI)
1.278
0.062
Literacy
1.107
0.062
Numeracy
1.198
0.056
Adaptive problem solving
1.108
0.054
Appendix
Table 12. The Model 1 proficiency coefficient under each benchmark, the composite GCPI versus each domain alone in its place.
Group
Estimate
SE
Dropped
233.3
3.0
Retained
273.9
0.7
Appendix
Table 13. Measured proficiency of respondents dropped by Effort-module non-administration versus those retained. The sample is positively selected on ability.
Domain
Estimate
SE
Literacy
0.654
0.010
Numeracy
0.678
0.008
Adaptive problem solving
0.652
0.009
Appendix
Table 14. Share of the analytic sample administered each cognitive domain directly, rather than by imputation from the background questionnaire.
Predictor
VIF
Most correlated with
∣r∣
GCPI
1.12
Assessment effort
0.19
ICT use at home
1.06
Extraversion
0.15
Conscientiousness
1.35
Emotional stability
0.35
Openness
1.08
Extraversion
0.17
Emotional stability
1.26
Conscientiousness
0.35
Extraversion
1.24
Conscientiousness
0.34
Appendix
Table 15. Variance-inflation factors and the largest absolute weighted correlation for each continuous predictor. No pair is collinear.
Term
Custom est.
Custom SE
svyolr est.
svyolr SE
GCPI
1.271
0.065
1.271
0.063
ICT use at home
0.053
0.039
0.053
0.039
Conscientiousness
0.052
0.037
0.052
0.037
Openness
0.079
0.038
0.079
0.038
Emotional stability
0.065
0.040
0.065
0.039
Extraversion
0.133
0.041
0.133
0.041
Appendix
Table 16. Cross-check of the custom replicate engine against survey::svyolr on one plausible value. Point estimates match and standard errors agree closely.
Threshold
Estimate
SE
Band ≥ 2
1.249
0.091
Band ≥ 3
1.293
0.069
Band ≥ 4
1.400
0.066
Band ≥ 5
1.470
0.081
Appendix
Table 17. The GCPI slope estimated separately at each band threshold, a check on the proportional-odds assumption. The slopes stay close, so the single ordered-logit summary is fair.
Band
Observed
SE
Predicted
SE
1
3.9
0.3
3.4
0.3
2
7.3
0.4
6.7
0.4
3
19.7
0.7
18.8
0.7
4
34.6
0.9
35.5
0.9
5
34.5
0.8
35.6
0.9
Appendix
Table 18. Predicted versus observed weighted share (%) in each self-assessment band, Model 1.
Ordered logit
Linear
Term
Estimate
SE
Estimate
SE
ICT use at home
0.057
0.041
0.026
0.016
Conscientiousness
0.055
0.038
0.021
0.015
Openness
0.085
0.039
0.028
0.015
Emotional stability
0.064
0.041
0.012
0.015
Extraversion
0.125
0.041
0.052
0.016
Appendix
Table 19. Model 1 as an ordered logit (log-odds) and as a linear regression of the band score (bands), side by side.
Policy
Description as presented to agents
Consumer privacy reform
The Government of Canada announces plans to modernize consumer privacy legislation. The proposed changes would recognize privacy as a fundamental right, strengthen people’s control over their personal information, and protect children’s information from exploitation. The government also plans to address inappropriate uses of personal data, including surveillance pricing.
Online safety legislation
The Government of Canada announces plans to introduce online safety laws addressing harms associated with digital technologies and AI. Its stated priorities include protecting children, providing legal tools to address deepfakes, making chatbot interactions safer, and holding those responsible for online harms accountable. It also commits to protecting elections against AI-enabled misinformation and foreign interference.
AI safety evaluation and certification
The Government of Canada announces plans to create a Canada Trusted AI Certification program intended to help people identify trustworthy AI products. It also commits CAD 50 million to expand the Canadian AI Safety Institute’s work on emerging risks, technical research, and transparent evaluations of AI models.
Identifying AI systems and generated content
The Government of Canada announces that it will work on measures to make AI use more transparent, including capabilities such as watermarking AI-generated content. The stated aim is to help people recognize when they are interacting with an AI system or encountering AI-generated material.
Free introductory AI education
The Government of Canada announces plans for a National AI Literacy Initiative offering free introductory AI education accessible to all Canadians. Public libraries and community organizations would help deliver learning opportunities, including in rural, remote, and northern communities. The initiative would cover practical AI use as well as recognizing bias, misinformation, privacy risks, and unsafe uses.
AI tools for postsecondary students
The Government of Canada announces a commitment to give all postsecondary students access to trusted AI agents. It also plans to bring AI literacy content to one million students entering postsecondary education and provide classroom AI learning resources and training to more than 3,000 educators.
Appendix
Table 20. The 20 policies identified in AI for All .
Challenge
Description as presented to agents
Low public trust and AI literacy
Many Canadians have limited AI understanding, training, or confidence. The strategy reports that 24% of Canadians have received AI training.
Employment disruption
Workers may need to reskill as jobs change, entry-level roles contract, and some female-dominated sectors face particular exposure.
Privacy, discrimination, and online harms
Risks include misuse of personal information, algorithmic discrimination, deepfake sexual violence, fraud, misinformation, and threats to democratic institutions.
Unequal access and disproportionate harms
Indigenous peoples, racialized communities, and people with disabilities may face distinct barriers or harms, alongside cultural and linguistic representation concerns.
Business adoption barriers
Businesses face costs, limited expertise, and uncertainty about useful applications. The strategy reports that 78% of non-adopting firms do not see how AI benefits their goods or services.
Dependence on foreign technology
Reliance on foreign cloud providers, chips, infrastructure, and models raises concerns about control, foreign legal regimes, vendor lock-in, and technological choice.
Appendix
Table 21. The nine landscape challenges identified in AI for All .
Agent
Assigned landscape challenges
PIAAC-4362
Low public trust and AI literacy
PIAAC-1754
Low public trust and AI literacy; Unequal access and disproportionate harms; Infrastructure and environmental constraints; Talent retention and commercialization; Privacy, discrimination, and online harms
PIAAC-10565
Fragmented and inaccessible data; Privacy, discrimination, and online harms; Dependence on foreign technology; Unequal access and disproportionate harms; Infrastructure and environmental constraints; Talent retention and commercialization; Low public trust and AI literacy; Employment disruption
Appendix
Figure 8. Examples of the landscape challenges assigned to different agents.
Support
Effectiveness
Benefit
Method
Same
Higher
Lower
Higher
Lower
Higher
Lower
TF-IDF logistic regression
98.4
0.9
0.6
9.2
1.3
48.7
12.0
XLM-RoBERTa
86.2
12.9
0.8
30.6
4.8
36.2
8.3
Appendix
Table 22. Inferred opinion change between first and last turns (percentages of speaker trajectories).
Figure 9. Dyad opinion viewer: one response at a time, but both agents remain visible.
Figure 10. PIAAC-persona-based agents’ support for different AI policies. Bars show the percentage of valid ratings at each scale point; blue bars mark the upper half (4–6).
Figure 11. PIAAC-persona-based agents’ perceived effectiveness of different AI policies for improving adult AI literacy. Blue bars mark the upper half (4–6).
Figure 12. PIAAC-persona-based agents’ perceived direct benefit of different AI policies. Blue bars mark the upper half (4–6).
Policy
Support (%)
Benefit (%)
Effectiveness (%)
All 20 policies
99.5
56.4
35.9
Consumer privacy reform
100.0
98.5
10.7
Online safety legislation
100.0
87.1
5.3
AI safety evaluation and certification
100.0
80.4
62.1
Identifying AI systems and generated content
100.0
95.2
94.9
Free introductory AI education
100.0
88.3
100.0
Appendix
Table 23. Share of valid ratings in the upper half of the scale (4–6), by policy and dimension.
AI literacy develops people's ability to understand, use, evaluate, and shape AI systems responsibly across everyday, educational, and professional settings.
AI-powered educational technologies have demonstrated measurable benefits for learners, but their design and evaluation have largely centered on K-12 contexts. As a result, many AI-supported learning systems remain poorly aligned with the needs, constraints, and goals of adult learners. To better understand how AI systems function in adult education, this paper examines the deployment of several AI learning technologies developed within a multidisciplinary, national research institute in the United States focused on adult learning and online education. Drawing on longitudinal deployment data, we conducted a reflexive thematic analysis to identify recurring challenges and design considerations across systems. These insights were synthesized into a set of 19 design guidelines intended to inform future AI-supported adult learning technologies. We demonstrate the utility of these guidelines through a heuristic evaluation of the deployed systems. Lastly, we present a guideline exploration tool that aids in the ideation of technologies by connecting the guidelines to stakeholder statements surfaced in the analysis process.
Jennifer M. Reddig, Glen R. Smith, Sanaz Ahmadzadeh Siyahrood +16
Georgia Institute of Technology · Vanderbilt University · Georgia State University +1
Agent benchmarks evaluate policy compliance but assume each policy determines a unique correct action. Natural-language policies can violate this assumption through silence, ambiguity, or contradiction, admitting multiple defensible readings that a single gold trajectory cannot capture. Auditing two τ2-bench domains, we develop a taxonomy of such policy loopholes and show that affected tasks produce unreliable scores: they lower scores across different models in different ways and make every model less consistent across repeated trials. A cross-domain comparison reveals that exploitability requires both policy ambiguity and tool permissiveness: when policy complexity exceeds what tools can enforce, agents resolve gaps inconsistently and scores become unreliable. Policy specification quality sets the ceiling on evaluation quality. Benchmark developers should audit policies before collecting gold annotations.
Real-world requests to AI agents are fundamentally underspecified. Natural human communication relies on shared context and unstated constraints that speakers expect listeners to infer. Current agentic benchmarks test explicit instruction-following but fail to evaluate whether agents can reason about implicit requirements spanning accessibility needs, privacy boundaries, catastrophic risks, and contextual constraints. We present Implicit Intelligence, an evaluation framework testing whether AI agents can move beyond prompt-following to become genuine goal-fulfillers, paired with Agent-as-a-World (AaW), a harness where interactive worlds are defined in human-readable YAML files and simulated by language models. Our scenarios feature apparent simplicity in user requests, hidden complexity in correct solutions, and discoverability of constraints through environmental exploration. Evaluating 16 frontier and open-weight models across 205 scenarios, we find that even the best-performing model achieves only 48.3% scenario pass rate, revealing substantial room for improvement in bridging the gap between literal instruction-following and human-like contextual reasoning.