Inference Energy and Latency in AI-Mediated Education: A Learning-per-Watt Analysis of Edge and Cloud Models

Kushal Khemani

cs.CYMar 4, 2026

Inference Energy and Latency in AI-Mediated Education: A Learning-per-Watt Analysis of Edge and Cloud Models

Authors: Kushal Khemani

Organizations: Billabong High International School, Hadapsar, Pune, 411028, Maharashtra, India.

Abstract

Immediate feedback is a foundational requirement of effective AI-mediated learning, yet the energy and latency costs of delivering it remain largely unexamined. This study investigates the latency-energy-learning trade-off in AI tutoring through an empirical comparison of two on-device inference configurations of Microsoft Phi-3 Mini (4k-instruct) on an NVIDIA T4 GPU: full-precision FP16 and 4-bit NormalFloat (NF4) quantisation. Both were evaluated under KV-cache-enabled inference across 500 educational prompts spanning five secondary school subject domains. Pedagogical quality was assessed for each of the 1000 generated responses by a hybrid panel of 10 Cambridge International teachers and three frontier AI systems using a four-dimension rubric. We introduce Learning-per-Watt (LpW), a novel metric quantifying pedagogical value per unit of energy over the learner's waiting window. Under realistic deployment, NF4 achieves lower per-inference energy than FP16 (329 J vs. 369 J) but higher latency (13.4 s vs. 9.2 s), yielding a modest FP16 advantage in LpW of 1.33x at a quality difference of 0.19 points. Under cache-disabled inference -- used in offline evaluation but absent from real deployments -- the gap widens to 7.4x, overstating the FP16 advantage by more than fivefold. Quantisation efficiency is hardware-dependent and inference-regime dependent, with significant implications for equitable AI tutoring deployment in low-resource settings.

Figures & tables

Dimension	Code	What is assessed
Conceptual Accuracy	CA	Factual correctness; absence of critical misconceptions that would mislead a learner
Clarity & Coherence	CC	Logical structure and readability of the explanation
Scaffolding Quality	SQ	Progressive knowledge building; use of examples, analogies, and step-by-step reasoning
Level Appropriateness	LA	Suitability of language and depth for secondary school learners (ages 14–18)

Table 1: Pedagogical scoring rubric applied by all 13 raters.

Weighting scheme	Mean Qped		Mean LpW ( $\times 10^{-3}$ )		Ratio
(CA / CC / SQ / LA)	FP16	NF4	FP16	NF4	FP16/NF4
Equal (0.25/0.25/0.25/0.25)	8.238	8.047	2.499	1.877	1.331 $\times$
CA+SQ heavy (0.35/0.15/0.35/0.15)	8.166	7.973	2.477	1.860	1.332 $\times$
CA dominant (0.40/0.20/0.25/0.15)	8.230	8.038	2.496	1.875	1.331 $\times$
SQ dominant (0.25/0.20/0.40/0.15)	8.116	7.917	2.461	1.846	1.333 $\times$
LA downweighted (0.30/0.30/0.30/0.10)	8.158	7.955	2.474	1.855	1.334 $\times$

Table 2: Sensitivity of Learning-per-Watt to rubric dimension weighting (cache-enabled,

n=500

per configuration). The FP16/NF4 LpW ratio ranges from

1.329\times

1.334\times

across all schemes, confirming that the efficiency conclusion is invariant to the choice of dimension weights.

Dimension	FP16		NF4
	$\alpha$	ICC(2,1)	$\alpha$	ICC(2,1)
Conceptual Accuracy (CA)	0.26	0.30	0.17	0.19
Clarity & Coherence (CC)	0.49	0.51	0.36	0.39
Scaffolding Quality (SQ)	0.45	0.47	0.32	0.34
Level Appropriateness (LA)	0.10	0.15	0.08	0.11
Overall	0.41	0.42	0.30	0.35

Table 3: Human-only inter-rater reliability (10 teachers).

Dimension	FP16		NF4
	$\alpha$	ICC(2,1)	$\alpha$	ICC(2,1)
Conceptual Accuracy (CA)	0.35	0.41	0.09	0.18
Clarity & Coherence (CC)	0.29	0.33	0.12	0.22
Scaffolding Quality (SQ)	0.23	0.28	0.05	0.15
Level Appropriateness (LA)	0.09	0.14	0.04	0.11
Overall	0.24	0.29	0.08	0.17

Table 4: AI-only inter-rater reliability (3 frontier models).

Dimension	FP16		NF4
	$\alpha$	ICC(2,1)	$\alpha$	ICC(2,1)
Conceptual Accuracy (CA)	0.21	0.39	0.04	0.33
Clarity & Coherence (CC)	0.42	0.45	0.23	0.37
Scaffolding Quality (SQ)	0.38	0.41	0.07	0.34
Level Appropriateness (LA)	0.10	0.28	0.05	0.31
Overall	0.37	0.42	0.18	0.35

Table 5: Combined inter-rater reliability (10 human + 3 AI raters).

Aggregation	FP16 LpW	NF4 LpW	FP16/NF4 Ratio
Human-only	$2.43\times 10^{-3}$	$1.82\times 10^{-3}$	1.34
AI-only	$2.61\times 10^{-3}$	$1.96\times 10^{-3}$	1.33
60/40 Weighted	$2.50\times 10^{-3}$	$1.88\times 10^{-3}$	1.33

Table 6: Sensitivity of Learning-per-Watt (LpW) to rater aggregation (cache-enabled,

n=500

per configuration).

Config	Precision	Mean	Mean Net	Mean	Mean LpW
		Latency (s)	Energy (J)	$\boldsymbol{Q_{\mathrm{ped}}}$	$\boldsymbol{(\times 10^{-3})}$
FP16 (edge)	16-bit	9.17	368.8	8.24	2.50
NF4 (edge)	4-bit	13.36	329.0	8.05	1.88
Ratio (FP16 / NF4)		$1.46\times$ faster	$10.8\,\%$ higher	$+0.19$	$1.33\times$ higher

Table 7: Aggregate on-device results: FP16 vs. NF4 (Phi-3 Mini, 500 prompts, NVIDIA T4, use_cache=True ).

	FP16				NF4
Category	Lat.	Energy	$Q_{\mathrm{ped}}$	LpW	Lat.	Energy	$Q_{\mathrm{ped}}$	LpW
	(s)	(J)		$(\times 10^{-3})$	(s)	(J)		$(\times 10^{-3})$
Mathematics	9.30	372.9	8.20	2.46	13.55	328.2	8.09	1.84
Science	9.11	366.8	8.27	2.60	13.40	333.2	8.01	1.82
Programming-CS	9.11	367.2	8.19	2.50	13.34	332.7	7.93	1.89
Humanities	9.14	367.5	8.27	2.48	13.20	319.9	8.12	1.97

Table 8: Per-category mean latency, energy,

Q_{\mathrm{ped}}

, and LpW for FP16 and NF4 configurations ( use_cache=True ,

n=100

prompts per category).

Configuration	Location	Mean	Client-side	DC Energy	LpW
		Latency (s)	Energy (J)	Included?	Basis
FP16 (Phi-3 Mini)	Edge T4	9.2	368.8	Yes	Fully measured
NF4 (Phi-3 Mini)	Edge T4	13.4	329.0	Yes	Fully measured
Cloud LLM (GPT-4)	Cloud API	2–4	$\sim$ few J	No	Estimated only

Table 9: Edge vs. cloud configurations: latency, client-side energy, and LpW implications.

Figure 1: Learning-per-Watt (LpW) distributions for on-device FP16 and NF4 configurations (Phi-3 Mini, NVIDIA T4 GPU, use_cache=True , $n=500$ prompts per configuration). (a) Frequency histogram of per-prompt LpW values. The NF4 distribution (red) is centred near $1.88\times 10^{-3}$ (J s) -1 ; the FP16 distribution (blue) near $2.50\times 10^{-3}$ (J s) -1 . The distributions partially overlap, in contrast to the complete separation observed under cache-disabled conditions (Appendix C). (b) Per-category box plots showing IQR and median LpW for each subject domain. The FP16–NF4 gap is consistent across all five categories, confirming that deployment precision rather than prompt domain is the primary determinant of LpW.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Model	VRAM	Mean	Mean Net	Mean LpW	Outcome
	(GB)	Lat. (s)	Energy (J)	$(\times 10^{-4})$
Phi-3 Mini 4B	7.2	16.73	652	7.55	Selected
Llama-3 8B Instruct	—	—	—	—	OOM
Mistral 7B Instruct	13.4	81.93	3,687	0.26	High latency
Gemma 7B Instruct	—	—	—	—	Load error
TinyLlama 1.1B Chat	2.2	7.51	432	24.6	Quality issues

Appendix

Table 10: Model selection experiment results on NVIDIA T4 GPU (FP16,

n=20

prompts). Phi-3 Mini figures are drawn from the main FP16 dataset on matching prompt IDs. LpW is computed using a reference

Q_{\mathrm{ped}}=8.24

for all models to isolate system-level efficiency. OOM = out-of-memory failure during loading.

Model	Server Energy	Energy	Client	Source
	(Wh/query)	(J)	Latency (s)
GPT-4o	0.43 $\pm$ 0.13	$\approx$ 1,550	1.9	Jegham et al. (2025) ; Vellum AI (2024)
Claude-3.5 Sonnet	n/a (eco-efficient)	est. $<$ 1,550	3.8	Jegham et al. (2025) ; Vellum AI (2024)
Gemini-1.5 Pro	0.24 a	$\approx$ 864	2–5	Google Cloud (2025)
a Google Cloud (2025) reports median per-query energy for Gemini models inclusive of PUE.
n/a = not publicly disclosed by provider.

Appendix

Table 11: Literature-derived server-side energy and client-side latency for cloud LLMs serving a short educational prompt (

\leq

200 output tokens). Energy figures are server-side estimates inclusive of PUE overhead; client-side energy is negligible (

<

1 J) and excluded. Latency figures are independently measured API response times.

Metric	Formula	FP16 mean	NF4 mean	Ratio	FP16 $>$ NF4?
LpW (primary)	$Q_{\mathrm{ped}}/(E\cdot L)$	$2.50\times 10^{-3}$	$1.88\times 10^{-3}$	$1.33\times$	✓
QpJ (energy only)	$Q_{\mathrm{ped}}/E$	$2.25\times 10^{-2}$	$2.46\times 10^{-2}$	$0.91\times$	×
QpS (latency only)	$Q_{\mathrm{ped}}/L$	$9.04\times 10^{-1}$	$6.06\times 10^{-1}$	$1.49\times$	✓
LpW geo (geometric)	$Q_{\mathrm{ped}}/\sqrt{E\cdot L}$	$1.43\times 10^{-1}$	$1.22\times 10^{-1}$	$1.17\times$	✓
QpJ = quality-per-Joule; QpS = quality-per-second; LpW geo uses geometric-mean denominator.

Appendix

Table 12: Sensitivity analysis: FP16 vs. NF4 under four alternative composite metrics (

n=500

prompts per configuration, use_cache=True ). QpJ favours NF4, confirming that NF4 achieves lower per-inference energy. Metrics that include latency all favour FP16. The direction and magnitude of the gap depends on which constraint is weighted.

Figure 2: Sensitivity and scenario analysis. (a) FP16/NF4 ratio under four alternative composite metrics ( use_cache=True ). QpJ favours NF4 ( $0.91\times$ ), reflecting NF4’s lower per-inference energy; all metrics that include latency favour FP16, ranging from $1.17\times$ (LpW geo ) to $1.49\times$ (QpS). (b) Cloud LpW under five server-energy scenarios (log scale), with on-device FP16 (blue dashed) and NF4 (red dotted) reference lines from the cache-enabled experiment. The highlighted bar (orange) is the central GPT-4o short-prompt estimate from Jegham et al. (2025) . Only the short-prompt central estimate places cloud LpW above on-device FP16; all higher-energy scenarios fall below both edge configurations.

Config	Cache	Lat. (s)	Energy (J)	$Q_{\mathrm{ped}}$	LpW
FP16	ON (primary)	9.17	368.8	8.24	$2.50\times 10^{-3}$
FP16	OFF (secondary)	16.47	648.0	7.97	$7.47\times 10^{-4}$
NF4	ON (primary)	13.36	329.0	8.05	$1.88\times 10^{-3}$
NF4	OFF (secondary)	49.40	1,882	7.89	$8.49\times 10^{-5}$
FP16/NF4 ratio — cache ON		$1.46\times$ faster	$10.8\,\%$ higher	$+0.19$	$1.33\times$
FP16/NF4 ratio — cache OFF		$3.00\times$ faster	$2.90\times$ higher	$+0.08$	$8.80\times$

Appendix

Table 14: Cache regime comparison: use_cache=True (primary study) vs. use_cache=False (secondary experiment),

n=500

prompts per configuration. LpW computed using empirically scored

Q_{\mathrm{ped}}

in both cases.

	FP16			NF4
Category	ON	OFF	Ratio	ON	OFF	Ratio
	$(\times 10^{-4})$	$(\times 10^{-4})$	ON/OFF	$(\times 10^{-4})$	$(\times 10^{-4})$	ON/OFF
Mathematics	24.6	7.35	$3.35\times$	18.4	0.783	$23.5\times$
Science	26.0	7.57	$3.44\times$	18.2	0.858	$21.2\times$
Programming-CS	25.0	7.37	$3.39\times$	18.9	0.855	$22.1\times$
Humanities	24.8	7.45	$3.33\times$	19.7	0.908	$21.7\times$

Appendix

Table 15: Per-category LpW (

\times 10^{-4}

) under cache=ON and cache=OFF for FP16 and NF4. The NF4 improvement from cache=OFF to cache=ON is consistently larger than the FP16 improvement across all five categories.

Figure 3: Cache regime comparison across three dimensions for FP16 and NF4 configurations of Phi-3 Mini on the NVIDIA T4 ( $n=500$ prompts per configuration). Dark bars show primary study results ( use_cache=True ); light bars show the secondary cache-disabled experiment ( use_cache=False ). (a) Mean latency: enabling the cache reduces FP16 latency by $1.80\times$ and NF4 latency by $3.70\times$ . (b) Mean net energy: the cache reduces FP16 energy by $1.76\times$ and NF4 energy by $5.72\times$ . (c) Mean LpW (log scale): the combined effect produces a $3.35\times$ LpW improvement for FP16 and a $22.1\times$ improvement for NF4, compressing the FP16–NF4 efficiency gap from $8.80\times$ (cache=OFF) to $1.33\times$ (cache=ON).

Category	Mean Latency (s)		Mean Tokens/s
	F16	Q4_K_M	F16	Q4_K_M
Mathematics	64.90	29.09	3.14	6.94
Science	70.21	27.61	2.85	7.24
Programming-CS	72.13	27.19	2.79	7.36
Humanities	72.14	25.83	2.78	7.74
Meta-cognition	67.00	25.59	3.03	7.82

Appendix

Table 16: Laptop inference results: F16 vs Q4_K_M on Intel Core i7 (11th gen) with Iris Xe, CPU-only backend (

n=100

prompts, 20 per category). Q4_K_M is

2.56\times

faster than F16—the opposite of the T4 result, where NF4 was

1.46\times

slower than FP16.

Laptop energy estimated at 20 W sustained package power; LpW ratio is power-invariant (see Section D.4).
Platform	Arch.	Config	Mean Lat. (s)	Energy (J)	LpW ( $\times 10^{-3}$ )	Ratio
NVIDIA T4	Turing GPU	FP16	9.17	368.8	2.50	$1.33\times$ (FP16)
NVIDIA T4	Turing GPU	NF4	13.36	329.0	1.88	$1.33\times$ (FP16)
Intel i7 11th gen	CPU (Iris Xe)	F16	69.28	est. 1,385 J	est. 0.088	${\approx}6.4\times$ (Q4)
Intel i7 11th gen	CPU (Iris Xe)	Q4_K_M	27.06	est. 541 J	est. 0.561	${\approx}6.4\times$ (Q4)
T4 values from primary study ( $n=500$ ). Laptop values from this appendix ( $n=100$ ).

Appendix

Table 17: Cross-platform comparison of the full-precision vs quantised efficiency relationship. The direction of the LpW advantage reverses between platforms: FP16/NF4 favours FP16 on the T4 (GPU, no native INT4); Q4_K_M/F16 favours quantisation on the laptop CPU. Both outcomes reflect the same underlying principle: quantisation efficiency depends on whether memory bandwidth or dequantisation overhead is the binding constraint.

Power-Efficient Inference

See all 14 papers

A Measurement Study of LLM Inference Trade-offs Across Edge Continuum Hardware
Sep 8, 2026Maysam Khatib, Moysis Symeonides, Demetris Trihinas +2LLM Inference Efficiency LLM Evaluation
University of Cyprus, Nicosia, Cyprus · University of Nicosia, Nicosia, Cyprus
Intelligence per Watt: Measuring Intelligence Efficiency of Local AI
Nov 11, 2025Jon Saad-Falcon, Avanika Narayan, Hakki Orhun Akengin +12LLM Inference Efficiency AI Accelerator Inference
Stanford University · Together AI
The Battery Price of edge AI: A study of the Environmental Impact of LLM Inference on Mobile Devices
Sep 14, 2026Édouard Guégain, Tristan CoignionOn-Device Language Model Inference Energy-Efficient ML
DRI, GREENSPECTOR · Univ. Bordeaux, CNRS, Bordeaux INP, LaBRI, UMR 5800, Talence, France

Sep 8, 2026J/K move · Enter open · S save

A Measurement Study of LLM Inference Trade-offs Across Edge Continuum Hardware

Maysam Khatib, Moysis Symeonides, Demetris Trihinas, George Pallis+1

University of Cyprus, Nicosia, Cyprus · University of Nicosia, Nicosia, Cyprus

LLM Inference Efficiency LLM Evaluation LLM Inference Edge Computing

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly used as backends for intelligent web services, but serving them across the edge continuum requires balancing quality, latency, model footprint, and energy. This paper presents a controlled measurement study of self-hosted LLM inference across edge and near-edge deployment nodes: an NVIDIA Jetson AGX Orin and a near-edge server with CPU-only and GPU-enabled inference modes. We evaluate multiple open-weight LLMs and quantization variants using a fixed question-answering workload, and compare them against GPT-4o as a cloud-hosted accuracy and latency reference. Our benchmarking pipeline reports accuracy, model footprint, per-token decoding latency, prefill latency, and overall execution energy. The results show that GPU-enabled server execution provides the lowest compute-side latency, while Jetson Orin shows lower measured energy, consistent with its lower platform power under our setup. CPU-only execution is consistently dominated in latency for our workload and shows higher measured energy. We also show that parameter count and downloaded weight-file size alone do not reliably predict observed accuracy or latency. Finally, using Pareto-frontier analysis, we study how deployment decisions may change under possible streamed-token delivery overheads, highlighting that compute-side inference metrics alone can lead to suboptimal placement for latency-sensitive interactive web services.

Abstract

Power-Efficient Inference

Explore similar work

A Measurement Study of LLM Inference Trade-offs Across Edge Continuum Hardware

Intelligence per Watt: Measuring Intelligence Efficiency of Local AI

The Battery Price of edge AI: A study of the Environmental Impact of LLM Inference on Mobile Devices