When Readability and Source Retention Diverge: An Evaluability Gap in AI Translation

Chenchen Mao; Hanjing Shi; Haiyan Jia; Emily Wegrzyn; Dominic DiFranzo

cs.HCAug 19, 2026

When Readability and Source Retention Diverge: An Evaluability Gap in AI Translation

Authors: Chenchen Mao, Hanjing Shi, Haiyan Jia, Emily Wegrzyn, Dominic DiFranzo

Organizations: Lehigh University, Bethlehem, Pennsylvania, United States

Abstract

Readable AI output can leave an evaluability gap: even when the source is shown, an overall-quality judgment may not reflect what an output preserves. We investigated how source-text condition and output rendering relate to perceived translation quality, and how output and system appraisals relate to trust and stated disclosure willingness in a plain-text interface. A focal 2 * 2 comparison (N=306) using TransLingo examined simple generated narratives and complex literary-philosophical prose alongside LLM-generated readability-oriented outputs and researcher-revised fidelity-oriented outputs. A descriptive stimulus audit indicated greater source retention in fidelity-oriented outputs in both source-text conditions. Factorial analyses showed a significant rendering-by-source-text-condition interaction in perceived quality. Participants rated fidelity-oriented outputs higher than readability-oriented outputs for the simple narratives, whereas no reliable rendering difference emerged for the complex prose. A corresponding source-condition-dependent pattern was observed for perceived intelligence, agency-oriented anthropomorphic attribution, and task-performance trust. A separate theory-ordered appraisal-structure SEM characterized concurrent associations among perceived quality, perceived intelligence, agency-oriented anthropomorphic attribution, task-performance trust, and stated disclosure willingness across six domains, with task-performance trust as the proximal correlate of stated willingness. The observed rating pattern distinguishes source access from source evaluability: for the complex stimuli, displaying the source did not ensure that one overall-quality rating reflected differences in retained content. It also separates support for evaluating translation output from data-handling support for decisions about what personal text to entrust to a system.

Figures & tables

Figure 1. Hypothesized model. Output rendering (readability-oriented vs. fidelity-oriented) is compared on perceived quality (RQ1a), with the rendering–quality relationship examined across the two source-text conditions (RQ1b). Perceived quality is expected to be positively associated with perceived intelligence (H1); perceived intelligence with perceived anthropomorphism (H2) and trust (H4); perceived anthropomorphism with trust (H3); and trust with disclosure willingness across six domains (H5a–H5f): tastes and interests, attitudes and opinions, work or studies, economic and social status, interpersonal relations and self-concept, and physical appearance and sex. Paths among the post-task appraisals represent hypothesized associations rather than temporal or causal effects. Conceptual path diagram. Output rendering and its interaction with source-text condition point to perceived quality. Perceived quality points to perceived intelligence, which points to anthropomorphism and trust. Anthropomorphism also points to trust, and trust points to disclosure willingness in six domains.

Source	Rendering	$\Delta$ Flesch	Jaccard	BERTScore	COMET
Simple	Readability-oriented	14.08	0.377	0.937	0.869
Simple	Fidelity-oriented	4.83	0.713	0.976	0.914
Complex	Readability-oriented	70.43	0.138	0.854	0.670
Complex	Fidelity-oriented	56.40	0.452	0.910	0.753

Table 1. Stimulus-level audit of the prepared-output contrast across the four focal cells.

\Delta

Flesch = change in Flesch reading-ease score from source to output; Jaccard = content-word Jaccard overlap with source; BERTScore = BERTScore F1 against source; COMET = learned MT quality metric. Higher

\Delta

Flesch indicates a larger readability gain; higher Jaccard and BERTScore indicate closer source retention, while higher COMET indicates higher estimated MT quality.

Figure 2. Snapshot of the TransLingo interface used in the experiment. Participants saw the English source, an intermediate Chinese text, and the final English output. Screenshot of the TransLingo prototype. The interface presents an English source-text panel, a Chinese intermediate-translation panel, and an English back-translation panel.

Outcome	Simple readability $M$ ( $SD$ )	Simple fidelity $M$ ( $SD$ )	Complex readability $M$ ( $SD$ )	Complex fidelity $M$ ( $SD$ )	$F$	$p$	$\eta_{p}^{2}$	FDR $q$
Perceived quality	5.269 (1.003)	5.853 (1.051)	5.210 (1.364)	5.229 (1.141)	4.51	$.034^{*}$	.015	—
Perceived intelligence	5.809 (.985)	6.213 (.572)	5.639 (1.059)	5.543 (.905)	5.66	$.018^{*}$	.018	$.027^{*}$
Agency-oriented anthropomorphism	5.769 (2.099)	6.820 (1.889)	6.219 (1.889)	5.721 (2.042)	11.56	$<.001^{***}$	.037	$.002^{**}$
Task-performance trust	6.698 (1.860)	7.429 (1.807)	6.734 (1.997)	6.573 (1.867)	4.20	$.041^{*}$	.014	$.041^{*}$
Tastes and interests	3.213 (.693)	3.382 (.574)	3.256 (.636)	3.317 (.563)	0.58	.447	.002	.447
Attitudes and opinions	2.650 (.915)	2.885 (.790)	2.750 (.791)	2.807 (.743)	0.90	.344	.003	.412

Table 2. Condition descriptives and rendering-by-source-condition interaction tests.

Figure 3. Standardized path estimates from the appraisal-structure model. The condition-to-quality paths are included to present the complete structural specification; RQ1a and RQ1b were formally evaluated using the factorial analyses reported in Section 4.2.1 . Perceived intelligence, anthropomorphism, and trust were modeled as latent constructs (circles). The six disclosure domains were modeled as separate observed domain-mean outcomes (rectangles). The nonsignificant source-condition path is omitted for visual clarity but is reported in Table 3 . Values shown are standardized coefficients. $***p<.001$ , $**p<.01$ , $*p<.05$ . Structural path diagram. Output rendering and its interaction with source-text condition point to perceived quality. Perceived quality points to perceived intelligence. Perceived intelligence points to anthropomorphism and trust, and anthropomorphism also points to trust. Trust points to six separate disclosure-willingness domains.

Path	$b$	$SE$	$\beta$	95% CI	$p$
Output rendering $\rightarrow$ Perceived quality	.584	.169	.247	$[.254,.915]$	$<.001$
Source-text condition $\rightarrow$ Perceived quality	$-$ .059	.183	$-$ .025	$[-.416,.299]$	.749
Rendering $\times$ source condition $\rightarrow$ Perceived quality	$-$ .566	.260	$-$ .202	$[-1.076,-.056]$	.030
Perceived quality $\rightarrow$ Perceived intelligence	1.025	.095	.770	$[.839,1.210]$	$<.001$
Perceived intelligence $\rightarrow$ Anthropomorphism	.546	.062	.650	$[.425,.666]$	$<.001$
Perceived intelligence $\rightarrow$ Trust	.400	.101	.290	$[.202,.599]$	$<.001$

Table 3. Appraisal-structure model. Reported

p

values are unadjusted; all six trust–disclosure associations remained significant after Benjamini–Hochberg FDR correction (all FDR-adjusted

p<.001

MT Evaluation

Machine translation evaluation measures translation quality, error types, and real-world suitability across languages, domains, and modalities.

See all 173 papers

AI translation of literary texts is "fine", but readers still prefer human translations
Jun 24, 2026Yves Ferstler, Adam Podoxin, Ty Brassington +5Machine Translation MT Evaluation
Université du Québec à Montréal · Simon Fraser University · Microsoft
Fluency and Faithfulness in Human and Machine Literary Translation
May 14, 2026Sarah Griebel, Ted UnderwoodMachine Translation MT Evaluation
School of Information Sciences University of Illinois Urbana–Champaign
CompactQE: Interpretable Translation Quality Estimation via Small Open-Weight LLMs
May 15, 2026Kamil Guttmann, Zofia Fraś, Artur Nowakowski +1Small Language Models MT Evaluation
Laniqo, Pozna´n, Poland · Faculty of Mathematics and Computer Science, Adam Mickiewicz University, Pozna´n, Poland

Jun 24, 2026J/K move · Enter open · S save

AI translation of literary texts is "fine", but readers still prefer human translations

Yves Ferstler, Adam Podoxin, Ty Brassington, Gaëlle Laperrière+4

Université du Québec à Montréal · Simon Fraser University · Microsoft

Machine Translation MT Evaluation

Open paper AskPDF arXiv

AI translation of literary works is increasingly common. While the content may be rendered adequately, we do not know enough about how readers experience it in terms of immersiveness and literary effect-aspects poorly captured by automatic metrics or human evaluation targeting fluency and adequacy. We ask 15 avid readers to compare recently published human translations (HT) to machine translations (MT) generated with an agentic language model-based pipeline, for 15 recent novels in French, Polish, and Japanese translated into English. Readers evaluated approximately 8K-word excerpts in two conditions: immersive reading of the whole excerpt (30 comparisons) and close reading of 386 aligned HT-MT chunk pairs (772 comparisons), with two readers per book. Overall, readers find MT "fine", but prefer HT (descriptively at the excerpt level, 19/30, and significantly at the chunk level, 522/772) for its ease, clarity, and immersive nature. Readers' highlights show that MT's quality varies more within one book than HT's does. Crucially, readers cannot reliably tell the two apart (17/30 guess correctly) and tend to prefer the version they believe to be human. Automatic metrics, including LLM-as-a-judge approaches, fail to recover reader preferences and favor MT. We release LAIT (Literary AI Translation), a reader-centered evaluation dataset with 1K reader comments, 2K judgments and preference ratings, and 7.2K span-level annotations, along with our evaluation protocol and supporting interface.

Abstract

MT Evaluation

Explore similar work

AI translation of literary texts is "fine", but readers still prefer human translations

Fluency and Faithfulness in Human and Machine Literary Translation

CompactQE: Interpretable Translation Quality Estimation via Small Open-Weight LLMs