Pictograms are a core modality in augmentative and alternative communication (AAC), yet recent multimodal large language models (MLLMs) cannot reliably interpret or produce pictogram sequences. Two obstacles explain this gap: the absence of a standardized pictogram grammar and the scarcity of parallel text-pictogram data. We introduce Retrieval-Augmented Pictogram Selection (RAPS), an annotation method that refines a rule-based transcription system's output with LLM-based concept extraction and retrieval over a pictogram vocabulary, handling cases where hand-written rules fail (named entities, numbers, polysemous words) without requiring new rules. Applied to French, RAPS yields a synthetic corpus of 1.4 million parallel pairs, of which 300k are publicly released. In a human ranking study where annotators order four pictogram transcriptions of the same sentence, RAPS outperforms the rule-based state of the art. Moreover, we show that standard reference-based metrics separate systems of clearly different quality but fail to rank strong systems against each other. Finally, we build an instruction-tuned model that supports interaction in several AAC modalities: it accepts pictogram or text input and answers in pictograms, speech, or text.
Figures & tables
Figure 1: Overview of our work. (a) RAPS recovers a concept missed by the rule-based grammar. (b) RAPS generates a pre-training corpus of 1.4M text-pictogram pairs used to train Panta-v1 . (c) Panta-v2 , initialized from Panta-v1 and instruction-tuned on 20k question-answer pairs
Figure 2: Overview of Retrieval-Augmented Pictogram Selection.
Dataset
# Pairs
Gutenberg Gerlach and Font-Clos (2020)
12707
Vikidia Hai-Nyzhnyk (2022)
90431
Cosmopedia Ben Allal et al. (2024)
215666
Non-Public
1038004
Single picto
40000
Total
1396808
Table 1: Number of text-pictogram pairs per source, after filtering. ∗ Used for pictogram-to-text only.
System
Rank ↓
Win ↑
Loss ↓
Human
2.48±1.23
32.4%
30.9%
PictoGrammar
3.00±1.05
13.0%
42.4%
RAPS
2.34±1.01
24.4%
15.6%
Panta-v1
2.18±0.99
30.2%
11.1%
Table 2: Human ranking of the text-to-pictogram systems. Rank: mean rank ± standard deviation. Win/Loss: percentage of first/last places.
System
Medical
Email
Book
AM
All
Human
2.05
3.11
2.89
2.65
2.48
PictoGrammar
3.28
2.68
2.62
2.92
3.00
RAPS
2.35
2.39
2.36
2.24
2.34
Panta-v1
2.32
1.83
2.12
2.19
2.18
Table 3: Mean rank per source corpus (lower is better). AM: Arasaac Material.
All sentences (121)
Medical only (58)
Metric
PG/Panta-v1
PG/RAPS
RAPS/Panta-v1
Mean
PG/Panta-v1
PG/RAPS
RAPS/Panta-v1
Mean
BLEU (lemmas)
0.333∗
0.333∗
0.172
0.279
0.720∗
0.608∗
0.306
0.545
METEOR (lemmas)
0.156
0.321∗
0.140
0.206
0.480∗
0.608∗
0.184
0.424
WER (lemmas)
0.528∗
0.500∗
0.038
0.355
0.808∗
0.755∗
0.105
0.556
chrF++ (lemmas)
0.162
0.197
0.097
0.152
0.500∗
0.491∗
0.179
0.390
BLEU (IDs)
0.200
0.333∗
0.149
0.227
0.619∗
0.644∗
0.167
0.477
Table 4: Agreement ( τ ) between metric and human preferences for each pair of systems (PG = PictoGrammar), using the human sequences as references. ∗ : significant agreement after Holm correction ( p<0.05 ).
Email
Medical
k
BS ↑
BLEU ↑
WER ↓
BS ↑
BLEU ↑
WER ↓
Ministral-3-8B (lemmas)
0
36.6
17.3
91.6
36.8
11.1
101.5
2
43.5
22.5
74.3
41.4
12.4
95.6
4
43.4
22.0
73.9
44.2
15.2
85.0
8
45.1
24.9
69.5
49.5
22.1
72.8
Table 5: Pictogram-to-text translation results
System
BLEU ↑
ROUGE-L ↑
BERTSc. ↑
Ministral 0-shot
12.37
34.48
46.36
Ministral 8-shot
16.44
37.34
50.69
Panta-v2
20.30
41.19
55.00
Panta-v2 (picto)
14.61
33.02
47.42
Table 6: Surface metrics on the Instruct dataset. Questions were given as text, except for Panta-v2 (picto) where questions were given as pictograms.
System
Rel. ↑
Corr. ↑
Simp. ↑
Overall ↑
Ministral 0-shot
0.861
0.913
0.732
0.835
Ministral 8-shot
0.850
0.900
0.807
0.852
Panta-v2
0.846
0.890
0.823
0.853
Panta-v2 (picto)
0.685
0.743
0.789
0.739
Table 7: LLM-judge scores: on the Dataset Instruct. Rel.: relevance; Corr.: correctness; Simp.: simplicity. Questions were given as text, except for Panta-v2 (picto) where questions were given as pictograms.
System
Correctness ↑
Simplicity ↑
Ministral-3-8B (8-shot)
0.850
0.788
Panta-v2
0.806
0.819
Reference answers
0.731
0.919
Table 8: Human evaluation of text answers.
Setting
Correctness ↑
Transcription ↑
Text Q → Picto A
0.53
0.65
Picto Q → Text A
0.68
0.59
Table 9: Human evaluation of Panta-v2 on cross-modal generation.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Component
Description
Proposed Model
Parameters
Pre-trained VLLM
Text-centric multimodal backbone based on a causal decoder architecture, responsible for cross-modal generation.
mistralai/Ministral-3-8B-Instruct-2512 Liu et al. (2026a)
7,952.68M
Pre-trained Image Encoder
Encodes visual inputs into a sequence of latent representations compatible with the language model.
Pixtral Liu et al. (2026a)
403.31M
Audio Codec
Converts continuous audio waveforms into discrete token sequences suitable for language model decoding.
BiCodec decoder part Wang et al. (2025)
92M
Audio Decoder
Generates discrete audio tokens autoregressively.
Initialized with LLM weights
1,359.04M
Pictogram Decoder
Generates sequences of pictograms from latent representations using a causal decoding strategy.
Initialized with LLM weights
928.53M
Appendix
Table 10: Core components of the proposed multimodal architecture with parameter counts.
Figure 3: Overview of our French multimodal system. A pretrained vision language model encodes French text and pictogram sequences into a shared representation. Two transformer decoders, initialised from the llm and trained in parallel predict speech and pictogram tokens, while text is decoded by the llm itself.
Figure 4: Stage 2 training curves of the selected run (LLM learning rate 10−6 , decoders 5⋅10−6 ), grouped by output modality. Solid lines: validation loss; dashed lines: training loss on an evaluated subset. The y-axis is in log scale.
Stage 1
Stage 2
Picto → Text
Text → Picto
Instruct
Initialization
Ministral-3-8B-Instruct
Ministral-3-8B-Instruct
Panta-v1
Trained modules
Adapter + Perceiver
Pictogram decoder
LLM + decoders
GPUs (H100)
4
4
8 (FSDP)
Global batch size
256
256
64
Steps
24k
20k
1.6k
Appendix
Table 11: Training hyperparameters. Stage 2 trains on five tasks (picto-to-text, text-to-picto, text-to-speech, and question answering from a text or a pictogram question), sampled uniformly with oversampling of the smaller datasets; the LLM is frozen for text-to-picto and text-to-speech batches.
Subset
# Examples
Train (Synthetic)
14965
Train (Non-Public)
4529
Total train
19494
Test (French-Instruct)
1285
Appendix
Table 12: Composition of the Instruct dataset.
Dataset
Total (kh)
Filtered (kh)
#Segments
MLS Pratap et al. (2020)
0.99
0.83
216 563
EPAC Esteve et al. (2010)
1.46
1.01
375 936
Audiocite Felice et al. (2024)
4.76
4.35
573 808
Yodas Li et al. (2023)
15.29
6.46
1 082 094
Total
22.50
12.65
2 248 401
Appendix
Table 13: Statistics of the text-to-speech training data, before and after filtering. Durations are given in thousands of hours (kh).
Tokenizer
PESQ ↑
STOI ↑
Mel ↓
STFT ↓
MOS ↑
Spk. ↑
Ground Truth
4.644
1.000
0.000
0.000
2.439
1.000
BiCodec
2.086
0.869
5.532
0.214
2.604
0.771
BiCodec FT
2.213
0.888
5.273
0.202
2.681
0.733
Appendix
Table 14: Reconstruction quality of the BiCodec tokenizer, before and after fine-tuning (FT). Spk.: speaker similarity. Best codec results in bold.
System
Input
Spk. sim. ↑
WER ↓
MOS ↑
Voxtral
Text
0.168
1.89
3.32
Panta-v2
Text
0.582
2.27
3.17
Panta-v2
Picto
0.581
2.41
3.17
Appendix
Table 15: Evaluation of text-to-speech.
Judge
Rel.
Corr.
Simp.
Overall
Ministral 0-shot
Mistral-Small
0.793
0.904
0.766
0.821
GPT-OSS
0.970
0.912
0.686
0.856
DeepSeek-R1
0.818
0.924
0.743
0.828
Mean
0.861
0.913
0.732
0.835
Ministral 8-shot
Appendix
Table 16: Per-judge LLM-judge scores on the 1,285 test questions. Scores from 1 to 5 are rescaled to [0,1] . Mean: average over the judges that returned a score.
Figure 6: Annotation interface, step 1: transcription of the pictogram sequence into French, without any textual context.
Figure 7: Annotation interface, step 2: adequacy of the answer to the question.
Figure 8: Annotation interface, step 3: consistency of the pictogram answer with the text answer, or fidelity of the pictogram question to the original question.
Figure 9: Annotation interface, step 4: correctness and simplicity of text answers to text questions.
Model / Tool
License
Backbone and RAPS pipeline
Ministral-3-8B-Instruct-2512 (inc. Pixtral)
Apache 2.0
gpt-oss-120b
Apache 2.0
Qwen3-32B
Apache 2.0
Qwen3-14B
Apache 2.0
Qwen3-Embedding-0.6B
Apache 2.0
Appendix
Table 17: Licenses of the models and tools used in this work.
Dataset / Resource
License
Pictogram vocabulary
ARASAAC
CC BY-NC-SA 4.0
WordNet
WordNet License
Text sources (pre-training corpus)
Vikidia
CC-BY-SA 3.0
Cosmopedia
Apache 2.0
Appendix
Table 18: Licenses of the datasets and resources used and released in this work.
Department Signal Theory, Telematics and Communications, University of Granada · Brain, Mind, and Behavior Research Center (CIMCYC), University of Granada · Faculty of Psychology and Education Sciences, University of Coimbra +1