cs.CLNov 15, 2026

FRAAC: A French Multimodal Large Language Model and Dataset forPictogram-Based Augmentative and Alternative Communication

Authors: Alexandre Audibert, François Portet, Benoit Favre, Maja Jonck Hjuler

Organizations: Univ. Grenoble Alpes, CNRS, Grenoble INP, LIG, 38000 Grenoble, France · NAVER LABS Europe, Meylan, France

Abstract

Pictograms are a core modality in augmentative and alternative communication (AAC), yet recent multimodal large language models (MLLMs) cannot reliably interpret or produce pictogram sequences. Two obstacles explain this gap: the absence of a standardized pictogram grammar and the scarcity of parallel text-pictogram data. We introduce Retrieval-Augmented Pictogram Selection (RAPS), an annotation method that refines a rule-based transcription system's output with LLM-based concept extraction and retrieval over a pictogram vocabulary, handling cases where hand-written rules fail (named entities, numbers, polysemous words) without requiring new rules. Applied to French, RAPS yields a synthetic corpus of 1.4 million parallel pairs, of which 300k are publicly released. In a human ranking study where annotators order four pictogram transcriptions of the same sentence, RAPS outperforms the rule-based state of the art. Moreover, we show that standard reference-based metrics separate systems of clearly different quality but fail to rank strong systems against each other. Finally, we build an instruction-tuned model that supports interaction in several AAC modalities: it accepts pictogram or text input and answers in pictograms, speech, or text.

Figures & tables

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Contextual Pictograms for Repairing Multilingual ASR Errors

    Nov 3, 2026Mariana Hernandez, Hector Luna, Carlos Bruno +2Automatic Speech RecognitionMultilingual ASR

  2. Robust Multilingual Text-to-Pictogram Mapping for Scalable Reading Rehabilitation

    Mar 25, 2026Soufiane Jhilal, Martina Galletti

  3. Parameter Efficient Multimodal Instruction Tuning for Romanian Vision Language Models

    Dec 16, 2025George-Andrei Dima, Răzvan-Alexandru Smădu, Dumitru-Clementin CercelVLM AdaptationImage Captioning