FederatedBytes: Tokenizer-Free, Byte-Level Small Language Models for Efficient Multimodal Federated Learning

Ali Sarabadani; Massimo Piccardi; Mehran Abolhasan; Farzad Tofigh; Ian Zhou; Justin Lipman

cs.CLOct 16, 2026

FederatedBytes: Tokenizer-Free, Byte-Level Small Language Models for Efficient Multimodal Federated Learning

Authors: Ali Sarabadani, Massimo Piccardi, Mehran Abolhasan, Farzad Tofigh, Ian Zhou, Justin Lipman

Organizations: Cyber Digital Centre, University of Technology Sydney

Abstract

Federated Learning (FL) is a powerful machine learning framework that allows building a global model from the contributions of multiple, distributed clients, while ensuring the privacy of their local data. However, its heavy computational and communication overheads challenge its use for contemporary AI workloads such as natural language processing (NLP) and multimodal analysis. For this reason, this paper presents a novel small language model that leverages byte-level text and multimodal data processing, enhanced by a linear attention mechanism, to enable manageable Federated Learning of NLP and multimodal analysis tasks. The proposed model, nicknamed FederatedBytes, manages to process text and any other data types, including images, by treating its input as a mere sequence of bytes, while its lean linear attention mechanism allows abating the overall memory and computational footprint. To prove the effectiveness of the proposed model we have carried out an extensive performance evaluation on sentiment analysis tasks in multilingual and multimodal settings over the Multilingual Amazon Reviews and ScienceQA datasets. The experimental results show that FederatedBytes has been able to outperform comparable models such as ByT5 and LSTM, achieving higher accuracy while significantly reducing memory and communication overheads.

Figures & tables

Model/Framework	Data Granularity	Learning Paradigm	Context Handling	Primary Contribution
FedNLP Lin et al. (2021)	Token-Level	Federated	Standard ( $O(L^{2})$ )	Benchmarking FL for token-based NLP.
ByT5 ( Xue et al., 2022 )	Byte-Level	Centralized	Standard ( $O(L^{2})$ )	State-of-the-art among early byte-level text processing.
Longformer ( Beltagy et al., 2020 )	Token-Level	Centralized	Efficient (Linearized)	Efficiently processing long token sequences.
BLT ( Pagnoni et al., 2024 )	Byte-Level	Centralized	Standard ( $O(L^{2})$ )	Scaling byte-level models for large datasets.
FederatedBytes (Ours)	Byte-Level	Federated	Efficient (Linearized)	Unified framework for private, efficient, byte-level FL.

Table 1: High-level comparison of key token- and byte-level models.

Model/Method	Key Innovation	Primary Use Case	Suitability for Federated Learning
FedAvg McMahan et al. (2017)	Decentralized model aggregation	General-purpose FL	Foundational, but model-agnostic and can be inefficient for large models.
ByT5 Xue et al. (2022)	Encoder-decoder Transformer on byte streams	Multilingual NLP in centralized settings	Low. High computational and memory requirements; not optimized for communication efficiency.
bGPT Wu et al. (2024)	Decoder-only Transformer on byte streams	Generative modeling on raw bytes	Low. Similar resource constraints to ByT5; large model size poses communication challenges.
BLT ( Pagnoni et al., 2024 )	Scaling byte-level models to very large parameter counts	Foundational research in large byte-level models	Very Low. Designed for large-scale, centralized data centers, making it impractical for on-device FL.
FederatedBytes (Ours)	Efficient attention on byte streams within a federated framework	Multilingual and multi-modal NLP on decentralized data	High. Explicitly designed to be resource-efficient, reduce communication, and operate without a central data store.

Table 2: Detailed comparison of specific models and methods. This table analyzes the key innovations and suitability for federated learning of the models discussed. While powerful, centralized models such as ByT5 and BLT are not inherently designed for the communication and resource constraints of FL.

Figure 1: End-to-end FederatedBytes Architecture. The pipeline operates in three sequential stages tailored for efficient federated learning: Stage 1 (Input Processing): Raw multilingual text is encoded into UTF-8 bytes. These bytes are compressed into patch embeddings using a Hierarchical Patch Encoder (1D Convolution $\rightarrow$ ReLU $\rightarrow$ Adaptive Pooling) to reduce sequence length. Stage 2 (Memory-Efficient Processing): The patch sequence is processed by a Transformer Encoder utilizing the Performer mechanism. This approximates the full attention matrix using Orthogonal Random Features (ORFs) Yu et al. (2016) , achieving linear $O(L)$ time and space complexity. Stage 3 (Prediction & Classification): The encoded representations are aggregated via global mean pooling, normalized via BatchNorm, and passed through a linear classification head to generate final class logits.

Figure 2: The FederatedBytes Inference Pipeline. An illustrative walkthrough of the running example (“This product is amazing…”) as it passes through the three stages of the model architecture. Stage 1: Input text is patched and processed by the Hierarchical Encoder (Conv1d) to reduce sequence length. Stage 2: Patches are contextualized via Performer linear attention. Stage 3: Features are pooled for final classification.

T (84)	h (104)	i (105)	s (115)	␣(32)	p (112)	r (114)	o (111)
d (100)	u (117)	c (99)	t (116)	␣(32)	i (105)	s (115)	␣(32)
a (97)	m (109)	a (97)	z (122)	i (105)	n (110)	g (103)	␣(32)
a (97)	n (110)	d (100)	␣(32)	w (119)	o (111)	r (114)	k (107)

Table 3: Byte content of the first 32 positions in Patch 1.

Byte	$e_{(1)}$	$e_{(2)}$	$e_{(3)}$	$e_{(4)}$
84 (T)	0.12	-0.08	0.33	0.44
104 (h)	-0.21	0.15	0.05	-0.36
105 (i)	0.19	-0.11	0.29	0.07
115 (s)	0.03	0.48	-0.22	-0.15
32 (space)	0.00	0.00	0.00	0.00
112 (p)	0.44	-0.05	0.19	0.09

Table 4: Example embeddings (first 4 dimensions) for the first 8 bytes of Patch 1.

	Patch 1	Patch 2	Patch 3
Patch 1	0.60	0.25	0.15
Patch 2	0.20	0.65	0.15
Patch 3	0.10	0.30	0.60

Table 5: Hypothetical attention distribution across the first 3 patches.

Data Element	Sample Values for Review ( $i=0$ )
Input Patches ( $X_{0}$ )	First patch: $[84,104,105,115,32,112,114,111,\dots,256]$ (128 integers)
Label ( $y_{0}$ )	$1$

Table 6: Sample input data for our example review within a training micro-batch.

Stage of Processing	Data Shape	Sample Value / Representation (first 4 dims shown)
1. Input Patch	‘(128,)‘	‘[84, 104, 105, 115, …]‘ (A single patch of 128 byte integers)
2. Embedding	‘(128, 768)‘	$\begin{pmatrix}0.12&-0.08&0.33&\dots\\ -0.21&0.15&0.05&\dots\\ \vdots&\vdots&\vdots&\ddots\end{pmatrix}$ (Matrix of 128 embedding vectors)
3. Hierarchical Encoding	‘(768,)‘	‘[0.07, 0.05, 0.08, 0.03, …]‘ (Encoded vector for the patch via Conv1d)
4. Attention	‘(128, 768)‘	‘[0.09, -0.01, 0.15, 0.02, …]‘ (Sequence of 128 contextualized vectors)
5. Global Pooling	‘(768,)‘	‘[0.12, -0.05, 0.07, 0.19, …]‘ (Mean of all 128 patch vectors)
6. Classification	‘(2,)‘	‘[-0.62, 1.45]‘ (Final 2-dim output logits for the sample)

Table 7: Tensor transformation for one sample during the forward pass.

Data Element	Sample Values for Review ( $i=0$ )
Input Patches ( $X_{0}$ )	First patch: $[84,104,105,115,32,\dots]$
Label ( $y_{0}$ )	$1$

Table 8: Sample input data for our example review within an evaluation batch.

Stage	Sample Value
Final Pooled Vector	$[0.15,-0.09,0.11,0.23,\dots]$
Generated Logits ( $\hat{y}_{\text{logits},0}$ )	$[-1.15,2.50]$
Probabilities	$\text{softmax}([-1.15,2.50])=[0.025,0.975]$
Prediction ( $\hat{y}_{\text{pred},0}$ )	$\text{argmax}([-1.15,2.50])=1$ (Positive)

Table 9: Prediction generation for our review sample during evaluation.

Aggregated List	State After Processing Our Sample
All Labels ( $y_{\text{all}}$ )	$[1,\dots]$
All Predictions ( $\hat{y}_{\text{all}}$ )	$[1,\dots]$
All Logits ( $\hat{y}_{\text{logits, all}}$ )	$[[-1.15,2.50],\dots]$

Table 10: Appending our example’s results to the aggregated evaluation lists.

Property	Multilingual Amazon Reviews	ScienceQA
Task	Binarized 5-Class Sentiment Analysis	Multimodal Question Answering
Domain	Product Reviews	Scientific Questions
Classes	5 (1-5 Stars)	Multiple Choice (varies)
Size	200k samples per language	21k samples
Languages	English, German, Chinese	English
Modality	Text	Text + Image

Table 11: Overview of the Datasets Used in the Experiments.

Language	# Train	# Val	# Test	Pos/Neg Ratio
English	200,000	5,000	5,000	40.0% / 60.0%
German	200,000	5,000	5,000	40.0% / 60.0%
Chinese	200,000	5,000	5,000	40.0% / 60.0%

Table 12: Amazon Reviews Dataset (3 Languages).

Model	Hidden Dim	Param Count
ByT5 (small)	512	300M
GPT-2 Medium	768	124M
LSTM (2-layer)	512	4.3M (approx.)
FederatedBytes-Base	512	18.6 (approx.)
FederatedBytes-Performer-30M	512	30M (approx.)
FederatedBytes-Performer-100M	512	100M (approx.)

Table 13: Model Size (Number of Trainable Parameters).

Language	Model	Model Params	Model Size (GB)	Acc (%)	F1 (%)	AUC (%)	Mem (CPU/GPU)	Time (min)
Byte-Level Models
Amazon (en)	FederatedBytes-Base	18.6M	0.07	64.00	60.26	65.78	0.00 GB / 1.56 GB	2.84
	FederatedBytes-Performer-30M	32.5M	0.12	73.60	73.76	79.68	0.00 GB / 0.69 GB	5.53
	FederatedBytes-Performer-100M	101.3M	0.38	70.40	70.56	80.50	0.05 GB / 1.96 GB	7.67
	ByT5	300.8M	1.12	59.80	44.76	58.45	0.00 GB / 19.99 GB	17.31
	LSTM	4.3M	0.02	59.80	44.94	50.08	0.01 GB / 0.28 GB	4.21

Table 14: Centralized performance of single models on the Amazon and ScienceQA datasets. Note: The top AUC scores within the Byte-Level model category are highlighted in bold. Text-Level baselines updated with GPT-2 Medium results.

Dataset	Model	Clients	Acc (%)	F1 (%)	AUC (%)	Time (min)	CPU (GB)	GPU (GB)	Net (GB)	Size (GB)
Amazon (en)	FederatedBytes-Performer-30M	5	59.70	44.71	76.79	2.27	0.13	0.46	2.27	0.11
	FederatedBytes-Performer-100M	5	59.80	44.76	74.80	7.55	0.17	0.72	7.55	0.38
	ByT5	5	59.80	44.75	60.61	22.32	0.28	1.71	22.32	1.12
	FederatedBytes-Performer-30M	10	59.80	44.76	72.04	5.67	0.13	0.46	5.67	0.11
	FederatedBytes-Performer-100M	10	59.80	44.75	72.04	18.87	0.17	0.72	11.32	0.38
	ByT5	10	59.80	44.76	56.54	55.81	0.27	1.71	55.81	1.12

Table 15: Federated learning results comparing FederatedBytes-Performer-30M, FederatedBytes-Performer-100M, and ByT5. The highest AUC for each comparison is bolded. CPU memory reflects the active process usage (total) rather than differential deltas.

Figure 3: FederatedBytes Scalability and Efficiency Analysis. Top: FederatedBytes maintains high AUC scores while ByT5 collapses as client counts increase. Bottom: FederatedBytes demonstrates significantly lower computational overhead, network traffic, and memory footprint compared to ByT5. FederatedBytes-Performer-30M demonstrates superior efficiency, maintaining competitive AUC scores with significantly lower training time, CPU memory, and network consumption compared to the larger ByT5 model, especially as the number of clients increases. Data points represent averages across Amazon (en), (zh), and (de) datasets.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value
General Optimization
Random Seed	42
Optimizer	AdamW
Learning Rate	$3.0\times 10^{-5}$
Weight Decay	0.01
Dropout	0.1

Appendix

Table 16: Key hyperparameters and configuration settings used in the experiments.

Byte-Level Language Model

Byte-level language models predict text directly from bytes, avoiding fixed token vocabularies while balancing character-level flexibility against sequence efficiency.

See all 30 papers

MLLM-LLaVA-FL: Multimodal Large Language Model Assisted Federated Learning
Sep 9, 2024Jianyi Zhang, Hao Frank Yang, Ang Li +5Multimodal Federated Learning Non-IID Federated Learning
Duke University · Johns Hopkins University · University of Maryland College Park +1
A Step Toward Federated Pretraining of Multimodal Large Language Models
Mar 25, 2026Baochen Xiong, Yifan Xu, Xiaoshan Yang +3Multimodal Federated Learning Multimodal Pretraining
MAIS, Institute of Automation, Chinese Academy of Sciences · Pengcheng Laboratory · School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS) +1
Federated Prompt Learning: A Unified Framework, Empirical Analysis, and Future Directions
Aug 14, 2026Qinglin Yang, Chen Qiu, Hongyuan Zhang +3Federated Learning Privacy-Preserving Language Models
Cyberspace Institute of Advanced Technology, Guangzhou University; Guangdong Key Laboratory of Industrial Control System Security; Huangpu Research School of Guangzhou University, Guangzhou, Guangdong, China · Iwate Biotechnology Research Center, Iwate, Japan · Nanjing University of Posts and Telecommunications, Nanjing, Jiangsu, China

Sep 9, 2024J/K move · Enter open · S save

MLLM-LLaVA-FL: Multimodal Large Language Model Assisted Federated Learning

Jianyi Zhang, Hao Frank Yang, Ang Li, Xin Guo+4

Duke University · Johns Hopkins University · University of Maryland College Park+1

Multimodal Federated Learning Non-IID Federated Learning Heterogeneous Federated Learning

Open paper AskPDF arXiv

Previous studies on federated learning (FL) often encounter performance degradation due to data heterogeneity among different clients. In light of the recent advances in multimodal large language models (MLLMs), such as GPT-4v and LLaVA, which demonstrate their exceptional proficiency in multimodal tasks, such as image captioning and multimodal question answering. We introduce a novel federated learning framework, named Multimodal Large Language Model Assisted Federated Learning (MLLM-LLaVA-FL), which employs powerful MLLMs at the server end to address the heterogeneous and long-tailed challenges. Owing to the advanced cross-modality representation capabilities and the extensive open-vocabulary prior knowledge of MLLMs, our framework is adept at harnessing the extensive, yet previously underexploited, open-source data accessible from websites and powerful server-side computational resources. Hence, the MLLM-LLaVA-FL not only enhances the performance but also avoids increasing the risk of privacy leakage and the computational burden on local devices, distinguishing it from prior methodologies. Our framework has three key stages. Initially, we conduct global visual-text pretraining of the model. This pretraining is facilitated by utilizing the extensive open-source data available online, with the assistance of MLLMs. Subsequently, the pretrained model is distributed among various clients for local training. Finally, once the locally trained models are transmitted back to the server, a global alignment is carried out under the supervision of MLLMs to further enhance the performance. Experimental evaluations on established benchmarks, show that our framework delivers promising performance in the typical scenarios with data heterogeneity and long-tail distribution across different clients in FL.

Abstract

Byte-Level Language Model

Explore similar work

MLLM-LLaVA-FL: Multimodal Large Language Model Assisted Federated Learning

A Step Toward Federated Pretraining of Multimodal Large Language Models

Federated Prompt Learning: A Unified Framework, Empirical Analysis, and Future Directions