DEEP: Docker-based Execution and Evaluation Platform

Sergio Gómez González; Miguel Domingo; Francisco Casacuberta

cs.CLFeb 23, 2026

DEEP: Docker-based Execution and Evaluation Platform

Authors: Sergio Gómez González, Miguel Domingo, Francisco Casacuberta

Organizations: PRHLT Research Center - Universitat Politècnica de València · ValgrAI - Valencian Graduate School and Research Network for Artificial Intelligence

Abstract

Comparative evaluation of several systems is a recurrent task in researching. It is a key step before deciding which system to use for our work, or, once our research has been conducted, to demonstrate the potential of the resulting model. Furthermore, it is the main task of competitive, public challenges evaluation. Our proposed software (DEEP) automates both the execution and scoring of machine translation and optical character recognition models. Furthermore, it is easily extensible to other tasks. DEEP is prepared to receive dockerized systems, run them (extracting information at that same time), and assess hypothesis against some references. With this approach, evaluators can achieve a better understanding of the performance of each model. Moreover, the software uses a clustering algorithm based on a statistical analysis of the significance of the results yielded by each model, according to the evaluation metrics. As a result, evaluators are able to identify clusters of performance among the swarm of proposals and have a better understanding of the significance of their differences. Additionally, we offer a visualization web-app to ensure that the results can be adequately understood and interpreted. Finally, we present an exemplary case of use of DEEP.

Figures & tables

Figure 1: Visual interface of DEEP with the results of an example.

Figure 2: Chart of the computation time needed by the proposals of a demo with DEEP.

Figure 3: Chart of time vs quality of hypothesis of a demo with DEEP.

Figure 4: Chart of average by cluster of a demo with DEEP.

Figure 5: Display of the evaluation data and download options.

System	BLEU	TER	chrF	Time (s)
Seed-x7b	$38.84$	$51.00$	$65.45$	$236.28$
MADLAD-400	$36.03$	$52.43$	$63.31$	$870.62$
NLLB-200	$33.60$	$55.34$	$60.17$	$774.95$
OPUS-MT	$30.02$	$57.81$	$58.80$	$57.60$
M2M-100	$27.70$	$60.68$	$55.04$	$417.13$
mBART	$27.68$	$59.81$	$57.15$	$228.66$

Table 1: Results of the demo of DEEP.

Automated Evaluation

Automated evaluation develops metrics, benchmarks, and AI judges to assess model outputs and system behavior at scale.

See all 396 papers

DEP: A Decentralized Large Language Model Evaluation Protocol
Mar 1, 2026Jianxiang Peng, Junhao Li, Hongxiang Wang +15LLM Evaluation Benchmark Contamination
TJUNLP Lab, Tianjin University, Tianjin, China · National Supercomputing Center in Tianjin · Yuanhui AI
DeepMTL2R: A Library for Deep Multi-task Learning to Rank
Feb 16, 2026Chaosheng Dong, Peiyao Xiao, Yijia Wang +1Learning to Rank Multi-Task Learning
Amazon USA
OmniStepBench: Deployment-Aware Agent Evaluation across Modalities and Task Complexity
Sep 5, 2026Yu Liu, Zhilin Liu, Zhiwei Yang +7Multimodal Agent Evaluation LLM Evaluation
Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences · MiLM Plus, Xiaomi Inc. +1

Mar 1, 2026J/K move · Enter open · S save

DEP: A Decentralized Large Language Model Evaluation Protocol

Jianxiang Peng, Junhao Li, Hongxiang Wang, Haocheng Lyu+14

TJUNLP Lab, Tianjin University, Tianjin, China · National Supercomputing Center in Tianjin · Yuanhui AI

LLM Evaluation Benchmark Contamination

Open paper AskPDF arXiv

With the rapid development of Large Language Models (LLMs), a large number of benchmarks have been proposed. However, most benchmarks lack unified evaluation standard and require the manual implementation of custom scripts, making results hard to ensure consistency and reproducibility. Furthermore, mainstream evaluation frameworks are centralized, with datasets and answers, which increases the risk of benchmark leakage. To address these issues, we propose a Decentralized Evaluation Protocol (DEP), a decentralized yet unified and standardized evaluation framework through a matching server without constraining benchmarks. The server can be mounted locally or deployed remotely, and once adapted, it can be reused over the long term. By decoupling users, LLMs, and benchmarks, DEP enables modular, plug-and-play evaluation: benchmark files and evaluation logic stay exclusively on the server side. In remote setting, users cannot access the ground truth, thereby achieving data isolation and leak-proof evaluation. To facilitate practical adoption, we develop DEP Toolkit, a protocol-compatible toolkit that supports features such as breakpoint resume, concurrent requests, and congestion control. We also provide detailed documentation for adapting new benchmarks to DEP. Using DEP toolkit, we evaluate multiple LLMs across benchmarks. Experimental results verify the effectiveness of DEP and show that it reduces the cost of deploying benchmark evaluations. As of February 2026, we have adapted over 60 benchmarks and continue to promote community co-construction to support unified evaluation across various tasks and domains.

Abstract

Automated Evaluation

Explore similar work

DEP: A Decentralized Large Language Model Evaluation Protocol

DeepMTL2R: A Library for Deep Multi-task Learning to Rank

OmniStepBench: Deployment-Aware Agent Evaluation across Modalities and Task Complexity