Natural Language Code Retrieval for 1C:Enterprise: An Open Benchmark and Efficient Bi-Encoder

Konstantin Chesnokov; Chingiz Mingazov

cs.CLAug 20, 2026

Natural Language Code Retrieval for 1C:Enterprise: An Open Benchmark and Efficient Bi-Encoder

Authors: Konstantin Chesnokov, Chingiz Mingazov

Organizations: Independent Researcher, Moscow, Russia · Independent Researcher, Kazan, Russia

Abstract

Natural language code retrieval is a rapidly evolving task in computer science. However, the 1C:Enterprise ecosystem combines Russian syntax with highly domain-specific terminology, for which open datasets and specialized models have been virtually non-existent. We present a comprehensive pipeline for 1C code retrieval: an open benchmark of 3,413 real-world, PII-scrubbed query-code pairs, a reproducible evaluation harness, and a specialized bi-encoder. To overcome scarce labeled data, we fine-tune on 784,057 synthetic triplets generated by google/gemma-4-26B-A4B-it from public code repositories, using Matryoshka Representation Learning (MRL) and a privacy-aware tokenizer. Because the benchmark subsets differ in size, we report balanced-subset macro, query-weighted micro, and forum-only results. Our model reaches 0.5992 balanced macro nDCG@10, 0.5044 micro, and 0.4617 on forum, versus 0.4932 macro for the baseline architecture and 0.5404 for google/embeddinggemma-300m. Removing every benchmark example flagged by the conservative exact/13-gram overlap audit leaves 0.6011 balanced macro (0.5010 micro), indicating that detected train-benchmark overlap does not explain the headline result. MRL truncation to 256 dimensions preserves 99.9% of retrieval quality while reducing dense-index storage and exact similarity arithmetic by a factor of three.

Figures & tables

Component	Content	Scale
PruhaNLP/1C-Ebench	Test pairs: question $\rightarrow$ code, subsets forum / fastcode	3,413 pairs
PruhaNLP/1C-RB	Evaluation harness: dense, BM25, metrics	CLI/Python
PruhaNLP/1C-Code-Train	Triplets $(q,d^{+},d^{-})$ with PII scrubbing	784,057 triplets
PruhaNLP/USER2-1C-code	Domain-adapted reference bi-encoder	768d, MRL

Table 1: Public resource stack.

Subset	Queries	Share	Source	Characterization
forum	2,883	84.5%	Forum discussions	Long questions, context, errors, inline code
fastcode	530	15.5%	Snippet catalog	Short queries, long ready-to-use snippets
Total	3,413	100%	–	Closed-set, single-gold qrels

Table 2: PruhaNLP/1C-Ebench subsets.

Figure 1: Construction of the public PruhaNLP/1C-Ebench benchmark and the dense-retrieval evaluation pipeline implemented in PruhaNLP/1C-RB .

Metric	forum	fastcode	all
Exact match: code	0.10%	0.00%	0.09%
Exact match: question	0.03%	8.68%	1.38%
Exact match: pair	0.00%	0.00%	0.00%
13-gram overlap: code	6.24%	16.23%	7.79%
13-gram overlap: question	0.03%	8.68%	1.38%
Input-only	0.03%	6.60%	1.05%

Table 3: Train–benchmark overlap audit.

Parameter	Value
Base model	deepvk/USER2-base
Architecture	ModernBERT/RuModernBERT sentence-transformers encoder
Embedding dimension	768
Maximum sequence length	8,192
Pooling	Mean pooling
Similarity	Cosine over L2-normalized embeddings

Table 4: PruhaNLP/USER2-1C-code configuration.

Figure 2: Construction of PruhaNLP/1C-Code-Train : from filtered public 1C code to synthetic queries, hard-negative triplets, PII scrubbing, and fine-tuning of PruhaNLP/USER2-1C-code .

Criterion	Mean	Median	Share $\geq 4$
Relevance	4.33	5	82.0%
Naturalness	4.17	5	76.3%
Clarity	4.37	5	82.0%

Table 5: LLM-as-a-judge scores for 300 synthetic query–code pairs.

Parameter	Value
Training samples	784,057 triplets
Batch size / MNRL mini-batch	256 / 32
Epochs / steps	3 / 9,189
Learning rate	$2\times 10^{-5}$
LR schedule	5% cosine warmup + decay
Max sequence length / precision	8,192 / FP16

Table 6: Training hyperparameters.

Model	Query prompt	Document prompt
PruhaNLP/USER2-1C-code	search_query	search_document
deepvk/USER2-base	search_query	search_document
google/embeddinggemma-300m	query	document
deepvk/USER-bge-m3	empty	empty
ibm-granite/granite-embedding-311m-multilingual-r2	query	document
microsoft/harrier-oss-v1-270m	web_search_query	empty

Table 7: Dense baselines and prompts. Empty means no prompt prefix.

Figure 3: nDCG@10 leaderboard on PruhaNLP/1C-Ebench (balanced-subset macro).

Rank	Model	forum	fastcode	macro	micro
1	PruhaNLP/USER2-1C-code	0.4617	0.7366	0.5992	0.5044
2	google/embeddinggemma-300m	0.3720	0.7080	0.5404	0.4242
3	deepvk/USER2-base	0.3670	0.6190	0.4932	0.4061
4	deepvk/USER-bge-m3	0.3430	0.6390	0.4910	0.3890
5	ibm-granite/granite-embedding-311m-multilingual-r2	0.3190	0.6510	0.4846	0.3706
6	microsoft/harrier-oss-v1-270m	0.3230	0.6360	0.4796	0.3716

Table 8: nDCG@10 leaderboard on PruhaNLP/1C-Ebench .

System	Subset	nDCG@10	Recall@10	MRR@10
PruhaNLP/USER2-1C-code	forum	0.4617	0.6008	0.4178
PruhaNLP/USER2-1C-code	fastcode	0.7366	0.9208	0.6774
BM25Okapi	forum	0.2865	0.3479	0.2670
BM25Okapi	fastcode	0.3334	0.4189	0.3067

Table 9: Full @10 metrics for the reference dense model and BM25.

Comparison	$\Delta$	95% CI / $P$
Macro: Ours vs. USER2-base	$+0.106$	$[0.087,\,0.125]$ , $P<0.001$
Macro: Ours vs. EmbeddingGemma	$+0.0588$	$[0.042,\,0.075]$ , $P<0.001$
RRF vs. dense ( Ours ), forum	$-0.062$	$[-0.075,\,-0.049]$ , $P<0.05$
RRF vs. dense ( Ours ), fastcode	$-0.277$	$[-0.319,\,-0.234]$ , $P<0.05$
RRF vs. dense ( Base ), forum	$-0.005$	$[-0.016,\,0.006]$ , n.s.
RRF vs. dense ( Base ), fastcode	$-0.200$	$[-0.237,\,-0.164]$ , $P<0.05$

Table 10: Paired-bootstrap nDCG@10 comparisons.

Type / ranks	Query, gold, interpretation
Win ( forum ) Ours 1; others $>$ 10	Q: Как обойти ограничения ФО на реквизит в динамическом списке, не трогая константу ФО? Gold: ОтключенныеПоля = Новый Массив; … FO $\times$ dynamic-list pattern; little lexical overlap with gold.
Win ( fastcode ) Ours 1; others $>$ 10	Q: Описание строки в таблице значений Gold: ст90 = новый ОписаниеТипов("строка",,Новый КвалификаторыСтроки(90)); Colloquial «описание строки» $\to$ type API; lexical baselines miss it.
Fail ( fastcode ) Ours $>$ 10; base 1	Q: Произношение текста голосом Gold: Voice = Новый COMObject("SAPI.SpVoice"); Voice.Speak("Привет!"); Rare COM API; general models recover Speak / SAPI .
Fail ( fastcode ) Ours 32; base 1	Q: Сжатие длинной Строки Gold: СжатиеДанных = Новый СжатиеДанных(9); …Base64. Short generic title; surface «сжатие»+«строка» wins.

Table 11: Qualitative wins and failures (original Russian query/code). Gold rank: 1 = best;

>

10 = outside top-10.

Figure 4: Matryoshka truncation: balanced-subset macro nDCG@10 by embedding dimension.

Dimension $m$	macro nDCG@10	Retention	Relative index size
768	0.599	100.0%	$1.00\times$
512	0.600	100.0%	$0.67\times$
384	0.600	100.0%	$0.50\times$
256	0.598	99.9%	$0.33\times$
128	0.584	97.5%	$0.17\times$
64	0.560	93.5%	$0.08\times$

Table 12: MRL truncation on PruhaNLP/1C-Ebench .

Code Search

See all 13 papers

Beyond Retrieval: A Multitask Benchmark and Model for Code Search
May 6, 2026Siqiao Xue, Zihan Liao, Jin Qin +4Learning to Rank Software Engineering Benchmarks
Ant Group · Alipay
Do not copy and paste! Rewriting strategies for code retrieval
May 8, 2026Andrea Gurioli, Federico Pennino, Maurizio GabbrielliQuery Reformulation Information Retrieval
DISI University of Bologna
Agent Retrieval Bench: Evaluating Repository Context Retrieval for Coding Agents
Jul 27, 2026Bowen Qin, Yi XieSoftware Engineering Agents AI Agent Benchmarks
1National University of Singapore (NUS) · 2Peking University

May 6, 2026J/K move · Enter open · S save

Beyond Retrieval: A Multitask Benchmark and Model for Code Search

Siqiao Xue, Zihan Liao, Jin Qin, Ziyin Zhang+3

Ant Group · Alipay

Learning to Rank Software Engineering Benchmarks Code Search Information Retrieval

Open paper AskPDF arXiv

Code search has usually been evaluated as first-stage retrieval, even though production systems rely on broader pipelines with reranking and developer-style queries. Existing benchmarks also suffer from data contamination, label noise, and degenerate binary relevance. In this paper, we introduce CoREB, a contamination-limited, multitask code retrieval and reranking benchmark, together with a fine-tuned code reranker, that goes beyond retrieval to cover the full code search pipeline. CoREB is built from counterfactually rewritten LiveCodeBench problems in five programming languages and delivered as timed releases with graded relevance judgments. We benchmark eleven embedding models and five rerankers across three tasks: text-to-code, code-to-text, and code-to-code. Our experiments reveal that: \circone code-specialised embeddings dominate code-to-code retrieval (

{\sim}2{\times}

over general encoders), yet no single model wins all three tasks; \circtwo short keyword queries, the format closest to real developer search, collapse every model to near-zero nDCG@10; \circthree off-the-shelf rerankers are task-asymmetric, with a 12-point swing on code-to-code and no baseline net-positive across all tasks; \circfour our fine-tuned CoREB-Reranker is the first to achieve consistent gains across all three tasks. The data and model are released.

Abstract

Code Search

Explore similar work

Beyond Retrieval: A Multitask Benchmark and Model for Code Search

Do not copy and paste! Rewriting strategies for code retrieval

Agent Retrieval Bench: Evaluating Repository Context Retrieval for Coding Agents