Making Bielik LLM Reason (Better): A Field Report

Adam Trybus; Bartosz Bartnicki; Remigiusz Kinas

cs.CLMar 11, 2026

Making Bielik LLM Reason (Better): A Field Report

Authors: Adam Trybus, Bartosz Bartnicki, Remigiusz Kinas

Organizations: Institute of Philosophy, Jagiellonian University, 52 Grodzka St., Kraków 31-044, Poland · Bielik.ai (Speakleash Foundation), 33/4 Niemczańska St., Wrocław 50-561, Poland

Abstract

This paper presents a research program dedicated to evaluating and advancing the reasoning capabilities of Bielik, a Polish large language model. The study describes a number of stages of work: initial benchmarking and creation of evaluation methodology, analyzing of comparative results with other LLMs and outlining of future prospects that take into account the limitations of the analyses conducted so far and aims to keep Bielik in the race give the ever-changing -- and competitive -- AI landscape.

Figures & tables

No.	Taxonomy	Deterministic
1	Abductive reasoning	no
2	Argument evaluation	no
3	Argument standardisation	no
4	Classical predicate logic	yes
5	Classical propositional logic	yes
6	Deductive reasoning (ASCII)	yes

Table 1: Benchmarking Taxonomies

Rank	Model	Score
1	gemini-3-pro-preview	87%
2	o3	87%
3	gemini-2.5-pro	84%
4	grok-4	83%
5	o4-mini	79%
6	deepseek-R1	78%

Table 2: Model Performance Ranking

Rank	Model	Avg. Reasoning Tokens
1	grok-3-mini	1 787
2	o4-mini	1 800
3	o3	2 001
4	o1	2 751
5	bielik-r:11b	3 152
6	Deepseek-R1	3 318

Table 3: Bielik-R and Commercial Models by Average Reasoning Tokens

Battle	Strategy	Result
#1–2	Learning basics	Partial success
#3	Sequential elimination	3 eliminations
#4	Aggressive Melee	4 eliminations (so close)
#5	Shoot enemy Rangers	1 elimination
#6	Defensive Melee	0 eliminations (failure)
#7	Return to 4. style	victory

Table 4: Battle Results Summary Generated by Claude Opus 4.5

LLM Reasoning

Large language model reasoning studies how models solve multi-step problems, verify intermediate steps, and use computation or tools reliably.

See all 926 papers

Competence-Based Evaluation of Logical Reasoning in Large Language Models
Oct 16, 2026Jize Jiang, Chengxiang ZhaiLLM Evaluation LLM Reasoning
University of Illinois Urbana-Champaign
PetriBench: Benchmarking LLM Reasoning over Dynamic State Spaces
Sep 17, 2026Pyrros Koussios, Benjamin Jäger, John Hua Yao +3LLM Evaluation Temporal Reasoning in Language Models
ETH Zürich · Stanford University
BrainBench: Exposing the Commonsense Reasoning Gap in Large Language Models
Mar 16, 2026Yuzhe TangCommonsense Reasoning Language Model Reasoning Evaluation
Georgia Institute of Technology

Oct 16, 2026J/K move · Enter open · S save

Competence-Based Evaluation of Logical Reasoning in Large Language Models

Jize Jiang, Chengxiang Zhai

University of Illinois Urbana-Champaign

LLM Evaluation LLM Reasoning Reasoning Consistency in Language Models

Open paper AskPDF arXiv

Recent advances in large language models (LLMs) have brought impressive performance across diverse tasks; however, shortcomings remain in logical reasoning, including inconsistencies under linguistic variation. To improve the reasoning capacity of LLMs, we must be able to accurately measure their reasoning capacity. Current benchmarks primarily report single accuracy numbers on selected reasoning tasks. Such a performance-based evaluation methodology says little about a model's generalization capacity in reasoning. To address this limitation, we argue for a different evaluation paradigm, called competence-based evaluation (CBE), which tests whether models reliably apply the same competence across a family of problems. A parameterized generator scales complexity while keeping the required competence fixed, and matched answer-preserving presentations test correctness beyond one formulation. We instantiate CBE on elementary ordering tasks, scaling input size while varying presentations of the same ordering rule. Across six GPT and Qwen models, performance is reasonably good over bounded complexity ranges, but even strong models fail the competence condition across the tested family. Chain-of-thought and GRPO extend strong performance without resolving all tested failures; supervised LoRA shows little held-out transfer. CBE provides diagnostic profiles and an evaluation protocol that can grow with future models.

Abstract

LLM Reasoning

Explore similar work

Competence-Based Evaluation of Logical Reasoning in Large Language Models

PetriBench: Benchmarking LLM Reasoning over Dynamic State Spaces

BrainBench: Exposing the Commonsense Reasoning Gap in Large Language Models