CollabSciReM: Aligning Language Models for Scientific Conversations via Science-Aware Reward Decomposition

Sasi Kiran Gaddipati; Gollam Rabby; Sören Auer

cs.CLOct 20, 2026

CollabSciReM: Aligning Language Models for Scientific Conversations via Science-Aware Reward Decomposition

Authors: Sasi Kiran Gaddipati, Gollam Rabby, Sören Auer

Organizations: TIB—Leibniz Information Centre for Science and Technology, Hannover, Germany · L3S Research Center, Hannover, Germany

Abstract

Large Language Models (LLMs) are increasingly used as research collaborators for hypothesis refinement, methodological planning, and technical analysis across multi-turn interactions. In this setting, a useful assistant must be accurate, methodologically justified, logically consistent, and token-efficient. Current alignment approaches rarely achieve all of these together. Instruction-tuned models gain conversational quality through verbosity, and we show that science-agnostic general collaborative rewards compress responses at the expense of scientific rigor. We propose CollabSciReM, a goal-conditioned reward framework that decomposes the session-level reward into conversational quality, scientific rigor, and structural efficiency. We also introduce SciQuality, a pairwise evaluation of multi-turn research conversations grounded in 300 held-out scientific papers and scored on six dimensions: methodological soundness, justification, logical consistency, factual correctness, novelty, and responsiveness. Fine-tuned on 8.62K synthetic research conversations, CollabSciReM is preferred on SciQuality over a science-agnostic reward baseline trained on the same data for all four Llama and Qwen models, including a 65.3% win rate at matched response length. For the 7B and 8B models, CollabSciReM outperforms the science-agnostic reward baseline on the primary metric in 8 of 10 single- and multi-turn benchmark settings. In a blind study, domain experts rated CollabSciReM significantly higher on novelty than the general-reward variant and reported fewer hallucinated responses than both the general-reward variant and the base model.

Figures & tables

Figure 1: Overview of the CollabSciReM Framework. The pipeline begins with Synthetic Data Generation, represented by dotted arrows, in which a simulated researcher and a teacher model engage in scientific dialogue. These interactions are evaluated by a Critic/Judge Model to compute multi-faceted rewards, including conversational, structural, and scientific metrics. The resulting preference-labeled dataset is used to train a Policy Model via Supervised Fine-Tuning (SFT) and Offline Identity Preference Optimization (IPO). In inference, a user query triggers the Learned Policy to generate a scientifically grounded response.

Figure 2: Pairwise scientific-quality win rates of CollabSciReM against (a) CollabLLMSci and (b) the base Instruct models on 300 held-out scientific papers. Bars show 95% confidence intervals. Meth. : Methodological soundness, Just. : Justification, Logic : Logical Consistency, Fact. : Factual Correctness, Novel : Novelty, Resp. : Responsiveness and Ovrl. : Overall

(a) Multi-turn benchmarks
	MT-Bench-STEM						MINT
Model	Overall ( $\uparrow$ )	Acc. ( $\uparrow$ )	Format ( $\uparrow$ )	Useful. ( $\uparrow$ )	Relev. ( $\uparrow$ )	TC ( $\downarrow$ )	SR@5 (%) ( $\uparrow$ )	TC ( $\downarrow$ )	APT ( $\uparrow$ )
Llama-3.1-8B-Instruct	6.83 $\pm$ 0.04	6.55 $\pm$ 0.05	7.92 $\pm$ 0.07	6.67 $\pm$ 0.08	8.37 $\pm$ 0.01	469.00 $\pm$ 8.00	47.11 $\pm$ 3.20	296.00 $\pm$ 14.06	159.20 $\pm$ 9.08
Llama-3.1-8B-CollabLLMSci	7.02 $\pm$ 0.19	6.69 $\pm$ 0.22	8.19 $\pm$ 0.01	6.88 $\pm$ 0.22	8.55 $\pm$ 0.02	298.00 $\pm$ 6.00	43.41 $\pm$ 3.22	123.91 $\pm$ 0.62	350.25 $\pm$ 24.14
Llama-3.1-8B-CollabSciReM	7.15 $\pm$ 0.20	6.84 $\pm$ 0.25	8.19 $\pm$ 0.04	6.94 $\pm$ 0.16	8.52 $\pm$ 0.08	310.00 $\pm$ 4.00	36.86 $\pm$ 0.89	147.60 $\pm$ 8.63	250.47 $\pm$ 15.27
Qwen-2.5-7B-Instruct	7.91 $\pm$ 0.09	7.62 $\pm$ 0.11	8.70 $\pm$ 0.07	7.80 $\pm$ 0.04	8.90 $\pm$ 0.03	555.00 $\pm$ 10.00	66.07 $\pm$ 1.03	226.34 $\pm$ 16.16	292.73 $\pm$ 16.98

Table 1: Performance of 7B and 8B models on (a) multi-turn benchmarks (MT-Bench-STEM, MINT) and (b) single-turn benchmarks (GPQA-D, MMLU-Valid, TheoremQA). Acc : accuracy (%). SR@5 : success rate within five interaction turns (%). TC : average tokens per response (per turn for MT-Bench-STEM). APT : accuracy per token, computed as

\text{Acc}/\text{TC}\times 1000

(SR@5 in place of Acc for MINT). Results are mean

\pm

sample standard deviation over three seeds; bold values indicate the best performance in each column within each model family.

(a) Multi-turn benchmarks
	MT-Bench-STEM						MINT
Reward Model	Overall ( $\uparrow$ )	Acc. ( $\uparrow$ )	Format ( $\uparrow$ )	Useful. ( $\uparrow$ )	Relev. ( $\uparrow$ )	TC ( $\downarrow$ )	SR@5 (%) ( $\uparrow$ )	TC ( $\downarrow$ )	APT ( $\uparrow$ )
$R_{\text{gen}}+R_{\text{struct}}$	6.81 $\pm$ 0.23	6.48 $\pm$ 0.23	8.03 $\pm$ 0.09	6.67 $\pm$ 0.26	8.38 $\pm$ 0.07	319 $\pm$ 3.00	40.59 $\pm$ 2.57	139.31 $\pm$ 1.87	291.45 $\pm$ 19.60
$R_{\text{gen}}+R_{\text{sci}}$	6.58 $\pm$ 0.26	6.33 $\pm$ 0.25	7.74 $\pm$ 0.27	6.46 $\pm$ 0.43	8.10 $\pm$ 0.26	622 $\pm$ 15.00	48.30 $\pm$ 4.45	699.95 $\pm$ 57.14	69.05 $\pm$ 4.17
$R_{\text{struct}}+R_{\text{sci}}$	6.92 $\pm$ 0.15	6.67 $\pm$ 0.10	8.06 $\pm$ 0.19	6.67 $\pm$ 0.12	8.30 $\pm$ 0.17	319 $\pm$ 8.00	40.74 $\pm$ 2.72	149.81 $\pm$ 0.99	271.88 $\pm$ 16.36
CollabSciReM $\mid 0.7\,\gamma_{\textsc{CSciReM}}$	7.14 $\pm$ 0.14	6.84 $\pm$ 0.08	8.27 $\pm$ 0.09	6.85 $\pm$ 0.15	8.52 $\pm$ 0.23	308 $\pm$ 6.00	41.78 $\pm$ 4.00	148.24 $\pm$ 6.28	282.92 $\pm$ 39.05

Table 2: Ablation and sensitivity analysis of reward components for CollabSciReM on Llama-3.1-8B-Instruct, fine-tuned with SFT followed by offline IPO. The upper block reports multi-turn benchmarks, MT-Bench-STEM and MINT, and the lower block reports single-turn benchmarks, GPQA-D, MMLU-Valid, and TheoremQA.

R_{\text{gen}}

R_{\text{struct}}

, and

R_{\text{sci}}

denote the conversational, structural, and science-aware rewards, respectively.

Panel of Figure 3 — Figure 3: Real-world user study results showcasing participant interactions with anonymized base Llama-3.1-8B-Instruct , general rewards (accuracy and interaction, w/o token penalty) and CollabSciReM models. Users rated (a) logical justification (1-10), (b) overall interactive experience (1-10), and (c) novelty (1-5).

Category	Metric ( $\phi$ )	CollabSciReM	CollabLLM	User study	Description
$\mathcal{R}_{\text{gen}}$	Accuracy	+1.3	+1.0	+1.0	Correctness of technical claims and data.
	Completeness	+0.9	–	–	Coverage of all user sub-questions.
	Clarity	+0.4	–	–	Syntactic and semantic ease of understanding.
	Interactivity	+0.3	+0.5	+0.5	Proactive engagement and user feedback.
$\mathcal{R}_{\text{sci}}$	Justification	+0.8	–	–	Provision of evidence, citations, or derivations.
	Logical Flow	+0.6	–	–	Internal consistency and step-by-step reasoning.

Table 3: Reward weight (

\gamma

) decomposition and metric definitions.

Hyperparameter	SFT	IPO
Hardware & Infrastructure
GPUs	4 $\times$ NVIDIA H100 (80GB)	4 $\times$ NVIDIA H100 (80GB)
Precision	bfloat16	bfloat16
Parallelism	DeepSpeed ZeRO-2	DeepSpeed ZeRO-2
LoRA Configuration
LoRA Rank ( $r$ )	32	32

Table 4: Hyperparameters for SFT and IPO training stages.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Synthetic dataset generation pipeline from a research paper to a conversational excerpt. (a) A research paper (excerpt from ( Yadav and Claeys Bouuaert, 2021 ) ) is transformed into (b) a structured representation in which research questions and methodologies are extracted. (c) These components are formalized into conversational excerpts, with user queries grounded in research questions and model responses derived from the associated methodologies.

Panel of Figure 5 — Figure 5: Snippets from the user study evaluation interface.

Model	Strengths	Weaknesses
Llama-3.1-8B-Instruct	“Clear and answered the questions appropriately.", “Strong structure and clarity."	“Flawed reasoning.", “The model’s approach was generic and lacked innovation, …", “The model provided too many options to try, and its initial response was not thorough at all. It felt as though I had to pry additional steps out of it."
General-reward Variant	“Decent response formatting." , “Clear, concise", “Clear language flow and formatting."	“I had to point it more specifically at research methodology over answering the question.", “It got some technical information wrong..", “Limited novelty."
CollabSciReM	“It was easy to understand; well structured and I liked how it was straight to the point and just gave an overview.", “Clear, structured, and readable"	“While methodologically solid, the responses rely heavily on standard qualitative techniques.", “Too concise…"

Appendix

Table 5: Qualitative Feedback from Human Evaluators.

(a) Multi-turn benchmarks
	MT-Bench-STEM						MINT
Model	Overall ( $\uparrow$ )	Acc. ( $\uparrow$ )	Format ( $\uparrow$ )	Useful. ( $\uparrow$ )	Relev. ( $\uparrow$ )	TC ( $\downarrow$ )	SR@5 (%) ( $\uparrow$ )	TC ( $\downarrow$ )	APT ( $\uparrow$ )
Llama-3.2-3B-Instruct	6.40 $\pm$ 0.38	6.02 $\pm$ 0.46	7.82 $\pm$ 0.20	6.25 $\pm$ 0.48	8.12 $\pm$ 0.29	447 $\pm$ 26	35.56 $\pm$ 1.94	265.06 $\pm$ 20.44	135.01 $\pm$ 17.10
Llama-3.2-3B-CollabLLMSci	6.48 $\pm$ 0.38	6.06 $\pm$ 0.42	7.93 $\pm$ 0.24	6.15 $\pm$ 0.25	8.24 $\pm$ 0.15	344 $\pm$ 9	31.85 $\pm$ 2.89	174.44 $\pm$ 4.47	182.94 $\pm$ 21.00
Llama-3.2-3B-CollabSciReM	6.24 $\pm$ 0.20	5.83 $\pm$ 0.24	7.82 $\pm$ 0.09	5.98 $\pm$ 0.21	8.26 $\pm$ 0.13	358 $\pm$ 16	35.41 $\pm$ 2.60	181.63 $\pm$ 3.06	195.13 $\pm$ 17.35
Qwen-2.5-3B-Instruct	7.44 $\pm$ 0.06	7.15 $\pm$ 0.02	8.54 $\pm$ 0.05	7.22 $\pm$ 0.09	8.82 $\pm$ 0.06	521 $\pm$ 23	52.15 $\pm$ 0.93	287.66 $\pm$ 1.29	181.29 $\pm$ 3.24

Appendix

Table 6: Performance of 3B models on (a) multi-turn benchmarks (MT-Bench-STEM, MINT) and (b) single-turn benchmarks (GPQA-D, MMLU-Valid, TheoremQA). Metrics and evaluation settings follow Table 1 . Results are mean

\pm

sample standard deviation over three seeds; bold values indicate the best performance in each column within each model family.

Dimension	Definition
Methodological soundness	Assesses whether the proposed study design is valid, with appropriate controls, careful handling of confounds, sound measurement and analysis, and a feasible plan.
Justification	Assesses whether each design choice is supported by specific, checkable reasons. Citations count only if they identify a real, relevant work, and invented, vague or unverifiable references count against the response.
Logical consistency	Assesses whether the response is free of contradictions within and across turns and whether later turns build coherently on earlier ones.
Factual correctness	Assesses the technical and factual accuracy of the response. The judge lists the concrete errors it finds for each side.
Novelty	Assesses whether the response offers non-obvious, useful ideas that go beyond a textbook answer.
Responsiveness	Assesses how well the assistant addresses the researcher’s evolving requests and questions across turns.

Appendix

Table 7: Scientific-quality rubric.

LLM Alignment

Large language model alignment studies methods for making models follow human intentions, values, and safety constraints across contexts.

See all 611 papers

COGTRL: Training LLMs for Scientific Discovery Assistance using Cognitive Traces via Reinforcement Learning
Aug 31, 2026Shrinidhi Kumbhar, Santosh Mashetty, Divij Handa +3RL for Language Model Reasoning LLM Post-Training
SciIntBench: Measuring LLM Compliance with Research Integrity Norms Under Adversarial Framing
May 28, 2026Almene De Meran Meguimtsop, Maria Leonor Pacheco, Daniel E. AcunaAI for Science Adversarial Attacks on LLMs
Department of Computer Science, University of Colorado Boulder, Boulder, CO 80309
Eigen-Agent: Adaptive Multi-Agent Scientific Reasoning with Monitor-Based RAG
Sep 25, 2025Xiangru Tang, Wanghan Xu, Yujie Wang +13Scientific Reasoning in Language Models Multi-Agent Reasoning
Yale University · Shanghai Jiao Tong University · Fudan University +4

Aug 31, 2026J/K move · Enter open · S save

COGTRL: Training LLMs for Scientific Discovery Assistance using Cognitive Traces via Reinforcement Learning

Shrinidhi Kumbhar, Santosh Mashetty, Divij Handa, Kevin Coutinho+2

RL for Language Model Reasoning LLM Post-Training Large Language Model-Guided Scientific Discovery

Open paper AskPDF arXiv

Large Language Models (LLMs) trained on extensive scientific research are increasingly integrated as assistants for scientific discovery. However, most research papers omit the fine-grained cognitive process of examining constraints, failed alternatives, and iterative decisions required to achieve the desired goal. Such cognitive processes are vital for real-world scientists working toward specific goals under constraints. In this paper, we show that LLMs, when trained to produce such cognitive traces, perform better as scientific discovery assistants than when trained solely on scientific literature. We propose COGTRL, a trajectory-level reinforcement learning framework that trains LLMs to emulate cognitively grounded reasoning by jointly optimizing cognitive traces and the scientific steps produced in an interleaved manner. Across two 3B-parameter models and two scientific domains (AI and Materials Science), COGTRL improves method quality by an average of 7.85 points over comparable 3B model baselines and achieves competitive performance relative to 70B parameter models. Moreover, analysis by domain experts shows a preference for methods generated by COGTRL over the baselines.

Abstract

LLM Alignment

Explore similar work

COGTRL: Training LLMs for Scientific Discovery Assistance using Cognitive Traces via Reinforcement Learning

SciIntBench: Measuring LLM Compliance with Research Integrity Norms Under Adversarial Framing

Eigen-Agent: Adaptive Multi-Agent Scientific Reasoning with Monitor-Based RAG