Halal-NeSyNet: Clause-Scoped Neuro-Symbolic Cross-Attention for Automated Food Ingredient Auditing and Risk-Aware Halal Logistics Triage

Mochamad Rizal Fauzan; Naila Nursyifa; Irgi Surya

cs.CLNov 1, 2026

Halal-NeSyNet: Clause-Scoped Neuro-Symbolic Cross-Attention for Automated Food Ingredient Auditing and Risk-Aware Halal Logistics Triage

Authors: Mochamad Rizal Fauzan, Naila Nursyifa, Irgi Surya

Organizations: National Taipei University of Technology, Taipei, Taiwan · Logistics Engineering, Universitas Pendidikan Indonesia, Bandung, Indonesia

Abstract

Verifying that a packaged food is Halal requires reading ingredient declarations and E-number additives before goods enter certified warehouses. Dual-origin additives such as E471, E422, E441 and E920 are Mushbooh (doubtful) unless a botanical, microbial or certified-Halal qualifier appears in the same ingredient clause, so a plant-based qualifier on one clause must not clear an unqualified additive in another. We call a failure to respect this scope cross-clause modifier confounding and build a benchmark of 3,199 items (277 Open Food Facts records and 2,922 template-generated clause-swap items, labelled by an explicit 50-code rule base rather than by a certification authority) with a 540-item counterfactual suite. We propose Halal-NeSyNet, which couples a DistilBERT encoder with a 16-dimensional rule-derived symbolic vector through cross-attention and gated fusion, trained with an asymmetric false-Halal risk loss. Over five seeds, text-only TF-IDF baselines reach 86.25-86.71 % accuracy with 13.42-21.73 % false-Halal leakage and 6.33-26.00 % counterfactual triad consistency, whereas Halal-NeSyNet reaches 97.04+/-0.34 % and 99.67+/-0.50 %. Gradient boosting given the same symbolic vector reaches 96.25+/-1.24 % (+0.59 points from Halal-NeSyNet, 95 % CI [-1.11, +2.28]), so most of the gain comes from clause-scoped symbolic features and not from the network, and the risk loss did not help. Holding out whole template families, text-only models fall to 61.91-78.35 %, Halal-NeSyNet to 90.87+/-11.76 % and logistic regression with the symbolic vector to 91.11+/-2.88 %, with far more spread for Halal-NeSyNet. A three-lane receiving simulation auto-clears 49.33 % of cargo at 0.76 % green-lane contamination. Because labels and features come from one rule engine and 91.3 % of the items are templated, the results measure fidelity to a rule policy, not religious certification.

Figures & tables

Work	Input	Method (as stated)	Output labels	Qualifier scope tested
Kartiwi et al. [ 21 ]	photo of ingredient list	OCR, then flag unlawful items	lawful / unlawful items	not stated
Fadhilah et al. [ 24 ]	photo of packaging	character CNN, word list of non-Halal compositions	non-Halal detection	not stated
Tarannum et al. [ 22 ]	photo of packaging	YOLOv5 + OCR, then ML, neural and rule-based classifiers	Halal / Haram	not stated
Bagundang and Gallo [ 25 ]	ingredient list (text)	Naive Bayes, LR, SVM, random forest	Halal / Haram	not stated
Mulyana et al. [ 26 ]	ingredient list (text)	BiLSTM, Word2Vec / GloVe	Halal / Haram	not stated
Hoang et al. [ 27 ]	cosmetics-ingredient graph	knowledge-graph completion	Halal status (cosmetics)	not applicable

TABLE I: Positioning against the closest automated Halal screening studies. Entries are taken from the abstracts of the cited works; “not stated” means that the abstract does not mention the item, not that the study lacks it.

Fig. 1: Cross-clause modifier confounding. A plant-based qualifier on a benign shortening clause (

c_{2}

) coexists with an unqualified E471 emulsifier in

c_{3}

. A bag of words merges the clauses and predicts Halal, a false-Halal leak; a clause-scoped parser keeps the qualifier with

c_{2}

, flags

c_{3}

as unresolved (

s_{i,6}=1

, Table III ) and routes the item to the amber lane.

Source	Halal	Mushbooh	Non-Halal	Total
Open Food Facts records	206	48	23	277
Template-generated items	1,455	880	587	2,922
All items	1,661	928	610	3,199

TABLE II: Benchmark composition by source and class (labels from the rule base).

Index	Feature
$s_{0},s_{1}$	explicit en:halal tag; en:vegan or en:vegetarian tag (0/1)
$s_{2}$	number of Haram E-codes (text or tags)
$s_{3}$	clauses with an un-negated Haram pattern or E-code
$s_{4}$	clauses with a negated Haram phrase (e.g. ginger beer)
$s_{5}$	number of Mushbooh E-codes
$s_{6}$	dual-origin clauses without a local qualifier (unresolved)

TABLE III: The 16-dimensional symbolic vector

\mathbf{s}_{i}

produced by the clause parser (raw counts unless stated;

m_{i}

is the number of clauses).

Fig. 2: Architecture of Halal-NeSyNet . (I) A parenthesis-balanced clause splitter and the rule engine turn a SKU declaration into clause-marked text and a 16-dimensional symbolic vector

\mathbf{s}_{i}

. (II) A DistilBERT text stream yields token states

\mathbf{H}_{\mathrm{text}}

and

\mathbf{h}_{\mathrm{cls}}

; a query MLP turns

\mathbf{s}_{i}

into four queries

\mathbf{Q}_{\mathrm{ont}}

and a direct embedding

\mathbf{e}_{\mathrm{sym}}

. (III) The queries attend over the token states, and a sigmoid gate blends

\mathbf{h}_{\mathrm{cls}}

with the pooled attention output. (IV) A classifier produces three logits, trained with

\mathcal{L}_{\mathrm{AHR}}

; at inference the maximum softmax confidence, compared with the threshold

\tau

, routes the item to the green, amber or red lane.

Model	Acc. (%)	Macro-F1 (%)	ROC-AUC (%)	FHLR (%) $\downarrow$	Haram $\to$ Halal (%) $\downarrow$
Logistic Regression (TF-IDF)	86.62 $\pm$ 1.45	87.81 $\pm$ 1.38	97.13 $\pm$ 0.63	13.42 $\pm$ 2.72	3.71 $\pm$ 2.51
Random Forest (TF-IDF)	86.25 $\pm$ 1.39	86.27 $\pm$ 1.45	96.22 $\pm$ 0.56	21.73 $\pm$ 1.92	6.11 $\pm$ 3.22
XGBoost (TF-IDF)	86.71 $\pm$ 1.45	87.64 $\pm$ 1.41	96.83 $\pm$ 0.45	15.24 $\pm$ 1.54	2.18 $\pm$ 1.54
Logistic Regression (TF-IDF + $\mathbf{s}$ )	94.54 $\pm$ 0.85	94.29 $\pm$ 0.89	99.42 $\pm$ 0.12	3.46 $\pm$ 0.31	2.62 $\pm$ 1.25
XGBoost (TF-IDF + $\mathbf{s}$ )	96.25 $\pm$ 1.24	96.24 $\pm$ 1.33	99.66 $\pm$ 0.15	3.81 $\pm$ 0.83	2.83 $\pm$ 2.11
Vanilla DistilBERT	93.71 $\pm$ 0.35	93.19 $\pm$ 0.21	99.21 $\pm$ 0.23	6.58 $\pm$ 1.85	5.24 $\pm$ 2.08

TABLE IV: Five-seed test results (mean

\pm

SD). FHLR: share of true Non-Halal or Mushbooh items predicted Halal; Haram

\to

Halal: share of true Non-Halal items predicted Halal (lower is better for both).

\mathbf{s}

: the 16-dimensional symbolic vector. Best mean per column in bold. The two “

+\,\mathbf{s}

” rows are the values of the original run.

Fig. 3: Trade-offs (mean

\pm

SD over five seeds). (a) Accuracy against false-Halal leakage; the best models sit at the upper left. (b) Share of cargo auto-cleared to the green lane against the contamination of that lane at

\tau=0.85

; the best models sit at the lower right. The models with the symbolic vector

\mathbf{s}

and Halal-NeSyNet are close in both panels.

Class	Support	LR	RF	XGB	DistilBERT	NeSyNet
Halal	249	88.45	89.80	89.41	95.07	97.05
Non-Halal	92	95.45	93.18	97.78	92.40	95.45
Mushbooh	139	80.85	80.82	81.48	92.20	96.73

TABLE V: Per-class F1 (%) on the test split of one run (seed 42); support is the number of test items.

Fig. 4: Confusion matrices on the seed-42 test split ( $N=480$ ; H: Halal, NH: Non-Halal, M: Mushbooh; shading is the row-normalised share).

Comparison	Metric	Mean difference	95% CI	Better in
NeSyNet $-$ Vanilla DistilBERT	Accuracy (pp)	+3.34	[+2.93, +3.75]	5/5 seeds
NeSyNet $-$ Vanilla DistilBERT	FHLR (pp)	-1.90	[-4.58, +0.77]	4/5 seeds
NeSyNet $-$ XGBoost (TF-IDF)	Accuracy (pp)	+10.33	[+8.13, +12.54]	5/5 seeds
NeSyNet $-$ XGBoost (TF-IDF)	FHLR (pp)	-10.56	[-13.02, -8.11]	5/5 seeds
NeSyNet $-$ XGBoost (TF-IDF + $\mathbf{s}$ )	Accuracy (pp)	+0.59	[-1.11, +2.28]	3/5 seeds
NeSyNet $-$ XGBoost (TF-IDF + $\mathbf{s}$ )	FHLR (pp)	+1.21	[-0.28, +2.71]	1/5 seeds

TABLE VI: Seed-paired differences over the five seeds (first model minus second; mean,

t

-based 95 % interval, number of seeds in which the first model is better). Rows with

\mathbf{s}

use the re-run of the ontology-augmented baselines. † The second model is also trained without

\mathcal{L}_{\mathrm{AHR}}

, on the plain text with

L=112

Fig. 5: Test accuracy of every seed (dots) and the five-seed mean (black bars); grey lines join the same seed. The models with

\mathbf{s}

and Halal-NeSyNet overlap; XGB+s uses the re-run.

Variant	Acc. (%)	Macro-F1 (%)	FHLR (%) $\downarrow$	Haram $\to$ Halal (%) $\downarrow$
Halal-NeSyNet (full)	97.04 $\pm$ 0.34	96.90 $\pm$ 0.35	4.68 $\pm$ 1.16	5.24 $\pm$ 2.36
w/o $\mathcal{L}_{\mathrm{AHR}}$ (weighted CE)	97.54 $\pm$ 0.40	97.47 $\pm$ 0.44	4.59 $\pm$ 1.09	4.58 $\pm$ 2.08
w/o cross-attention †	96.25 $\pm$ 0.87	96.03 $\pm$ 0.99	6.41 $\pm$ 1.24	5.89 $\pm$ 2.12
w/o symbolic stream (= Vanilla DistilBERT) ‡	93.71 $\pm$ 0.35	93.19 $\pm$ 0.21	6.58 $\pm$ 1.85	5.24 $\pm$ 2.08

TABLE VII: Ablation of Halal-NeSyNet (mean

\pm

SD over five seeds). † Also trained without

\mathcal{L}_{\mathrm{AHR}}

, on the plain text with

L=112

. ‡ Plain text,

L=96

, learning rate

2.5\times 10^{-5}

, no symbolic stream. The rows differ in more than one factor, so this is not a clean single-factor study.

Additive	$N$	LR	RF	XGB	DistilBERT	NeSyNet
E120	20	95.0 / 5.0	85.0 / 15.0	100.0 / 0.0	85.0 / 10.0	95.0 / 0.0
E471	143	90.2 / 8.3	91.6 / 10.7	93.0 / 7.1	97.9 / 2.4	98.6 / 1.2
E422	38	79.0 / 18.8	89.5 / 25.0	84.2 / 12.5	89.5 / 18.8	92.1 / 6.2
E441	39	89.7 / 4.3	84.6 / 17.4	92.3 / 4.3	92.3 / 8.7	97.4 / 4.3

TABLE VIII: Accuracy / false-Halal leakage (%) on test items that mention a sensitive additive (seed 42).

N

is the subset size. LR: logistic regression, RF: random forest, XGB: XGBoost, all on TF-IDF features.

Model	CF acc. (%)	Triad cons. (%)	CF FHLR (%) $\downarrow$
LR (TF-IDF)	71.89 $\pm$ 2.38	26.00 $\pm$ 8.05	26.83 $\pm$ 4.10
RF (TF-IDF)	58.07 $\pm$ 5.28	6.33 $\pm$ 6.91	28.67 $\pm$ 8.55
XGB (TF-IDF)	62.15 $\pm$ 2.17	7.33 $\pm$ 4.94	21.17 $\pm$ 3.26
LR (TF-IDF + $\mathbf{s}$ )	99.89 $\pm$ 0.17	99.67 $\pm$ 0.50	0.00 $\pm$ 0.00
XGB (TF-IDF + $\mathbf{s}$ )	93.59 $\pm$ 3.16	82.11 $\pm$ 8.05	0.00 $\pm$ 0.00
Vanilla DistilBERT	99.22 $\pm$ 1.15	97.67 $\pm$ 3.45	1.17 $\pm$ 1.73

TABLE IX: Counterfactual suite (540 items, 180 triplets; mean

\pm

SD over five seeds). Triad consistency requires all three members of a triplet to be predicted correctly; CF FHLR is the false-Halal leakage on the suite. The two “

+\,\mathbf{s}

” rows are the values of the original run. LR: logistic regression, RF: random forest, XGB: XGBoost;

\mathbf{s}

: symbolic vector.

Fig. 6: Counterfactual accuracy and strict triad consistency (mean

\pm

SD over five seeds).

Model	Auto-cleared to green lane (%)	Green-lane contamination (%) $\downarrow$
LR (TF-IDF)	22.25 $\pm$ 1.85	0.00 $\pm$ 0.00
RF (TF-IDF)	18.63 $\pm$ 1.26	0.00 $\pm$ 0.00
XGB (TF-IDF)	35.96 $\pm$ 2.59	4.11 $\pm$ 1.68
LR (TF-IDF + $\mathbf{s}$ )	42.96 $\pm$ 0.86	0.96 $\pm$ 0.75
XGB (TF-IDF + $\mathbf{s}$ )	48.54 $\pm$ 0.91	1.46 $\pm$ 0.48
Vanilla DistilBERT	47.13 $\pm$ 2.63	1.26 $\pm$ 1.43

TABLE X: Three-lane triage at

\tau=0.85

(mean

\pm

SD over five seeds). Rows with

\mathbf{s}

are from the re-run of the ontology-augmented baselines. LR: logistic regression, RF: random forest, XGB: XGBoost;

\mathbf{s}

: symbolic vector.

Model	Acc. (%)	Macro-F1 (%)	FHLR (%) $\downarrow$	Haram $\to$ Halal (%) $\downarrow$	CF triad (%)
Logistic Regression (TF-IDF)	78.35 $\pm$ 11.92	79.94 $\pm$ 11.30	9.35 $\pm$ 9.29	10.86 $\pm$ 15.38	2.96 $\pm$ 5.38
Random Forest (TF-IDF)	61.91 $\pm$ 14.77	60.23 $\pm$ 14.82	21.32 $\pm$ 16.84	19.02 $\pm$ 17.34	0.00 $\pm$ 0.00
XGBoost (TF-IDF)	67.10 $\pm$ 18.26	63.54 $\pm$ 18.18	23.14 $\pm$ 23.39	30.68 $\pm$ 32.67	0.00 $\pm$ 0.00
Logistic Regression (TF-IDF + $\mathbf{s}$ )	91.11 $\pm$ 2.88	90.75 $\pm$ 3.17	7.30 $\pm$ 5.88	6.80 $\pm$ 9.63	84.44 $\pm$ 28.80
XGBoost (TF-IDF + $\mathbf{s}$ )	86.70 $\pm$ 5.50	84.66 $\pm$ 8.15	9.87 $\pm$ 9.69	12.38 $\pm$ 18.64	52.96 $\pm$ 47.65
Vanilla DistilBERT	86.85 $\pm$ 14.89	85.07 $\pm$ 17.09	15.10 $\pm$ 23.18	7.76 $\pm$ 12.88	79.63 $\pm$ 38.17

TABLE XI: Leave-one-template-family-out results (mean

\pm

SD over the ten held-out families, seeds averaged within a family). CF triad: strict triad consistency on the counterfactual triplets of the held-out family (six families).

Fig. 7: Leave-one-template-family-out. (a) Test accuracy and (b) false-Halal leakage on the held-out family for each model; each dot is one family (mean of the seeds), grey lines join the same family and the black bar is the mean over the ten families.

Held-out family	LR	RF	XGB	LR+s	XGB+s	DistilBERT	NeSyNet	NeSyNet w/o AHR
E471	72.0	61.7	76.4	91.2	91.7	94.7	97.4	98.2
E472e	82.3	77.9	65.2	89.2	86.8	77.2	87.0	92.2
E441	57.8	68.8	70.0	87.1	86.6	59.5	64.6	67.4
E422	79.2	46.0	39.9	88.3	84.4	95.2	98.0	97.7
rennet	69.9	47.7	63.8	87.4	77.8	94.2	97.7	97.9
E920	74.9	57.4	69.0	91.9	79.9	95.9	96.7	97.1

TABLE XII: Test accuracy (%) on each held-out template family (mean of 3 seeds); best per family in bold.

Comparison	Metric	Mean difference	95% CI	Better in
NeSyNet $-$ Vanilla DistilBERT	Accuracy (pp)	+4.02	[+0.94, +7.10]	9/10 families
NeSyNet $-$ Vanilla DistilBERT	FHLR (pp)	-6.51	[-12.39, -0.63]	9/10 families
NeSyNet $-$ XGBoost (TF-IDF + $\mathbf{s}$ )	Accuracy (pp)	+4.18	[-5.27, +13.63]	8/10 families
NeSyNet $-$ XGBoost (TF-IDF + $\mathbf{s}$ )	FHLR (pp)	-1.28	[-12.51, +9.95]	8/10 families
NeSyNet $-$ LR (TF-IDF + $\mathbf{s}$ )	Accuracy (pp)	-0.23	[-8.34, +7.87]	7/10 families
NeSyNet $-$ LR (TF-IDF + $\mathbf{s}$ )	FHLR (pp)	+1.30	[-10.94, +13.54]	8/10 families

TABLE XIII: Family-paired differences over the ten held-out families (first model minus second; mean,

t

-based 95 % interval, number of families in which the first model is better).

Neuro-Symbolic Reasoning

Neuro-symbolic reasoning combines learned neural models with explicit logic, rules, knowledge structures, or executable tools to solve structured tasks.

See all 284 papers

Feb 19, 2026

HalalBench: A Multilingual OCR Benchmark for Food Packaging Ingredient Extraction

No standardized benchmark exists for evaluating OCR on food packaging, despite its critical role in automated halal food verification. Existing benchmarks target documents or scene text, missing the unique challenges of ingredient labels: curved surfaces, dense multilingual text, and sub-8pt fonts. We present HalalBench, the first open multilingual benchmark for food packaging OCR, comprising 1,043 images (50 real, 993 synthetic) with 36,438 annotations in COCO format spanning 14 languages. We evaluate four engines: docTR achieves F1=0.193, ML Kit 0.180, EasyOCR 0.167, while all fail on Japanese (F1=0.000). A clustering ablation shows 36% F1 improvement from our post-processing algorithm. We validate findings through HalalLens (https://halallens.no), a production halal scanner serving 20+ countries. Dataset and code are released under open licenses.

Hasan Arief

HalalLens Research

Multilingual OCR Optical Character Recognition

Ask PDF

Aug 4, 2026

Detecting Hallucinations and Recovering Verified Answers in Arabic Islamic Question Answering

Large language models can generate fluent responses to Islamic questions while introducing factual errors that are difficult to identify. This paper presents our system for HalluScoring 2026 Task 2.1, Islamic Hallucination Detection and Find the Truth. The task requires a unified two-step prediction: determining whether an Arabic answer generated by an LLM is hallucinated and selecting the verified answer from six closely related candidate options. We use the Islamic knowledge dataset provided by the shared task, which contains 600 question--answer instances, including 341 hallucinated and 259 non-hallucinated answers. Our system is based on the fine-tuned google/gemma-4-12B-it model and uses deterministic decoding during inference. The generated outputs are normalized to extract the hallucination label and the selected option. The system achieves a Macro-F1 score of 0.928 and a label accuracy of 0.935 for hallucination detection, together with an option accuracy of 0.895 for answer selection. These results yield a combined score of 0.912, demonstrating strong performance across both stages of the task. The lower option-selection accuracy indicates that distinguishing the verified answer from plausible alternatives remains more challenging than detecting hallucinated responses.

Khaled Ziani

Independent Researcher, Paris, France

LLM Answer Verification Question Answering LLM Hallucination Detection

Ask PDF

Sep 29, 2026

Halluscoring 2026: The first shared task on llms hallucination detection and answer verification

We present HalluScoring 2026, a shared task for evaluating hallucination detection and factual verification in Arabic question answering under challenging generalization settings. Its four subtasks are organized into two tasks. Task~1 evaluates binary hallucination detection, considering generalization to unseen questions (Subtask 1.1) and responses generated by unseen LLMs (Subtask 1.2). Task 2 extends the evaluation beyond detection by requiring the systems to additionally identify the correct factual answer from six related candidates, covering Islamic knowledge (Subtask 2.1) and general knowledge (Subtask 2.2). The shared task is based on two Arabic datasets: HalluScore and HalluTruthQA. A total of 13 teams participated in the shared task, ten of which submitted system description papers. The results of Task 1 demonstrate that hallucination detection remains challenging. On the Task 1 test sets, the top-ranked systems achieved AUC-ROC scores of 0.7717 for Subtask 1.1 (REGLAT) and 0.7670 for Subtask 1.2 (NAMAA). Under assisted evaluation, the highest combined detection and answer-selection scores for Subtasks 2.1 and 2.2 were 0.8824 and 0.8565, respectively.

Aisha Alansari, Abdessalam Bouchekif, Ahmed Hasanaath +9

King Fahd University of Petroleum and Minerals, Saudi Arabia · Hamad Bin Khalifa University, Qatar · University of the Basque Country, Spain +5

LLM Evaluation LLM Answer Verification LLM Hallucination Detection

Ask PDF

Abstract

Neuro-Symbolic Reasoning

Explore similar work

HalalBench: A Multilingual OCR Benchmark for Food Packaging Ingredient Extraction

Detecting Hallucinations and Recovering Verified Answers in Arabic Islamic Question Answering

Halluscoring 2026: The first shared task on llms hallucination detection and answer verification