Halal-NeSyNet: Clause-Scoped Neuro-Symbolic Cross-Attention for Automated Food Ingredient Auditing and Risk-Aware Halal Logistics Triage

Mochamad Rizal Fauzan; Naila Nursyifa; Irgi Surya

cs.CLNov 1, 2026

Halal-NeSyNet: Clause-Scoped Neuro-Symbolic Cross-Attention for Automated Food Ingredient Auditing and Risk-Aware Halal Logistics Triage

Authors: Mochamad Rizal Fauzan, Naila Nursyifa, Irgi Surya

Organizations: National Taipei University of Technology, Taipei, Taiwan · Logistics Engineering, Universitas Pendidikan Indonesia, Bandung, Indonesia

Abstract

Verifying that a packaged food is Halal requires reading ingredient declarations and E-number additives before goods enter certified warehouses. Dual-origin additives such as E471, E422, E441 and E920 are Mushbooh (doubtful) unless a botanical, microbial or certified-Halal qualifier appears in the same ingredient clause, so a plant-based qualifier on one clause must not clear an unqualified additive in another. We call a failure to respect this scope cross-clause modifier confounding and build a benchmark of 3,199 items (277 Open Food Facts records and 2,922 template-generated clause-swap items, labelled by an explicit 50-code rule base rather than by a certification authority) with a 540-item counterfactual suite. We propose Halal-NeSyNet, which couples a DistilBERT encoder with a 16-dimensional rule-derived symbolic vector through cross-attention and gated fusion, trained with an asymmetric false-Halal risk loss. Over five seeds, text-only TF-IDF baselines reach 86.25-86.71 % accuracy with 13.42-21.73 % false-Halal leakage and 6.33-26.00 % counterfactual triad consistency, whereas Halal-NeSyNet reaches 97.04+/-0.34 % and 99.67+/-0.50 %. Gradient boosting given the same symbolic vector reaches 96.25+/-1.24 % (+0.59 points from Halal-NeSyNet, 95 % CI [-1.11, +2.28]), so most of the gain comes from clause-scoped symbolic features and not from the network, and the risk loss did not help. Holding out whole template families, text-only models fall to 61.91-78.35 %, Halal-NeSyNet to 90.87+/-11.76 % and logistic regression with the symbolic vector to 91.11+/-2.88 %, with far more spread for Halal-NeSyNet. A three-lane receiving simulation auto-clears 49.33 % of cargo at 0.76 % green-lane contamination. Because labels and features come from one rule engine and 91.3 % of the items are templated, the results measure fidelity to a rule policy, not religious certification.

Figures & tables

Work	Input	Method (as stated)	Output labels	Qualifier scope tested
Kartiwi et al. [ 21 ]	photo of ingredient list	OCR, then flag unlawful items	lawful / unlawful items	not stated
Fadhilah et al. [ 24 ]	photo of packaging	character CNN, word list of non-Halal compositions	non-Halal detection	not stated
Tarannum et al. [ 22 ]	photo of packaging	YOLOv5 + OCR, then ML, neural and rule-based classifiers	Halal / Haram	not stated
Bagundang and Gallo [ 25 ]	ingredient list (text)	Naive Bayes, LR, SVM, random forest	Halal / Haram	not stated
Mulyana et al. [ 26 ]	ingredient list (text)	BiLSTM, Word2Vec / GloVe	Halal / Haram	not stated
Hoang et al. [ 27 ]	cosmetics-ingredient graph	knowledge-graph completion	Halal status (cosmetics)	not applicable

TABLE I: Positioning against the closest automated Halal screening studies. Entries are taken from the abstracts of the cited works; “not stated” means that the abstract does not mention the item, not that the study lacks it.

Fig. 1: Cross-clause modifier confounding. A plant-based qualifier on a benign shortening clause (

c_{2}

) coexists with an unqualified E471 emulsifier in

c_{3}

. A bag of words merges the clauses and predicts Halal, a false-Halal leak; a clause-scoped parser keeps the qualifier with

c_{2}

, flags

c_{3}

as unresolved (

s_{i,6}=1

, Table III ) and routes the item to the amber lane.

Source	Halal	Mushbooh	Non-Halal	Total
Open Food Facts records	206	48	23	277
Template-generated items	1,455	880	587	2,922
All items	1,661	928	610	3,199

TABLE II: Benchmark composition by source and class (labels from the rule base).

Index	Feature
$s_{0},s_{1}$	explicit en:halal tag; en:vegan or en:vegetarian tag (0/1)
$s_{2}$	number of Haram E-codes (text or tags)
$s_{3}$	clauses with an un-negated Haram pattern or E-code
$s_{4}$	clauses with a negated Haram phrase (e.g. ginger beer)
$s_{5}$	number of Mushbooh E-codes
$s_{6}$	dual-origin clauses without a local qualifier (unresolved)

TABLE III: The 16-dimensional symbolic vector

\mathbf{s}_{i}

produced by the clause parser (raw counts unless stated;

m_{i}

is the number of clauses).

Fig. 2: Architecture of Halal-NeSyNet . (I) A parenthesis-balanced clause splitter and the rule engine turn a SKU declaration into clause-marked text and a 16-dimensional symbolic vector

\mathbf{s}_{i}

. (II) A DistilBERT text stream yields token states

\mathbf{H}_{\mathrm{text}}

and

\mathbf{h}_{\mathrm{cls}}

; a query MLP turns

\mathbf{s}_{i}

into four queries

\mathbf{Q}_{\mathrm{ont}}

and a direct embedding

\mathbf{e}_{\mathrm{sym}}

. (III) The queries attend over the token states, and a sigmoid gate blends

\mathbf{h}_{\mathrm{cls}}

with the pooled attention output. (IV) A classifier produces three logits, trained with

\mathcal{L}_{\mathrm{AHR}}

; at inference the maximum softmax confidence, compared with the threshold

\tau

, routes the item to the green, amber or red lane.

Model	Acc. (%)	Macro-F1 (%)	ROC-AUC (%)	FHLR (%) $\downarrow$	Haram $\to$ Halal (%) $\downarrow$
Logistic Regression (TF-IDF)	86.62 $\pm$ 1.45	87.81 $\pm$ 1.38	97.13 $\pm$ 0.63	13.42 $\pm$ 2.72	3.71 $\pm$ 2.51
Random Forest (TF-IDF)	86.25 $\pm$ 1.39	86.27 $\pm$ 1.45	96.22 $\pm$ 0.56	21.73 $\pm$ 1.92	6.11 $\pm$ 3.22
XGBoost (TF-IDF)	86.71 $\pm$ 1.45	87.64 $\pm$ 1.41	96.83 $\pm$ 0.45	15.24 $\pm$ 1.54	2.18 $\pm$ 1.54
Logistic Regression (TF-IDF + $\mathbf{s}$ )	94.54 $\pm$ 0.85	94.29 $\pm$ 0.89	99.42 $\pm$ 0.12	3.46 $\pm$ 0.31	2.62 $\pm$ 1.25
XGBoost (TF-IDF + $\mathbf{s}$ )	96.25 $\pm$ 1.24	96.24 $\pm$ 1.33	99.66 $\pm$ 0.15	3.81 $\pm$ 0.83	2.83 $\pm$ 2.11
Vanilla DistilBERT	93.71 $\pm$ 0.35	93.19 $\pm$ 0.21	99.21 $\pm$ 0.23	6.58 $\pm$ 1.85	5.24 $\pm$ 2.08

TABLE IV: Five-seed test results (mean

\pm

SD). FHLR: share of true Non-Halal or Mushbooh items predicted Halal; Haram

\to

Halal: share of true Non-Halal items predicted Halal (lower is better for both).

\mathbf{s}

: the 16-dimensional symbolic vector. Best mean per column in bold. The two “

+\,\mathbf{s}

” rows are the values of the original run.

Fig. 3: Trade-offs (mean

\pm

SD over five seeds). (a) Accuracy against false-Halal leakage; the best models sit at the upper left. (b) Share of cargo auto-cleared to the green lane against the contamination of that lane at

\tau=0.85

; the best models sit at the lower right. The models with the symbolic vector

\mathbf{s}

and Halal-NeSyNet are close in both panels.

Class	Support	LR	RF	XGB	DistilBERT	NeSyNet
Halal	249	88.45	89.80	89.41	95.07	97.05
Non-Halal	92	95.45	93.18	97.78	92.40	95.45
Mushbooh	139	80.85	80.82	81.48	92.20	96.73

TABLE V: Per-class F1 (%) on the test split of one run (seed 42); support is the number of test items.

Fig. 4: Confusion matrices on the seed-42 test split ( $N=480$ ; H: Halal, NH: Non-Halal, M: Mushbooh; shading is the row-normalised share).

Comparison	Metric	Mean difference	95% CI	Better in
NeSyNet $-$ Vanilla DistilBERT	Accuracy (pp)	+3.34	[+2.93, +3.75]	5/5 seeds
NeSyNet $-$ Vanilla DistilBERT	FHLR (pp)	-1.90	[-4.58, +0.77]	4/5 seeds
NeSyNet $-$ XGBoost (TF-IDF)	Accuracy (pp)	+10.33	[+8.13, +12.54]	5/5 seeds
NeSyNet $-$ XGBoost (TF-IDF)	FHLR (pp)	-10.56	[-13.02, -8.11]	5/5 seeds
NeSyNet $-$ XGBoost (TF-IDF + $\mathbf{s}$ )	Accuracy (pp)	+0.59	[-1.11, +2.28]	3/5 seeds
NeSyNet $-$ XGBoost (TF-IDF + $\mathbf{s}$ )	FHLR (pp)	+1.21	[-0.28, +2.71]	1/5 seeds

TABLE VI: Seed-paired differences over the five seeds (first model minus second; mean,

t

-based 95 % interval, number of seeds in which the first model is better). Rows with

\mathbf{s}

use the re-run of the ontology-augmented baselines. † The second model is also trained without

\mathcal{L}_{\mathrm{AHR}}

, on the plain text with

L=112

Fig. 5: Test accuracy of every seed (dots) and the five-seed mean (black bars); grey lines join the same seed. The models with

\mathbf{s}

and Halal-NeSyNet overlap; XGB+s uses the re-run.

Variant	Acc. (%)	Macro-F1 (%)	FHLR (%) $\downarrow$	Haram $\to$ Halal (%) $\downarrow$
Halal-NeSyNet (full)	97.04 $\pm$ 0.34	96.90 $\pm$ 0.35	4.68 $\pm$ 1.16	5.24 $\pm$ 2.36
w/o $\mathcal{L}_{\mathrm{AHR}}$ (weighted CE)	97.54 $\pm$ 0.40	97.47 $\pm$ 0.44	4.59 $\pm$ 1.09	4.58 $\pm$ 2.08
w/o cross-attention †	96.25 $\pm$ 0.87	96.03 $\pm$ 0.99	6.41 $\pm$ 1.24	5.89 $\pm$ 2.12
w/o symbolic stream (= Vanilla DistilBERT) ‡	93.71 $\pm$ 0.35	93.19 $\pm$ 0.21	6.58 $\pm$ 1.85	5.24 $\pm$ 2.08

TABLE VII: Ablation of Halal-NeSyNet (mean

\pm

SD over five seeds). † Also trained without

\mathcal{L}_{\mathrm{AHR}}

, on the plain text with

L=112

. ‡ Plain text,

L=96

, learning rate

2.5\times 10^{-5}

, no symbolic stream. The rows differ in more than one factor, so this is not a clean single-factor study.

Additive	$N$	LR	RF	XGB	DistilBERT	NeSyNet
E120	20	95.0 / 5.0	85.0 / 15.0	100.0 / 0.0	85.0 / 10.0	95.0 / 0.0
E471	143	90.2 / 8.3	91.6 / 10.7	93.0 / 7.1	97.9 / 2.4	98.6 / 1.2
E422	38	79.0 / 18.8	89.5 / 25.0	84.2 / 12.5	89.5 / 18.8	92.1 / 6.2
E441	39	89.7 / 4.3	84.6 / 17.4	92.3 / 4.3	92.3 / 8.7	97.4 / 4.3

TABLE VIII: Accuracy / false-Halal leakage (%) on test items that mention a sensitive additive (seed 42).

N

is the subset size. LR: logistic regression, RF: random forest, XGB: XGBoost, all on TF-IDF features.

Model	CF acc. (%)	Triad cons. (%)	CF FHLR (%) $\downarrow$
LR (TF-IDF)	71.89 $\pm$ 2.38	26.00 $\pm$ 8.05	26.83 $\pm$ 4.10
RF (TF-IDF)	58.07 $\pm$ 5.28	6.33 $\pm$ 6.91	28.67 $\pm$ 8.55
XGB (TF-IDF)	62.15 $\pm$ 2.17	7.33 $\pm$ 4.94	21.17 $\pm$ 3.26
LR (TF-IDF + $\mathbf{s}$ )	99.89 $\pm$ 0.17	99.67 $\pm$ 0.50	0.00 $\pm$ 0.00
XGB (TF-IDF + $\mathbf{s}$ )	93.59 $\pm$ 3.16	82.11 $\pm$ 8.05	0.00 $\pm$ 0.00
Vanilla DistilBERT	99.22 $\pm$ 1.15	97.67 $\pm$ 3.45	1.17 $\pm$ 1.73

TABLE IX: Counterfactual suite (540 items, 180 triplets; mean

\pm

SD over five seeds). Triad consistency requires all three members of a triplet to be predicted correctly; CF FHLR is the false-Halal leakage on the suite. The two “

+\,\mathbf{s}

” rows are the values of the original run. LR: logistic regression, RF: random forest, XGB: XGBoost;

\mathbf{s}

: symbolic vector.

Fig. 6: Counterfactual accuracy and strict triad consistency (mean

\pm

SD over five seeds).

Model	Auto-cleared to green lane (%)	Green-lane contamination (%) $\downarrow$
LR (TF-IDF)	22.25 $\pm$ 1.85	0.00 $\pm$ 0.00
RF (TF-IDF)	18.63 $\pm$ 1.26	0.00 $\pm$ 0.00
XGB (TF-IDF)	35.96 $\pm$ 2.59	4.11 $\pm$ 1.68
LR (TF-IDF + $\mathbf{s}$ )	42.96 $\pm$ 0.86	0.96 $\pm$ 0.75
XGB (TF-IDF + $\mathbf{s}$ )	48.54 $\pm$ 0.91	1.46 $\pm$ 0.48
Vanilla DistilBERT	47.13 $\pm$ 2.63	1.26 $\pm$ 1.43

TABLE X: Three-lane triage at

\tau=0.85

(mean

\pm

SD over five seeds). Rows with

\mathbf{s}

are from the re-run of the ontology-augmented baselines. LR: logistic regression, RF: random forest, XGB: XGBoost;

\mathbf{s}

: symbolic vector.

Model	Acc. (%)	Macro-F1 (%)	FHLR (%) $\downarrow$	Haram $\to$ Halal (%) $\downarrow$	CF triad (%)
Logistic Regression (TF-IDF)	78.35 $\pm$ 11.92	79.94 $\pm$ 11.30	9.35 $\pm$ 9.29	10.86 $\pm$ 15.38	2.96 $\pm$ 5.38
Random Forest (TF-IDF)	61.91 $\pm$ 14.77	60.23 $\pm$ 14.82	21.32 $\pm$ 16.84	19.02 $\pm$ 17.34	0.00 $\pm$ 0.00
XGBoost (TF-IDF)	67.10 $\pm$ 18.26	63.54 $\pm$ 18.18	23.14 $\pm$ 23.39	30.68 $\pm$ 32.67	0.00 $\pm$ 0.00
Logistic Regression (TF-IDF + $\mathbf{s}$ )	91.11 $\pm$ 2.88	90.75 $\pm$ 3.17	7.30 $\pm$ 5.88	6.80 $\pm$ 9.63	84.44 $\pm$ 28.80
XGBoost (TF-IDF + $\mathbf{s}$ )	86.70 $\pm$ 5.50	84.66 $\pm$ 8.15	9.87 $\pm$ 9.69	12.38 $\pm$ 18.64	52.96 $\pm$ 47.65
Vanilla DistilBERT	86.85 $\pm$ 14.89	85.07 $\pm$ 17.09	15.10 $\pm$ 23.18	7.76 $\pm$ 12.88	79.63 $\pm$ 38.17

TABLE XI: Leave-one-template-family-out results (mean

\pm

SD over the ten held-out families, seeds averaged within a family). CF triad: strict triad consistency on the counterfactual triplets of the held-out family (six families).

Fig. 7: Leave-one-template-family-out. (a) Test accuracy and (b) false-Halal leakage on the held-out family for each model; each dot is one family (mean of the seeds), grey lines join the same family and the black bar is the mean over the ten families.

Held-out family	LR	RF	XGB	LR+s	XGB+s	DistilBERT	NeSyNet	NeSyNet w/o AHR
E471	72.0	61.7	76.4	91.2	91.7	94.7	97.4	98.2
E472e	82.3	77.9	65.2	89.2	86.8	77.2	87.0	92.2
E441	57.8	68.8	70.0	87.1	86.6	59.5	64.6	67.4
E422	79.2	46.0	39.9	88.3	84.4	95.2	98.0	97.7
rennet	69.9	47.7	63.8	87.4	77.8	94.2	97.7	97.9
E920	74.9	57.4	69.0	91.9	79.9	95.9	96.7	97.1

TABLE XII: Test accuracy (%) on each held-out template family (mean of 3 seeds); best per family in bold.

Comparison	Metric	Mean difference	95% CI	Better in
NeSyNet $-$ Vanilla DistilBERT	Accuracy (pp)	+4.02	[+0.94, +7.10]	9/10 families
NeSyNet $-$ Vanilla DistilBERT	FHLR (pp)	-6.51	[-12.39, -0.63]	9/10 families
NeSyNet $-$ XGBoost (TF-IDF + $\mathbf{s}$ )	Accuracy (pp)	+4.18	[-5.27, +13.63]	8/10 families
NeSyNet $-$ XGBoost (TF-IDF + $\mathbf{s}$ )	FHLR (pp)	-1.28	[-12.51, +9.95]	8/10 families
NeSyNet $-$ LR (TF-IDF + $\mathbf{s}$ )	Accuracy (pp)	-0.23	[-8.34, +7.87]	7/10 families
NeSyNet $-$ LR (TF-IDF + $\mathbf{s}$ )	FHLR (pp)	+1.30	[-10.94, +13.54]	8/10 families

TABLE XIII: Family-paired differences over the ten held-out families (first model minus second; mean,

t

-based 95 % interval, number of families in which the first model is better).

Neuro-Symbolic Reasoning

Neuro-symbolic reasoning combines learned neural models with explicit logic, rules, knowledge structures, or executable tools to solve structured tasks.

See all 284 papers

HalalBench: A Multilingual OCR Benchmark for Food Packaging Ingredient Extraction
Feb 19, 2026Hasan AriefMultilingual OCR Optical Character Recognition
HalalLens Research
Detecting Hallucinations and Recovering Verified Answers in Arabic Islamic Question Answering
Aug 4, 2026Khaled ZianiLLM Answer Verification Question Answering
Independent Researcher, Paris, France
Halluscoring 2026: The first shared task on llms hallucination detection and answer verification
Sep 29, 2026Aisha Alansari, Abdessalam Bouchekif, Ahmed Hasanaath +9LLM Evaluation LLM Answer Verification
King Fahd University of Petroleum and Minerals, Saudi Arabia · Hamad Bin Khalifa University, Qatar · University of the Basque Country, Spain +5

Feb 19, 2026J/K move · Enter open · S save

HalalBench: A Multilingual OCR Benchmark for Food Packaging Ingredient Extraction

Hasan Arief

HalalLens Research

Multilingual OCR Optical Character Recognition

Open paper AskPDF arXiv

No standardized benchmark exists for evaluating OCR on food packaging, despite its critical role in automated halal food verification. Existing benchmarks target documents or scene text, missing the unique challenges of ingredient labels: curved surfaces, dense multilingual text, and sub-8pt fonts. We present HalalBench, the first open multilingual benchmark for food packaging OCR, comprising 1,043 images (50 real, 993 synthetic) with 36,438 annotations in COCO format spanning 14 languages. We evaluate four engines: docTR achieves F1=0.193, ML Kit 0.180, EasyOCR 0.167, while all fail on Japanese (F1=0.000). A clustering ablation shows 36% F1 improvement from our post-processing algorithm. We validate findings through HalalLens (https://halallens.no), a production halal scanner serving 20+ countries. Dataset and code are released under open licenses.

Abstract

Neuro-Symbolic Reasoning

Explore similar work

HalalBench: A Multilingual OCR Benchmark for Food Packaging Ingredient Extraction

Detecting Hallucinations and Recovering Verified Answers in Arabic Islamic Question Answering

Halluscoring 2026: The first shared task on llms hallucination detection and answer verification