Organizations: SocialX · Center of Excellence SAKTI, Research Institute Intelligent Business & Sustainable Economy, Telkom University · Biomedical Engineering Study Program, School of Electrical Engineering, Telkom University · Research Center for Smart Mechatronics, National Research and Innovation Agency (BRIN)
Determining whether a piece of text is relevant to a given topic is a fundamental task in natural language processing, yet it remains largely unexplored for Bahasa Indonesia. Unlike sentiment analysis or named entity recognition, relevancy classification requires the model to reason about the relationship between two inputs simultaneously: a topical context and a candidate text. We introduce IndoBERT-Relevancy, a context-conditioned relevancy classifier built on IndoBERT Large (335M parameters) and trained on a novel dataset of 31,360 labeled pairs spanning 188 topics. Through an iterative, failure-driven data construction process, we demonstrate that no single data source is sufficient for robust relevancy classification, and that targeted synthetic data can effectively address specific model weaknesses. Our final model achieves an F1 score of 0.948 and an accuracy of 96.5%, handling both formal and informal Indonesian text. The model is publicly available at HuggingFace.
Figures & tables
Register
Example Text
Formal news
“Pertamina resmi menaikkan harga Pertalite menjadi Rp10.000 per liter”
Informal news
“BBM naik lagi, Pertamina umumkan harga baru”
Social media
“gila bensin naik lagi, dompet makin tipis nih”
Highly informal
“isi full tank skrg kayak bayar cicilan motor wkwk”
Table 1: The same topic expressed across the register spectrum. All four texts are relevant to “fuel price increases,” but they share almost no vocabulary.
Domain
Contexts
Social & Community Issues
22
Industry & Business
20
Economics & Finance
17
Health
13
Law & Crime
12
Environment & Disasters
12
Table 2: Distribution of 188 topical contexts across 12 thematic domains.
Stage
Pairs
Relevant
Not Rel.
Contexts
1. Formal text
18,798
4,584
14,214
188
2. Informal text
7,884
3,022
4,862
40
3. Implicit text
4,678
3,094
1,584
30
Combined
31,360
10,700
20,660
188
Table 3: Summary of the three-stage dataset construction.
Figure 1: The iterative, failure-driven data construction process. Each stage is motivated by a specific weakness identified in the previous model. Dashed red arrows indicate the identified failure that motivated the next stage.
Training Data
Accuracy
Precision
Recall
F1
V1
Formal only (18.8K)
93.3%
87.8%
84.5%
0.860
V2
+ Informal (26.7K)
95.6%
93.0%
91.4%
0.922
V3
+ Implicit (31.4K)
96.5%
94.8%
94.8%
0.948
Table 4: Model performance at each stage of data construction.
Figure 2: Model performance improves with each data source. The largest single improvement comes from adding informal text (+6.2 F1), while implicit text provides a targeted boost (+2.6 F1).
Predicted
Not Relevant
Relevant
Actual
Not Relevant
3,015
84
Relevant
83
1,522
Table 5: Confusion matrix on the validation set. The near-symmetric error distribution (84 false positives vs. 83 false negatives) confirms balanced precision and recall.
Figure 3: Validation metrics across training epochs for the final model (V3).
Context
Text
Pred.
Score
Explicit, formal:
Monetary policy
BI tahan suku bunga acuan 6%
Rel.
1.000
Oil prices
OPEC pangkas produksi, minyak melonjak
Rel.
1.000
Explicit, informal:
Fuel prices
gila bensin naik lagi, dompet makin tipis
Rel.
1.000
Tech layoffs
temen gw di tokped kena layoff semua divisi
Rel.
0.997
Table 6: Qualitative evaluation across text registers and relevance types. The model correctly handles explicit formal text, informal slang, and correctly rejects off-topic text.
SocialX · Center of Excellence SAKTI, Research Institute Intelligent Business & Sustainable Economy, Telkom University · Biomedical Engineering Study Program, School of Electrical Engineering, Telkom University +1