TripleGrader: A Web-Based Platform Supporting Collaborative Triple Annotation with Quality Grading and Inter-Rater Agreement Assessment
Organizations: Department of Computer Science Charleston Southern University Charleston, SC, USA
Abstract
Knowledge graphs need validated relational triples, and gold benchmarks of such triples are often built by several annotators. Most text annotation platforms focus on span tagging and compute agreement on categorical labels. They are not designed for ordinal quality grading of triples that have already been extracted. We present TripleGrader, an open-source web platform for grading subject-predicate-object triples on a 0 to 10 ordinal scale. In TripleGrader, each triple is scored independently, and peer scores stay hidden until the grader has rated it. Graders then discuss entries in threaded comments and propose revisions, which the session owner approves and publishes as new versions. For agreement, the platform computes quadratic-weighted Cohen's kappa for two raters and a quadratic-weighted Fleiss-type kappa for three or more. Graded sessions and the final benchmark are exported in common formats. We evaluated the platform in three ways. We ran 72 automated tests, including 18 workflow scenarios through the REST API. We also injected 10 error types to check whether prescribed corrections could be applied while preserving unaffected triples. On two small corpora with existing gold benchmarks, we then used the platform to curate the triples extracted by four large language models into each benchmark. Only 10% to 32% of the candidate triples could be kept unchanged, and 53% to 71% needed corrections, partly to match the benchmarks' naming conventions. Per model, 11 to 29 missing triples had to be added. On these corpora, the extracted triples therefore needed substantial human review. The source code is available at https://github.com/razz0n/triple-grader-tool.
Figures & tables
| Feature | TripleGrader | brat [ 7 ] | Label Studio [ 8 ] | INCEpTION [ 9 ] | Potato [ 19 ] |
| Primary task | Grading SPO triples | Span/relation tagging | Multi-modal labeling | Span/relation annotation | Configurable item annotation |
| Input unit | Pre-extracted triples | Raw text | Text, image, audio, video | Text with optional pre-annotations | Text, image, audio, video items |
| Ordinal score per item | 0–10 grade | No | Rating control | Numeric features | Likert, slider |
| Built-in agreement | Cohen/Fleiss-type | No | Commercial edition | Cohen, Fleiss, Krippendorff, Gwet | Cohen, Fleiss, Krippendorff |
| Weighted agreement for ordinal scores | Quadratic | No | No | No | Linear, quadratic |
| Grades dropped when a triple is revised | On version publish | n/a | No | No | No |
| Component | Scenarios | Tests | Checked behavior |
|---|---|---|---|
| Grades and sentence ratings | T02–T05 | 4 | Each grade and sentence rating is stored under its grader’s ID; clearing one grade keeps the others |
| Suggestions, votes, approvals | T06–T10 | 5 | Suggestion stored with its author; self-vote and non-author deletion rejected (HTTP 403); owner approval and rollback |
| Removal and additions | T11–T12 | 2 | Triples marked for removal leave the gold export; added triples enter it only after approval |
| Comments and versions | T13–T15 | 3 | Comment and reply persist; draft edits leave the published version unchanged; only the owner can delete a version, and a session always keeps at least one |
| Creation, export, robustness | T01, T16–T18 | 4 | JSON session creation; gold JSON and CSV list the same triples; two graders’ saves both kept; unknown ID rejected |
| Total | 18 | 14 permitted, 4 prohibited |
| Corpus | Model | Candidates | Exact | Modified | Deleted | Newly suggested |
|---|---|---|---|---|---|---|
| Software ethics | ChatGPT-5.6-High | 59 | 11 (18.6%) | 37 (62.7%) | 11 (18.6%) | 18 (27.3%) |
| ( ) | Gemini-3.1-Pro | 49 | 10 (20.4%) | 35 (71.4%) | 4 (8.2%) | 21 (31.8%) |
| GPT-OSS 120B | 47 | 12 (25.5%) | 25 (53.2%) | 10 (21.3%) | 29 (43.9%) | |
| Qwen 3.8 27B | 60 | 13 (21.7%) | 36 (60.0%) | 11 (18.3%) | 17 (25.8%) | |
| Biomedical BEL | ChatGPT-5.6-High | 60 | 19 (31.7%) | 34 (56.7%) | 7 (11.7%) | 16 (23.2%) |
| ( ) | Gemini-3.1-Pro | 67 | 21 (31.3%) | 37 (55.2%) | 9 (13.4%) | 11 (15.9%) |
Human-in-the-Loop Annotation
Human-in-the-loop annotation combines human judgment and AI assistance to create, review, and improve labeled data.
See all 149 papers