SocialX: A Modular Platform for Multi-Source Big Data Research in Indonesia
Organizations: SocialX · Center of Excellence SAKTI, Research Institute Intelligent Business & Sustainable Economy, Telkom University · Biomedical Engineering Study Program, School of Electrical Engineering, Telkom University · Research Center for Smart Mechatronics, National Research and Innovation Agency (BRIN)
Abstract
Big data research in Indonesia is constrained by a fundamental fragmentation: relevant data is scattered across social media, news portals, e-commerce platforms, review sites, and academic databases, each with different formats, access methods, and noise characteristics. Researchers must independently build collection pipelines, clean heterogeneous data, and assemble separate analysis tools, a process that often overshadows the research itself. We present SocialX, a modular platform for multi-source big data research that integrates heterogeneous data collection, language-aware preprocessing, and pluggable analysis into a unified, source-agnostic pipeline. The platform separates concerns into three independent layers (collection, preprocessing, and analysis) connected by a lightweight job-coordination mechanism. This modularity allows each layer to grow independently: new data sources, preprocessing methods, or analysis tools can be added without modifying the existing pipeline. We describe the design principles that enable this extensibility, detail the preprocessing methodology that addresses challenges specific to Indonesian text across registers, and demonstrate the platform's utility through a walkthrough of a typical research workflow. SocialX is publicly accessible as a web-based platform at https://www.socialx.id.
Figures & tables
| Approach | Multi-Source | Preprocessing | Analysis | Indonesian | No-Code |
|---|---|---|---|---|---|
| Collection | NLP | ||||
| Brandwatch / Meltwater | Social only | Keyword | Sentiment | Limited | Yes |
| GATE | No | Yes | Yes | No | No |
| Orange | No | Limited | Yes | No | Yes |
| snscrape / Twint | Single source | No | No | No | No |
| IndoBERT (standalone) | No | No | Single task | Yes | No |
| Stage | Records | Removed | Reduction |
|---|---|---|---|
| Raw collected data | 12,847 | ||
| After deduplication | 10,203 | 2,644 | 20.6% |
| After language detection | 9,451 | 752 | 7.4% |
| After keyword filtering | 7,832 | 1,619 | 17.1% |
| After relevancy classification | 5,614 | 2,218 | 28.3% |
| Total | 5,614 | 7,233 | 56.3% |
| Step | Action | Description |
|---|---|---|
| 1 | Select sources | Choose data sources from catalog. Configure keywords, date range, and source-specific parameters. |
| 2 | Collect data | Platform runs scrapers automatically. Output is normalized to the common schema and stored. |
| 3 | Preprocess | Select collected datasets as input. Enable and configure filters: deduplication, language detection, keyword, relevancy. |
| 4 | Analyze | Choose an analyzer (sentiment, network, trend, etc.). Results are rendered as interactive visualizations in the browser. |
| Step | Manual Approach | SocialX |
|---|---|---|
| Collection | Install and configure separate scraping tools per platform. Handle authentication, rate limits, and errors in code. | Select sources and configure keywords through web interface. |
| Format reconciliation | Write scripts to parse and unify different output formats into a single structure. | Handled automatically by collection connectors. |
| Preprocessing | Write cleaning scripts. Install and run NLP models (language detection, relevancy) locally. | Configure filters through web interface. ML models run on backend. |
| Analysis | Load pre-trained models, write inference code, build visualizations. | Select analyzer, submit job, view results in browser. |
| Requirements | Python, NLP libraries, GPU (optional), programming skills. | Web browser. |
Computational Social Science
Computational social science uses data and computational models to study social behavior, institutions, communication, and collective outcomes.
See all 165 papersCommonMorph: Participatory Morphological Documentation Platform
CommonMorph, a comprehensive platform that streamlines morphological data collection development through a three-tiered approach: expert linguistic definition, contributor elicitation, and community validation. The platform minimises manual work by incorporating active learning, annotation suggestions, and tools to import and adapt materials from related languages. It accommodates diverse morphological systems, including fusional, agglutinative, and root-and-pattern morphologies. Its open-source design and UniMorph-compatible outputs ensure accessibility and interoperability with NLP tools. Our platform is accessible at https://common-morph.com, offering a replicable model for preserving linguistic diversity through collaborative technology.