Statistical Foundations for a Google Play User-Review Sentiment Index: Signal Fusion, Shrinkage, Distributional Validation, and Dynamic Smoothing

Marco Mandap

stat.MESep 25, 2026

Statistical Foundations for a Google Play User-Review Sentiment Index: Signal Fusion, Shrinkage, Distributional Validation, and Dynamic Smoothing

Authors: Marco Mandap

Organizations: Bulacan State University · Bulacan State University, McArthur Highway, Capitol Compound, Guinhawa, Malolos City, 3000, Bulacan, Philippines.

Abstract

The decision problem is whether valence among the users who write reviews of an app has changed enough to warrant investigation of a release, outage, or support issue. We propose a statistical specification for that decision, applied to written Google Play reviews collected under a declared locale, language, sort order, retrieval cap, and window rule. That endpoint returns a selected collection rather than a random sample, so the index describes review writers under the declared protocol and not all app users. What is new is not a sentiment classifier but an auditable interface: an ordinal threshold model is the preferred representation of star ratings, a calibrated two-signal generalized least-squares estimator is retained as an interpretable linear approximation when held-out labels support its conditional-unbiasedness assumption, the estimand is declared before the estimator, and every downstream module--uncertainty, diagnostics, and regularization--is tied to that declaration. The framework separates the latent average of a fixed collected set from the mean of a protocol-defined population of review writers, so measurement and between-review variation are not conflated, and treats helpfulness and recency weights as a different policy target. Shrinkage, cross-app empirical Bayes, histogram and dependence diagnostics, and dynamic smoothing are specified as separate modules with defaults and mandatory sensitivity analyses. A reference implementation, a declared 6000-review three-app collection, and a synthetic stress test exercise the displayed equations: fusion reduced error in clean and small-window scenarios, coordinated contamination caused severe bias and interval failure, and the collected sample reproduces the abstention rules. No real-review validation is claimed.

Figures & tables

Field	Statistical role	Use and limitation
content	Text input	A calibrated sentiment model produces a score; uncertainty must be estimated by that model or by held-out residuals.
score	Ordinal rating signal	Maps a one-to-five-star response to a working scale; equal spacing is a modeling assumption, and the discreteness that equal spacing leaves untreated is why the ordinal threshold model is preferred.
thumbsUpCount	Social endorsement	May define an optional prominence weight, but lacks impressions and negative-vote denominators, so a helpfulness ratio is not a Wilson proportion Wilson (1927) .
at	Review time	Defines time windows and optional recency weights.
reviewCreatedVersion , appVersion	Release context	Supports version-level grouping and associational change-point analyses.
replyContent , repliedAt	Developer response	Supplies optional responsiveness covariates after timestamp and missing-value validation.

Table 1: Google Play fields used in the sentiment-index framework.

Figure 1: Implemented framework stages. Protocol and calibration choices precede fusion; target-specific aggregation and diagnostics precede any regularization.

Stage	Operation
Input	Supply review records $(r_{i},s_{i},u_{i},t_{i})$ , disjoint windows $\mathcal{W}_{t}(\pi)$ , validation-fitted ordinal/text calibration and $\bm{\Sigma}_{i}$ , repeated rating-histogram snapshots, and the declared policy parameters $(\alpha,q_{\max},\lambda)$ .
1	Record $\pi$ , scraper version and commit, retrieved and app-level totals; deduplicate reviewId , detect edits across snapshots, enforce timestamp rules, and use fixed-duration calendar windows with version overlays.
2	Fit or apply the ordinal–continuous measurement model. For the documented linear approximation, compute calibrated $\bm{y}_{i}$ , the constrained weights in equation ( 52 ), and $(\hat{\theta}_{i},\nu_{i},\delta_{i})$ . Report non-identity calibration and held-out coverage; otherwise label the fusion illustrative.
3	Compute the primary unit-weighted $I_{t}$ and separately compute the policy index from equation ( 54 ). Report the half-life and $n_{\mathrm{eff},t}$ for each.
4	Report $M_{t}^{2}$ for the finite records and $S_{t}^{2}$ for the writer population. Pool $\kappa_{t}^{2}$ across small windows. Form duplicate-text clusters and replace ordinary uncertainty by equation ( 61 ) when $\mathrm{DEFF}>2$ or the daily-count Fano factor exceeds 2.
5	Compute $f_{t}$ , the sensitivity band in equation ( 62 ), and the guardrail in equation ( 63 ). A flagged boundary is not interpreted as a sentiment trend without composition analysis.

Table 2: Pseudocode for estimating and diagnosing the review-sentiment index.

Review	Stars	$z_{\rho i}$	$z_{si}$	$\hat{\theta}_{i}$	$\delta_{i}$	$q_{i}$
Synthetic 1	5	0.930	0.300	0.694	0.630	6.481
Synthetic 2	2	$-0.495$	$-0.420$	$-0.467$	0.075	3.565
Synthetic 3	4	0.455	0.840	0.599	0.385	5.762

Table 3: Synthetic three-review calculation using calibrated signals.

Scenario	Method	Bias	RMSE	Coverage	Mean width
Clean	Star only	$-0.004$	0.040	0.998	0.265
Clean	Text only	$-0.036$	0.049	0.985	0.221
Clean	Fusion	$-0.016$	0.032	1.000	0.232
Clean	Count Bayes	$-0.059$	0.066	0.896	0.187
Clean	Declared prior	$-0.023$	0.035	1.000	0.228
Small $E(n)=8$	Fusion	$-0.014$	0.063	0.998	0.511

Table 4: Selected deterministic simulation results. Coverage is the nominal 95% interval coverage for the realized protocol-defined writer mean.

Figure 2: Left: simulated shrinkage-to-raw MSE ratio against the theoretical risk-radius scale. Right: one synthetic filtered path with conditional intervals.

App	$n$	(interior)	$\bar{I}_{t}$	only	only	score	$\hat{\kappa}_{t}^{2}$	DEFF	Fano
		Windows		Star	Text	App		max
Google Maps	2000	7 (5)	0.310	0.366	0.216	3.217	0.407	1.33	11.2
Spotify	2000	4 (2)	0.399	0.450	0.314	4.347	0.347	1.15	–
WhatsApp	2000	2 (0)	0.476	0.589	0.287	4.617	0.247	1.61	–

Table 5: Collected-sample summary by app.

\bar{I}_{t}

is the mean raw window index, Star only and Text only are the mandatory baselines, App score is the contemporaneous app-detail score on the one-to-five scale,

\hat{\kappa}_{t}^{2}

is pooled across the app’s windows by the DerSimonian–Laird diagnostic, DEFF is the maximum near-duplicate design effect over windows, and Fano is the arrival count factor over interior windows only.

Figure 3: Collected-sample window index with 95% writer-population intervals (blue) and the filtered series with conditional intervals (red). Open squares mark the first and last day of each sample, which are partial by construction; the dashed line is neutral valence.

Jul 5, 2026

evalci: A Python Library for Statistically Rigorous Comparison of Language Model Evaluations

The dominant practice in language model evaluation is to report a single accuracy number per model and declare the higher one better, without testing whether the gap could plausibly be sampling noise. On benchmarks of a few thousand items, and under temperature sampling where a model can differ from itself run to run by more than the reported gap between models, this practice routinely overstates confidence in headline claims. The statistical machinery to fix this -- confidence intervals, paired significance tests, power analysis, clustered standard errors, multiple-comparison correction -- is well established, but no standard, pip-installable tool packages it in the shape an evaluation actually takes: a per-item results table. We present evalci, a pure-Python library (numpy/scipy/pandas only) that turns a per-item results table into a publication-ready claim -- e.g., "Model A beats Model B,

Δ=3.1

pts, 95% CI [1.2, 5.0], paired permutation

p=0.002

n=1{,}319

" -- in one function call, with adapters for lm-evaluation-harness and HELM output. Every routine is validated against an independent reference (statsmodels, or brute-force exact enumeration) rather than only against itself. As a case study, we re-analyze a public comparison of nine language models' MMLU accuracy and find that 3 of the 8 adjacent leaderboard-rank gaps are not statistically significant after correcting for the 36 pairwise comparisons the ranking implies. evalci is available at https://pypi.org/project/evalci/ (source: https://github.com/Shreyaskc/evalci, DOI: https://doi.org/10.5281/zenodo.21201815)

Shreyas K Chandrahas

LLM Evaluation Pairwise Comparison Statistical Power Analysis Statistical Hypothesis Testing

Ask PDF

Sep 17, 2026

Prediction-Powered Smoothing and Validation for Disaggregated AI Evaluation

Evaluating an AI system requires disaggregated assessment, as performance varies across domains such as benchmark task types or conversation types in deployed agents. Exhaustive testing is expensive, so evaluation rests on a sample of labeled units. We treat the evaluation set as a finite population and seek accurate point and interval estimates of each domain mean. Direct estimators, including prediction-powered inference (PPI), use only a domain's own labels and are imprecise where labels are few. Small area estimation addresses this problem, and we build on it to develop an integrated workflow for estimation and validation. For estimation, we propose prediction-powered smoothing (PP-S), a Bayesian model fit to each domain's prediction-powered estimate, with an extension that borrows strength across a reporting taxonomy (PP-TS). For validation, we derive a new, approximately unbiased design-based cross-validation score for choosing among direct and smoothed estimators. We study a curated benchmark with verifiable grading and deployed agent traffic graded by humans, each with every outcome observed. In both, the proposed estimators improve on the direct estimators in point and interval estimation, with near-nominal coverage. At the same sampling budget, our score selects as well as an independent validation sample does and estimates the selected estimator's error far more accurately.

Sho Kawano, Zehang Richard Li, Paul A. Parker

Department of Statistics, University of California, Santa Cruz, CA, USA

AI Agent Evaluation Prediction-Powered Inference

Ask PDF

Mar 17, 2026

A Multi-Model Approach to English-Bangla Sentiment Classification of Government Mobile Banking App Reviews

For millions of users in developing economies who depend on mobile banking as their primary gateway to financial services, app quality directly shapes financial access. The study analyzed 5,652 Google Play reviews in English and Bangla (filtered from 11,414 raw reviews) for four Bangladeshi government banking apps. The authors used a hybrid labeling approach that combined use of the reviewer's star rating for each review along with a separate independent XLM-RoBERTa classifier to produce moderate inter-method agreement (kappa = 0.459). Traditional models outperformed transformer-based ones: Random Forest produced the highest accuracy (0.815), while Linear SVM produced the highest weighted F1 score (0.804); both were higher than the performance of fine-tuned XLM-RoBERTa (0.793). McNemar's test confirmed that all classical models were significantly superior to the off-the-shelf XLM-RoBERTa (p < 0.05), while differences with the fine-tuned variant were not statistically significant. DeBERTa-v3 was applied to analyze the sentiment at the aspect level across the reviews for the four apps; the reviewers expressed their dissatisfaction primarily with the speed of transactions and with the poor design of interfaces; eJanata app received the worst ratings from the reviewers across all apps. Three policy recommendations are made based on these findings - remediation of app quality, trust-centred release management, and Bangla-first NLP adoption - to assist state-owned banks in moving towards improving their digital services through data-driven methods. Notably, a 16.1-percentage-point accuracy gap between Bangla and English text highlights the need for low-resource language model development.

Md. Naim Molla, Md Muhtasim Munif Fahim, Md. Binyamin +4

Data Science Research Lab, Department of Statistics, University of Rajshahi, Rajshahi-6205, Bangladesh · Department of Statistics, Mawlana Bhashani Science and Technology University, Tangail-1902, Bangladesh · Department of Statistics, University of Rajshahi, Rajshahi-6205, Bangladesh

Multilingual Sentiment Analysis Sentiment Analysis Financial Services

Ask PDF

Abstract

Explore similar work

evalci: A Python Library for Statistically Rigorous Comparison of Language Model Evaluations

Prediction-Powered Smoothing and Validation for Disaggregated AI Evaluation

A Multi-Model Approach to English-Bangla Sentiment Classification of Government Mobile Banking App Reviews