Statistical Foundations for a Google Play User-Review Sentiment Index: Signal Fusion, Shrinkage, Distributional Validation, and Dynamic Smoothing

Marco Mandap

stat.MESep 25, 2026

Statistical Foundations for a Google Play User-Review Sentiment Index: Signal Fusion, Shrinkage, Distributional Validation, and Dynamic Smoothing

Authors: Marco Mandap

Organizations: Bulacan State University · Bulacan State University, McArthur Highway, Capitol Compound, Guinhawa, Malolos City, 3000, Bulacan, Philippines.

Abstract

The decision problem is whether valence among the users who write reviews of an app has changed enough to warrant investigation of a release, outage, or support issue. We propose a statistical specification for that decision, applied to written Google Play reviews collected under a declared locale, language, sort order, retrieval cap, and window rule. That endpoint returns a selected collection rather than a random sample, so the index describes review writers under the declared protocol and not all app users. What is new is not a sentiment classifier but an auditable interface: an ordinal threshold model is the preferred representation of star ratings, a calibrated two-signal generalized least-squares estimator is retained as an interpretable linear approximation when held-out labels support its conditional-unbiasedness assumption, the estimand is declared before the estimator, and every downstream module--uncertainty, diagnostics, and regularization--is tied to that declaration. The framework separates the latent average of a fixed collected set from the mean of a protocol-defined population of review writers, so measurement and between-review variation are not conflated, and treats helpfulness and recency weights as a different policy target. Shrinkage, cross-app empirical Bayes, histogram and dependence diagnostics, and dynamic smoothing are specified as separate modules with defaults and mandatory sensitivity analyses. A reference implementation, a declared 6000-review three-app collection, and a synthetic stress test exercise the displayed equations: fusion reduced error in clean and small-window scenarios, coordinated contamination caused severe bias and interval failure, and the collected sample reproduces the abstention rules. No real-review validation is claimed.

Figures & tables

Field	Statistical role	Use and limitation
content	Text input	A calibrated sentiment model produces a score; uncertainty must be estimated by that model or by held-out residuals.
score	Ordinal rating signal	Maps a one-to-five-star response to a working scale; equal spacing is a modeling assumption, and the discreteness that equal spacing leaves untreated is why the ordinal threshold model is preferred.
thumbsUpCount	Social endorsement	May define an optional prominence weight, but lacks impressions and negative-vote denominators, so a helpfulness ratio is not a Wilson proportion Wilson (1927) .
at	Review time	Defines time windows and optional recency weights.
reviewCreatedVersion , appVersion	Release context	Supports version-level grouping and associational change-point analyses.
replyContent , repliedAt	Developer response	Supplies optional responsiveness covariates after timestamp and missing-value validation.

Table 1: Google Play fields used in the sentiment-index framework.

Figure 1: Implemented framework stages. Protocol and calibration choices precede fusion; target-specific aggregation and diagnostics precede any regularization.

Stage	Operation
Input	Supply review records $(r_{i},s_{i},u_{i},t_{i})$ , disjoint windows $\mathcal{W}_{t}(\pi)$ , validation-fitted ordinal/text calibration and $\bm{\Sigma}_{i}$ , repeated rating-histogram snapshots, and the declared policy parameters $(\alpha,q_{\max},\lambda)$ .
1	Record $\pi$ , scraper version and commit, retrieved and app-level totals; deduplicate reviewId , detect edits across snapshots, enforce timestamp rules, and use fixed-duration calendar windows with version overlays.
2	Fit or apply the ordinal–continuous measurement model. For the documented linear approximation, compute calibrated $\bm{y}_{i}$ , the constrained weights in equation ( 52 ), and $(\hat{\theta}_{i},\nu_{i},\delta_{i})$ . Report non-identity calibration and held-out coverage; otherwise label the fusion illustrative.
3	Compute the primary unit-weighted $I_{t}$ and separately compute the policy index from equation ( 54 ). Report the half-life and $n_{\mathrm{eff},t}$ for each.
4	Report $M_{t}^{2}$ for the finite records and $S_{t}^{2}$ for the writer population. Pool $\kappa_{t}^{2}$ across small windows. Form duplicate-text clusters and replace ordinary uncertainty by equation ( 61 ) when $\mathrm{DEFF}>2$ or the daily-count Fano factor exceeds 2.
5	Compute $f_{t}$ , the sensitivity band in equation ( 62 ), and the guardrail in equation ( 63 ). A flagged boundary is not interpreted as a sentiment trend without composition analysis.

Table 2: Pseudocode for estimating and diagnosing the review-sentiment index.

Review	Stars	$z_{\rho i}$	$z_{si}$	$\hat{\theta}_{i}$	$\delta_{i}$	$q_{i}$
Synthetic 1	5	0.930	0.300	0.694	0.630	6.481
Synthetic 2	2	$-0.495$	$-0.420$	$-0.467$	0.075	3.565
Synthetic 3	4	0.455	0.840	0.599	0.385	5.762

Table 3: Synthetic three-review calculation using calibrated signals.

Scenario	Method	Bias	RMSE	Coverage	Mean width
Clean	Star only	$-0.004$	0.040	0.998	0.265
Clean	Text only	$-0.036$	0.049	0.985	0.221
Clean	Fusion	$-0.016$	0.032	1.000	0.232
Clean	Count Bayes	$-0.059$	0.066	0.896	0.187
Clean	Declared prior	$-0.023$	0.035	1.000	0.228
Small $E(n)=8$	Fusion	$-0.014$	0.063	0.998	0.511

Table 4: Selected deterministic simulation results. Coverage is the nominal 95% interval coverage for the realized protocol-defined writer mean.

Figure 2: Left: simulated shrinkage-to-raw MSE ratio against the theoretical risk-radius scale. Right: one synthetic filtered path with conditional intervals.

App	$n$	(interior)	$\bar{I}_{t}$	only	only	score	$\hat{\kappa}_{t}^{2}$	DEFF	Fano
		Windows		Star	Text	App		max
Google Maps	2000	7 (5)	0.310	0.366	0.216	3.217	0.407	1.33	11.2
Spotify	2000	4 (2)	0.399	0.450	0.314	4.347	0.347	1.15	–
WhatsApp	2000	2 (0)	0.476	0.589	0.287	4.617	0.247	1.61	–

Table 5: Collected-sample summary by app.

\bar{I}_{t}

is the mean raw window index, Star only and Text only are the mandatory baselines, App score is the contemporaneous app-detail score on the one-to-five scale,

\hat{\kappa}_{t}^{2}

is pooled across the app’s windows by the DerSimonian–Laird diagnostic, DEFF is the maximum near-duplicate design effect over windows, and Fano is the arrival count factor over interior windows only.

Figure 3: Collected-sample window index with 95% writer-population intervals (blue) and the filtered series with conditional intervals (red). Open squares mark the first and last day of each sample, which are partial by construction; the dashed line is neutral valence.

evalci: A Python Library for Statistically Rigorous Comparison of Language Model Evaluations
Jul 5, 2026Shreyas K ChandrahasLLM Evaluation Pairwise Comparison
Prediction-Powered Smoothing and Validation for Disaggregated AI Evaluation
Sep 17, 2026Sho Kawano, Zehang Richard Li, Paul A. ParkerAI Agent Evaluation Prediction-Powered Inference
Department of Statistics, University of California, Santa Cruz, CA, USA
A Multi-Model Approach to English-Bangla Sentiment Classification of Government Mobile Banking App Reviews
Mar 17, 2026Md. Naim Molla, Md Muhtasim Munif Fahim, Md. Binyamin +4Multilingual Sentiment Analysis Sentiment Analysis
Data Science Research Lab, Department of Statistics, University of Rajshahi, Rajshahi-6205, Bangladesh · Department of Statistics, Mawlana Bhashani Science and Technology University, Tangail-1902, Bangladesh · Department of Statistics, University of Rajshahi, Rajshahi-6205, Bangladesh

Jul 5, 2026J/K move · Enter open · S save

evalci: A Python Library for Statistically Rigorous Comparison of Language Model Evaluations

Shreyas K Chandrahas

LLM Evaluation Pairwise Comparison Statistical Power Analysis Statistical Hypothesis Testing

Open paper AskPDF arXiv

The dominant practice in language model evaluation is to report a single accuracy number per model and declare the higher one better, without testing whether the gap could plausibly be sampling noise. On benchmarks of a few thousand items, and under temperature sampling where a model can differ from itself run to run by more than the reported gap between models, this practice routinely overstates confidence in headline claims. The statistical machinery to fix this -- confidence intervals, paired significance tests, power analysis, clustered standard errors, multiple-comparison correction -- is well established, but no standard, pip-installable tool packages it in the shape an evaluation actually takes: a per-item results table. We present evalci, a pure-Python library (numpy/scipy/pandas only) that turns a per-item results table into a publication-ready claim -- e.g., "Model A beats Model B,

Δ=3.1

pts, 95% CI [1.2, 5.0], paired permutation

p=0.002

n=1{,}319

" -- in one function call, with adapters for lm-evaluation-harness and HELM output. Every routine is validated against an independent reference (statsmodels, or brute-force exact enumeration) rather than only against itself. As a case study, we re-analyze a public comparison of nine language models' MMLU accuracy and find that 3 of the 8 adjacent leaderboard-rank gaps are not statistically significant after correcting for the 36 pairwise comparisons the ranking implies. evalci is available at https://pypi.org/project/evalci/ (source: https://github.com/Shreyaskc/evalci, DOI: https://doi.org/10.5281/zenodo.21201815)

Abstract

Explore similar work

evalci: A Python Library for Statistically Rigorous Comparison of Language Model Evaluations

Prediction-Powered Smoothing and Validation for Disaggregated AI Evaluation

A Multi-Model Approach to English-Bangla Sentiment Classification of Government Mobile Banking App Reviews