When Does Design Thinking Work? A Convergence Theorem for the Iterate-Test Core
Authors: Paweł Kuraś, Adrian Michalski, Alicja Gerka, Patryk Organiściak
Organizations: Institute of Informatics, Kozminski University, ul. Jagiellońska 57/59, 03-301 Warsaw, Poland · Faculty of Mathematics and Applied Sciences, Rzeszów University of Technology, al. Powstańców Warszawy 12, 35-959 Rzeszów, Poland · Department of Complex Systems, Rzeszów University of Technology, al. Powstańców Warszawy 12, 35-959 Rzeszów, Poland
Design thinking is widely taught as an iterative loop of ideation, prototyping, and testing, yet conditions under which such a loop must improve innovation outcomes remain unspecified. Taking problem framing as given, we formalize the iterate-test core as search over a finite prototype space whose utility is learned through noisy evaluations. We prove that the prototype a team believes to be best converges almost surely to a local optimum under five conditions: finite framing, local ideation, non-summable discovery rates, unbiased repeated testing, and retention of tested prototypes. Bounded test bias or criteria drift degrades the guarantee by a bounded amount, and multiple stakeholders yield convergence to a locally Pareto-optimal set. Finite-horizon simulations across three landscape classes (additive valuation, NK rugged landscapes, and clustered modular architectures) show that asymptotic limits are rarely reached in practice. Teams face an optimizer's curse, overestimating their chosen concept by 4-8% and frequently discarding superior prototypes. Testing every prototype once proves to be the worst allocation; optimal performance requires building one to three prototypes per five tests and repeatedly re-testing leaders. Pre-screening concepts yields greater performance gains than tuning testing policies.
Figures & tables
Figure 1: The two levels of the model. Empathising and defining produce a frame, a set of features F and a utility U . Within the frame the team repeats a round of ideation, prototyping and testing, and the theorem concerns this loop. A reframe (dashed) returns the team to the outer level and starts a new framing epoch.
F , 2F
features admitted by the frame; prototype space
U(p) , U∗
utility of prototype p , in [0,1] ; its maximum
N(p)
variants of p that ideation can reach in one step
Pk , P∞
prototypes built and tested by round k ; their eventual set
Fk
history up to the end of round k
Mkp
belief about p after round k ( Mkp under biased tests)
pk∗ , Xk
the favourite, argmaxp∈PkMkp , and the belief about it
Table 1: Notation.
Figure 2: The baseline loop on the main landscape (250 runs; bands are 95% confidence intervals). The team’s belief about its favourite exceeds the optimum from round 18 on, while the favourite is worth less than the best prototype the team has tested. A prior calibrated to the landscape (dashed) removes most of the overestimate.
Figure 3: Short and long runs (main landscape: 120 runs; DVF and NK families: 10 landscapes each, 24 runs per landscape). (a) Utility shipped (solid) and best utility among tested prototypes (dashed). (b) Share of runs in which the favourite (solid) or the best tested prototype (dashed) is a local optimum, determined exactly by enumeration.
Figure 4: Shipped utility after 60 rounds of exactly five tests each, as a function of how many of the five tests go to new prototypes, for four levels of test noise (five ideas per round; error bars are 95% confidence intervals; circles mark the best allocation).
Assumption
Changed element
Main
DVF family
NK family
Baseline
—
94.9±0.3
95.7±0.5
93.7±0.9
Local ideation
variants within 1 feature
94.5±0.3
95.2±0.4
92.5±1.1
variants within 3 features
94.7±0.3
95.9±0.4
93.3±0.8
unbounded variants
94.2±0.3
94.9±0.5
92.4±0.8
Generativity
1 idea per round
94.5±0.3
94.4±0.5
89.6±1.2
2 ideas per round
94.8±0.3
95.2±0.4
92.0±0.9
Table 2: One change at a time: utility shipped after 80 rounds, as a percentage of U∗ , with 95% confidence half-widths (main landscape: 200 runs; families: 16 runs on each landscape). The baseline uses variants within two features, five ideas per round, full retention, one new prototype, three re-tests of leading prototypes and one rotating re-test per round, and σ=0.10 .
Figure 5: Change in shipped utility after 80 rounds when each round’s re-test of the favourite is inflated by b , at three levels of test noise (main landscape: 240 runs per point; families: 16 runs per landscape; error bars are 95% confidence intervals). The dashed line marks b=σ .
Main
DVF family
NK family
Policy
shipped
shipped
diff.
shipped
diff.
Random search
91.0±0.4
87.9±0.8
−7.4±0.8
80.2±0.9
−12.0±0.9
Hill-climbing
91.0±0.5
88.7±0.8
−6.6±0.8
83.8±1.4
−8.4±1.6
ε -greedy
94.0±0.3
94.3±0.6
−1.1±0.3
89.9±1.1
−2.3±0.6
UCB
94.5±0.3
94.7±0.5
−0.6±0.3
90.2±1.1
−2.1±0.5
Thompson sampling
93.9±0.3
94.4±0.7
−0.9±0.4
89.6±1.1
−2.7±0.6
Table 3: Utility shipped after 300 tests, as a percentage of U∗ , on landscapes not used for tuning (main landscape: 250 runs; families: 20 landscapes each, 30 runs per landscape; ± is the 95% confidence half-width). “Diff.” is the mean paired difference from the loop across landscapes.
Cost of a reframe (rounds)
Policy
4
10
20
35
Reframes
Never
0.694
0.694
0.694
0.694
0
Every 15 rounds
0.700
0.697
0.693
0.694
5.0
Every 25 rounds
0.703
0.698
0.698
0.698
4.0
Every 40 rounds
0.702
0.704
0.703
0.698
2.0
Once, after 75 rounds
0.701
0.702
0.699
0.699
1.0
Table 4: True utility of the prototype shipped after 150 rounds in a world of six frames, by reframing policy and by the cost of a reframe in rounds (240 paired runs; 95% half-widths are at most 0.004 ). The last column is the mean number of reframes per run at a cost of 10 rounds; the criterion rows differ in the prior used for Gref . Bold marks the best policy at each cost.
Assumption
In practice
When it fails
Simulated departure and its cost
Finite frame
The brief and the vocabulary of features stay fixed during an epoch
No fixed space in which an optimum is defined
Not simulated
Local ideation
New ideas are variants of what has been built
With unbounded jumps the limit is global but progress is slower
Unbounded variants: −0.7 to −1.3
Persistent generativity
The team keeps proposing variants, even late
The process can stop short of a local optimum
One idea per round in place of five: −0.4 to −4.0
Unbiased, repeated testing
Tests can contradict the team; favourites are tested again
A lucky prototype can stay on top; under bias B the result is only 2B -locally optimal
No prototype tested twice: −2.3 to −8.2 . Bias b=0.2 : −1.0 to −3.5
Retention
Rivals to the favourite are kept
The best prototype found can be lost
Only the favourite kept: −1.2 to −3.6
Table 5: The five assumptions of Theorem 1 : what each means in practice, what the theory says when it fails, and what a departure in its direction cost after 80 rounds (range over the three landscape classes, in points of U∗ ; from Table 2 and Figure 5 ).
User research and system testing reveal whether products and services meet user needs, yet recruiting specialized participants, designing controlled what-if comparisons, and repeating studies as designs evolve often bottleneck design iteration and data-driven decision-making. We present a framework and execution infrastructure that translates real-world industry inquiries into controlled experiments with simulated users. Our methodology formally defines the target cohort, experimental stimuli, and decision-relevant metrics, while the pipeline supports large-scale concurrent runs over cohort-stimulus-environment combinations (survey, web, chatbot, and OS app), auditable trial-level logging, cohort-level outcome aggregation, and cross-cohort and cross-variant statistical reports. Across twelve industry partner cases spanning five domains, we show how constructing hard-to-reach or what-if persona cohorts supports exploratory studies and distills simulated responses into testable design hypotheses. Cross-model comparisons on selected cases largely preserve primary conclusions, whereas perturbing or ablating persona attributes alters service outcomes, effect sizes, and inter-group differences, indicating that grounding target cohorts in detailed personas materially shapes simulation results. Together, these studies show that persona-enabled agents can test specified cohorts at design speed, diagnose systems under test, and deliver directional evidence that focuses problem definition, steers design and strategic decisions, and guides further study.
Experimental design has emerged as a powerful approach for improving the sample efficiency of A/B testing, yet existing designs rely critically on correctly specified models. We study robust sequential experimental design under model misspecification and develop a unified framework that covers both contextual bandit and dynamic settings. Theoretically, we prove that our design bounds the worst-case mean squared error of the estimated treatment effect. Empirically, we demonstrate the effectiveness of the proposed approach using synthetic and real-world datasets from a leading technology company.
Qianglin Wen, Xiangkun Wu, Chengchun Shi +4
Yunnan Key Laboratory of Statistical Modeling and Data Analysis, Yunnan University, Kunming, China · School of Mathematical Sciences, Zhejiang University, Hangzhou, China · Department of Statistics, London School of Economics and Political Science, London, United Kingdom +3
The scenario approach is an established data-driven design framework that comes equipped with a powerful theory linking design complexity to generalization properties. In this approach, data are simultaneously used both for design and for certifying the design's reliability, without resorting to a separate test dataset. This paper takes a step further by guaranteeing additional properties, useful in post-design usage but not considered during the design phase. To this end, we introduce a two-level framework of appropriateness: baseline appropriateness, which guides the design process, and post-design appropriateness, which serves as a criterion for a posteriori evaluation. We provide distribution-free upper bounds on the risk of failing to meet the post-design appropriateness; these bounds are computable without using any additional test data. Under additional assumptions, lower bounds are also derived. As part of an effort to demonstrate the usefulness of the proposed methodology, the paper presents two practical examples in H2 and pole-placement problems. Moreover, a method is provided to infer comprehensive distributional knowledge of relevant performance indexes from the available dataset.
Algo Carè, Marco C. Campi, Simone Garatti
Department of Information Engineering - University of Brescia, via Branze 38, 25123 Brescia, Italia · Dipartimento di Elettronica, Informazione e Bioingegneria - Politecnico di Milano, piazza Leonardo da Vinci 32, 20133 Milano, Italia