Rethinking Opinion Measurement Interfaces for Human-in-the-Loop Optimization

Changkun Ou; Sven Mayer; Daniel Buschek; Andreas Butz

Figure 1. Different opinion measurement interfaces: UI1) Single item Rating Scale (1-RS); UI2) Two Alternative Non-Forced Choice (2-ANFC); UI3)

n

-items Rating Scales (

n

-RS); UI4)

n

-Alternative Forced Ranking (

n

-AFR); UI5)

n

-Alternative Non-Forced Ranking (

n

-ANFR); UI6)

n

-Alternative Non-Forced Ranking with Distance (

n

-ANFRD). The 1-RS and

n

-RS interfaces are rating scale-based measurements, and the 2-ANFC,

n

-AFR, and

n

-ANFR are preference-based ranking interfaces. As a combination, the

n

-ANFRD mixes absolute ratings and preferential rankings, which permits expressing not only ranking orders but also local ordinal ranking distance. All interfaces in this gallery also permit users to express their incomplete preference through “I don’t know,” and to signal when a satisfactory result is achieved through “I’m satisfied.”

Optimizer	Input Form	Feedback Type	Options	Modeled Indifference/Incompleteness
Naive BO ( Močkus, 1975 )	Rating	Utility	Pointwise	No/No
2GC ( Brochu et al., 2007a )	Selection	Preferential	Pairwise	No/No
PBO ( González et al., 2017 )	Selection	Preferential	Pairwise	No/No
EUBO ( Lin et al., 2022 )	Selection	Preferential	Pairwise	No/No
4GC ( Brochu et al., 2010 )	Rating	Preferential	Listwise	No/No
SLS ( Koyama et al., 2017 )	Selection	Preferential	Listwise	No/No

Table 1. Examples of different Bayesian Optimization (BO) frameworks. Types of feedback can be either utility-based or preference-based. Most of the BO frameworks do not model indifference, and none of them consider incomplete preferences. Note that 1) rating can be considered either as absolute utility or preferential ranking distance depending on the underlying preference model, 2) selection is a special case of ranking where one item is strictly preferred over other options, and the ranking order of those unselected options is not specified, which means that their preference relation may be considered incomplete.

Figure 2. Real-world measured opinions in different interfaces: a) Commonly used rating scales to measure opinion regarding one item; b) Two alternative candidates (non-)forced choice; c) Music album song orders without specified criteria, and forced ranking of conferences and movies using a distance.

Figure 3. Examples of opinion measurement interfaces being used in HITL optimizations: a) A user chooses from a list of candidates to optimize illumination ( Marks et al., 1997 ) ; b) A user is asked to choose items that are closer to a given reference ( Wilber et al., 2014 ) ; c) A user is required to decide which of the two given options is closer to their desired material ( Brochu et al., 2007a ) ; d) A user is asked to choose which of two presented texts is more likely to be written on Wikipedia by a human ( Welleck et al., 2020 ) ; e) A user is required to choose which trained machine’s back-flip is better ( Christiano et al., 2017 ) , note that the interface uses a “can’t tell”; f) A user rates four presented graphics animations using a 3-step rating scale to show how close an animation is to what they are looking for ( Brochu et al., 2010 ) ; g) A user rates which of the given paired designs is more preferred ( Koyama et al., 2014 ) ; h) A user must select one target out of a list of choice alternatives ( Koyama et al., 2020 ) . i) A user ranks the given design variations, and each rank position has an associated ordinal rating label ( Ou et al., 2023 ) , and ties are allowed. These interfaces fit in a design space and are discussed in Section 3.1 .

	Alternative Options ( $n\in\mathbb{N}$ )
Feedback Type	Pointwise ( $n=1$ )	Pairwise ( $n=2$ )	Listwise ( $n>2$ )
Utility	1-RS ( UI1 )	2-RS	$n$ -RS ( UI3 )
Preferential (strict)	N/A	2-AFC	$n$ -AFR ( UI4 )
Preferential (weak)	N/A	2-ANFC ( UI2 )	$n$ -ANFR ( UI5 )
Hybrid (strict)	N/A	N/A	$n$ -AFRD
Hybrid (weak)	N/A	N/A	$n$ -ANFRD ( UI6 )

Table 2. A design space of opinion measurement interfaces concerning feedback type, ordering type, and batch size. The numbers in bold are those of the interfaces implemented in our study. All of them allow users to express incomplete preferences using an “I don’t know” checkbox.

Figure 4. The twelve UIs we implemented in our studies for text summarization and image color enhancement, with (a) a news article selected from the CNN daily email dataset, and (b) a photo taken from Sequential Gallery ( Koyama et al., 2020 ) .

Figure 5. The study procedure. Each participant completed three news articles or three photos, and each task item underwent at least 4 initial iterations for exploring the participant’s preference prior.

Figure 6. Comparisons between opinion measurement interfaces, from left to right: 1) Decision time : Listwise approaches (UI3, UI4, UI5, UI6) significantly increased decision time per iteration. However, decision time per item in listwise approaches does not differ from the pairwise (UI2); 2) Iterations : Participants iterate significantly more in the pointwise (UI1) than in a pairwise (UI2) or listwise (UI3) approach; 3) Ranking interactions : There is a significant difference between listwise interfaces, when indifference is not permitted (UI4 vs. UI6) in the text domain, but not in the image domain; 4) Indifference : Participants express more indifference in listwise approaches if they can (UI5 and UI6) than in the pairwise approach (UI2.) Note that UI1 and UI4 cannot express indifference; 5) Incomplete preferences : Participants express incomplete preferences significantly more when they cannot express indifference in listwise approaches (UI4 vs. UI5 and UI6.)

	Decision								Interaction				Preference
	Time								Behavior				Type
	per iteration				per item				number of interactions				indifference				incompleteness
Text Sum.	W	r	BF 10	p	W	r	BF 10	p	W	r	BF 10	p	W	r	BF 10	p	W	r	BF 10	p
H1 : UI1 vs. UI2	$2.140$	$-.040$	$.085$	$.168$	$2.930$	$.310$	$-$	$<.001$	$.049$	$.210$	$-$	$.007$	$-$	$-$	$-$	$-$	$.037$	$-.080$	$.314$	$.202$
H2a : UI1 vs. UI3	$1.650$	$-.290$	$-$	$<.001$	$1.360$	$-.360$	$-$	$<.001$	$.047$	$.180$	$-$	$.028$	$-$	$-$	$-$	$-$	$.036$	$-.100$	$.386$	$.128$
H2b : UI2 vs. UI5	$3.220$	$.390$	$-$	$<.001$	$2.150$	$.010$	$.072$	$.697$	$.043$	$.050$	$.188$	$.506$	$.030$	$-.270$	$-$	$.002$	$.043$	$.070$	$.227$	$.255$

Table 3. Statistics of participants’ behaviors and interactions. Results for H1, H2a, and H2b are tested using Wilcoxon (W

\times 10^{5}

), and LMMs are used for H3 and H4. Note that 1)

BF_{10}

are provided for non-significant results; 2) it is not possible to express indifference in UI1 and UI4; hence H1, H2a, and H3b statistics are omitted; 3) ranking behavior is only measured for listwise interfaces (UI4-UI6: H4c and H4d), and their results are shown in Section 6.1.4 .

Figure 7. Comparing objective quality between interfaces. The blue line is the average utility. The red line is a linear regression of the blue line, with each individual slope indicated in red.

				Opinion Measurement Interfaces (UI6 vs.)
	Progress			UI1			UI2			UI3			UI4			UI5
	$\beta$	SE	p	$\beta$	SE	p	$\beta$	SE	p	$\beta$	SE	p	$\beta$	SE	p	$\beta$	SE	p
Text Summerization
User Feedback	$.005$	$.001$	$<.001$	$-.078$	$.031$	$.012$	–	–	–	$-.042$	$.031$	$.169$	–	–	–	–	–	–
Learned Latent Utility	$.011$	$.001$	$<.001$	–	–	–	$.229$	$.028$	$<.001$	–	–	–	$-.007$	$.028$	$.780$	$-.024$	$.028$	$.390$
Length	$-.087$	$.020$	$<.001$	$.260$	$.427$	$.544$	$-.914$	$.427$	$.033$	$-.596$	$.427$	$.164$	$-.949$	$.427$	$.027$	$-.803$	$.427$	$.061$

Table 4. Statistics of the optimization loop. The main effect of iteration is evaluated for all measured metrics in text summarization and image color enhancement. Similarly, the main effect Interface is evaluated by the comparison between UI6 to all other interfaces. User input utilities are only compared between rating interfaces (UI1/UI3.) Learned latent utilities are only compared between preferential interfaces (UI2/UI4/UI5.) Results indicate UI6 outperforms all other interfaces in utility, and the progress over iterations showed that the optimization loop has a consistent trend. See also Figure 7 , Figure 8 , and Figure 9 for visualizations.

Figure 8. The user feedback (user input rating utility through the UIs directly) throughout the optimization loop. The blue line is the average user feedback , and the red line is a linear regression of the blue line. An increasing user feedback trends indicate the optimization goes more toward the users’ preferences. Thus, increasing their feedback over time. Here, UI6 outperforms UI1 and UI3 in both domains after completing the optimization.

Figure 9. The optimizer learned latent utility (inferred ranking utility through preferential Bayesian optimization) throughout the optimization loop. The blue line is the average utility, and the red line is a second-order polynomial regression of the blue line. An increasing utility (either direct or latent) trend indicates the optimized text is more toward users’ preferences. In latent utility comparisons of the two domains, UI6 outperforms UI2, UI4, and UI5.

Text Summarization
	Satisfaction (Q1-Q4)				Incompleteness Indication (Q5)				Feedback Clarity (Q6)
	W	r	BF 10	p	W	r	BF 10	p	W	r	BF 10	p
H1 : UI1 vs. UI2	$.028$	$-.300$	–	$<.001$	$.039$	$-.050$	$.221$	$.578$	$.039$	$-.050$	$.196$	$.588$
H2a : UI1 vs. UI3	$.155$	$-.040$	$.274$	$.511$	$.135$	$-.160$	–	$.008$	$.160$	$-.008$	$.189$	$.902$
H2b : UI2 vs. UI5	$.053$	$.310$	–	$<.001$	$.041$	$.003$	$.174$	$.970$	$.047$	$.150$	$.498$	$.076$
	$\beta$	SE	t	p	$\beta$	SE	t	p	$\beta$	SE	t	p

Table 5. Statistics of ending questionnaire answers. Results for H1, H2a, and H2b are tested using Wilcoxon (W

\times 10^{5}

), and LMMs are used for H3 and H4. Note that

BF_{10}

are provided for non-significant results.

Figure 10. Questionnaire results. Participants answered six questions per task and were collected using a bipolar slider-based Likert scale from text summarization and image color enhancement tasks. 100 means “Strongly agree“, and 0 means “Strongly disagree”. Participants also provided open-ended responses, which are quantified to sentiment ratings. The results indicate significant differences between the pointwise (UI1) and pairwise (UI2) approaches, and the hybrid approach (UI6) outperforms other listwise approaches (UI3, UI4, and UI5) but shares similar satisfaction and sentiment compared to the pairwise interface (UI2). When using UI6, participants considered expressing incomplete preference using “I don’t know” as the least useful.

Rethinking Opinion Measurement Interfaces for Human-in-the-Loop Optimization

Abstract

Figures & tables

Appendix

Human-in-the-Loop AI

Human-in-the-Loop Pareto Optimization: Trade-off Characterization for Assist-as-Needed Training and Performance Evaluation

Simultaneous Forward and Inverse Human-in-the-Loop Optimization

PerceptUI: LLM Agents as Human-Aligned Synthetic Users for UI/UX Evaluation

Abstract

Human-in-the-Loop AI

Explore similar work

Human-in-the-Loop Pareto Optimization: Trade-off Characterization for Assist-as-Needed Training and Performance Evaluation

Simultaneous Forward and Inverse Human-in-the-Loop Optimization

PerceptUI: LLM Agents as Human-Aligned Synthetic Users for UI/UX Evaluation