CIM-Embed: Learning Instruction-Conditioned Multimodal Embeddings for Clustering

Yinghua Xu; Pengyang Xie; Yanxiang Zhou; Chuanzhi Tong; Xiao Xiao; Chaoqun Feng; Xiaoqing Wang

cs.AIOct 15, 2026

CIM-Embed: Learning Instruction-Conditioned Multimodal Embeddings for Clustering

Authors: Yinghua Xu, Pengyang Xie, Yanxiang Zhou, Chuanzhi Tong, Xiao Xiao, Chaoqun Feng, Xiaoqing Wang

Organizations: College of Control Science and Engineering, Zhejiang University · Alibaba International Digital Commerce

Abstract

Multimodal objects can require different partitions under different instructions, as when products in an e-commerce catalog are grouped by color, material, or both. Retrieval-oriented embeddings may not recover these partitions, while generative clustering often relies on intermediate descriptions or iterative decisions. This paper presents CIM-Embed, a framework for learning instruction-conditioned multimodal embeddings for clustering. Supervised contrastive learning uses same-group positives and negatives from other groups, optionally including other tasks; heterogeneous structured batches accommodate tasks with different group counts. At inference, the model maps each object's image, text, and instruction directly to an embedding; HDBSCAN then forms a partition without the ground-truth cluster count. We construct ProductMIC from product metadata in Amazon Reviews 2023, comprising training data and an evaluation benchmark for instruction-conditioned clustering. The benchmark contains 5,373 products and 449 single-attribute and compositional grouping tasks spanning 50 leaf categories excluded from our training. With full-dimensional embeddings and HDBSCAN, direct within-task SupCon training achieves a mean per-task V-measure of 95.01, compared with 53.06 for WeMM 9B, the strongest general-purpose embedding baseline evaluated. Frozen product-trained encoders also improve out-of-domain conditional grouping, while controlled transfer experiments distinguish partition transfer from instruction response. We will release the code and dataset upon acceptance.

Figures & tables

Figure 1: Illustration of grouping the same eight faucets by color, handle type, or both. All partitions are schematic rather than empirical model predictions.

Figure 2: CIM-Embed training and inference. The illustrated batch combines two tasks with 20 object–instruction instances. Within SupCon compares groups within each task; Cross SupCon additionally includes instances from other tasks. At inference, image, text, and instruction produce embeddings for HDBSCAN. Partitions are schematic.

General-purpose models
Method	$V\uparrow$	ARI $\uparrow$	$Q\uparrow$	Coverage $\uparrow$	Clusters $\to 4.73$	$R_{\mathrm{dist}}\downarrow$	Align- ment $\downarrow$	Unifor- mity $\downarrow$
Tongyi Flash	45.62	0.1662	0.0462	73.14	7.57	0.9106	0.2533	-0.5755
Tongyi Plus	47.45	0.1808	0.0483	74.41	7.76	0.8954	0.2455	-0.5667
Gemini Embedding 2	49.68	0.2044	0.0345	74.82	7.77	0.8756	0.2525	-0.5929
Qwen3-VL 2B (official)	47.91	0.1882	0.0390	74.84	7.70	0.8911	0.5255	-1.1854
Qwen3-VL 8B (official)	50.89	0.2185	0.0374	75.00	7.67	0.8634	0.5460	-1.2665

Table 1: Task-wise conditional clustering and representation statistics, equally averaged over

449

task units for general-purpose models, external methods, and direct and staged training configurations.

Configuration	Task-wise			Pooled			Category
	$V\uparrow$	ARI $\uparrow$	$Q\uparrow$	$V\uparrow$	ARI $\uparrow$	$Q\uparrow$	$V\uparrow$	ARI $\uparrow$	$Q\uparrow$
Qwen3-VL 8B	50.89	0.2185	0.0374	59.21	0.0631	0.0364	89.32	0.6000	0.1132
InfoNCE	93.24	0.8943	0.4498	86.74	0.7919	0.2266	71.02	0.1423	0.0297
Within	95.01	0.9238	0.5561	91.85	0.8805	0.4279	75.46	0.2528	0.0618
Cross	94.68	0.9183	0.5551	93.86	0.9081	0.5207	85.02	0.5141	0.1704
Within + replay	94.23	0.9104	0.5471	90.85	0.8653	0.4021	96.96	0.8878	0.7430

Table 2: Three evaluation scopes on Qwen3-VL 8B. V is a percentage; ARI and Q are unscaled. Task-wise and pooled Q use

Q_{\mathrm{target}}

; category Q uses

Q_{\mathrm{leaf}}

. Replay rows include prior generic adaptation. Full results appear in Table 1 and Appendix Tables 16 and 17 .

Appendix figures & tables28 assets

Supplementary material from the paper’s appendix.

Appendix

Label source	Training pool	Evaluation
Details-verified	420,274	12,610
Evidence-supplemented non-boolean	106,498	952
Supplementary boolean	3,457	67
Total	530,229	13,629

Appendix

Table 3: Label provenance. The training column counts the accepted-label pool after excluding evaluation leaf categories and before training sampling. Evaluation labels are deduplicated by product ID and key.

Statistic	One attribute	Two attributes	Three attributes	Overall
Leaf categories	50	50	14	50
Task units	230	189	30	449
Distinct products	5,053	3,065	512	5,373
Product–task records	9,105	5,211	681	14,997
Products per task	10/40/98	16/24/72	16/22/30	10/30/98
Ground-truth clusters/task ( $K$ )	2/4/7	4/6/6	4/5/6	2/5/7

Appendix

Table 4: Evaluation-set and task-grouping statistics. Products are deduplicated by ID, with overlap across attribute orders. Product and record counts include only the instructed condition. The final three rows report minimum/median/maximum values; cluster sizes are computed over ground-truth clusters within tasks.

Schema	$P$	$N_{\mathrm{intra}}$	$N_{\mathrm{cross}}$	$D_{\mathrm{full}}$
C5M4	3	16	0	19
C4M5	4	15	0	19
C2M4+C3M4	3 / 3	4 / 8	12 / 8	19 / 19
C2M5+C2M5	4 / 4	5 / 5	10 / 10	19 / 19

Appendix

Table 5: Per-anchor comparison counts for four complete batch configurations at

B=20

under Cross SupCon.

P

counts positives;

N_{\mathrm{intra}}

and

N_{\mathrm{cross}}

count within-unit and cross-unit negatives;

D_{\mathrm{full}}

is the total candidate count. Slashes separate the counts for the two units.

Model	$V$	ARI	Coverage	Clusters
Gemini Embedding 2	49.68 $\to$ 53.80	0.2044 $\to$ 0.2796	74.82 $\to$ 97.92	7.77 $\to$ 8.11
Qwen3-VL 2B (official)	47.91 $\to$ 51.99	0.1882 $\to$ 0.2471	74.84 $\to$ 97.86	7.70 $\to$ 8.62
Qwen3-VL 8B (official)	50.89 $\to$ 56.59	0.2185 $\to$ 0.3085	75.00 $\to$ 97.75	7.67 $\to$ 8.31
WeMM 2B (official)	50.70 $\to$ 54.70	0.2120 $\to$ 0.2794	76.78 $\to$ 98.02	7.85 $\to$ 8.43
WeMM 9B (official)	53.06 $\to$ 60.53	0.2418 $\to$ 0.3575	74.06 $\to$ 97.93	7.52 $\to$ 8.25
Direct InfoNCE	93.24 $\to$ 92.64	0.8943 $\to$ 0.8824	97.75 $\to$ 99.93	4.90 $\to$ 5.21

Appendix

Table 6: UMAP comparison: task-wise clustering. Each cell reports full dimensions

\to

UMAP.

Model	$V$	ARI	Coverage	Clusters
Gemini Embedding 2	59.04 $\to$ 57.86	0.0417 $\to$ 0.0511	94.48 $\to$ 90.48	3806 $\to$ 3625
Qwen3-VL 2B (official)	59.37 $\to$ 58.35	0.0354 $\to$ 0.0493	94.60 $\to$ 91.72	4164 $\to$ 3681
Qwen3-VL 8B (official)	59.21 $\to$ 57.70	0.0631 $\to$ 0.0692	91.59 $\to$ 88.45	4002 $\to$ 3625
WeMM 2B (official)	59.34 $\to$ 57.65	0.0476 $\to$ 0.0587	93.61 $\to$ 89.00	4154 $\to$ 3702
WeMM 9B (official)	57.95 $\to$ 57.37	0.0932 $\to$ 0.1253	86.71 $\to$ 86.48	3915 $\to$ 3687
Direct InfoNCE	86.74 $\to$ 86.20	0.7919 $\to$ 0.7780	90.16 $\to$ 95.12	2006 $\to$ 2332

Appendix

Table 7: UMAP comparison: pooled conditional clustering. Each cell reports full dimensions

\to

UMAP.

Model	$V$	ARI	Coverage	Clusters
Gemini Embedding 2	83.40 $\to$ 75.27	0.4274 $\to$ 0.2019	84.03 $\to$ 90.12	70 $\to$ 340
Qwen3-VL 2B (official)	86.08 $\to$ 74.14	0.4797 $\to$ 0.1686	85.71 $\to$ 87.64	75 $\to$ 334
Qwen3-VL 8B (official)	89.32 $\to$ 74.83	0.6000 $\to$ 0.1847	91.68 $\to$ 88.83	52 $\to$ 328
WeMM 2B (official)	87.24 $\to$ 75.51	0.5305 $\to$ 0.2132	87.46 $\to$ 89.88	70 $\to$ 325
WeMM 9B (official)	86.67 $\to$ 73.78	0.4955 $\to$ 0.1559	85.58 $\to$ 88.11	75 $\to$ 345
Direct InfoNCE	71.02 $\to$ 72.71	0.1423 $\to$ 0.1447	76.12 $\to$ 93.37	276 $\to$ 489

Appendix

Table 8: UMAP comparison: generic category clustering. Each cell reports full dimensions

\to

UMAP.

Model	64	128	Full	1D	2D	3D	No-inst.	$\Delta V$
InfoNCE MRL frozen (old)	93.39	93.46	93.61	92.65	94.65	94.37	41.93	51.68
InfoNCE MRL unfrozen (old)	93.04	93.22	93.25	91.92	94.62	94.86	42.05	51.20
Direct InfoNCE full	93.08	93.57	93.24	91.86	94.64	95.00	42.35	50.89
Within MRL frozen (old)	94.94	94.72	94.83	93.43	96.26	96.46	46.27	48.56
Within MRL unfrozen (old)	94.77	94.67	94.81	93.18	96.44	96.92	43.75	51.05
Within system MRL (old)	94.64	94.75	94.52	92.64	96.57	96.10	43.90	50.63

Appendix

Table 9: Prefix, instruction-arity, and no-instruction configuration comparisons.

Method	$V\uparrow$	ARI $\uparrow$	$Q\uparrow$	Coverage $\uparrow$	Mean clusters $\to 4.73$	$R_{\mathrm{dist}}\downarrow$
$C2M12$	90.12	0.8333	0.5719	97.50	4.59	0.1396
$C3M8$	92.72	0.8817	0.5443	97.53	4.74	0.1344
$C4M6$	92.77	0.8836	0.5474	97.54	4.77	0.1296
$C6M4$	92.55	0.8831	0.5416	97.14	4.74	0.1323
Mixed C/M	93.17	0.8922	0.5420	97.83	4.86	0.1294
Multi-Unit balanced (Within)	92.82	0.8880	0.5443	97.54	4.82	0.1325

Appendix

Table 10: CM24 batch composition: task-wise conditional clustering.

Method	$V\uparrow$	$Q\uparrow$	Coverage $\uparrow$	Clusters $\to 2123$	$R_{\mathrm{dist}}\downarrow$	Align- ment $\downarrow$	Unifor- mity $\downarrow$
$C2M12$	86.80	0.3255	93.24	1887	0.5478	0.0525	-2.1646
$C3M8$	88.70	0.3636	93.07	1962	0.4253	0.0815	-2.6152
$C4M6$	89.20	0.3455	92.72	1935	0.4396	0.0797	-2.6548
$C6M4$	89.51	0.3579	93.49	1965	0.4123	0.0944	-2.9466
Mixed C/M	89.48	0.3592	94.39	2018	0.4148	0.0800	-2.6738
Multi-Unit balanced (Within)	89.72	0.3621	93.60	1973	0.4141	0.0904	-2.8390

Appendix

Table 11: CM24 batch composition: pooled conditional clustering.

Method	$V\uparrow$	$Q\uparrow$	Coverage $\uparrow$	Clusters $\to 50$
$C2M12$	63.25	0.0621	70.52	337
$C3M8$	61.95	0.0458	72.64	566
$C4M6$	60.15	0.0459	68.97	552
$C6M4$	59.57	0.0458	73.07	1091
Mixed C/M	60.39	0.0465	69.72	581
Multi-Unit balanced (Within)	61.24	0.0503	71.21	601

Appendix

Table 12: CM24 batch composition: generic category clustering.

Method	$V\uparrow$	ARI $\uparrow$	$Q\uparrow$	Coverage $\uparrow$	Mean clusters $\to 4.73$	$R_{\mathrm{dist}}\downarrow$
Qwen 2B Cross+Replay	92.76	0.8842	0.4974	97.90	4.83	0.1765
Qwen 2B Hier	41.21	0.1416	0.1519	78.79	6.65	0.9948
WeMM 2B Hier	42.60	0.1468	0.1183	76.95	6.99	0.9678
WeMM 9B Hier	44.01	0.1619	0.1238	79.60	7.13	0.9500
Earlier InfoNCE replay	93.22	0.8939	0.4689	98.04	4.92	0.2208
Hier SupCon V3 (earlier checkpoint)	44.34	0.1750	0.1631	82.54	6.84	0.9627

Appendix

Table 13: Additional model evaluation: task-wise conditional clustering.

Method	$V\uparrow$	$Q\uparrow$	Coverage $\uparrow$	Clusters $\to 2123$	$R_{\mathrm{dist}}\downarrow$	Align- ment $\downarrow$	Unifor- mity $\downarrow$
Qwen 2B Cross+Replay	90.43	0.4182	95.80	2089	0.3390	0.0933	-3.1460
Qwen 2B Hier	58.59	0.0817	94.39	3978	1.2767	0.0514	-2.1335
WeMM 2B Hier	59.36	0.0842	94.99	4156	1.2166	0.0646	-1.9737
WeMM 9B Hier	59.30	0.0894	95.32	4087	1.2115	0.0606	-2.0496
Earlier InfoNCE replay	87.80	0.2766	92.78	2054	0.4680	0.1886	-3.2836
Hier SupCon V3 (earlier checkpoint)	57.95	0.0835	94.64	3893	1.2596	0.0502	-2.0962

Appendix

Table 14: Additional model evaluation: pooled conditional clustering.

Method	$V\uparrow$	$Q\uparrow$	Coverage $\uparrow$	Clusters $\to 50$
Qwen 2B Cross+Replay	97.67	0.6963	99.03	53
Qwen 2B Hier	96.46	0.7054	98.34	58
WeMM 2B Hier	96.49	0.6493	97.75	51
WeMM 9B Hier	97.40	0.7427	99.74	50
Earlier InfoNCE replay	94.49	0.4669	97.54	63
Hier SupCon V3 (earlier checkpoint)	96.49	0.7880	98.88	57

Appendix

Table 15: Additional model evaluation: generic category clustering.

Method	$V\uparrow$	ARI $\uparrow$	$Q\uparrow$	Coverage $\uparrow$	Clusters	$R_{\mathrm{dist}}\downarrow$
General-purpose models
Tongyi Flash	59.45	0.0446	0.0802	95.01	3999	1.1008
Tongyi Plus	59.32	0.0505	0.0677	94.13	3979	1.0788
Gemini Embedding 2	59.04	0.0417	0.0504	94.48	3806	1.0030
Qwen3-VL 2B (official)	59.37	0.0354	0.0669	94.60	4164	1.0773
Qwen3-VL 8B (official)	59.21	0.0631	0.0364	91.59	4002	0.9705

Appendix

Table 16: Pooled conditional clustering with task-specific ground-truth groups

(U,y_{i}^{U})

retained across

449

tasks (

2{,}123

groups). Model groups follow Table 1 .

Method	$V\uparrow$	ARI $\uparrow$	$Q\uparrow$	Coverage $\uparrow$	Clusters	$R_{\mathrm{dist}}\downarrow$
General-purpose models
Tongyi Flash	84.85	0.4199	0.1214	82.93	79	0.6217
Tongyi Plus	87.99	0.5559	0.1393	88.57	74	0.6002
Gemini Embedding 2	83.40	0.4274	0.0609	84.03	70	0.7234
Qwen3-VL 2B (official)	86.08	0.4797	0.1264	85.71	75	0.6452
Qwen3-VL 8B (official)	89.32	0.6000	0.1132	91.68	52	0.6180

Appendix

Table 17: Generic category clustering with

50

leaf categories as targets. Model groups follow Table 1 ; generic inputs specify no target attribute.

General-purpose models
Method	Align- ment $\downarrow$	Unifor- mity $\downarrow$
Tongyi Flash	0.2533	-1.3519
Tongyi Plus	0.2455	-1.4391
Gemini Embedding 2	0.2525	-1.3503
Qwen3-VL 2B (official)	0.5255	-2.7918
Qwen3-VL 8B (official)	0.5460	-2.9168

Appendix

Table 18: Pooled representation diagnostics. Model order, selection, and aggregation follow Table 16 .

General-purpose models
Method	Align- ment $\downarrow$	Unifor- mity $\downarrow$
Tongyi Flash	0.3228	-1.3505
Tongyi Plus	0.3336	-1.4376
Gemini Embedding 2	0.4251	-1.4288
Qwen3-VL 2B (official)	0.8524	-3.2855
Qwen3-VL 8B (official)	0.6874	-2.8275

Appendix

Table 19: Category representation diagnostics. Model order, selection, and aggregation follow Table 17 .

Method	Overall $V$	Coverage	ARI	1D $V$	2D $V$	3D $V$
GCPM G1	57.32	76.06	0.3123	52.05	62.07	67.74
GCPM G2	57.61	78.05	0.3083	52.59	61.62	70.81
GCPM G3	74.85	100.00	0.5786	75.09	74.04	78.21
GCPM G4	74.17	100.00	0.5683	75.30	72.36	76.87
ABC	54.52	86.80	0.3221	48.38	60.20	65.86
CLAY	51.86	85.99	0.2415	43.72	58.96	69.54

Appendix

Table 20: External methods by instruction dimensionality under their original evaluation configurations. V-measure and coverage are percentages; ARI uses its original scale. Struck-through G3/G4 coverage marks complete assignment and is excluded from coverage ranking; the largest coverage among the remaining methods is bold.

Method	Fixed $V$	Grid $V$	CV $V$	CV ARI	CV Cov.
Qwen 8B Base	50.89	50.89	50.89	0.2185	75.00
WeMM 9B Base	53.06	53.06	53.06	0.2418	74.06
GCPM G1	57.32	57.32	57.32	0.3123	76.06
GCPM G2	57.61	57.61	57.61	0.3083	78.05
ABC	54.52	54.52	54.52	0.3221	86.80
CLAY	51.86	51.86	51.86	0.2415	85.99

Appendix

Table 21: Conditional clustering under a unified HDBSCAN selection protocol. Fixed and Grid report V-measure with fixed

(2,1)

parameters and shared-grid selection on the full task pool, respectively. CV reports five-fold leaf-grouped cross-validation. All metrics are task-macro averages. V-measure and coverage are percentages; ARI is unscaled. Unless specified otherwise, Direct configurations use Qwen3-VL 8B.

Method	Requests	Input tokens	Output tokens
ABC	0	0	0
CLAY	2,056	2,427,707	1,118,584
Multi-Sub	2,042	1,922,029	856,531
ITGC	6,265	12,273,548	1,969,358
GCPM G1/2	14,998	7,179,554	1,421,060
GCPM G3	12,695	6,128,699	1,006,614

Appendix

Table 22: Additional strong-LLM requests and tokens in the evaluation pipeline. GCPM G3/G4 rows report the merging stage; standalone branch totals also include shared proxy generation. Proxy input-token counts cover text only.

Method	$R_{\mathrm{dist}}\downarrow$	Alignment	$U_{\mathrm{task}}$	$U_{\mathrm{pool}}$
GCPM G1	0.7817	0.2116	-0.5581	-1.9754
GCPM G2	0.7797	0.3690	-0.9585	-2.9360
GCPM G3	0.7817	0.2116	-0.5581	-1.9754
GCPM G4	0.7797	0.3690	-0.9585	-2.9360
ABC	0.7184	0.0915	-0.3987	-3.6362
CLAY	0.8188	1.5644	-3.1463	-3.9322

Appendix

Table 23: External-method geometry recovered from saved representations. All task-wise quantities average the same 449 complete units.

Model	Rank $V$	Suit $V$	Mean $V$	ARI	Cov.	$(c,s)$
Qwen 8B Direct Within	88.80	88.29	88.55	0.8748	94.53	(5, –)
WeMM 2B Direct Within	88.55	86.94	87.75	0.8703	92.13	(5, –)
Earlier MRL Cross	88.18	86.00	87.09	0.8551	93.29	(5, –)
Qwen 8B Direct Cross	87.10	85.00	86.05	0.8359	91.93	(5, –)
WeMM 9B Cross + replay	85.32	83.74	84.53	0.8042	88.50	(5, –)
WeMM 9B Direct Within	92.90	75.44	84.17	0.8320	90.32	(5, –)

Appendix

Table 24: Cards conditional clustering with a 12-point grid selected using target labels.

Model	$V$	ARI	$Q$	Cov.	Clusters	$R_{\mathrm{dist}}$	Align.	Unif.
Qwen 8B Base	42.07	0.1701	0.0308	84.94	517.50	0.8218	0.5564	-1.4372
Direct InfoNCE	60.93	0.3298	0.1991	95.45	28.00	0.5015	0.2233	-1.1758
Direct Within	64.19	0.4796	0.2094	84.15	10.00	0.3822	0.0613	-0.6255
Direct Cross	54.09	0.2826	0.1705	93.16	42.00	0.4934	0.1721	-0.9979
Stage2 InfoNCE	35.41	0.1531	0.1132	84.34	1006.75	0.8008	0.5547	-1.3047
Within + Mix	66.46	0.4702	0.2038	91.25	14.50	0.3676	0.0638	-0.6026

Appendix

Table 25: CLEVR-4: equal-weight means over four conditions. V and its differences are in percentage points; coverage is a percentage.

Model	$V_{\mathrm{correct}}$	$V_{\mathrm{none}}$	$\Delta V_{\mathrm{none}}$	$\Delta V_{\mathrm{other}}$
Qwen 8B Base	42.07	35.86	+6.21	+11.81
Direct InfoNCE	60.93	23.83	+37.10	+48.05
Direct Within	64.19	7.61	+56.58	+59.22
Direct Cross	54.09	24.78	+29.31	+35.42
Stage2 InfoNCE	35.41	31.33	+4.08	+5.22
Within + Mix	66.46	23.94	+42.53	+60.18

Appendix

Table 26: CLEVR-4: instruction controls on the fixed image pool. V and its differences are in percentage points; coverage is a percentage.

Model	$V$	ARI	$Q$	Cov.	Clusters	$R_{\mathrm{dist}}$	Align.	Unif.
Qwen 8B Base	12.44	0.0042	0.0235	38.55	7246.50	0.8750	1.1154	-2.3130
Direct InfoNCE	32.33	0.1413	0.0555	44.46	9.00	0.7445	0.5569	-1.2517
Direct Within	31.76	0.1227	0.0751	48.84	8.50	0.6287	0.2241	-0.5551
Direct Cross	31.75	0.1033	0.0451	49.03	8.50	0.6375	0.4023	-0.9462
Stage2 InfoNCE	13.35	0.0159	0.0286	21.76	3.00	0.7732	0.3639	-0.7788
Within + Mix	27.28	0.0860	0.0631	42.31	10.50	0.6357	0.2073	-0.5037

Appendix

Table 27: CIFAR10-MC: equal-weight means over two conditions. V and its differences are in percentage points; coverage is a percentage.

Model	$V_{\mathrm{Type}}$	$V_{\mathrm{Env.}}$	$\Delta V_{\mathrm{Type}}$	$\Delta V_{\mathrm{Env.}}$
Qwen 8B Base	10.86	14.02	-0.03	+0.04
Direct InfoNCE	33.01	31.65	+0.60	-1.53
Direct Within	29.74	33.79	-6.51	+0.73
Direct Cross	29.79	33.72	-6.37	+0.27
Stage2 InfoNCE	20.76	5.93	+10.99	-4.89
Within + Mix	26.26	28.29	-4.66	-0.69

Appendix

Table 28: CIFAR10-MC: per-condition results and cross-instruction controls. V and its differences are in percentage points; coverage is a percentage.

Figure 3: Computer Case representations with and without instructions. Models, projections, and evaluation conventions follow Figure 4 . Hollow circles and filled triangles denote representations without and with instructions, respectively; colors denote ground-truth groups. Both conditions are jointly projected within each cell, with independent fits across cells. Product pools are identical across models within a task but can differ across tasks. Instructed

V

is original-space HDBSCAN V-measure under the evaluation-label-tuned protocol.

Figure 4: Real representations with and without instructions. Single- and joint-attribute tasks on Wheel. Rows denote models and columns denote single- or joint-attribute tasks. Hollow circles indicate no instruction, filled triangles indicate an instruction, and colors denote ground-truth groups. Within each cell, the two conditions are jointly projected with UMAP after L2 normalization; projections are fitted independently across models and tasks, so positions and distances should not be compared across cells. Each task uses the same product pool across models, but pools can differ across tasks. Instructed

V

is HDBSCAN V-measure (0–100) on the original instructed 4,096-dimensional representations under the evaluation-label-tuned protocol used in the main tables, not a score computed from the two-dimensional points.

Clustering

Clustering groups data points by shared structure, developing methods to discover, validate, and interpret groups across diverse data and constraints.

See all 335 papers

Mar 6, 2026

Cluster-R1: Large Reasoning Models Are Instruction-following Clustering Agents

General-purpose embedding models excel at recognizing semantic similarities but fail to capture the characteristics of texts specified by user instructions. In contrast, instruction-tuned embedders can align embeddings with textual instructions yet cannot autonomously infer latent corpus structures, such as determining the optimal number of clusters. To address both limitations, we reframe instruction-following clustering as a generative task and train large reasoning models (LRMs) as autonomous clustering agents. Our reasoning-driven training pipeline enables LRMs to interpret high-level clustering instructions and then infer the corresponding latent groupings. To evaluate this paradigm, we introduce ReasonCluster, a comprehensive benchmark comprising 28 diverse tasks spanning daily dialogue, legal cases, and financial reports. Experiments across diverse datasets and clustering scenarios show that our approach consistently outperforms strong embedding-based methods and LRM baselines, demonstrating that explicit reasoning fosters more faithful and interpretable instruction-based clustering.

Peijun Qing, Puneet Mathur, Nedim Lipka +5

Dartmouth College, USA · Adobe Research, San Jose, USA

Clustering Unsupervised Clustering Large Reasoning Models

Ask PDF

Jul 24, 2026

Controlling Embedding Spaces with Text-Conditioned Transformations

Multimodal embedding spaces in models like CLIP enable powerful capabilities such as semantic similarity retrieval and cross-modal zero-shot classification. These embeddings compress high-level semantics into a single vector, which comes at the cost of primarily expressing a dominant semantics like main object while suppressing other important attributes such as camera angle or color tone. We propose a text-conditioned transformation of visual embeddings that makes such attributes explicitly accessible. Given a natural language description of an attribute category (e.g., "color" or "art style"), a network generates an affine transformation that emphasizes the specified attribute. Conditioning on text enables it to learn many attributes simultaneously, accessing them at inference time through an intuitive interface. The network is trained to align transformed embeddings with the frozen latent space, enabling retrieval using existing large-scale embeddings without any re-encoding. When applied to a full set, the same mechanism transforms the latent space for attribute disentanglement tasks such as multi-clustering. By operating directly in latent space, our method provides a unified and efficient framework for controlling embedding spaces, demonstrating state-of-the-art performance across both attribute-based retrieval and multi-attribute organization tasks with near-zero inference cost. Project page: https://joefioresi718.github.io/ControlEmbed_webpage/

Joseph Fioresi, Fabian Caba Heilbron, Pankaj Nathani +2

Institute of Artificial Intelligence, University of Central Florida, USA · ⋆Majority of work done as an intern at Adobe Research · Adobe Research, USA

Representation Disentanglement Multimodal Embedding Cross-Modal Retrieval

Ask PDF

Jun 23, 2026

Universal Guideline-Driven Image Clustering via a Hybrid LLM Agent

Unifying image clustering across different clustering scenarios remains challenging due to fundamental gaps among tasks. We introduce a Guideline-Driven Image Clustering Agent, the first universal framework that bridges these gaps through textual guidelines. To incorporate complex guidelines without task-specific training, we propose Generative Concept Proxy Modeling, which generates guideline-aware embeddings via concept proxy extraction. For scenarios requiring automatic cluster discovery, we introduce LLM Traversal based on Minimum Spanning Tree that selectively applies LLM reasoning for complex semantic judgments. Our method generalizes across diverse clustering scenarios spanning from general to fine-grained categorization, from global to local criteria, and from balanced to long-tail distributions. Our framework consistently outperforms specialized methods across diverse clustering tasks.

Wenliang Zhong, Rob Barton, Lucas Goncalves +7

1The University of Texas at Arlington · 2Amazon

Clustering Unsupervised Clustering Image Clustering Semantic Clustering

Ask PDF

Abstract

Clustering

Explore similar work

Cluster-R1: Large Reasoning Models Are Instruction-following Clustering Agents

Controlling Embedding Spaces with Text-Conditioned Transformations

Universal Guideline-Driven Image Clustering via a Hybrid LLM Agent