TISC: A Text-Driven Image Semantic Communication System for Faithful Reconstruction

Feifan Zhang; Yuyang Du; Xiaoyan Liu; Soung Chang Liew

cs.CVAug 17, 2026

TISC: A Text-Driven Image Semantic Communication System for Faithful Reconstruction

Authors: Feifan Zhang, Yuyang Du, Xiaoyan Liu, Soung Chang Liew

Organizations: Department of Information Engineering, The Chinese University of Hong Kong, HKSAR, China

Abstract

Generative image semantic communication converts an image into a text description and then performs text-to-image reconstruction at the receiver via diffusion-based generative models. This paradigm has attracted broad attention due to its extremely low bandwidth cost. However, existing methods still face two critical bottlenecks across image-to-text (I2T) semantic extraction at the transmitter and text-to-image (T2I) semantic reconstruction at the receiver: (i) semantic loss and distortion in I2T, where holistic image descriptions may omit fine-grained object attributes and spatial-position information, causing the generated text to deviate from the original image semantics; and (ii) insufficient semantic faithfulness in T2I, where even with the same semantically faithful text description, different initial noise settings may lead diffusion-based reconstruction to produce images with different levels of semantic consistency with the original image. These issues jointly limit the semantic faithfulness of image reconstruction. To address them, we propose TISC, a text-driven image semantic communication framework tailored for faithful reconstruction. TISC incorporates two key designs: (1) Tree-Structured Attribute Semantic Extraction (TSASE), which decomposes semantic extraction into global scene, background, and object-level attribute descriptions, covering spatial position, shape/pose, color, material, and other physical attributes for each detected object; and (2) an Initial Noise Optimization (INO) mechanism, which selects an initial noise seed at the transmitter according to a comprehensive similarity score that jointly considers visual and semantic consistency. Experiments on multiple datasets show that TSASE improves object-position recovery and semantic description faithfulness, while the INO parameter study supports the adopted configuration for noise selection.

Figures & tables

Fig. 1: Comparative illustrations of conventional methods versus our generative semantic communication framework from two aspects: (a) I2T semantic distortion: conventional methods may suffer from attribute-level semantic distortions such as missing or incorrect color and spatial information, which impair semantic extraction, while our approach mitigates such distortions and improves the reliability of semantic extraction; (b) insufficient semantic faithfulness in T2I reconstruction: conventional methods randomly sample the initial noise, so different noises may lead to reconstruction results with different semantic faithfulness even under the same text description. The proposed method evaluates multiple candidate initial noises at the transmitter and selects the one that produces the most faithful reconstruction within the candidate set.

Fig. 2: Overall architecture of the proposed TISC framework. At the transmitter, TSASE extracts the structured semantic description $D$ , while INO selects the initial noise seed $s^{*}$ . These two components are introduced in Sections II and III , respectively. The communication network delivers $D$ and $s^{*}$ to the receiver, where they are used to guide LLM-grounded Diffusion-based image reconstruction.

Fig. 3: Example of the tree-structured attribute semantic extraction process.

Fig. 4: An example of the structured semantic description produced by TSASE.

Fig. 5: Illustration of IoU calculation for spatial matching. (a) Original image, where the yellow box denotes the bounding box $\bm{b_{i}}$ provided by the spatial-position attribute of the $i$ -th object description $d_{i}^{S}$ . (b) Candidate reconstruction, where the blue box denotes the bounding box $\bm{b_{j}^{\prime}}$ provided by the spatial-position attribute of the $j$ -th object description ${d^{\prime}}_{j}^{S}$ . (c) Spatial overlap between $\bm{b_{i}}$ and $\bm{b_{j}^{\prime}}$ , which is used to compute their IoU.

Fig. 6: Illustration of matched, missed, and hallucinated objects based on spatial matching. The left panel shows the original image, and the right panel shows the reconstructed image. Green boxes indicate matched objects that appear in both images with sufficient spatial overlap. The red box indicates a missed object that appears only in the original image, while the orange box indicates a hallucinated object that appears only in the reconstructed image.

Fig. 7: Comparison of average IoU between TISC and the w/o TSASE ablation across tested datasets.

Fig. 8: Average TIFA scores (

\overline{\mathrm{TIFA}}

) of TISC and w/o TSASE across five tested datasets.

Fig. 10: Empirical distribution of comprehensive similarity scores induced by random initial noise seeds. The three subfigures correspond to COCO image IDs 99, 93, and 71, respectively. For each image, we fix the semantic description, generation model, and inference configuration, and generate 500 candidate reconstructions using 500 different initial noise seeds. The comprehensive similarity score of each candidate is computed according to Eq. ( 6 ). The blue histogram shows the empirical distribution of the 500 comprehensive similarity scores, where the y-axis denotes probability density rather than sample count, computed as the bin count divided by the product of the total number of samples and the bin width, so that the total area of the histogram is normalized to one. The x-axis represents the comprehensive similarity score. The red curve shows the Gaussian distribution fitted using the empirical mean and standard deviation of the comprehensive similarity scores.

Fig. 11: MLLM-guided evaluation of different weighting configurations.

Fig. 12: End-to-end noise robustness comparison over an AWGN channel. To characterize the channel resource usage of different schemes, we report the average number of channel symbols used per image. For TISC and the JPEG baseline, this number is counted from the output symbols after BPSK modulation. For DeepJSCC, since its end-to-end model implicitly includes source coding, channel coding, and modulation-related operations, we directly count the number of channel symbols produced by its encoder. Under this experimental setting, TISC, the JPEG baseline, and DeepJSCC use approximately 12.6k, 72.2k, and 16.4k channel symbols per image on average, respectively. This indicates that the performance advantage of TISC is not obtained by using more channel resources.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Fig. 13: Comparison between JPEG-based image transmission and the proposed text-driven semantic transmission scheme in terms of compressed bits and the comprehensive similarity score. The x-axis reports the average number of compressed bits per image, and the y-axis reports the dataset-level average comprehensive similarity score computed according to Eq. 6 .

Explore similar work

May 27, 2025

ISAC: Training-Free Instance-to-Semantic Attention Control for Multi-Instance Generation

Recent open-weight text-to-image (T2I) diffusion models still struggle with multi-instance prompts, often omitting or merging instances and mixing semantics among similar objects. We trace these failures to early denoising steps, before instance boundaries are reliably stabilized. Existing training-free guidance is largely driven by cross-attention or other token-conditioned semantic signals. Such guidance can separate concepts at the token level, but largely assumes that distinct instance regions have already emerged. In early denoising steps, it cannot reliably carve out these regions, so count failures and semantic mixing persist. By contrast, self-attention exposes class-agnostic instance layouts during early denoising. To exploit this asymmetry, we propose

\textbf{ISAC}

(

\textbf{I}

nstance-to-

\textbf{S}

emantic

\textbf{A}

ttention

\textbf{C}

ontrol), a training-free, model-agnostic objective that first stabilizes self-attention layouts and then binds cross-attention semantics within them, without fine-tuning or external vision models. Across T2I-CompBench, HRS-Bench, and our newly curated IntraCompBench, ISAC consistently outperforms prior training-free methods. Furthermore, ISAC enhances layout-to-image controllers by refining coarse, overlapping bounding boxes into dense instance masks. Code and IntraCompBench are available at https://shjo-april.github.io/ISAC.

Sanghyun Jo, Wooyeol Lee, Ziseok Lee +3

OGQ, Seoul, Korea · Seoul National University, Seoul, Korea

Diffusion Model Guidance T2I Generation Attention Control in Diffusion Models Compositional T2I Generation Attention Mechanisms Controllable Image Generation

Ask PDF

Jul 21, 2026

One Rewrite to Fix Them All? Type-Aware Repair Allocation for Text-to-Image Prompt Optimization

Text-to-image (T2I) generators often fail to follow their prompts faithfully, producing wrong counts, swapped attributes, ambiguous relations, and illegible text. Prompt optimization repairs such failures by rewriting the user prompt, requiring no generator retraining, and has yielded promising results. However, existing optimizers absorb heterogeneous failures into one uniform prompt expansion, even though each calls for different repair language. We formulate semantic prompt optimization as atomic repair allocation: each failed proposition is routed to a type-conditioned repair operator before the resulting local constraints are compiled into one executable prompt. We instantiate this formulation in the training-free Type-Aware Repair Allocation (TARA) framework, which separates diagnosis, allocation, compilation, and a semantic repair gate, an accept-or-revert controller over exactly one prescribed repair that prevents semantic regressions. Extensive experiments on DSG and TIFA across four frozen generators demonstrate that TARA achieves the best semantic accuracy in all eight benchmark-generator cells, improving over VisualPrompter by 5.6 and 2.6 points on DSG and TIFA, respectively, while maintaining image quality and running fastest in our matched local setting at 16.0 seconds versus 20.0 seconds per prompt.

Haoyue Liu, Xiaoyu Ma, Ye Chen +2

School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen 518172, China · XJTU-POLIMI Joint School, Xi’an Jiaotong University, Xi’an 710049, China · Shenzhen Future Network of Intelligence Institute (FNii-Shenzhen)

T2I Generation Automatic Prompt Optimization Prompt Optimization for T2I Generation

Ask PDF

Jun 9, 2026

STEDiff: Strengthening Text Embedding for Text-to-Image Alignment in Diffusion Model

Although pretrained text-to-image (T2I) generation models can produce high-quality images, they often fail to faithfully reflect the semantic intent of complex prompts due to stochastic noise and inherent model limitations. This issue frequently manifests as the model overlooking specific objects or failing to correctly bind attributes to their corresponding entities, a challenge referred to as semantic alignment. Unlike existing approaches that rely on computationally expensive fine-tuning or labor-intensive layout priors, we propose STEDiff, a training-free method designed to enhance semantic representations directly within the text-embedding space. Specifically, we introduce a method that primarily leverages the [EOT] token to strengthen the relevant semantics of sub-sentences and then replaces the corresponding tokens in the original prompt. Furthermore, a novel semantic enhancement loss is incorporated to enforce spatial constraints, ensuring that the semantics of each entity are precisely mapped to their respective image regions. Extensive quantitative and qualitative evaluations on the T2I-CompBench demonstrate that our method notably improves semantic consistency and generation integrity in complex scenarios.

Hailan Zhang, Haipeng Liu, Bo Fu +1

School of Computer Science and Information Engineering, Hefei University of Technology, Hefei, China · School of Computer and Artificial Intelligence, Liaoning Normal University, Dalian, China

Text Embeddings Text-Image Alignment Compositional T2I Generation

Ask PDF