Multimodal embedding spaces in models like CLIP enable powerful capabilities such as semantic similarity retrieval and cross-modal zero-shot classification. These embeddings compress high-level semantics into a single vector, which comes at the cost of primarily expressing a dominant semantics like main object while suppressing other important attributes such as camera angle or color tone. We propose a text-conditioned transformation of visual embeddings that makes such attributes explicitly accessible. Given a natural language description of an attribute category (e.g., “color” or “art style”), a network generates an affine transformation that emphasizes the specified attribute. Conditioning on text enables it to learn many attributes simultaneously, accessing them at inference time through an intuitive interface. The network is trained to align transformed embeddings with the frozen latent space, enabling retrieval using existing large-scale embeddings without any re-encoding. When applied to a full set, the same mechanism transforms the latent space for attribute disentanglement tasks such as multi-clustering. By operating directly in latent space, our method provides a unified and efficient framework for controlling embedding spaces, demonstrating state-of-the-art performance across both attribute-based retrieval and multi-attribute organization tasks with near-zero inference cost.
ControlEmbed maps a natural-language attribute description such as “color,” “camera angle,” or “mood” to an affine transformation of frozen visual embeddings. Applying the transform only to a query enables attribute-aware retrieval over an unchanged gallery while applying it globally reorganizes the same embedding set for attribute-specific clustering.
Text-conditioned transformation. ControlEmbed leaves the pretrained vision-language model frozen and learns one small generator of feature-space transformations. An attribute phrase such as “color” or “camera angle” is encoded as \(\mathbf{t}\), and the hypernetwork \(H_{\phi}\) converts that representation into an affine matrix \(\mathbf{W}\) and bias \(\mathbf{b}\). The resulting operator acts directly on a frozen image embedding \(\mathbf{v}\):
Changing the text input selects a different comparison axis without producing a new encoder or attribute-specific checkpoint. Because the vision and text encoders remain frozen, the method operates entirely on stored features and does not require image re-encoding.
Multi-attribute training. At each optimization step, one attribute taxonomy is chosen and its labels determine the positive groups in the batch. A transformed query is compared against unmodified base embeddings: samples sharing the selected attribute label are treated as positives, while the remaining samples provide negatives. Rotating through taxonomies lets the same hypernetwork learn many attribute controls without forcing unrelated label spaces into a single fixed grouping.
Here, the feature vectors are \(\ell_2\)-normalized, \(N\) is the batch size, \(\tau\) controls the contrastive temperature, and \(\mathds{1}_{i,j}\) identifies pairs with the same label under the active taxonomy. The loss pulls each transformed query toward matching frozen features while preserving the original space as the retrieval target. A multimodal alignment term additionally relates transformed image features to text embeddings of the attribute labels, grounding the learned transformations in language.
Two inference modes. Query mode changes only the query vector, leaving gallery features untouched so existing indexes can be reused. Global mode applies the generated operator to every stored vector, yielding an attribute-specific space for clustering. Both modes use the same trained hypernetwork and differ only in where the transformation is applied.
Attribute-aware retrieval. ControlEmbed transforms only the query while searching the original frozen gallery, preserving compatibility with precomputed large-scale indexes. Base features reach 33.8 mAP on Clevr-4, 43.7 on Stanford40, and 25.2 on ShotBench, while our query mode reaches 77.0, 86.3, and 41.0 mAP, respectively. A single hypernetwork supports multiple categories instead of storing one adapter per attribute.
Multi-clustering. The same learned transform can be applied to the full embedding set, followed by Faiss k-means and Hungarian matching. Our global mode reaches 73.0 clustering accuracy on Clevr-4, 80.0 on Stanford40, and 51.6 on ShotBench with near-instant inference time. Our proposed query-focused training method naturally handles this attribute disentanglement setting and achieves the highest average clustering accuracy across datasets while using a single transformation network trained once for all attributes.
Figure 4 shows near-peak Stanford40 performance with six training samples per class. Figure 5 shows less than 1.5% variation while scaling one hypernetwork to 19 attributes and 187 subclasses.
Figure 6 shows gains across a diverse set of both uni- and multi-modal backbones: CLIP, SigLIP, DINOv2, and Qwen3-VL-Embed.
Figure 7 shows shot-size labels changing from entangled clusters to a continuous, interpretable axis after transformation.
Figure 3 shows color- and location-conditioned similarity on MS-COCO images, despite training the corresponding transforms on smaller source datasets.
We presented a text-conditioned transformation framework that enables fine-grained control over pretrained embedding spaces without modifying encoder weights or recomputing gallery features. A single hypernetwork learns to generate attribute-specific affine transformations from natural-language descriptions, supporting both attribute-aware retrieval and multi-attribute organization with near-zero inference overhead. While the method generalizes across diverse attributes, its expressiveness is currently bounded by the range of categories seen during training. Future work may explore scaling up training to find emergent transformation properties across attributes completely unseen during training.
@inproceedings{fioresi2026controlling,
title={Controlling Embedding Spaces with Text-Conditioned Transformations},
author={Fioresi, Joseph and Caba Heilbron, Fabian and Nathani, Pankaj and Shah, Mubarak and Kafle, Kushal},
booktitle={European Conference on Computer Vision (ECCV)},
year={2026}
}