ViQ: Bridging the Gap Between Semantics and Detail in Visual Quantized Representations

Representing images as discrete tokens, akin to text, has long been a goal for multimodal AI, it promises a unified representation space that simplifies model architecture and reduces computational overhead. But the path is fraught with information loss. Reconstruction-focused representations capture fine-grained pixels but miss semantic structure, while semantically rich features often discard visual detail. A new paper from Tencent-Hunyuan, titled "ViQ: Text-Aligned Visual Quantized Representations at Any Resolution," proposes a framework that aims to reconcile these competing demands.

The Core Challenge: Quantization Without Sacrifice

Existing approaches to discrete visual representation fall into two camps. Those optimized for reconstruction, such as VQ-VAE and its variants, can faithfully reproduce pixel-level details but fail to encode the high-level concepts that language models rely on. Conversely, representations derived from contrastive language-image pre-training (like CLIP) carry strong semantic signals but compress or discard visual granularity, making them unsuitable for tasks that require precise reconstruction.

ViQ addresses this by structuring quantization learning into two distinct stages. The first stage, termed text-aligned pre-training, exposes the visual encoder to semantic-rich supervision from a pretrained language model. Critically, this stage is designed to handle native-resolution inputs, meaning images are not cropped, resized, or patchified before encoding, preserving all original detail. The second stage, feature discretization, uses a novel proximal representation learning strategy to progressively compact the feature space, combined with a position-aware head-wise quantization mechanism that adapts to arbitrary resolutions.

Technical Innovation: Proximal Learning and Head-Wise Quantization

The proximal representation learning strategy is the paper's key algorithmic contribution. Rather than forcing the entire feature space into a fixed-size codebook in one pass, the method applies iterative compression, gradually clustering and quantizing features while preserving neighborhoods from the continuous space. This preserves both local detail and global structure during quantization.

The position-aware head-wise quantization mechanism further enhances flexibility. Standard vector quantization treats each feature position independently, which fails to capture spatial relationships in images of varying size. ViQ introduces multiple quantization heads, each attending to a different spatial scale, and weights their outputs based on positional context. This allows the encoder to allocate more bits to semantically important regions without wasting capacity on uniform backgrounds.

Empirical Performance: Competitive Quality with Major Speed Gains

Extensive experiments across multimodal benchmarks, including image captioning, visual question answering, and text-to-image generation, show that ViQ's discrete representations achieve performance within 1 to 3 percentage points of state-of-the-art continuous vision encoders (such as SigLIP and DINOv2), while using significantly fewer parameters and producing discrete tokens. On low-level reconstruction tasks, ViQ maintains high precision, outperforming other discrete methods by a substantial margin.

Perhaps the most striking finding is the training acceleration. When ViQ's quantized representations are used as input for multimodal LLM training, the authors report speedups of 20% to 70% depending on the base LLM size and training recipe. The acceleration stems from the reduced dimensionality and simpler architecture of discrete tokens, which slash both memory bandwidth and matrix multiplication costs during forward passes.

"ViQ is the first framework to demonstrate that discrete visual representations can simultaneously match continuous encoders on multimodal tasks and offer dramatically faster training," the paper states. The authors attribute the efficiency gains to the compactness of quantized tokens and the removal of expensive continuous-to-discrete projection layers.

Implications for Multimodal AI

If ViQ's results hold up in practice, the framework could accelerate the development of large-scale multimodal models that process images and text through a single token space. Unified representations are crucial for next-generation assistants, robotics perception, and real-time vision-language systems where latency and compute budget are tightly constrained.

The ability to handle native-resolution inputs is particularly important. Many existing vision encoders require resizing images to a fixed resolution, which both discards detail and introduces distortion. ViQ's resolution-agnostic design means it can process high-resolution documents, microscopy images, or satellite photos without pre-processing, retaining information that would otherwise be lost.

The paper also notes that ViQ's discrete tokens are compatible with existing transformer architectures and can be fed directly into LLMs without architectural modifications, a practical advantage for teams looking to integrate vision without rewriting their model stack.

Limitations and Open Questions

The study's primary limitation is that all experiments were run on curated, clean datasets (COCO, Flickr30k, and a proprietary internal dataset). Performance on noisy, domain-shifted, or low-quality images, common in real-world deployments, remains untested. Additionally, the codebook size and quantization overhead at inference time are not thoroughly analyzed; if the number of heads or codebook entries grows too large, the efficiency gains could erode.

The paper does not release a pre-trained model or code at time of publication, which limits reproducibility. External validation will be essential to confirm that the reported speedups translate to production workflows.

Conclusion

ViQ represents a thoughtful step toward a unified representation space for vision and language, tackling the long-standing trade-off between semantic richness and visual detail in discrete tokenization. By combining text-aligned pre-training with progressive quantization and position-aware heads, Tencent-Hunyuan's team achieves competitive multimodal performance while delivering substantial training speedups. As multimodal models grow in scale and ambition, efficient and detail-preserving discrete representations may become a key enabler.