Academic

TSEmbed: Unlocking Task Scaling in Universal Multimodal Embeddings

arXiv:2603.04772v1 Announce Type: new Abstract: Despite the exceptional reasoning capabilities of Multimodal Large Language Models (MLLMs), their adaptation into universal embedding models is significantly impeded by task conflict. To address this, we propose TSEmbed, a universal multimodal embedding framework that synergizes Mixture-of-Experts (MoE) with Low-Rank Adaptation (LoRA) to explicitly disentangle conflicting task objectives. Moreover, we introduce Expert-Aware Negative Sampling (EANS), a novel strategy that leverages expert routing distributions as an intrinsic proxy for semantic similarity. By dynamically prioritizing informative hard negatives that share expert activation patterns with the query, EANS effectively sharpens the model's discriminative power and refines embedding boundaries. To ensure training stability, we further devise a two-stage learning paradigm that solidifies expert specialization before optimizing representations via EANS. TSEmbed achieves state-of-t

Y
Yebo Wu, Feng Liu, Ziwei Xie, Zhiyuan Liu, Changwang Zhang, Jun Wang, Li Li · · 1 min read · 2 views
#cs.CL #cs.AI

arXiv:2603.04772v1 Announce Type: new Abstract: Despite the exceptional reasoning capabilities of Multimodal Large Language Models (MLLMs), their adaptation into universal embedding models is significantly impeded by task conflict. To address this, we propose TSEmbed, a universal multimodal embedding framework that synergizes Mixture-of-Experts (MoE) with Low-Rank Adaptation (LoRA) to explicitly disentangle conflicting task objectives. Moreover, we introduce Expert-Aware Negative Sampling (EANS), a novel strategy that leverages expert routing distributions as an intrinsic proxy for semantic similarity. By dynamically prioritizing informative hard negatives that share expert activation patterns with the query, EANS effectively sharpens the model's discriminative power and refines embedding boundaries. To ensure training stability, we further devise a two-stage learning paradigm that solidifies expert specialization before optimizing representations via EANS. TSEmbed achieves state-of-the-art performance on both the Massive Multimodal Embedding Benchmark (MMEB) and real-world industrial production datasets, laying a foundation for task-level scaling in universal multimodal embeddings.

Executive Summary

The article proposes TSEmbed, a universal multimodal embedding framework that addresses task conflict in Multimodal Large Language Models (MLLMs) by combining Mixture-of-Experts (MoE) with Low-Rank Adaptation (LoRA). It introduces Expert-Aware Negative Sampling (EANS) to refine embedding boundaries and a two-stage learning paradigm for training stability. TSEmbed achieves state-of-the-art performance on the MMEB and industrial production datasets, enabling task-level scaling in universal multimodal embeddings.

Key Points

  • ▸ Introduction of TSEmbed, a universal multimodal embedding framework
  • ▸ Combination of MoE and LoRA to disentangle conflicting task objectives
  • ▸ Proposal of EANS to sharpen the model's discriminative power

Merits

Improved Performance

TSEmbed achieves state-of-the-art performance on benchmark datasets, demonstrating its effectiveness in universal multimodal embeddings.

Demerits

Complexity

The combination of MoE, LoRA, and EANS may add complexity to the model, potentially increasing computational requirements and training time.

Expert Commentary

The proposed TSEmbed framework demonstrates a significant advancement in universal multimodal embeddings, addressing the long-standing issue of task conflict. The introduction of EANS and the two-stage learning paradigm are notable contributions, as they enable more effective and efficient representation learning. However, further research is needed to fully explore the potential of TSEmbed and its applications, particularly in real-world scenarios. The complexity of the model may also require additional computational resources, which could be a limiting factor in some cases.

Recommendations

  • ✓ Further evaluation of TSEmbed on diverse datasets and applications to assess its generalizability and robustness
  • ✓ Investigation of potential extensions or modifications to TSEmbed, such as incorporating additional modalities or tasks, to further improve its performance and versatility

Sources

Related Articles