ICLR 2026

Co-LoRA

Collaborative Model Personalization on
Heterogeneous Multi-Modal Clients

Minhyuk Seo^1,2,* Taeheon Kim^2,* Hankook Lee³ Jonghyun Choi^2,† Tinne Tuytelaars^1,†

¹KU Leuven ²Seoul National University ³Sungkyunkwan University

OpenReview PDF arXiv Code

The setup

Personalize different multimodal models—together.

Each client learns its own stream of private multimodal tasks and may use a different model architecture. A central server coordinates knowledge sharing so every client can improve its own model without requiring identical data, tasks, or model dimensions.

A central server coordinates clients with different multimodal task streams and different model architectures. — **Setup.** Clients face both **data heterogeneity** (different private task streams) and **model heterogeneity** (different model architectures), yet collaborate through a shared server. Open full size ↗

The problem

Standard LoRA cannot be averaged across heterogeneous models.

LoRA’s input and output matrices inherit each model’s hidden dimension. When clients use different architectures, those matrices no longer have compatible shapes. Co-LoRA moves collaboration into the shared low-rank space instead.

Model-dependent LoRA Different shapes

Model 1 · d₁ A₁ × B₁

Model 2 · d₂ A₂ × B₂

Dimension-invariant Co-LoRA Same rank-space shape

Model 1 · d₁ A₁ P₁, Q₁ B₁

Model 2 · d₂ A₂ P₂, Q₂ B₂

Comparison of conventional LoRA and Co-LoRA. Conventional LoRA trains model-dependent A and B, while Co-LoRA freezes A and B and trains dimension-invariant P and Q. — **Figure 3.** Conventional LoRA trains model-dependent A and B. Co-LoRA freezes them and updates only the dimension-invariant P and Q. Open full size ↗

The core idea

Train what is compatible. Align what is not.

Co-LoRA inserts a dimension-invariant matrix P and vector Q between the original LoRA projections. The model-specific A and B stay frozen; only P and Q are optimized and exchanged.

h′ = B(PAh + Q)

Dimension-invariant sharing

P ∈ ℝ^r×r and Q ∈ ℝ^r depend only on the common LoRA rank r, not on model width.

Compatible initialization

Public samples align A in the shared rank space with MSE, then align width-dependent B through CCA.

Relative-depth matching

Models are split into the same number of blocks. Co-LoRA modules at corresponding relative depths are aggregated.

Where it fits. The full method, FedMosaic, uses Co-LoRA to share knowledge across heterogeneous models and is evaluated on the 40-task DRAKE benchmark .

Key results

Collaboration improves both personalization and transfer.

FedMosaic combines two complementary ideas:

RELATask-relevance-aware aggregation for data heterogeneity.
Co-LoRADimension-invariant adapter sharing for model heterogeneity.

Experiment

Table 2

Heterogeneous models

DRAKE-Dynamic · LLaVA-Llama 1B / 3B

Bars show mean A_last (%) in the method order reported by the paper. Error estimates and A_AUC are available in the full tables.

Adaptation curves on four unseen downstream tasks comparing FedMosaic with federated learning baselines. — **Figure 7.** Models aggregated with **FedMosaic** adapt faster on unseen DRAKE downstream tasks. Open full size ↗

Accuracy versus relative FLOPs for federated personalization methods. FedMosaic achieves higher accuracy with substantially fewer FLOPs than high-compute baselines. — **Figure 8.** Accuracy versus relative FLOPs on DRAKE-Dynamic. **FedMosaic** reaches a stronger operating point with a reported 2.6% gain and 60% fewer FLOPs than the highlighted comparison. Open full size ↗

67.86 / 51.16

Better personalization and transfer

Strong performance on both clients’ own data (Self) and other clients’ data (Others).

Llama ↔ Qwen

Sharing works across model families

Co-LoRA extends beyond size heterogeneity to cross-family collaboration.

1B ↔ 3B

Small and large models both benefit

Knowledge transfer remains useful across different hidden widths—not only between identical clients.

See the paper for confidence intervals, AUC metrics, full ablations, and additional settings.

Want the details?

Read the full paper.

Full formulations, alignment derivations, benchmark design, ablations, and additional results are available in the paper.

OpenReview PDF arXiv version View code on GitHub