🌟 Parameter-Efficient Finetuning (PEFT) of Vision-Language Models (VLMs) for resource-constrained Federated Learning.
🌟 Uses principal eigenvalue of layerwise Neural Tangent Kernels (NTKs) as client-specific layer importance score.
🌟 Distributed layer participation: Encourages diverse layer selection across clients for optimal performance.
🌟 Joint optimization: Uses multi-objective meta-heuristics on the server to optimize both importance and diversity.
🌟 MedVQA-FL Dataset: 707,962 VQA triplets, 9 modality-specific clients, 12 anatomical categories.
🌟 Extensive Experiments: 10,000+ client-level runs, 6 FL settings, 58 datasets, 4 VLMs.
(1) Resource Constraints: Fine-tuning foundation models exceeds client resources.
(2) Layer Selection: Naive selection overfits and ignores client-specific needs.
(3) Competing Goals: Balancing client adaptation and collaborative layer usage is hard.
Client Layer Importance: NTK spectral analysis scores each layer per client.
Server-Side Layer Balancing: Meta-heuristics maximize adaptation and distribute layer updates.
Privacy-Preserving: Optimization happens without sharing raw data.
 (2).png)
(a) Vanilla: selects layers only on local client data, neglecting collaboration.
(b) F3OCUS: jointly maximizes client-specific layer importance and distributes layer adaptation across clients.
We use the Layerwise Neural Tangent Kernel (LNTK) to score which layers should be adapted for each client. The LNTK captures how sensitive the model’s output is to each layer's parameters on a client’s data.
For neural network function \(f\) trained on client data \(\mathcal{X}\):
$$ \dot{f} = -\eta \Theta(\mathcal{X}, \mathcal{X}) \nabla_{f} F (\mathcal{X}; \theta_t) $$
The NTK matrix:
$$ \Theta(\mathcal{X}, \mathcal{X}) = \nabla_{\theta} f(\mathcal{X}; \theta_t) \nabla_{\theta} f(\mathcal{X}; \theta_t)^T $$
...can be split into per-layer (LNTK) components:
$$ \Theta(\mathcal{X}, \mathcal{X}) = \sum_{l=1}^L \Theta^l(\mathcal{X}, \mathcal{X}) $$ $$ \Theta^l(\mathcal{X}, \mathcal{X}) = \nabla_{\theta^l} f (\mathcal{X}; \theta^l) \nabla_{\theta^l} f(\mathcal{X}; \theta^l)^T $$
The principal eigenvalue \(\lambda^l_1\) of each layer’s LNTK determines how quickly that layer learns and aligns with the client’s data.
 (2).png)
We define the importance score for layer \(l\) on client \(i\) as:
$$ S^l_i = \frac{\lambda^l_{i, 1}}{\sum_{k=1}^L \lambda^k_{i, 1}} $$
This enables selective, efficient adaptation of the most relevant layers for each client.
 (2).png)
On the server, we refine layer selection to jointly maximize adaptation (via client-specific importance) and balance layer usage across clients. We formalize this as a multi-objective problem:
Objectives:
Constraint: Each client can only select up to their own budget of layers.
$$ \begin{cases} \mathbf{max} & \sum\limits_{i=1}^{N} \sum\limits_{l=1}^L S_{i}^{l} \\ \mathbf{min} & \frac{1}{L} \sum\limits_{l=1}^{L} ( n_l - \bar{n} )^2 \\ \text{s.t.} & \sum_{l \in \mathcal{L}_i} m_{i}^{l} \leq L_{\text{i,max}}, \ \forall i \end{cases} $$
 (1).png)
Why Meta-heuristics? Traditional methods struggle as the search space is huge and no client data is available on the server. We use five representative meta-heuristic algorithms to find a set of optimal trade-offs:
 (2).png)
Ultra-MedVQA is a large-scale, federated medical VQA benchmark featuring 707,962 image–QA triplets across 9 imaging modalities and 12 anatomical categories, distributed as 9 simulated clients. It maximizes real medical image utility and supports robust federated VQA evaluation.
Ultra-MedVQA provides a challenging, multi-modal federated VQA setting with fine-grained evaluation across anatomical regions (Colon, Lung, Skin, Eye, Breast, Kidney, Blood, Femur, Heart, Liver, Pancreas, Spleen) and modalities.
We evaluate F3OCUS for selective fine-tuning of layers and adapters across four vision-language models (ViLT, ALBEF, Llava-1.5, BLIP-2) and multiple federated learning settings:
Device Heterogeneity: We vary the number of trainable layers per client/task to simulate diverse resource budgets.
 (2).png)
 (1).png)
Explanation: The t-SNE plots above visualize the learned feature spaces for eight different medical imaging modalities. Compared to Federated Dropout and the standard approach of fine-tuning the last K layers, F3OCUS produces more distinct and well-separated feature clusters, demonstrating improved class discriminability and representation quality across clients. This highlights the effectiveness of our selective layer adaptation strategy for heterogeneous federated learning.
Explanation: The table above summarizes the accuracy (Tasks 1-3) and F1-scores (Tasks 4-6) across different federated learning settings, including domain gap, modality gap, label shift, and task heterogeneity. We benchmark our method, F3OCUS, against a wide range of state-of-the-art baselines including pruning, layer selection, and meta-heuristic approaches across ViLT and BLIP models.
F3OCUS consistently outperforms all competing methods in every task and model setting, demonstrating superior adaptation and collaboration in heterogeneous federated scenarios. Notably, our approach achieves higher scores with fewer trainable parameters, highlighting its efficiency for real-world resource-constrained federated learning. The reduction in computation and communication needs is shown in the Table below for Task 1:
Explanation: The table above presents a comprehensive comparison between F3OCUS and 5 leading PEFT baselines: LayerNorm Tuning (LN), LoRA, Bias Tuning, Prompt Tuning, and FedDAT. We evaluate each method in terms of communication overhead, computational requirements, parameter efficiency, and accuracy on all clients.
F3OCUS consistently surpasses all PEFT baselines except adapters and FedDAT. Notably, FedDAT and adapter-based methods fine-tune all adapters in each client, while F3OCUS selectively adapts only 4 adapter layers per client. This leads to a substantial reduction in communication cost (down to 9.7 MBits) and computation (80.6 GFLOPs), while maintaining or exceeding performance across clients.
Conclusion: These results highlight the efficiency and scalability of F3OCUS for real-world federated learning scenarios, where client resource constraints are critical.
 (1).png)
@inproceedings{saha2025f3ocus,
title={F3OCUS: Federated Finetuning of Vision-Language Foundation Models with Optimal Client Layer Updating Strategy via Multi-objective Meta-Heuristics},
author={Pramit Saha and Felix Wagner and Divyanshu Mishra and Can Peng and Anshul Thakur and David A. Clifton and Konstantinos Kamnitsas and J. Alison Noble},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
year={2025}
}