Yedi Hu
EMNLP 2025 Findings

Explore Model Kinship For Merging Large Language Models

Yedi Hu, Yunzhi Yao, Ningyu Zhang, Shumin Deng, Huajun Chen

Model kinship connects biological evolution and model evolution, helping predict merge gains and steer continual model merging away from local optima.

Read on ArXiv Code Poster Slides Notebook LM
Featured figure for the paper

Abstract

Model merging has become one of the key technologies for enhancing the capabilities and efficiency of Large Language Models (LLMs). The open-source community has driven model evolution by iteratively merging existing models. However, a principled understanding of the expected gains and underlying factors in model merging remains lacking.

In this work, we examine model evolution through continual merging, analogous to biological evolution, and introduce the concept of model kinship, the degree of similarity or relatedness between LLMs. With comprehensive empirical analysis, we find that there is a certain relationship between model kinship and the performance gains after model merging, which can help guide our selection of candidate models.

Inspired by this, we propose Top-k Greedy Merging with Model Kinship, which yields better performance on benchmark datasets. Using model kinship as a criterion helps continuously perform model merging while alleviating degradation caused by local optima.

NotebookLM Speech

This speech is generated by NotebookLM, and some points may not be explained clearly.

Method

Motivated by the parallel between biological reproduction and model evolution, the paper extends the notion of kinship to describe relationships between language models through their weight-space evolution. Building on ideas related to task arithmetic, model kinship is used as a metric for assessing similarity based on changes in weights during model evolution.

Method figure

Results

The study presents both statistical analysis of community model-merging evolution and main experiments that reproduce optimization issues caused by greedy iterative merging. The paper argues that kinship correlates with merge outcomes and that increasing kinship among top models may indicate convergence toward local optima.

By integrating a kinship-based exploration process into Top-k Greedy Merging, the method searches beyond high-kinship candidates and improves the chance of discovering better merged models.

Results figure

Reference

Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. ICLR 2023.