自适应路由与双阈值剪枝的多模态大模型检索增强感知

doi:10.3778/j.issn.1673-9418.2508035

摘要/Abstract

摘要： 检索增强感知算法能有效提升多模态大模型对高分辨率图像的感知能力，具有重要应用价值。但是现有算法存在检索时间过长问题，难以满足系统实时性需求。为此，提出一种融合自适应路由机制与双阈值剪枝搜索策略的多模态大模型检索增强感知算法，以优化处理效率。设置了自适应路由机制，通过计算整图任务可行性概率，并结合问题空间复杂度与模型规模自适应设定动态阈值，实现对简单样本的有效预筛选，使其无需分块处理即可直接获得答案，从而从源头规避无效计算。针对必须处理的复杂样本，在树搜索过程中采用双阈值剪枝的搜索策略：第一级剪枝基于语义质量评分的动态衰减约束，提前终止低质量分支的扩展；第二级剪枝则基于置信度评分差异，对通过第一级剪枝的节点，进一步合并那些决策稳定性高、置信度相近的冗余路径，从而有效抑制搜索空间的膨胀。实验结果表明，在V*Bench、HR-Bench等数据集上，该方案在保持感知精度（准确度仅损失2个百分点以内）的同时，实现了检索效率的显著提升，在LLaVA-ov-0.5B模型上检索速度最高提升达48.3%，尤其适用于低资源场景下的部署应用。

关键词: 多模态大模型, 检索增强感知, 自适应路由机制, 双阈值剪枝搜索策略

Abstract: The retrieval-augmented perception algorithm can effectively enhance the perception ability of multimodal large language models for high-resolution images and has important application value. However, existing algorithms suffer from long retrieval time, failing to meet the real-time requirements of the system. To address this problem, this paper proposes a retrieval-augmented perception algorithm for multimodal large language models, which integrates an adaptive routing mechanism and a dual-threshold pruning search strategy to optimize processing efficiency. Firstly, an adaptive routing mechanism is set up. By calculating the feasibility probability of the entire graph task and combining it with the problem space complexity and model size to dynamically set thresholds, it can effectively pre-screen simple samples, enabling them to obtain answers directly without block processing, thus avoiding invalid calculations from the source. Secondly, for complex samples that must be processed, a dual-threshold pruning search strategy is adopted during tree search: the first-level pruning is based on the dynamic attenuation constraint of semantic quality scores to terminate the expansion of low-quality branches in advance; the second-level pruning is based on the difference in confidence scores, and for nodes that have passed the first-level pruning, it further merges redundant paths with high decision stability and similar confidence, effectively suppressing the expansion of the search space. Experimental results show that on datasets such as V*Bench and HR-Bench, the proposed method significantly improves retrieval efficiency while maintaining perception accuracy (with an accuracy loss of no more than 2 percentage points). On the LLaVA-ov-0.5B model, the retrieval speed is increased by up to 48.3%, making it particularly suitable for deployment in low-resource scenarios.

Key words: multimodal large language models, retrieval-augmented perception, adaptive routing mechanism, dual-threshold pruning search strategy

徐国愚, 张一丹, 魏笑, 毛洋敏. 自适应路由与双阈值剪枝的多模态大模型检索增强感知[J]. 计算机科学与探索, 2025, 19(12): 3257-3266.

XU Guoyu, ZHANG Yidan, WEI Xiao, MAO Yangmin. Retrieval-Augmented Perception in Multimodal Large Language Models via Adaptive Routing and Dual-Threshold Pruning[J]. Journal of Frontiers of Computer Science and Technology, 2025, 19(12): 3257-3266.

参考文献

[1] DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16x16 words: transformers for image recognition at scale[EB/OL]. [2025-06-21]. https://arxiv.org/abs/2010.11929.
[2] ZHOU S J, ZHANG R Y, ZHU H S, et al. Multimodal LLMs as customized reward models for text-to-image generation[EB/OL]. [2025-08-01]. https://arxiv.org/abs/2507.21391.
[3] YAO H J, ZHANG R F, HUANG J X, et al. A survey on agentic multimodal large language models[EB/OL]. [2025-10-15]. https://arxiv.org/abs/2510.10991.
[4] YIN S K, FU C Y, ZHAO S R, et al. A survey on multimodal large language models[EB/OL]. [2025-06-27]. https://arxiv.org/abs/2306.13549.
[5] 秦小林, 古徐, 李弟诚, 等. 大语言模型综述与展望[J]. 计算机应用, 2025, 45(3): 685-696.
QIN X L, GU X, LI D C, et al. Survey and prospect of large language models[J]. Journal of Computer Applications, 2025, 45(3): 685-696.
[6] 吴信东, 黄满宗, 卜晨阳. BEKO: 大语言模型与知识图谱的双向增强[J]. 计算机学报, 2025, 48(7): 1572-1588.
WU X D, HUANG M Z, BU C Y. BEKO: bidirectional enhancement with a knowledge ocean for LLMs and KGs[J]. Chinese Journal of Computers, 2025, 48(7): 1572-1588.
[7] 任泽裕, 王振超, 柯尊旺, 等. 多模态数据融合综述[J]. 计算机工程与应用, 2021, 57(18): 49-64.
REN Z Y, WANG Z C, KE Z W, et al. Survey of multimodal data fusion[J]. Computer Engineering and Applications, 2021, 57(18): 49-64.
[8] 姜丽梅, 李秉龙. 面向图像文本的多模态处理方法综述[J]. 计算机应用研究, 2024, 41(5): 1281-1290.
JIANG L M, LI B L. Comprehensive review of multimodal processing methods for image-text[J]. Application Research of Computers, 2024, 41(5): 1281-1290.
[9] 杜佳俊, 兰红. 基于多模态特征融合的图像编辑模型[J]. 计算机科学与应用, 2024, 14(6): 164-176.
DU J J, LAN H. Image editing model based on multi-model feature fusion[J]. Computer Science and Application, 2024, 14(6): 164-176.
[10] BAI J Z, BAI S, YANG S S, et al. Qwen-VL: a versatile vision-language model for understanding, localization, text reading, and beyond[EB/OL]. [2025-06-27]. https://arxiv.org/abs/2308.12966.
[11] ABDIN M, ANEJA J, AWADALLA H, et al. Phi-3 technical report: a highly capable language model locally on your phone[EB/OL]. [2025-06-27]. https://arxiv.org/abs/2404.14219.
[12] 梁敏, 刘佳艺, 李杰. 融合迭代反馈与注意力机制的图像超分辨重建方法[J]. 计算机应用, 2023, 43(7): 2280-2287.
LIANG M, LIU J Y, LI J. Image super-resolution reconstruction method based on iterative feedback and attention mechanism[J]. Journal of Computer Applications, 2023, 43(7): 2280-2287.
[13] LIU H, LI C, LI Y, et al. LLaVA-NeXT: improved reasoning, OCR, and world knowledge[EB/OL]. [2025-07-22]. https://llava-vl.github.io/blog/2024-01-30-llava-next/.
[14] CHEN Z, WANG W Y, TIAN H, et al. How far are we to GPT-4V? Closing the gap to commercial multimodal models with open-source suites[EB/OL]. [2025-07-31]. https://arxiv.org/abs/2404.16821.
[15] LUO G, ZHOU Y Y, ZHANG Y X, et al. Feast your eyes: mixture-of-resolution adaptation for multimodal large language models[EB/OL]. [2025-07-31]. https://arxiv.org/abs/2403.03003.
[16] GE C J, CHENG S J, WANG Z M, et al. ConvLLaVA: hierarchical backbones as visual encoder for large multimodal models[EB/OL]. [2025-07-31]. https://arxiv.org/abs/2405.15738.
[17] KIRILLOV A, MINTUN E, RAVI N, et al. Segment anything[C]//Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision. Piscataway: IEEE, 2023: 3992-4003.
[18] WANG W, DING L, ZENG M, et al. Divide, conquer and combine: a trainingfree framework for high-resolution image perception in multimodal large language models[C]//Proceedings of the 39th AAAI Conference on Artificial Intelligence and the 37th Conference on Innovative Applications of Artificial Intelligence and the 15th Symposium on Educational Advances in Artificial Intelligence. Palo Alto: AAAI, 2025.
[19] SHEN H, ZHAO K, ZHAO, et al. ZoomEye: enhancing multimodal LLMs with human-like zooming capabilities through tree-based image exploration[C]//Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. Stroudsburg: ACL, 2025.
[20] WANG W B, JING Y C, DING L, et al. Retrieval-augmented perception: high-resolution image perception meets visual RAG[EB/OL]. [2025-07-22]. https://arxiv.org/abs/2503.01222.
[21] YU S, TANG C Y, XU B K, et al. VisRAG: vision-based retrieval-augmented generation on multi-modality documents[EB/OL]. [2025-07-28]. https://arxiv.org/abs/2410.10594.
[22] WU P H, XIE S N. V*: guided visual search as a core mechanism in multimodal LLMs[C]//Proceedings of the 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2024: 13084-13094.
[23] GU S H, LUGMAYR A, DANELLJAN M, et al. DIV8K: diverse 8K resolution image dataset[C]//Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision. Piscataway: IEEE, 2020: 3512-3516.
[24] LI B, ZHANG Y H, GUO D, et al. LLaVA-OneVision: easy visual task transfer[EB/OL]. [2025-08-12]. https://arxiv.org/abs/2408.03326.
[25] LIU H T, LI C Y, LI Y H, et al. Improved baselines with visual instruction tuning[C]//Proceedings of the 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2024: 26286-26296.
[26] 陆春霞, 马少辉. 基于网格搜索的船体不规则分段动态堆放方法[J]. 计算机应用, 2013, 33(2): 333-337.
LU C X, MA S H. Approach on irregular block dynamic stacking in shipbuilding based on grid technology[J]. Journal of Computer Applications, 2013, 33(2): 333-337.