Depth Selective Adapter Allocation for Polarity Robust Medical Vision Language Inference

• Nader Trabelsi

  Institut Supérieur d'Informatique de Mahdia, Département des Sciences Informatiques, Route de Rejiche, Mahdia 5121, Tunisia

  Author

• Yassine Jebali

  École Supérieure des Sciences et de la Technologie de Hammam Sousse, Département des Technologies de l’Information et de la Communication, Avenue de la République, Hammam Sousse 4011, Tunisia

  Author

Medical visual inference models often receive questions that contain polarity cues such as absence, presence, exclusion, or uncertainty. These cues can be short, syntactically fragile, and clinically important, because a mistaken polarity interpretation can reverse the intended answer. This study evaluates whether adapter parameters should be distributed uniformly across a multimodal language backbone or assigned according to measurable layer-level sensitivity to polarity transformations. We introduce a depth-selective adapter allocation strategy for binary medical vision-language inference and test it on 18,720 image-question pairs derived from radiography, ultrasound, and dermoscopy tasks. The evaluation uses paired affirmative, negative, exclusion, and uncertainty-framed prompts with identical image labels. We estimate layerwise polarity transport through Jacobian energy, centered kernel alignment, and signed margin propagation, then allocate low-rank adapter capacity using a constrained optimization rule. Compared with uniform adapter placement, final-block placement, and random layer allocation, the proposed allocation reduces polarity reversal errors from 10.8% to 4.9% while preserving mean task accuracy within 0.4 percentage points. It also improves negative predictive calibration in exclusion-framed prompts and reduces the variance of signed margin changes across prompt forms. Ablation experiments indicate that adapter rank distribution matters more than total trainable parameter count once a minimum capacity threshold is reached. These findings suggest that layer-sensitive parameter allocation can improve robustness to polarity-bearing clinical wording without increasing model size.

• PDF

Copyright (c) 2026 International Journal of Science, Engineering and Applied Technologies

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.