DetPO: In-Context Learning with Multi-Modal LLMs
for Few-Shot Object Detection

Abstract

Multi-Modal LLMs (MLLMs) demonstrate strong visual grounding capabilities on popular object detection benchmarks like OdinW-13 and RefCOCO. However, state-of-the-art models still struggle to generalize to out-of-distribution classes, tasks, and imaging modalities not typically found in their pre-training. While in-context prompting is a common strategy to improve performance across diverse tasks, we find that it often yields lower detection accuracy than prompting with class names alone. This suggests that current MLLMs cannot yet effectively leverage few-shot visual examples and rich textual descriptions for object detection.

Since frontier MLLMs are typically only accessible via APIs, and state-of-the-art open-weights models are prohibitively expensive to fine-tune on consumer-grade hardware, we instead explore black-box prompt optimization for few-shot object detection. To this end, we propose Detection Prompt Optimization (DetPO), a gradient-free test-time optimization approach that refines text-only prompts by maximizing detection accuracy on few-shot visual training examples while calibrating prediction confidence. Our proposed approach yields consistent improvements across generalist MLLMs on Roboflow20-VL and LVIS, outperforming prior black-box approaches by up to 9.7%.

We benchmark several state-of-the-art MLLMs and find that naively adding few-shot visual examples to the prompt consistently hurts detection accuracy. We posit this arises from rigid prompt structures used during post-training, making it difficult to exploit additional multi-modal contextual information during zero-shot inference.

Contrastive Prompt Refinement

Unlike traditional detectors trained primarily on true positive examples, we find that MLLMs achieve higher detection accuracy when provided with corner cases and negative examples, similar to how human annotators learn from examples of what not to annotate. DetPO iteratively refines class descriptions by identifying the worst false positive (highest confidence incorrect detection) and worst false negative (lowest-IoU missed detection), and uses these to instruct the model to explicitly exclude incorrect examples and include missed instances.

Confidence Score Estimation via VQA Score

Current MLLMs often overpredict bounding boxes and lack per-box confidence scores. We find that prompting models to self-report per-box scores improves detection accuracy. Optionally, we post-process detections with VQA Score: for each predicted bounding box overlaid on the test image, we ask "Is this bounding box an instance of class {CLS}?" and use the normalized probability of the Yes token as the final confidence score. This significantly down-weights false positives and improves mAP.

We evaluate on Roboflow20-VL (RF20-VL), a few-shot object detection benchmark spanning 20 diverse domains including aerial imagery, X-rays, medical imaging, and wildlife monitoring. Each dataset provides 10-shot training examples and rich annotation instructions per class.

We show how DetPO progressively refines class descriptions. The highlighted text indicates newly added discriminative details: