Tencent OpenSearch-VL explained: the open-source recipe for multimodal AI search agents

How the agent searches with images

A standard vision-language model may describe an image or answer questions based on visible pixels. OpenSearch-VL is designed for a tool loop. The paper describes agents that can call web search, reverse image search, OCR, cropping, sharpening, super-resolution and perspective-correction tools ^[17].

That matters for search tasks because an image may contain partial or degraded evidence: small text, an object seen from the wrong angle, a cropped landmark or a visual clue that needs external confirmation. In the OpenSearch-VL setup, the model can decide what evidence is missing, apply a retrieval or image-processing tool, and fold the result into later reasoning steps ^[17].

The training recipe: SFT, RL and fault-aware tool use

OpenSearch-VL includes two trajectory datasets in the paper: SearchVL-SFT with 36,000 supervised fine-tuning trajectories and SearchVL-RL with 8,000 reinforcement-learning trajectories ^[17]. It also introduces Multi-round Fault-Aware GRPO, a training method intended for multi-step tool-use trajectories where intermediate actions can fail, partially help or require correction ^[17].

The emphasis on trajectories is important. A multimodal search agent does not only need to know what an image contains; it must learn when to search, when to transform an image, when to read text with OCR and when to stop gathering evidence. OpenSearch-VL packages those decisions as trainable examples rather than leaving the tool-use process implicit ^[17].

Reported performance, with an important caveat

The headline performance claim is strong: the paper reports an average improvement of more than 10 percentage points across seven multimodal deep-search benchmarks and says OpenSearch-VL is comparable to leading closed-source commercial models on some tasks ^[17].

That is not the same as proven product parity with OpenAI or Google systems. The provided evidence consists of the authors’ paper and launch coverage, not an independent reproduction or a public, like-for-like audit of production systems ^[1][17]. The current case for OpenSearch-VL is therefore best read as promising and technically useful, but still preliminary for real-world reliability, latency, safety behavior and long-horizon failure recovery.

How it compares with proprietary OpenAI and Google systems

For readers comparing OpenSearch-VL with proprietary systems from OpenAI and Google, the clearest confirmed difference is openness. OpenSearch-VL is presented as an open recipe and open-source training scheme, while the cited materials do not expose equivalent training stacks for those closed commercial products ^[17][21].

That makes OpenSearch-VL especially relevant for researchers and developers who want to inspect how multimodal search agents are trained, how tool-use trajectories are constructed and where multi-step visual reasoning fails. Its benchmark claims make it a serious open challenger, but the public evidence does not yet settle whether it matches closed systems in production settings ^[17].

What to watch next

The next meaningful tests are practical ones: whether outside researchers can reproduce the reported benchmark results, whether the recipe works across domains beyond the paper’s evaluation set, and whether agents trained this way handle tool errors reliably in longer real-world searches.

Until then, OpenSearch-VL’s biggest contribution is transparency. It gives the AI community a concrete open recipe for building multimodal search agents—and a clearer way to test whether open systems can close the gap with proprietary AI search products ^[17][18][21].