The Limits of Text-Only Agents
Text-only agents are severely limited when dealing with UI testing and autonomous web browsing. Giving an LLM the raw HTML DOM tree is inefficient and often strips away crucial visual layout context (e.g., "Is this button hidden behind a modal?").
We built a multimodal agent that takes a screenshot of the webpage via Playwright, parses the DOM tree, and uses a Vision-Language Model (VLM) from Hugging Face (specifically a fine-tuned variant of Qwen-VL) to decide where to click next.
The Set-of-Mark Technique
The hardest part wasn't the reasoning model—it was the coordinate mapping. VLMs are notoriously bad at outputting exact (x, y) pixel coordinates. We implemented the "Set-of-Mark" prompting technique.
from playwright.async_api import async_playwright
import cv2
async def annotate_page(page):
# Extract interactive elements
elements = await page.evaluate('''() => {
return Array.from(document.querySelectorAll('a, button, input')).map(el => {
let rect = el.getBoundingClientRect();
return {x: rect.x, y: rect.y, w: rect.width, h: rect.height};
});
}''')
# Draw numbered bounding boxes on screenshot using OpenCV before sending to VLM
# ...We overlay numbered, brightly colored bounding boxes on the screenshot before feeding it to the VLM. The prompt simply asks the model: "Output the ID number of the element you want to interact with." This bypasses the coordinate regression problem entirely and results in 99% click accuracy.