GuidesAI and agents

An AI agent that drives a browser: useful, slow and to be kept on a leash

By the Banzai Labs teamUpdated on 2 min read

OrderAgentclickscreenshotSee the page, pick an action, repeat: slow but universal
The agent looks at a screenshot of the page, chooses where to click, then repeats.

The loop

  1. 1LookScreenshot or page structure.
  2. 2DecideThe model picks the next action.
  3. 3ActClick, type, scroll.
  4. 4CheckNew screenshot, and so on.

Browser or API

Two ways to act on a service
API
  • Fast and precise
  • Few tokens
  • Only if the service has one
Driven browser
  • Works on almost any site
  • Slow, one screenshot per step
  • Breaks when pages change

Safety rules

Rule Why
Isolated browser No access to your personal accounts
Allowlist of sites The agent does not wander off
Confirmation before purchase or sending A booby-trapped page does not decide
Step budget A stuck loop stops
Screenshots recorded Understand what happened

The basics: the AI agent loop. The risks: prompt injection.

Frequently asked questions

How does the agent see the page?
Through screenshots, through the page structure (accessibility tree, DOM), or both. The model then decides the action: click, type, scroll.
Is it reliable?
Less than an API. A pop-up, a button that moves or a captcha can block the agent. Plan for retries and a check of the result.
What are the risks?
A page can contain hidden instructions that hijack the agent. Use it in an isolated browser, with no personal sessions logged in, and with confirmation before any purchase or message.

Read next

All guides