I'm a QA engineer with 10+ years of automation experience, and I've been using real operational systems I have hands-on access to as test subjects to stay sharp with modern QA tooling: Playwright, TypeScript, CI/CD, and now AI agents.

This experiment started with this question, can an AI explore a booking website the way a real user would, and find problems my automated tests can't?

Why I built this

My Playwright test suite already verifies the booking journey

  • Does the booking button work?
  • Does navigation behave correctly?
  • Do forms validate?

Those are important tests, but they don't answer the question that matters to the business, can a first-time visitor understand the website well enough to make a booking?

Instead of writing a scripted journey, I gave an AI agent a goal to "Visit this site as a first-time guest and decide whether you'd book." The agent explored the website on its own, reading content, comparing pages, following links, and reporting what it found confusing or inconsistent.

The technical approach

A few design decisions made this experiment practical.

Playwright MCP instead of scripted automation

I already use Playwright for my automation framework, so it became the browser layer for the agent. Rather than generating Playwright code, the agent interacts through Playwright MCP, giving it a clean interface to navigate, inspect, and interact with pages. A typical run completes in around 11 browser interactions, making it both fast and inexpensive.

Claude Sonnet instead of a larger model

This task isn't solving complex reasoning problems, it's exploring a small marketing website. One lesson I've learned with AI engineering is to use the cheapest model that reliably solves the problem.

Manual execution

This isn't something I'd run on every commit, the Playwright suite remains the fast safety net during development. The agentic test runs manually after significant content or UX changes, when I want an independent review of the user experience.

What the agent found

One short run uncovered several issues that my Playwright suite would never detect.

  • Room names on the marketing website didn't match those in the booking engine.
  • A minimum stay policy was present but written ambiguously.
  • Clicking Book Now unexpectedly redirected guests to an external booking site without warning.
  • Room pricing appeared inconsistent between different parts of the booking journey.

These were user experience problems that could easily reduce booking confidence.

The biggest lesson

The most interesting part wasn't what the agent found, it was what happened when it misunderstood my instructions. At one point I asked it to investigate the booking flow. I meant, "Observe what happens," the agent interpreted that as, "Complete an actual booking."

It opened the booking engine, selected dates, adjusted guest numbers, and kept exploring until it exhausted its limit, with no useful report.

That failure taught me two lessons about building AI agents. First, scope matters. I rewrote the prompt to explicitly tell the agent to observe the booking flow rather than complete a reservation. Second, I stopped relying on instructions given only at the beginning of the conversation. Instead, just before the interaction limit is reached, the framework injects another instruction telling the agent to stop exploring and produce its report.

That small change improved reliability, the same test now completes in 11 interactions that cost $0.50.

What it costs

The experiment turned out to be more affordable than expected.

  • Successful run: ~$0.50
  • Poorly scoped run: ~$1.25
  • Several manual runs each month: under $5

What I'd build next

This prototype currently evaluates the site as a first-time guest. I'd like to expand it with additional personas, including,

  • Budget traveller
  • Family traveller
  • Digital nomad
  • Mount Kenya climber
  • Price consistency validator

Each persona asks different questions and should uncover different usability issues without requiring additional Playwright scripts.

Final thoughts

Traditional automation verifies whether software behaves correctly. Agentic testing explores whether the experience makes sense to a user, both of which complement each other.

In summary, building good agentic tests is about defining clear goals, controlling scope, managing cost, and designing guardrails so the agent knows when to stop exploring and start reporting.