Time: an afternoon for the sample run, and a day to set it up properly for your own app. Cost: nothing beyond the AI assistant you already pay for. Difficulty: you need to be comfortable starting a program from a terminal. You don't need to write any code.
On Saturday 10 October 2026, eighteen AI assistants played my browser game at the same time, as brand-new players, for three hours. They found the bugs I'd have needed a room full of beta testers to find, argued with each other about the game's economy, and filed reports that were more thorough than most people's. Their findings were in a live patch the next afternoon. I've written up the story of that run separately. This is the how-to.
The method works for any web app with a front end, not just games. To prove that, and to give you something to practise on, I built a small kit and a deliberately broken sample shop, and ran three AI testers against it on Sunday 11 October. Everything below comes from that run or from the Saturday one, and I've said which.
At the end of this you will have
- A harness: a small program that holds a real browser open and lets an AI drive it with plain commands, seeing only what a person would see.
- A brief that turns a general-purpose AI assistant into a blind first-time user with a character, a journal and a report to write.
- A shared chat the testers talk in, which turns out to be the best dashboard you'll have.
- A way to start them all at once, a way to check the harness works before anyone starts, and a checklist for turning their reports into a list your builder can act on.
- A sample app, Corner Shop, with five planted problems, so you can see what a crowd finds before you point one at something you care about.
Why a crowd, and why through the screen
I needed two things from Saturday's test, and the reasons are the whole method.
Volume. A lot of software only misbehaves with several people using it at once. In a trading game it's obvious: one trader can't crash a market. In a shop it's the last item in stock; in a booking system it's two people wanting the same slot. Billy, the agent in my fleet who tests our apps, could realistically run two players at once. I needed a lot more than two, and I wasn't willing to make up the numbers by bothering real beta users.
The screen. I didn't want to know whether the server returned the right numbers. I have tests for that. I wanted to know what worked and what didn't as a person saw it. Billy put it better than I can: in a game, the screen is the truth. Most bugs that hurt people live in the gap between what happened and what they were told. A request that returns 200 tells you nothing about a button that went grey without saying why.
So the testers drive real browsers, and they're not allowed to look behind the curtain: no source code, no database, no admin screens. If you'd have to be a developer to know it, they don't know it.
Before you start
Have
- An AI assistant that can run terminal commands. I use Claude Code. Anything that can run
curland write a file will do. Each tester is one assistant session. - Node.js 18 or later on the machine that will hold the browsers. That machine needs enough memory for one headless browser per tester. On Saturday a separate box held 38 browsers and spent most of the afternoon two-thirds idle; three testers on a laptop is nothing.
- The app, running somewhere the browsers can reach, and a test account per tester if it needs logins. Use a copy, not production. A crowd changes things.
Decide
- Who your users are. On Saturday it was the game's five playstyles. For a shop it might be a careful buyer, a bulk buyer and a first-timer. Each tester gets one, and it changes how they use the app, which is where the breadth comes from.
- Who is foreman. Someone has to write the brief, start everyone, watch, and collate. On Saturday it was Billy. In a small run it's you. And the foreman is not a tester. On Saturday Billy also flew two "control" players while foremanning, and between 14:56 and 15:45 he was too busy to touch them, so the control told us nothing. One job each.
- Who you won't use. I have about two dozen agents and only eighteen flew. Some of mine exist for one very specific job and I won't fill their memories with something that has nothing to do with it. One, the Geordie, was needed to watch the server instead. If your assistant keeps memory between sessions, think about what a three-hour game does to it.
Step 1: Get the kit
Download it: a-crowd-of-testers.zip (32KB). It's five small files, the sample app, and the journals, reports and chat from the run described below, so you can see what good output looks like. The two files that matter most are printed in full at the end of this guide.
a-crowd-of-testers/
harness.js one tester's browsers, driven by plain HTTP
launch.sh start one harness per tester, check each answers; ./launch.sh stop
selftest.sh prove every harness can see the app before anyone starts
BRIEF.md the tester brief, with {BLANKS} to fill in
COLLATE.md the foreman's checklist for turning reports into a list
sample-app/
server.js Corner Shop: one file, no dependencies
ANSWERS.md the five planted problems. Don't show your testers.
Unzip it, and in the kit folder:
npm install
npx playwright install chromium
Keep the version pinned. The kit pins Playwright 1.58.2. Playwright needs the exact browser build it was made for. When I installed the latest version on 11 October it went looking for a browser build I didn't have, and every single command came back no browser called p1. Either pin a version and install its browser, as above, or upgrade both together.
The two .sh scripts need a bash shell. On a Mac or Linux that's your terminal. On Windows, use Git Bash, and run them as bash launch.sh … if unzipping has lost their permission to run.
What you should see: npx playwright install chromium ends without an error, and node -e "require('playwright')" prints nothing.
Step 2: Start the app
node sample-app/server.js
What you should see: Corner Shop on http://localhost:8080. Open that in your own browser and have a look: four items, a name box, a Buy button per row. Don't go looking for the problems. You'll spoil the fun of watching someone else find them.
For your own app, start it the way you normally would, on a copy of its data.
Step 3: Drive the harness yourself first
Before you hand the controls to anyone else, use them. Start one harness:
TZ_NAME=Europe/London node harness.js me 7501 http://localhost:8080 ./run-practice
Then, in another terminal, talk to it:
curl -s localhost:7501 -d 'p1 open'
curl -s localhost:7501 -d 'p1 fill #name|Me'
curl -s localhost:7501 -d 'p1 press #signin'
curl -s localhost:7501 -d 'p1 press [data-buy="tea"]'
What you should see (this is my own run, with the current kit):
opened p1 at http://localhost:8080
filled #name
the app said: 09:00:11 Welcome, Me
the app said: 09:00:11 Bought 1 for 2
That last word matters. The harness clicks once, then waits up to ten seconds for the app to say something: a message, a toast, an error. It reports what was said, with the time. If the app says nothing, the harness says so in capitals:
NOTHING: no new message within 10 s. The button is now DISABLED, with no reason shown. The app may still answer: wait, then run events.
That one line is the most valuable thing the harness does. A tester that only checks whether a click "worked" will never notice a button that worked eventually and silently. Saturday's worst bug was exactly that, and Billy's first harness reported it as "CONFIRMED", because the click had happened. Clicked is not the same as done.
NOTHING means nothing yet. On Corner Shop a slow purchase can answer fifteen seconds later, after the harness has given up, and the only place you'll see "Bought 5 for 25" is events. So the brief tells testers: after NOTHING, wait, run events, and only then decide.
Three more rules are built in. Pressing the same button again while it's still waiting, or within a second and a half of it finishing, is refused, because a double-click is not a decision. A button that's disabled or hidden isn't clicked at all, and the harness says which, so a busy button doesn't get reported as a broken one. And there is no command to run code in the page or jump to an arbitrary address. A tester who can't cheat has to report what a person would see.
Step 4: Write the brief
Copy BRIEF.md and fill in the blanks. Most of it is the controls table. The parts that make the method work are short:
- You are a brand-new user. What confuses you is the data. Nobody wants a polite report.
- Through the screen only. No source, no docs folder, no database, no logs.
- A character. One line, and it should change behaviour: "a retired bookkeeper who checks every price against what she's charged" uses a shop differently from "a busy parent buying in bulk who hates waiting". On Saturday each of my agents also brought its own personality, and I told them to talk like themselves. That's a lot of why the reports disagreed with each other, which is what you want.
- A journal every ten actions, in a fixed shape: doing, learned (and how), stuck, next.
- SEEN or HEARD. Anything learned from chat gets marked as learned from chat. Otherwise one tester's sighting, repeated by four others, arrives on your desk as five.
- Other people's accounts. Decide before the run whether a tester may touch another tester's account, and say so. In the sample run below, one tester found she could see another's account, rightly pressed nothing, and reported it. Leave it unsaid and eventually someone won't be so careful.
Then check every fact the brief states about your app, not just the selectors. A fact in the brief is a test case the testers will run whether you meant it or not. Billy's Saturday brief told the players that the Max buttons "disarm in about six seconds". They don't, and the players who believed him reported a game bug that was really a brief bug.
Step 5: Launch the harnesses and prove they work
TZ_NAME=Europe/London ./launch.sh http://localhost:8080 ./run-1 7511 ada bert cleo
./selftest.sh "Corner Shop" 7511 7512 7513
Set TZ_NAME to the time zone the people reading the logs are in. It puts the browsers' clocks and the harness's timestamps in the same zone, which matters as soon as the browsers run on a server somewhere else (more on that below).
What you should see:
ada -> port 7511
bert -> port 7512
cleo -> port 7513
ready: 7511
ready: 7512
ready: 7513
and then OK for every port from the self-test. Don't skip the self-test. A harness that can't see the app fails in the first minute, and the tester writes it up as a bug in your app.
Three things that trip people up the first time. On Windows, the firewall asks whether to allow each harness the first time it starts listening, so be at the keyboard to click Allow, or the launch just hangs. Keep the commands you give testers plain ASCII, because accented letters and symbols like "·" can arrive mangled from Windows. And reset the app between runs. Corner Shop keeps its stock in memory, so restart it. For your own app, re-seed the test accounts.
On Saturday Billy ran a self-test of his harness before the run, and it still had faults that only a crowd found. More on those below.
Step 6: Start everyone at once
"At once" matters. Testers who start ten minutes apart are a queue, not a crowd.
How I did it on Saturday. Each agent had already been sent the brief and its assignment by Billy, and was sitting in its own session, waiting. Scotty, who looks after my infrastructure, had written a small plugin for Claude Code: when an agent sends another an urgent message through our fleet's internal mail, the plugin also pushes it straight into the recipient's running session, if they have one, so they act on it immediately instead of finding it later. The mail is always stored first, so a push that fails never loses the message; it just waits in the inbox like any other. Billy started the run by sending each tester an urgent mail that said GO. Eighteen of them went out between 14:12 and 14:19, in waves of four, five and nine, each wave only after Scotty had confirmed the servers were comfortable with the last. Scotty's go-aheads were counted in players, not testers ("OK 10", "OK 20", "OK 38"), because most testers flew two. One of the eighteen GOs went to Scotty himself, who was flying as well as minding the servers. The extensions and the final "TIME" went out the same way. I never had to tell a single one of them to start. The night before, Billy, the Geordie and I had been using the same plugin to work on the release together in real time.
How you can do it without a fleet. Two ways, both of which I've used:
- One session per tester. Open a terminal per tester, start your assistant in each, and paste that tester's filled-in brief as its first message. Paste them as close together as you can. With three or four testers this is fine.
- One assistant, many sub-agents. Some assistants can start sub-agents that work in parallel. Claude Code can. Give your assistant the filled-in briefs and ask it to start one sub-agent per tester, all at once. That's how I ran the 11 October sample. One instruction, three testers, launched in the same second.
Step 7: Watch the chat
Every tester was told to read and post in a shared chat every few actions. The harness keeps it in one file, run-1/chat.log, and any tester can read it with chat and post with say.
This was the surprise of Saturday. I'd expected to spend the afternoon typing into twenty terminals asking how it was going. Instead I sat with galaxy chat open and could see, almost in real time, what everyone was doing: personalities first, then tips, then banter, then a lot of "ha, beat you". When one of them found something, the others knew within minutes, and I could see them testing it. It went so smoothly that at one point I went shopping with Lynne for forty-five minutes.
Be honest with yourself about one thing, though. Real users don't usually behave like this unless they're in a guild or a team. Strangers don't share their best discovery with everyone ten minutes after finding it. A crowd of AIs is more cooperative than a crowd of people. That's useful for coverage, because they spread findings and then test each other's, and misleading for anything social, like how fast a market gets crowded out. Keep that in mind when you read the economics.
Step 8: Collate
When time's up, stop the harnesses with ./launch.sh stop, and check it says every port is down. On Saturday the first version of Billy's stop command said "stopped" for every port and did close the browsers, but left all nineteen harness servers behind them still listening, and Scotty had to kill them by hand. The version in the kit stops each harness by its process number and then checks the port actually closed.
Then work through COLLATE.md. The short version:
- Count sightings, not mentions. SEEN counts. HEARD doesn't.
- Reproduce the top findings yourself, with the same harness, before you hand them over, and write down when you did it. Do it on a quiet machine, and date log lines by their timestamps, never by where they sit in the file.
- Say what you proved and what you only think. What happened on screen can usually be proved. Why it happened usually can't, not from the screen.
- Separate bugs from design questions. "The quote said one thing and I was charged another" is a bug. "Loyalty points should depend on price" is a decision for whoever owns the app. Neither gets dropped.
- Keep the disagreements, and correct yourself in writing.
That last point earned its place on Saturday. After the run, Billy reproduced the slow-button bug, timed it at 36 seconds, and blamed contention between the game's tick and its trade locks. The Geordie checked the server's own timestamped log and showed that Billy's reproduction had run while the Geordie's automated test suite was holding the database, so the 36 seconds was the test suite, not the game. Billy had also dated some deadlock retries by where they sat in the log rather than by their timestamps. He struck his own explanation through in the report and wrote "My error". The button's behaviour was real: any slow trade greyed it out with no reason and no time limit, and the trade then landed silently later. Why trades were slow on the day is still unproven, and the report says so. Because he left the correction visible, nobody went chasing it.
What three testers found on 11 October
Here's the sample run, measured. Three testers, each a sub-agent of one assistant session, were started in the same second at 16:54:42 against a freshly started Corner Shop. Each had the brief above and a character:
- Ada, a retired bookkeeper who checks every price against what she's charged.
- Bert, a busy parent buying in bulk for a school fete, who hates waiting.
- Cleo, fifteen, who has never shopped online, presses everything, and wants the loyalty points.
The last of them signed off at 17:02:56. Between them they sent 407 commands, took 88 screenshots, posted 11 chat messages, and wrote about 5,600 words of journals and reports. None of them knew there were planted problems.
| Planted problem | Ada | Bert | Cleo |
|---|---|---|---|
| Buy greys out for 9–15 s with no message | SEEN | SEEN | SEEN |
| Hidden 10% charge on 10 or more | heard | SEEN, and found the cut-off | SEEN |
| Invisible "thanks" layer swallows clicks | SEEN | SEEN | SEEN |
| First Cancel does nothing | heard, then SEEN | heard, then SEEN | SEEN first |
| Loyalty points: one per item, whatever it costs | listed it as working well | a problem | a problem |
All five were found, and every planted bug was seen first-hand by at least two testers. Bert pinned down the hidden charge the way a person would: nine teas cost exactly the quote, ten cost exactly ten per cent more, and he posted it in chat. "Buying 9 twice is cheaper than 18 once."
Then they found things I hadn't planted, which is the real argument for the method:
- There are no passwords. Cleo typed Bert's name into her second browser and could see his cash and his orders, with working Cancel buttons. She pressed nothing and reported it. Bert then confirmed it on his own account. I'd written the shop with a name box and no login because it was a toy. A crowd treats it as a shop.
- Names are case-sensitive, and every new name gets 100 free. "cleo" and "Cleo" are different customers. "Free money glitch?" asked Cleo.
- Half a jar of jam gave three different answers. All three of them tried a quantity of 1.5 and got a quote for 1.5, a message saying "Bought 1.5", and an order line saying "1 × jam".
- The crowd moved the price. Ada was quoted 14.35 for seven teas and charged 17.15. That's below the ten-item line, so it wasn't the planted charge. While her button sat grey, Bert's twenty teas emptied the shelf, and the price rose under her. She worked that out herself: it only happened "while Bert was buying 20 tea", and when nobody else was buying the same thing she paid the quote. Saturday's worst money bug in the game had the same shape, and you can't see it with one tester.
- Smaller ones: a negative quantity shows a quote of −10, signing in with an empty name does nothing at all, and an open page never refreshes its prices.
They also disagreed, which is the part I care about. Ada listed the loyalty points under "what worked well". Cleo, whose character was there for the points, called them "a number that goes up and does nothing", and pointed out that a kettle earns the same as a tea. Bert and Ada each found an overcharge and each gave a different cause: his was the bulk charge, hers was the crowd. Both were right. Two causes produced the same symptom, and a single tester would have found one and stopped.
And one finding was wrong. Cleo reported that "the buttons only work if you use lowercase, like tea not Tea". That's not the shop. That's the selectors in my brief, which are case-sensitive. A tester can't tell a harness fault from an app fault, and that's exactly why the foreman reads the journals and reproduces things before handing them on.
All three said they wouldn't come back. Ada put it best: "A shop whose receipts I have to audit is not a shop I go back to." All three, independently, named the same first fix: charge the quote, or stop and ask. That's the shape of a useful crowd. They didn't agree about everything, but they agreed about what mattered most.
The ways it lied to me
Every one of these reported success, or reported nothing, while something was wrong.
- "CONFIRMED" meant clicked, not done. Billy's Saturday harness reported a press as confirmed as soon as the click landed. The app might have refused it ("price moved") or still be waiting. Fix: the harness waits for the app to say something, quotes it, and says NOTHING in capitals when it doesn't.
- "Stopped" didn't stop. The stop command closed the browsers, printed "stopped", and deleted its list of processes, while the harness servers behind them went on listening. The status command then said "nothing started" over a machine with nineteen ports still open. Fix: stop each harness by its process number, check the port is really closed, and keep the list until it is.
- The clocks were an hour out. The browsers on Saturday ran on UTC and the testers on British Summer Time, so every message's time was an hour behind the journal entries, and several testers were misled. Fix: give the browser and the logs the same named time zone as the people reading them. In the kit that's
TZ_NAME. - One request at a time per tester. A long
waitblocked everything else that tester sent. Fix: keep waits short. The kit caps them at thirty seconds. - A selector that was too generous. The commands the players were given picked market rows by "contains the text", so "Water" also matched "Deep-Water Catch", and three players bought the wrong thing. Fix: target buttons by an id or data attribute, not by visible text. Where the app gives you nothing better than text, match the longest sentence that is unique on the page, not a name. Flying in the game on 15 October, I picked a destination by its name three times in systems with more than one dock, and three times my harness flew me to the wrong one. The description line under each dock was unique. The name wasn't.
- A confirm that didn't read the price. Some apps arm an expensive action and then ask you to confirm it. In the game, the confirmation expired faster than an AI tester can read it and reply, so on 15 October I bundled both presses into one command. On 21 October that command pressed straight through "Confirm · 38,252 cr" and bought 146 hull plates at a price I'd never have accepted. The game had shown me the number. My harness hadn't read it. Fix: if the app needs a second press, have the command quote the confirmation text and accept a limit from the tester ("confirm only if under 20,000"), so it still happens in one round trip, but not blind. Corner Shop has no confirm step, so the kit doesn't need this. Your app might.
- A brief that leaks into the findings. In the 11 October run, Cleo reported that the shop's buttons "only work if you use lowercase". They don't. My brief's example selectors are case-sensitive. Fix: give testers exact selectors, or tell them that if a command fails they should check the command before blaming the app. And reproduce before you file.
- Mine, found by Billy after I'd written this. I asked him to check this guide, and he ran the kit and proved three faults in it. If an app said the same thing twice in a row, my harness reported the second time as NOTHING, because it only noticed when the message changed. My double-click guard was timed from the start of a press, so against a slow app, which is exactly when people press twice, it never fired. And a disabled button and a hidden one both came back as a bare "Timeout 3000ms exceeded", which a tester reads as "the button is broken". Fix: all three are fixed in the kit, and I re-ran his reproductions against it. The harness now reports every message the app writes, times the guard from when a press finishes, and says "disabled" or "not visible" in words.
- Mine on the day: the wrong browser build. An unpinned install fetched a Playwright that wanted a browser I didn't have. Every command after that failed with "no browser called p1", which reads like the tester's mistake. Fix: pin the version, and run the self-test, which would have said FAIL on every port.
The check that catches all of them
Every item in that list is a harness reporting something other than what happened. So before a run, and after any change to the harness, do one action whose answer you already know, through every port, and compare.
./selftest.sh "Corner Shop" 7511 7512 7513
That covers "can each harness see the app". For "does the harness tell the truth about an action", add one known action to your own checklist. With Corner Shop, sign in, buy one tea, and check the harness says the app said: … Bought 1 for 2 and that the cash line on the screen dropped by exactly that much. Then buy six biscuits and check the harness says NOTHING, with the button disabled, rather than claiming success. If the harness can't see a problem you planted, it can't see one you didn't.
What it costs
Measured on the 11 October sample run: 8 minutes 14 seconds from the three testers starting to the last one signing off, and about 275,000 tokens between them (90,630, 90,997 and 93,073 by the sub-agent counters). On a subscription plan, that's part of what you already pay for. On pay-per-token pricing it depends on the model, so check your provider's current rates rather than trusting a number from me. The harness costs nothing: Playwright and Chromium are free, and three headless browsers on a laptop barely register.
Saturday's run was eighteen testers for about three hours, on my own machines, inside the subscriptions I already pay for. The real cost was the evening Billy spent writing the brief and the harness, and the afternoon Scotty spent checking the servers could take it. With this kit, most of that is done for you.
Getting an assistant to do this for you
You don't have to write any of the kit yourself. I didn't write Billy's harness, and Billy didn't write the plugin. The useful prompts are about the rules, not the code:
- "Write me a small program that holds a browser open and lets another program drive it with plain text commands over HTTP. A press must click once and then report exactly what the page said within ten seconds, or say NOTHING if it said nothing. No command may run code in the page."
- "Here's my app and the three kinds of people who use it. Write a tester brief for each: a one-line character that changes how they'd use it, the rules (screen only, no source), a journal every ten actions, and a report that marks every finding SEEN or HEARD."
- "Here are the reports from N testers. Group the findings, count how many SAW each one, separate bugs from design questions, and list every place two testers disagreed."
Then check what comes back the way you'd check a new colleague's work. Plant one problem you know about, and see whether the crowd finds it.
What I actually use it for
The first time was a game, the night before a big release. The bugs it found went into a patch that shipped the next afternoon. The one I think about most is the mining laser. A new miner started with 5,000 credits and the laser cost 9,000, and nothing told them to earn the difference first. All three of the miner testers hit it, and two of them were complaining about it in chat within half an hour of starting. My closed-beta players had been playing for weeks and none of them had mentioned it. I'd never even noticed it myself. That's what a blind first-time user is for. Just as useful were the things it found that I decided not to change, because I could see exactly what the testers were reacting to and decide it was what I wanted.
The bigger thing it gave me is knowing this is possible. I'll be using it on the next app with a front end, whatever it is. A crowd of distinct, opinionated testers, using the same screen a person would, gives you depth and breadth that one careful tester, human or AI, can't. And it doesn't all come back as one voice, which was the part I was least sure of.
The code
Both are in the zip. They're here so you can read them before you run them, which you should do with anything from the internet, mine included.
harness.js
// harness.js: one tester's browsers, held open and driven by plain HTTP, so an AI assistant can play a web app
// through the same screen a person sees. Start one per tester, each on its own port:
//
// TZ_NAME=Europe/London node harness.js <name> <port> <app-url> [shared-dir=./run]
// then: curl -s localhost:<port> -d 'p1 open' (p1, p2 … are browser names the tester chooses)
//
// Ops (b = browser name):
// b open a fresh browser (no cookies) at the app's front page
// b shot <label> screenshot to <shared-dir>/<name>/<label>.png AND every bit of visible text, back to you
// b text <css> the text of matching elements
// b press <css> click ONCE, then wait up to 10 s for the app to say something. Reports what it said,
// or NOTHING (which means nothing YET: wait, then run events). Pressing the same thing
// again while it is still waiting, or within 1.5 s of it finishing, is refused (a
// double-click is not a decision). A disabled or invisible button is reported as such,
// not clicked. (click = press)
// b fill <css>|<value> type a value into a field
// b key <Key> press a keyboard key (Escape, Enter …)
// b events every message the app showed since you last asked, with times
// b wait <ms> pause (max 30 s)
// b close close that browser
// say <text> post to the shared tester chat (everyone in this run sees it)
// chat [n=15] read the last n lines of the shared chat
//
// What it will NOT do, on purpose: run JavaScript in the page, navigate to arbitrary URLs, or read anything but
// the screen. A tester who can't look behind the curtain has to report what a person would see.
// Every command and the first line of its answer goes to <shared-dir>/<name>/commands.log.
const fs = require('fs');
const path = require('path');
const http = require('http');
const { chromium } = require('playwright');
const [NAME, PORT, APP, SHARED = './run'] = process.argv.slice(2);
if (!NAME || !PORT || !APP) { console.error('usage: node harness.js <name> <port> <app-url> [shared-dir]'); process.exit(1); }
// What counts as "the app said something": toasts, alerts, status lines. Override for your own app.
const MSG = process.env.MSG_SELECTOR || '[role=status], [role=alert], .toast, .notice, .error';
const dir = path.join(SHARED, NAME);
const chatFile = path.join(SHARED, 'chat.log');
fs.mkdirSync(dir, { recursive: true });
// One named time zone for the browsers AND the logs, so a message's time matches the journal written beside it.
// On Saturday's run the browsers were on UTC and the testers on British Summer Time, and every time was an hour out.
// Set TZ_NAME to the zone the PEOPLE reading the logs are in (e.g. TZ_NAME=Europe/London); it defaults to this
// machine's zone, which is wrong if the browsers run on a server somewhere else.
const TZ = process.env.TZ_NAME || Intl.DateTimeFormat().resolvedOptions().timeZone;
const now = () => new Date().toLocaleTimeString('en-GB', { hour12: false, timeZone: TZ });
const log = (line) => fs.appendFileSync(path.join(dir, 'commands.log'), `${now()} ${line}\n`);
let browser;
const tabs = {}; // b -> { ctx, page, events: [], seen: 0 }
let lastPress = { key: '', at: 0 };
async function open(b) {
browser ||= await chromium.launch();
const ctx = await browser.newContext({ viewport: { width: 1280, height: 900 }, timezoneId: TZ });
const page = await ctx.newPage();
const rec = { ctx, page, events: [], seen: 0 };
tabs[b] = rec;
// Watch for messages from the page's side, so a toast that comes and goes in three seconds is still caught.
// It reports every WRITE to a message element, not just a change of text: an app that says "Nothing to buy."
// twice in a row has said something twice, and a watcher that only notices changes reports the second as NOTHING.
await page.exposeFunction('__said', (txt) => rec.events.push(`${now()} ${txt}`));
await page.addInitScript((sel) => {
const shown = (el) => el.offsetParent !== null || getComputedStyle(el).position === 'fixed' && getComputedStyle(el).display !== 'none';
const dirty = new Set();
let timer = null;
const flush = () => {
timer = null;
for (const el of dirty) {
const t = (el.innerText || '').replace(/\s+/g, ' ').trim();
if (el.isConnected && shown(el) && t) window.__said(t);
}
dirty.clear();
};
new MutationObserver((muts) => {
for (const m of muts) {
const node = m.target.nodeType === 1 ? m.target : m.target.parentElement;
const el = node && node.closest(sel);
if (el) dirty.add(el);
// A message element added to the page whole (an app that appends a fresh toast each time).
for (const n of m.addedNodes || []) {
if (n.nodeType !== 1) continue;
if (n.matches(sel)) dirty.add(n);
for (const inner of n.querySelectorAll(sel)) dirty.add(inner);
}
}
// One write usually arrives as several mutations (text, then style). Report it once.
if (dirty.size && !timer) timer = setTimeout(flush, 50);
}).observe(document, { subtree: true, childList: true, characterData: true, attributes: true, attributeFilter: ['style', 'class', 'hidden'] });
}, MSG);
await page.goto(APP);
return `opened ${b} at ${APP}`;
}
// One click, then wait for the app to say something. Before clicking, say plainly why a button can't be pressed:
// "page.click: Timeout 3000ms exceeded" is what Playwright says for disabled AND invisible AND missing, and a
// tester who reads it reports "the button is broken" when it is busy.
async function pressOnce(t, page, arg) {
const target = page.locator(arg).first();
if (await page.locator(arg).count() === 0) return `CLICK FAILED: nothing on the page matches ${arg}. Check the command before blaming the app.`;
if (!(await target.isVisible())) return 'CLICK FAILED: that button is on the page but not visible.';
if (await target.isDisabled()) return 'CLICK FAILED: that button is disabled right now, so it was not pressed. It may be busy: try events, then look again.';
const before = t.events.length;
try { await target.click({ timeout: 3000 }); }
catch (e) {
const m = String(e.message).match(/(<[a-z][^>]*>)[^\n]*?intercepts pointer events/i);
const why = m ? `something on top of it is catching the click: ${m[1]}` : String(e.message).split('\n')[0];
return `CLICK FAILED: ${why}`;
}
for (let i = 0; i < 50 && t.events.length === before; i++) await page.waitForTimeout(200);
const said = t.events.slice(before);
t.seen = Math.max(t.seen, t.events.length);
if (said.length) return 'the app said: ' + said.join(' | ');
// NOTHING means "nothing YET". Say what is still on screen, and whether the button is busy, so the tester
// knows to wait and check events before calling anything broken.
const showing = (await page.locator(MSG).filter({ visible: true }).allInnerTexts().catch(() => []))
.map((s) => s.replace(/\s+/g, ' ').trim()).filter(Boolean);
const disabled = await target.isDisabled().catch(() => null);
return 'NOTHING: no new message within 10 s.'
+ (disabled ? ' The button is now DISABLED, with no reason shown.' : '')
+ (showing.length ? ` Still showing from before: "${showing.join(' | ')}".` : '')
+ ' The app may still answer: wait, then run events.';
}
async function run(cmd) {
const sp = cmd.indexOf(' ');
const first = sp < 0 ? cmd : cmd.slice(0, sp);
const restAll = sp < 0 ? '' : cmd.slice(sp + 1);
if (first === 'say') { fs.appendFileSync(chatFile, `${now()} ${NAME}: ${restAll}\n`); return 'said'; }
if (first === 'chat') {
const lines = fs.existsSync(chatFile) ? fs.readFileSync(chatFile, 'utf8').trim().split('\n') : [];
return lines.slice(-(Number(restAll) || 15)).join('\n') || '(nobody has said anything yet)';
}
const b = first;
const sp2 = restAll.indexOf(' ');
const op = sp2 < 0 ? restAll : restAll.slice(0, sp2);
const arg = sp2 < 0 ? '' : restAll.slice(sp2 + 1);
if (op === 'open') return open(b);
const t = tabs[b];
if (!t) return `ERROR: no browser called ${b}. Start with: ${b} open`;
const page = t.page;
switch (op) {
case 'shot': {
const file = path.join(dir, `${(arg || 'shot').replace(/[^\w-]/g, '_')}.png`);
await page.screenshot({ path: file });
return `[${file}]\n` + (await page.innerText('body'));
}
case 'text': return (await page.locator(arg).allInnerTexts()).join('\n') || '(nothing matches)';
case 'fill': { const [css, val] = arg.split('|'); await page.fill(css, val, { timeout: 3000 }); return `filled ${css}`; }
case 'key': await page.keyboard.press(arg); return `pressed key ${arg}`;
case 'wait': await page.waitForTimeout(Math.min(Number(arg) || 1000, 30000)); return 'waited';
case 'events': { const out = t.events.slice(t.seen); t.seen = t.events.length; return out.join('\n') || '(nothing new)'; }
case 'close': await t.ctx.close(); delete tabs[b]; return `closed ${b}`;
case 'press': case 'click': {
// The double-click guard is timed from when the last press FINISHED, not when it started. A slow app
// takes the full 10 s to answer, and a guard timed from the start would never fire in exactly the case
// it exists for: a person pressing again because nothing seemed to happen.
const key = b + ' ' + arg;
if (lastPress.key === key && (lastPress.busy || Date.now() - lastPress.at < 1500)) {
return 'REFUSED: you pressed this again ' + (lastPress.busy ? 'while the last press was still waiting' : 'within 1.5 s of the last one finishing') + '. That is a double-click, not a decision.';
}
lastPress = { key, at: Date.now(), busy: true };
try { return await pressOnce(t, page, arg); }
finally { if (lastPress.key === key) lastPress = { key, at: Date.now(), busy: false }; }
}
default: return `ERROR: unknown op "${op}". Try: open, shot, text, press, fill, key, events, wait, close, say, chat`;
}
}
http.createServer((req, res) => {
let body = '';
req.on('data', (c) => (body += c));
req.on('end', async () => {
const cmd = body.trim();
let out;
try { out = String(await run(cmd)); } catch (e) { out = 'ERROR: ' + String(e.message).split('\n')[0]; }
log(`${cmd} -> ${out.split('\n')[0].slice(0, 200)}`);
res.end(out + '\n');
});
}).listen(Number(PORT), () => console.log(`${NAME}'s harness on :${PORT}, driving ${APP}, clocks on ${TZ}`));
sample-app/server.js
// sample-app/server.js: "Corner Shop", a deliberately imperfect little web shop to point a crowd of testers at.
// No dependencies. Run: node sample-app/server.js then open http://localhost:8080
//
// Everyone shares ONE stock room, so testers compete for the same shelves, the way players crowd a market.
// It has five planted problems. Don't tell your testers what they are; that's the point. They're listed in
// sample-app/ANSWERS.md, for you, after the run.
const http = require('http');
const PORT = Number(process.env.PORT || 8080);
const START_CASH = 100;
const items = {
tea: { name: 'Tea', price: 2, stock: 60 },
biscuit: { name: 'Biscuits', price: 3, stock: 40 },
jam: { name: 'Jam', price: 5, stock: 25 },
kettle: { name: 'Kettle', price: 24, stock: 6 },
};
const users = {}; // name -> { cash, points, bag: {id: qty}, orders: [] }
// Price rises as the shelf empties: the last few of anything cost more.
const priceOf = (id) => {
const it = items[id];
const start = { tea: 60, biscuit: 40, jam: 25, kettle: 6 }[id];
return Math.round(it.price * (1 + 0.5 * (1 - it.stock / start)) * 100) / 100;
};
const quote = (id, qty) => Math.round(priceOf(id) * qty * 100) / 100;
const user = (name) => (users[name] ||= { cash: START_CASH, points: 0, bag: {}, orders: [] });
const json = (res, code, body) => { res.writeHead(code, { 'Content-Type': 'application/json' }); res.end(JSON.stringify(body)); };
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
async function api(req, res, url, body) {
const u = body.name ? user(String(body.name).slice(0, 20)) : null;
if (url === '/api/state') {
return json(res, 200, { items: Object.fromEntries(Object.entries(items).map(([k, v]) => [k, { ...v, now: priceOf(k) }])), me: u });
}
if (!u) return json(res, 400, { error: 'Who are you? Sign in first.' });
if (url === '/api/quote') return json(res, 200, { total: quote(body.id, Number(body.qty) || 0) });
if (url === '/api/buy') {
const id = body.id, qty = Math.floor(Number(body.qty));
const it = items[id];
if (!it || !(qty > 0)) return json(res, 400, { error: 'Nothing to buy.' });
// PLANTED 1: bigger orders are slow. Nothing on screen says so.
if (qty >= 5) await sleep(9000 + Math.random() * 6000);
if (it.stock < qty) return json(res, 409, { error: `Only ${it.stock} left.` });
let total = quote(id, qty);
// PLANTED 2: orders of 10+ quietly pick up a 10% "handling" charge the quote never showed.
if (qty >= 10) total = Math.round(total * 1.1 * 100) / 100;
if (u.cash < total) return json(res, 402, { error: `That's ${total}. You have ${u.cash}.` });
it.stock -= qty; u.cash = Math.round((u.cash - total) * 100) / 100;
u.bag[id] = (u.bag[id] || 0) + qty;
// PLANTED 5 (a design question, not a bug): a loyalty point per ITEM, whatever it costs.
u.points += qty;
u.orders.push({ n: u.orders.length + 1, id, qty, total, status: 'placed' });
return json(res, 200, { ok: true, total, me: u });
}
if (url === '/api/cancel') {
const o = u.orders.find((x) => x.n === Number(body.n));
if (!o || o.status !== 'placed') return json(res, 400, { error: 'No such order.' });
// PLANTED 4: the first cancel is acknowledged but does nothing. Only a second one works.
o.tries = (o.tries || 0) + 1;
if (o.tries < 2) return json(res, 200, { ok: true, message: 'Cancellation requested.' });
o.status = 'cancelled';
items[o.id].stock += o.qty; u.bag[o.id] -= o.qty; u.points -= o.qty;
u.cash = Math.round((u.cash + o.total) * 100) / 100;
return json(res, 200, { ok: true, message: `Order ${o.n} cancelled, ${o.total} refunded.`, me: u });
}
json(res, 404, { error: 'Unknown.' });
}
const PAGE = `<!doctype html><html><head><meta charset="utf-8"><title>Corner Shop</title><style>
body{font:16px system-ui,sans-serif;max-width:760px;margin:24px auto;padding:0 16px;color:#222}
table{width:100%;border-collapse:collapse}td,th{padding:8px;border-bottom:1px solid #ddd;text-align:left}
button{padding:6px 12px}button:disabled{opacity:.5}input[type=number]{width:60px}
#toast{position:fixed;top:12px;right:12px;background:#222;color:#fff;padding:10px 14px;border-radius:6px;display:none}
#thanks{position:fixed;inset:0;background:rgba(255,255,255,.01);display:none}
#thanks div{position:absolute;bottom:8px;right:8px;font-size:11px;color:#bbb}
</style></head><body>
<h1>Corner Shop</h1>
<p id="who"><label>Your name <input id="name"></label> <button id="signin">Sign in</button></p>
<p id="wallet"></p>
<table><thead><tr><th>Item</th><th>Price</th><th>On shelf</th><th>Qty</th><th>Quote</th><th></th></tr></thead><tbody id="rows"></tbody></table>
<h2>Your orders</h2><ul id="orders"></ul>
<div id="toast" role="status"></div>
<div id="thanks"><div>Thanks for shopping with us! <button id="ok">ok</button></div></div>
<script>
let me=null,name='';const $=s=>document.querySelector(s);
const post=(u,b)=>fetch(u,{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify({name,...b})}).then(r=>r.json());
function toast(t){const e=$('#toast');e.textContent=t;e.style.display='block';clearTimeout(e._t);e._t=setTimeout(()=>e.style.display='none',3000)}
async function load(){const s=await post('/api/state',{});me=s.me;
$('#wallet').textContent=me?('Cash: '+me.cash+' · Loyalty points: '+me.points):'Sign in to shop.';
$('#rows').innerHTML=Object.entries(s.items).map(([k,v])=>'<tr><td>'+v.name+'</td><td>'+v.now+'</td><td>'+v.stock+'</td><td><input type=number min=1 value=1 data-q="'+k+'"></td><td data-quote="'+k+'">'+v.now+'</td><td><button data-buy="'+k+'">Buy</button></td></tr>').join('');
$('#orders').innerHTML=me?me.orders.map(o=>'<li>#'+o.n+' '+o.qty+' × '+o.id+' for '+o.total+' — '+o.status+(o.status==='placed'?' <button data-cancel="'+o.n+'">Cancel</button>':'')+'</li>').join(''):'';}
document.addEventListener('input',async e=>{const k=e.target.dataset.q;if(!k||!name)return;const q=await post('/api/quote',{id:k,qty:e.target.value});document.querySelector('[data-quote="'+k+'"]').textContent=q.total});
document.addEventListener('click',async e=>{const b=e.target;
if(b.id==='signin'){name=$('#name').value.trim();if(name){await load();toast('Welcome, '+name)}return}
if(b.id==='ok'){$('#thanks').style.display='none';return}
if(b.dataset.buy){const k=b.dataset.buy,qty=document.querySelector('[data-q="'+k+'"]').value;b.disabled=true;
const r=await post('/api/buy',{id:k,qty});b.disabled=false;
if(r.error){toast(r.error);return}
await load();toast('Bought '+qty+' for '+r.total);
if(me.orders.length===1)$('#thanks').style.display='block';return}
if(b.dataset.cancel){const r=await post('/api/cancel',{n:b.dataset.cancel});toast(r.message||r.error);await load()}});
load();
</script></body></html>`;
http.createServer((req, res) => {
const url = req.url.split('?')[0];
if (req.method === 'GET' && url === '/') { res.writeHead(200, { 'Content-Type': 'text/html; charset=utf-8' }); return res.end(PAGE); }
let raw = '';
req.on('data', (c) => (raw += c));
req.on('end', () => { let b = {}; try { b = JSON.parse(raw || '{}'); } catch {} api(req, res, url, b).catch((e) => json(res, 500, { error: String(e) })); });
}).listen(PORT, () => console.log('Corner Shop on http://localhost:' + PORT));