feat: network idle, state persistence, iframe support, chain pipe format (v0.12.1.0) (#516)

* feat: network idle detection + chain pipe format

- Upgrade click/fill/select from domcontentloaded to networkidle wait
  (2s timeout, best-effort). Catches XHR/fetch triggered by interactions.
- Add pipe-delimited format to chain as JSON fallback:
  $B chain 'goto url | click @e5 | snapshot -ic'
- Add post-loop networkidle wait in chain when last command was a write.
- Frame-aware: commands use target (getActiveFrameOrPage) for locator ops,
  page-only ops (goto/back/forward/reload) guard against frame context.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: $B state save/load + $B frame — new browse commands

- state save/load: persist cookies + URLs to .gstack/browse-states/{name}.json
  File perms 0o600, name sanitized to [a-zA-Z0-9_-]. V1 skips localStorage
  (breaks on load-before-navigate). Load replaces session via closeAllPages().
- frame: switch command context to iframe via CSS selector, @ref, --name, or
  --url. 'frame main' returns to main frame. Execution target abstraction
  (getActiveFrameOrPage) across read-commands, snapshot, and write-commands.
- Frame context cleared on tab switch, navigation, resume, and handoff.
- Snapshot shows [Context: iframe src="..."] header when in frame.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* test: add tests for network idle, chain pipe format, state, and frame

- Network idle: click on fetch button waits for XHR, static click is fast
- Chain pipe: pipe-delimited commands, quoted args, JSON still works
- State: save/load round-trip, name sanitization, missing state error
- Frame: switch to iframe + back, snapshot context header, fill in frame,
  goto-in-frame guard, usage error

New fixtures: network-idle.html (fetch + static buttons), iframe.html (srcdoc)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: review fixes — iframe ref scoping, detached frame recovery, state validation

- snapshot.ts: ref locators, cursor-interactive scan, and cursor locator
  now use target (frame-aware) instead of page — fixes @ref clicking in iframes
- browser-manager.ts: getActiveFrameOrPage auto-recovers from detached frames
  via isDetached() check
- meta-commands.ts: state load resets activeFrame, elementHandle disposed after
  contentFrame(), state file schema validation (cookies + pages arrays),
  filter empty pipe segments in chain tokenizer
- write-commands.ts: upload command uses target.locator() for frame support

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: regenerate SKILL.md files + rebuild binary

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: bump version and changelog (v0.12.1.0)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-03-26 11:15:12 -06:00
committed by GitHub
parent 10046ecdcb
commit ee21f2fc90
14 changed files with 571 additions and 64 deletions
+26
View File
@@ -1,5 +1,31 @@
# Changelog
## [0.12.1.0] - 2026-03-26 — Smarter Browsing: Network Idle, State Persistence, Iframes
Every click, fill, and select now waits for the page to settle before returning. No more stale snapshots because an XHR was still in-flight. Chain accepts pipe-delimited format for faster multi-step flows. You can save and restore browser sessions (cookies + open tabs). And iframe content is now reachable.
### Added
- **Network idle detection.** `click`, `fill`, and `select` auto-wait up to 2s for network requests to settle before returning. Catches XHR/fetch triggered by interactions. Uses Playwright's built-in `waitForLoadState('networkidle')`, not a custom tracker.
- **`$B state save/load`.** Save your browser session (cookies + open tabs) to a named file, load it back later. Files stored at `.gstack/browse-states/{name}.json` with 0o600 permissions. V1 saves cookies + URLs only (not localStorage, which breaks on load-before-navigate). Load replaces the current session, not merge.
- **`$B frame` command.** Switch command context into an iframe: `$B frame iframe`, `$B frame --name checkout`, `$B frame --url stripe`, or `$B frame @e5`. All subsequent commands (click, fill, snapshot, etc.) operate inside the iframe. `$B frame main` returns to the main page. Snapshot shows `[Context: iframe src="..."]` header. Detached frames auto-recover.
- **Chain pipe format.** Chain now accepts `$B chain 'goto url | click @e5 | snapshot -ic'` as a fallback when JSON parsing fails. Pipe-delimited with quote-aware tokenization.
### Changed
- **Chain post-loop idle wait.** After executing all commands in a chain, if the last was a write command, chain waits for network idle before returning.
### Fixed
- **Iframe ref scoping.** Snapshot ref locators, cursor-interactive scan, and cursor locators now use the frame-aware target instead of always scoping to the main page.
- **Detached frame recovery.** `getActiveFrameOrPage()` checks `isDetached()` and auto-recovers.
- **State load resets frame context.** Loading a saved state clears the active frame reference.
- **elementHandle leak in frame command.** Now properly disposed after getting contentFrame.
- **Upload command frame-aware.** `upload` uses the frame-aware target for file input locators.
## [0.12.0.0] - 2026-03-26 — Headed Mode + Sidebar Agent
You can now watch Claude work in a real Chrome window and direct it from a sidebar chat.
+2
View File
@@ -591,6 +591,7 @@ Refs are invalidated on navigation — run `snapshot` again after `goto`.
| Command | Description |
|---------|-------------|
| `chain` | Run commands from JSON stdin. Format: [["cmd","arg1",...],...] |
| `frame <sel|@ref|--name n|--url pattern|main>` | Switch to iframe context (or main to return) |
| `inbox [--clear]` | List messages from sidebar scout inbox |
| `watch [stop]` | Passive observation — periodic snapshots while user browses |
@@ -611,6 +612,7 @@ Refs are invalidated on navigation — run `snapshot` again after `goto`.
| `handoff [message]` | Open visible Chrome at current page for user takeover |
| `restart` | Restart server |
| `resume` | Re-snapshot after user takeover, return control to AI |
| `state save|load <name>` | Save/load browser state (cookies + URLs) |
| `status` | Health check |
| `stop` | Shutdown server |
+9 -13
View File
@@ -80,17 +80,14 @@ May replace `/setup-browser-cookies` for most use cases since the user's real co
**Effort:** S
**Priority:** P3
### State persistence
### State persistence — SHIPPED
**What:** Save/load cookies + localStorage to JSON files for reproducible test sessions.
~~**What:** Save/load cookies + localStorage to JSON files for reproducible test sessions.~~
**Why:** Enables "resume where I left off" for QA sessions and repeatable auth states.
`$B state save/load` ships in v0.12.1.0. V1 saves cookies + URLs only (not localStorage, which breaks on load-before-navigate). Files at `.gstack/browse-states/{name}.json` with 0o600 permissions. Load replaces session (closes all pages first). Name sanitized to `[a-zA-Z0-9_-]`.
**Context:** The `saveState()`/`restoreState()` helpers from the handoff feature (browser-manager.ts) already capture cookies + localStorage + sessionStorage + URLs. Adding file I/O on top is ~20 lines.
**Effort:** S
**Priority:** P3
**Depends on:** Sessions
**Remaining:** V2 localStorage support (needs pre-navigation injection strategy).
**Completed:** v0.12.1.0 (2026-03-26)
### Auth vault
@@ -102,14 +99,13 @@ May replace `/setup-browser-cookies` for most use cases since the user's real co
**Priority:** P3
**Depends on:** Sessions, state persistence
### Iframe support
### Iframe support — SHIPPED
**What:** `frame <sel>` and `frame main` commands for cross-frame interaction.
~~**What:** `frame <sel>` and `frame main` commands for cross-frame interaction.~~
**Why:** Many web apps use iframes (embeds, payment forms, ads). Currently invisible to browse.
`$B frame` ships in v0.12.1.0. Supports CSS selector, @ref, `--name`, and `--url` pattern matching. Execution target abstraction (`getActiveFrameOrPage()`) across all read/write/snapshot commands. Frame context cleared on navigation, tab switch, resume. Detached frame auto-recovery. Page-only operations (goto, screenshot, viewport) throw clear error when in frame context.
**Effort:** M
**Priority:** P4
**Completed:** v0.12.1.0 (2026-03-26)
### Semantic locators
+1 -1
View File
@@ -1 +1 @@
0.12.0.0
0.12.1.0
+2
View File
@@ -474,6 +474,7 @@ Refs are invalidated on navigation — run `snapshot` again after `goto`.
| Command | Description |
|---------|-------------|
| `chain` | Run commands from JSON stdin. Format: [["cmd","arg1",...],...] |
| `frame <sel|@ref|--name n|--url pattern|main>` | Switch to iframe context (or main to return) |
| `inbox [--clear]` | List messages from sidebar scout inbox |
| `watch [stop]` | Passive observation — periodic snapshots while user browses |
@@ -494,5 +495,6 @@ Refs are invalidated on navigation — run `snapshot` again after `goto`.
| `handoff [message]` | Open visible Chrome at current page for user takeover |
| `restart` | Restart server |
| `resume` | Re-snapshot after user takeover, return control to AI |
| `state save|load <name>` | Save/load browser state (cookies + URLs) |
| `status` | Health check |
| `stop` | Shutdown server |
+39
View File
@@ -402,6 +402,7 @@ export class BrowserManager {
switchTab(id: number): void {
if (!this.pages.has(id)) throw new Error(`Tab ${id} not found`);
this.activeTabId = id;
this.activeFrame = null; // Frame context is per-tab
}
getTabCount(): number {
@@ -531,6 +532,42 @@ export class BrowserManager {
return this.customUserAgent;
}
// ─── Lifecycle helpers ───────────────────────────────
/**
* Close all open pages and clear the pages map.
* Used by state load to replace the current session.
*/
async closeAllPages(): Promise<void> {
for (const page of this.pages.values()) {
await page.close().catch(() => {});
}
this.pages.clear();
this.clearRefs();
}
// ─── Frame context ─────────────────────────────────
private activeFrame: import('playwright').Frame | null = null;
setFrame(frame: import('playwright').Frame | null): void {
this.activeFrame = frame;
}
getFrame(): import('playwright').Frame | null {
return this.activeFrame;
}
/**
* Returns the active frame if set, otherwise the current page.
* Use this for operations that work on both Page and Frame (locator, evaluate, etc.).
*/
getActiveFrameOrPage(): import('playwright').Page | import('playwright').Frame {
// Auto-recover from detached frames (iframe removed/navigated)
if (this.activeFrame?.isDetached()) {
this.activeFrame = null;
}
return this.activeFrame ?? this.getPage();
}
// ─── State Save/Restore (shared by recreateContext + handoff) ─
/**
* Capture browser state: cookies, localStorage, sessionStorage, URLs, active tab.
@@ -789,6 +826,7 @@ export class BrowserManager {
resume(): void {
this.clearRefs();
this.resetFailures();
this.activeFrame = null;
}
getIsHeaded(): boolean {
@@ -818,6 +856,7 @@ export class BrowserManager {
page.on('framenavigated', (frame) => {
if (frame === page.mainFrame()) {
this.clearRefs();
this.activeFrame = null; // Navigation invalidates frame context
}
});
+6
View File
@@ -34,6 +34,8 @@ export const META_COMMANDS = new Set([
'connect', 'disconnect', 'focus',
'inbox',
'watch',
'state',
'frame',
]);
export const ALL_COMMANDS = new Set([...READ_COMMANDS, ...WRITE_COMMANDS, ...META_COMMANDS]);
@@ -109,6 +111,10 @@ export const COMMAND_DESCRIPTIONS: Record<string, { category: string; descriptio
'inbox': { category: 'Meta', description: 'List messages from sidebar scout inbox', usage: 'inbox [--clear]' },
// Watch
'watch': { category: 'Meta', description: 'Passive observation — periodic snapshots while user browses', usage: 'watch [stop]' },
// State
'state': { category: 'Server', description: 'Save/load browser state (cookies + URLs)', usage: 'state save|load <name>' },
// Frame
'frame': { category: 'Meta', description: 'Switch to iframe context (or main to return)', usage: 'frame <sel|@ref|--name n|--url pattern|main>' },
};
// Load-time validation: descriptions must cover exactly the command sets
+129 -8
View File
@@ -11,6 +11,8 @@ import * as Diff from 'diff';
import * as fs from 'fs';
import * as path from 'path';
import { TEMP_DIR, isPathWithin } from './platform';
import { resolveConfig } from './config';
import type { Frame } from 'playwright';
// Security: Path validation to prevent path traversal attacks
const SAFE_DIRECTORIES = [TEMP_DIR, process.cwd()];
@@ -23,6 +25,25 @@ export function validateOutputPath(filePath: string): void {
}
}
/** Tokenize a pipe segment respecting double-quoted strings. */
function tokenizePipeSegment(segment: string): string[] {
const tokens: string[] = [];
let current = '';
let inQuote = false;
for (let i = 0; i < segment.length; i++) {
const ch = segment[i];
if (ch === '"') {
inQuote = !inQuote;
} else if (ch === ' ' && !inQuote) {
if (current) { tokens.push(current); current = ''; }
} else {
current += ch;
}
}
if (current) tokens.push(current);
return tokens;
}
export async function handleMetaCommand(
command: string,
args: string[],
@@ -187,35 +208,54 @@ export async function handleMetaCommand(
case 'chain': {
// Read JSON array from args[0] (if provided) or expect it was passed as body
const jsonStr = args[0];
if (!jsonStr) throw new Error('Usage: echo \'[["goto","url"],["text"]]\' | browse chain');
if (!jsonStr) throw new Error(
'Usage: echo \'[["goto","url"],["text"]]\' | browse chain\n' +
' or: browse chain \'goto url | click @e5 | snapshot -ic\''
);
let commands: string[][];
try {
commands = JSON.parse(jsonStr);
if (!Array.isArray(commands)) throw new Error('not array');
} catch {
throw new Error('Invalid JSON. Expected: [["command", "arg1", "arg2"], ...]');
// Fallback: pipe-delimited format "goto url | click @e5 | snapshot -ic"
commands = jsonStr.split(' | ')
.filter(seg => seg.trim().length > 0)
.map(seg => tokenizePipeSegment(seg.trim()));
}
if (!Array.isArray(commands)) throw new Error('Expected JSON array of commands');
const results: string[] = [];
const { handleReadCommand } = await import('./read-commands');
const { handleWriteCommand } = await import('./write-commands');
let lastWasWrite = false;
for (const cmd of commands) {
const [name, ...cmdArgs] = cmd;
try {
let result: string;
if (WRITE_COMMANDS.has(name)) result = await handleWriteCommand(name, cmdArgs, bm);
else if (READ_COMMANDS.has(name)) result = await handleReadCommand(name, cmdArgs, bm);
else if (META_COMMANDS.has(name)) result = await handleMetaCommand(name, cmdArgs, bm, shutdown);
else throw new Error(`Unknown command: ${name}`);
if (WRITE_COMMANDS.has(name)) {
result = await handleWriteCommand(name, cmdArgs, bm);
lastWasWrite = true;
} else if (READ_COMMANDS.has(name)) {
result = await handleReadCommand(name, cmdArgs, bm);
lastWasWrite = false;
} else if (META_COMMANDS.has(name)) {
result = await handleMetaCommand(name, cmdArgs, bm, shutdown);
lastWasWrite = false;
} else {
throw new Error(`Unknown command: ${name}`);
}
results.push(`[${name}] ${result}`);
} catch (err: any) {
results.push(`[${name}] ERROR: ${err.message}`);
}
}
// Wait for network to settle after write commands before returning
if (lastWasWrite) {
await bm.getPage().waitForLoadState('networkidle', { timeout: 2000 }).catch(() => {});
}
return results.join('\n\n');
}
@@ -410,6 +450,87 @@ export async function handleMetaCommand(
return lines.join('\n');
}
// ─── State ────────────────────────────────────────
case 'state': {
const [action, name] = args;
if (!action || !name) throw new Error('Usage: state save|load <name>');
// Sanitize name: alphanumeric + hyphens + underscores only
if (!/^[a-zA-Z0-9_-]+$/.test(name)) {
throw new Error('State name must be alphanumeric (a-z, 0-9, _, -)');
}
const config = resolveConfig();
const stateDir = path.join(config.stateDir, 'browse-states');
fs.mkdirSync(stateDir, { recursive: true });
const statePath = path.join(stateDir, `${name}.json`);
if (action === 'save') {
const state = await bm.saveState();
// V1: cookies + URLs only (not localStorage — breaks on load-before-navigate)
const saveData = {
version: 1,
cookies: state.cookies,
pages: state.pages.map(p => ({ url: p.url, isActive: p.isActive })),
};
fs.writeFileSync(statePath, JSON.stringify(saveData, null, 2), { mode: 0o600 });
return `State saved: ${statePath} (${state.cookies.length} cookies, ${state.pages.length} pages — treat as sensitive)`;
}
if (action === 'load') {
if (!fs.existsSync(statePath)) throw new Error(`State not found: ${statePath}`);
const data = JSON.parse(fs.readFileSync(statePath, 'utf-8'));
if (!Array.isArray(data.cookies) || !Array.isArray(data.pages)) {
throw new Error('Invalid state file: expected cookies and pages arrays');
}
// Close existing pages, then restore (replace, not merge)
bm.setFrame(null);
await bm.closeAllPages();
await bm.restoreState({
cookies: data.cookies,
pages: data.pages.map((p: any) => ({ ...p, storage: null })),
});
return `State loaded: ${data.cookies.length} cookies, ${data.pages.length} pages`;
}
throw new Error('Usage: state save|load <name>');
}
// ─── Frame ───────────────────────────────────────
case 'frame': {
const target = args[0];
if (!target) throw new Error('Usage: frame <selector|@ref|--name name|--url pattern|main>');
if (target === 'main') {
bm.setFrame(null);
bm.clearRefs();
return 'Switched to main frame';
}
const page = bm.getPage();
let frame: Frame | null = null;
if (target === '--name') {
if (!args[1]) throw new Error('Usage: frame --name <name>');
frame = page.frame({ name: args[1] });
} else if (target === '--url') {
if (!args[1]) throw new Error('Usage: frame --url <pattern>');
frame = page.frame({ url: new RegExp(args[1]) });
} else {
// CSS selector or @ref for the iframe element
const resolved = await bm.resolveRef(target);
const locator = 'locator' in resolved ? resolved.locator : page.locator(resolved.selector);
const elementHandle = await locator.elementHandle({ timeout: 5000 });
frame = await elementHandle?.contentFrame() ?? null;
await elementHandle?.dispose();
}
if (!frame) throw new Error(`Frame not found: ${target}`);
bm.setFrame(frame);
bm.clearRefs();
return `Switched to frame: ${frame.url()}`;
}
default:
throw new Error(`Unknown meta command: ${command}`);
}
+23 -15
View File
@@ -7,7 +7,7 @@
import type { BrowserManager } from './browser-manager';
import { consoleBuffer, networkBuffer, dialogBuffer } from './buffers';
import type { Page } from 'playwright';
import type { Page, Frame } from 'playwright';
import * as fs from 'fs';
import * as path from 'path';
import { TEMP_DIR, isPathWithin } from './platform';
@@ -57,7 +57,7 @@ export function validateReadPath(filePath: string): void {
* Extract clean text from a page (strips script/style/noscript/svg).
* Exported for DRY reuse in meta-commands (diff).
*/
export async function getCleanText(page: Page): Promise<string> {
export async function getCleanText(page: Page | Frame): Promise<string> {
return await page.evaluate(() => {
const body = document.body;
if (!body) return '';
@@ -77,10 +77,12 @@ export async function handleReadCommand(
bm: BrowserManager
): Promise<string> {
const page = bm.getPage();
// Frame-aware target for content extraction
const target = bm.getActiveFrameOrPage();
switch (command) {
case 'text': {
return await getCleanText(page);
return await getCleanText(target);
}
case 'html': {
@@ -90,13 +92,19 @@ export async function handleReadCommand(
if ('locator' in resolved) {
return await resolved.locator.innerHTML({ timeout: 5000 });
}
return await page.innerHTML(resolved.selector);
return await target.locator(resolved.selector).innerHTML({ timeout: 5000 });
}
return await page.content();
// page.content() is page-only; use evaluate for frame compat
const doctype = await target.evaluate(() => {
const dt = document.doctype;
return dt ? `<!DOCTYPE ${dt.name}>` : '';
});
const html = await target.evaluate(() => document.documentElement.outerHTML);
return doctype ? `${doctype}\n${html}` : html;
}
case 'links': {
const links = await page.evaluate(() =>
const links = await target.evaluate(() =>
[...document.querySelectorAll('a[href]')].map(a => ({
text: a.textContent?.trim().slice(0, 120) || '',
href: (a as HTMLAnchorElement).href,
@@ -106,7 +114,7 @@ export async function handleReadCommand(
}
case 'forms': {
const forms = await page.evaluate(() => {
const forms = await target.evaluate(() => {
return [...document.querySelectorAll('form')].map((form, i) => {
const fields = [...form.querySelectorAll('input, select, textarea')].map(el => {
const input = el as HTMLInputElement;
@@ -136,7 +144,7 @@ export async function handleReadCommand(
}
case 'accessibility': {
const snapshot = await page.locator("body").ariaSnapshot();
const snapshot = await target.locator("body").ariaSnapshot();
return snapshot;
}
@@ -144,7 +152,7 @@ export async function handleReadCommand(
const expr = args[0];
if (!expr) throw new Error('Usage: browse js <expression>');
const wrapped = wrapForEvaluate(expr);
const result = await page.evaluate(wrapped);
const result = await target.evaluate(wrapped);
return typeof result === 'object' ? JSON.stringify(result, null, 2) : String(result ?? '');
}
@@ -155,7 +163,7 @@ export async function handleReadCommand(
if (!fs.existsSync(filePath)) throw new Error(`File not found: ${filePath}`);
const code = fs.readFileSync(filePath, 'utf-8');
const wrapped = wrapForEvaluate(code);
const result = await page.evaluate(wrapped);
const result = await target.evaluate(wrapped);
return typeof result === 'object' ? JSON.stringify(result, null, 2) : String(result ?? '');
}
@@ -170,7 +178,7 @@ export async function handleReadCommand(
);
return value;
}
const value = await page.evaluate(
const value = await target.evaluate(
([sel, prop]) => {
const el = document.querySelector(sel);
if (!el) return `Element not found: ${sel}`;
@@ -195,7 +203,7 @@ export async function handleReadCommand(
});
return JSON.stringify(attrs, null, 2);
}
const attrs = await page.evaluate((sel) => {
const attrs = await target.evaluate((sel: string) => {
const el = document.querySelector(sel);
if (!el) return `Element not found: ${sel}`;
const result: Record<string, string> = {};
@@ -253,7 +261,7 @@ export async function handleReadCommand(
if ('locator' in resolved) {
locator = resolved.locator;
} else {
locator = page.locator(resolved.selector);
locator = target.locator(resolved.selector);
}
switch (property) {
@@ -283,10 +291,10 @@ export async function handleReadCommand(
if (args[0] === 'set' && args[1]) {
const key = args[1];
const value = args[2] || '';
await page.evaluate(([k, v]) => localStorage.setItem(k, v), [key, value]);
await target.evaluate(([k, v]: string[]) => localStorage.setItem(k, v), [key, value]);
return `Set localStorage["${key}"]`;
}
const storage = await page.evaluate(() => ({
const storage = await target.evaluate(() => ({
localStorage: { ...localStorage },
sessionStorage: { ...sessionStorage },
}));
+16 -7
View File
@@ -17,7 +17,7 @@
* Later: "click @e3" look up Locator locator.click()
*/
import type { Page, Locator } from 'playwright';
import type { Page, Frame, Locator } from 'playwright';
import type { BrowserManager, RefEntry } from './browser-manager';
import * as Diff from 'diff';
import { TEMP_DIR, isPathWithin } from './platform';
@@ -136,15 +136,18 @@ export async function handleSnapshot(
): Promise<string> {
const opts = parseSnapshotArgs(args);
const page = bm.getPage();
// Frame-aware target for accessibility tree
const target = bm.getActiveFrameOrPage();
const inFrame = bm.getFrame() !== null;
// Get accessibility tree via ariaSnapshot
let rootLocator: Locator;
if (opts.selector) {
rootLocator = page.locator(opts.selector);
rootLocator = target.locator(opts.selector);
const count = await rootLocator.count();
if (count === 0) throw new Error(`Selector not found: ${opts.selector}`);
} else {
rootLocator = page.locator('body');
rootLocator = target.locator('body');
}
const ariaText = await rootLocator.ariaSnapshot();
@@ -205,11 +208,11 @@ export async function handleSnapshot(
let locator: Locator;
if (opts.selector) {
locator = page.locator(opts.selector).getByRole(node.role as any, {
locator = target.locator(opts.selector).getByRole(node.role as any, {
name: node.name || undefined,
});
} else {
locator = page.getByRole(node.role as any, {
locator = target.getByRole(node.role as any, {
name: node.name || undefined,
});
}
@@ -233,7 +236,7 @@ export async function handleSnapshot(
// ─── Cursor-interactive scan (-C) ─────────────────────────
if (opts.cursorInteractive) {
try {
const cursorElements = await page.evaluate(() => {
const cursorElements = await target.evaluate(() => {
const STANDARD_INTERACTIVE = new Set([
'A', 'BUTTON', 'INPUT', 'SELECT', 'TEXTAREA', 'SUMMARY', 'DETAILS',
]);
@@ -287,7 +290,7 @@ export async function handleSnapshot(
let cRefCounter = 1;
for (const elem of cursorElements) {
const ref = `c${cRefCounter++}`;
const locator = page.locator(elem.selector);
const locator = target.locator(elem.selector);
refMap.set(ref, { locator, role: 'cursor-interactive', name: elem.text });
output.push(`@${ref} [${elem.reason}] "${elem.text}"`);
}
@@ -394,5 +397,11 @@ export async function handleSnapshot(
// Store for future diffs
bm.setLastSnapshot(snapshotText);
// Add frame context header when operating inside an iframe
if (inFrame) {
const frameUrl = bm.getFrame()?.url() ?? 'unknown';
output.unshift(`[Context: iframe src="${frameUrl}"]`);
}
return output.join('\n');
}
+23 -13
View File
@@ -18,9 +18,13 @@ export async function handleWriteCommand(
bm: BrowserManager
): Promise<string> {
const page = bm.getPage();
// Frame-aware target for locator-based operations (click, fill, etc.)
const target = bm.getActiveFrameOrPage();
const inFrame = bm.getFrame() !== null;
switch (command) {
case 'goto': {
if (inFrame) throw new Error('Cannot use goto inside a frame. Run \'frame main\' first.');
const url = args[0];
if (!url) throw new Error('Usage: browse goto <url>');
await validateNavigationUrl(url);
@@ -30,16 +34,19 @@ export async function handleWriteCommand(
}
case 'back': {
if (inFrame) throw new Error('Cannot use back inside a frame. Run \'frame main\' first.');
await page.goBack({ waitUntil: 'domcontentloaded', timeout: 15000 });
return `Back → ${page.url()}`;
}
case 'forward': {
if (inFrame) throw new Error('Cannot use forward inside a frame. Run \'frame main\' first.');
await page.goForward({ waitUntil: 'domcontentloaded', timeout: 15000 });
return `Forward → ${page.url()}`;
}
case 'reload': {
if (inFrame) throw new Error('Cannot use reload inside a frame. Run \'frame main\' first.');
await page.reload({ waitUntil: 'domcontentloaded', timeout: 15000 });
return `Reloaded ${page.url()}`;
}
@@ -73,15 +80,14 @@ export async function handleWriteCommand(
if ('locator' in resolved) {
await resolved.locator.click({ timeout: 5000 });
} else {
await page.click(resolved.selector, { timeout: 5000 });
await target.locator(resolved.selector).click({ timeout: 5000 });
}
} catch (err: any) {
// Enhanced error guidance: clicking <option> elements always fails (not visible / timeout)
const isOption = 'locator' in resolved
? await resolved.locator.evaluate(el => el.tagName === 'OPTION').catch(() => false)
: await page.evaluate(
(sel: string) => document.querySelector(sel)?.tagName === 'OPTION',
(resolved as { selector: string }).selector
: await target.locator(resolved.selector).evaluate(
el => el.tagName === 'OPTION'
).catch(() => false);
if (isOption) {
throw new Error(
@@ -90,8 +96,8 @@ export async function handleWriteCommand(
}
throw err;
}
// Wait briefly for any navigation/DOM update
await page.waitForLoadState('domcontentloaded').catch(() => {});
// Wait for network to settle (catches XHR/fetch triggered by clicks)
await page.waitForLoadState('networkidle', { timeout: 2000 }).catch(() => {});
return `Clicked ${selector} → now at ${page.url()}`;
}
@@ -103,8 +109,10 @@ export async function handleWriteCommand(
if ('locator' in resolved) {
await resolved.locator.fill(value, { timeout: 5000 });
} else {
await page.fill(resolved.selector, value, { timeout: 5000 });
await target.locator(resolved.selector).fill(value, { timeout: 5000 });
}
// Wait for network to settle (form validation XHRs)
await page.waitForLoadState('networkidle', { timeout: 2000 }).catch(() => {});
return `Filled ${selector}`;
}
@@ -116,8 +124,10 @@ export async function handleWriteCommand(
if ('locator' in resolved) {
await resolved.locator.selectOption(value, { timeout: 5000 });
} else {
await page.selectOption(resolved.selector, value, { timeout: 5000 });
await target.locator(resolved.selector).selectOption(value, { timeout: 5000 });
}
// Wait for network to settle (dropdown-triggered requests)
await page.waitForLoadState('networkidle', { timeout: 2000 }).catch(() => {});
return `Selected "${value}" in ${selector}`;
}
@@ -128,7 +138,7 @@ export async function handleWriteCommand(
if ('locator' in resolved) {
await resolved.locator.hover({ timeout: 5000 });
} else {
await page.hover(resolved.selector, { timeout: 5000 });
await target.locator(resolved.selector).hover({ timeout: 5000 });
}
return `Hovered ${selector}`;
}
@@ -154,11 +164,11 @@ export async function handleWriteCommand(
if ('locator' in resolved) {
await resolved.locator.scrollIntoViewIfNeeded({ timeout: 5000 });
} else {
await page.locator(resolved.selector).scrollIntoViewIfNeeded({ timeout: 5000 });
await target.locator(resolved.selector).scrollIntoViewIfNeeded({ timeout: 5000 });
}
return `Scrolled ${selector} into view`;
}
await page.evaluate(() => window.scrollTo(0, document.body.scrollHeight));
await target.evaluate(() => window.scrollTo(0, document.body.scrollHeight));
return 'Scrolled to bottom';
}
@@ -183,7 +193,7 @@ export async function handleWriteCommand(
if ('locator' in resolved) {
await resolved.locator.waitFor({ state: 'visible', timeout });
} else {
await page.waitForSelector(resolved.selector, { timeout });
await target.locator(resolved.selector).waitFor({ state: 'visible', timeout });
}
return `Element ${selector} appeared`;
}
@@ -248,7 +258,7 @@ export async function handleWriteCommand(
if ('locator' in resolved) {
await resolved.locator.setInputFiles(filePaths);
} else {
await page.locator(resolved.selector).setInputFiles(filePaths);
await target.locator(resolved.selector).setInputFiles(filePaths);
}
const fileInfo = filePaths.map(fp => {
+235 -7
View File
@@ -1323,13 +1323,12 @@ describe('Errors', () => {
}
});
test('chain with invalid JSON throws', async () => {
try {
await handleMetaCommand('chain', ['not json'], bm, async () => {});
expect(true).toBe(false);
} catch (err: any) {
expect(err.message).toContain('Invalid JSON');
}
test('chain with invalid JSON falls back to pipe format', async () => {
// Non-JSON input is now treated as pipe-delimited format
// 'not json' → [["not", "json"]] → "not" is unknown command → error in result
const result = await handleMetaCommand('chain', ['not json'], bm, async () => {});
expect(result).toContain('ERROR');
expect(result).toContain('Unknown command: not');
});
test('chain with no arg throws', async () => {
@@ -1834,3 +1833,232 @@ describe('Chain with cookie-import', () => {
}
});
});
// ─── Network Idle Detection ─────────────────────────────────────
describe('Network idle', () => {
test('click on fetch button waits for XHR to complete', async () => {
await handleWriteCommand('goto', [baseUrl + '/network-idle.html'], bm);
// Click the button that triggers a fetch → networkidle waits for it
await handleWriteCommand('click', ['#fetch-btn'], bm);
// The DOM should be updated by the time click returns
const result = await handleReadCommand('js', ['document.getElementById("result").textContent'], bm);
expect(result).toContain('Data loaded');
});
test('click on static button has no latency penalty', async () => {
await handleWriteCommand('goto', [baseUrl + '/network-idle.html'], bm);
const start = Date.now();
await handleWriteCommand('click', ['#static-btn'], bm);
const elapsed = Date.now() - start;
// Static click should complete well under 2s (the networkidle timeout)
// networkidle resolves immediately when no requests are in flight
expect(elapsed).toBeLessThan(1500);
const result = await handleReadCommand('js', ['document.getElementById("static-result").textContent'], bm);
expect(result).toBe('Static action done');
});
test('fill triggers networkidle wait', async () => {
await handleWriteCommand('goto', [baseUrl + '/forms.html'], bm);
// fill should complete without error (networkidle resolves immediately on static page)
const result = await handleWriteCommand('fill', ['#email', 'idle@test.com'], bm);
expect(result).toContain('Filled');
});
});
// ─── Chain Pipe Format ──────────────────────────────────────────
describe('Chain pipe format', () => {
test('pipe-delimited commands work', async () => {
const result = await handleMetaCommand(
'chain',
[`goto ${baseUrl}/basic.html | js document.title`],
bm,
async () => {}
);
expect(result).toContain('[goto]');
expect(result).toContain('[js]');
expect(result).toContain('Test Page - Basic');
});
test('pipe format with quoted args', async () => {
const result = await handleMetaCommand(
'chain',
[`goto ${baseUrl}/forms.html | fill #email "pipe@test.com"`],
bm,
async () => {}
);
expect(result).toContain('[fill]');
expect(result).toContain('Filled');
// Verify the fill actually worked
const val = await handleReadCommand('js', ['document.querySelector("#email").value'], bm);
expect(val).toBe('pipe@test.com');
});
test('JSON format still works', async () => {
const commands = JSON.stringify([
['goto', baseUrl + '/basic.html'],
['js', 'document.title'],
]);
const result = await handleMetaCommand('chain', [commands], bm, async () => {});
expect(result).toContain('[goto]');
expect(result).toContain('Test Page - Basic');
});
test('pipe format with unknown command includes error', async () => {
const result = await handleMetaCommand(
'chain',
['bogus command'],
bm,
async () => {}
);
expect(result).toContain('ERROR');
expect(result).toContain('Unknown command: bogus');
});
});
// ─── State Persistence ──────────────────────────────────────────
describe('State persistence', () => {
test('state save and load round-trip', async () => {
await handleWriteCommand('goto', [baseUrl + '/basic.html'], bm);
// Set a cookie so we can verify it persists
await handleWriteCommand('cookie', ['state_test=hello'], bm);
// Save state
const saveResult = await handleMetaCommand('state', ['save', 'test-roundtrip'], bm, async () => {});
expect(saveResult).toContain('State saved');
expect(saveResult).toContain('treat as sensitive');
// Navigate away
await handleWriteCommand('goto', [baseUrl + '/forms.html'], bm);
// Load state — should restore to basic.html with cookie
const loadResult = await handleMetaCommand('state', ['load', 'test-roundtrip'], bm, async () => {});
expect(loadResult).toContain('State loaded');
// Verify we're back on basic.html
const url = await handleReadCommand('js', ['location.pathname'], bm);
expect(url).toContain('basic.html');
// Clean up
try {
const { resolveConfig } = await import('../src/config');
const config = resolveConfig();
fs.unlinkSync(`${config.stateDir}/browse-states/test-roundtrip.json`);
} catch {}
});
test('state save rejects invalid names', async () => {
try {
await handleMetaCommand('state', ['save', '../../evil'], bm, async () => {});
expect(true).toBe(false);
} catch (err: any) {
expect(err.message).toContain('alphanumeric');
}
});
test('state save accepts valid names', async () => {
const result = await handleMetaCommand('state', ['save', 'my-state_1'], bm, async () => {});
expect(result).toContain('State saved');
// Clean up
try {
const { resolveConfig } = await import('../src/config');
const config = resolveConfig();
fs.unlinkSync(`${config.stateDir}/browse-states/my-state_1.json`);
} catch {}
});
test('state load rejects missing state', async () => {
try {
await handleMetaCommand('state', ['load', 'nonexistent-state-xyz'], bm, async () => {});
expect(true).toBe(false);
} catch (err: any) {
expect(err.message).toContain('State not found');
}
});
test('state requires action and name', async () => {
try {
await handleMetaCommand('state', [], bm, async () => {});
expect(true).toBe(false);
} catch (err: any) {
expect(err.message).toContain('Usage');
}
});
});
// ─── Frame (Iframe Support) ─────────────────────────────────────
describe('Frame', () => {
test('frame switch to iframe and back', async () => {
await handleWriteCommand('goto', [baseUrl + '/iframe.html'], bm);
// Verify we're on the main page
const mainTitle = await handleReadCommand('js', ['document.getElementById("main-title").textContent'], bm);
expect(mainTitle).toBe('Main Page');
// Switch to iframe by CSS selector
const switchResult = await handleMetaCommand('frame', ['#test-frame'], bm, async () => {});
expect(switchResult).toContain('Switched to frame');
// Verify we can read iframe content
const frameTitle = await handleReadCommand('js', ['document.getElementById("frame-title").textContent'], bm);
expect(frameTitle).toBe('Inside Frame');
// Switch back to main
const mainResult = await handleMetaCommand('frame', ['main'], bm, async () => {});
expect(mainResult).toBe('Switched to main frame');
// Verify we're back on the main page
const mainTitleAgain = await handleReadCommand('js', ['document.getElementById("main-title").textContent'], bm);
expect(mainTitleAgain).toBe('Main Page');
});
test('snapshot shows frame context header', async () => {
await handleWriteCommand('goto', [baseUrl + '/iframe.html'], bm);
await handleMetaCommand('frame', ['#test-frame'], bm, async () => {});
const snap = await handleMetaCommand('snapshot', ['-i'], bm, async () => {});
expect(snap).toContain('[Context: iframe');
// Clean up — return to main
await handleMetaCommand('frame', ['main'], bm, async () => {});
});
test('goto throws error when in frame context', async () => {
await handleWriteCommand('goto', [baseUrl + '/iframe.html'], bm);
await handleMetaCommand('frame', ['#test-frame'], bm, async () => {});
try {
await handleWriteCommand('goto', ['https://example.com'], bm);
expect(true).toBe(false);
} catch (err: any) {
expect(err.message).toContain('Cannot use goto inside a frame');
}
await handleMetaCommand('frame', ['main'], bm, async () => {});
});
test('frame requires argument', async () => {
try {
await handleMetaCommand('frame', [], bm, async () => {});
expect(true).toBe(false);
} catch (err: any) {
expect(err.message).toContain('Usage');
}
});
test('fill works inside iframe', async () => {
await handleWriteCommand('goto', [baseUrl + '/iframe.html'], bm);
await handleMetaCommand('frame', ['#test-frame'], bm, async () => {});
const result = await handleWriteCommand('fill', ['#frame-input', 'hello from frame'], bm);
expect(result).toContain('Filled');
const value = await handleReadCommand('js', ['document.getElementById("frame-input").value'], bm);
expect(value).toBe('hello from frame');
await handleMetaCommand('frame', ['main'], bm, async () => {});
});
});
+30
View File
@@ -0,0 +1,30 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Test Page - Iframe</title>
<style>
body { font-family: sans-serif; padding: 20px; }
iframe { border: 1px solid #ccc; width: 400px; height: 200px; }
</style>
</head>
<body>
<h1 id="main-title">Main Page</h1>
<iframe id="test-frame" name="testframe" srcdoc='
<!DOCTYPE html>
<html>
<body>
<h1 id="frame-title">Inside Frame</h1>
<button id="frame-btn">Frame Button</button>
<input id="frame-input" type="text" placeholder="Type here">
<div id="frame-result"></div>
<script>
document.getElementById("frame-btn").addEventListener("click", () => {
document.getElementById("frame-result").textContent = "Frame button clicked";
});
</script>
</body>
</html>
'></iframe>
</body>
</html>
+30
View File
@@ -0,0 +1,30 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Test Page - Network Idle</title>
<style>
body { font-family: sans-serif; padding: 20px; }
#result { margin-top: 10px; color: green; }
</style>
</head>
<body>
<button id="fetch-btn">Load Data</button>
<div id="result"></div>
<button id="static-btn">Static Action</button>
<div id="static-result"></div>
<script>
document.getElementById('fetch-btn').addEventListener('click', async () => {
// Simulate an XHR that takes 200ms
const res = await fetch('/echo');
const data = await res.json();
document.getElementById('result').textContent = 'Data loaded: ' + Object.keys(data).length + ' headers';
});
document.getElementById('static-btn').addEventListener('click', () => {
// No network activity — purely client-side
document.getElementById('static-result').textContent = 'Static action done';
});
</script>
</body>
</html>