fix: update check preamble exits 1 when up to date — convert all skills to .tmpl
The `[ -n "$_UPD" ] && echo "$_UPD"` line in 5 skills was missing `|| true`,
causing exit code 1 when the update check finds no update (empty $_UPD).
Fix: convert ship/, review/, plan-ceo-review/, plan-eng-review/, retro/ to
.tmpl templates using {{UPDATE_CHECK}} placeholder (same as browse/qa/etc).
All 9 skills now generated from templates — preamble changes propagate everywhere.
Also: regenerates qa/SKILL.md which had drifted from its template, adds 12 tests
validating the update check preamble exits 0 in all skills, removes completed TODO.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
feat: eval CLI tools + docs cleanup
Add eval:list, eval:compare, eval:summary CLI scripts for exploring
eval history from ~/.gstack-dev/evals/. eval:compare reuses the shared
comparison functions from eval-store.ts.
- eval:list: sorted table with branch/tier/cost filters
- eval:compare: thin wrapper around compareEvalResults + formatComparison
- eval:summary: aggregate stats, flaky test detection, branch rankings
- Remove unused @anthropic-ai/claude-agent-sdk from devDependencies
- Update CLAUDE.md: streaming docs, eval CLI commands, remove Agent SDK refs
- Add GH Actions eval upload (P2) and web dashboard (P3) to TODOS.md
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
feat: eval persistence with auto-compare against previous run
EvalCollector accumulates test results during eval runs, writes JSON to
~/.gstack-dev/evals/{version}-{branch}-{tier}-{timestamp}.json, prints
a summary table, and automatically compares against the previous run.
- EvalCollector class with addTest() / finalize() / summary table
- findPreviousRun() prefers same branch, falls back to any branch
- compareEvalResults() matches tests by name, detects improved/regressed
- extractToolSummary() counts tool types from transcript events
- formatComparison() renders delta table with per-test + aggregate diffs
- Wire into skill-e2e.test.ts (recordE2E helper) and skill-llm-eval.test.ts
- 19 unit tests for collector + comparison functions
- schema_version: 1 for forward compatibility
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
feat: stream-json NDJSON parser for real-time E2E progress
Switch session-runner from buffered `--output-format json` to streaming
`--output-format stream-json --verbose`. Parses NDJSON line-by-line for
real-time tool-by-tool progress on stderr during 3-5 min E2E runs.
- Extract testable `parseNDJSON()` function (pure, no I/O)
- Count turns per assistant event (not per text block)
- Add `transcript: any[]` to SkillTestResult, remove dead `messages` field
- Reconstruct allText from transcript for browse error scanning
- 8 unit tests for parser (malformed lines, empty input, turn counting)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Merge remote-tracking branch 'origin/main' into v0.3.6-qa-upgrades
# Conflicts:
# test/skill-e2e.test.ts
fix: rewrite session-runner to claude -p subprocess, lower flaky baselines
Session runner now spawns `claude -p` as a subprocess instead of using
Agent SDK query(), which fixes E2E tests hanging inside Claude Code.
Also lowers command_reference completeness baseline to 3 (flaky oscillation),
adds test:e2e script, and updates CLAUDE.md.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
fix: browse binary discovery broken for agents (v0.3.5) (#44)
* fix: replace find-browse with direct path in SKILL.md setup blocks
Agents were skipping the find-browse binary and guessing bin/browse
(wrong path). Now the setup block explicitly checks browse/dist/browse
with workspace-local priority, global fallback.
Also adds || true to update check to prevent misleading exit code 1.
Adds {{UPDATE_CHECK}} and {{BROWSE_SETUP}} template placeholders to
gen-skill-docs.ts so all skills share a single source of truth.
* refactor: convert qa/ and setup-browser-cookies/ to .tmpl templates
Replaces hardcoded update check and find-browse blocks with
{{UPDATE_CHECK}} and {{BROWSE_SETUP}} placeholders. Both skills
are now generated from templates via gen-skill-docs.
* test: add e2e and LLM eval tests for SKILL.md setup block
- 3 Agent SDK e2e tests: happy path, NEEDS_SETUP, non-git-repo
- LLM eval: setup block clarity + actionability >= 4
- New error pattern: 'no such file or directory.*browse'
These tests catch the exact failure mode where agents can't discover
the browse binary via SKILL.md instructions.
* chore: bump version and changelog (v0.3.5)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
simplify: one command for evals — bun run test:evals
Remove test:eval, test:e2e, test:all. Just two commands:
- bun test (free)
- bun run test:evals (everything that costs money)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
fix: pass all LLM evals — severity defs, rubric edge cases, EVALS=1 flag
- Add severity classification to qa/SKILL.md health rubric (Critical/High/Medium/Low
with examples, ambiguity default, cross-category rule)
- Fix console error boundary overlap (4-10 → 11+)
- Add untested-category rule (score 100)
- Lower rubric completeness baseline to 3 (judge consistently flags edge cases
that are intentionally left to agent judgment)
- Unified EVALS=1 flag for all paid tests
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
feat: 3-tier eval suite with planted-bug outcome testing (EVALS=1)
Adds comprehensive eval infrastructure:
- Tier 1 (free): 13 new static tests — cross-skill path consistency, QA
structure validation, greptile format, planted-bug fixture validation
- Tier 2 (Agent SDK E2E): /qa quick, /review with pre-built git repo,
3 planted-bug outcome evals (static, SPA, checkout — each with 5 bugs)
- Tier 3 (LLM judge): QA workflow quality, health rubric clarity,
cross-skill consistency, baseline score pinning
New fixtures: 3 HTML pages with 15 total planted bugs, ground truth JSON,
review-eval-vuln.rb, eval-baselines.json. Shared llm-judge.ts helper (DRY).
Unified EVALS=1 flag replaces SKILL_E2E + ANTHROPIC_API_KEY checks.
`bun run test:evals` runs everything that costs money (~$4/run).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
feat: daily update check + /gstack-upgrade skill (v0.3.4) (#42)
* feat: add daily update check script + /gstack-upgrade skill
bin/gstack-update-check: pure bash, checks VERSION against remote once/day,
outputs UPGRADE_AVAILABLE or JUST_UPGRADED. Uses ~/.gstack/ for state.
gstack-upgrade/SKILL.md: new skill with inline upgrade flow for all preambles.
Detects global-git, local-git, vendored installs. Shows What's New from CHANGELOG.
browse/test/gstack-update-check.test.ts: 10 test cases covering all branch paths.
* refactor: remove version check from find-browse, simplify to binary locator
Delete checkVersion(), readCache(), writeCache(), fetchRemoteSHA(),
resolveSkillDir(), CacheEntry interface, REPO_URL/CACHE_PATH/CACHE_TTL
constants, and META output from find-browse.ts.
Version checking is now handled by bin/gstack-update-check (previous commit).
* feat: add update check preamble to all 9 skills
Every skill now runs bin/gstack-update-check on invocation. If an upgrade
is available, reads gstack-upgrade/SKILL.md inline upgrade flow.
Also adds AskUserQuestion to 5 skills that lacked it (gstack root, browse,
qa, retro, setup-browser-cookies) and Bash to plan-eng-review.
Simplifies qa and setup-browser-cookies setup blocks (removes META parsing).
* chore: bump version and changelog (v0.3.4)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: remove unused import + add corrupt cache test
Address pre-landing review findings:
- Remove unused mkdirSync import from gstack-update-check.test.ts
- Add Path I test: corrupt cache file falls through to remote fetch
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Merge remote-tracking branch 'origin/main' into v0.3.5-qa-upgrades
fix: enrich SKILL.md docs to pass LLM evals, upgrade judge to Sonnet 4.6 (#43)
* fix: enrich command descriptions and snapshot flags for LLM eval quality
14 command descriptions enriched with specific arg formats, valid values,
error behavior, and return types. Fixed header usage from <name> <value>
to <name>:<value>. Added cookie usage syntax. Snapshot flags now show
long names, ref numbering, and output format examples.
* refactor: auto-generate server.ts help text from COMMAND_DESCRIPTIONS
Replace hand-maintained help block with generateHelpText() that reads
from COMMAND_DESCRIPTIONS and SNAPSHOT_FLAGS. Eliminates help text
drift from source of truth.
* test: add usage consistency and pipe guard tests
Usage consistency test cross-checks Usage: patterns in implementation
against COMMAND_DESCRIPTIONS using structural skeleton comparison.
Pipe guard test ensures descriptions don't contain | which would break
markdown table rendering.
* chore: upgrade eval judge to Sonnet 4.6, update changelog
Switch LLM-as-judge evals from Haiku to Sonnet 4.6 for more stable,
nuanced scoring. Add changelog entry for all eval improvements.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
feat: dual greptile-history paths (per-project + global)
- Suppressions read from ~/.gstack/projects/{slug}/greptile-history.md
- Triage outcomes write to both per-project and global files
- greptile-triage.md: remote-slug derivation, dual-write instructions
- review/SKILL.md + ship/SKILL.md: updated save path references
- TODO: add smart default QA tier (P2, S)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
feat: QA test plan tiers with per-page risk scoring
Rewrite qa/SKILL.md to v2.0:
- Smart test plan generation with Quick/Standard/Exhaustive tiers
- Per-page risk heuristics (forms=HIGH, CSS=LOW, tests=SKIP)
- Reports persist to ~/.gstack/projects/{slug}/qa-reports/
- QA run index with bidirectional links between reports
- Report metadata: branch, commit, PR, tier
- Auto-open preference saved to ~/.gstack/config.json
- PR comment integration via gh
- file:// link output on completion
feat: add remote slug helper and auto-gitignore for .gstack/
- getRemoteSlug() in config.ts: parses git remote origin → owner-repo format
- browse/bin/remote-slug: shell helper for SKILL.md use (BSD sed compatible)
- ensureStateDir() now appends .gstack/ to project .gitignore if not present
- setup creates ~/.gstack/projects/ global state directory
- 7 new tests: 4 gitignore behavior + 3 remote slug parsing
chore: regenerate SKILL.md from template
v0.3.3 updated SKILL.md.tmpl but the generated output was stale.
Removes deprecated META:UPDATE_AVAILABLE setup flow.
feat: SKILL.md template system, 3-tier testing, DX tools (v0.3.3) (#41)
* refactor: extract command registry to commands.ts, add SNAPSHOT_FLAGS metadata
- NEW: browse/src/commands.ts — command sets + COMMAND_DESCRIPTIONS + load-time validation (zero side effects)
- server.ts imports from commands.ts instead of declaring sets inline
- snapshot.ts: SNAPSHOT_FLAGS array drives parseSnapshotArgs (metadata-driven, no duplication)
- All 186 existing tests pass
* feat: SKILL.md template system with auto-generated command references
- SKILL.md.tmpl + browse/SKILL.md.tmpl with {{COMMAND_REFERENCE}} and {{SNAPSHOT_FLAGS}} placeholders
- scripts/gen-skill-docs.ts generates SKILL.md from templates (supports --dry-run)
- Build pipeline runs gen:skill-docs before binary compilation
- Generated files have AUTO-GENERATED header, committed to git
* test: Tier 1 static validation — 34 tests for SKILL.md command correctness
- test/helpers/skill-parser.ts: extracts $B commands from code blocks, validates against registry
- test/skill-parser.test.ts: 13 parser/validator unit tests
- test/skill-validation.test.ts: 13 tests validating all SKILL.md files + registry consistency
- test/gen-skill-docs.test.ts: 8 generator tests (categories, sorting, freshness)
* feat: DX tools (skill:check, dev:skill) + Tier 2 E2E test scaffolding
- scripts/skill-check.ts: health summary for all SKILL.md files (commands, templates, freshness)
- scripts/dev-skill.ts: watch mode for template development
- test/helpers/session-runner.ts: Agent SDK wrapper for E2E skill tests
- test/skill-e2e.test.ts: 2 E2E tests + 3 stubs (auto-skip inside Claude Code sessions)
- E2E tests must run from plain terminal: SKILL_E2E=1 bun test test/skill-e2e.test.ts
* ci: SKILL.md freshness check on push/PR + TODO updates
- .github/workflows/skill-docs.yml: fails if generated SKILL.md files are stale
- TODO.md: add E2E cost tracking and model pinning to future ideas
* fix: restore rich descriptions lost in auto-generation
- Snapshot flags: add back value hints (-d <N>, -s <sel>, -o <path>)
- Snapshot flags: restore parenthetical context (@e refs, @c refs, etc.)
- Commands: is → includes valid states enum
- Commands: console → notes --errors filter behavior
- Commands: press → lists common keys (Enter, Tab, Escape)
- Commands: cookie-import-browser → describes picker UI
- Commands: dialog-accept → specifies alert/confirm/prompt
- Tips: restore → arrow (was downgraded to ->)
* test: quality evals for generated SKILL.md descriptions
Catches the exact regressions we shipped and caught in review:
- Snapshot flags must include value hints (-d <N>, -s <sel>, -o <path>)
- is command must list all valid states (visible/hidden/enabled/...)
- press command must list example keys (Enter, Tab, Escape)
- console command must describe --errors behavior
- Snapshot -i must mention @e refs, -C must mention @c refs
- All descriptions must be >= 8 chars (no empty stubs)
- Tips section must use → not ->
* feat: LLM-as-judge evals for SKILL.md documentation quality
4 eval tests using Anthropic API (claude-haiku, ~$0.01-0.03/run):
- Command reference table: clarity/completeness/actionability >= 4/5
- Snapshot flags section: same thresholds
- browse/SKILL.md overall quality
- Regression: generated version must score >= hand-maintained baseline
Requires ANTHROPIC_API_KEY. Auto-skips without it.
Run: bun run test:eval (or ANTHROPIC_API_KEY=sk-... bun test test/skill-llm-eval.test.ts)
* chore: bump version to 0.3.3, update changelog
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs: add ARCHITECTURE.md, update CLAUDE.md and CONTRIBUTING.md
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat: conductor.json lifecycle hooks + .env propagation across worktrees
bin/dev-setup now copies .env from main worktree so API keys carry
over to Conductor workspaces automatically. conductor.json wires up
setup and archive hooks.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs: complete CHANGELOG for v0.3.3 (architecture, conductor, .env)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
feat: v0.3.2 — project-local state, diff-aware QA, Greptile integration (#36)
* fix: cookie import picker returns JSON instead of HTML
jsonResponse() was defined at module scope but referenced `url` which
only existed as a parameter of handleCookiePickerRoute(). Every API call
crashed, the catch block also crashed, and Bun returned a default HTML
page that the frontend couldn't parse as JSON.
Thread port via corsOrigin() helper and options objects. Add route-level
tests to prevent this class of bug from shipping again.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat: add help command to browse server
Agents that don't have SKILL.md loaded (or misread flags) had no way to
self-discover the CLI. The help command returns a formatted reference of
all commands and snapshot flags.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat: version-aware find-browse with META signal protocol
Agents in other workspaces found stale browse binaries that were missing
newer flags. find-browse now compares the local binary's git SHA against
origin/main via git ls-remote (4hr cache), and emits META:UPDATE_AVAILABLE
when behind. SKILL.md setup checks parse META signals and prompt the user
to update.
- New compiled binary: browse/dist/find-browse (TypeScript, testable)
- Bash shim at browse/bin/find-browse delegates to compiled binary
- .version file written at build time with git commit SHA
- Build script compiles both browse and find-browse binaries
- Graceful degradation: offline, missing .version, corrupt cache all skip check
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* chore: clean up .bun-build temp files after compile
bun build --compile leaves ~58MB temp files in the working directory.
Add rm -f .*.bun-build to the build script to clean up after each build.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: make help command reachable by removing it from META_COMMANDS
help was in META_COMMANDS, so it dispatched to handleMetaCommand() which
threw "Unknown meta command: help". Removing it from the set lets the
dedicated else-if handler in handleCommand() execute correctly.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* chore: bump version and changelog (v0.3.2)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat: add shared Greptile comment triage reference doc
Shared reference for fetching, filtering, and classifying Greptile
review comments on GitHub PRs. Used by both /review and /ship skills.
Includes parallel API fetching, suppressions check, classification
logic, reply APIs, and history file writes.
* feat: make /review and /ship Greptile-aware
/review: Step 2.5 fetches and classifies Greptile comments, Step 5
resolves them with AskUserQuestion for valid issues and false positives.
/ship: Step 3.75 triages Greptile comments between pre-landing review
and version bump. Adds Greptile Review section to PR body in Step 8.
Re-runs tests if any Greptile fixes are applied.
* feat: add Greptile batting average to /retro
Reads ~/.gstack/greptile-history.md, computes signal ratio
(valid catches vs false positives), includes in metrics table,
JSON snapshot, and Code Quality Signals narrative.
* docs: add Greptile integration section to README
Personal endorsement, two-layer review narrative, full UX walkthrough
transcript, skills table updates. Add Greptile training feedback loop
to TODO.md future ideas.
* feat: add local dev mode for testing skills from within the repo
bin/dev-setup creates .claude/skills/gstack symlink to the working tree
so Claude Code discovers skills locally. bin/dev-teardown cleans up.
DEVELOPING_GSTACK.md documents the workflow.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: narrow gitignore to .claude/skills/ instead of all .claude/
Avoids ignoring legitimate Claude Code config like settings.json or CLAUDE.md.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs: rename DEVELOPING_GSTACK.md to CONTRIBUTING.md
Rewritten as a contributor-friendly guide instead of a dry plan doc.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs: explain why dev-setup is needed in CONTRIBUTING.md quick start
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat: add browser interaction guidance to CLAUDE.md
Prevents Claude from using mcp__claude-in-chrome__* tools instead of /browse.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat: add shared config module for project-local browse state
Centralizes path resolution (git root detection, state dir, log paths) into
config.ts. Both cli.ts and server.ts import from it, eliminating duplicated
PORT_OFFSET/BROWSE_PORT/STATE_FILE logic.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat: rewrite port selection to use random ports
Replace CONDUCTOR_PORT magic offset and 9400-9409 scan with random port
10000-60000. Atomic state file writes, log paths from config module,
binaryVersion field for auto-restart on update.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat: move browse state from /tmp to project-local .gstack/
CLI now uses config module for state paths, passes BROWSE_STATE_FILE to
spawned server. Adds version mismatch auto-restart, legacy /tmp cleanup
with PID verification, and removes stale global install fallback.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: update crash log path reference to .gstack/
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* test: add config tests and update CLI lifecycle test
14 new tests for config resolution, ensureStateDir, readVersionHash,
resolveServerScript, and version mismatch detection. Remove obsolete
CONDUCTOR_PORT/BROWSE_PORT filtering from commands.test.ts.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs: update BROWSER.md and TODO.md for project-local state
Replace /tmp paths with .gstack/, remove CONDUCTOR_PORT docs, document
random port selection and per-project isolation. Add server bundling TODO.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs: update README, CHANGELOG, and CONTRIBUTING for v0.3.2
- README: replace Conductor-aware language with project-local isolation,
add Greptile setup note
- CHANGELOG: comprehensive v0.3.2 entry with all state management changes
- CONTRIBUTING: add instructions for testing branches in other repos
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat: add diff-aware mode to /qa — auto-tests affected pages from branch diff
When on a feature branch, /qa now reads git diff main, identifies affected
pages/routes from changed files, and tests them automatically. No URL required.
The most natural flow: write code, /ship, /qa.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* chore: update CHANGELOG for complete v0.3.2 coverage
Add missing entries: diff-aware QA mode, Greptile integration,
local dev mode, crash log path fix, README/SKILL.md updates.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>