feat: Gemini CLI E2E tests (v0.9.2.0) (#252)
* feat: add Gemini CLI session runner + JSONL parser
Subprocess wrapper for `gemini -p --output-format stream-json --yolo`
that spawns the Gemini CLI and parses NDJSON events (init, message,
tool_use, tool_result, result) into a structured GeminiResult.
Includes 10 unit tests for parseGeminiJSONL covering happy path,
malformed input, empty input, missing fields, and multi-tool scenarios.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: add Gemini CLI E2E tests
Two E2E tests (gemini-discover-skill, gemini-review-findings) that
verify gstack skills work when invoked by the Gemini CLI. Follows
the same pattern as codex-e2e.test.ts — gated by EVALS=1 + binary
availability, diff-based selection via touchfiles, eval persistence.
- Add test/gemini-e2e.test.ts
- Add Gemini entries to E2E_TOUCHFILES and GLOBAL_TOUCHFILES
- Add test:gemini and test:gemini:all scripts to package.json
- Add gemini-e2e.test.ts to test:evals, test:e2e, and ignore list
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: bump version and changelog (v0.9.2.0)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
feat: adversarial spec review loop + skill chaining (v0.9.1.0) (#249)
* feat: add {{SPEC_REVIEW_LOOP}}, {{DESIGN_SKETCH}}, benefits-from resolvers
Three new resolvers in gen-skill-docs.ts:
- {{SPEC_REVIEW_LOOP}}: adversarial subagent reviews documents on 5
dimensions (completeness, consistency, clarity, scope, feasibility)
with convergence guard, quality score, and JSONL metrics
- {{DESIGN_SKETCH}}: generates rough HTML wireframes for UI ideas using
DESIGN.md constraints and design principles, renders via $B
- {{BENEFITS_FROM}}: parses benefits-from frontmatter and generates
skill chaining offer prose (one-hop-max, never blocks)
Also extends TemplateContext with benefitsFrom field and adds inline
YAML frontmatter parsing for the new field.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: /office-hours spec review loop + visual sketch phases
- Phase 4.5 ({{DESIGN_SKETCH}}): for UI ideas, generates rough HTML
wireframe using design principles from {{DESIGN_METHODOLOGY}} and
DESIGN.md, renders via $B, presents screenshot for iteration
- Phase 5.5 ({{SPEC_REVIEW_LOOP}}): adversarial subagent reviews the
design doc before user sees it — catches gaps in completeness,
consistency, clarity, scope, and feasibility
- Adds {{BROWSE_SETUP}} for $B availability in sketch phase
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: skill chaining — plan reviews offer /office-hours
- plan-ceo-review: benefits-from office-hours, offers /office-hours when
no design doc found, mid-session detection when user seems lost,
spec review loop on CEO plan documents
- plan-eng-review: benefits-from office-hours, offers /office-hours when
no design doc found
- One-hop-max chaining: never blocks, max one offer per session
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* test: add validation + E2E tests for spec review, sketch, benefits-from
Unit tests (32 new assertions):
- SPEC_REVIEW_LOOP: 5 dimensions, Agent dispatch, 3 iterations, quality
score, metrics path, convergence guard, graceful failure
- DESIGN_SKETCH: DESIGN.md awareness, wireframe, $B goto/screenshot,
rough aesthetic, skip conditions
- BENEFITS_FROM: prerequisite offer in CEO + eng review, graceful
decline, skills without benefits-from don't get offer
- office-hours structure: spec review loop, adversarial dimensions,
visual sketch section
E2E tests (2 new):
- office-hours-spec-review: verifies agent understands the spec review
loop from SKILL.md
- plan-ceo-review-benefits: verifies agent understands the skill
chaining offer
Touchfiles updated for diff-based test selection.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: bump version and changelog (v0.9.1.0)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
feat: multi-agent support — gstack works on Codex, Gemini CLI, and Cursor (v0.9.0) (#226)
* refactor: host-aware gen-skill-docs + --host codex generation
Refactor gen-skill-docs.ts for multi-agent support:
- Add Host type, HostPaths interface, HOST_PATHS config
- Decompose generatePreamble() into 7 composable sub-functions
- Replace all hardcoded .claude/skills/gstack paths with ctx.paths
- Replace static findTemplates() list with dynamic filesystem scan
- Add --host codex|agents flag (aliases, same output)
- Add processTemplate host routing to .agents/skills/gstack-*/
- Add codexSkillName() with double-prefix prevention
- Add transformFrontmatter() — keeps only name + description for Codex
- Add extractHookSafetyProse() — converts hooks to inline advisory
- Add body text path rewriting for remaining hardcoded paths
- Exclude /codex skill from Codex generation (self-referential)
Claude output is unchanged (verified via --dry-run).
SKILL.md is an open standard: .agents/skills/ works on Codex, Gemini CLI, and Cursor.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: generate Codex/Gemini/Cursor skills into .agents/skills/
Generated 21 skill files for the open SKILL.md standard:
- Output: .agents/skills/gstack-*/SKILL.md (one per skill)
- Frontmatter: name + description only (no allowed-tools/version)
- No .claude/skills/ paths in any generated file
- /codex skill excluded (Claude wrapper, self-referential on Codex)
- Hook skills (careful/freeze/guard) get inline safety prose
- Build script generates both hosts: bun run build
Supported agents (all read .agents/skills/):
- Codex CLI
- Gemini CLI
- Cursor
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: dual-host setup + find-browse for Codex/Gemini/Cursor
- setup: add --host codex|claude|auto flag, install to ~/.codex/skills/
when targeting Codex, auto-detect installed agents
- find-browse: priority chain .codex > .agents > .claude (both
workspace-local and global)
- dev-setup/teardown: create .agents/skills/gstack symlinks for dev mode
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* test: Codex generation tests + CI + docs for multi-agent support
Tests (28 new):
- Codex output path routing, frontmatter validation (name+description only)
- No .claude/skills/ path leaks in Codex output (regression guard)
- /codex skill exclusion, hook→prose conversion, multiline YAML
- --host agents alias, dynamic template discovery
- Codex skill validation + $B command validation
- find-browse priority chain verification
- Replace static ALL_SKILLS list with dynamic filesystem scan
CI:
- Add Codex freshness check to skill-docs workflow
Docs:
- AGENTS.md: Codex-facing project instructions
- README: multi-agent installation section
- CONTRIBUTING: dual-host development workflow
- CHANGELOG: v0.9.0 multi-agent support entry
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: Codex E2E test harness — verify skills work on Codex CLI
New test infrastructure:
- CodexSessionRunner: spawns codex exec, parses JSONL stream, returns
structured results (output, reasoning, toolCalls, tokens)
- JSONL parser ported from Python (codex/SKILL.md.tmpl) to TypeScript
- Temp HOME skill installation for Codex discovery testing
E2E tests (gated behind EVALS=1 + codex + OPENAI_API_KEY):
- codex-discover-skill: installs skill, verifies Codex finds it
- codex-review-findings: runs gstack-review via Codex, validates output
Integrates with existing eval infrastructure:
- Diff-based test selection via touchfiles
- Eval persistence via EvalCollector
- bun run test:codex / test:codex:all convenience scripts
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: bump VERSION to 0.9.0 to match CHANGELOG
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: Codex sidecar paths + setup installs generated skills
Two bugs found by Codex adversarial review:
1. Sidecar path mismatch: generated Codex skills referenced
.agents/skills/gstack-review/checklist.md but setup creates
sidecars at .agents/skills/gstack/review/. Fixed path rewriter
to emit .agents/skills/gstack/review/ (matching setup layout).
2. Setup installed Claude-format source dirs for Codex global
install instead of the generated Codex-format skills. Split
link_skill_dirs into link_claude_skill_dirs (source dirs for
Claude) and link_codex_skill_dirs (generated .agents/skills/
gstack-* dirs for Codex).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* test: comprehensive Codex path rewriting + setup install tests
17 new tests covering:
- Sidecar path rewriting: .claude/skills/review → .agents/skills/gstack/review/
(catches the bug where checklist.md was unreachable at gstack-review/)
- All 4 path rewrite rules tested individually across all skills
- Greptile triage sidecar path correctness
- Ship skill sidecar paths for pre-landing review
- Claude output regression guard: zero Codex paths in any Claude skill
- Setup script validation: separate link functions for Claude vs Codex,
link_codex_skill_dirs reads from .agents/skills/, create_agents_sidecar
links runtime assets (bin, browse, review, qa)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: regenerate Codex skills after investigate rename merge
Remove stale gstack-debug, add gstack-investigate, regenerate all
Codex skills to pick up changes merged from main (investigate rename,
platform-agnostic templates, review helpers).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: Codex E2E uses ~/.codex/ auth, not OPENAI_API_KEY
- Remove OPENAI_API_KEY gate from test prerequisites
- Copy real ~/.codex/ auth config into temp HOME so codex can authenticate
- Increase review test timeout to 540s (codex does thorough 60+ tool call reviews)
- Document in CLAUDE.md that Codex uses its own auth config
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
fix: /qa never refuses browser testing on backend-only changes (#202)
* feat: QA skill never refuses browser testing
Add anti-refusal guardrails to /qa and /qa-only skills. When the user
invokes /qa, the skill must always use the browser — even if the diff
shows only backend/config changes with no obvious UI surface. Falls
back to Quick mode (homepage + top 5 nav targets) when no specific
pages are identified from the diff.
Adds LLM-as-judge eval to verify the anti-refusal behavior.
* chore: bump version and changelog (v0.8.1)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
feat: /codex skill — multi-AI second opinion + proactive suggestions (#197)
* feat: /codex skill — multi-AI second opinion (review, challenge, consult)
Three modes: code review with pass/fail gate, adversarial challenge mode,
and conversational consult with session continuity. First multi-AI skill
in gstack, wrapping OpenAI's Codex CLI.
* feat: integrate /codex into /review, /ship, /plan-eng-review + dashboard
/review offers Codex second opinion after completing its own review.
/ship offers Codex review as optional gate before pushing.
/plan-eng-review offers Codex plan critique after scope challenge.
Review Readiness Dashboard shows Codex Review as optional row.
* chore: bump version and changelog (v0.8.0)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* test: codex skill validation (12 stub tests) + E2E eval test
Stub tests (free tier): verify template content — three modes, gate verdict,
session continuity, cost tracking, cross-model comparison, binary discovery,
error handling, mktemp usage, and integrations into /review, /ship, /plan-eng-review.
E2E test (paid tier): runs /codex review on vulnerable fixture repo via
session-runner, verifies output contains findings and GATE verdict.
* fix: codex auth error message — use codex login, not OPENAI_API_KEY
Codex authenticates via ChatGPT OAuth (codex login), not an env var.
* feat: codex uses high reasoning effort by default
gpt-5.2-codex is the only model available with ChatGPT login.
All commands now use model_reasoning_effort="high" for maximum
depth — the whole point is a thorough second opinion.
* feat: crank codex reasoning to xhigh (maximum)
* feat: per-mode reasoning (high for review/consult, xhigh for challenge) + web search
Review and consult use high reasoning — thorough but not slow.
Challenge (adversarial) uses xhigh — maximum depth for breaking code.
All modes enable web_search_cached so Codex can look up docs/APIs.
* refactor: don't hardcode model — use codex default (always latest)
* feat: JSONL output for codex challenge + consult modes
Use --json flag to parse codex's JSONL events, extracting reasoning
traces ([codex thinking]), tool calls ([codex ran]), and token counts.
This gives richer output than the -o flag alone — you can see what
codex thought through before its answer.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: only persist codex-review log when code review actually ran
Don't write a codex-review entry to reviews.jsonl when only the
adversarial challenge (option B) was selected — there's no gate
verdict to record, and a false entry misleads the Review Readiness
Dashboard into thinking a code review happened.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat: add codex plan review option to /plan-eng-review
After scope challenge (Step 0), offer to have Codex independently
review the plan with a brutally honest tech reviewer persona.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* test: update e2e test for codex skill
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: codex integration bugs — plan content, review persistence, quoting, stderr
- plan-eng-review: Codex now reads the plan file itself instead of inlining
content as a CLI arg (avoids ARG_MAX for large plans)
- review: add missing echo to persist codex-review results to reviews.jsonl
- codex: consult mode uses $TMPERR (mktemp) instead of hardcoded stderr path
- codex + review: quote $SLUG/$BRANCH_SLUG in review log paths
- codex: scope plan lookup to current project, warn on cross-project fallback
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: add .context/ to .gitignore to prevent session ID leaks
Codex consult mode stores session IDs in .context/codex-session-id.
Without this ignore rule, session IDs could leak into commits.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: proactive skill suggestions + opt-out + trigger phrase tests
- Preamble reads proactive config via gstack-config
- Root SKILL.md.tmpl has lifecycle map (stage → skill suggestion)
- Users can opt out ("stop suggesting") / opt in ("be proactive again")
- Restored trigger phrase validation tests (16 skills × "Use when" check)
- Added missing "Use when" trigger phrases to /debug and /office-hours
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: update changelog for v0.8.0 — add proactive suggestions note
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
feat: natural language skill routing + proactive suggestions (v0.7.1) (#195)
* feat: add trigger phrases to /debug and /office-hours
These two skills had zero "Use when asked to..." phrases, making them
completely invisible to natural language. Users saying "debug this" or
"brainstorm an idea" would get no skill invocation.
* feat: add proactive triggers to all workflow skills
Every skill now has "Proactively suggest when..." language so Claude
surfaces skills at natural moments — not just when the user says
specific trigger phrases.
* feat: lifecycle map + proactive preference system
Root gstack description now includes a developer workflow guide mapping
12 stages to skills. Preamble reads proactive preference via gstack-config.
Users can opt out with "stop suggesting things" and re-enable with
"be proactive again" — natural language toggle, no CLI needed.
* test: 11 journey-stage E2E routing tests + trigger phrase validation
Each test simulates a real development stage (ideation, plan review,
debug, QA, ship, retro...) with realistic project context and verifies
the right skill fires from natural language alone. 11/11 pass.
* chore: bump version and changelog (v0.7.1)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
feat: interactive /plan-design-review + CEO invokes designer + 100% coverage (v0.6.4) (#149)
* refactor: rename qa-design-review → design-review
The "qa-" prefix was confusing — this is the live-site design audit with
fix loop, not a QA-only report. Rename directory and update all references
across docs, tests, scripts, and skill templates.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: interactive /plan-design-review + CEO invokes designer
Rewrite /plan-design-review from report-only grading to an interactive
plan-fixer that rates each design dimension 0-10, explains what a 10
looks like, and edits the plan to get there. Parallel structure with
/plan-ceo-review and /plan-eng-review — one issue = one AskUserQuestion.
CEO review now detects UI scope and invokes the designer perspective
when the plan has frontend/UX work, so you get design review
automatically when it matters.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* test: validation + touchfile entries for 100% coverage
Add design-consultation to command/snapshot flag validation. Add 4
skills to contributor mode validation (plan-design-review,
design-review, design-consultation, document-release). Add 2 templates
to hardcoded branch check. Register touchfile entries for 10 new
LLM-judge tests and 1 new E2E test.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* test: LLM-judge for 10 skills + gstack-upgrade E2E
Add LLM-judge quality evals for all uncovered skills using a DRY
runWorkflowJudge helper with section marker guards. Add real E2E
test for gstack-upgrade using mock git remote (replaces test.todo).
Add plan-edit assertion to plan-design-review E2E.
14/15 skills now at full coverage. setup-browser-cookies remains
deferred (needs real browser).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs: add bisect commit style to CLAUDE.md
All commits should be single logical changes, split before pushing.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: bump version and changelog (v0.6.4.0)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
feat: design review lite in /review and /ship + gstack-diff-scope (v0.6.3) (#142)
* feat: gstack-diff-scope helper + design review checklist
bin/gstack-diff-scope categorizes branch changes into SCOPE_FRONTEND,
SCOPE_BACKEND, SCOPE_PROMPTS, SCOPE_TESTS, SCOPE_DOCS, SCOPE_CONFIG.
review/design-checklist.md is a 20-item code-level checklist with
HIGH/MEDIUM/LOW confidence tags for detecting design anti-patterns
from source code.
* feat: integrate design review lite into /review and /ship
Add generateDesignReviewLite() resolver, insert {{DESIGN_REVIEW_LITE}}
partial in review Step 4.5 and ship Step 3.5. Update dashboard to
recognize design-review-lite entries. Ship pre-flight uses
gstack-diff-scope for smarter design review recommendations.
* test: E2E eval for design review lite detection
Planted CSS/HTML fixtures with 7 design anti-patterns. E2E test
verifies /review catches >= 4 of 7 (Papyrus font, 14px body text,
outline:none, !important, purple gradient, generic hero copy,
3-column feature grid).
* chore: bump version and changelog (v0.6.3.0)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
feat: diff-based test selection for E2E and LLM-judge evals (v0.6.1.0) (#139)
* feat: diff-based test selection for E2E and LLM-judge evals
Each test declares file dependencies in a TOUCHFILES map. The test runner
checks git diff against the base branch and only runs tests whose
dependencies were modified. Global touchfiles (session-runner, eval-store,
gen-skill-docs) trigger all tests.
New scripts: test:e2e:all, test:evals:all, eval:select
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* chore: bump version and changelog (v0.6.1.0)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: plan-design-review-audit eval — bump turns to 30, add efficiency hints
The test was flaky at 20 turns because the agent reads a 300-line SKILL.md,
navigates, extracts design data, and writes a report. Added hints to skip
preamble/batch commands/write early while still testing the real SKILL.md.
Now completes in ~13 turns consistently.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
feat: QA restructure, browser ref staleness, eval efficiency metrics (v0.4.0) (#83)
* feat: browser ref staleness detection via async count() validation
resolveRef() now checks element count to detect stale refs after page
mutations (e.g. SPA navigation). RefEntry stores role+name metadata
for better diagnostics. 3 new snapshot tests for staleness detection.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat: qa-only skill, qa fix loop, plan-to-QA artifact flow
Add /qa-only (report-only, Edit tool blocked), restructure /qa with
find-fix-verify cycle, add {{QA_METHODOLOGY}} DRY placeholder for
shared methodology. /plan-eng-review now writes test-plan artifacts
to ~/.gstack/projects/<slug>/ for QA consumption.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat: eval efficiency metrics — turns, duration, commentary across all surfaces
Add generateCommentary() for natural-language delta interpretation,
per-test turns/duration in comparison and summary output, judgePassed
unit tests, 3 new E2E tests (qa-only, qa fix loop, plan artifact).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* chore: bump version and changelog (v0.4.0)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs: update ARCHITECTURE, BROWSER, CONTRIBUTING, README for v0.4.0
- ARCHITECTURE: add ref staleness detection section, update RefEntry type
- BROWSER: add ref staleness paragraph to snapshot system docs
- CONTRIBUTING: update eval tool descriptions with commentary feature
- README: fix missing qa-only in project-local uninstall command
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs: add user-facing benefit descriptions to v0.4.0 changelog
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Merge pull request #55 from garrytan/v0.3.6-qa-upgrades
feat: E2E observability + eval infrastructure + all skills templated
fix: auto-clear stale heartbeat when process is dead
Add PID to heartbeat file. eval-watch checks process.kill(pid, 0) and
auto-deletes the heartbeat when the PID is no longer alive — no manual
cleanup needed after crashed/killed E2E runs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
fix: never clean up observability artifacts — partial file persists after finalize
Removing the _partial-e2e.json deletion from finalize(). These are small files
on a local disk and their persistence is the whole point of observability.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
fix: detect is_error from claude -p result line (ConnectionRefused was PASS)
claude -p can return subtype="success" with is_error=true when the API is
unreachable. Previously we only checked subtype, so API failures silently
passed. Now check is_error first and report as 'error_api'.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
feat: eval-watch dashboard + observability unit tests (15 tests, 11 codepaths)
eval-watch: live terminal dashboard reads heartbeat + partial file every 1s,
shows completed/running tests, stale detection (>10min), --tail flag for
progress.log tail. Pure renderDashboard() function for testability.
observability.test.ts: unit tests for sanitizeTestName, heartbeat schema,
progress.log format, NDJSON file naming, savePartial() with _partial flag,
finalize() cleanup, diagnostic fields, watcher rendering, stale detection,
and non-fatal I/O guarantees.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
feat: add E2E observability — heartbeat, progress.log, NDJSON persistence, savePartial()
session-runner: atomic heartbeat file (e2e-live.json), per-run log directory
(~/.gstack-dev/e2e-runs/{runId}/), progress.log + per-test NDJSON persistence,
failure transcripts to persistent run dir instead of tmpdir.
eval-store: 3 new diagnostic fields (exit_reason, timeout_at_turn, last_tool_call),
savePartial() writes _partial-e2e.json after each addTest() for crash resilience,
finalize() cleans up partial file.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
fix: remove false-positive Exit code 1 pattern, fix NEEDS_SETUP test, update QA tests
- Remove /Exit code 1/ from BROWSE_ERROR_PATTERNS — too broad, matches any
bash command exit code in the transcript (e.g., git diff, test commands).
Remaining patterns (Unknown command, Unknown snapshot flag, binary not found,
server failed, no such file) are specific to browse errors.
- Fix NEEDS_SETUP E2E test — accepts READY when global binary exists at
~/.claude/skills/gstack/browse/dist/browse (which it does on dev machines).
Test now verifies the setup block handles missing local binary gracefully.
- Update QA skill structure validation tests to match current qa/SKILL.md
template content (phases renamed, modes replaced tiers, output structure).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
feat: eval persistence with auto-compare against previous run
EvalCollector accumulates test results during eval runs, writes JSON to
~/.gstack-dev/evals/{version}-{branch}-{tier}-{timestamp}.json, prints
a summary table, and automatically compares against the previous run.
- EvalCollector class with addTest() / finalize() / summary table
- findPreviousRun() prefers same branch, falls back to any branch
- compareEvalResults() matches tests by name, detects improved/regressed
- extractToolSummary() counts tool types from transcript events
- formatComparison() renders delta table with per-test + aggregate diffs
- Wire into skill-e2e.test.ts (recordE2E helper) and skill-llm-eval.test.ts
- 19 unit tests for collector + comparison functions
- schema_version: 1 for forward compatibility
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
feat: stream-json NDJSON parser for real-time E2E progress
Switch session-runner from buffered `--output-format json` to streaming
`--output-format stream-json --verbose`. Parses NDJSON line-by-line for
real-time tool-by-tool progress on stderr during 3-5 min E2E runs.
- Extract testable `parseNDJSON()` function (pure, no I/O)
- Count turns per assistant event (not per text block)
- Add `transcript: any[]` to SkillTestResult, remove dead `messages` field
- Reconstruct allText from transcript for browse error scanning
- 8 unit tests for parser (malformed lines, empty input, turn counting)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Merge remote-tracking branch 'origin/main' into v0.3.6-qa-upgrades
# Conflicts:
# test/skill-e2e.test.ts