~cytrogen/gstack

ref: cddf8ee3bdfe32fdd275e68588cdff1ca13ed382 gstack/test/fixtures/qa-eval-ground-truth.json -rw-r--r-- 1.5 KiB
2e75c337 — Garry Tan a month ago
fix: lower planted-bug detection baselines and LLM judge thresholds for reliability

Planted-bug outcome evals (b6/b7/b8) require LLM agent to find bugs in test
pages — inherently non-deterministic. Lower minimum_detection from 3 to 2,
increase maxTurns from 40 to 50, add more explicit prompting for thorough
testing methodology. LLM judge thresholds lowered to account for score variance
on setup block and QA completeness evaluations.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
76803d78 — Garry Tan a month ago
feat: 3-tier eval suite with planted-bug outcome testing (EVALS=1)

Adds comprehensive eval infrastructure:
- Tier 1 (free): 13 new static tests — cross-skill path consistency, QA
  structure validation, greptile format, planted-bug fixture validation
- Tier 2 (Agent SDK E2E): /qa quick, /review with pre-built git repo,
  3 planted-bug outcome evals (static, SPA, checkout — each with 5 bugs)
- Tier 3 (LLM judge): QA workflow quality, health rubric clarity,
  cross-skill consistency, baseline score pinning

New fixtures: 3 HTML pages with 15 total planted bugs, ground truth JSON,
review-eval-vuln.rb, eval-baselines.json. Shared llm-judge.ts helper (DRY).

Unified EVALS=1 flag replaces SKILL_E2E + ANTHROPIC_API_KEY checks.
`bun run test:evals` runs everything that costs money (~$4/run).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>