~cytrogen/gstack

ref: b2b380bfcc39107f9fdafc17e61bd74cc34fb537 gstack/test/telemetry.test.ts -rw-r--r-- 16.4 KiB
cdd6f786 — Garry Tan 10 days ago
feat: community wave — 7 fixes, relink, sidebar Write, discoverability (v0.13.5.0) (#641)

* test: add 16 failing tests for 6 community fixes

Tests-first for all fixes in this PR wave:
- #594 discoverability: gstack tag in descriptions, 120-char first line
- #573 feature signals: ship/SKILL.md Step 4 detection
- #510 context warnings: no preemptive warnings in generated files
- #474 Safety Net: no find -delete in generated files
- #467 telemetry: JSONL writes gated by _TEL conditional
- #584 sidebar: Write in allowedTools, stderr capture
- #578 relink: prefixed/flat symlinks, cleanup, error, config hook

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: replace find -delete with find -exec rm for Safety Net (#474)

-delete is a non-POSIX extension that fails on Safety Net environments.
-exec rm {} + is POSIX-compliant and works everywhere.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: gate local JSONL writes by telemetry setting (#467)

When telemetry is off, nothing is written anywhere — not just remote,
but local JSONL too. Clean trust contract: off means off everywhere.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: remove preemptive context warnings from plan-eng-review (#510)

The system handles context compaction automatically. Preemptive warnings
waste tokens and create false urgency. Skills should not warn about
context limits — just describe the compression priority order.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: add (gstack) tag to skill descriptions for discoverability (#594)

Every SKILL.md.tmpl description now contains "gstack" on the last line,
making skills findable in Claude Code's command palette. First-line hooks
stay under 120 chars. Split ship description to fix wrapping.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: auto-relink skill symlinks on prefix config change (#578)

New bin/gstack-relink creates prefixed (gstack-*) or flat symlinks
based on skill_prefix config. gstack-config auto-triggers relink
when skill_prefix changes. Setup guards against recursive calls
with GSTACK_SETUP_RUNNING env var.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: add feature signal detection to version bump heuristic (#573)

/ship Step 4 now checks for feature signals (new routes, migrations,
test+source pairs, feat/ branches) when deciding version bumps.
PATCH requires no feature signals. MINOR asks the user if any signal
is detected or 500+ lines changed.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: sidebar Write tool, stderr capture, cross-platform URL opener (#584)

Add Write to sidebar allowedTools (both sidebar-agent.ts and server.ts).
Write doesn't expand attack surface beyond what Bash already provides.
Replace empty stderr handler with buffer capture for better error
diagnostics. New bin/gstack-open-url for cross-platform URL opening.

Does NOT include Search Before Building intro flow (deferred).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: update sidebar-security test for Write tool addition

The fallback allowedTools string now includes Write, matching the
sidebar-agent.ts change from commit 68dc957.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: bump version and changelog (v0.13.5.0)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: prevent gstack-relink from double-prefixing gstack-upgrade

gstack-relink now checks if a skill directory is already named gstack-*
before prepending the prefix. Previously, setting skill_prefix=true would
create gstack-gstack-upgrade, breaking the /gstack-upgrade command.

Matches setup script behavior (setup:260) which already has this guard.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: add double-prefix fix to changelog

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: remove .factory/ from git tracking and add to .gitignore

Generated Factory Droid skills are build output, same as .agents/.
They should not be committed to the repo.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
7450b516 — Garry Tan 12 days ago
fix: security audit remediation — 12 fixes, 20 tests (v0.13.1.0) (#595)

* fix: remove auth token from /health, secure extension bootstrap (CRITICAL-02 + HIGH-03)

- Remove token from /health response (was leaked to any localhost process)
- Write .auth.json to extension dir for Manifest V3 bootstrap
- sidebar-agent reads token from state file via BROWSE_STATE_FILE env var
- Remove getToken handler from extension (token via health broadcast)
- Extension loads token before first health poll to prevent race condition

* fix: require auth on cookie-picker data routes (CRITICAL-01)

- Add Bearer token auth gate on all /cookie-picker/* data/action routes
- GET /cookie-picker HTML page stays unauthenticated (UI shell)
- Token embedded in served HTML for picker's fetch calls
- CORS preflight now allows Authorization header

* fix: add state file TTL and plaintext cookie warning (HIGH-02)

- Add savedAt timestamp to state save output
- Warn on load if state file older than 7 days
- Auto-delete stale state files (>7 days) on server startup
- Warning about plaintext cookie storage in save message

* fix: innerHTML XSS in extension content script and sidepanel (MEDIUM-01)

- content.js: replace innerHTML with createElement/textContent for ref panel
- sidepanel.js: escape entry.command with escapeHtml() in activity feed
- Both found by security audit + Codex adversarial red team

* fix: symlink bypass in validateReadPath (MEDIUM-02)

- Always resolve to absolute path first (fixes relative path bypass)
- Use realpathSync to follow symlinks before boundary check
- Throw on non-ENOENT realpathSync failures (explicit over silent)
- Resolve SAFE_DIRECTORIES through realpathSync (macOS /tmp → /private/tmp)
- Resolve directory part for non-existent files (ENOENT with symlinked parent)

* fix: freeze hook symlink bypass and prefix collision (MEDIUM-03)

- Add POSIX-portable path resolution (cd + pwd -P, works on macOS)
- Fix prefix collision: /project-evil no longer matches /project freeze dir
- Use trailing slash in boundary check to require directory boundary

* fix: shell script injection in gstack-config and telemetry (MEDIUM-04)

- gstack-config: validate keys (alphanumeric+underscore only)
- gstack-config: use grep -F (fixed string) instead of -E (regex)
- gstack-config: escape sed special chars in values, drop newlines
- gstack-telemetry-log: sanitize REPO_SLUG and BRANCH via json_safe()

* test: 20 security tests for audit remediation

- server-auth: verify token removed from /health, auth on /refs, /activity/*
- cookie-picker: auth required on data routes, HTML page unauthenticated
- path-validation: symlink bypass blocked, realpathSync failure throws
- gstack-config: regex key rejected, sed special chars preserved
- state-ttl: savedAt timestamp, 7-day TTL warning
- telemetry: branch/repo with quotes don't corrupt JSON
- adversarial: sidepanel escapes entry.command, freeze prefix collision

* chore: bump version and changelog (v0.13.1.0)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* docs: tone down changelog — defense in depth, not catastrophic bugs

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
b343ba27 — Garry Tan 13 days ago
fix: community PRs + security hardening + E2E stability (v0.12.7.0) (#552)

* fix(security): skip hidden directories in skill template discovery

discoverTemplates() scans subdirectories for SKILL.md.tmpl files but
only skips node_modules, .git, and dist. Hidden directories like
.claude/, .agents/, and .codex/ (which contain symlinked skill
installs) were being scanned, allowing a malicious .tmpl in a
symlinked skill to inject into the generation pipeline.

Fix: add !d.name.startsWith('.') to the subdirs() filter. This skips
all dot-prefixed directories, matching the standard convention that
hidden dirs are not source code.

* fix(security): sanitize telemetry JSONL inputs against injection

SKILL, OUTCOME, SESSION_ID, SOURCE, and EVENT_TYPE values go directly
into printf %s for JSONL output. If any contain double quotes,
backslashes, or newlines, the JSON breaks — or worse, injects
arbitrary fields.

Fix: strip quotes, backslashes, and control characters from all
string fields before JSONL construction via json_safe() helper.

* fix(security): validate JSON input in gstack-review-log

gstack-review-log appends its argument directly to a JSONL file with
no validation. Malformed or crafted input could corrupt the review log
or inject arbitrary content.

Fix: validate input is parseable JSON via python3 before appending.
Reject with exit 1 and stderr message if invalid.

* fix: treat relative dot-paths as file paths in screenshot command

Closes #495

* fix: use host-specific co-author trailer in /ship and /document-release

Codex-generated skills hardcoded a Claude co-author trailer in commit
messages. Users running gstack under Codex pushed commits attributed
to the wrong AI assistant.

Add {{CO_AUTHOR_TRAILER}} resolver that emits the correct trailer
based on ctx.host:
  - claude: Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
  - codex:  Co-Authored-By: OpenAI Codex <noreply@openai.com>

Replace hardcoded trailers in ship/SKILL.md.tmpl and
document-release/SKILL.md.tmpl with the resolver placeholder.

Fixes #282. Fixes #383.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: auto-upgrade marker no longer masks newer remote versions

When a just-upgraded-from marker persists across sessions, the update
check would write UP_TO_DATE to cache and exit immediately — never
fetching the remote VERSION. Users silently miss updates that landed
after their last upgrade.

Remove the early exit and premature cache write so the script falls
through to the remote check after consuming the marker. This ensures
JUST_UPGRADED is still emitted for the preamble, while also detecting
any newer versions available upstream.

Fixes #515

* fix: decouple doc generation from binary compilation in build script

The build script chains gen:skill-docs and bun build --compile with &&,
so a doc generation failure (e.g. missing Codex host config, template
error) prevents the browse binary from being compiled. Users end up
with a broken install where setup reports the binary is missing.

Replace && with ; for the two gen:skill-docs steps so they run
independently of the compilation chain. Doc generation errors are still
visible in stderr, but no longer block binary compilation.

Fixes #482

* fix: extend security sanitization + add 10 tests for merged community PRs

- Extend json_safe() to ERROR_CLASS and FAILED_STEP fields
- Improve ERROR_MESSAGE escaping to handle backslashes and newlines
- Replace python3 with bun for JSON validation in gstack-review-log
- Add 7 telemetry injection prevention tests
- Add 2 review-log JSON validation tests
- Add 1 discover-skills hidden directory filtering test

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: stabilize flaky E2E tests (browse-basic, ship-base-branch, dashboard-via)

browse-basic: bump maxTurns 5→7 (agent reads PNG per SKILL.md instruction)
ship-base-branch: extract Step 0 only instead of full 1900-line ship/SKILL.md
dashboard-via: extract dashboard section only + increase timeout 90s→180s

Root cause: copying full SKILL.md files into test fixtures caused context bloat,
leading to timeouts and flaky turn limits. Extracting only the relevant section
cut dashboard-via from timing out at 240s to finishing in 38s.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: add E2E fixture extraction rule to CLAUDE.md

Never copy full SKILL.md files into E2E test fixtures. Extract only
the section the test needs. Also: run targeted evals in foreground,
never pkill and restart mid-run.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: stabilize journey-think-bigger routing test

Use exact trigger phrases from plan-ceo-review skill description
("think bigger", "expand scope", "ambitious enough") instead of
the ambiguous "thinking too small". Reduce maxTurns 5→3 to cut
cost per attempt ($0.12 vs $0.25). Test remains periodic tier
since LLM routing is inherently non-deterministic.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* remove: delete journey-think-bigger routing test

Never passed reliably. Tests ambiguous routing ("think bigger" →
plan-ceo-review) but Claude legitimately answers directly instead
of invoking a skill. The other 10 journey tests cover routing
with clear, actionable signals.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: bump version and changelog (v0.12.7.0)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Arun Kumar Thiagarajan <arunkt.bm14@gmail.com>
Co-authored-by: bluzername <bluzer@gmail.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Greg Jackson <gregario@users.noreply.github.com>
2b85b1df — Garry Tan a month ago
fix: random UUID installation_id + verify-rls.sh edge cases (v0.11.16.1) (#462)

* fix: random UUID installation_id + gitignore supabase/.temp

Replace SHA-256(hostname+user) with random UUID v4 stored in
~/.gstack/installation-id. Not derivable from public inputs.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: random UUID installation_id + verify-rls.sh edge cases (v0.11.16.1)

Replace SHA-256(hostname+user) with random UUID v4 stored in
~/.gstack/installation-id. Gitignore supabase/.temp/.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
64d5a3e4 — Garry Tan a month ago
fix: Supabase telemetry security lockdown (v0.11.16.0) (#460)

* fix: drop all anon RLS policies + revoke view access + add cache table

Migration 002 locks down the Supabase telemetry backend:
- Drops all SELECT, INSERT, UPDATE policies for the anon role
- Explicitly revokes SELECT on crash_clusters and skill_sequences views
- Drops stale error_message/failed_step columns (exist live but not in migration)
- Creates community_pulse_cache table for server-side aggregation caching

* feat: extend community-pulse with full dashboard data + server-side cache

community-pulse now returns top skills, crash clusters, version distribution,
and weekly active count in a single aggregated response. Results are cached
in the community_pulse_cache table (1-hour TTL) to prevent DoS via repeated
expensive queries.

* fix: route all telemetry through edge functions, not PostgREST

- gstack-telemetry-sync: POST to /functions/v1/telemetry-ingest instead of
  /rest/v1/telemetry_events. Removes sed field-renaming (edge function expects
  raw JSONL names). Parses inserted count — holds cursor if zero inserted.
- gstack-update-check: POST to /functions/v1/update-check.
- gstack-community-dashboard: calls community-pulse edge function instead of
  direct PostgREST queries.
- config.sh: removes GSTACK_TELEMETRY_ENDPOINT, fixes misleading comment.

* test: RLS smoke test + telemetry field name verification

- verify-rls.sh: 9-check smoke test (5 reads + 3 inserts + 1 update)
  verifying anon key is fully locked out after migration.
- telemetry.test.ts: verifies JSONL uses raw field names (v, ts, sessions)
  that the edge function expects, not Postgres column names.
- README.md: fixes privacy claim to match actual RLS policy.

* chore: bump version and changelog (v0.11.16.0)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: pre-landing review fixes — JSONB field order, version filter, RLS verification

- Dashboard JSON parsing: use per-object grep instead of field-order-dependent
  regex (JSONB doesn't preserve key order)
- Version distribution: filter to skill_run events only (was counting all types)
- verify-rls.sh: only 401/403 count as PASS (not empty 200 or 5xx); add
  Authorization header to test as anon role properly
- Remove dead empty loop in community-pulse

* chore: untrack browse/dist binaries — 116MB of arm64-only Mach-O

These compiled Bun binaries only work on arm64 macOS, and ./setup
already rebuilds from source for every platform. They were tracked
despite .gitignore due to being committed before the ignore rule.
Untracking stops them from appearing as modified in every diff.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* docs: tone down changelog — security hardening, not incident report

* fix: keep INSERT policies for old client compat, preserve extra columns

- Keep anon INSERT policies so pre-v0.11.16 clients can still sync
  telemetry via PostgREST while new clients use edge functions
- Add error_message/failed_step columns to migration (reconcile repo
  with live schema) instead of dropping them
- Security fix still lands: SELECT and UPDATE policies are dropped

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: sync package.json version with VERSION file (0.11.16.0)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
3b22fc39 — Garry Tan a month ago
feat: opt-in usage telemetry + community intelligence platform (v0.8.6) (#210)

* feat: add gstack-telemetry-log and gstack-analytics scripts

Local telemetry infrastructure for gstack usage tracking.
gstack-telemetry-log appends JSONL events with skill name, duration,
outcome, session ID, and platform info. Supports off/anonymous/community
privacy tiers. gstack-analytics renders a personal usage dashboard
from local data.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: add telemetry preamble injection + opt-in prompt + epilogue

Extends generatePreamble() with telemetry start block (config read,
timer, session ID, .pending marker), opt-in prompt (gated by
.telemetry-prompted), and epilogue instructions for Claude to log
events after skill completion. Adds 5 telemetry tests.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: regenerate all SKILL.md files with telemetry blocks

Automated regeneration from gen-skill-docs.ts changes. All skills
now include telemetry start block, opt-in prompt, and epilogue.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: add Supabase schema, edge functions, and SQL views

Telemetry backend infrastructure: telemetry_events table with RLS
(insert-only), installations table for retention tracking,
update_checks for install pings. Edge functions for update-check
(version + ping), telemetry-ingest (batch insert), and
community-pulse (weekly active count). SQL views for crash
clustering and skill co-occurrence sequences.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: add telemetry-sync, community-dashboard, and integration tests

gstack-telemetry-sync: fire-and-forget JSONL → Supabase sync with
privacy tier field stripping, batch limits, and cursor tracking.
gstack-community-dashboard: CLI tool querying Supabase for skill
popularity, crash clusters, and version distribution.
19 integration tests covering all telemetry scripts.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: session-specific .pending markers + crash_clusters view fix

Addresses Codex review findings:
- .pending race condition: use .pending-$SESSION_ID instead of
  shared .pending file to prevent concurrent session interference
- crash_clusters view: add total_occurrences and anonymous_occurrences
  columns since anonymous tier has no installation_id
- Added test: own session pending marker is not finalized

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: dual-attempt update check with Supabase install ping

Fires a parallel background curl to Supabase during the slow-path
version fetch. Logs upgrade_prompted event only on fresh fetches
(not cached replays) to avoid overcounting. GitHub remains the
primary version source — Supabase ping is fire-and-forget.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: integrate telemetry usage stats into /retro output

Retro now reads ~/.gstack/analytics/skill-usage.jsonl and includes
gstack usage metrics (skill run counts, top skills, success rate)
in the weekly retrospective output.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: move 'Skill usage telemetry' to Completed in TODOS.md

Implemented in this branch: local JSONL logging, opt-in prompt,
privacy tiers, Supabase backend, community dashboard, /retro
integration.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: wire Supabase credentials and expose tables via Data API

Add supabase/config.sh with project URL and publishable key (safe to
commit — RLS restricts to INSERT only). Update telemetry-sync,
community-dashboard, and update-check to source the config and
include proper auth headers for the Supabase REST API.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: add SELECT RLS policies to migration for community dashboard reads

All telemetry data is anonymous (no PII), so public reads via the
publishable key are safe. Needed for the community dashboard to
query skill popularity, crash clusters, and version distribution.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: bump version and changelog (v0.8.6)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: analytics backward-compatible with old JSONL format

Handle old-format events (no event_type field) alongside new format.
Skip hook_fire events. Fix grep -c whitespace issues and unbound
variable errors.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: map JSONL field names to Postgres columns in telemetry-sync

Local JSONL uses short names (v, ts, sessions) but the Supabase
table expects full names (schema_version, event_timestamp,
concurrent_sessions). Add sed mapping during field stripping.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: address Codex adversarial findings — cursor, opt-out, queries

- Sync cursor now advances on HTTP 2xx (not grep for "inserted")
- Update-check respects telemetry opt-out before pinging Supabase
- Dashboard queries use correct view column names (total_occurrences)
- Sync strips old-format "repo" field to prevent privacy leak

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: add Privacy & Telemetry section to README

Transparent disclosure of what telemetry collects, what it never sends,
how to opt out, and a link to the schema so users can verify.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>