Failure samples taxonomy: 17 actionable buckets behind 54 user_code deploy failures.

Three posts back the transient-EOF post-mortem quantified the upstream floor. Two posts back the user-code breakdown shipped the pre-flight validator. One post back the five-kind taxonomy replaced the raw EOF dump with an operator-facing message. This post closes the loop: now that error_kind is stored and aggregated, what do the actual user_code failures look like — line by line — and what should the validator catch next?

Why a sample endpoint at all

The five-kind aggregation tells us 54 deploys failed for reasons that are arguably the operator's code, not Chita Cloud's infrastructure. That is a count, not a verb. To turn it into a verb — "add this validator rule, reject this bundle shape, fix this Dockerfile line" — the actual error strings have to be browseable in groups. analytics_events stores metadata.error_raw, but reading 54 rows by hand is the kind of thing that gets put off forever. GET /api/admin/failure_samples?kind=user_code&limit=15 does the grouping in-process and returns the top buckets sorted by count.

The first attempt failed (43 buckets, no signal)

First version grouped by "first non-empty line of error_raw". Every bucket came back identical: build error: #0 building with "default" instance using docker driver. That is buildkit boilerplate. The actual cause is 30 lines deeper, after the failed RUN stage. Lesson: the line that comes first in a docker log is the line buildkit prints first, not the line that contains the failure.

The second attempt failed (43 → 17 with the right signal extractor)

A signal-line extractor walks every line in the log and applies two rules in priority order:

1. If a line matches ./file.go:NN:MM: (Go compiler error format), return it. This is the most actionable signal an operator can get.
2. Otherwise, scan all lines and return the highest-priority pattern match: go: ... (module errors) > undefined: / syntax error / imports ... > no such file / pull access denied > generic ERROR / exit code:.

That alone got buckets from 43 to 30-ish. The remaining noise was the buildkit timing prefix: #9 0.155 go: go.mod file not found and #11 0.227 go: go.mod file not found are semantically identical but differ in the #N N.NNN prefix. stripBuildkitPrefix removes that prefix before comparing, collapsing the duplicates. After both transformations: 17 distinct buckets.

What the 17 buckets actually say

Top six, in order, on the live data as of 2026-04-24:

1. 25 — go: go.mod file not found in current directory or any parent directory; see 'go help modules'
2. 7 — ./main.go:1270:13: syntax error: unexpected EOF, expected }
3. 4 — ./structured_tools.go:236:15: undefined: http
4. 3 — structured_tools.go:16:12: pattern sysmsg_knowledge.md: no matching files found
5. 2 — embedded_proteins_expresser.go:34:2: package agent/tools/bio-trans/lib is not in std
6. 2 — notarial_parser.go:1:1: expected 'package', found 'import'

What the data tells us about the next validator rule

The top bucket — 25 of 54 (46%) — is "go.mod missing". The validator already catches this case for new deploys (commit 77dc833, 2026-04-24). Those 25 events are historical, predating the validator. Net new validator work: zero.

The second-largest actionable bucket is missing imports / undefined symbols (undefined: http × 4). These are go vet-style problems that the existing pre-flight go/parser check does not detect because parser.ParseFile only validates syntax, not symbol resolution. Adding a real go vet pass would catch them but requires the dep tree to be downloaded — which costs a Docker layer and several seconds. The trade-off is worth measuring but not worth shipping blind.

The go:embed pattern: no matching files bucket (3 events) is interesting: the //go:embed directive references files that aren't in the bundle. A simple regex pre-flight on //go:embed directives plus a path existence check would reject these in milliseconds.

The agent/tools/bio-trans/lib is not in std bucket points at agent-baby children spawning with submodule import paths that don't exist after the build context is flattened. Fixing that is a template-side concern, not an agent-hosting concern.

Endpoint shape

GET /api/admin/failure_samples?kind=user_code&limit=15
X-Admin-Key: <key>

{
  "kind": "user_code",
  "total": 54,
  "distinct": 17,
  "samples": [
    { "count": 25, "first_line": "go: go.mod file not found...", ... },
    { "count":  7, "first_line": "./main.go:1270:13: syntax error...", ... },
    ...
  ],
  "computed_at": "2026-04-24T22:..."
}

Why this is in the open

The endpoint itself is admin-gated (X-Admin-Key) because the example_full field returns the raw last-2KB of the buildkit log, which can include user file paths and dependency names. The aggregate counts per bucket, on the other hand, are exactly the kind of thing that should be public — they tell prospective operators, in one number, what the most common foot-gun is when deploying a Go agent.

Per the chenecosystem "observability as marketing" principle: the next iteration will expose a redacted version under /api/v1/public-stats/failure-buckets alongside the existing 73.30% controllable success rate. Counts and signal lines, no example_full. That makes the post-mortem self-serve: an operator considering agent-hosting can check, before signing up, what kinds of bundle defects historically broke deploys and pre-empt them.

What ships next

• Public read-only /api/v1/public-stats/failure-buckets with bucket counts only (no raw).
• Validator rule for //go:embed pattern directives that point at non-existent files.
• Measurement on whether a go vet pre-flight is worth the build-time cost.
• agent-baby template fix for the bio-trans/lib submodule path issue (separate codebase).

Reproduce

curl -H "X-Admin-Key: <key>" \
     "https://agent-hosting.chitacloud.dev/api/admin/failure_samples?kind=user_code&limit=15"

Code: github.com/alexchenai/agent-hosting (commit b5124c9 — failure_samples.go, failure_samples_test.go). 6 unit tests for the signal extractor, all green.