Data that holds up.
AI with human judgment.

21 of 21 entries

1. Oct 28, 2026Problem solved

   Not every drop on the scoreboard is a regression.

   A release scoreboard showed nine metrics getting worse, which looked like a regression. Instead of reverting, I split the results: every metric that dropped belonged to the end-to-end agent run, while retrieval, loading and checks were identical. A blind re-judging of both versions' answers, mixed together with the same judge, ruled out judge drift, and the baseline agent dropped even more than the new one. The release was fine; the noise came from the agent run itself.

   Outcome: 9 apparent regressions traced to agent-run noise, with zero retrieval metric change; margin over baseline grew from +5 to +8 of 20.

   Technical detail

   • Scoreboard v-prev → v-new: 9 improved, 13 equal, 9 worse. All 9 regressions sit in the agent-vs-judge block (20 questions); retrieval metrics (R@10, nDCG, MRR, hybrid and re-ranked nDCG) and data checks were identical.
   • Control: the production-style baseline agent ran on the same code and database in both rounds and fell further (15→9 accepted of 20) than the new system (20→17). If retrieval had changed, the retrieval metrics would have moved.
   • Judge hypothesis closed: answers from both versions were re-judged together, blind, in the same batches with the same judge; the drop persisted.
   • Remaining variables: run-to-run variance of a small model, and the briefing date changing between rounds. The 3 lost questions were reasoning errors on correctly opened sources (date inferred instead of read).
   • Noise floor: with 20 questions each is worth 5 points, and the previous round was at a 20/20 ceiling, so it could only go down.
   • Reading that holds: compare within the same round. The margin over the baseline grew (+5 → +8 questions).
   • Next round: run both versions' agents the same day and pin the briefing date as a recorded run parameter.

   evals · llm-as-judge · regression-analysis · retrieval · release-engineering#
2. Oct 28, 2026Problem solved

   A better prompt needs evidence.

   While reviewing an open-source app with an AI chat assistant, I wanted evidence that a rewritten prompt was actually better, not just a feeling. I built a local test bench: 96 answers, each produced in isolation so cases could not contaminate each other, then graded blind by independent judges. The rewrite clearly helped the stronger model, mainly by stopping it from inventing facts, and made no reliable difference on the smaller one.

   Outcome: Rewritten prompt raised ungrounded-claim-free answers from 75% to 92% on Sonnet (sign test p≈0.04) with no reliable gain on Haiku.

   Technical detail

   Design: 2 models (Haiku, Sonnet) × 2 prompt variants (A baseline, B rewritten) × 8 fixture cases × 3 repetitions = 96 responses. Fixtures were generated by calling the app's own context-builder code on seed data, so the model saw the exact context production would send. One subagent per response, reading only the prompt and a single case file, to eliminate cross-case leakage. Four blind Opus judges scored 1–5 and flagged claims not grounded in the supplied context.

   Results (mean score / ≥4 / no ungrounded claim): Haiku A 3.75 / 54% / 58%; Haiku B 4.08 / 71% / 62%; Sonnet A 4.54 / 96% / 75%; Sonnet B 4.92 / 100% / 92%. Between-repetition spread was 0.07–0.14, small enough to separate prompt effect from noise.

   Statistics: paired by case and repetition, sign test. Sonnet: B won 10, lost 2, tied 12 (p≈0.04). Haiku: 9/5/10 (p≈0.4), not supported. Model choice mattered more than prompt: worst Sonnet beat best Haiku.

   Findings: a counting error repeated in 6/6 Haiku answers regardless of prompt, so the fix is to send the number precomputed from the server rather than prompt harder. Variant B's examples induced a side effect (4/24 answers exceeded the bold-text limit). Judges also exposed an inconsistency in the demo fixture.

   Trade-offs/caveats: subagents receive the prompt as a file, not as a real system prompt, and the target deployment runs a different vendor's model, so results transfer to Claude only; next step is a cross-vendor run. ~4.5M subagent tokens, mostly harness overhead; a direct API run with the true system prompt would be far cheaper and more faithful.

   prompt-eval · llm-as-judge · subagents · experiment-design · rag#
3. Oct 16, 2026Decision

   Talking to a coding agent by voice.

   Wanted a hands-free voice conversation with a coding agent, like ChatGPT's voice mode: press talk, speak, the agent answers out loud. The first version kept the agent 'waiting' for the next sentence, which broke as soon as the session sat idle and needed an extra keypress. Instead of patching the waiting trick, the delivery mechanism was replaced: a small plugin inside each session now submits the transcribed speech as a normal prompt, so nothing has to stay blocked and no Enter is needed.

   Outcome: Speech now reaches an idle session with no Enter and no 240s wait window.

   Technical detail

   Local daemon: VAD closes utterances on silence, Parakeet transcribes on-device, macOS say speaks replies (mic paused while speaking to avoid self-hearing). v1 delivered speech through a blocking Stop hook (decision: block with the transcript as reason, up to 240s wait) plus PostToolUse for mid-work interjections. Failure: after the wait expired or the session idled, speech showed as 'listening' but never reached the agent. v2: a Claude Code hooks-module plugin (loaded via CLAUDE_CODE_PLUGIN_DIRS) polls the daemon /pending every 400ms and calls $.prompt.submit — the same path as a typed prompt, valid on an idle session. Stop hook wait set to 0. Target session = last one that replied; widget shows connected-session count. Trade-off: polling in every session and dependence on a plugin API vs. a hook-only approach that is stateless but time-boxed.

   claude-code · plugins · voice · hooks · local-first#
4. Oct 15, 2026Problem solved

   The uncertainty was understated.

   An eval estimates how often an AI system is right by combining a small set of human-labeled cases with a large set scored automatically. A code reviewer built a case where the reported uncertainty was badly wrong. The cause: the labeled cases were a subset of the scored ones, so the two averages moved together, and the formula treated them as independent. Adding the missing correlation term fixed the estimate, and a review-driven test now guards it.

   Outcome: Variance on the reviewer's adversarial case went from 5/12 (wrong) to exactly 1/12; full suite 662 passed, 1 xfail.

   Technical detail

   Prediction-powered inference (PPI) estimator: the variance was computed as var(f)/N + var(r)/n, which assumes the labeled sample and the scored sample are independent. In this design the labeled cases are a subset of the scored ones, so both means share the same f values and covary. Corrected variance: var(f)/N + var(r)/n + 2·cov(f_labeled, r)/N. Reviewer's adversarial case (all cases labeled, y = 1 − f) now yields exactly 1/12 instead of 5/12; three edge cases are pinned in tests. Earlier tests stayed valid because their labeled f values were constant (zero covariance). Same review round also reclassified timeout/interrupt/sink-failure events as ambiguous loss (a cancelled task does not undo a committed write), bounding the call count to an interval and returning a PARTIAL answer; only build-failed and write-refused prove loss. Trade-off: wider, honest intervals over tighter but overconfident ones.

   statistics · evals · prediction-powered-inference · code-review · python#
5. Oct 14, 2026Decision

   Measuring must not break what it measures.

   In a framework that records which pieces of content AI agents consult, a record could carry a link to a content unit that looked valid but actually pointed at the wrong kind of thing. Elsewhere in the same system the fix was to reject such requests outright. Here, the recording code runs inside the caller's own path, so rejecting would let the measuring tool break the thing it measures. The decision was to keep the row but drop only the unsafe link, collapsing two scattered checks into a single rule.

   Outcome: Two divergent pointer-writing sites unified into one rule; 552 tests passing, no new type errors.

   Technical detail

   Single pure function unit_pointer(type, key, ontology) placed next to the record-shape guard. The unit record pointer is written only when the key is record-shaped AND the table it names equals the ontology-declared table for that unit type; anything else writes NONE while unit_key stays faithful. Both writers (event builder for ref rows, usage evidence links) now call it; the record-id regex moved out of the registry. Deliberate asymmetry with the read path: trace/neighbors fail fast on a mismatch before querying, while the write-side instrument degrades gracefully (row kept, join withheld). Tests: pure-rule cases without a server; integration proving a look-alike key in another table yields NONE in both tables, a real key yields a pointer, and a custom table name works live. Full suite 552 passed, 1 xfailed; mypy error count unchanged.

   data-integrity · api-design · surrealdb · observability · python#
6. Oct 8, 2026Decision

   Cap the batch by bytes, not by records.

   A bulk-ingestion routine sent records to the database in fixed-size batches, and the open question was how big a batch should be. Instead of guessing, I ran a controlled experiment against a real database server and found the real limit was set by payload bytes, not record count. A batch size that looks safe is only safe by accident of how large each record happens to be, so the recommendation was to cap batches by bytes.

   Outcome: Fixed batches of 100+ failed 5/5 against the real server; the byte-bounded batch ran at 601 stmts/s with zero 413s.

   Technical detail

   Setup: real SurrealDB 3.1.5 over POST /sql, statements generated by the production upsert builder, a 417-unit corpus (real passages plus synthetic short/medium/long ones) with precomputed 384-dim embeddings. Byte caps were decided and recorded before the runs. Each strategy ran with a warm-up plus 5 repetitions in rotated order, a fresh database per run, and separate phases (initial load, re-ingest, rebuild). A local proxy with a 256 KiB limit simulated a clean 413.

   Findings:

   • The body limit is exactly 1 MiB, found by binary search with padding payloads. With ~10 KB statements dominated by the embedding, the ceiling falls between 50 and 100 statements per batch. Fixed batches of 100 and 200 failed 5/5, and the existing default of 200 overshoots by a wide margin.
   • Above the limit, rejection degrades from a readable 413 (1 MiB+1B: 6/6 clean) to a connection closed mid-upload (3 MiB: 6/6 BrokenPipeError). So an adaptive 'split only on 413' strategy failed 5/5 against the direct server. Behind a proxy that returns a clean 413 it works, but resends 1,275 statements and transfers 4x the payload (16.7 MB vs 4.1 MB).
   • Partial state is real: a 413 after 20 successful batches left the earlier 400 rows persisted, with no rollback.
   • Idempotency held everywhere: identical content hashes across initial load, re-ingest and rebuild for every strategy.

   Decision: a byte-bounded batch with headroom (768 KiB, 75% of the limit) was fastest (0.70s, 601 stmts/s, 6 requests), had no 413s by construction, and was the only strategy that failed cleanly when a single statement exceeds the cap (it refuses before sending). A fixed batch of 20-50 is an acceptable simple option, but its safety is a coincidence of statement size. Caveats stated in the report: small corpus, localhost, concurrency 1, default server limit, so the numbers don't extrapolate, only the shape does.

   surrealdb · batching · experiment-design · reliability · http#
7. Oct 8, 2026Decision

   Two citation metrics, side by side.

   Before implementing an analytics metric that says how well an AI agent uses the evidence it is given, I asked whether the obvious definition was the right one. Two candidates looked equivalent: the share of calls that produced a citation, or the share of delivered evidence that got cited. Three parallel sub-agents examined the contract, simulated 13 scenarios and evaluated how people would read each version. Following either metric alone led to the wrong action in several cases, so the recommendation was to show both side by side, with the gap between them as the signal.

   Outcome: 13 synthetic scenarios computed; 3 showed a single metric leading to the wrong action, which led to the recommendation to present both.

   Technical detail

   Setup: read-only investigation (no code or repo changes) split across three sub-agents: (1) contract and validity: numerators, denominators, eligibility, deduplication, observation limits; (2) synthetic scenario matrix with computed rates; (3) heuristic usability evaluation plus a user-test protocol.

   Candidates: A = calls with ≥1 directly attributed, cited evidence / eligible calls. B = delivered-and-cited evidence / delivered evidence, in three aggregations: per-call ratio, mean of per-call ratios (macro), and pooled ratio over all evidence (micro). These weight calls differently and are not one metric.

   Findings:

   • The current schema already supports A and all B variants; only the metric code is missing (the existing branch is an unconditional stub).
   • "Delivered" must be the consumer-asserted delivery link, never the refs recorded by the decorator.
   • In 3 of 13 scenarios a single metric drives the wrong action: A hides 90% of unused evidence (10 delivered, 1 cited); the macro variant inverts rankings and masks dead calls; the pooled variant penalizes top-k tools by arithmetic (ceiling ≈ 1/k), which could suggest disabling the retrieval that feeds everything.
   • B depends far more on attribution ergonomics than A, so A degrades more gracefully when attribution coverage is partial.

   Rules kept: citation ≠ quality or causality; missing attribution is reported as an interval, never silently zero; no automatic causal credit from a search to a later open; no "last global event" fallback.

   Decision: show both, named so the denominator is explicit, per tool and stage; the divergence (high A, low B) diagnoses result excess. Confidence: medium. Heuristic evaluation is labeled as such, and a 15-participant Latin-square protocol (8 tasks with answer keys, 3 trap tasks, pre-registered criteria) was delivered. No user validation was claimed.

   metrics · analytics · multi-agent · llm-observability · evaluation-design#
8. Oct 8, 2026Decision

   Not every mistake is observable.

   A user of an open-source Python framework noticed that a decorated function called outside a session ran fine but left no trace, and the open question was whether the library should warn about it. Instead of adding a warning, I ran three independent subagents (opt-in diagnostic, doctor/test alternatives, adversarial critic) over the same nine scenarios, with consistency criteria defined before any test ran. The finding: 'forgot the session' is not an observable state. It looks exactly like a legitimate call without a session. So no diagnostic can honestly claim it, and the smallest reliable fix was documentation plus a test against the production entrypoint, with the library untouched.

   Outcome: Nine scenarios × five alternatives compared; the one library change considered was cut down to an 8-line passive counter or nothing.

   Technical detail

   Method: read-only investigation; throwaway prototypes in a scratchpad; shared scenario matrix (legit no-session, forgot session, incomplete registry/sink, never called, persisted, instrumentation failure, return, raise, repeated/concurrent). Determinism checked with 5 reps × 3 processes.

   Alternatives: A) tutorial + persistence test; B) doctor/check; C1) passive counter in the wrapper's no-session branch (+8 lines); C2) warn-once; C3) raise in dev mode.

   Findings:

   • C1 preserves return/exception identity and is deterministic, but only separates 'session opened, nothing called' from 'called without session'. C2 adds no information and is invisible under green pytest or ERROR-level logging. C3 stops the tool from running and masks its exception, so it was discarded.
   • The doctor output was byte-identical for the legit, forgotten and never-called cases.
   • The 'natural' persistence test (open a session around the call) passes while production is blind; only a test that calls the production entrypoint catches it.
   • Side findings: 20 concurrent calls with a session persisted 5 and dropped 15 (circuit breaker, by design); a drop counter flushed every 100 writes hid early drops; docs claimed 'no session ⇒ raise' while the decorator passes through.

   Decision: keep the wrapper untouched, ship docs + entrypoint test, and keep C1 as the only admissible library change once something reads it. The report separates observed facts, inferences and proposals, and states what the microbenchmarks do not prove (scale, real onboarding).

   decision-making · observability · python · testing · adversarial-review#
9. Oct 8, 2026Decision

   A debate settled by experiment, not taste.

   A team debate over how tools should reference content (a structured object vs. a plain "type:key" string) was settled with a controlled experiment instead of taste. Two sub-agents built the same consumer in isolation, one per format, against a shared contract and adversarial fixtures, and a third audited the equivalence. Both formats turned out equally correct, and the experiment surfaced real bugs in the library underneath, which was the more useful result.

   Outcome: Both formats passed all deterministic cases (29/29 and 30/30 + 1 strict xfail) with 0 malformed refs in 14 agent runs; 3 library defects found and reproduced.

   Technical detail

   Design: coordinator fixed fixtures, a functional contract and common criteria first; implementers A (JSON object) and B (string) worked in separate directories without reading each other's code; an auditor ran adversarial cases. Both pinned to the same library revision (not the local checkout, not an open PR). Fixtures made the stable key, typed ref and graph record ID deliberately different, with hostile keys (:/::, Unicode, spaces, record-ID lookalikes, key equal to the type name, same key in two types, composite key).

   Deterministic layer: 29 and 30 tests passed respectively, covering round-trip search→open→navigate→cite, malformed/unknown/incompatible refs, source removed after search, content change with stable identity, and a closed error vocabulary so none of them collapses into "no information".

   Agent layer: 7 tasks × 2 scenarios, fresh claude -p sessions, same model, alternating order, stream-json transcripts. Round 1: 6/7 vs 7/7 round-trips, zero malformed or altered refs in 14 runs, 100% valid citations. The single failure was giving up after empty searches (BM25 AND semantics vs. Portuguese queries), not the format. Replications with a second order and a weaker model were still pending when the session ended, so the report marks them as such.

   Findings owned by the library, not the format: the evidence link unit was inferred from the key by regex (a key shaped like a record ID produced a pointer to the wrong record); cite and validate_claims disagreed on whether a candidate is citeable; accepted ref shapes were inconsistent across three entry points.

   Recommendation: no experimental basis to prefer either; decide on what the experiment doesn't measure (the library already speaks type:key; the object allows closed schemas), and fix the inference bug regardless.

   llm-agents · experiment-design · api-design · mcp · evals#
10. Oct 6, 2026Problem solved

    Fixing sign-in without opening an account takeover.

    After an enterprise client's users started failing to sign in through single sign-on, the root cause turned out to be old accounts created before SSO existed: the new login tried to create a duplicate record and hit a unique-email constraint. The obvious fix, matching the old account by email, would have let anyone who controls that email in the identity provider take over an existing account, including admin ones. The fix was designed around that risk: accounts are only adopted when safe, privileged accounts require proof of ownership, and admin rights come from the stored account type instead of the email.

    Outcome: SSO login fixed for all affected legacy accounts, with account takeover through email reuse closed; shipped across 4 reviewed PRs.

    Technical detail

    Root cause: the upsert looked up users by external_id; legacy users had no external_identity, so it fell through to CREATE and collided with a global unique index on email. The recovery path only re-selected by external_id, so it re-raised. About 70% of the tenant's users were in that state, so every one would fail on first SSO login.

    Fix 1 (adopt by email), with review guardrails: only users with no identity are adoptable (a reused email at the provider fails closed), the email is normalized on both lookup and write, the lookup is scoped per tenant, and a cross-tenant diagnostic log was added. A census showed no cross-tenant duplicates.

    Follow-up after learning the provider allows free email changes (no verification): email alone is not proof of ownership. Staff accounts are never auto-adopted; they link only if the (provider id, email) pair is on an allowlist held in a secret. is_admin is now stamped into the signed session from the account kind in the database, and promotion by email list was removed. The env-var shortcut had only been there for a fast release.

    Trade-off: the allowlist needs a manual step per staff member (collect provider ids from the client) in exchange for closing account takeover. A reviewer's concern that swallowing the collision error would disarm an accidental protection against session theft led us to ship the error handling in a separate PR, only after the admin change.

    Diagnosis: Cloud Run logs read through gcloud logging.

    sso · auth · security · incident-response · multi-tenant#
11. Sep 29, 2026Decision

    Read-only enforced by the database, not a filter.

    An evaluation harness lets AI agents run their own database queries against a shared test dataset. Code review found that a simple 'is this query read-only?' check could be fooled by wrapping a delete inside a harmless-looking select, which silently corrupted the data every later run was scored against. Instead of making the filter smarter and hoping, the fix moved the guarantee into the database itself: agent queries now run as a user who is only allowed to read. The text filter stayed, but only to give friendly error messages.

    Outcome: Viewer-role DELETE against seeded data verified as a no-op; shared eval fixture can no longer be mutated by agent queries.

    Technical detail

    Bug: the gateway validated only the first keyword, so SELECT * FROM (DELETE ...) passed and mutated the shared fixture; later runs scored against golds the data no longer matched. The scorer's introspection check had the same flaw (anchored at position 0, so SELECT 1; INFO FOR DB; was credited as correct).

    Decision: make the role the guarantee, not the parser. Agent queries run under a VIEWER-role user bootstrapped once with root; a word-level scan over every statement (string literals stripped) remains only as the friendly early error. Trade-off: one extra bootstrap step and a second credential, versus an ever-growing denylist that a creative query can always route around. Verified that a viewer DELETE against seeded data is a no-op.

    Related hardening from the same review: HTTPError caught before URLError so the 400 body with the real parse error is not lost; generators refuse to overwrite a round that already has a manifest unless --force; report regeneration rewrites only the region between generated markers so hand-written analysis survives.

    evals · security · surrealdb · defense-in-depth · llm-agents#
12. Jul 21, 2026Shipped

    LLM insights grounded in facts.

    Delivered the first milestone of an LLM-powered insights generation API built for an enterprise client. The system takes structured survey data, atomizes it into grounded facts, and produces layered briefings — with a guardrail that enforces every claim traces back to a source. Validated on real survey data: 29 insights generated with 100% grounding correctness and zero guardrail failures.

    Outcome: 29 insights generated on real survey data; 100% grounding correctness; 0 guardrail failures.

    Technical detail

    Built in FastAPI with Pydantic AI for parsing. Core pipeline: fact extraction → multi-pass atomization → document generation (H3 pattern). Grounding guardrail rejects any claim without direct record provenance. Multi-provider architecture (OpenAI and Anthropic validated interchangeably via model-as-config). DAG-aware build sequencing: parallel cards where dependencies allowed, sequential where schemas needed to stabilize first. Empirically closed one architectural question — on-demand fact calculation was scoped out after zero occurrences appeared in real data runs, avoiding premature abstraction. Six stacked PRs structured for clean merge sequence.

    fastapi · llm-pipeline · pydantic-ai · grounding · insights-generation#
13. Jul 21, 2026Decision

    Pick the strategy before the model.

    When asked to extract structured insights from thousands of open-ended survey responses about a consumer mobile app, the first question wasn't which model to use — it was which extraction *strategy* to use. Three approaches were benchmarked end-to-end: a two-stage atomic-claims pipeline, a single-pass document approach, and a hybrid with a pre-draft refinement step. The winning strategy wasn't the most sophisticated one.

    Outcome: H3 selected as default extraction strategy: lowest cost ($0.0216/doc), zero altitude violations, 100% quote grounding across all test documents.

    Technical detail

    Spike comparing three LLM extraction flows for qualitative open-ended survey data using Pydantic AI and Jinja-templated prompts (via ai-prompter):

    • H3 (two-stage: facts → atomic claims → synthesis): $0.0216/doc, 0 altitude violations, 100% quote grounding
    • H1+separate-doc: $0.0296/doc, 1 altitude violation, 71% fact-family coverage
    • H3+predraft (H3 + refinement draft): $0.0348/doc, 3 altitude violations, 60% more claims but noisier

    Key trade-off: H3+predraft generated 60% more claims than H3 but introduced 3 'altitude violations' (over-generalized statements unsupported by the data). H1 had better fact coverage (71% vs 57% of families) but higher cost and one violation. H3 won on every weighted criterion.

    Pipeline also includes a deterministic PII guard (emails, phones, CPF, social handles) running before any LLM call — 20% of real responses contained PII, making this non-optional. A quote_grounding validator confirms every generated claim traces back to a verbatim quote, making the pipeline fully auditable. Fact index covers 67 facts across 7 families (boolean, categorical, cooccurrence, cross_tab, matrix, multi_select, persona).

    llm · pydantic-ai · prompt-engineering · qualitative-analysis · data-pipeline#
14. Jul 21, 2026Problem solved

    A migration that only looked like it worked.

    While validating a database schema migration, discovered that SurrealDB v3 silently ignores `DEFINE FIELD` statements on fields that already exist — the operation appears to succeed, but the schema is left unchanged. A custom migration runner was compounding the problem by swallowing the underlying error and marking the migration as applied. The bug was proven empirically by querying table metadata in an isolated namespace and cross-checking field types before and after the migration ran.

    Technical detail

    SurrealDB v3 behavior: DEFINE FIELD (without IF NOT EXISTS) on an already-existing field emits a silent failure — The field already exists — but does not raise an exception that would halt execution. The custom migration runner (sbl-migrate, built on surreal_basics) catches this at the transaction level and records the migration as [APPLIED], meaning all field redefinitions across the M0→M1 migration were no-ops. Discovered by querying INFO FOR TABLE in an isolated namespace (test_sup513b) and comparing field type metadata. Fix: replace each bare DEFINE FIELD on an existing field with an explicit REMOVE FIELD IF EXISTS + DEFINE FIELD sequence. The DOWN migration (using REMOVE TABLE IF EXISTS) was correctly implemented and unaffected.

    surrealdb · schema-migration · debugging · database · python#
15. Jul 16, 2026Shipped

    When a missing field is information too.

    Built a second LLM-powered parser for a survey analytics platform — this one converts a plain-text analyst description of audience personas into a validated structured spec. The interesting constraint: the downstream clustering factory checks for feature configuration using Python's `'key' in dict` (key presence), not `dict.get('key')` (value check). That distinction forced a non-obvious model design choice.

    Outcome: 16 tests passing (2 skipped pending API key), ruff clean — parser, models, route, and test suite shipped as one cohesive unit.

    Technical detail

    The parser uses Pydantic AI with output_type=PersonaSpec, and PersonaSpec contains PersonaVariable objects with optional order and scale fields. The design choice: both fields use Optional[list] = None rather than list[str] = []. Reason: the downstream factory calls model_dump(exclude_none=True) and then checks 'order' in var — an empty list [] would be falsy in a boolean check but truthy for in, silently breaking ordinal variable handling. With None defaults, exclude_none=True removes the keys entirely, and 'order' in var correctly returns False for categorical variables. The test suite uses Pydantic AI's TestModel(custom_output_args={...}) to override the LLM with deterministic structured output — no API calls needed in CI. One test (test_parse_persona_spec_model_dump_exclui_nulos) explicitly documents the key-absence invariant. Stack: Pydantic AI, FastAPI, Pydantic v2, pytest.

    pydantic-ai · llm-parsers · python · fastapi · api-design#
16. Jul 16, 2026Problem solved

    A shared namespace was corrupting migrations.

    Wrote a reversible database schema migration for a new data model milestone. Along the way, discovered that sharing a SurrealDB namespace between two local projects caused silent migration tracker corruption — the migration history table accumulated ghost entries from a different project, making rollback fail. Isolated the root cause, cleaned the state manually, and documented the isolation pattern.

    Outcome: Migration applied and rolled back cleanly: ruff clean, 10/10 tests passing, all field types verified via INFO FOR TABLE.

    Technical detail

    SurrealDB v3, Python, uv. The migration adds new SCHEMAFULL tables and re-types several existing fields. Key constraint: re-typing a field on a pre-existing SCHEMAFULL table requires DEFINE FIELD OVERWRITE — a plain DEFINE FIELD silently no-ops or errors depending on SurrealDB version. The down migration reverses in strict dependency order and explicitly removes array sub-fields to avoid residual ghost sub-fields after table cleanup.

    Environment bug: two projects sharing the same SurrealDB namespace/database made their migration tracking tables collide. The other project's runner had inserted rows into the shared table; when rolling back, the runner couldn't find the corresponding migration files in the current project's directory and aborted. Immediate fix: manual deletion of the contaminating rows. Long-term fix: each project gets its own namespace/database pair in local dev.

    surrealdb · schema-migration · database · dev-environment#
17. Jul 14, 2026Decision

    Measure before debating architecture.

    Designing an API that turns survey data into business insights forced a core question: how do you make AI-generated findings trustworthy enough to act on? Instead of debating in the abstract, we measured first — then decided. The answer landed on two independent verification strategies that keep the AI accountable to the source data at every step, without adding embedding infrastructure or discarding the runs that fail.

    Technical detail

    Architecture session for an async FastAPI service (Python / Pydantic AI / SurrealDB) that processes survey data into structured, auditable insights — replacing a legacy claude -p shell pipeline.

    Measure before decide: Before committing to a retrieval/selection pipeline, wrote a scratch script querying the live SurrealDB instance to measure the actual persisted fact space. Result: 12–14 k tokens (6% of context window). No pipeline was justified until that symptom appeared in production — a concrete number closed a debate that would otherwise have been speculative.

    Hybrid deterministic/LLM regime: Split the insight core into two invariant regimes:

    • Closed questions (Likert, NPS, multiple-choice): pure Python, fully deterministic, golden-testable — LLM never touches this path.
    • Open-ended / qualitative: LLM identifies themes and frequency patterns; a deterministic quote-grounding check (verbatim substring match against the source CSV) validates every claim before persistence. Hallucination caught without embeddings — zero cost, fully auditable. The invariant: every qualitative fact must cite a traceable source fragment.

    Guardrail failure as eval signal: Pydantic AI output validator retries once on failure; persistent failures are stored with a guardrail_failed flag rather than discarded. Rationale: in early production runs, failure rate is the most useful quality signal — silently discarding failed outputs hides exactly what you need to learn.

    Fact space recomputed per run: Each run rebuilds its own fact space from source specs and persists it independently. Trade-off: redundant computation vs. full reproducibility — any run can be re-audited in isolation regardless of future schema or data changes.

    architecture · pydantic-ai · llm-guardrails · survey-analysis · api-design#
18. Jul 11, 2026Shipped

    From real work to a published portfolio.

    Shipped the engine behind this very site: a bilingual portfolio whose content is curated automatically from real work sessions. A session-end hook evaluates each Claude Code session against a relevance rubric; when something clears the bar, it is written up in two languages, sanitized, and committed as a draft that publishes with one command.

    Outcome: The site you are reading was produced by this pipeline.

    Technical detail

    Three components: an Astro static site (i18n EN/PT, feed/projects/CV rendered from JSON at build time), a Python MCP server exposing five tools (log_activity, review_and_publish, add_project, update_now, get_portfolio_state) that write through the GitHub Contents API, and a Stop hook that embeds the rubric verbatim so criteria changes require no code changes. Validation runs three layers deep: JSON Schema, bilingual completeness (a missing translation fails the build, by design), and a local denylist as a deterministic backstop behind the LLM's category-based sanitization.

    astro · mcp · python · claude-code · i18nPart of Living Portfolio#
19. Jul 11, 2026Learning

    A timeline that draws itself without JavaScript.

    First real-world use of CSS scroll-driven animations: this feed's timeline draws itself as you scroll, with zero JavaScript. The motion concept for the site — pulse, trace, breath — had to communicate 'alive' without a single animation library.

    Outcome: Feed timeline draws on scroll in ~20 lines of CSS; zero JS, zero libraries.

    Technical detail

    animation-timeline: view() maps an animation's progress to an element's position in the viewport — the timeline border scales from 0 to full height as each entry crosses it. Wrapped in @supports so unsupported browsers just see the static border, and disabled under prefers-reduced-motion. The rest of the motion system is a ~30-line inline IntersectionObserver (staggered reveal) gated behind an html.js class, so a JavaScript failure can never leave content hidden.

    css · scroll-driven-animations · progressive-enhancementPart of Living Portfolio#
20. Jul 10, 2026Problem solved

    A zero-cost LLM, no API key.

    Needed AI-powered route re-ranking for a personal transit app but had no API key — only a Claude Code subscription. Built a zero-cost LLM integration by calling the Claude CLI in headless mode via subprocess. Hit a non-obvious bug: global session hooks were being inherited by the subprocess, injecting evaluation text into the JSON output and silently corrupting every response. Traced the corruption, found the root cause, and fixed it with a single flag.

    Outcome: M4 milestone complete: AI re-ranking working without an API key; 57 tests green, 6/6 ranking outputs valid.

    Technical detail

    • Stack: Python + subprocess, Claude Code CLI (claude -p --output-format json)
    • Added ROTAS_LLM_PROVIDER=claude-code provider alongside the existing API-based path, so both can coexist
    • Root cause: global Stop hooks (used for portfolio session evaluation) were inherited by the headless CLI child process, appending their evaluation output to the result field and breaking JSON parsing downstream
    • Fix: --settings '{"disableAllHooks": true}' passed to the subprocess invocation
    • JSON parser made tolerant of surrounding markdown prose as a defensive measure against future prompt drift
    • Trade-off accepted: ~95s per query vs 5-10s with a real API key — cost-zero outweighs latency for a personal prototype; UX shows a 'slow AI' warning
    • Acceptance criteria met: 6/6 itinerary IDs correct, justifications ≤280 chars, contextual re-ranking correctly inverted priority order and flagged a late-night street transfer as eliminatory

    llm-integration · subprocess · transit-routing · bug-fix · zero-costPart of Rotas-SP — transit routing that thinks past the obvious#
21. Jul 9, 2026Decision

    Git as the database. Review as the boundary.

    My portfolio needed to update itself without becoming a server to maintain. The decision: no CMS, no database, no backend — content lives as JSON in a git repository, every update is a commit, and the site is a static build triggered by pushes. A drafts branch acts as a mandatory buffer: automation can write, but nothing goes public without a human merge.

    Outcome: Zero infrastructure to operate; every content change is auditable in git history.

    Technical detail

    A competent peer might reasonably have chosen a headless CMS or a small API. Git-as-database won on three fronts: full history and rollback for free, zero infrastructure cost, and a deploy pipeline (push → rebuild) that static hosts provide out of the box. The drafts branch exists for confidentiality, not relevance — client work must never auto-publish without a human glance. Trade-off accepted: no real-time updates, which a portfolio doesn't need.

    architecture · static-site · git · astroPart of Living Portfolio#

Nothing in this slice. Try another term or type.