Research framework for LLM-guided fuzzing seed corpus synthesis. The framework is the contribution — what gets measured (and how the random baseline is forced to score 0%) is more interesting than any single number it produces.

What it does

Given a coverage-instrumented target binary and its upstream test suite, the framework:

1. Identifies hard-to-reach branches (M2 target set, frozen per target).
2. Prompts an LLM (across 7 candidates × 5 context variants × 7 prompt strategies) to synthesize inputs targeting those branches.
3. Evaluates the resulting seeds by replaying them under instrumentation:
   • M1 — total edges hit
   • M2 — fraction of the frozen hard-branch set hit by the seed corpus

A hard branch is defined as one hit by struct-guided seeds but never by the random baseline. By construction the random baseline scores 0% M2, so any model gain on M2 is provable signal, not measurement noise.

Why the framework matters

LLM-eval frameworks usually fail in one of three ways:

1. Hidden caching cost (model-level + provider-level). Mine has a SHA-256-keyed cache that’s reproducible across processes; you can see exactly what each experiment cost.
2. Hand-wavy statistical claims. The framework wires Mann-Whitney, Vargha-Delaney, and Friedman-Nemenyi tests so claims of “model A beats model B” survive multiple testing corrections.
3. Prompt-strategy comparisons that aren’t actually orthogonal to context. The strategy axis (default, CoT, few-shot, self-critique, chain, tool-use, tool-use+retrieval) is a separate dimension from the context axis (none, src, src+tests, all, src+gaps) — both are explicit in the design.

Tool use, properly

One of the 7 prompt strategies is oracle-gated tool use: the model calls a validity tool mid-generation, receives a verdict (valid input format / parse error / coverage prediction), and revises its candidate seed. This isn’t “agentic”; it’s a deliberately scoped self-correction loop with measurable cost.

Cost transparency

There’s a cost_audit.py that walks the SHA-256 cache directory and reports per-provider, per-day spend. It’s the single source of truth for “what did this campaign cost.” Current cumulative: $100.09 across 14,161 cached calls (Anthropic $86.25, UF LiteLLM proxy $13.83). The demo above shows it live.

Headline results (from the framework’s experiment-2 report)

• RE2 (text format): best M2 = 0.867 at v1_src / nemotron-120b
• RE2 best M1: +335 edges (+28.9%) at v2_src_tests / codestral-22b
• harfbuzz (binary format): best M1 = +444 edges (+80%) at v0_none / Sonnet
• Strongest free model overall: codestral-22b on text-format targets
• Source code is the load-bearing context variable: every context with src available outperforms every context without it on every model

(Raw per-cell results live in the repo under results/ablation_*/summary.json.)