Research fuzzing framework that treats AFL++ mutation selection as a sequential decision problem and learns which of 47 mutation primitives to fire next based on execution context.

What I built

Each model variant is two cooperating processes:

• C mutator (src/mutator_m*.c) — an AFL++ custom-mutator plugin. Each fuzzing step, it reads trace_bits, writes the state vector into a memory-mapped shared buffer, polls for an action, and applies the selected mutation primitive.
• Python RL server (scripts/rl_server.py) — hosts a DQN with a replay buffer and ε-greedy exploration, reads state from SHM, computes an action, runs the training loop, writes back.

The two processes coordinate via mmap + GCC atomic-builtin sequence numbers (__ATOMIC_RELEASE on the producer side, __ATOMIC_ACQUIRE on the consumer side) — keeps the per-execution decision loop sub-millisecond, no syscalls in the hot path.

The model variants

I shipped 6 variants distinguished by their state representation:

• M0_0 — 3 dims: [coverage_n, new_edges_n, crashes_n]
• M1_0 — 12 dims: edge-stability distribution over all 65,536 edges
• M1_1 — 13 dims: edge-stability over visited edges only + visit count
• M1_2 — 64 dims: M1_1 + input-buffer features (length, entropy, byte histogram)
• M2 — 97 dims: per-mutator trace-bit magnitudes (47 enabled + 47 disabled means)
• M3_0 — 13 dims: differential-analysis-derived (heat distribution, entropy, timing, velocity)

The reward function is straightforward — coverage delta plus log-scaled crash delta — but the interesting design question is what features make the policy learnable. Higher state dimensionality buys some signal, but past a certain point training noise dominates. M3_0 was an attempt to fix that with hand-crafted features derived from comparing buggy vs fixed versions of libxml2.

The action space

The 47 actions cover AFL++‘s full deterministic and havoc mutation menu: deterministic bit/byte flips (16), interesting-value substitutions (5), havoc random mutations (18), dictionary insert/overwrite (4), and meta (2). Refactored up from an initial 7-action design because action collapse made the smaller space behave like a static schedule under late-training.

Validated against

6 FuzzBench targets: jsoncpp, freetype2, libxml2, re2, harfbuzz, libpng. Each model is compared against vanilla AFL++ on same-steps and same-time axes, with statistical aggregation across multi-run trials.

Bugs I had to chase

• Inverted reward signs causing policy divergence in M1_1 (training reward up, evaluation reward down for ~50k steps before it became obvious).
• An ε-decay schedule that hit zero too early on small targets, causing premature plateau detection.
• Action collapse at evaluation time on M2 — the larger state space made certain Q-values nearly identical, and argmax consistently picked the same action.
• IPC-bound throughput limits (~1.3k exec/s in early M0_0 vs ~58k for vanilla AFL++), traced to Python-side overhead in the loop. Mitigated but never fully closed in the framework as it stands.