# Fuzzer Benchmark Report (from bug-count CSV)

- Time budget: **4.00h**

> **Warning — trial run.** This benchmark was executed with fewer than 10 instances per fuzzer and/or a time budget shorter than 24h. Results from trial runs are meant for debugging purposes and are not valid for extracting conclusions across different fuzzers.

## Executive summary
This report is derived solely from cumulative bugs-found over time across repeated runs per fuzzer. It emphasizes robust, distribution-based metrics (median/IQR, success rates, time-to-k) and shape-based behavior (plateau time, late discovery share) instead of single-run time-to-first-bug.

## Bugs found at fixed time budgets (median [IQR])
| Fuzzer | Runs | 1h | 4h |
|---|---|---|---|
| medusa | 2 | 14 [14,14] | 14 [14,14] |
| echidna | 2 | 14 [14,14] | 14 [14,15] |
| foundry | 2 | 12 [12,12] | 14 [13,14] |
| recon-fuzzer | 2 | 6 [6,6] | 6 [6,6] |

## Overall metrics
| Fuzzer | AUC (norm) | Plateau time | Late discovery share | Final median | Final IQR |
|---|---|---|---|---|---|
| medusa | 0.898 | 0.40h | 0.000 | 14 | 0.00 |
| echidna | 0.889 | 1.30h | 0.000 | 14 | 0.50 |
| foundry | 0.825 | 2.00h | 0.000 | 14 | 0.50 |
| recon-fuzzer | 0.392 | 0.10h | 0.000 | 6 | 0.00 |

## Milestones: time-to-k and success rates
| Fuzzer | time-to-1 (p50) | time-to-3 (p50) | time-to-5 (p50) | reach-1 rate | reach-3 rate | reach-5 rate |
|---|---|---|---|---|---|---|
| medusa | 0.00h | 0.10h | 0.10h | 100.0% | 100.0% | 100.0% |
| echidna | 0.10h | 0.10h | 0.10h | 100.0% | 100.0% | 100.0% |
| foundry | 0.10h | 0.10h | 0.10h | 100.0% | 100.0% | 100.0% |
| recon-fuzzer | 0.00h | 0.10h | 0.10h | 100.0% | 100.0% | 100.0% |

## Throughput metrics (if supported by log format)
Values are run-level rates aggregated per fuzzer; `n/a` indicates the parser could not recover that metric from logs.
| Fuzzer | Runs | Tx/s runs | Tx/s p50 [p25,p75] | Gas/s runs | Gas/s p50 [p25,p75] |
|---|---|---|---|---|---|
| medusa | 2 | 2 | 4848.00 [4761.50,4934.50] | 2 | 1483285370.00 [1468525720.00,1498045020.00] |
| echidna | 2 | 2 | 793.35 [778.10,808.60] | 2 | 9723997054.00 [9039376017.00,10408618091.00] |
| foundry | 2 | 2 | 25930.58 [23388.79,28472.38] | 2 | 6378039431.01 [6212524989.01,6543553873.01] |
| recon-fuzzer | 2 | 2 | 604.70 [585.05,624.35] | 2 | 88277160.50 [80386847.75,96167473.25] |

## Progress metrics from logs (fuzzer-specific proxies)
Sequence-rate and corpus values are parsed from native progress output and are useful for within-tool trend context.
| Fuzzer | Runs | Seq/s runs | Seq/s p50 [p25,p75] | Corpus runs | Corpus p50 [p25,p75] |
|---|---|---|---|---|---|
| medusa | 2 | 2 | 48.50 [47.75,49.25] | 2 | 139.00 [136.00,142.00] |
| echidna | 2 | 0 | n/a | 2 | 62.00 [61.50,62.50] |
| foundry | 2 | 0 | n/a | 2 | 0.00 [0.00,0.00] |
| recon-fuzzer | 2 | 0 | n/a | 2 | 84.00 [71.00,97.00] |

## Coverage over time (opt-in, fuzzer-native signals)
- Status: **enabled**
- Source-based coverage is preferred, but the current timestamped progress formats expose only native bytecode/instrumentation proxies. The signals below have different units and must not be ranked, pooled, or plotted on a shared scale.
- Foundry showmap artifacts are a separate, opt-in post-campaign edge replay; they are not real-time source coverage.

| Fuzzer | Native signal | Source-based? | Live availability | Runs with signal | Temporal observations | Observed parser provenance | Final native signal p50 [p25,p75] | Limitation |
|---|---|---|---|---|---|---|---|---|
| medusa | `branches hit` (branches) | No (runtime proxy) | Medusa status lines | 2/2 | live; 9574 samples; first 0.00-0.00h, last 4.00-4.00h of 4.00h budget | text-metrics:branches_hit | 5204.00 [5203.50,5204.50] | Branch identities depend on compiled bytecode/config; no source locations. |
| echidna | `cov` coverage points (coverage points) | No (runtime proxy) | Echidna status lines | 2/2 | live; 9567 samples; first 0.00-0.00h, last 4.00-4.00h of 4.00h budget | text-metrics:cov | 44530.50 [44528.25,44532.75] | Tool/config-specific counter; no source locations in the progress log. |
| foundry | `cumulative_edges_seen` (instrumented edges) | No (runtime proxy) | Compatible Foundry JSON pulse builds | 2/2 | live; 5731 samples; first 0.00-0.00h, last 3.98-3.98h of 4.00h budget | json-metrics:cumulative_edges_seen | 0.00 [0.00,0.00] | Edge identities depend on the build/instrumentation; showmap replay is separate and post-campaign. |
| recon-fuzzer | `Unique instructions` (instructions) | No (runtime proxy) | End-of-run Recon summary (not a live series) | 2/2 | end-of-run only; 2 sample(s); excluded from time-series chart | text-metrics:unique_instructions | 45237.00 [45228.50,45245.50] | Tool/version-specific final instruction count; no source locations or intermediate observations. |

## Function selector sanity checks

Counts are occurrences in unique saved corpus sequences, not runtime execution frequencies The complete per-instance distribution is in `selector_distribution.csv`; `selector_summary.json` preserves statuses and provenance.

- Expected-set status: **available** (67 selector(s))
- Expected-set provenance: peer-consensus heuristic: observed in every available corpus for at least two independent evidence families (Echidna and Recon count as one related typed-corpus family); not benchmark ground truth

| Fuzzer | Status | Instances with telemetry | Saved-corpus calls | Unique selectors |
|---|---|---:|---:|---:|
| echidna | available | 2/2 | 12329 | 68 |
| foundry | unavailable | 0/2 | 0 | 0 |
| medusa | available | 2/2 | 10744 | 67 |
| recon-fuzzer | available | 2/2 | 16128 | 67 |

### Most common saved-corpus selectors

| Fuzzer | Selector | Function | Calls | Share |
|---|---|---|---:|---:|
| echidna | `0x0325f4e1` | `addSplitsReceiver(uint8,uint8,uint32)` | 240 | 1.95% |
| echidna | `0xed0c06d6` | `removeAllSplits(uint8)` | 232 | 1.88% |
| echidna | `0x81b0058c` | `testSqueezeViewVsActual(uint8,uint8)` | 231 | 1.87% |
| echidna | `0xb89fedbc` | `addStreamWithClamping(uint8,uint8,uint160,uint32,uint32,int128)` | 227 | 1.84% |
| echidna | `0xfabaa91d` | `testSetStreamBalanceWithdrawAllShouldNotRevert(uint8)` | 215 | 1.74% |
| echidna | `0x82ce0f6d` | `testReceiveStreamsShouldNotRevert(uint8)` | 210 | 1.70% |
| echidna | `0x8319b8db` | `receiveStreamsSplitAndCollectToSelf(uint8)` | 210 | 1.70% |
| echidna | `0x45b9b0ba` | `testReceiveStreamsViewVsActual(uint8,uint32)` | 208 | 1.69% |
| echidna | `0x79f368d0` | `squeezeAllSenders(uint8)` | 203 | 1.65% |
| echidna | `0x1a3ed582` | `setStreams(uint8,uint8,uint160,uint32,uint32,int128)` | 201 | 1.63% |
| medusa | `0x0325f4e1` | `addSplitsReceiver(uint8,uint8,uint32)` | 349 | 3.25% |
| medusa | `0x1ba3a23c` | `testReceiveStreamsViewConsistency(uint8,uint32)` | 306 | 2.85% |
| medusa | `0x8d8b7f3b` | `setSplits(uint8,uint8,uint32)` | 293 | 2.73% |
| medusa | `0x462664e4` | `testRemoveStreamShouldNotRevert(uint8,uint256)` | 285 | 2.65% |
| medusa | `0x218be192` | `invariantSumAmtDeltaIsZero(uint8)` | 280 | 2.61% |
| medusa | `0x1a3ed582` | `setStreams(uint8,uint8,uint160,uint32,uint32,int128)` | 279 | 2.60% |
| medusa | `0x47bc779b` | `testAddSplitsShouldNotRevert(uint8,uint8,uint32)` | 268 | 2.49% |
| medusa | `0x3bfbc3bd` | `setStreamBalance(uint8,int128)` | 261 | 2.43% |
| medusa | `0xec13b1ca` | `testAddStreamShouldNotRevert(uint8,uint8,uint160,uint32,uint32,int128)` | 253 | 2.35% |
| medusa | `0x2e0c28fb` | `setMaxEndHints(uint32,uint32)` | 248 | 2.31% |
| recon-fuzzer | `0xbcb054fd` | `addStreamImmediatelySqueezable(uint8,uint8,uint160)` | 324 | 2.01% |
| recon-fuzzer | `0xe2b3d196` | `splitAndCollectToSelf(uint8)` | 324 | 2.01% |
| recon-fuzzer | `0x92da0f7a` | `testSqueezeWithFuzzedHistoryShouldNotRevert(uint8,uint8,uint256,bytes32)` | 320 | 1.98% |
| recon-fuzzer | `0xed0c06d6` | `removeAllSplits(uint8)` | 318 | 1.97% |
| recon-fuzzer | `0x2b95b898` | `invariantWithdrawAllTokensShouldNotRevert()` | 314 | 1.95% |
| recon-fuzzer | `0x662fe867` | `invariantAmtPerSecVsMinAmtPerSec(uint8,uint256)` | 312 | 1.93% |
| recon-fuzzer | `0xb777536f` | `squeezeToSelf(uint8)` | 309 | 1.92% |
| recon-fuzzer | `0x19ca05b3` | `testSplittableAfterSplit(uint8)` | 297 | 1.84% |
| recon-fuzzer | `0x13a600d0` | `addStream(uint8,uint8,uint160,uint32,uint32,int128)` | 285 | 1.77% |
| recon-fuzzer | `0x218be192` | `invariantSumAmtDeltaIsZero(uint8)` | 284 | 1.76% |

### Selector diagnostic findings

- **INFO**: i-093a6dd5d2b926065-recon-v0.4.6: peer-heuristic gap (diagnostic, not ground truth): absent 1 selector(s): 0x21d76db7
- **INFO**: i-0ff08ba021a054554-recon-v0.4.6: peer-heuristic gap (diagnostic, not ground truth): absent 1 selector(s): 0x21d76db7

### Selector telemetry limitations

- i-0064e4a60c38595e5-foundry-git-02c05d9: Foundry selector distribution is unavailable because no persisted corpus was present; failure-event selectors are intentionally not used as a distribution
- i-058a92fa40dc71e39-foundry-git-02c05d9: Foundry selector distribution is unavailable because no persisted corpus was present; failure-event selectors are intentionally not used as a distribution

## Statistical comparison (Mann-Whitney U and Vargha-Delaney A12)

Pairwise Mann-Whitney U tests and Vargha-Delaney A12 effect sizes on end-of-budget bug counts.
The tests are two-sided, with Bonferroni correction applied for 6 comparison(s). Significance level: alpha = 0.05.

**Warnings:**
- One or more fuzzers have fewer than 5 runs. Statistical power may be limited.

| Fuzzer A | Fuzzer B | Median A | Median B | U statistic | A12 (A over B) | Effect magnitude | p-value | p (corrected) | Significant |
|---|---|---|---|---|---|---|---|---|---|
| medusa | echidna | 14 | 14.5 | 1.0 | 0.250 | large | 0.6171 | 1.0000 | no |
| medusa | foundry | 14 | 13.5 | 3.0 | 0.750 | large | 0.6171 | 1.0000 | no |
| medusa | recon-fuzzer | 14 | 6 | 4.0 | 1.000 | large | 0.1939 | 1.0000 | no |
| echidna | foundry | 14.5 | 13.5 | 3.5 | 0.875 | large | 0.4142 | 1.0000 | no |
| echidna | recon-fuzzer | 14.5 | 6 | 4.0 | 1.000 | large | 0.2207 | 1.0000 | no |
| foundry | recon-fuzzer | 13.5 | 6 | 4.0 | 1.000 | large | 0.2207 | 1.0000 | no |

No pairwise comparison reached significance after Bonferroni correction.

> **Note:** A12 is the probability that a randomly selected Fuzzer A run
> has a higher final bug count than a randomly selected Fuzzer B run,
> counting ties as half. Values above 0.5 favor A, values below 0.5 favor B,
> and 0.5 means equal tendency. Magnitude thresholds by distance from 0.5
> are negligible (<0.06), small (<0.14), medium (<0.21), and large
> (>=0.21). Small sample sizes (fewer than 5 runs) reduce statistical power.
> A12 and its magnitude label are descriptive; the thresholds are rules of
> thumb. Neither the effect size nor statistical significance establishes
> practical importance, causation, or performance beyond the observed runs.

## Shape-based interpretation (rules of thumb)
- **Fast-start / early-plateau**: high early checkpoint median + early plateau time + low late discovery share.
- **Steady**: moderate AUC, later plateau, consistent improvements across checkpoints, moderate variance.
- **Slow-burn / late-surge**: low early checkpoints but high late discovery share and later plateau time; often higher final median.

## Limitations
- Core count-based charts use normalized event identities stored under the legacy `bugs_found` column. They do not count crash inputs, but they are not confirmed root-cause bug counts.
- Use `known_bug_report.md`, `known_bug_summary.csv`, and `known_bug_findings.csv` for evidence-backed known-bug hit rates; use `broken_invariants.md` / `broken_invariants.csv` for raw invariant identities.
- Severity, exploitability, and root-cause uniqueness cannot be measured directly without richer per-bug metadata.
- Harness design still affects results; mitigate by keeping harness identical across fuzzers and reporting many runs.
