# Fuzzer Benchmark Report (from bug-count CSV)

- Time budget: **2.00h**

> **Warning — trial run.** This benchmark was executed with fewer than 10 instances per fuzzer and/or a time budget shorter than 24h. Results from trial runs are meant for debugging purposes and are not valid for extracting conclusions across different fuzzers.

## ⚠️ Run health

The following fuzzer legs show signs of not having run for the full time budget. Their results below are **not comparable** to healthy legs.

- ⚠️ **foundry**: no bugs and no runtime activity samples were parsed from any instance — the fuzzer most likely never executed (startup/compile failure). Inspect the instance bootstrap logs before drawing conclusions from this leg.

## Executive summary
This report is derived solely from cumulative bugs-found over time across repeated runs per fuzzer. It emphasizes robust, distribution-based metrics (median/IQR, success rates, time-to-k) and shape-based behavior (plateau time, late discovery share) instead of single-run time-to-first-bug.

## Bugs found at fixed time budgets (median [IQR])
| Fuzzer | Runs | 1h |
|---|---|---|
| medusa | 4 | 6 [5,6] |
| recon-fuzzer | 4 | 4 [4,4] |
| echidna | 4 | 4 [4,4] |
| foundry | 4 | 0 [0,0] |

## Overall metrics
| Fuzzer | AUC (norm) | Plateau time | Late discovery share | Final median | Final IQR |
|---|---|---|---|---|---|
| medusa | 0.656 | 1.70h | 0.214 | 7 | 0.50 |
| recon-fuzzer | 0.500 | 1.70h | 0.200 | 5 | 2.00 |
| echidna | 0.475 | 0.10h | 0.000 | 4 | 0.25 |
| foundry | 0.000 | 0.00h | 0.000 | 0 | 0.00 |

## Milestones: time-to-k and success rates
| Fuzzer | time-to-1 (p50) | time-to-3 (p50) | time-to-5 (p50) | reach-1 rate | reach-3 rate | reach-5 rate |
|---|---|---|---|---|---|---|
| medusa | 0.10h | 0.10h | 0.10h | 100.0% | 100.0% | 100.0% |
| recon-fuzzer | 0.00h | 0.10h | 1.05h | 100.0% | 100.0% | 50.0% |
| echidna | 0.10h | 0.10h | 1.20h | 100.0% | 100.0% | 25.0% |
| foundry | inf | inf | inf | 0.0% | 0.0% | 0.0% |

## Throughput metrics (if supported by log format)
Values are run-level rates aggregated per fuzzer; `n/a` indicates the parser could not recover that metric from logs.
| Fuzzer | Runs | Tx/s runs | Tx/s p50 [p25,p75] | Gas/s runs | Gas/s p50 [p25,p75] |
|---|---|---|---|---|---|
| medusa | 4 | 4 | 1415.50 [1390.25,1432.00] | 4 | 285991466.00 [284969623.00,289126579.75] |
| recon-fuzzer | 4 | 4 | 6574.85 [6547.34,6614.40] | 4 | 21645414.00 [20288338.50,23547804.25] |
| echidna | 4 | 4 | 2695.21 [2690.43,2718.04] | 4 | 0.00 [0.00,0.00] |

## Progress metrics from logs (fuzzer-specific proxies)
Coverage/corpus values are parsed from each fuzzer's native progress output and are useful for trend context, not strict cross-fuzzer equivalence.
| Fuzzer | Runs | Seq/s runs | Seq/s p50 [p25,p75] | Coverage runs | Coverage p50 [p25,p75] | Corpus runs | Corpus p50 [p25,p75] |
|---|---|---|---|---|---|---|---|
| medusa | 4 | 4 | 14.00 [13.75,14.00] | 4 | 12179.50 [12173.50,12213.50] | 4 | 547.50 [545.75,549.50] |
| recon-fuzzer | 4 | 0 | n/a | 4 | 143819.00 [143791.25,143822.50] | 4 | 584.00 [577.75,589.50] |
| echidna | 4 | 0 | n/a | 4 | 136659.00 [136392.50,136907.00] | 4 | 506.50 [500.75,512.00] |

## Statistical comparison (Mann-Whitney U test)

Pairwise Mann-Whitney U tests on end-of-budget bug counts (two-sided).
Bonferroni correction applied for 6 comparison(s). Significance level: alpha = 0.05.

**Warnings:**
- One or more fuzzers have fewer than 5 runs. Statistical power may be limited.

| Fuzzer A | Fuzzer B | Median A | Median B | U statistic | p-value | p (corrected) | Significant |
|---|---|---|---|---|---|---|---|
| medusa | recon-fuzzer | 7 | 5 | 15.0 | 0.0515 | 0.3091 | no |
| medusa | echidna | 7 | 4 | 16.0 | 0.0256 | 0.1535 | no |
| medusa | foundry | 7 | 0 | 16.0 | 0.0202 | 0.1212 | no |
| recon-fuzzer | echidna | 5 | 4 | 11.0 | 0.4047 | 1.0000 | no |
| recon-fuzzer | foundry | 5 | 0 | 16.0 | 0.0194 | 0.1162 | no |
| echidna | foundry | 4 | 0 | 16.0 | 0.0177 | 0.1062 | no |

No pairwise comparison reached significance after Bonferroni correction.

> **Note:** The Mann-Whitney U test assesses whether one fuzzer tends to find
> more bugs than another across runs. It does not measure effect size or
> practical importance. Small sample sizes (fewer than 5 runs) reduce
> statistical power.

## Shape-based interpretation (rules of thumb)
- **Fast-start / early-plateau**: high early checkpoint median + early plateau time + low late discovery share.
- **Steady**: moderate AUC, later plateau, consistent improvements across checkpoints, moderate variance.
- **Slow-burn / late-surge**: low early checkpoints but high late discovery share and later plateau time; often higher final median.

## Limitations
- Core metrics in this section are count-based; use `broken_invariants.md` / `broken_invariants.csv` for invariant identities.
- Severity, exploitability, and root-cause uniqueness cannot be measured directly without richer per-bug metadata.
- Harness design still affects results; mitigate by keeping harness identical across fuzzers and reporting many runs.
