# Fuzzer Benchmark Report (from bug-count CSV)

- Time budget: **4.00h**

> **Warning — trial run.** This benchmark was executed with fewer than 10 instances per fuzzer and/or a time budget shorter than 24h. Results from trial runs are meant for debugging purposes and are not valid for extracting conclusions across different fuzzers.

## Executive summary
This report is derived solely from cumulative bugs-found over time across repeated runs per fuzzer. It emphasizes robust, distribution-based metrics (median/IQR, success rates, time-to-k) and shape-based behavior (plateau time, late discovery share) instead of single-run time-to-first-bug.

## Bugs found at fixed time budgets (median [IQR])
| Fuzzer | Runs | 1h | 4h |
|---|---|---|---|
| medusa | 2 | 15 [15,15] | 15 [15,15] |
| echidna | 2 | 14 [14,15] | 15 [14,16] |
| recon-fuzzer | 2 | 14 [14,14] | 14 [14,14] |
| foundry | 2 | 14 [13,14] | 14 [14,14] |

## Overall metrics
| Fuzzer | AUC (norm) | Plateau time | Late discovery share | Final median | Final IQR |
|---|---|---|---|---|---|
| medusa | 0.916 | 0.10h | 0.000 | 15 | 0.00 |
| echidna | 0.902 | 1.40h | 0.000 | 15 | 1.00 |
| recon-fuzzer | 0.856 | 0.20h | 0.000 | 14 | 0.00 |
| foundry | 0.844 | 1.20h | 0.000 | 14 | 0.00 |

## Milestones: time-to-k and success rates
| Fuzzer | time-to-1 (p50) | time-to-3 (p50) | time-to-5 (p50) | reach-1 rate | reach-3 rate | reach-5 rate |
|---|---|---|---|---|---|---|
| medusa | 0.00h | 0.10h | 0.10h | 100.0% | 100.0% | 100.0% |
| echidna | 0.10h | 0.10h | 0.10h | 100.0% | 100.0% | 100.0% |
| recon-fuzzer | 0.00h | 0.05h | 0.10h | 100.0% | 100.0% | 100.0% |
| foundry | 0.10h | 0.10h | 0.10h | 100.0% | 100.0% | 100.0% |

## Throughput metrics (if supported by log format)
Values are run-level rates aggregated per fuzzer; `n/a` indicates the parser could not recover that metric from logs.
| Fuzzer | Runs | Tx/s runs | Tx/s p50 [p25,p75] | Gas/s runs | Gas/s p50 [p25,p75] |
|---|---|---|---|---|---|
| medusa | 2 | 2 | 47991.00 [47830.50,48151.50] | 2 | 2514082134.00 [2513608802.00,2514555466.00] |
| echidna | 2 | 2 | 5596.10 [5543.79,5648.41] | 2 | 27230235923.50 [13615117961.75,40845353885.25] |
| recon-fuzzer | 2 | 2 | 857.44 [853.21,861.66] | 2 | 4945544.50 [4156940.75,5734148.25] |
| foundry | 2 | 2 | 185197.04 [182087.17,188306.90] | 2 | 13993386024.76 [13951408468.82,14035363580.70] |

## Progress metrics from logs (fuzzer-specific proxies)
Sequence-rate and corpus values are parsed from native progress output and are useful for within-tool trend context.
| Fuzzer | Runs | Seq/s runs | Seq/s p50 [p25,p75] | Corpus runs | Corpus p50 [p25,p75] |
|---|---|---|---|---|---|
| medusa | 2 | 2 | 479.50 [478.25,480.75] | 2 | 206.50 [204.25,208.75] |
| echidna | 2 | 0 | n/a | 2 | 26.50 [26.25,26.75] |
| recon-fuzzer | 2 | 0 | n/a | 2 | 22.00 [21.50,22.50] |
| foundry | 2 | 0 | n/a | 2 | 0.00 [0.00,0.00] |

## Coverage over time (opt-in, fuzzer-native signals)
- Status: **enabled**
- Source-based coverage is preferred, but the current timestamped progress formats expose only native bytecode/instrumentation proxies. The signals below have different units and must not be ranked, pooled, or plotted on a shared scale.
- Foundry showmap artifacts are a separate, opt-in post-campaign edge replay; they are not real-time source coverage.

| Fuzzer | Native signal | Source-based? | Live availability | Runs with signal | Temporal observations | Observed parser provenance | Final native signal p50 [p25,p75] | Limitation |
|---|---|---|---|---|---|---|---|---|
| medusa | `branches hit` (branches) | No (runtime proxy) | Medusa status lines | 2/2 | live; 9592 samples; first 0.00-0.00h, last 4.00-4.00h of 4.00h budget | text-metrics:branches_hit | 1546.00 [1546.00,1546.00] | Branch identities depend on compiled bytecode/config; no source locations. |
| echidna | `cov` coverage points (coverage points) | No (runtime proxy) | Echidna status lines | 2/2 | live; 9574 samples; first 0.00-0.00h, last 4.00-4.00h of 4.00h budget | text-metrics:cov | 15377.00 [15373.00,15381.00] | Tool/config-specific counter; no source locations in the progress log. |
| recon-fuzzer | `Unique instructions` (instructions) | No (runtime proxy) | End-of-run Recon summary (not a live series) | 2/2 | end-of-run only; 2 sample(s); excluded from time-series chart | text-metrics:unique_instructions | 16459.50 [16457.75,16461.25] | Tool/version-specific final instruction count; no source locations or intermediate observations. |
| foundry | `cumulative_edges_seen` (instrumented edges) | No (runtime proxy) | Compatible Foundry JSON pulse builds | 2/2 | live; 5734 samples; first 0.00-0.00h, last 3.98-3.98h of 4.00h budget | json-metrics:cumulative_edges_seen | 0.00 [0.00,0.00] | Edge identities depend on the build/instrumentation; showmap replay is separate and post-campaign. |

## Function selector sanity checks

Counts are occurrences in unique saved corpus sequences, not runtime execution frequencies The complete per-instance distribution is in `selector_distribution.csv`; `selector_summary.json` preserves statuses and provenance.

- Expected-set status: **available** (46 selector(s))
- Expected-set provenance: peer-consensus heuristic: observed in every available corpus for at least two independent evidence families (Echidna and Recon count as one related typed-corpus family); not benchmark ground truth

| Fuzzer | Status | Instances with telemetry | Saved-corpus calls | Unique selectors |
|---|---|---:|---:|---:|
| echidna | available | 2/2 | 5289 | 47 |
| foundry | unavailable | 0/2 | 0 | 0 |
| medusa | available | 2/2 | 11607 | 46 |
| recon-fuzzer | available | 2/2 | 4218 | 45 |

### Most common saved-corpus selectors

| Fuzzer | Selector | Function | Calls | Share |
|---|---|---|---:|---:|
| echidna | `0xd5afe6cd` | `optIn(uint8)` | 168 | 3.18% |
| echidna | `0xd89cecc4` | `testNonRebasingSupplyVsTotalSupply()` | 157 | 2.97% |
| echidna | `0xf24a44c9` | `assert_canary_ASSERTION_CANARY(uint256)` | 145 | 2.74% |
| echidna | `0xdad9f8f0` | `approve(uint8,uint8,uint256)` | 140 | 2.65% |
| echidna | `0x0a1fc5e7` | `testMintBalance(uint8,uint256)` | 139 | 2.63% |
| echidna | `0x43a45e75` | `testTransferBalanceSentLess(uint8,uint8,uint256)` | 138 | 2.61% |
| echidna | `0x6900d46e` | `transfer(uint8,uint8,uint256)` | 137 | 2.59% |
| echidna | `0xc7df1ada` | `testTransferFromShouldNotRevert(uint8,uint8,uint8,uint256)` | 135 | 2.55% |
| echidna | `0x1a042ae7` | `invariant_canary()` | 132 | 2.50% |
| echidna | `0x178e5c32` | `testChangeSupply(uint256)` | 130 | 2.46% |
| medusa | `0x00a3c5d1` | `testTransferWithinBalanceDoesNotRevert(uint8,uint8,uint256)` | 415 | 3.58% |
| medusa | `0x9164cd26` | `testTransferBalanceSentMore(uint8,uint8,uint256)` | 405 | 3.49% |
| medusa | `0x657808a3` | `testBalanceOfShouldNotRevert(uint8)` | 404 | 3.48% |
| medusa | `0x379db993` | `testTransferBalanceReceivedLessRounding(uint8,uint8,uint256)` | 364 | 3.14% |
| medusa | `0xc7df1ada` | `testTransferFromShouldNotRevert(uint8,uint8,uint8,uint256)` | 358 | 3.08% |
| medusa | `0x324b65f5` | `testApprove(uint8,uint8,uint256)` | 350 | 3.02% |
| medusa | `0x43a45e75` | `testTransferBalanceSentLess(uint8,uint8,uint256)` | 324 | 2.79% |
| medusa | `0x9de07e5a` | `testOptInBalanceRounding(uint8)` | 322 | 2.77% |
| medusa | `0x7fc05f0a` | `testOptOutTotalSupply(uint8)` | 309 | 2.66% |
| medusa | `0x6dcc9dcc` | `testRebasingCreditsPerTokenNotIncreased()` | 301 | 2.59% |
| recon-fuzzer | `0x13ffd369` | `testBurnBalance(uint8,uint256)` | 169 | 4.01% |
| recon-fuzzer | `0x0a1fc5e7` | `testMintBalance(uint8,uint256)` | 137 | 3.25% |
| recon-fuzzer | `0xe4840d4f` | `testTransferBalanceReceivedMore(uint8,uint8,uint256)` | 128 | 3.03% |
| recon-fuzzer | `0x43a45e75` | `testTransferBalanceSentLess(uint8,uint8,uint256)` | 127 | 3.01% |
| recon-fuzzer | `0x178e5c32` | `testChangeSupply(uint256)` | 123 | 2.92% |
| recon-fuzzer | `0x72a96677` | `transferFrom(uint8,uint8,uint8,uint256)` | 123 | 2.92% |
| recon-fuzzer | `0x8f287234` | `testBurnAllBalanceShouldNotRevert(uint8)` | 108 | 2.56% |
| recon-fuzzer | `0xd89cecc4` | `testNonRebasingSupplyVsTotalSupply()` | 107 | 2.54% |
| recon-fuzzer | `0x379db993` | `testTransferBalanceReceivedLessRounding(uint8,uint8,uint256)` | 103 | 2.44% |
| recon-fuzzer | `0x7fc05f0a` | `testOptOutTotalSupply(uint8)` | 103 | 2.44% |

### Selector diagnostic findings

- **INFO**: i-00286b19f02049aa2-recon-v0.4.6: peer-heuristic gap (diagnostic, not ground truth): absent 2 selector(s): 0x12b58349, 0x54113357
- **INFO**: i-0922c2f8649781881-recon-v0.4.6: peer-heuristic gap (diagnostic, not ground truth): absent 2 selector(s): 0x12b58349, 0x54113357

### Selector telemetry limitations

- i-024a9959ae72deb9f-foundry-git-02c05d9: Foundry selector distribution is unavailable because no persisted corpus was present; failure-event selectors are intentionally not used as a distribution
- i-0eb2f56ba7c7e57aa-foundry-git-02c05d9: Foundry selector distribution is unavailable because no persisted corpus was present; failure-event selectors are intentionally not used as a distribution

## Statistical comparison (Mann-Whitney U and Vargha-Delaney A12)

Pairwise Mann-Whitney U tests and Vargha-Delaney A12 effect sizes on end-of-budget bug counts.
The tests are two-sided, with Bonferroni correction applied for 6 comparison(s). Significance level: alpha = 0.05.

**Warnings:**
- One or more fuzzers have fewer than 5 runs. Statistical power may be limited.

| Fuzzer A | Fuzzer B | Median A | Median B | U statistic | A12 (A over B) | Effect magnitude | p-value | p (corrected) | Significant |
|---|---|---|---|---|---|---|---|---|---|
| medusa | echidna | 15 | 15 | 2.0 | 0.500 | negligible | 1.0000 | 1.0000 | no |
| medusa | recon-fuzzer | 15 | 14 | 4.0 | 1.000 | large | 0.1939 | 1.0000 | no |
| medusa | foundry | 15 | 14 | 4.0 | 1.000 | large | 0.1939 | 1.0000 | no |
| echidna | recon-fuzzer | 15 | 14 | 3.0 | 0.750 | large | 0.6171 | 1.0000 | no |
| echidna | foundry | 15 | 14 | 3.0 | 0.750 | large | 0.6171 | 1.0000 | no |
| recon-fuzzer | foundry | 14 | 14 | 2.0 | 0.500 | negligible | 1.0000 | 1.0000 | no |

No pairwise comparison reached significance after Bonferroni correction.

> **Note:** A12 is the probability that a randomly selected Fuzzer A run
> has a higher final bug count than a randomly selected Fuzzer B run,
> counting ties as half. Values above 0.5 favor A, values below 0.5 favor B,
> and 0.5 means equal tendency. Magnitude thresholds by distance from 0.5
> are negligible (<0.06), small (<0.14), medium (<0.21), and large
> (>=0.21). Small sample sizes (fewer than 5 runs) reduce statistical power.
> A12 and its magnitude label are descriptive; the thresholds are rules of
> thumb. Neither the effect size nor statistical significance establishes
> practical importance, causation, or performance beyond the observed runs.

## Shape-based interpretation (rules of thumb)
- **Fast-start / early-plateau**: high early checkpoint median + early plateau time + low late discovery share.
- **Steady**: moderate AUC, later plateau, consistent improvements across checkpoints, moderate variance.
- **Slow-burn / late-surge**: low early checkpoints but high late discovery share and later plateau time; often higher final median.

## Limitations
- Core count-based charts use normalized event identities stored under the legacy `bugs_found` column. They do not count crash inputs, but they are not confirmed root-cause bug counts.
- Use `known_bug_report.md`, `known_bug_summary.csv`, and `known_bug_findings.csv` for evidence-backed known-bug hit rates; use `broken_invariants.md` / `broken_invariants.csv` for raw invariant identities.
- Severity, exploitability, and root-cause uniqueness cannot be measured directly without richer per-bug metadata.
- Harness design still affects results; mitigate by keeping harness identical across fuzzers and reporting many runs.
