# Fuzzer Benchmark Report (from bug-count CSV)

- Time budget: **4.00h**

> **Warning — trial run.** This benchmark was executed with fewer than 10 instances per fuzzer and/or a time budget shorter than 24h. Results from trial runs are meant for debugging purposes and are not valid for extracting conclusions across different fuzzers.

## Executive summary
This report is derived solely from cumulative bugs-found over time across repeated runs per fuzzer. It emphasizes robust, distribution-based metrics (median/IQR, success rates, time-to-k) and shape-based behavior (plateau time, late discovery share) instead of single-run time-to-first-bug.

## Bugs found at fixed time budgets (median [IQR])
| Fuzzer | Runs | 1h | 4h |
|---|---|---|---|
| recon-fuzzer | 2 | 8 [7,8] | 9 [9,9] |
| echidna | 2 | 6 [6,7] | 7 [7,7] |
| foundry | 2 | 6 [6,7] | 6 [6,7] |
| medusa | 2 | 6 [6,6] | 6 [6,7] |

## Overall metrics
| Fuzzer | AUC (norm) | Plateau time | Late discovery share | Final median | Final IQR |
|---|---|---|---|---|---|
| recon-fuzzer | 0.896 | 1.50h | 0.000 | 9 | 0.00 |
| echidna | 0.719 | 1.20h | 0.000 | 7 | 0.00 |
| foundry | 0.678 | 0.90h | 0.000 | 6 | 0.50 |
| medusa | 0.647 | 2.80h | 0.077 | 6 | 0.50 |

## Milestones: time-to-k and success rates
| Fuzzer | time-to-1 (p50) | time-to-3 (p50) | time-to-5 (p50) | reach-1 rate | reach-3 rate | reach-5 rate |
|---|---|---|---|---|---|---|
| recon-fuzzer | 0.00h | 0.10h | 0.40h | 100.0% | 100.0% | 100.0% |
| echidna | 0.10h | 0.10h | 0.30h | 100.0% | 100.0% | 100.0% |
| foundry | 0.10h | 0.10h | 0.15h | 100.0% | 100.0% | 100.0% |
| medusa | 0.10h | 0.10h | 0.25h | 100.0% | 100.0% | 100.0% |

## Throughput metrics (if supported by log format)
Values are run-level rates aggregated per fuzzer; `n/a` indicates the parser could not recover that metric from logs.
| Fuzzer | Runs | Tx/s runs | Tx/s p50 [p25,p75] | Gas/s runs | Gas/s p50 [p25,p75] |
|---|---|---|---|---|---|
| recon-fuzzer | 2 | 2 | 4049.21 [3960.51,4137.91] | 2 | 17521558.50 [15392029.25,19651087.75] |
| echidna | 2 | 2 | 2021.31 [1979.67,2062.95] | 2 | 8108372492.50 [4054186246.25,12162558738.75] |
| foundry | 2 | 2 | 11671.27 [11650.25,11692.30] | 2 | 7989702792.82 [7696331835.63,8283073750.01] |
| medusa | 2 | 2 | 1270.00 [1261.50,1278.50] | 2 | 149613096.00 [147741559.50,151484632.50] |

## Progress metrics from logs (fuzzer-specific proxies)
Sequence-rate and corpus values are parsed from native progress output and are useful for within-tool trend context.
| Fuzzer | Runs | Seq/s runs | Seq/s p50 [p25,p75] | Corpus runs | Corpus p50 [p25,p75] |
|---|---|---|---|---|---|
| recon-fuzzer | 2 | 0 | n/a | 2 | 83.00 [80.50,85.50] |
| echidna | 2 | 0 | n/a | 2 | 263.50 [232.75,294.25] |
| foundry | 2 | 0 | n/a | 2 | 0.00 [0.00,0.00] |
| medusa | 2 | 2 | 11.50 [11.25,11.75] | 2 | 228.50 [228.25,228.75] |

## Coverage over time (opt-in, fuzzer-native signals)
- Status: **enabled**
- Source-based coverage is preferred, but the current timestamped progress formats expose only native bytecode/instrumentation proxies. The signals below have different units and must not be ranked, pooled, or plotted on a shared scale.
- Foundry showmap artifacts are a separate, opt-in post-campaign edge replay; they are not real-time source coverage.

| Fuzzer | Native signal | Source-based? | Live availability | Runs with signal | Temporal observations | Observed parser provenance | Final native signal p50 [p25,p75] | Limitation |
|---|---|---|---|---|---|---|---|---|
| recon-fuzzer | `Unique instructions` (instructions) | No (runtime proxy) | End-of-run Recon summary (not a live series) | 2/2 | end-of-run only; 2 sample(s); excluded from time-series chart | text-metrics:unique_instructions | 71810.50 [71581.75,72039.25] | Tool/version-specific final instruction count; no source locations or intermediate observations. |
| echidna | `cov` coverage points (coverage points) | No (runtime proxy) | Echidna status lines | 2/2 | live; 9519 samples; first 0.03-0.03h, last 4.00-4.00h of 4.00h budget | text-metrics:cov | 119496.00 [109098.00,129894.00] | Tool/config-specific counter; no source locations in the progress log. |
| foundry | `cumulative_edges_seen` (instrumented edges) | No (runtime proxy) | Compatible Foundry JSON pulse builds | 2/2 | live; 5726 samples; first 0.00-0.00h, last 3.98-3.98h of 4.00h budget | json-metrics:cumulative_edges_seen | 0.00 [0.00,0.00] | Edge identities depend on the build/instrumentation; showmap replay is separate and post-campaign. |
| medusa | `branches hit` (branches) | No (runtime proxy) | Medusa status lines | 2/2 | live; 9498 samples; first 0.00-0.00h, last 3.98-3.98h of 4.00h budget | text-metrics:branches_hit | 5041.00 [5016.50,5065.50] | Branch identities depend on compiled bytecode/config; no source locations. |

## Function selector sanity checks

Counts are occurrences in unique saved corpus sequences, not runtime execution frequencies The complete per-instance distribution is in `selector_distribution.csv`; `selector_summary.json` preserves statuses and provenance.

- Expected-set status: **available** (23 selector(s))
- Expected-set provenance: peer-consensus heuristic: observed in every available corpus for at least two independent evidence families (Echidna and Recon count as one related typed-corpus family); not benchmark ground truth

| Fuzzer | Status | Instances with telemetry | Saved-corpus calls | Unique selectors |
|---|---|---:|---:|---:|
| echidna | available | 2/2 | 51930 | 44 |
| foundry | unavailable | 0/2 | 0 | 0 |
| medusa | available | 2/2 | 23482 | 63 |
| recon-fuzzer | available | 2/2 | 16436 | 32 |

### Most common saved-corpus selectors

| Fuzzer | Selector | Function | Calls | Share |
|---|---|---|---:|---:|
| echidna | `0xae22f4b8` | `iAaveOracle_setPrice(uint256,uint256)` | 2028 | 3.91% |
| echidna | `0xf24a44c9` | `assert_canary_ASSERTION_CANARY(uint256)` | 1821 | 3.51% |
| echidna | `0x5c9d5cb7` | `iSpoke_supply_ASSERTION_SUPPLY_DOS(uint256,uint256)` | 1751 | 3.37% |
| echidna | `0xe6fb1960` | `iSpoke_updateReserveConfig(uint256,(uint24,bool,bool,bool,bool,bool))` | 1738 | 3.35% |
| echidna | `0x4b43b035` | `invariant_totalBorrowedLessThanSupplied_v0()` | 1698 | 3.27% |
| echidna | `0xd3fabfac` | `iSpoke_withdraw_ASSERTION_WITHDRAW_DOS(uint256,uint256)` | 1690 | 3.25% |
| echidna | `0xf9f2ec54` | `iSpoke_borrow(uint256,uint256)` | 1686 | 3.25% |
| echidna | `0xf7d31edc` | `invariant_hubAddedAssetsGreaterThanSpokeSum()` | 1674 | 3.22% |
| echidna | `0x7aca1cb2` | `invariant_supplySharePriceAndDrawnIndexMonotonic()` | 1670 | 3.22% |
| echidna | `0xb722c6ed` | `iHub_updateAssetConfig(uint256,(address,uint16,address,address),(uint16,uint32,uint32,uint32))` | 1660 | 3.20% |
| medusa | `0x5c9d5cb7` | `iSpoke_supply_ASSERTION_SUPPLY_DOS(uint256,uint256)` | 1199 | 5.11% |
| medusa | `0xa89ef612` | `iSpoke_updateUserDynamicConfig()` | 1189 | 5.06% |
| medusa | `0xb722c6ed` | `iHub_updateAssetConfig(uint256,(address,uint16,address,address),(uint16,uint32,uint32,uint32))` | 1137 | 4.84% |
| medusa | `0xdfbe4d80` | `iSpoke_repay_ASSERTION_REPAY_DOS(uint256,uint256)` | 1125 | 4.79% |
| medusa | `0x5d55852e` | `iSpoke_addDynamicReserveConfig(uint256,(uint16,uint32,uint16))` | 1099 | 4.68% |
| medusa | `0x6801b82e` | `switchActor(uint256)` | 1092 | 4.65% |
| medusa | `0xae22f4b8` | `iAaveOracle_setPrice(uint256,uint256)` | 1084 | 4.62% |
| medusa | `0x20bea9dd` | `iHub_mintFeeShares_ASSERTION_MINT_FEE_SHARES_PPS_CHANGE(uint256)` | 1073 | 4.57% |
| medusa | `0x8aa3bc37` | `switch_oracle(uint256)` | 1029 | 4.38% |
| medusa | `0x256aeede` | `iSpoke_updateLiquidationConfig((uint128,uint64,uint16))` | 1018 | 4.34% |
| recon-fuzzer | `0x27dd847c` | `iSpoke_liquidationCall_ASSERTION_LIQUIDATION_CALL_DOS(uint256,uint256,uint256,uint256,bool)` | 699 | 4.25% |
| recon-fuzzer | `0x4896f9c3` | `iHub_updateSpokeConfig(uint256,uint256,(uint40,uint40,uint24,bool,bool))` | 663 | 4.03% |
| recon-fuzzer | `0x7aca1cb2` | `invariant_supplySharePriceAndDrawnIndexMonotonic()` | 648 | 3.94% |
| recon-fuzzer | `0x256aeede` | `iSpoke_updateLiquidationConfig((uint128,uint64,uint16))` | 630 | 3.83% |
| recon-fuzzer | `0xca2664f9` | `iSpoke_setUsingAsCollateral(uint256,bool)` | 605 | 3.68% |
| recon-fuzzer | `0x5d55852e` | `iSpoke_addDynamicReserveConfig(uint256,(uint16,uint32,uint16))` | 597 | 3.63% |
| recon-fuzzer | `0x66d9a9a0` | `targetArtifactSelectors()` | 577 | 3.51% |
| recon-fuzzer | `0x44a0611b` | `iHub_setInterestRateData(uint256,(uint16,uint32,uint32,uint32))` | 574 | 3.49% |
| recon-fuzzer | `0x77494a8e` | `invariant_hubAddedSharesMatchesSpokeSum()` | 573 | 3.49% |
| recon-fuzzer | `0x6801b82e` | `switchActor(uint256)` | 570 | 3.47% |

### Selector diagnostic findings

- **INFO**: i-05e6717728e1919df-recon-v0.4.6: peer-heuristic gap (diagnostic, not ground truth): absent 1 selector(s): 0xb0464fdc
- **INFO**: i-0e67ab59a18b228f6-recon-v0.4.6: peer-heuristic gap (diagnostic, not ground truth): absent 1 selector(s): 0xb0464fdc

### Selector telemetry limitations

- i-0a9505372fea882f4-foundry-git-02c05d9: Foundry selector distribution is unavailable because no persisted corpus was present; failure-event selectors are intentionally not used as a distribution
- i-0cc6af8d1beaaef2e-foundry-git-02c05d9: Foundry selector distribution is unavailable because no persisted corpus was present; failure-event selectors are intentionally not used as a distribution

## Statistical comparison (Mann-Whitney U and Vargha-Delaney A12)

Pairwise Mann-Whitney U tests and Vargha-Delaney A12 effect sizes on end-of-budget bug counts.
The tests are two-sided, with Bonferroni correction applied for 6 comparison(s). Significance level: alpha = 0.05.

**Warnings:**
- One or more fuzzers have fewer than 5 runs. Statistical power may be limited.

| Fuzzer A | Fuzzer B | Median A | Median B | U statistic | A12 (A over B) | Effect magnitude | p-value | p (corrected) | Significant |
|---|---|---|---|---|---|---|---|---|---|
| recon-fuzzer | echidna | 9 | 7 | 4.0 | 1.000 | large | 0.1939 | 1.0000 | no |
| recon-fuzzer | foundry | 9 | 6.5 | 4.0 | 1.000 | large | 0.2207 | 1.0000 | no |
| recon-fuzzer | medusa | 9 | 6.5 | 4.0 | 1.000 | large | 0.2207 | 1.0000 | no |
| echidna | foundry | 7 | 6.5 | 3.0 | 0.750 | large | 0.6171 | 1.0000 | no |
| echidna | medusa | 7 | 6.5 | 3.0 | 0.750 | large | 0.6171 | 1.0000 | no |
| foundry | medusa | 6.5 | 6.5 | 2.0 | 0.500 | negligible | 1.0000 | 1.0000 | no |

No pairwise comparison reached significance after Bonferroni correction.

> **Note:** A12 is the probability that a randomly selected Fuzzer A run
> has a higher final bug count than a randomly selected Fuzzer B run,
> counting ties as half. Values above 0.5 favor A, values below 0.5 favor B,
> and 0.5 means equal tendency. Magnitude thresholds by distance from 0.5
> are negligible (<0.06), small (<0.14), medium (<0.21), and large
> (>=0.21). Small sample sizes (fewer than 5 runs) reduce statistical power.
> A12 and its magnitude label are descriptive; the thresholds are rules of
> thumb. Neither the effect size nor statistical significance establishes
> practical importance, causation, or performance beyond the observed runs.

## Shape-based interpretation (rules of thumb)
- **Fast-start / early-plateau**: high early checkpoint median + early plateau time + low late discovery share.
- **Steady**: moderate AUC, later plateau, consistent improvements across checkpoints, moderate variance.
- **Slow-burn / late-surge**: low early checkpoints but high late discovery share and later plateau time; often higher final median.

## Limitations
- Core count-based charts use normalized event identities stored under the legacy `bugs_found` column. They do not count crash inputs, but they are not confirmed root-cause bug counts.
- Use `known_bug_report.md`, `known_bug_summary.csv`, and `known_bug_findings.csv` for evidence-backed known-bug hit rates; use `broken_invariants.md` / `broken_invariants.csv` for raw invariant identities.
- Severity, exploitability, and root-cause uniqueness cannot be measured directly without richer per-bug metadata.
- Harness design still affects results; mitigate by keeping harness identical across fuzzers and reporting many runs.
