Amsterdam Comparison
Human and LLM Complexity Evaluations
A descriptive comparison of published human assessments with the study’s LLM assessments (Claude Opus 5.5, checklist revision 3) for 12 Amsterdam Execution Layer EIPs.
Why Amsterdam? It is currently the only fork with a completed human complexity evaluation; the separate manual evaluation for Hegotá is still underway (15 of 35 Hegotá candidates have a scored human checklist, and each Hegotá EIP page shows its Human status and a comparison where a scored checklist exists). This page compares the Amsterdam LLM-based and human assessments to examine how closely they align and whether their disagreements reflect systematic differences in how the complexity criteria were interpreted. It is a structured sanity check, not evidence that the LLM is calibrated to human judgment.
Different checklist revisions, same substance. The human reviewers used checklist revision 1 (24 criteria); the LLM used revision 3 (28 criteria), which phrases the same criteria more precisely. Per-criterion differences cover the 23 criteria both revisions share. Totals are compared as published, so the LLM total also includes five criteria revision 1 does not have: 4.4 points per EIP on average here. Both evaluated the same historical EIP revision.
EIPs compared
12
23 shared criteria per EIPMean Δ total (LLM − Human)
+6.58
mean |Δ| 7.58 · median |Δ| 6.5Who scored higher
11 LLM · 1 Human
0 equal totalsSame tier
7 of 12
each evaluator’s own tier thresholds (revision 1: <10 · 10–19 · ≥20; revision 3: <12 · 12–22 · ≥23)Complexity profiles per EIP
Human (revision 1) and LLM (revision 3) profiles for each EIP on one shared scale, ranked by the Human total. Δ is LLM minus Human. Open an EIP for its per-criterion comparison with the rationale from both evaluators.
Where the Evaluators Differ by Criterion
Mean per-criterion difference across the 12 EIPs, LLM minus Human, for the 23 criteria both revisions share. Positive values mean the LLM scored the criterion higher on average.
| Cross-EIP interactions | +1.42 | 1.58 | 11 | 1 | 0 |
|---|---|---|---|---|---|
| EVM Gas rule changes | −1.08 | 1.08 | 0 | 5 | 7 |
| Encoding changes (RLP/SSZ) | +0.75 | 0.75 | 3 | 0 | 9 |
| Security risks | +0.67 | 1.17 | 9 | 2 | 1 |
| New or modified transaction validity mechanisms | −0.42 | 0.75 | 2 | 4 | 6 |
| Block syncing changes | +0.42 | 0.42 | 3 | 0 | 9 |
| Performance risks | +0.33 | 0.67 | 3 | 2 | 7 |
| Edge/boundary conditions | +0.33 | 0.50 | 4 | 1 | 7 |
| Modified system contracts | −0.25 | 0.25 | 0 | 2 | 10 |
| New EVM gas refund | −0.25 | 0.25 | 0 | 2 | 10 |
| New block / header fields | +0.25 | 0.25 | 1 | 0 | 11 |
| Modified opcodes | +0.17 | 0.67 | 2 | 1 | 9 |
| New fork activation mechanism | +0.17 | 0.17 | 1 | 0 | 11 |
| Patterns affecting pre-existing tests | −0.17 | 1.00 | 5 | 2 | 5 |
| Added system contracts | −0.08 | 0.08 | 0 | 1 | 11 |
| Blob gas accounting changes | −0.08 | 0.08 | 0 | 1 | 11 |
| Added opcodes | 0 | 0.00 | 0 | 0 | 12 |
| Added precompiles | 0 | 0.00 | 0 | 0 | 12 |
| Modified precompiles | 0 | 0.00 | 0 | 0 | 12 |
| New transaction types | 0 | 0.00 | 0 | 0 | 12 |
| Engine API changes | 0 | 0.17 | 1 | 1 | 10 |
| Transition-tool interface changes | 0 | 0.50 | 2 | 2 | 8 |
| Cryptography | 0 | 0.00 | 0 | 0 | 12 |
Totals per EIP
| EIP-7928 | Block-Level Access Lists | 29 | 41 | +12 | High |
| EIP-8037 | State Creation Gas Cost Increase | 28 | 43 | +15 | High |
| EIP-8038 | State-access gas cost update | 20 | 14 | −6 | High vs Medium |
| EIP-2780 | Resource-based intrinsic transaction gas | 13 | 25 | +12 | Medium vs High |
| EIP-7778 | Block Gas Accounting without Refunds | 10 | 13 | +3 | Medium |
| EIP-7708 | ETH transfers emit a log | 9 | 17 | +8 | Low vs Medium |
| EIP-7610 | Revert creation in case of non-empty storage | 7 | 10 | +3 | Low |
| EIP-7843 | SLOTNUM opcode | 7 | 19 | +12 | Low vs Medium |
| EIP-7981 | Increase Access List Cost | 6 | 11 | +5 | Low |
| EIP-8024 | Backward compatible SWAPN, DUPN, EXCHANGE | 6 | 11 | +5 | Low |
| EIP-7976 | Increase Calldata Floor Cost | 5 | 8 | +3 | Low |
| EIP-7997 | Deterministic Factory Contract | 5 | 12 | +7 | Low vs Medium |
What Differed Systematically?
- The LLM scored higher overall. Its total exceeded the human total for 11 of 12 EIPs, by +6.58 points on average. About 4.4 points per EIP come from the five criteria that revision 1 does not have.
- Cross-EIP interactions and security drove the shared-criterion excess. The LLM scored Cross-EIP interactions higher for 11 of 12 EIPs and Security risks higher for 9.
- Humans scored EVM gas rules higher. The human checklists scored more gas-accounting work on five EIPs; the LLM never exceeded them on that criterion.
- Tiers agreed for 7 of 12 EIPs, each against its own revision’s thresholds; most disagreements are human Low against LLM Medium.
These are descriptive differences across 12 EIPs. The comparison crosses checklist revisions and evaluation times, so it neither validates nor calibrates either evaluator.
Criterion legend for the revision-3 checklist
Every stacked bar, comparison matrix, and criterion table on this site uses the same criterion colours, abbreviations, and order. Colour marks the criterion group; the abbreviation and name identify the criterion. Scores are 0–3 per criterion (4 is exceptional; cross-EIP interactions is uncapped).
EVM surface
Opcodes, precompiles, and system contracts that are added or modified.
- Added opcodesIntroduces new opcodes
- Modified opcodesModifies pre-existing opcodes
- Added precompilesIntroduces new precompiles
- Modified precompilesModifies pre-existing precompiles logic or gas-accounting
- Added system contractsIntroduces new system contract, stateful or not
- Modified system contractsModifies pre-existing system contracts
Gas and accounting
Execution, blob, and state gas rules, refunds, and where charges happen inside opcodes.
- EVM Gas rule changesNew EVM gas accounting rules
- State-access ordering within opcode execution · not in checklist revision 1Changes *where inside an opcode's execution* state is accessed, or where gas is charged relative to that access. Because a state access is recorded in the block-level access list only if execution had enough gas to reach it, this ordering is consensus-critical: moving it changes the BAL at every gas boundary of every affected opcode.
- Blob gas accounting changesNew Blob gas accounting rules which potentially affect pre-existing tests
- State gas accounting changes · not in checklist revision 1New state gas accounting rules. State gas is the cost of *writing* state, as opposed to accessing or executing it: `StateGasCosts`, `COST_PER_STATE_BYTE`, the block-level state gas budget, and the spill path into execution gas.
- New EVM gas refundNew gas-refund mechanism
Blocks, transactions, and encoding
Transaction types and validity, block and header fields, encodings, syncing, and activation-time changes.
- New transaction typesIntroduces a new transaction type
- New or modified transaction validity mechanismsCreates new or modifies pre-existing transaction types' validation mechanisms
- New block / header fieldsIntroduces new block or block header fields
- Encoding changes (RLP/SSZ)Introduces encoding changes at the transaction/block/interfaces level
- Block syncing changesModifies block RLP validation mechanisms that require test client syncing.
- New fork activation mechanismModifies state, internal variables, or similar, at the fork activation block
Client interfaces
Engine API and transition-tool interface changes.
- Engine API changesIntroduces new fields to the Engine API directives
- Transition-tool interface changesModifies or adds new fields to the transition tool interface.
Testing impact
Rework, new invariants, and new primitives required in the test framework.
- Patterns affecting pre-existing testsImplements a new validation mechanism or rule that translates in reworking pre-existing tests
- New invariant on pre-existing tests · not in checklist revision 1Tests that are **not about this EIP** must nonetheless assert something this EIP produces. Their logic does not change; they gain a new thing to check.
- New test-framework primitives · not in checklist revision 1Requires new abstractions in the test framework itself — expectation types, modifiers, helpers — beyond writing test functions with what already exists.
Risk and validation
Security, performance, boundary conditions, and cryptography that need validation.
- Security risksIntroduces or modifies mechanisms that could compromise the security of the chain, users, validators, or other stakeholders, if not implemented properly.
- Performance risksIntroduces or modifies mechanisms and requires performance validation.
- Edge/boundary conditionsFeature contains edge/boundary conditions.
- CryptographyIntroduces new cryptography mechanisms or modifies existing functionality that involves cryptography
Coordination
Cross-EIP interactions and behavior that clients must agree on before tests exist.
- Cross-EIP interactionsIntroduces or modifies mechanisms that affect other EIPs in either the same or past forks.
- Unspecified behavior requiring cross-client consensus · not in checklist revision 1The EIP text does not determine the answer for cases a test can construct. Clients must agree on a previously unspecified detail before tests can be baselined. The cost here is coordination and re-baselining, not test writing.