Retrospective LLM-Based Complexity Evaluations

EIP complexity assessment

EIP-3855: PUSH0 instruction

Assessed in Shanghai / Shapella. The score describes the EIP text available at the assessment cutoff, not the EIP as it stands today.

RetrospectiveShanghai / ShapellaAssessment cutoff 2022-02-04Included by cutoffLayers: execution
LLM Completescore 5
Human Not available· Human complexity assessments were not produced for this fork; only the LLM assessment exists.

LLM assessment

Evaluated on: · Spec revision: 2021-09-22 · 6caab461df

Scope at the cutoff. This revision of EIP-3855 adds the instruction PUSH0 at opcode 0x5f, starting at a fork block HF. PUSH0 has no immediate data and pops nothing. It pushes the constant 0 onto the stack and costs a fixed 2 gas, which the text labels "verylow". The listed test cases cover a single PUSH0, 1024 consecutive PUSH0s (succeeds) and 1025 (stack overflow). The Security Considerations section says jumpdest analysis is unaffected because there is no immediate data. Already-deployed code containing 0x5f may behave differently after the fork.

5LowLow
Evaluator
LLMChecklist v3
Confidence
High
Under-specified at assessment cutoff
Yes — 1 criterion affected
Plausible range
5–6 (Low)
Assessment cutoff
2022-02-04 · EIP revision 6caab461df (2021-09-22)
Score bands · Checklist revision 3
  • Low <12
  • Medium 12–22
  • High ≥23

28 criteria scored 0–3 (4 in exceptional cases; cross-EIP interactions is uncapped); nominal maximum 84.

Complexity profile

Each segment is one criterion's contribution to the LLM total. Hover or focus a segment for its score and rationale.

Top complexity drivers

  1. Added opcodes1
  2. Patterns affecting pre-existing tests1
  3. Security risks1
  4. Edge/boundary conditions1

Under-specified at assessment cutoff: Yes

The EIP text available at the assessment cutoff left material behavior unresolved. The affected criteria and the plausible total range record that uncertainty.

Why: The gas cost says "2 gas (aka verylow)". In conventional tier naming, verylow is 3 gas and 2 gas is the base tier. The explicit number and the comparison to ADDRESS and ORIGIN support 2 gas.

Plausible total

5–6
recorded score 5 · plausible tiers Low

Unresolved questions at the cutoff (2)
  • Is the intended cost 2 gas (the explicit number) or the verylow tier value (3 gas)?
  • The fork is identified only as an HF block number.
Notable ambiguities noted by the assessor (2)
  • The "verylow" label conflicts with the stated 2-gas cost. The cited comparison instructions (ADDRESS, ORIGIN) cost 2, which suggests the label is wrong rather than the number.
  • Activation is given as BLOCK_NUMBER >= HF, with no concrete fork block.

Criterion breakdown

EIP-3855 Shanghai / Shapella: LLM criterion scores and rationale
CriterionScoreWhy this scoreEvidence / uncertainty
Added opcodes1Exactly one simple instruction is introduced.
  • eip.md · Specification — "It has no immediate data, pops no items from the stack, and places a single item" One new instruction with no immediate data, fixed stack effects and constant gas.
Confidence: High
Patterns affecting pre-existing tests1Rework is limited to the one parameter value 0x5f within the undefined/invalid-opcode test family. That is localized rework in one behavioral family.
  • eip.md · Backwards Compatibility — "Already deployed contracts using this opcode could change their behaviour" Before the fork, 0x5f is undefined. Baseline tests that expect it to abort as an invalid opcode must change.
Confidence: Medium
Uncertainty: How much rework is needed depends on whether baseline invalid-opcode tests enumerate 0x5f. No test suite was supplied.
Security risks1The security conditions (jumpdest analysis does not skip bytes after 0x5f, and pre-fork vs post-fork validity) can be checked locally.
  • eip.md · Security Considerations — "jumpdest-analysis is unaffected, as `PUSH0` has no immediate data bytes" Jumpdest analysis must not treat 0x5f as having immediate data.
  • eip.md · Backwards Compatibility Deployed code containing 0x5f changes behavior from aborting to succeeding.
Confidence: Medium
Edge/boundary conditions1There is one boundary-sensitive mechanism: the new push instruction's validity and stack-limit behavior, plus fork activation and exact gas or out-of-gas. There is no elevated matrix.
  • eip.md · Test Cases — "(1025 times) -- execution aborts due to out of stack" The outcome flips at the 1024-item stack limit for the new instruction.
  • eip.md · Specification — "Starting `BLOCK_NUMBER >= HF`" Whether 0x5f is valid depends on the fork boundary.
Confidence: Medium
Uncertainty: The stack limit itself is an existing rule. Someone could argue for 0, but the new instruction's own overflow behavior needs boundary cases.
Unspecified behavior requiring cross-client consensusUnder-specified1The tier label is inconsistent, but the explicit number and the comparison instructions point to one intended outcome: 2 gas. That makes this a localized omission or detail at level 1.
  • eip.md · Specification — "The cost of this instruction is 2 gas (aka `verylow`)" In standard gas-tier naming, 2 gas is the base tier, while verylow is 3 gas. The label conflicts with the number.
  • eip.md · Rationale — Gas cost — "such as `ADDRESS`, `ORIGIN`" The instructions cited as comparisons cost 2 gas, which supports the explicit value of 2.
Confidence: Medium
Uncertainty: If an implementer followed the tier name, gas results would differ (3 vs 2). That reading could justify level 2.
Show 23 zero-score criteria
Zero-score criteria (Checklist revision 3)
CriterionScoreWhy this scoreEvidence / uncertainty
Modified opcodes00x5f was previously undefined. Defining it counts as an added opcode, not a modification of an existing one.
  • eip.md · Motivation — RETURNDATASIZE example Changing other instructions is given only as motivation. No existing instruction's semantics are modified.
Added precompiles0None.
  • eip.md · Specification No precompile is added.
Modified precompiles0None.
  • eip.md · Specification No precompile is changed.
Added system contracts0None introduced.
  • eip.md · Specification No system contract is introduced.
Modified system contracts0None modified.
  • eip.md · Specification No system contract is affected.
EVM Gas rule changes0A new instruction priced at an existing constant tier is not a new accounting mechanism, and it does not change any existing gas rule or baseline expectation.
  • eip.md · Specification — "The cost of this instruction is 2 gas (aka `verylow`)" The new instruction uses a constant cost from an existing gas tier. No accounting rule changes.
Uncertainty: The tier name conflicts with the number 2 (see UNSP). Either way it is a constant from an existing tier, so this score is unaffected.
State-access ordering within opcode execution0No state access occurs, and no ordering rule changes.
  • eip.md · Specification — "pops no items from the stack, and places a single item with the value 0" PUSH0 only touches the stack and does not access state.
Blob gas accounting changes0Blob-gas accounting does not change.
  • eip.md · Specification Only an EVM instruction is specified. Blob gas is not mentioned.
State gas accounting changes0State-gas accounting does not change.
  • eip.md · Specification There is no state-writing or state-gas rule.
New EVM gas refund0No new refund mechanism is introduced.
  • eip.md · Specification No refund is defined.
New transaction types0None.
  • eip.md · Specification No transaction type is introduced.
New or modified transaction validity mechanisms0None.
  • eip.md · Specification No transaction-validity or intrinsic-gas changes.
New block / header fields0None.
  • eip.md · Specification No header fields are added.
Encoding changes (RLP/SSZ)0No schema or codec changes.
  • eip.md · Specification No serialized schema changes.
Block syncing changes0Only an execution rule changes.
  • eip.md · Specification No block-structure or RLP validation rules change.
New fork activation mechanism0No activation-specific state transition is needed.
  • eip.md · Specification — "Starting `BLOCK_NUMBER >= HF`" Activation only selects which rules apply. There is no state migration.
Engine API changes0The Engine API does not change.
  • eip.md · Specification The Engine API is not involved.
Transition-tool interface changes0The tool interface does not change. Fork selection covers the new behavior.
  • eip.md · Specification Only an EVM instruction changes. No t8n inputs or outputs are affected.
New invariant on pre-existing tests0Baseline tests need no new assertion.
  • eip.md · Specification No new log, header, receipt or other output is produced.
New test-framework primitives0Baseline primitives (bytecode, stack-overflow and gas expectations) are enough. Adding a PUSH0 mnemonic to the opcode list is a new value, not a new abstraction.
  • eip.md · Test Cases The tests are plain bytecode with stack, success and stack-overflow expectations.
Uncertainty: One could argue the opcode list needs a local extension (level 1).
Performance risks0No additional performance-validation requirement is established.
  • eip.md · Rationale — Gas cost A trivial constant push priced like other constant-pushing instructions.
Cryptography0No cryptographic mechanism changes.
  • eip.md · Specification No cryptography is involved.
Cross-EIP interactions0PUSH0 behaves the same regardless of any other EIP. The EIP-2733 citation is motivation only.
  • eip.md · Motivation — "This was proposed as a way to chain transactions (i.e. EIP-2733)" EIP-2733 is cited only as motivation for not depending on RETURNDATASIZE being zero. PUSH0 does not interact with its behavior.
Assessment provenance
Assessed EIP revision
ethereum/EIPs@6caab461df EIPS/eip-3855.md committed 2021-09-22 · information cutoff 2022-02-04
Current master · File history · blob e86efc6e13 · sha256 f89cc30713d2
Rubric
Checklist revision 3 · ethspecs/pm@fe2f793b03
Evaluator
Opus 5.5 (claude-opus-5-5) at high effort, one tool-less call per EIP · isolation bubblewrap_claude_p_no_tools_v1
Source record
Frozen research record research/tasks/10-opus-v3-reassessment/retrospective/outputs/assessments/shanghai/eip-3855.yaml · sha256 72fefbd84bbf
Compare EIP-3855 with other assessmentsAll Shanghai / Shapella assessments
Criterion legend and glossary

Every stacked bar, comparison matrix, and criterion table on this site uses the same criterion colours, abbreviations, and order. Colour marks the criterion group; the abbreviation and name identify the criterion. Scores are 0–3 per criterion (4 is exceptional; cross-EIP interactions is uncapped).

EVM surface

Opcodes, precompiles, and system contracts that are added or modified.

  • Added opcodes
    Introduces new opcodes
    Score anchors
    0
    No new opcodes are introduced.
    1
    A new simple opcode is introduced (no data portion, no complex stack mechanics, and a constant gas cost).
    2
    Multiple new simple opcodes are introduced, or a single new complex opcode is introduced (has data portion, or complex stack mechanics, or a dynamic gas cost).
    3
    Multiple new opcodes are introduced, and at least one of them is complex (has data portion, or complex stack mechanics, or a dynamic gas cost).
    • Cryptography opcodes are not considered complex by default. Refer to the "Cryptography" section for a separate assessment.
  • Modified opcodes
    Modifies pre-existing opcodes
    Score anchors
    0
    No pre-existing opcode modifications are introduced.
    3
    At least one pre-existing opcode's behavior is modified (not including gas changes) or a pre-existing opcode is deprecated.
  • Added precompiles
    Introduces new precompiles
    Score anchors
    0
    No new precompiles are introduced.
    1
    A new simple precompile is introduced (constant input length, constant gas cost).
    2
    Multiple new simple precompiles are introduced, or a single new complex precompile is introduced (dynamic input length or dynamic gas cost).
    3
    Multiple new precompiles are introduced, and at least one of them is complex (dynamic input length or dynamic gas cost).
    • Cryptography precompiles are not considered complex by default. Refer to the "Cryptography" for a separate assessment.
  • Modified precompiles
    Modifies pre-existing precompiles logic or gas-accounting
    Score anchors
    0
    No pre-existing precompiles are modified.
    1
    At least one pre-existing precompile has its gas schedule modified.
    2
    Multiple pre-existing precompiles have their gas schedule modified, or a single pre-existing precompile has its behavior modified.
    3
    The behavior of multiple pre-existing precompiles, or a single complex pre-existing precompile modified.
  • Added system contracts
    Introduces new system contract, stateful or not
    Score anchors
    0
    No new system contracts are introduced.
    1
    A new system contract is introduced that is not stateful nor does it trigger a new system action (e.g. requests to the consensus layer).
    2
    Multiple new system contracts are introduced or a single new system contract that is either stateful or triggers a new system action (e.g. requests to the consensus layer).
    3
    Multiple new system contracts are introduced and at least one of them is either stateful or triggers a new system action (e.g. requests to the consensus layer).
  • Modified system contracts
    Modifies pre-existing system contracts
    Score anchors
    0
    No modifications to pre-existing system contracts are introduced, directly or indirectly.
    1
    Does not directly modify any system contract, but its behavior has minor indirect effects on one or more system contracts.
    2
    Does not directly modify any system contract, but its behavior has major indirect effects on one or more system contracts.
    3
    At least one pre-existing system contract code or state is modified, which would involve irregular state transition or a similarly complex transition methodology.

Gas and accounting

Execution, blob, and state gas rules, refunds, and where charges happen inside opcodes.

  • EVM Gas rule changes
    New EVM gas accounting rules
    Score anchors
    0
    No gas accounting changes.
    1
    Existing gas accounting mechanism is updated.
    2
    A new gas accounting mechanism is introduced but it does not affect existing mechanisms nor does it affect existing tests.
    3
    A new gas accounting mechanism is introduced and affects existing mechanisms which in turn affect existing tests.
  • State-access ordering within opcode execution · not in checklist revision 1
    Changes *where inside an opcode's execution* state is accessed, or where gas is charged relative to that access. Because a state access is recorded in the block-level access list only if execution had enough gas to reach it, this ordering is consensus-critical: moving it changes the BAL at every gas boundary of every affected opcode.
    Score anchors
    0
    No change to where state is accessed, or to where gas is charged relative to a state access, within any opcode.
    1
    A single opcode's state-access or gas-charge ordering changes.
    2
    Multiple opcodes' ordering changes, or a new state-accessing operation is introduced whose position in the order must be settled.
    3
    The ordering rule changes for a whole class of state-accessing opcodes at once, or what counts as a recordable state access is redefined — requiring existing BAL vectors to be re-derived across opcodes and forks.
    • Distinct from "Modified opcodes", which asks whether an opcode's **result** changed. This row asks about the **path to the result**, which is observable even when the result is identical. An EIP can be 0 on that row and 3 on this one.
    • Score changes **to** the ordering. Do not score the fact that state accesses are observable — they always are.
    • Each boundary must be re-tested against every other dimension that can change the answer (cold/warm, static/non-static, delegated/direct, revert/success), so the case count grows multiplicatively rather than additively. Note this explicitly under Special Considerations.
  • Blob gas accounting changes
    New Blob gas accounting rules which potentially affect pre-existing tests
    Score anchors
    0
    No blob gas accounting changes.
    1
    Existing blob gas accounting mechanism is updated.
    2
    A new blob gas accounting mechanism is introduced but it does not affect existing mechanisms nor does it affect existing tests.
    3
    A new blob gas accounting mechanism is introduced and affects existing mechanisms which in turn affect existing tests.
  • State gas accounting changes · not in checklist revision 1
    New state gas accounting rules. State gas is the cost of *writing* state, as opposed to accessing or executing it: `StateGasCosts`, `COST_PER_STATE_BYTE`, the block-level state gas budget, and the spill path into execution gas.
    Score anchors
    0
    No state gas accounting changes.
    1
    An existing state gas cost or `STATE_BYTES_PER_*` rate is adjusted.
    2
    A new state-gas-charging site is introduced, or the block-level state gas budget or reservoir allocation is modified.
    3
    A new state gas charging mechanism is introduced, or the spill interaction between state gas and execution gas is modified, affecting existing gas tests.
    • Harder to test than blob gas: the spill path means state gas cannot be metered independently of execution gas, and some costs (e.g. `NEW_ACCOUNT`) are state-dependent.
  • New EVM gas refund
    New gas-refund mechanism
    Score anchors
    0
    No new gas-refund mechanisms are introduced.
    1
    A new simple gas-refund mechanism is introduced that does not affect either existing tests or existing gas-refund mechanisms.
    2
    A new complex gas-refund mechanism is introduced or a simple mechanism that affects existing tests or existing gas-refund mechanisms.
    3
    A new complex gas-refund mechanism is introduced that affects existing tests or existing gas-refund mechanisms.

Blocks, transactions, and encoding

Transaction types and validity, block and header fields, encodings, syncing, and activation-time changes.

  • New transaction types
    Introduces a new transaction type
    Score anchors
    0
    No new transaction types are introduced.
    3
    A new transaction type is introduced.
  • New or modified transaction validity mechanisms
    Creates new or modifies pre-existing transaction types' validation mechanisms
    Score anchors
    0
    No changes are introduced to the validity rules of existing transaction types or to their intrinsic gas cost calculation.
    1
    Minor adjustments are introduced to validity rules or intrinsic gas cost calculation, but they do not significantly affect existing tests.
    2
    Changes to validity rules or intrinsic gas cost calculation affect existing tests, but require only limited updates to test cases and no redesign of the testing infrastructure.
    3
    Changes to validity rules or intrinsic gas cost calculation require extensive rework or redesign of the tests or testing infrastructure.
  • New block / header fields
    Introduces new block or block header fields
    Score anchors
    0
    No new block or header fields are introduced.
    3
    A new block or header field is introduced.
  • Encoding changes (RLP/SSZ)
    Introduces encoding changes at the transaction/block/interfaces level
    Score anchors
    0
    No encoding changes are introduced at the transaction, block, or interfaces levels.
    3
    An encoding change is introduced at transaction, block or interfaces level (e.g. RLP -> SSZ).
    • "Interfaces level" includes the Engine API. Score an Engine API encoding change (e.g. JSON -> SSZ) here.
  • Block syncing changes
    Modifies block RLP validation mechanisms that require test client syncing.
    Score anchors
    0
    No new RLP validation mechanism is introduced.
    1
    A single simple RLP validation mechanism is introduced.
    2
    Multiple simple RLP validation mechanisms are introduced or a single complex one.
    3
    Multiple RLP validation mechanisms are introduced and at least one of them is deemed complex.
  • New fork activation mechanism
    Modifies state, internal variables, or similar, at the fork activation block
    Score anchors
    0
    No state modifications, internal variables or similar are modified at the fork activation block.
    3
    Either a state modification or internal variables are modified at the fork activation block.
    • Initialization of new internal variable is not considered a modification.

Client interfaces

Engine API and transition-tool interface changes.

  • Engine API changes
    Introduces new fields to the Engine API directives
    Score anchors
    0
    No new fields or communication mechanisms are introduced to the Engine API.
    1
    A single new field is introduced in one of the Engine API endpoints.
    2
    Multiple fields are introduced to one or multiple Engine API end points, or a new Engine API end-point is introduced.
    3
    Multiple fields are introduced to one or multiple Engine API end points and a new Engine API end-point is introduced.
  • Engine API encoding changes · Checklist revision 1 only
    Engine API encoding changes (the revision-1 template defines no anchor text for this row).
  • Transition-tool interface changes
    Modifies or adds new fields to the transition tool interface.
    Score anchors
    0
    No modifications to the transition tool interface are required.
    1
    A single new field needs to be introduced to the transition tool interface.
    2
    Multiple new fields or a new mechanism has to be introduced to the transition tool interface.
    3
    Multiple new fields and a new mechanism has to be introduced to the transition tool interface.
    • Special consideration must be paid to this section if the EIP introduces a mechanism that requires the state transition tool to be aware whether the block it is processing is the fork-activation block.

Testing impact

Rework, new invariants, and new primitives required in the test framework.

  • Patterns affecting pre-existing tests
    Implements a new validation mechanism or rule that translates in reworking pre-existing tests
    Score anchors
    0
    No pre-existing tests are affected by this change.
    1
    Minor subset of existing tests are affected by this change.
    2
    Considerable subset of existing tests are affected by this change but involves only a contrived category of tests.
    3
    Major subset of existing tests are affected, including diverse category of tests (benchmarks, static, multiple forks, etc.).
  • New invariant on pre-existing tests · not in checklist revision 1
    Tests that are **not about this EIP** must nonetheless assert something this EIP produces. Their logic does not change; they gain a new thing to check.
    Score anchors
    0
    Pre-existing tests assert nothing new.
    1
    A narrow, contrived category of pre-existing tests gains a new assertion.
    2
    A broad category gains a new assertion, applied mechanically.
    3
    Every test in the fork gains the assertion regardless of what it tests, and pre-fork vectors must be re-derived to satisfy it.
    • Paired with the row above, and easy to confuse with it. "Patterns affecting pre-existing tests" asks whether existing tests must be **reworked**; this row asks whether they must **additionally assert something new**. Score both — an EIP can be low on one and high on the other.
  • New test-framework primitives · not in checklist revision 1
    Requires new abstractions in the test framework itself — expectation types, modifiers, helpers — beyond writing test functions with what already exists.
    Score anchors
    0
    Existing test primitives suffice.
    1
    Existing primitives need minor extension.
    2
    New expectation or modifier primitives are required, reusable within this EIP's own test suite.
    3
    New framework-level primitives are required that become a permanent part of the framework and are used by other EIPs' tests.

Risk and validation

Security, performance, boundary conditions, and cryptography that need validation.

  • Security risks
    Introduces or modifies mechanisms that could compromise the security of the chain, users, validators, or other stakeholders, if not implemented properly.
    Score anchors
    0
    No new mechanisms are introduced that could pose a security risk.
    1
    The introduced mechanisms are self-contained, can be validated in isolation, and do not alter existing invariants that could pose a security risk for any stakeholders.
    2
    The introduced mechanisms interact with a limited number of existing components, slightly altering their security assumptions and requiring a targeted security review or fuzzing.
    3
    The introduced mechanisms interact with multiple existing components, including critical ones, substantially altering their security assumptions and requiring an extensive security review and fuzzing.
  • Performance risks
    Introduces or modifies mechanisms and requires performance validation.
    Score anchors
    0
    No new mechanisms are introduced that require performance validation.
    1
    The introduced mechanisms can be benchmarked in isolation and do not affect existing performance behavior.
    2
    The introduced mechanisms cannot be fully benchmarked in isolation, but they only have a limited impact on the existing performance benchmarks.
    3
    The introduced mechanisms cannot be benchmarked in isolation and have a substantial impact on existing performance benchmarks or have complex interactions with existing mechanisms.
  • Edge/boundary conditions
    Feature contains edge/boundary conditions.
    Score anchors
    0
    No discernible edge cases or boundary conditions are introduced.
    1
    A single edge-case or boundary-condition prone mechanism is introduced.
    2
    Multiple edge-case or boundary-condition prone mechanisms are introduced, but none of them requires an elevated number of cases to test.
    3
    Multiple edge-case or boundary-condition prone mechanisms are introduced and at least one of them requires an elevated number of cases to test.
  • Cryptography
    Introduces new cryptography mechanisms or modifies existing functionality that involves cryptography
    Score anchors
    0
    No cryptography mechanisms are introduced.
    1
    A new cryptography mechanism is introduced but it is a well known mechanism that is known to have vast resources to aid on its testing.
    2
    Multiple new cryptography mechanisms are introduced that are well-known or a single but novel mechanism is introduced that is either untested or has limited resources.
    3
    Multiple new cryptography mechanisms are introduced and at least one of them is a novel mechanism.

Coordination

Cross-EIP interactions and behavior that clients must agree on before tests exist.

  • Cross-EIP interactions
    Introduces or modifies mechanisms that affect other EIPs in either the same or past forks.
    Score anchors
    0
    Fully self-contained EIP that does not depend on, modify, or conflict with any other EIP.
    1
    The EIP interacts with one or more other EIPs in a non-critical and limited way but can be tested independently for the most part.
    2
    The EIP depends on or modifies one or more other EIPs such that coordinated testing and consideration is required, but interactions are limited in scope and not complex.
    3
    The EIP has strong interdependencies with multiple EIPs, requiring extensive coordinated cross-EIP testing as well as potential re-design of existing test vectors.
    • +1 for every 3 additional interacting EIPs beyond the first 3, each of which requires its own coordinated test cases. List the EIPs in the rationale.
    • This row is intentionally uncapped, unlike every other anchor: each interacting EIP is another axis of the test matrix, so a ceiling would make a 12-EIP product indistinguishable from a 3-EIP one.
  • Unspecified behavior requiring cross-client consensus · not in checklist revision 1
    The EIP text does not determine the answer for cases a test can construct. Clients must agree on a previously unspecified detail before tests can be baselined. The cost here is coordination and re-baselining, not test writing.
    Score anchors
    0
    The EIP text determines the answer for every case a test could construct.
    1
    A few details are unspecified but have an obvious intended reading.
    2
    Details require client agreement before tests can be written, but they are localized.
    3
    A previously unspecified *and previously unobservable* behavior becomes consensus-critical; expect tests to be re-baselined on each round of EIP amendment.
    • Score this from the EIP's state at assessment time: whether it has client implementations, whether it has been through a devnet, and how many open questions remain on its discussion thread.