Skip to content

Commit 10cefc8

Browse files
committed
Move testing instructions from README to dedicated developer docs
1 parent f906e71 commit 10cefc8

File tree

2 files changed

+130
-114
lines changed

2 files changed

+130
-114
lines changed

README.md

Lines changed: 2 additions & 114 deletions
Original file line numberDiff line numberDiff line change
@@ -88,121 +88,9 @@ The VM processes state changes at many levels.
8888
The opFns for `CREATE`, `CALL`, and `CALLCODE` call back up to `runCall`.
8989

9090

91-
# TESTING
92-
93-
### Running Tests
94-
95-
Tests can be found in the ``tests`` directory, with ``FORK_CONFIG`` set in ``tests/tester.js``. There are test runners for [State tests](http://www.ethdocs.org/en/latest/contracts-and-transactions/ethereum-tests/state_tests/index.html) and [Blockchain tests](http://www.ethdocs.org/en/latest/contracts-and-transactions/ethereum-tests/blockchain_tests/index.html). VM tests are disabled since Frontier gas costs are not supported any more. Tests are then executed by the [ethereumjs-testing](https://github.com/ethereumjs/ethereumjs-testing) utility library using the official client-independent [Ethereum tests](https://github.com/ethereum/tests).
96-
97-
For a wider picture about how to use tests to implement EIPs you can have a look at this [reddit post](https://www.reddit.com/r/ethereum/comments/6kc5g3/ethereumjs_team_is_seeking_contributors/)
98-
or the associated YouTube video introduction to [core development with Ethereumjs-vm](https://www.youtube.com/watch?v=L0BVDl6HZzk&feature=youtu.be).
99-
100-
#### Running different Test Types
101-
102-
Running all the tests:
103-
104-
`npm test`
105-
106-
Running the State tests:
107-
108-
`node ./tests/tester -s`
109-
110-
Running the Blockchain tests:
111-
112-
`node ./tests/tester -b`
113-
114-
State tests and Blockchain tests can also be run against the ``dist`` folder (default: ``lib``):
115-
116-
`node ./tests/tester -b --dist`
117-
118-
State tests run significantly faster than Blockchain tests, so it is often a good choice to start fixing State tests.
119-
120-
#### Running Specific Tests
121-
122-
Running all the blockchain tests in a file:
123-
124-
`node ./tests/tester -b --file='randomStatetest303'`
125-
126-
Running tests from a specific directory:
127-
128-
`node ./tests/tester -b --dir='bcBlockGasLimitTest'`
129-
130-
Running a specific state test case:
131-
132-
`node ./tests/tester -s --test='stackOverflow'`
133-
134-
Only run test cases with selected ``data``, ``gas`` and/or ``value`` values (see
135-
[attribute description](http://ethereum-tests.readthedocs.io/en/latest/test_types/state_tests.html) in
136-
test docs), provided by the index of the array element in the test ``transaction`` section:
137-
138-
`node tests/tester -s --test='CreateCollisionToEmpty' --data=0 --gas=1 --value=0`
139-
140-
Run a state test from a specified source file not under the ``tests`` directory:
141-
`node ./tests/tester -s --customStateTest='{path_to_file}'`
142-
143-
#### Running tests with a reporter/formatter
144-
145-
`npm run formatTest -t [npm script name OR node command]` will pipe to `tap-spec` by default.
146-
147-
To pipe the results of the API tests through `tap-spec`:
148-
149-
`npm run formatTest -- -t testAPI`
150-
151-
To pipe the results of tests run with a node command through `tap-spec`:
152-
153-
`npm run formatTest -- -t "./tests/tester -b --dir='bcBlockGasLimitTest'"`
154-
155-
The `-with` flag allows the specification of a formatter of your choosing:
156-
157-
`npm install -g tap-mocha-reporter`
158-
`npm run formatTest -- -t testAPI -with 'tap-mocha-reporter json'`
159-
160-
#### Skipping Tests
161-
162-
There are three types of skip lists (``BROKEN``, ``PERMANENT`` and ``SLOW``) which
163-
can be found in ``tests/tester.js``. By default tests from all skip lists are omitted.
164-
165-
You can change this behaviour with:
166-
167-
`node tests/tester -s --skip=BROKEN,PERMANENT`
168-
169-
to skip only the ``BROKEN`` and ``PERMANENT`` tests and include the ``SLOW`` tests.
170-
There are also the keywords ``NONE`` or ``ALL`` for convenience.
171-
172-
It is also possible to only run the tests from the skip lists:
173-
174-
`node tests/tester -s --runSkipped=SLOW`
175-
176-
### CI Test Integration
177-
178-
Tests are run on ``CircleCI`` on every PR, configuration can be found in ``.circleci/config.yml``.
179-
180-
### Debugging
181-
182-
#### Local Debugging
183-
184-
For state tests you can use the ``--jsontrace`` flag to output opcode trace information.
185-
186-
Blockchain tests support `--debug` to verify the postState:
187-
188-
`node ./tests/tester -b --debug --test='ZeroValue_SELFDESTRUCT_ToOneStorageKey_OOGRevert_d0g0v0_EIP158'`
189-
190-
All/most State tests are replicated as Blockchain tests in a ``GeneralStateTests`` [sub directory](https://github.com/ethereum/tests/tree/develop/BlockchainTests/GeneralStateTests) in the Ethereum tests repo, so for debugging single test cases the Blockchain test version of the State test can be used.
191-
192-
#### Debugging Tools
193-
194-
For comparing ``EVM`` traces [here](https://gist.github.com/cdetrio/41172f374ae32047a6c9e97fa9d09ad0) are some instructions for setting up ``pyethereum`` to generate corresponding traces for state tests.
195-
196-
Compare TAP output from blockchain/state tests and produces concise diff of the differences between them (example):
197-
198-
```
199-
curl https://gist.githubusercontent.com/jwasinger/6cef66711b5e0787667ceb3db6bea0dc/raw/0740f03b4ce90d0955d5aba1e0c30ce698c7145a/gistfile1.txt > output-wip-byzantium.txt
200-
curl https://gist.githubusercontent.com/jwasinger/e7004e82426ff0a7137a88d273f11819/raw/66fbd58722747ebe4f7006cee59bbe22461df8eb/gistfile1.txt > output-master.txt
201-
python utils/diffTestOutput.py output-wip-byzantium.txt output-master.txt
202-
```
203-
204-
An extremely rich and powerful toolbox is the [evmlab](https://github.com/holiman/evmlab) from ``holiman``, both for debugging and creating new test cases or example data.
91+
# DEVELOPMENT
20592

93+
Developer documentation - currently mainly with information on testing and debugging - can be found [here](./docs/developer.md).
20694

20795
# LICENSE
20896
[MPL-2.0](https://www.mozilla.org/MPL/2.0/)

docs/developer.md

Lines changed: 128 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,128 @@
1+
# Developer Documentation
2+
3+
## TESTING
4+
5+
### Running Tests
6+
7+
Tests can be found in the ``tests`` directory, with ``FORK_CONFIG`` set in ``tests/tester.js``. There are test runners for [State tests](http://www.ethdocs.org/en/latest/contracts-and-transactions/ethereum-tests/state_tests/index.html) and [Blockchain tests](http://www.ethdocs.org/en/latest/contracts-and-transactions/ethereum-tests/blockchain_tests/index.html). VM tests are disabled since Frontier gas costs are not supported any more. Tests are then executed by the [ethereumjs-testing](https://github.com/ethereumjs/ethereumjs-testing) utility library using the official client-independent [Ethereum tests](https://github.com/ethereum/tests).
8+
9+
For a wider picture about how to use tests to implement EIPs you can have a look at this [reddit post](https://www.reddit.com/r/ethereum/comments/6kc5g3/ethereumjs_team_is_seeking_contributors/)
10+
or the associated YouTube video introduction to [core development with Ethereumjs-vm](https://www.youtube.com/watch?v=L0BVDl6HZzk&feature=youtu.be).
11+
12+
#### Running different Test Types
13+
14+
Running all the tests:
15+
16+
`npm test`
17+
18+
Running the State tests:
19+
20+
`node ./tests/tester -s`
21+
22+
Running the Blockchain tests:
23+
24+
`node ./tests/tester -b`
25+
26+
State tests and Blockchain tests can also be run against the ``dist`` folder (default: ``lib``):
27+
28+
`node ./tests/tester -b --dist`
29+
30+
State tests run significantly faster than Blockchain tests, so it is often a good choice to start fixing State tests.
31+
32+
#### Running Specific Tests
33+
34+
Running all the blockchain tests in a file:
35+
36+
`node ./tests/tester -b --file='randomStatetest303'`
37+
38+
Running tests from a specific directory:
39+
40+
`node ./tests/tester -b --dir='bcBlockGasLimitTest'`
41+
42+
Running a specific state test case:
43+
44+
`node ./tests/tester -s --test='stackOverflow'`
45+
46+
Only run test cases with selected ``data``, ``gas`` and/or ``value`` values (see
47+
[attribute description](http://ethereum-tests.readthedocs.io/en/latest/test_types/state_tests.html) in
48+
test docs), provided by the index of the array element in the test ``transaction`` section:
49+
50+
`node tests/tester -s --test='CreateCollisionToEmpty' --data=0 --gas=1 --value=0`
51+
52+
Run a state test from a specified source file not under the ``tests`` directory:
53+
`node ./tests/tester -s --customStateTest='{path_to_file}'`
54+
55+
#### Running tests with a reporter/formatter
56+
57+
`npm run formatTest -t [npm script name OR node command]` will pipe to `tap-spec` by default.
58+
59+
To pipe the results of the API tests through `tap-spec`:
60+
61+
`npm run formatTest -- -t testAPI`
62+
63+
To pipe the results of tests run with a node command through `tap-spec`:
64+
65+
`npm run formatTest -- -t "./tests/tester -b --dir='bcBlockGasLimitTest'"`
66+
67+
The `-with` flag allows the specification of a formatter of your choosing:
68+
69+
`npm install -g tap-mocha-reporter`
70+
`npm run formatTest -- -t testAPI -with 'tap-mocha-reporter json'`
71+
72+
#### Skipping Tests
73+
74+
There are three types of skip lists (``BROKEN``, ``PERMANENT`` and ``SLOW``) which
75+
can be found in ``tests/tester.js``. By default tests from all skip lists are omitted.
76+
77+
You can change this behaviour with:
78+
79+
`node tests/tester -s --skip=BROKEN,PERMANENT`
80+
81+
to skip only the ``BROKEN`` and ``PERMANENT`` tests and include the ``SLOW`` tests.
82+
There are also the keywords ``NONE`` or ``ALL`` for convenience.
83+
84+
It is also possible to only run the tests from the skip lists:
85+
86+
`node tests/tester -s --runSkipped=SLOW`
87+
88+
### CI Test Integration
89+
90+
Tests are run on ``CircleCI`` on every PR, configuration can be found in ``.circleci/config.yml``.
91+
92+
### Debugging
93+
94+
#### Local Debugging
95+
96+
For state tests you can use the ``--jsontrace`` flag to output opcode trace information.
97+
98+
Blockchain tests support `--debug` to verify the postState:
99+
100+
`node ./tests/tester -b --debug --test='ZeroValue_SELFDESTRUCT_ToOneStorageKey_OOGRevert_d0g0v0_EIP158'`
101+
102+
All/most State tests are replicated as Blockchain tests in a ``GeneralStateTests`` [sub directory](https://github.com/ethereum/tests/tree/develop/BlockchainTests/GeneralStateTests) in the Ethereum tests repo, so for debugging single test cases the Blockchain test version of the State test can be used.
103+
104+
#### Comparing Stack Traces
105+
106+
Other client implementations often also provide functionality for output trace information.
107+
108+
A convenient way is to use a local ``geth`` installation (can be the binary installation and doesn't has to be build from source or something) and then use the included ``evm`` tool like:
109+
110+
```shell
111+
evm --json --nomemory statetest node_modules/ethereumjs-testing/tests/GeneralStateTests/stCreate2/create2collisionCode2.json
112+
```
113+
114+
If you want to have only the output for a specific fork you can go into the referenced json test file and temporarily delete the ``post`` section for the non-desired fork outputs (or, more safe and also more convenient on triggering later: copy the test files you are interested in to your working directory and then modify without further worrying).
115+
116+
#### Debugging Tools
117+
118+
For comparing ``EVM`` traces [here](https://gist.github.com/cdetrio/41172f374ae32047a6c9e97fa9d09ad0) are some instructions for setting up ``pyethereum`` to generate corresponding traces for state tests.
119+
120+
Compare TAP output from blockchain/state tests and produces concise diff of the differences between them (example):
121+
122+
```
123+
curl https://gist.githubusercontent.com/jwasinger/6cef66711b5e0787667ceb3db6bea0dc/raw/0740f03b4ce90d0955d5aba1e0c30ce698c7145a/gistfile1.txt > output-wip-byzantium.txt
124+
curl https://gist.githubusercontent.com/jwasinger/e7004e82426ff0a7137a88d273f11819/raw/66fbd58722747ebe4f7006cee59bbe22461df8eb/gistfile1.txt > output-master.txt
125+
python utils/diffTestOutput.py output-wip-byzantium.txt output-master.txt
126+
```
127+
128+
An extremely rich and powerful toolbox is the [evmlab](https://github.com/holiman/evmlab) from ``holiman``, both for debugging and creating new test cases or example data.

0 commit comments

Comments
 (0)