|
| 1 | +# Developer Documentation |
| 2 | + |
| 3 | +## TESTING |
| 4 | + |
| 5 | +### Running Tests |
| 6 | + |
| 7 | +Tests can be found in the ``tests`` directory, with ``FORK_CONFIG`` set in ``tests/tester.js``. There are test runners for [State tests](http://www.ethdocs.org/en/latest/contracts-and-transactions/ethereum-tests/state_tests/index.html) and [Blockchain tests](http://www.ethdocs.org/en/latest/contracts-and-transactions/ethereum-tests/blockchain_tests/index.html). VM tests are disabled since Frontier gas costs are not supported any more. Tests are then executed by the [ethereumjs-testing](https://github.com/ethereumjs/ethereumjs-testing) utility library using the official client-independent [Ethereum tests](https://github.com/ethereum/tests). |
| 8 | + |
| 9 | +For a wider picture about how to use tests to implement EIPs you can have a look at this [reddit post](https://www.reddit.com/r/ethereum/comments/6kc5g3/ethereumjs_team_is_seeking_contributors/) |
| 10 | +or the associated YouTube video introduction to [core development with Ethereumjs-vm](https://www.youtube.com/watch?v=L0BVDl6HZzk&feature=youtu.be). |
| 11 | + |
| 12 | +#### Running different Test Types |
| 13 | + |
| 14 | +Running all the tests: |
| 15 | + |
| 16 | +`npm test` |
| 17 | + |
| 18 | +Running the State tests: |
| 19 | + |
| 20 | +`node ./tests/tester -s` |
| 21 | + |
| 22 | +Running the Blockchain tests: |
| 23 | + |
| 24 | +`node ./tests/tester -b` |
| 25 | + |
| 26 | +State tests and Blockchain tests can also be run against the ``dist`` folder (default: ``lib``): |
| 27 | + |
| 28 | +`node ./tests/tester -b --dist` |
| 29 | + |
| 30 | +State tests run significantly faster than Blockchain tests, so it is often a good choice to start fixing State tests. |
| 31 | + |
| 32 | +#### Running Specific Tests |
| 33 | + |
| 34 | +Running all the blockchain tests in a file: |
| 35 | + |
| 36 | +`node ./tests/tester -b --file='randomStatetest303'` |
| 37 | + |
| 38 | +Running tests from a specific directory: |
| 39 | + |
| 40 | +`node ./tests/tester -b --dir='bcBlockGasLimitTest'` |
| 41 | + |
| 42 | +Running a specific state test case: |
| 43 | + |
| 44 | +`node ./tests/tester -s --test='stackOverflow'` |
| 45 | + |
| 46 | +Only run test cases with selected ``data``, ``gas`` and/or ``value`` values (see |
| 47 | +[attribute description](http://ethereum-tests.readthedocs.io/en/latest/test_types/state_tests.html) in |
| 48 | +test docs), provided by the index of the array element in the test ``transaction`` section: |
| 49 | + |
| 50 | +`node tests/tester -s --test='CreateCollisionToEmpty' --data=0 --gas=1 --value=0` |
| 51 | + |
| 52 | +Run a state test from a specified source file not under the ``tests`` directory: |
| 53 | +`node ./tests/tester -s --customStateTest='{path_to_file}'` |
| 54 | + |
| 55 | +#### Running tests with a reporter/formatter |
| 56 | + |
| 57 | +`npm run formatTest -t [npm script name OR node command]` will pipe to `tap-spec` by default. |
| 58 | + |
| 59 | +To pipe the results of the API tests through `tap-spec`: |
| 60 | + |
| 61 | +`npm run formatTest -- -t testAPI` |
| 62 | + |
| 63 | +To pipe the results of tests run with a node command through `tap-spec`: |
| 64 | + |
| 65 | +`npm run formatTest -- -t "./tests/tester -b --dir='bcBlockGasLimitTest'"` |
| 66 | + |
| 67 | +The `-with` flag allows the specification of a formatter of your choosing: |
| 68 | + |
| 69 | +`npm install -g tap-mocha-reporter` |
| 70 | +`npm run formatTest -- -t testAPI -with 'tap-mocha-reporter json'` |
| 71 | + |
| 72 | +#### Skipping Tests |
| 73 | + |
| 74 | +There are three types of skip lists (``BROKEN``, ``PERMANENT`` and ``SLOW``) which |
| 75 | +can be found in ``tests/tester.js``. By default tests from all skip lists are omitted. |
| 76 | + |
| 77 | +You can change this behaviour with: |
| 78 | + |
| 79 | +`node tests/tester -s --skip=BROKEN,PERMANENT` |
| 80 | + |
| 81 | +to skip only the ``BROKEN`` and ``PERMANENT`` tests and include the ``SLOW`` tests. |
| 82 | +There are also the keywords ``NONE`` or ``ALL`` for convenience. |
| 83 | + |
| 84 | +It is also possible to only run the tests from the skip lists: |
| 85 | + |
| 86 | +`node tests/tester -s --runSkipped=SLOW` |
| 87 | + |
| 88 | +### CI Test Integration |
| 89 | + |
| 90 | +Tests are run on ``CircleCI`` on every PR, configuration can be found in ``.circleci/config.yml``. |
| 91 | + |
| 92 | +### Debugging |
| 93 | + |
| 94 | +#### Local Debugging |
| 95 | + |
| 96 | +For state tests you can use the ``--jsontrace`` flag to output opcode trace information. |
| 97 | + |
| 98 | +Blockchain tests support `--debug` to verify the postState: |
| 99 | + |
| 100 | +`node ./tests/tester -b --debug --test='ZeroValue_SELFDESTRUCT_ToOneStorageKey_OOGRevert_d0g0v0_EIP158'` |
| 101 | + |
| 102 | +All/most State tests are replicated as Blockchain tests in a ``GeneralStateTests`` [sub directory](https://github.com/ethereum/tests/tree/develop/BlockchainTests/GeneralStateTests) in the Ethereum tests repo, so for debugging single test cases the Blockchain test version of the State test can be used. |
| 103 | + |
| 104 | +#### Comparing Stack Traces |
| 105 | + |
| 106 | +Other client implementations often also provide functionality for output trace information. |
| 107 | + |
| 108 | +A convenient way is to use a local ``geth`` installation (can be the binary installation and doesn't has to be build from source or something) and then use the included ``evm`` tool like: |
| 109 | + |
| 110 | +```shell |
| 111 | +evm --json --nomemory statetest node_modules/ethereumjs-testing/tests/GeneralStateTests/stCreate2/create2collisionCode2.json |
| 112 | +``` |
| 113 | + |
| 114 | +If you want to have only the output for a specific fork you can go into the referenced json test file and temporarily delete the ``post`` section for the non-desired fork outputs (or, more safe and also more convenient on triggering later: copy the test files you are interested in to your working directory and then modify without further worrying). |
| 115 | + |
| 116 | +#### Debugging Tools |
| 117 | + |
| 118 | +For comparing ``EVM`` traces [here](https://gist.github.com/cdetrio/41172f374ae32047a6c9e97fa9d09ad0) are some instructions for setting up ``pyethereum`` to generate corresponding traces for state tests. |
| 119 | + |
| 120 | +Compare TAP output from blockchain/state tests and produces concise diff of the differences between them (example): |
| 121 | + |
| 122 | +``` |
| 123 | +curl https://gist.githubusercontent.com/jwasinger/6cef66711b5e0787667ceb3db6bea0dc/raw/0740f03b4ce90d0955d5aba1e0c30ce698c7145a/gistfile1.txt > output-wip-byzantium.txt |
| 124 | +curl https://gist.githubusercontent.com/jwasinger/e7004e82426ff0a7137a88d273f11819/raw/66fbd58722747ebe4f7006cee59bbe22461df8eb/gistfile1.txt > output-master.txt |
| 125 | +python utils/diffTestOutput.py output-wip-byzantium.txt output-master.txt |
| 126 | +``` |
| 127 | + |
| 128 | +An extremely rich and powerful toolbox is the [evmlab](https://github.com/holiman/evmlab) from ``holiman``, both for debugging and creating new test cases or example data. |
0 commit comments