defaultimplfuzzers: atpointer dump dump_raw_tape element minify parser print_json

implement_fuzzer(fuzz_element)

#include <vector>

#include <string_view>

#include <cstring> //memcpy

inline std::vector<std::string_view> split(const char* Data, size_t Size) {

std::vector<std::string_view> ret;

using namespace std::literals;

constexpr auto sep="\n~~\n"sv;

std::string_view all(Data,Size);

auto pos=all.find(sep);

while(pos!=std::string_view::npos) {

ret.push_back(all.substr(0,pos));

all=all.substr(pos+sep.size());

pos=all.find(sep);

}

ret.push_back(all);

return ret;

}

struct FuzzData {

// data may not be null, even if size is zero.

FuzzData(const uint8_t* data,

size_t size) : Data(data),Size(size){}

///range is inclusive

template<int Min, int Max>

int getInt() {

static_assert (Min<Max,"min must be <max");

// make this constexpr, can't overflow because that is UB and is forbidden

// in constexpr evaluation

constexpr int range=(Max-Min)+1;

constexpr unsigned int urange=range;

// don't use std::uniform_int_distribution, we don't want to pay for

// over consumption of random data. Accept the slightly non-uniform distribution.

if(range<256)

return Min+static_cast<int>(get<uint8_t>()%urange);

if(range<65536)

return Min+static_cast<int>(get<uint16_t>()%urange);

return Min+static_cast<int>(get<uint32_t>()%urange);

}

template<typename T>

T get() {

const auto Nbytes=sizeof(T);

T ret{};

if(Size<Nbytes) {

//don't throw, signal with null instead.

Data=nullptr;

Size=0;

return ret;

}

std::memcpy(&ret,Data,Nbytes);

Data+=Nbytes;

Size-=Nbytes;

return ret;

}

// gets a string view with length in [Min,Max]

template<int Min, int Max>

std::string_view get_stringview() {

static_assert (Min>=0,"Min must be positive");

const int len=getInt<Min,Max>();

const unsigned int ulen=static_cast<unsigned int>(len);

if(ulen<Size) {

std::string_view ret(chardata(),ulen);

Data+=len;

Size-=ulen;

return ret;

}

//mark that there is too little data to fulfill the request

Data=nullptr;

Size=0;

return {};

}

// split the remainder of the data into string views,

std::vector<std::string_view> splitIntoStrings() {

std::vector<std::string_view> ret;

if(Size>0) {

ret=split(chardata(),Size);

// all data consumed.

Data+=Size;

Size=0;

}

return ret;

}

//are we good?

explicit operator bool() const { return Data!=nullptr;}

//we are a URBG

// https://en.cppreference.com/w/cpp/named_req/UniformRandomBitGenerator

//The type G satisfies UniformRandomBitGenerator if Given

// T, the type named by G::result_type

// g, a value of type G

//

// The following expressions must be valid and have their specified effects

// Expression Return type Requirements

// G::result_type T T is an unsigned integer type

using result_type=uint8_t;

// G::min() T Returns the smallest value that G's operator() may return. The value is strictly less than G::max(). The function must be constexpr.

static constexpr result_type min() {return 0;}

// G::max() T Returns the largest value that G's operator() may return. The value is strictly greater than G::min(). The function must be constexpr.

static constexpr result_type max() {return 255;}

// g() T Returns a value in the closed interval [G::min(), G::max()]. Has amortized constant complexity.

result_type operator()() {

if(Size==0) {

// return something varying, otherwise uniform_int_distribution may get

// stuck

return failcount++;

}

const result_type ret=Data[0];

Data++;

Size--;

return ret;

}

// returns a pointer to data as const char* to avoid those cstyle casts

const char* chardata() const {return static_cast<const char*>(static_cast<const void*>(Data));}

// members

const uint8_t* Data;

size_t Size;

uint8_t failcount=0;

};

There is both "normal" fuzzers just feeding the api with fuzz data, as well as **differential** fuzzers. The differential fuzzers feed the same data to the multiple implementations (haswell, westmere and fallback) and ensure the same results are achieved. This makes sure the user will always get the same answer regardless of which implementation is in use.

The fuzzers are used in several ways.

* local fuzzing - for developers testing their changes before pushing and/or during development of the fuzzers themselves.

* CI fuzzing - for weeding out those easy to find bugs in pull requests, before they are merged.

* oss-fuzz - heavy duty 24/7 fuzzing provided by the google driven oss-fuzz project

Just invoke fuzz/quick_check.sh, it will download the latest corpus from bintray (kept up to date by the CI fuzzers) and run the fuzzers for a short time. In case you want to run the fuzzers for longer, modify the timeout value in the script or invoke the fuzzer directly.

This requires linux with clang and cmake installed (recent Debian and Ubuntu are known to work fine).

It is also possible to run the full oss-fuzz setup by following [these oss-fuzz instructions](https://google.github.io/oss-fuzz/getting-started/new-project-guide/#testing-locally) with PROJECT_NAME set to simdjson. You will need rights to run docker.

There is a CI job which builds and runs the fuzzers. This is aimed to catch the "easy to fuzz" bugs quickly, without having to wait until pull requests are merged and eventually built and run by oss-fuzz.

The CI job does the following

- builds a fast fuzzer, with full optimization but less checks which is good at rapidly exploring the input space

- builds a heavily sanitized fuzzer, which is good at detecting errors

- downloads the stored corpus

- runs the fast fuzzer build for a while, to grow the corpus

- runs the sanitizer fuzzer for a while, using the input found by the fast fuzzer

- using a reproduce build (uninstrumented), executes a subset of the test cases in the corpus through valgrind

- minimizes the corpus and uploads it (if on the master branch)

- stores the corpus and valgrind output as artifacts

The job is available under the actions tab, here is a [direct link](https://github.com/simdjson/simdjson/actions?query=workflow%3A%22Fuzz+and+run+valgrind%22).

The corpus will grow over time and easy to find bugs will be detected already during the pull request stage. Also, it will keep the fuzzer builds from bit rot.

There is also a job running the fuzzers on arm64 (see .drone.yml) to make sure also the arm specific parts are fuzzed. This does not update the corpus, it just reuses what the x64 job finds.

The simdjson library is continuously fuzzed on [oss-fuzz](https://github.com/google/oss-fuzz). In case a bug is found, the offending input is minimized and tested for reproducibility. A report with the details is automatically filed, and the contact persons at simdjson are notified via email. An issue is opened at the oss-fuzz bugtracker with restricted view access. When the bug is fixed, the issue is automatically closed.

Bugs are automatically made visible to the public after a period of time. An example of a bug that was found, fixed and closed can be seen here: [oss-fuzz 18714](https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=18714).

You can find the currently open bugs (if any) at [bugs.chromium.org](https://bugs.chromium.org/p/oss-fuzz/issues/list?sort=-opened&q=proj%3Asimdjson&can=2): make sure not to miss the "Open Issues" selector. Bugs that are fixed by follow-up commits are automatically closed.

Changes to the integration with oss-fuzz are made by making pull requests against the oss-fuzz github repo. An example can be seen at [oss-fuzz pull request 3013](https://github.com/google/oss-fuzz/pull/3013).

As little code as possible is kept at oss-fuzz since it is inconvenient to change. The [oss-fuzz build script](https://github.com/google/oss-fuzz/blob/b96dd54183f727a5d90c786e0fb01ec986c74d30/projects/simdjson/build.sh#L18) invokes [the script from the simdjson repo](https://github.com/simdjson/simdjson/blob/master/fuzz/ossfuzz.sh).

Keeping the coverage up is a never ending job. See [issue 368](https://github.com/simdjson/simdjson/issues/368)

To reproduce a test case, use the local build instruction. Then invoke the fuzzer (the fuzz_parser is shown as an example below) with the testcase as a command line argument:

build-sanitizers/fuzz/fuzz_parser my_testcase.json

In case this does not reproduce the bug, you may want to proceed with reproducing using the oss-fuzz tools. See the instructions [here](https://google.github.io/oss-fuzz/advanced-topics/reproducing/).

#include "simdjson.h"

// Split data into two strings, json pointer and the document string.

// Might end up with none, either or both being empty, important for

// covering edge cases such as

// https://github.com/simdjson/simdjson/issues/1142 Inputs missing the

// separator line will get an empty json pointer but the all the input put in

// the document string. This means test data from other fuzzers that take json

// input works for this fuzzer as well.

FuzzData fd(Data, Size);

auto strings = fd.splitIntoStrings();

while (strings.size() < 2) {

strings.emplace_back();

}

assert(strings.size() >= 2);

simdjson::dom::parser parser;

// parse without exceptions, for speed

auto res = parser.parse(strings[0]);

if (res.error())

return 0;

simdjson::dom::element root;

if (res.get(root))

return 0;

auto maybe_leaf = root.at_pointer(strings[1]);

if (maybe_leaf.error())

return 0;

simdjson::dom::element leaf;

if (maybe_leaf.get(leaf))

return 0;

std::string_view sv;

if (leaf.get_string().get(sv))

return 0;