What do you think about the style?

I've provided some early feedback in spectrum, let's chat about it there :)

this snapshot-based system looks nice. it seems like ESLint also supports endLine and endColumn information for errors. can that be surfaced in the snapshots somehow (we do it in TSLint)?

I love this idea (you know I wanted to do something like this in the past).
Great work building out a POC!

Some feedback I have (note this is based off not running the code....):

• we probably want to separate valid/invalid snapshot files
  this lets us be explicit in defining what should and shouldn't fail; which makes it easier to review PRs.
  Right now something might be erroneously passing, and we wouldn't have any idea.

• we need a nice way to separate test cases
  ESP for the rule you've picked (which has multiline test cases), it's hard to tell when one test case starts and another finishes. Without this, it will make it hard to review and maintain if we need to move cases around / duplicate cases for different option configs.

• needs a nice way to set options for a specific test case
  Could do via eslint comments, but the problem is that comments just turn a rule on or off indefinitely, so you have to then also have a way to turn off the rule after a test case.
  (unless of course your runner will treat each case as a separate run through eslint, then just an enable comment is all you need :)

• the snapshot needs to have better delimiters
  It needs to be easy to review! IMO it needs to have two clear sections for each error case: Failing code case, and error output.
  The former will heavily rely on the previous point of clear separation of cases.
  For the error output, it'd be good to dump out as much of the eslint error output as you can (like @adidahiya said - message, line, column, start/end, range). The more you dump, the more likely we are to catch regressions.

• we probably want to separate valid/invalid snapshot files
• we need a nice way to separate test cases
• needs a nice way to set options for a specific test case

Perhaps we should do it like TSLint does? Have a directory for each rule, with subdirectories for each test. Each could have a valid.ts, invalid.ts, options.json (or YAML/TOML/JSON5/CSON?) and errors.snap in it, which would either contain an array of options to spread, or a primitive/object containing the single option to pass.

we probably want to separate valid/invalid snapshot files

Personally I don't find this worth the effort... If the test snapshot file uses a clear enough syntax (TSLint array-type example), you can see where lint errors are expected in the file and where they aren't. But the decision of using many small files vs. fewer larger ones is often a matter of personal style preference, so I don't feel strongly about that.

the snapshot needs to have better delimiters

+1, this change shouldn't be too hard to incorporate into this PR, though, I think it's just a matter of making a new (let's call it "testSnap") formatter which generates more reviewable snapshots?

needs a nice way to set options for a specific test case

+1 for this, I think using subdirectories for each different rule option configuration (rather than each individual test case) is a good middle ground. Instead of options.json, how about using a standard .eslintrc in each subdirectory to adjust the rule options?

@adidahiya if you've got a format like that, that certainly helps a lot in terms of clarity!

Having separate valid/invalid files might be good because it's more explicit in cases where there's multiple lines.
But then again I'm a sucker for being explicit :)

Here’s the data ESLint gives us:

{
    fatal: false,
    ruleId: "semi",
    severity: 2,
    line: 1,
    column: 23,
    message: "Expected a semicolon.",
    fix: {
        range: [1, 15],
        text: ";"
    }
}

Huh, is that all? Well then we only want line, column and message.
Fixer information doesn't really help if we've got separate fix tests (what would these look like?)

one other question I've got - are we thinking of supporting an api for this to generate tests?

For example, for indent I did a bit of generation to create valid and invalid + fixer tests instead of copy+pasting the examples out: https://github.com/typescript-eslint/typescript-eslint/blob/master/packages/eslint-plugin/tests/lib/rules/indent.js

Huh, is that all? Well then we only want line, column and message.

Ah, they do have endLine and endColumn if they’re available.

Fixer information doesn't really help if we've got separate fix tests (what would these look like?)

Maybe a .fixed.ts file?

one other question I've got - are we thinking of supporting an api for this to generate tests?

I wasn’t planning on it — if you want to do something fancy, you can use the RuleTester API. I’d rather not make this too complicated.

Maybe a .fixed.ts file?

I wish to see the fixed code beside the original code. How about a comment on the next line of the original code?

foo.indexOf("bar") === 0
/*fixedTo:
foo.startsWith("bar")
*/

needs a nice way to set options for a specific test case

We can write options in code:

/*eslint arbitrary-rule-name: [error, { option: true }] */

some code

So I think it's enough we can separate tests to multiple files.

tests/rules
└ rule-name
   ├ default.src.ts
   ├ an-option.src.ts
   └ another-option.src.ts

I'm going to close this for housekeeping purposes.