Parameterized suite, parameterized test + cmd commands #124

killoctal · 2016-08-25T16:12:42Z

Hi,
I spent some hours on my free time and at my work to develop two functionalities :

Parameterized tests (more exactly -> parameters for tests)
Cmd commands (--test, --suite, --ignoreparam, feel free to add a --help if you want to)

Actually I made cmd for the needs of the parameterized tests.

Important : Please note that I did not testes it on other platforms than Windows, in *.exe mode because I think you have more knowledges than me for this kind of testing :) Anyway the code is fully tested (see unittest-cpp\tests\TestParameterizedTest.cpp). Can I let you update the code for it works as a *.dll, *.lib, etc and for other platforms ?

For your test program you have to set the main this way :

int main(int argc, char**argv)
{
    return UnitTest::RunTestsCmd(argc, argv); // <- new entry point, replaces UnitTest::RunAllTests
}

And here is a basic example, the only macro you need is SET_TEST_PARAMETER :

// File: MyParameters.h
// Note: I recommand prefix your parameter name with "pz"
SET_TEST_PARAMETER(string, pzOneTwo, {
    values.push_back("one");
    values.push_back("two");
});

// File: MyTests.cpp
string result;
TEST(OneTwoIsFollowing)
{
    result += pzOneTwo();
    result += " ";
}
TEST(OneTwoIsFollowing_Verify)
{
    CHECK_EQUAL("one two ", result);
}

See file unittest-cpp\tests\TestParameterizedTest.cpp for more complex examples.

Best regards

killoctal · 2016-08-26T08:33:06Z

Apparently it does not compile with older version than Visual Studio 2010, I will fix it soon

pjohnmeyer · 2016-08-26T14:15:45Z

Thanks for the PR.

I'm a bit concerned about the size and scope of this. I realize that I don't have a contribution guide posted (bad maintainer, no Twinkie) -- but there are three (possibly more?) things going on here that are not all related:

Changing int to size_t in several places, including interfaces
The new runner
The new parameterized method of running a test with multiple inputs

At the least, they need to be broken up into separate PRs so the changes can be considered individually. That said, I don't want to waste your time on doing all that without offering some feedback first.

Changing int to size_t is likely the right thing to do, but I have to check my notes -- I seem to recall there's an issue with using size_t here for certain, shall we say, odd compilers that I try to maintain support for. Given that a 2.0 release is forthcoming, now would be a great time to make that change if we can.

The new runner is something many people have done from outside of the framework, and I would not be opposed to adding an "official" one to the project. I don't think I grasp, though, why this one is as complicated as it is, or why the RunTestsCmd function has a signature that takes more than argc and argv. Can you explain it to me? I don't see any unit tests for it that might help me understand.

Finally, parameterized tests. I have a lot of issues with this feature as implemented, and frankly I feel that it's just... out of place. I think, if we worked together, we could come up with an implementation myself and the users would be happy with, but this is the kind of new feature that would really lend itself to a discussion ticket before being implemented. Of course, having a proposed implementation to start is helpful for the discussion.

I don't want to discourage you from contributing, but I am closing this PR. Please feel free to continue the discussion here or on the issue board, and by all means open up separate PRs as you see fit.

killoctal · 2016-08-26T15:39:15Z

Hi John,

Thank you for your review, I am a bit disappointed but not discouraged so I will improve until you can validate the PR ;)

I will shortly tell you the story of the tests parameters :
I began to wrote the first version at my work, locally in my project. I needed the functionality quickly and I was not thinking "open source" but when I realized I did a cool stuff I ask to the boss if I can publish the functionality and surprisingly he said yes :) Then I forked the repo and integrated the functionality in the project. I added some functionalities and here we are.

About the size_t, I must admit that I do not remember why I had to replace int with size_t. I think it was not that much necessary and for simplification of my next PR I will revert that.

About the runner, I wrote it because of test parameters needs (for the parameter --ignoreparam, which is helpful). It is that much complicated because I tried to respect the way of the original RunAllTests was written, but I did really wanted to make it simpler, in a dedicated class and using strings etc. If you want I can make a simpler version in a dedicated PR ?

I opened an issue, for discuss about the modifications that you would like.

Best regards

pjohnmeyer · 2016-08-27T04:24:06Z

If you'd like to put the int->size_t changes into a pull request I'll take a look, although now that you point it out I may go after it if I don't see that PR in a few days.

As for the runner, I'd be more interested in a simpler version that the user can just pass argc and argv straight through. It'd probably be good to put only one or two options on it at first so that it can be built up incrementally. How about just building one that accepts one --suite SuiteName argument for starters?

killoctal · 2016-08-29T07:48:19Z

I will try to take time some time for make these two PRs this week

killoctal added 30 commits May 11, 2016 10:33

Imported files from other project

d083237

removed dependency to <functional> (and lambdas)

68118d6

pass values in constructor instead of template argument

d8a5370

removed cascading constructor

d01652f

removed dependency to <memory> and unique_pointer

2a3c4ef

fixed uncoherent test name

313830b

added endl after warning

a05e709

less C++11

a6c56fe

created macro SET_SUITE_PARAMETERS

ab3cc22

updated tests for using macros

1dbb31f

little macro syntax change

0a3c699

created macro PARAMETERIZED_SUITE (usage is like SUITE marco)

abb0667

created test

1966eee

changed params order

8800546

added some const methods

f9f3523

splitted class in .h/.cpp

17ec122

created empty class ParameterizedTest

f6f0777

method for retrieve a test from its details

a1510a1

throw exception if test not retrieved

908f7b5

non static members

0be4fef

coded feature

183209f

created simple test (not passing yet)

e525453

fixed test

9ec5722

prevent for empty values

1f2c10d

bug fix

77fef1d

fix: nested tests

e558ada

updated macro for use test instead of suite

2102d05

added callback feature

817fbc8

migrated tests

29bb785

totally removed class "ParameterizedSuite"

1a1e041

killoctal added 22 commits July 12, 2016 13:27

minor chage: const ref

90c678e

do not throw exception if index out of range

7ed0732

minor simplifications

f7b47b6

macro SET_TEST_PARAMETER_LISTENER(Type, Name, ListenerPtr, SetUpBody)

b697f9c

method for check if a parameter is ignored

ebe11e5

method rename

87146c1

added possibility to set auto or force local/global ignore

4543b15

added feature ignore index by parameter name

672b17b

created tests

0f7dd82

simplification: added SuitePredicate::addAll

57cc963

cleaned params argument and now mandatory param "--test"

c077a46

analyze ignore by array name

c91ff62

added tests

923cd55

added argument "--ignoreparam" (ex: --ignoreparam pzToto[0,2,6])

62378a6

added comment about how to use commands

924c8ed

fixed comment

16e0dc2

generalization

69ccda4

specify "--test" is optional

21b1a59

removed non pertinant and problematic condition

a77b0c7

Merge branch 'master' into ParameterizedSuite

2647991

added long/short macro

b48bd98

forgot to remove this test after commit a77b0c7

17d6e82

pjohnmeyer closed this Aug 26, 2016

killoctal mentioned this pull request Aug 26, 2016

Parameterized tests, parameterized suites (parameters for tests) #125

Open

pjohnmeyer mentioned this pull request Dec 16, 2016

Run tests/suites using custom selection from command line #132

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Parameterized suite, parameterized test + cmd commands #124

Parameterized suite, parameterized test + cmd commands #124

Uh oh!

killoctal commented Aug 25, 2016

Uh oh!

killoctal commented Aug 26, 2016

Uh oh!

pjohnmeyer commented Aug 26, 2016

Uh oh!

killoctal commented Aug 26, 2016

Uh oh!

pjohnmeyer commented Aug 27, 2016

Uh oh!

killoctal commented Aug 29, 2016

Uh oh!

Uh oh!

Parameterized suite, parameterized test + cmd commands #124

Parameterized suite, parameterized test + cmd commands #124

Uh oh!

Conversation

killoctal commented Aug 25, 2016

Uh oh!

killoctal commented Aug 26, 2016

Uh oh!

pjohnmeyer commented Aug 26, 2016

Uh oh!

killoctal commented Aug 26, 2016

Uh oh!

pjohnmeyer commented Aug 27, 2016

Uh oh!

killoctal commented Aug 29, 2016

Uh oh!

Uh oh!