Variable argument count plays badly with choices.

Example:
========

>>> import argparse
>>> parser = argparse.ArgumentParser()
>>> parser.add_argument('choices', nargs='*', default='a', choices=['a',
'b', 'c'])
>>> args = parser.parse_args()
>>> print type(args.choices)
<type 'str'>
>>> args = parser.parse_args(['a'])
>>> print type(args.choices)
<type 'list'>

If the user specifies the value on the command line then a list is used, but if the value comes from the default a string is used.
Unfortunately, changing default to a list value gives an error:
error: argument choices: invalid choice: ['a'] (choose from 'a', 'b',
'c')

Additionally, this means it is also not possible to use default=['a', 'c'].

The current workaround is to create a custom type:

def my_type(string):
    if string not in ['a', 'b', 'c']:
        raise TypeError
    return string

Maybe it will sound strange, but what is this task REALLY about? I mean - I can see two problems here, but no clear information about which problem is a real problem and - if it is - what is the expected behavior.

Problems I can see are:

1. Type of returned value changes depending on the value source (list for user provided or list for default)
2. It's impossible to set list as 'default'

I understand that this task concentrates on 1st one, however changing behavior described in 2nd would - in my oppinion - fix 1st one too - am I right? User would "define" the returned type by defining the 'default' as a list or string.

But - if we assume that we only discuss 1st problem here - what is the expected behavior? Provided workaround is suggesting that we expect string, but - basing on current nargs behavior and "intuition" - I'd rather expect to get a list, when I define nargs with "*" or "+". This sounds "natural" for me.

Could someone explain me the problem and the expected behavior in a clear way?

Sorry if I'm not clear, but it's my first post here and maybe I've missed something in the description or I assume something that is not true (especially that the task is quite old... ;) ).

Of course I've made a mistake:

"list for user provided or list for default"

should be:

"list for user provided or STRING for default"

The real issue is that the choices flag does not work with a default flag and * nargs.

The following works as expected:
>>> parser.add_argument('chosen', nargs='*', default=['a'])
>>> print(parser.parse_args())
Namespace(chosen=['a'])
>>> print(parser.parse_args(['a', 'b']))
Namespace(chosen=['a', 'b'])

Introducing a choices constraint breaks down when using the defaults:
>>> parser.add_argument('chosen', nargs='*', default=['a'], choices=['a', 'b'])
>>> print(parser.parse_args(['a']))
Namespace(chosen=['a'])
>>> print(parser.parse_args())
error: argument chosen: invalid choice: ['a'] (choose from 'a', 'b')

I would expect instead to have Namespace.chosen populated with the default list as before, but the choices constraint check does not validate correctly.

I think that changing the choices constraint logic to iterate over the default values if nargs results in a list would be a possible solution.

I agree that this looks like a bug. I think the fix is something like the attached patch, but it needs some tests to make sure that it fixes your problem.

Here is a new version of the patch with tests

ping

I've added 2 more tests,

one with default='c', which worked before.

one with default=['a','b'], which only works with this change.

http://bugs.python.org/issue16878 is useful reference, since it documents
the differences between nargs="?" and nargs="*", and their handling of
their defaults.

Change "choices='abc'" to "choices=['a', 'b', 'c']", as discussed in bpo-16977 (use of string choices is a bad example)

The patch I just posted to http://bugs.python.org/issue16468 uses this fix.

There's a complicating issue - should these default values be passed through the type function?

In most cases in _get_values, the string first goes through _get_value, and then to _check_value.

For example the 'else:' case:

            value = [self._get_value(action, v) for v in arg_strings]
            for v in value:
                self._check_value(action, v)

The '*' positional case could coded the same way, allowing:

parser.add_argument('foo',
     nargs='*',
     type=int,
     choices=range(5),
     default=['0',1,'2'])

and objecting to

         default=[6,'7','a'] # out of range string or int or invalid value

This does impose a further constraint on the 'type' function, that it accepts a converted value. e.g. int(1) is as valid as int('1').

But we need to be careful that this case is handled in a way that is consistent with other defaults (including the recent change that delayed evaluating defaults till the end).

Recent StackOverFlow question related to this issue

http://stackoverflow.com/questions/41750896/python-argparse-type-inconsistencies-when-combining-choices-nargs-and-def/41751730#41751730

Another workaround for who might ever need it. The benefit of this solution comparing to a custom type is that argparse will generate the help string properly with the choices, and this solution does workaround the place when the bug happens:

    class Choices(tuple):
        # Python bug https://bugs.python.org/issue27227
        def __new__(cls, *args, **kwargs):
            x = tuple.__new__(cls, *args, **kwargs)
            Choices.__init__(x, *args, **kwargs)
            return x

        def __init__(self, *args, **kwargs):
            self.default = []

        def __contains__(self, item):
            return tuple.__contains__(self, item) or item is self.default

    choices = Choices(("value1", "value2", "value3", ...))

    parser.add_argument(
        ...,
        nargs="*",
        choices=choices,
        default=choices.default,
        ...)

It looks like choices are broken for nargs='*' even without using default:

>>> import argparse
>>> parser = argparse.ArgumentParser()
>>> parser.add_argument('test', nargs='*', choices=['foo', 'bar', 'baz'])
_StoreAction(option_strings=[], dest='test', nargs='*', const=None, default=None, type=None, choices=['foo', 'bar', 'baz'], help=None, metavar=None)
>>> parser.parse_args([])
usage: [-h] [{foo,bar,baz} [{foo,bar,baz} ...]]
: error: argument test: invalid choice: [] (choose from 'foo', 'bar', 'baz')

I think there is a same issue with "nargs='+'" - if you are aware of the, please ignore me.

$ python3 --version
Python 3.7.5

With "choices=...", there seems to be a problem:

$ cat bbb.py 
#!/usr/bin/env python3

import argparse

parser = argparse.ArgumentParser()
parser.add_argument('choices', nargs='*',
                    default=['a', 'b', 'c'],
                    choices=['a', 'b', 'c'])
args = parser.parse_args()

print(args)
$ ./bbb.py 
usage: bbb.py [-h] [{a,b,c} [{a,b,c} ...]]
bbb.py: error: argument choices: invalid choice: ['a', 'b', 'c'] (choose from 'a', 'b', 'c')
$ ./bbb.py a c
Namespace(choices=['a', 'c'])

but without choices it works:

$ cat bbb.py 
#!/usr/bin/env python3

import argparse

parser = argparse.ArgumentParser()
parser.add_argument('choices', nargs='*',
                    default=['a', 'b', 'c'])
args = parser.parse_args()

print(args)
$ ./bbb.py 
Namespace(choices=['a', 'b', 'c'])
$ ./bbb.py a c
Namespace(choices=['a', 'c'])

As I said, if this is a known issue, I'm sorry for the noise.

I would love to get this issue resolved; it seems like everyone agrees that it's a bug. It came up for me recently: https://bugs.python.org/issue41047. Judging from the comments above, the consensus is that the relevant line, self._check_value(action, value) should either be replaced with something like if isinstance(value, collections.abc.Sequence): for v in value: self._check_value(action, v) or be removed entirely.

I think the line should just be removed. I think it's fair to assume that users of argparse know what they're doing, so I think they should be allowed to pass default values that conflict with choices. Also, removing the line makes the behavior consistent with the optionals, which don't check whether default values are in choices. See the below script:

import argparse


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--foo", nargs="+", default=[-1], choices=range(10))
    parser.add_argument("--bar", nargs="*", default=-1, choices=range(10))
    parser.add_argument("pos", nargs="?", default=-1, choices=range(10))
    args = parser.parse_args()
    print(args)


if __name__ == '__main__':
    main()

Which yields:

$ python argparse_test.py 
Namespace(foo=[-1], bar=-1, pos=-1)

Paul, what is your current thinking on this?

Another workaround for who might ever need it. The benefit of this solution comparing to a custom type is that argparse will generate the help string properly with the choices, and this solution does workaround the place when the bug happens:

    class Choices(tuple):
    ...

Thanks @JooEiras, this is a very nice workaround, but I think it can be simplified:

class Choices(tuple):

    def __init__(self, _iterable=None, default=None):
        # _iterable is already handled by tuple.__new__
        self.default = default or []

    def __contains__(self, item):
        return super().__contains__(item) or item == self.default

This has the added benefit of being able to set default when creating the object, and working when omitting default=choices.default from parser.add_argument(...), if no default is set for the Choices object.

Any concerns about this change?

PS: I think Choices.__init__ is actually ran twice in the original workaround. I don't think there's a reason for this.

I think that we should allow arbitrary default and should omit checks if the default value is used. It can fail only if the programmer provides default that does not pass the check. The check does not check input, it restricts the programmer. In this case it always fail because default and choices are always incompatible if nargs='*'.

This also allows to distinguish the default value from the specified value if nargs='?'.