Skip to content

Argparse: improve parse_known_args() doc #106235

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
vidraj opened this issue Jun 29, 2023 · 5 comments · Fixed by #126921
Closed

Argparse: improve parse_known_args() doc #106235

vidraj opened this issue Jun 29, 2023 · 5 comments · Fixed by #126921
Assignees
Labels
docs Documentation in the Doc dir

Comments

@vidraj
Copy link

vidraj commented Jun 29, 2023

Bug report

When the ArgumentParser.parse_known_args() method is used to parse a parameter with action='append' set, the parameter will be parsed in the whole argument list, including arguments in the unknown section.

Example code showing the issue

import argparse

p = argparse.ArgumentParser()
p.add_argument("--foo", action='append')

print(p.parse_known_args(["--foo", "a", "STOP", "--foo", "b"]))

Expected output

Parsing stops at the first unknown arg (STOP) and all following arguments are left untouched, as they might have to be parsed by a different parser elsewhere.

(Namespace(foo=['a']), ['STOP', '--foo', 'b'])

Actual output

All instances of --foo are parsed, including the ones following the first unknown arg.

(Namespace(foo=['a', 'b']), ['STOP'])

Your environment

  • CPython versions tested on: 3.11.3, 3.10.6, 3.9.16
  • Operating system and architecture: x86_64-pc-linux-gnu

Linked PRs

@vidraj vidraj added the type-bug An unexpected behavior, bug, or error label Jun 29, 2023
@terryjreedy terryjreedy moved this to Bugs in Argparse issues Jun 29, 2023
@terryjreedy
Copy link
Member

The behavior is not a bug as it is documented. The parse_known_args entry has an example something like yours with interleaved known and unknowns and all are processed and assigned to the namespace or unknown tuple. I presume that this example intentionally makes this point.

ArgumentParser.parse_known_args(args=None, namespace=None)

Sometimes a script may only parse a few of the command-line arguments, passing the remaining arguments on to another script or program. In these cases, the parse_known_args() method can be useful. It works much like parse_args() except that it does not produce an error when extra arguments are present. Instead, it returns a two item tuple containing the populated namespace and the list of remaining argument strings.

>>> parser = argparse.ArgumentParser()
>>> parser.add_argument('--foo', action='store_true')
>>> parser.add_argument('bar')
>>> parser.parse_known_args(['--foo', '--badger', 'BAR', 'spam'])
(Namespace(bar='BAR', foo=True), ['--badger', 'spam'])

Perhaps you are implicitly making a feature request, but I do not know argparse well enough to know if what you want is already possible or not.

@terryjreedy terryjreedy added pending The issue will be closed if no feedback is provided and removed type-bug An unexpected behavior, bug, or error labels Jun 29, 2023
@vidraj
Copy link
Author

vidraj commented Jul 6, 2023

Ooh, I misread the example, which indeed makes this behavior clear. Thank you for the explanation! The text of the documentation only talks about “remaining arguments”, making it seem that it means “the rest after some point” and not “any unknown ones anywhere”.

Argparse actually supports what I want it to do, using argparse.ArgumentParser.add_argument("unknown_args", nargs=argparse.REMAINDER) as the last positional parameter, but the argparse.REMAINDER is only obliquely mentioned in the documentation and doesn't seem to be properly documented anywhere. The only downside I see is that it makes the unknown-args argument show up in the help, but that can be solved by adding help=argparse.SUPPRESS.

In that case, I guess I'm making a documentation improvement request. :-) My suggested change would be amendment of the last sentence as follows: “Instead, it skips over any unknown arguments in the command line, only parsing the known ones, and returns a two item tuple containing the populated namespace and the list of skipped argument strings.”

Perhaps I would also add another paragraph as follows:
“Note that parsing the skipped arguments in a later pass may produce unexpected results. If the --badger argument in the example below is supposed to be taking a parameter, it would get spam as the parameter instead of BAR, which was already consumed by parse_known_args().”

And maybe also the following:
“If you want to parse known arguments from the start of the argument list and stop at the first unknown one, you can add an unknown_args positional parameter using argparse.ArgumentParser.add_argument("unknown_args", nargs=argparse.REMAINDER, help=argparse.SUPPRESS) and using parse_args() instead of this function.”

@ericvsmith
Copy link
Member

Can you create a pull request?

@terryjreedy terryjreedy added docs Documentation in the Doc dir and removed pending The issue will be closed if no feedback is provided labels Jul 9, 2023
@terryjreedy terryjreedy changed the title argparse action 'append' used with parse_known_args() consumes unknown arguments Argparse: improve parse_known_args() doc Jul 9, 2023
@vidraj
Copy link
Author

vidraj commented Jul 16, 2023

Yes, I'll get to it in a few days.

@hpaulj
Copy link

hpaulj commented Jul 26, 2023

Documentation for REMAINDER has intentionally been removed, since its use has been somewhat buggy. It is still allowed, but we aren't encouraging its use.

Note that parse_args also uses parse_known_args, only it raises an error when the 'extra' list is not empty.

In any case, parsing tries to handle all strings, putting ones it does not recognize in the namespace under the "UNRECOGNIZED" dest. It does not have a 'short-circuit' option, which would stop parsing the rest. One hallmark of argparse parsing is it tries to handle arguments, at least the optionals, in what ever order the end user provides.

Besides REMAINDER, '--' can act to short-circuit this regular parsing.

@savannahostrowski savannahostrowski moved this to Doc issues in Argparse issues Sep 23, 2024
@savannahostrowski savannahostrowski self-assigned this Nov 17, 2024
savannahostrowski added a commit to savannahostrowski/cpython that referenced this issue Nov 17, 2024
miss-islington pushed a commit to miss-islington/cpython that referenced this issue May 30, 2025
… "remaining" (pythonGH-126921)

(cherry picked from commit a425141)

Co-authored-by: Savannah Bailey <savannahostrowski@gmail.com>
miss-islington pushed a commit to miss-islington/cpython that referenced this issue May 30, 2025
… "remaining" (pythonGH-126921)

(cherry picked from commit a425141)

Co-authored-by: Savannah Bailey <savannahostrowski@gmail.com>
savannahostrowski added a commit that referenced this issue May 30, 2025
…g "remaining" (GH-126921) (#134913)

GH-106235: Clarify `parse_known_args` documentation by removing "remaining" (GH-126921)
(cherry picked from commit a425141)

Co-authored-by: Savannah Bailey <savannahostrowski@gmail.com>
savannahostrowski added a commit that referenced this issue May 30, 2025
…g "remaining" (GH-126921) (#134914)

GH-106235: Clarify `parse_known_args` documentation by removing "remaining" (GH-126921)
(cherry picked from commit a425141)

Co-authored-by: Savannah Bailey <savannahostrowski@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
docs Documentation in the Doc dir
Projects
Status: Doc issues
Development

Successfully merging a pull request may close this issue.

5 participants