Skip to content

#5186 env var expansion #5434

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 49 additions & 0 deletions atest/testdata/cli/argumentfile/expandvarsinargfile.robot
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
*** Settings ***
Documentation Tests for environment variable expansion in argument files
Library OperatingSystem
Library Process

*** Variables ***
${ARGDIR} ${EXECDIR}${/}atest/testdata/cli/argumentfile

*** Test Cases ***
No Expansion By Default
${result}= Run Process python -m robot --argumentfile ${ARGDIR}${/}test_default.args shell=True
Log ${result.stdout}
Log ${result.stderr}
${combined_output}= Catenate SEPARATOR=\n ${result.stdout} ${result.stderr}
Should Contain ${combined_output} Example
Should Not Contain ${combined_output} $TEST_DIR/output.xml
Should Not Contain ${combined_output} \${TEST_DIR}/log.html

Expansion With Pragma True
Set Environment Variable TEST_DIR test_output
${result}= Run Process python -m robot --argumentfile ${ARGDIR}${/}test_expandtrue.args shell=True
Log ${result.stdout}
Log ${result.stderr}
${combined_output}= Catenate SEPARATOR=\n ${result.stdout} ${result.stderr}
Should Contain ${combined_output} Example
Should Contain ${combined_output} test_output/output.xml
Should Contain ${combined_output} test_output/log.html


Double Expansion With Pragma True
Set Environment Variable TEST_DIR test_output
Set Environment Variable TEST_DIR2 test_output2
${result}= Run Process python -m robot --argumentfile ${ARGDIR}${/}test_doubleexpandtrue.args shell=True
Log ${result.stdout}
Log ${result.stderr}
${combined_output}= Catenate SEPARATOR=\n ${result.stdout} ${result.stderr}
Should Contain ${combined_output} Example
Should Contain ${combined_output} test_output/test_output2/output.xml
Should Contain ${combined_output} test_output/test_output2/log.html

Expansion With Pragma False
Set Environment Variable TEST_DIR test_output
${result}= Run Process python -m robot --argumentfile ${ARGDIR}${/}test_expandfalse.args shell=True
Log ${result.stdout}
Log ${result.stderr}
${combined_output}= Catenate SEPARATOR=\n ${result.stdout} ${result.stderr}
Should Contain ${combined_output} Example
Should Contain ${combined_output} $TEST_DIR/output.xml
Should Contain ${combined_output} \${TEST_DIR}/log.html
3 changes: 3 additions & 0 deletions atest/testdata/cli/argumentfile/test_default.args
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
--output output.xml
--log log.html
atest/testdata/cli/argumentfile/testfile.robot
4 changes: 4 additions & 0 deletions atest/testdata/cli/argumentfile/test_doubleexpandtrue.args
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# expandvars: true
--output $TEST_DIR/$TEST_DIR2/output.xml
--log ${TEST_DIR}/${TEST_DIR2}/log.html
atest/testdata/cli/argumentfile/testfile.robot
4 changes: 4 additions & 0 deletions atest/testdata/cli/argumentfile/test_expandfalse.args
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# expandvars: false
--output $TEST_DIR/output.xml
--log ${TEST_DIR}/log.html
atest/testdata/cli/argumentfile/testfile.robot
4 changes: 4 additions & 0 deletions atest/testdata/cli/argumentfile/test_expandtrue.args
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# expandvars: true
--output $TEST_DIR/output.xml
--log ${TEST_DIR}/log.html
atest/testdata/cli/argumentfile/testfile.robot
4 changes: 4 additions & 0 deletions atest/testdata/cli/argumentfile/testfile.robot
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
*** Test Cases ***
Example
Log Example test

2 changes: 1 addition & 1 deletion src/robot/utils/application.py
Original file line number Diff line number Diff line change
Expand Up @@ -147,4 +147,4 @@ def __enter__(self):
pass

def __exit__(self, *exc_info):
pass
pass
52 changes: 46 additions & 6 deletions src/robot/utils/argumentparser.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@
import sys
import warnings
from pathlib import Path
from string import Template

from robot.errors import DataError, FrameworkError, Information
from robot.version import get_full_version
Expand Down Expand Up @@ -361,7 +362,8 @@ def _raise_invalid_args(self, min_args, max_args, arg_count):
raise DataError(f"{expectation}, got {arg_count}.")


class ArgFileParser:
class ArgFileParser:

def __init__(self, options):
self._options = options

Expand Down Expand Up @@ -407,23 +409,61 @@ def _read_from_stdin(self):

def _process_file(self, content):
args = []
for line in content.splitlines():
lines = content.splitlines()

# Extract pragma handling to dedicated function
expand_vars = self._parse_expandvars_pragma(lines)

for line in lines:
line = line.strip()
if line.startswith("-"):
args.extend(self._split_option(line))
args.extend(self._split_option(line, expand_vars))
elif line and not line.startswith("#"):
args.append(line)
args.append(self._expand_vars(line) if expand_vars else line)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be simpler to expand variables from the whole file before even splitting it into lines than doing that for each line. It would make it impossible to expand variables only from some parts of the lines, but I actually don't think that's a good idea in general. I'll comment that separately later.

return args

def _split_option(self, line):

def _parse_expandvars_pragma(self, lines):
#
#
# design resume:
# - must be on the first line
# - default is off (False)
# - values 'false', 'no', 'off' are considered False
# - empty values are considered False
# - all other values are considered True
if not lines:
return False

first_line = lines[0].strip().lower()
if not first_line.startswith('# expandvars:'):
return False

pragma_value = first_line.split(':', 1)[1].strip().lower()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we also support #expandvars:? What about # expandvars: where the space is actually (and most likely accidentally) a non-break space? An easy way to support them would be using regular expressions. They are often complicated, but in this case I consider the resulting code simpler:

match = re.fullmatch('#\s*expandvars:\s*(.*?)\s*', lines[0], flags=re.IGNORECASE)
if not match:
    return False
value = match.group(1).lower()
return value not in ("false", "no", "off", "")

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think that would be : .upper() if we are goind to use FALSE_STRINGS instead of ("false", "no", "off", "")
otherwise , resolved


if not pragma_value:
return False

if pragma_value in ('false', 'no', 'off'):
return False

return True

def _split_option(self, line, expand_vars=False):
separator = self._get_option_separator(line)
if not separator:
return [line]
option, value = line.split(separator, 1)
if separator == " ":
value = value.strip()
if expand_vars:
value = self._expand_vars(value)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This implementation expands variable only from option values, not from option names. I don't think that's a good idea for the following reasons:

  1. On the commend line you can use --$name value just fine so the implementation is inconsistent with that. I see no problems with supporting variables in option names either.
  2. The implementation doesn't work correctly when using short options without a separator like -N$value. Trying to fix this would be pretty complicated, because short options can be combined like -XN$value. Related to this, at least the -N$value usage should get a test!
  3. If variables are parsed everywhere, it can be done before splitting content to lines which simplifies the overall implementation.
  4. If variables are parsed as the first step, it allows conditional usage like
    $COND --name Example
    
    where $COND could be set to # to exclude the line. Not sure is this that useful, but could come handy in some cases.

return [option, value]

def _expand_vars(self, value):
# Handle escaping with Template.safe_substitute
# (fail gracefully without raising errors for missing variable
return Template(value).safe_substitute(os.environ)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Using Template.safe_substitute instead of os.path.expandvars is probably a good idea, but the latter would support %name% substitution on Windows. If that's important, we ought to be able to to extend Template to support it.
  2. One benefit of using Template is that we could extend it to support ${name=default} style somewhat easily. It could even be part of this enhancement, but it can also be added later.
  3. No need to explain how safe_substiture works here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for the review

i see the trade-oof between using Template.safe_substitute and os.path.expandvars

1- i thought a bit about extending Template to support both ${var} and %var% formats, the solution i saw is :
creating like " EnvVarTemplate subclass " that extend functionality through inheritance to support %var% formats. I think that this would require regex pattern. however, I'm still wondering if adding windows style variable support critical for the current enhancement? I think that i can push this enhancement later since it will requires more tests

2- i' am considering adding this support ( ${name=default} ) but i don't see another solution other then this :

    def _expand_vars(self, value):
        result = Template(value).safe_substitute(os.environ)
        regexpatter, = r'\${([^}]+?)=([^}]*)}'
        def replace_with_default_value(match):
            var, default = match.groups()
            return os.environ.get(var, default)
        return re.sub(regexpatter, replace_with_default_value, result)

3- intended to make a comment for design reminder


def _get_option_separator(self, line):
if " " not in line and "=" not in line:
return None
Expand Down