Skip to content

Non-ASCII characters in os.environ cause silent failures in test_httpservers #88813

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
ambv opened this issue Jul 15, 2021 · 7 comments
Closed

Non-ASCII characters in os.environ cause silent failures in test_httpservers #88813

ambv opened this issue Jul 15, 2021 · 7 comments
Labels
3.9 only security fixes 3.10 only security fixes 3.11 only security fixes type-bug An unexpected behavior, bug, or error

Comments

@ambv
Copy link
Contributor

ambv commented Jul 15, 2021

BPO 44647
Nosy @ambv, @miss-islington
PRs
  • bpo-44647: Fix test_httpservers failing on Unicode characters in os.environ on Windows #27161
  • [3.10] bpo-44647: Fix test_httpservers failing on Unicode characters in os.environ on Windows (GH-27161) #27169
  • [3.9] bpo-44647: Fix test_httpservers failing on Unicode characters in os.environ on Windows (GH-27161) #27170
  • bpo-44647: Add a permanent Unicode-valued env var to regrtest  #27187
  • [3.10] bpo-44647: Add a permanent Unicode-valued env var to regrtest (GH-27187) #27191
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2021-07-15.20:13:07.907>
created_at = <Date 2021-07-15.17:50:20.422>
labels = ['type-bug', '3.9', '3.10', '3.11']
title = 'Non-ASCII characters in os.environ cause silent failures in test_httpservers'
updated_at = <Date 2021-07-16.13:55:33.744>
user = 'https://github.com/ambv'

    bugs.python.org fields:

    activity = <Date 2021-07-16.13:55:33.744>
actor = 'lukasz.langa'
assignee = 'none'
closed = True
closed_date = <Date 2021-07-15.20:13:07.907>
closer = 'lukasz.langa'
components = []
creation = <Date 2021-07-15.17:50:20.422>
creator = 'lukasz.langa'
dependencies = []
files = []
hgrepos = []
issue_num = 44647
keywords = ['patch']
message_count = 6.0
messages = ['397572', '397576', '397578', '397579', '397621', '397630']
nosy_count = 2.0
nosy_names = ['lukasz.langa', 'miss-islington']
pr_nums = ['27161', '27169', '27170', '27187', '27191']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue44647'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']
    @ambv
    Copy link
    Contributor Author

    ambv commented Jul 15, 2021

    #67826 introduced a new test for Accept: headers in CGI HTTP servers. This test serializes all of os.environ on the server side. For non-UTF8 locales this can fail for some Unicode characters found in environment variables.

    This started failing this week on Azure Pipelines with their rollout of a new Windows 2019 image version that included a "BUILD_SOURCEVERSIONAUTHOR" env variable. For me specifically it includes a leading Unicode character so all my PRs started failing on Azure Pipelines Windows 2019 alone.

    The result was truncated output from the CGI HTTP server, like:

    ======================================================================
    FAIL: test_accept (test.test_httpservers.CGIHTTPServerTestCase) [OrderedDict([('Accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8')])]
    ----------------------------------------------------------------------

    Traceback (most recent call last):
  File "D:\a\1\s\lib\test\test_httpservers.py", line 860, in test_accept
    self.assertIn(expected.encode('ascii'), res.read())
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: b"'HTTP_ACCEPT': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'" not found in b''

    The root cause is that the CGI script in question (cgi_file6 in test_httpservers.py) is crashing on server-side reaching the print(repr(os.environ)) line. However, this exception isn't visible on the client side where the test is running. The only visible issue is truncated output.

    I suggest adding ENSURE_UNICODE_WORKS=Łukasz to the testing env so that this never regresses.

    @ambv ambv added 3.9 only security fixes 3.10 only security fixes 3.11 only security fixes type-bug An unexpected behavior, bug, or error labels Jul 15, 2021
    @ambv
    Copy link
    Contributor Author

    ambv commented Jul 15, 2021

    New changeset 82b218f by Łukasz Langa in branch 'main':
    bpo-44647: Fix test_httpservers failing on Unicode characters in os.environ on Windows (GH-27161)
    82b218f

    @ambv
    Copy link
    Contributor Author

    ambv commented Jul 15, 2021

    New changeset 356bdff by Miss Islington (bot) in branch '3.10':
    bpo-44647: Fix test_httpservers failing on Unicode characters in os.environ on Windows (GH-27161) (bpo-27169)
    356bdff

    @ambv
    Copy link
    Contributor Author

    ambv commented Jul 15, 2021

    New changeset 95596d5 by Miss Islington (bot) in branch '3.9':
    bpo-44647: Fix test_httpservers failing on Unicode characters in os.environ on Windows (GH-27161) (bpo-27170)
    95596d5

    @ambv ambv closed this as completed Jul 15, 2021
    @ambv ambv closed this as completed Jul 15, 2021
    @ambv
    Copy link
    Contributor Author

    ambv commented Jul 16, 2021

    New changeset 7915c96 by Łukasz Langa in branch 'main':
    bpo-44647: Add a permanent Unicode-valued env var to regrtest (bpo-27187)
    7915c96

    @ambv
    Copy link
    Contributor Author

    ambv commented Jul 16, 2021

    New changeset 93d36a5 by Miss Islington (bot) in branch '3.10':
    bpo-44647: Add a permanent Unicode-valued env var to regrtest (GH-27187) (bpo-27191)
    93d36a5

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    @serhiy-storchaka
    Copy link
    Member

    It introduced a regression. See #91904.

    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees
    No one assigned
    Labels
    3.9 only security fixes 3.10 only security fixes 3.11 only security fixes type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    2 participants
    @ambv @serhiy-storchaka