gh-92936: allow double quote in cookie values #113663

nburns · 2024-01-02T19:47:15Z

As detailed more extensively in: #92936 it's not uncommon to see cookies with json and therefore double quotes in them. While IMO a well behaved application would base64 or otherwise encode those values, there are numerous services putting double quotes in the cookies, and it would be nice to have support for this directly in python so various http server/client/util libraries don't need to implement their own cookie parsing. Additionally browsers are tolerant of and handle double quotes in cookie values so SimpleCookie should also.

Before this change SimpleCookie will without error drop cookie values that appear after a value with a double quote in them, which can lead to some very confusing and hard to debug issues when implementing http clients and servers.
After this change SimpleCookie allows a double quote character in the values section of the cookie while making no attempt to determine if the value is valid json or anything else (since that's a application/usecase specific concern)

Downstream issues:
aio-libs/aiohttp#7993
yt-dlp/yt-dlp#4780

Issue: SimpleCookie() fails for json-like values with embedded double-quotes #92936

nburns · 2024-01-03T00:14:07Z

Lib/http/cookies.py

@@ -442,7 +442,7 @@ def OutputString(self, attrs=None):
    (                              # Optional group: there may not be a value.
    \s*=\s*                          # Equal Sign
    (?P<val>                         # Start of group 'val'
-    "(?:[^\\"]|\\.)*"                  # Any doublequoted string


webob handles cookies with quotes more robustly: https://github.com/Pylons/webob/blob/main/src/webob/cookies.py#L335

nburns · 2024-01-12T15:55:23Z

@orsenthil sorry for the random mention, but I saw you were active on some other cookie related issues in the past and wondered if you could review this or help me get pointed in the direction to find someone to review this?

danbirken · 2025-02-22T07:09:23Z

I have nothing to add to this other than to say that I spent about 4 hours today debugging an issue caused and it almost drove me to madness before I found the problem. This is a particularly pernicious issue because the error is caused by a level of the infrastructure you expect to never fail.

I'm sure this impacts people all the time and they have no idea why their thing doesn't work and just give up.

Keep fighting the good fight trying to get this merged! I have no power to help you but the PR looks good to me.

orsenthil

Added an additional test case demonstrating the need for a lazy match in regex.

orsenthil

Formatted the test case.
Added an additional test for lazy matching of quoted characters, which would fail if the match was greedy.

Lib/test/test_http_cookies.py

orsenthil · 2025-08-08T09:52:06Z

@nburns ,and @danbirken - thanks for raising this issue. Apologize for not looking into this or responding to the ping.

Lib/test/test_http_cookies.py

python-cla-bot · 2025-08-08T16:13:54Z

All commit authors signed the Contributor License Agreement.

Co-authored-by: Senthil Kumaran <senthil@python.org>

nburns · 2025-08-08T18:28:23Z

@orsenthil Thanks for taking a look, I applied your suggestion and fixed up the tests, also updated the news entry and signed the CLA, is there anything else I can do to help move this PR forward?

orsenthil · 2025-08-08T19:06:49Z

I applied your suggestion and fixed up the tests, also updated the news entry

Thank you for updating and fixing the tests. I didn't realize the failure when I ran it yesterday. Thanks.

picnixz · 2025-08-08T19:25:07Z

Can we have a follow-up PR here that adds a What's New entry and updates the documentation of http.cookies please?

picnixz · 2025-08-08T19:28:21Z

Also, please be careful but we have some examples with quotes on the online docs so maybe they should be updated.

nburns · 2025-08-08T20:11:26Z

Sure, I'll take a look

bedevere-bot · 2025-08-08T20:28:52Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 RHEL8 Refleaks 3.x (tier-1) has failed when building commit d7dbde8.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/259/builds/3132) and take a look at the build logs.
Check if the failure is related to this commit (d7dbde8) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/259/builds/3132

Failed tests:

test.test_multiprocessing_fork.test_processes

Failed subtests:

test_interrupt - test.test_multiprocessing_fork.test_processes.WithProcessesTestProcess.test_interrupt

Summary of the results of the build (if available):

==

Click to see traceback logs

Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.refleak/build/Lib/test/_test_multiprocessing.py", line 588, in test_interrupt
    exitcode = self._kill_process(multiprocessing.Process.interrupt)
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.refleak/build/Lib/test/_test_multiprocessing.py", line 570, in _kill_process
    self.assertEqual(join(), None)
                     ~~~~^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.refleak/build/Lib/test/_test_multiprocessing.py", line 250, in __call__
    return self.func(*args, **kwds)
           ~~~~~~~~~^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.refleak/build/Lib/multiprocessing/process.py", line 156, in join
    res = self._popen.wait(timeout)
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.refleak/build/Lib/multiprocessing/popen_fork.py", line 44, in wait
    return self.poll(os.WNOHANG if timeout == 0.0 else 0)
           ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.refleak/build/Lib/multiprocessing/popen_fork.py", line 28, in poll
    pid, sts = os.waitpid(self.pid, flag)
               ~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.refleak/build/Lib/test/_test_multiprocessing.py", line 566, in handler
    raise RuntimeError('join took too long: %s' % p)
RuntimeError: join took too long: <Process name='Process-2103' pid=185712 parent=169200 started daemon>


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.refleak/build/Lib/multiprocessing/process.py", line 320, in _bootstrap
    self.run()
    ~~~~~~~~^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.refleak/build/Lib/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.refleak/build/Lib/test/_test_multiprocessing.py", line 518, in _sleep_some_event
    time.sleep(100)
    ~~~~~~~~~~^^^^^
KeyboardInterrupt
k


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.refleak/build/Lib/test/_test_multiprocessing.py", line 588, in test_interrupt
    exitcode = self._kill_process(multiprocessing.Process.interrupt)
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.refleak/build/Lib/test/_test_multiprocessing.py", line 570, in _kill_process
    self.assertEqual(join(), None)
                     ~~~~^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.refleak/build/Lib/test/_test_multiprocessing.py", line 250, in __call__
    return self.func(*args, **kwds)
           ~~~~~~~~~^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.refleak/build/Lib/multiprocessing/process.py", line 156, in join
    res = self._popen.wait(timeout)
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.refleak/build/Lib/multiprocessing/popen_fork.py", line 44, in wait
    return self.poll(os.WNOHANG if timeout == 0.0 else 0)
           ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.refleak/build/Lib/multiprocessing/popen_fork.py", line 28, in poll
    pid, sts = os.waitpid(self.pid, flag)
               ~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.refleak/build/Lib/test/_test_multiprocessing.py", line 566, in handler
    raise RuntimeError('join took too long: %s' % p)
RuntimeError: join took too long: <Process name='Process-1' pid=196164 parent=196162 started daemon>

nburns · 2025-08-08T20:36:39Z

updated the docs in #137566

bedevere-app bot added the awaiting review label Jan 2, 2024

bedevere-app bot mentioned this pull request Jan 2, 2024

SimpleCookie() fails for json-like values with embedded double-quotes #92936

Open

nburns force-pushed the allow-double-quote-in-cookie-values branch from 4e125ac to 1b3b200 Compare January 3, 2024 00:12

nburns commented Jan 3, 2024

View reviewed changes

nburns force-pushed the allow-double-quote-in-cookie-values branch 2 times, most recently from 78dce80 to 1b78ed1 Compare January 8, 2024 18:22

nburns force-pushed the allow-double-quote-in-cookie-values branch from 1b78ed1 to 3eae48a Compare January 12, 2024 15:51

nburns force-pushed the allow-double-quote-in-cookie-values branch from 95af939 to e9b3cd5 Compare September 4, 2024 16:45

nburns force-pushed the allow-double-quote-in-cookie-values branch from e9b3cd5 to 6e3d1d7 Compare August 6, 2025 21:32

nburns mentioned this pull request Aug 6, 2025

http.cookies.SimpleCookie.load() fails to consistently handle malformed cookies #127195

Open

orsenthil reviewed Aug 8, 2025

View reviewed changes

Lib/test/test_http_cookies.py Outdated Show resolved Hide resolved

orsenthil reviewed Aug 8, 2025

View reviewed changes

Lib/test/test_http_cookies.py Outdated Show resolved Hide resolved

nburns force-pushed the allow-double-quote-in-cookie-values branch 2 times, most recently from f88d0a7 to 72c88f9 Compare August 8, 2025 16:42

nburns and others added 2 commits August 8, 2025 16:50

allow double quote in cookie values

423efc5

Update Lib/test/test_http_cookies.py

081ace2

Co-authored-by: Senthil Kumaran <senthil@python.org>

nburns force-pushed the allow-double-quote-in-cookie-values branch from 72c88f9 to 081ace2 Compare August 8, 2025 16:50

orsenthil approved these changes Aug 8, 2025

View reviewed changes

bedevere-app bot added awaiting merge and removed awaiting review labels Aug 8, 2025

orsenthil merged commit d7dbde8 into python:main Aug 8, 2025
46 checks passed

bedevere-app bot removed the awaiting merge label Aug 8, 2025

Dreamsorcerer mentioned this pull request Aug 8, 2025

Fix cookie parsing issues aio-libs/aiohttp#11112

Merged

nburns mentioned this pull request Aug 8, 2025

gh-92936: update http.cookies docs post GH-113663 #137566

Open

Uh oh!

gh-92936: allow double quote in cookie values #113663

gh-92936: allow double quote in cookie values #113663

Uh oh!

Conversation

nburns commented Jan 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nburns Jan 3, 2024

Choose a reason for hiding this comment

Uh oh!

nburns commented Jan 12, 2024

Uh oh!

danbirken commented Feb 22, 2025

Uh oh!

orsenthil left a comment

Choose a reason for hiding this comment

Uh oh!

orsenthil left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

orsenthil commented Aug 8, 2025

Uh oh!

Uh oh!

python-cla-bot bot commented Aug 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nburns commented Aug 8, 2025

Uh oh!

orsenthil commented Aug 8, 2025

Uh oh!

Uh oh!

picnixz commented Aug 8, 2025

Uh oh!

picnixz commented Aug 8, 2025

Uh oh!

nburns commented Aug 8, 2025

Uh oh!

bedevere-bot commented Aug 8, 2025

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

nburns commented Aug 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

nburns commented Jan 2, 2024 •

edited

Loading

python-cla-bot bot commented Aug 8, 2025 •

edited

Loading

nburns commented Aug 8, 2025 •

edited

Loading