bpo-4963: Fix for initialization and non-deterministic behavior issues in mimetypes #3062

davidkhess · 2017-08-10T14:00:06Z

Tracked in:

https://bugs.python.org/issue4963

In particular, here's my thoughts on it which led to this PR:

https://bugs.python.org/msg293626

https://bugs.python.org/issue4963

…avior.

brettcannon · 2017-08-11T21:13:37Z

Doc/library/mimetypes.rst

@@ -88,6 +88,10 @@ behavior of the module.
   Specifying an empty list for *files* will prevent the system defaults from
   being applied: only the well-known values will be present from a built-in list.

+   If *files* is not specified or is ``None`` the internal data structure is completely


Since files is already set to None you don't have to explicitly call out what happens if you don't specify an argument.

Bring in the new .json entry from master.

davidkhess · 2017-08-15T00:31:12Z

I see a new check. Should I write a news entry for this? Or will the committer?

brettcannon · 2017-08-15T16:15:38Z

@davidkhess It's easier for us if you create the news entry because if a core dev decides to merge this they can edit the file easily through GitHub's web UI while having to add a file is much more burdensome.

davidkhess · 2017-08-15T16:26:12Z

News entry added.

davidkhess · 2017-09-04T20:59:51Z

@bitdancer, anything I can do to help this along?

davidkhess · 2017-10-23T16:48:53Z

@bitdancer, anything I can do to help this along?

davidkhess · 2018-04-07T17:26:06Z

@brettcannon @bitdancer, anything I can do to help this along?

I'm willing to resolve the conflict but I don't want to waste my time if this PR is never going to be merged.

brettcannon · 2018-04-09T17:59:35Z

Unfortunately I don't know this module well enough to feel comfortable reviewing it. I would try pinging the issue to ask if someone could review it.

davidkhess · 2018-04-09T18:07:40Z

Pinged.

serhiy-storchaka · 2018-04-10T05:58:35Z

Lib/mimetypes.py

@@ -32,6 +32,8 @@
 except ImportError:
    _winreg = None

+from collections import OrderedDict


This no longer needed in 3.7+.

Should I remove all usage of OrderedDict from the PR and let others add it back for back porting it to 2.7? Or should I do something like set OrderedDict = dict if Python version >= 3.7?

The order of the items in the list is a critical part of the fix - I'd vote for retaining the explicit use of OrderedDict.

Adding OrderedDict makes this PR more complex, and the resulting code clumsy, and this is absolutely unneeded in 3.7+. I think it would be better to merge a simple fix into master and 3.7, and prepare more complex backport for 3.6.

Since 3.6 is only receiving security fixes, there's no need to worry about the backport to 3.6 anymore. So, you can remove the OrderedDict as this will only be in 3.7 and 3.8.

tiran

Please resolve the merge conflict

bedevere-bot · 2018-07-28T09:58:56Z

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

zooba · 2018-07-28T12:33:44Z

Lib/mimetypes.py

@@ -32,6 +32,8 @@
 except ImportError:
    _winreg = None

+from collections import OrderedDict


The order of the items in the list is a critical part of the fix - I'd vote for retaining the explicit use of OrderedDict.

zooba · 2018-07-28T12:36:13Z

The merge conflict does need to be fixed, and it looks like it'll be a bit of work to figure out exactly what needs merging.

csabella · 2019-05-21T12:05:20Z

@davidkhess, you can remove the OrderedDict as this point since this won't be applied to 3.6. Since this was close to being merged before, pending the discussion of the OrderedDict, if you fix the merge conflict and remove the OrderedDict, I think this will be able to be merged. Thank you for your contribution and for your patience. 🙂

auvipy · 2019-05-21T13:36:11Z

@davidkhess If you allow me to push on your branch I can help to get this merged

davidkhess · 2019-05-21T13:38:02Z

I'm making the requested changes now.

davidkhess · 2019-05-21T14:26:24Z

@auvipy @csabella good to go.

csabella · 2019-05-21T14:30:12Z

@serhiy-storchaka and @zooba, the merge conflicts have been resolved. I know this was approved last year, but I wanted to make sure you were still OK with it before merging. Thanks!

Merge conflict has been resolved

miss-islington · 2019-06-24T23:47:02Z

Thanks @davidkhess for the PR, and @zooba for merging it 🌮🎉.. I'm working now to backport this PR to: 3.8.
🐍🍒⛏🤖

miss-islington · 2019-06-24T23:47:07Z

Sorry @davidkhess and @zooba, I had trouble checking out the 3.8 backport branch.
Please backport using cherry_picker on command line.
cherry_picker 9fc720e5e4f772598013ea48a3f0d22b2b6b04fa 3.8

…s in mimetypes (pythonGH-3062)

bedevere-bot · 2019-06-25T14:37:52Z

GH-14375 is a backport of this pull request to the 3.8 branch.

…s in mimetypes (pythonGH-3062)

adeadman · 2022-06-01T14:36:57Z

This change has resulted in a difference in behaviour from previous versions of Python in that even when specifying files as a parameter to init(), the known files are loaded, which is not the case in previous versions. This causes problems when an application wishes to use the mimetypes database with an internal-only mapping of mimetypes, to work around inconsistencies with mimetypes in the known files list (which can vary between systems).

Unfortunately this change wasn't documented in the changelog either, nor in the docstring comments, so it can only be found by examining the commit history for the code in cpython.

Would it be appropriate to add another kwarg to init() indicating that knownfiles are to be ignored/excluded from the DB initialization?

pombredanne · 2022-06-01T15:02:43Z

@adeadman posted a message inline as a reply to mine at 9fc720e#r45794801

In essence this fixe was supposed to bring in a deterministic behaviour and actually had the complete opposite effect. I still consider this a regression. I stopped using mimetypes for now as it is useless as it is now.

tiran · 2022-06-01T15:08:00Z

Could you please open a new bug report? Comments on closed PRs tend to get lost.

pombredanne · 2022-06-01T16:30:51Z

@adeadman @tiran let me create a new issue #93417

davidkhess added 3 commits May 13, 2017 17:30

Addresses mimetypes initialization problems and non-deterministic beh…

ecabb1c

…avior.

Fix edge case with init.

0e8c48f

Fix another edge case.

10abe19

davidkhess requested a review from bitdancer as a code owner August 10, 2017 14:00

the-knights-who-say-ni added the CLA signed label Aug 10, 2017

davidkhess added 3 commits August 10, 2017 09:10

Fix whitespace issue with added test.

2486b92

Capture the new init() behavior in the documentation.

384178c

Merge remote-tracking branch 'python/master' into fix-issue-4963

32cd666

brettcannon reviewed Aug 11, 2017

View reviewed changes

davidkhess added 2 commits August 11, 2017 16:51

Tweak to the documentation per review feedback.

d2097be

Merge remote-tracking branch 'python/master' into fix-issue-4963

1f45e83

Bring in the new .json entry from master.

brettcannon added the awaiting review label Aug 15, 2017

Add news entry.

f9e59f9

serhiy-storchaka reviewed Apr 10, 2018

View reviewed changes

tiran previously requested changes Jul 28, 2018

View reviewed changes

bedevere-bot added awaiting changes and removed awaiting review labels Jul 28, 2018

zooba approved these changes Jul 28, 2018

View reviewed changes

bedevere-bot added awaiting merge and removed awaiting changes labels Jul 28, 2018

davidkhess added 2 commits May 21, 2019 08:46

Remove use of OrderedDict

59d1128

Merge remote-tracking branch 'python/master' into fix-issue-4963

799ce54

davidkhess requested a review from a team as a code owner May 21, 2019 14:07

csabella requested review from serhiy-storchaka and zooba May 21, 2019 14:28

auvipy approved these changes May 21, 2019

View reviewed changes

zooba added the needs backport to 3.8 label Jun 24, 2019

zooba merged commit 9fc720e into python:master Jun 24, 2019

bedevere-bot removed the awaiting merge label Jun 24, 2019

miss-islington assigned zooba Jun 24, 2019

zooba pushed a commit to zooba/cpython that referenced this pull request Jun 25, 2019

bpo-4963: Fix for initialization and non-deterministic behavior issue…

71ae6f1

…s in mimetypes (pythonGH-3062)

zooba pushed a commit to zooba/cpython that referenced this pull request Jun 25, 2019

bpo-4963: Fix for initialization and non-deterministic behavior issue…

0dd9791

…s in mimetypes (pythonGH-3062)

bedevere-bot removed the needs backport to 3.8 label Jun 25, 2019

lisroach pushed a commit to lisroach/cpython that referenced this pull request Sep 10, 2019

bpo-4963: Fix for initialization and non-deterministic behavior issue…

7d3e52d

…s in mimetypes (pythonGH-3062)

DinoV pushed a commit to DinoV/cpython that referenced this pull request Jan 14, 2020

bpo-4963: Fix for initialization and non-deterministic behavior issue…

df8c848

…s in mimetypes (pythonGH-3062)

pombredanne mentioned this pull request Jun 1, 2022

Regression on initialization and new non-deterministic behavior issue in mimetypes #93417

Open

Uh oh!

bpo-4963: Fix for initialization and non-deterministic behavior issues in mimetypes #3062

bpo-4963: Fix for initialization and non-deterministic behavior issues in mimetypes #3062

Uh oh!

Conversation

davidkhess commented Aug 10, 2017 • edited by bedevere-bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

brettcannon Aug 11, 2017

Choose a reason for hiding this comment

Uh oh!

davidkhess commented Aug 15, 2017

Uh oh!

brettcannon commented Aug 15, 2017

Uh oh!

davidkhess commented Aug 15, 2017

Uh oh!

davidkhess commented Sep 4, 2017

Uh oh!

davidkhess commented Oct 23, 2017

Uh oh!

davidkhess commented Apr 7, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

brettcannon commented Apr 9, 2018

Uh oh!

davidkhess commented Apr 9, 2018

Uh oh!

serhiy-storchaka Apr 10, 2018

Choose a reason for hiding this comment

Uh oh!

davidkhess Apr 10, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zooba Jul 28, 2018

Choose a reason for hiding this comment

Uh oh!

serhiy-storchaka Jul 28, 2018

Choose a reason for hiding this comment

Uh oh!

csabella May 21, 2019

Choose a reason for hiding this comment

Uh oh!

tiran left a comment

Choose a reason for hiding this comment

Uh oh!

bedevere-bot commented Jul 28, 2018

Uh oh!

zooba Jul 28, 2018

Choose a reason for hiding this comment

Uh oh!

zooba commented Jul 28, 2018

Uh oh!

csabella commented May 21, 2019

Uh oh!

auvipy commented May 21, 2019

Uh oh!

davidkhess commented May 21, 2019

Uh oh!

davidkhess commented May 21, 2019

Uh oh!

csabella commented May 21, 2019

Uh oh!

miss-islington commented Jun 24, 2019

Uh oh!

miss-islington commented Jun 24, 2019

Uh oh!

bedevere-bot commented Jun 25, 2019

Uh oh!

adeadman commented Jun 1, 2022

Uh oh!

pombredanne commented Jun 1, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tiran commented Jun 1, 2022

Uh oh!

pombredanne commented Jun 1, 2022

Uh oh!

Uh oh!

davidkhess commented Aug 10, 2017 •

edited by bedevere-bot

Loading

davidkhess commented Apr 7, 2018 •

edited

Loading

davidkhess Apr 10, 2018 •

edited

Loading

pombredanne commented Jun 1, 2022 •

edited

Loading