PEP 277: Unicode file name support #37017

loewis · 2002-08-12T11:33:37Z

BPO	594001
Nosy	@gvanrossum, @tim-one, @loewis, @mhammond
Files	pep277.txt pep277.txt: Version 2 unicode_comments.txt: Comments on the patch pep277-2.txt: Better version of that patch that doesn't crash anything! doc.txt

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/mhammond'
closed_at = <Date 2002-10-05.09:47:07.000>
created_at = <Date 2002-08-12.11:33:37.000>
labels = ['OS-windows']
title = 'PEP 277: Unicode file name support'
updated_at = <Date 2002-10-05.09:47:07.000>
user = 'https://github.com/loewis'

bugs.python.org fields:

activity = <Date 2002-10-05.09:47:07.000>
actor = 'loewis'
assignee = 'mhammond'
closed = True
closed_date = None
closer = None
components = ['Windows']
creation = <Date 2002-08-12.11:33:37.000>
creator = 'loewis'
dependencies = []
files = ['4506', '4507', '4508', '4509', '4510']
hgrepos = []
issue_num = 594001
keywords = ['patch']
message_count = 25.0
messages = ['40910', '40911', '40912', '40913', '40914', '40915', '40916', '40917', '40918', '40919', '40920', '40921', '40922', '40923', '40924', '40925', '40926', '40927', '40928', '40929', '40930', '40931', '40932', '40933', '40934']
nosy_count = 5.0
nosy_names = ['gvanrossum', 'tim.peters', 'loewis', 'mhammond', 'nnorwitz']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue594001'
versions = []

loewis · 2002-08-12T11:33:37Z

This patch is in an updated version of the patch [1]
mentioned in the PEP. In addition to merging it with
the CVS, it fixes a few formatting problems.

gvanrossum · 2002-08-12T13:38:09Z

Logged In: YES
user_id=6380

I'd make unicode_filenames() a macro that expands to 0 on
platforms without this wart. I'd also test for wfunc!=NULL
before calling unicode_filenames().

There's a lot of hairy code here. Are you sure that there
are test cases in the test suite that exercise all of it?

Aren't there some #ifdefs missing? posix_[12]str have code
that's only relevant for Windows but isn't #ifdef'ed out
like it is elsewhere.

There should probably be a separate #define in pyport.h to
test for this that's equivalent to defined(MS_WINDOWS) &&
!defined(Py_UNICODE_WIDE), so this can be uniformly tested
to see whether this code is necessary.

loewis · 2002-09-04T12:06:52Z

Logged In: YES
user_id=21627

This is a new version:

wrapped all Windows wide API code with #ifdef
Py_WIN_WIDE_FILENAMES
added test case

Mark, can you please take a look at this?

loewis · 2002-09-22T15:31:11Z

Logged In: YES
user_id=21627

Tim, can you review this?

tim-one · 2002-09-25T19:43:24Z

Logged In: YES
user_id=31435

Sorry, Martin, bouncing back to Mark -- I really don't know
anything about the Windows filename APIs.

loewis · 2002-09-26T06:24:42Z

Logged In: YES
user_id=21627

Mark, can you please indicate whether you will review this
patch?

mhammond · 2002-09-30T01:03:52Z

Logged In: YES
user_id=14198

Will review this instant - sorry for the delay.

mhammond · 2002-09-30T06:33:51Z

Logged In: YES
user_id=14198

Uploading text file with my comments on the patch. Also
uploading a new patch with these corrections and even better
(?) error handling.

mhammond · 2002-09-30T06:34:56Z

Logged In: YES
user_id=14198

Back to Martin for comments on my comments ;)

mhammond · 2002-09-30T06:35:47Z

Logged In: YES
user_id=14198

Back to martin for comments on my comments ;)

loewis · 2002-09-30T10:43:00Z

Logged In: YES
user_id=21627

Thanks for your comments, just shouting at me to correct all
these problems would have been fine :-)

In what case do you still get an exception?

mhammond · 2002-09-30T12:20:25Z

Logged In: YES
user_id=14198

I get an exception running test_unicode_file.py - we end up
with a MBCS encoded string passed to
PyUnicode_FromEncodedObject(), that attempts to decode using
the system default encoding rather than the file-system
default. The error is a NULL pointer deref so should be
obvious - unfortunately the fix requires a local
reimplementation of PyUnicode_FromEncodedObject(), passing
an explicit encoding for string objects.

mhammond · 2002-09-30T12:21:56Z

Logged In: YES
user_id=14198

I missed the fact this was re-assigned to me - should I
check my new patch in?

mhammond · 2002-09-30T12:52:02Z

Logged In: YES
user_id=14198

Sorry - not sure what I was thinking - obviously can't check
in a patch known to crash the test suite!

Attaching a new patch. This fixes the crash in
test_unicode_file I mentioned. It also fixes another error
I introduced where PyErr_SetWithFilename functions can have
NULL filenames. Also added an assert for symmetry with one
I added before.

Martin - back to me if you want me to check it in.

mhammond · 2002-09-30T12:54:44Z

Logged In: YES
user_id=14198

Sorry - not sure what I was thinking - obviously can't check
in a patch known to crash the test suite!

Attaching a new patch - fixes the problem I mentioned with
test_unicode_files.py, fixes an error I introduced where the
"WithFilename" error functions could be passed a NULL
filename, and adding an assert to match the one I added
previously.

mhammond · 2002-09-30T23:48:33Z

Logged In: YES
user_id=14198

Although SF implies otherwise, a bad filename in a
submission appears to only prevent the file being uploaded -
the rest of the comments etc come through. Hence my
double-comments and strange status changes you can see.

So, back to Martin as I don't think he is waiting on me for
anything. Happy to check in if you like, but would like to
see someone run a debug build of this through Linux first to
see if any asserts fire - some asserts did indeed fire in
the first version of this patch, and Linux may trigger
others, or even the 2 that I added.

mhammond · 2002-10-01T09:59:47Z

Logged In: YES
user_id=14198

The block:

+ if (!wpath1 || !wpath2)
+ return NULL;

Should include Py_XDECREF calls for wpath1 and wpath2.

Can't be bothered with a new patch just for this :)

loewis · 2002-10-01T17:41:20Z

Logged In: YES
user_id=21627

The patch passes the test suite on Solaris. I've also
attached a documentation change, so I think the patch is
ready now. Back to Mark for checkin.

mhammond · 2002-10-03T05:15:36Z

Logged In: YES
user_id=14198

/cvsroot/python/python/dist/src/Include/pyerrors.h,v <--
pyerrors.h
new revision: 2.61; previous revision: 2.60
/cvsroot/python/python/dist/src/Lib/test/test_unicode_file.py,v
<-- test_unicode_file.py
new revision: 1.6; previous revision: 1.5
/cvsroot/python/python/dist/src/Modules/posixmodule.c,v <--
posixmodule.c
new revision: 2.261; previous revision: 2.260
/cvsroot/python/python/dist/src/Objects/fileobject.c,v <--
fileobject.c
new revision: 2.169; previous revision: 2.168
/cvsroot/python/python/dist/src/PC/pyconfig.h,v <-- pyconfig.h
new revision: 1.15; previous revision: 1.14
/cvsroot/python/python/dist/src/Python/errors.c,v <-- errors.c
new revision: 2.72; previous revision: 2.71

mhammond · 2002-10-03T07:00:55Z

Logged In: YES
user_id=14198

Oops - I missed checking in Lib/test/test_pep277.py.

With this file in place, the test suite fails for me:
test_pep277
test test_pep277 produced unexpected output:
...

However, I seem to recall that new tests do not need output
files. Should I use the old technique of "regrtest -g ..."
and check both the test and output file in, or is there
something else I should be doing?

loewis · 2002-10-03T08:08:58Z

Logged In: YES
user_id=21627

Generating the output file with -g and committing it is the
right thing to do. I noticed that a .sort call is missing
parenthesis, which would need to be added.

mhammond · 2002-10-03T23:15:47Z

Logged In: YES
user_id=14198

/cvsroot/python/python/dist/src/Lib/test/test_pep277.py,v
<-- test_pep277.py
initial revision: 1.1
RCS file:
/cvsroot/python/python/dist/src/Lib/test/output/test_pep277,v
/cvsroot/python/python/dist/src/Lib/test/output/test_pep277,v
<-- test_pep277
initial revision: 1.1

nnorwitz · 2002-10-03T23:43:27Z

Logged In: YES
user_id=33168

Mark, I had a bunch of code review notes I sent to
python-checkins. I'm not sure you saw them, since my mail
to you @ users.sf.net apparently doesn't want to go out.

mhammond · 2002-10-03T23:53:10Z

Logged In: YES
user_id=14198

I will try and fix my sf.net mail - in tne meantime Neal, if
you could forward them to my skippinet address I will
address them.

loewis · 2002-10-05T09:47:07Z

Logged In: YES
user_id=21627

/cvsroot/python/python/dist/src/Doc/lib/libos.tex,v <--
libos.tex
new revision: 1.97; previous revision: 1.96
done
Checking in Misc/NEWS;
/cvsroot/python/python/dist/src/Misc/NEWS,v <-- NEWS
new revision: 1.493; previous revision: 1.492

loewis mannequin closed this as completed Aug 12, 2002

loewis mannequin assigned mhammond Aug 12, 2002

loewis mannequin added the OS-windows label Aug 12, 2002

loewis mannequin closed this as completed Aug 12, 2002

loewis mannequin assigned mhammond Aug 12, 2002

loewis mannequin added the OS-windows label Aug 12, 2002

ezio-melotti transferred this issue from another repository Apr 9, 2022

Uh oh!

PEP 277: Unicode file name support #37017

PEP 277: Unicode file name support #37017

Comments

loewis mannequin commented Aug 12, 2002

loewis mannequin commented Aug 12, 2002

Uh oh!

gvanrossum commented Aug 12, 2002

Uh oh!

loewis mannequin commented Sep 4, 2002

Uh oh!

loewis mannequin commented Sep 22, 2002

Uh oh!

tim-one commented Sep 25, 2002

Uh oh!

loewis mannequin commented Sep 26, 2002

Uh oh!

mhammond commented Sep 30, 2002

Uh oh!

mhammond commented Sep 30, 2002

Uh oh!

mhammond commented Sep 30, 2002

Uh oh!

mhammond commented Sep 30, 2002

Uh oh!

loewis mannequin commented Sep 30, 2002

Uh oh!

mhammond commented Sep 30, 2002

Uh oh!

mhammond commented Sep 30, 2002

Uh oh!

mhammond commented Sep 30, 2002

Uh oh!

mhammond commented Sep 30, 2002

Uh oh!

mhammond commented Sep 30, 2002

Uh oh!

mhammond commented Oct 1, 2002

Uh oh!

loewis mannequin commented Oct 1, 2002

Uh oh!

mhammond commented Oct 3, 2002

Uh oh!

mhammond commented Oct 3, 2002

Uh oh!

loewis mannequin commented Oct 3, 2002

Uh oh!

mhammond commented Oct 3, 2002

Uh oh!

nnorwitz mannequin commented Oct 3, 2002

Uh oh!

mhammond commented Oct 3, 2002

Uh oh!

loewis mannequin commented Oct 5, 2002

Uh oh!