Skip to content

bpo-37476: Adding a unit test of unicode in test_unicode.py #14531

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Jul 20, 2019

Conversation

shihai1991
Copy link
Member

@shihai1991 shihai1991 commented Jul 1, 2019

@mangrisano
Copy link
Contributor


self.assertEqual(unicode_asutf8('abc'), 'abc')
self.assertEqual(unicode_asutf8('abc\0'), 'abc')
self.assertEqual(unicode_asutf8('abc\0abc'), 'abc')
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's more like you are testing decodeUTF8. I'd suggest add more encode samples, BMP, non-BMP, just take from other tests.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

np, xiang core.

Copy link
Member

@vstinner vstinner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you add tests, please test also PyUnicode_AsUTF8AndSize() which would allow embedded NUL characters/bytes.

def test_asutf8(self):
from _testcapi import unicode_asutf8

self.assertEqual(unicode_asutf8('abc'), 'abc')
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The function encodes to UTF-8, so the result should be a bytes string, not Unicode. Use PyString_FromString() rather than PyUnicode_FromString() in _testcapi.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The others similar C API tests also return unicodes other than bytes which in some sense also tests the counter part decoding function, though I don't very understand why designed like this.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Either fix other tests, or leave them unchanged. But I would prefer to not add new buggy tests :-)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So returning the decoded bytes string is right way? I am not sure I understand it clearly.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Return bytes, not unicodes. Test decode functionality separately.

@shihai1991
Copy link
Member Author

If you add tests, please test also PyUnicode_AsUTF8AndSize() which would allow embedded NUL characters/bytes.

np, I would continue to add test.

@tiran
Copy link
Member

tiran commented Jul 2, 2019

FYI, I cancelled the Travis CI run to free some resources for upcoming 3.8b2 and 3.7.4rc2 releases.

@shihai1991 shihai1991 changed the title bpo-37476: Adding a unit test of unicode in test_unicode.py [WIP]bpo-37476: Adding a unit test of unicode in test_unicode.py Jul 3, 2019
@shihai1991 shihai1991 changed the title [WIP]bpo-37476: Adding a unit test of unicode in test_unicode.py bpo-37476: Adding a unit test of unicode in test_unicode.py Jul 4, 2019
@zhangyangyu zhangyangyu added the tests Tests in the Lib/test dir label Jul 8, 2019
@shihai1991
Copy link
Member Author

@vstinner hi, victor. Pls help me review this patch again, thanks ;)

@zhangyangyu zhangyangyu merged commit 5623ac8 into python:master Jul 20, 2019
lisroach pushed a commit to lisroach/cpython that referenced this pull request Sep 10, 2019
DinoV pushed a commit to DinoV/cpython that referenced this pull request Jan 14, 2020
websurfer5 pushed a commit to websurfer5/cpython that referenced this pull request Jul 20, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
skip news tests Tests in the Lib/test dir
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants