Skip to content

Add JSON field type and include it in TEXT_TYPES list to prevent encoding problems #477

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jun 24, 2016

Conversation

bmamouri
Copy link
Contributor

Note: This issue is not related to the following pull request: https://github.com/PyMySQL/PyMySQL/pull/428/commits

MySQL 5.7 added JSON types. In _get_descriptions method we are setting the encoding of any field type that is not textual to 'ascii'. This is clearly wrong as the JSON field types are actually BINARY column that contains strings.

Because of wrong encoding problem, in _read_row_from_packet function we encode the JSON field type to 'ascii'. Here is the related code:

    def _read_row_from_packet(self, packet):
        row = []
        for encoding, converter in self.converters:
            try:
                data = packet.read_length_coded_string()
            except IndexError:
                # No more columns in this row
                # See https://github.com/PyMySQL/PyMySQL/pull/434
                break
            if data is not None:
                if encoding is not None:  # Since encoding is set 'ascii' the content is encoding to 'ascii'
                    data = data.decode(encoding)
                if DEBUG: print("DEBUG: DATA = ", data)
                if converter is not None:
                    data = converter(data)
            row.append(data)
        return tuple(row)

Because of this problem, querying JSON field types that contain Unicode characters will fail.

The solution to the problem is relatively easy. Add the JSON field types to TEXT_TYPES.

…ing the charset of JSON field types to 'ascii' and treat JSON field type like BINARY types to prevent encoding issue
@coveralls
Copy link

coveralls commented Jun 24, 2016

Coverage Status

Coverage decreased (-2.5%) to 85.926% when pulling 13c44c4 on bmamouri:master into de3b658 on PyMySQL:master.

@coveralls
Copy link

coveralls commented Jun 24, 2016

Coverage Status

Coverage decreased (-0.1%) to 88.332% when pulling 13c44c4 on bmamouri:master into de3b658 on PyMySQL:master.

@methane methane merged commit f2e79cd into PyMySQL:master Jun 24, 2016
@methane
Copy link
Member

methane commented Jun 24, 2016

Thanks!

@methane
Copy link
Member

methane commented Jul 4, 2016

I'm sorry, but I revert this.

You should use encoding="utf8mb4" or use_unicode=False to avoid this issue.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Aug 3, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants