Skip to content

Possible pluggable C extension for performance #1066

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 15 commits into
base: main
Choose a base branch
from

Conversation

kesmit13
Copy link

@kesmit13 kesmit13 commented Sep 1, 2022

I'm creating this PR more as a start to a conversation as opposed to a request that I expect to be accepted any time soon. We like being able to use PyMySQL because of the ease of installation and the fact that it can be used without any system-level libraries being installed. However, the performance leaves a lot to be desired when dealing with larger result sets.

After some investigation, it looked like most of the time was being spent in the row data fetching and data conversion, which all stems from one method in the connection (`read_rowdata_packet'). An idea came to mind to replace that method (and anything that it calls) with a C extension to see how much performance could be increased. It ended up working much better than expected and the performance improvements actually made the client (arguably) the fastest MySQL Python client available (arguably because the Mariadb client is pretty fast too). It is definitely the fastest Python client that doesn't require any external libraries to run.

I thought I'd bring the work up to your team to see if you might want to collaborate on it as a possible plugin for PyMySQL, or to see if it should remain a separate project.

@methane methane marked this pull request as draft September 9, 2022 06:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant