Skip to content

gh-100414: Add SQLite backend to dbm #114481

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 48 commits into from
Feb 14, 2024
Merged

Conversation

erlend-aasland
Copy link
Contributor

@erlend-aasland erlend-aasland commented Jan 23, 2024

@erlend-aasland
Copy link
Contributor Author

cc. @presidento

@erlend-aasland
Copy link
Contributor Author

I suggest to focus this PR on the basic, required functionality only. We can optimise things in follow-up PRs.

@erlend-aasland erlend-aasland marked this pull request as ready for review January 23, 2024 22:51
@erlend-aasland
Copy link
Contributor Author

I think the basic functionality is ready for an initial round of reviews.

Tests

There's a basic test suite in Lib/test_dbm.py, and I also added some dbm.sqlite3 specific tests in Lib/test_dbm_sqlite3.py. We can expand the latter in follow-up PRs.

Docs

I also added som basic docs; unless significant information is missing, I suggest we work on the docs in follow-up PRs.

@erlend-aasland
Copy link
Contributor Author

I made a core dev poll on Discourse regarding if the SQLite backend should be the default one; 16 out of 21 was in favour (Skip changed his mind post voting). This PR does not make the SQLite backend the default. If we want to change the default, we can do it post merging the feature itself. Another possibility, suggested by @ethanfurman:

Shouldn’t we make it the default now so it can be tested in the alphas? We can undo the default portion if it turns out to be a problem.

@serhiy-storchaka, should we land this?

@serhiy-storchaka
Copy link
Member

It seems that currently it is not compatible with other dbm implementations. They all are bytes oriented, but accept strings as input (it is a Python 2 legacy). Most of the dbm tests use string input, so I expect that many user code does it too. But the output always expected to be bytes.

Supporting other types, supported by Sqlite looks attractive, but we implement a backend to dbm, so it should be compatible. We can later add an option to alter the behavior, but it should not be default.

@erlend-aasland
Copy link
Contributor Author

It seems that currently it is not compatible with other dbm implementations. They all are bytes oriented, but accept strings as input (it is a Python 2 legacy). Most of the dbm tests use string input, so I expect that many user code does it too. But the output always expected to be bytes.

Supporting other types, supported by Sqlite looks attractive, but we implement a backend to dbm, so it should be compatible. We can later add an option to alter the behavior, but it should not be default.

As discussed in DM, I agree with this. I addressed your concern in e782fad.

Copy link
Member

@serhiy-storchaka serhiy-storchaka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM if all operations still accept strings.

# (raw, coerced)
(42, b"42"),
(3.14, b"3.14"),
("string", b"string"),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it work with non-ASCII strings?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mean such a test?

diff --git a/Lib/test/test_dbm_sqlite3.py b/Lib/test/test_dbm_sqlite3.py
index 7bc2a03035..3d2dc5fd55 100644
--- a/Lib/test/test_dbm_sqlite3.py
+++ b/Lib/test/test_dbm_sqlite3.py
@@ -214,6 +214,7 @@ class DataTypes(_SQLiteDbmTests):
         (3.14, b"3.14"),
         ("string", b"string"),
         (b"bytes", b"bytes"),
+        ("æøå", b"\xc3\xa6\xc3\xb8\xc3\xa5"),
     )
 
     def setUp(self):

@erlend-aasland
Copy link
Contributor Author

With 34930cb, the dbm.sqlite3 backend now aligns with the other backends; for example, the general code snippet in the dbm docs now works as expected with the new backend.

@erlend-aasland
Copy link
Contributor Author

Let's land this, @serhiy-storchaka. There is plenty of opportunities to tweak the implementation and to expand the test suite.

Thanks @smontanaro for the initial idea, @rhettinger for the initial implementation, and everyone else involved for ideas, reviews and critique. Thanks to @serhiy-storchaka for the, as always, extremely thorough review of the feature and the implementation.

@erlend-aasland erlend-aasland enabled auto-merge (squash) February 14, 2024 11:02
@erlend-aasland erlend-aasland merged commit dd5e4d9 into python:main Feb 14, 2024
@erlend-aasland erlend-aasland deleted the sqlite/dbm branch February 14, 2024 11:14
@bedevere-bot

This comment was marked as resolved.

@bedevere-bot

This comment was marked as duplicate.

@erlend-aasland
Copy link
Contributor Author

erlend-aasland commented Feb 14, 2024

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

[...]

Failed tests:

  • test_dbm_sqlite3

[...]
ModuleNotFoundError: No module named '_sqlite3'

Proposed fixes:

@bedevere-bot

This comment was marked as duplicate.

erlend-aasland added a commit to erlend-aasland/cpython that referenced this pull request Feb 14, 2024
@bedevere-bot

This comment was marked as resolved.

fsc-eriker pushed a commit to fsc-eriker/cpython that referenced this pull request Feb 14, 2024
Co-authored-by: Raymond Hettinger <rhettinger@users.noreply.github.com>
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
Co-authored-by: Mariusz Felisiak <felisiak.mariusz@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants