deps: Drop support for Python 3.7 and 3.8 #337

chalmerlowe · 2025-04-17T19:49:39Z

This branch is generated automagically via AI as an experiment.
Requires a comprehensive human review and comment.
Do not merge until approved by an authorized reviewer.

Removes support for Python 3.7 and 3.8, establishing Python 3.9 as the new minimum supported version.

This change involves:

Updating python_requires and classifiers in setup.py.
Modifying Python versions in noxfile.py (default, unit tests, system tests) and ensuring constraint file logic remains correct.
Updating the GitHub Actions workflow (unittest.yml) matrix, runner, and coverage job version.
Deleting constraint files for Python 3.7 and 3.8 (testing/constraints-3.7.txt, testing/constraints-3.8.txt).
Removing Kokoro sample configuration directories (.kokoro/samples/python3.7/, .kokoro/samples/python3.8/).
Updating supported version mentions in README.rst.
Removing 3.7 and 3.8 from the ALL_VERSIONS list in samples/snippets/noxfile.py.

BEGIN_COMMIT_OVERRIDE
deps: Drop support for Python 3.7 and 3.8
END_COMMIT_OVERRIDE

Removes support for Python 3.7 and 3.8, establishing Python 3.9 as the new minimum supported version. This change involves: - Updating `python_requires` and classifiers in `setup.py`. - Modifying Python versions in `noxfile.py` (default, unit tests, system tests) and ensuring constraint file logic remains correct. - Updating the GitHub Actions workflow (`unittest.yml`) matrix, runner, and coverage job version. - Deleting constraint files for Python 3.7 and 3.8 (`testing/constraints-3.7.txt`, `testing/constraints-3.8.txt`). - Removing Kokoro sample configuration directories (`.kokoro/samples/python3.7/`, `.kokoro/samples/python3.8/`). - Updating supported version mentions in `README.rst`. - Removing 3.7 and 3.8 from the `ALL_VERSIONS` list in `samples/snippets/noxfile.py`.

Linchin · 2025-04-23T17:06:42Z

Just a suggestion, maybe use deps!: so this commit shows up in release notes? (I'm not 100% sure if a breaking change "chore" will appear in the release, though)

chalmerlowe · 2025-04-23T21:13:14Z

Just a suggestion, maybe use deps!: so this commit shows up in release notes? (I'm not 100% sure if a breaking change "chore" will appear in the release, though)

Conventional Commit points at this page as reference for prefixes:

'build',
'chore',
'ci',
'docs',
'feat',
'fix',
'perf',
'refactor',
'revert',
'style',
'test'

If Google also uses deps, then sure.

Linchin · 2025-04-23T21:48:19Z

Wow, I suggested it because we used it in the commit for python-bigquery. Apparently it's not part of convetional commits, but it worked.

Edit:
It seems conventional commits doesn't enforce a whitelist of commit types

tswast · 2025-05-05T18:55:07Z

There is a class in pandas_backports.py that did not have tests (and which fact coverage ignored apparently).

The purpose of that class is to support older versions of pandas (and numpy, I believe). If it's not covered, it's likely because we aren't testing / supporting such versions anymore. Please delete dead code rather than mark as no cover.

tswast · 2025-05-05T19:01:34Z

db_dtypes/pandas_backports.py

+@import_default("pandas.core.arrays._mixins")
+class NDArrayBackedExtensionArray(
+    pandas.core.arrays.base.ExtensionArray
+):  # pragma: NO COVER


If these isn't used, then we should delete it. Per previous:

@import_default("pandas.core.arrays._mixins", pandas_release < (1, 3))

This was only used in pandas 1.2 tests, so it's not used anymore that the pandas minimum version is bumped to 1.5.3.

@tswast

I am struggling with this. I would like to suggest that this specific issue is tangential to the scope of this PR and should be handled after this PR is merged as part of a new refactoring PR.

Regarding the comment: "If it's not covered, it's likely because we aren't testing / supporting such versions anymore"

It appears that we never had tests for this class, so the lack of tests seems to be unrelated. I do not know why coverage did not flag this, but it flags it now, hence the pragma as a temporary solution to allow the work to proceed.

I know it was marked as < pandas 1.3 in the preceding comment.

In those comments it was also noted it could be dropped when NDArrayBackedExtension is added to pandas (presumably in 1.5/2.0) but that doesn't appear to have happened.

We use end up referring to this class as a dependency via subclassing in
in BaseDateTimeArray. BaseDateTimeArray is subclassed in TimeArray and DateArray. BaseDateTimeArray thus carries with it a number of attributes and methods from NDArrayBackedExtension that are relied upon across a plethora of tests throughout our test suite in multiple files (including unit, compliance, and prerelease tests)

If I remove this class and attempt to subclass BaseDateTimeArray on something like pandas.api.extensions.ExtensionArray, ExtensionArray does not have all the attributes and methods that are included in NDArrayBackedExtension and so assorted tests continue to fail.

Even if I replace those specific missing attributes and methods, that does not solve all the problems because ExtensionArray is somehow different and we still have other tests that also fail (I stopped there in my attempts to resolve this comment).

It feels like a catch-22, if I take this out we end up going down a rabbit hole to try and figure all the nuances of why this breaks so many things. That task is tangential to the scope of this PR which is to remove the dependencies on 3.7 and 3.8. It will also likely greatly increase the size and complexity of this PR.

If I try to create a separate PR to just refactor this portion of the code before we deprecate 3.7 and 3.8, I will likely have to do at least some deprecation of 3.7 and 3.8 in the new PR to ensure that we are comparing apples to apples and capturing all the failure modes, which means we end up duplicating a subset of the work from this PR.

I know it was marked as < pandas 1.3 in the preceding comment.

Right. It was only used in such environments. Now such an environment should not be possible. We can remove these backports since they aren't needed with the bump in mimumum versions.

In those comments it was also noted it could be dropped when NDArrayBackedExtension is added to pandas (presumably in 1.5/2.0) but that doesn't appear to have happened.

It is there, it's just not part of the public interface. It would be good to revisit that but at this point the interface is quite stable and we've mostly moved on to pyarrow-based extension arrays instead.

methods that are included in NDArrayBackedExtension and so assorted tests continue to fail.

Why would you change the superclass? The backport thing was just to move some code around. Just use the real class directly.

If I try to create a separate PR to just refactor this portion of the code before we deprecate 3.7 and 3.8,

Just delete the unused code. It's not a refactor.

I deleted the unused code and adjusted the signature for BaseDateTimeArray class to use the pandas.core.array._mixins.NDArrayBackedExtensionArray directly.

Linchin · 2025-05-05T22:14:25Z

db_dtypes/__init__.py

@@ -47,15 +46,6 @@
 _NP_BOX_DTYPE = "datetime64[us]"


-# To use JSONArray and JSONDtype, you'll need Pandas 1.5.0 or later. With the removal
-# of Python 3.7 compatibility, the minimum Pandas version will be updated to 1.5.0.
-if packaging.version.Version(pandas.__version__) >= packaging.version.Version("1.5.0"):


From the comments, it seems removing Python 3.7 could ensure pandas >= 1.5.0, and thus ensure JSONArray and JSONDtype to be available. Do we still need _determine_all()?

Linchin · 2025-05-05T22:21:55Z

tests/unit/test__init__.py

+    assert "JSONDtype" in result
+    assert "JSONArray" in result
+    assert "JSONArrowType" in result


Nit: these don't seem necessary if we already verified that the sets are equal.

Linchin · 2025-05-05T22:45:00Z

tests/unit/test_json.py

+
+    import db_dtypes.json  # noqa: F401
+
+    assert True, "Module re-import completed without error, except block likely worked."


I think the string after the comma is part of the raised error if the assertion failed, but the wording describes the expected behavior.

Linchin · 2025-05-05T22:49:29Z

tests/unit/test_json.py

+
+
+@pytest.fixture
+def cleanup_json_module_for_reload():


I understand that this fixture makes sure db_dtypes.json is removed before any unit test that might want to import it with a clean slate, and we have added two unit tests verifying that it works. But I don't see this fixture being used in any tests. Should we add it to where we intend to use it?

If we really use this fixture, we probably can get rid of all the # pragma: NO COVER inside.

tswast · 2025-05-07T14:12:52Z

db_dtypes/pandas_backports.py

+# TODO: use public API once NDArrayBackedExtensionArray is added to the
+# pandas.


It is in pandas. That's what import_default is doing. It's importing from pandas.core.arrays._mixins. We don't need this class anymore. Just use pandas.core.arrays._mixins.NDArrayBackedExtensionArray directly.

https://github.com/pandas-dev/pandas/blob/7595ed503df845f3bdf4d26b9beea5ec031b619d/pandas/core/arrays/_mixins.py#L93

Used the suggested class directly.

tswast · 2025-05-07T14:16:27Z

db_dtypes/pandas_backports.py

+# Also: Right now none of this is tested in test_pandas_backports.
+# Temporarily marking this as # pragma: NO COVER just to see how this
+# affects unittest coverage
+@import_default("pandas.core.arrays._mixins")
+class NDArrayBackedExtensionArray(
+    pandas.core.arrays.base.ExtensionArray
+):  # pragma: NO COVER


If we aren't changing the imports in other files, just add the class here like this:

Suggested change

# Also: Right now none of this is tested in test_pandas_backports.

# Temporarily marking this as # pragma: NO COVER just to see how this

# affects unittest coverage

@import_default("pandas.core.arrays._mixins")

class NDArrayBackedExtensionArray(

pandas.core.arrays.base.ExtensionArray

): # pragma: NO COVER

NDArrayBackedExtensionArray = pandas.core.arrays._mixins.NDArrayBackedExtensionArray

We only referenced NDArrayBackedExtensionArray in a single class: BaseDateTimeArray.
I updated that class signature directly.

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

noxfile.py

chalmerlowe added the do not merge Indicates a pull request not ready for merge, due to either quality or timing. label Apr 17, 2025

chalmerlowe self-assigned this Apr 17, 2025

chalmerlowe requested review from a team as code owners April 17, 2025 19:49

chalmerlowe requested review from gericdong and Linchin April 17, 2025 19:49

product-auto-label bot added size: m Pull request size is medium. api: bigquery Issues related to the googleapis/python-db-dtypes-pandas API. samples Issues that are directly related to samples. labels Apr 17, 2025

chalmerlowe added 6 commits April 18, 2025 09:46

Updates python version in lint.yml

dccef48

Updates owlbot, removing reference to 3.8

f81c738

Updates CONTRIBUTING.rst

5cb30ca

updates pytest warnings

7f44424

Removes test_samples-impl ref to older virtualenv package

3e4e6f6

Removes references to pandas older than 1.5.0

7beea0d

product-auto-label bot added size: l Pull request size is large. and removed size: m Pull request size is medium. labels Apr 18, 2025

chalmerlowe added 6 commits April 18, 2025 12:42

Removes pandas older than 1.5 and misc changes

32d7b8f

updates pandas in setup.py

e871d73

more updates related to pandas

2bdf3ef

still broken

d902e31

Updates FutureWarning tests to account for unittest coverage

a0543b9

Updates json array type tests to account for unittest coverage

6a5d943

updates python version checks to ensure coverage

1bbd5f7

chalmerlowe changed the title ~~chore!: Drop support for Python 3.7 and 3.8 (AI Experiment)~~ deps!: Drop support for Python 3.7 and 3.8 (AI Experiment) Apr 23, 2025

tswast reviewed May 5, 2025

View reviewed changes

Linchin reviewed May 5, 2025

View reviewed changes

removes determine all and module reload tests

4c522b3

tswast reviewed May 7, 2025

View reviewed changes

chalmerlowe and others added 14 commits May 7, 2025 15:05

updates re: ndarrybackedextensionarray

cbfcd1a

testing blacken as part of owlbot processing using 3.8

2f9fc09

🦉 Updates from OwlBot post-processor

7b967e7

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

updates blacken to 3.10

18098f0

update python version in lint.yml

20e6565

🦉 Updates from OwlBot post-processor

f39b97f

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

updates owlbot.py

c23d435

🦉 Updates from OwlBot post-processor

a52c46d

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

updates owlbot.py

4ebe910

testing lint.yml

cd4942e

testing linting issue

32245de

testing linting issue

695c33d

🦉 Updates from OwlBot post-processor

9ea13d3

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

updates lint_setup session

2ee0533

chalmerlowe commented May 7, 2025

View reviewed changes

noxfile.py Outdated Show resolved Hide resolved

Update noxfile.py

f59ee72

tswast approved these changes May 7, 2025

View reviewed changes

chalmerlowe merged commit 66f3f0b into main May 7, 2025
24 checks passed

chalmerlowe deleted the remove-python-37-38 branch May 7, 2025 19:57

release-please bot mentioned this pull request May 7, 2025

chore(main): release 1.4.3 #332

Merged

chalmerlowe changed the title ~~deps!: Drop support for Python 3.7 and 3.8 (AI Experiment)~~ deps: Drop support for Python 3.7 and 3.8 May 12, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

deps: Drop support for Python 3.7 and 3.8 #337

deps: Drop support for Python 3.7 and 3.8 #337

chalmerlowe commented Apr 17, 2025 •

edited

Loading

Linchin commented Apr 23, 2025

chalmerlowe commented Apr 23, 2025

Linchin commented Apr 23, 2025 •

edited

Loading

tswast commented May 5, 2025 •

edited

Loading

tswast May 5, 2025

chalmerlowe May 7, 2025 •

edited

Loading

tswast May 7, 2025

chalmerlowe May 7, 2025

Linchin May 5, 2025 •

edited

Loading

chalmerlowe May 7, 2025

Linchin May 5, 2025

chalmerlowe May 7, 2025

Linchin May 5, 2025

chalmerlowe May 7, 2025

Linchin May 5, 2025

Linchin May 5, 2025 •

edited

Loading

chalmerlowe May 7, 2025

tswast May 7, 2025

chalmerlowe May 7, 2025

tswast May 7, 2025

chalmerlowe May 7, 2025


		import db_dtypes.json # noqa: F401

		assert True, "Module re-import completed without error, except block likely worked."

		# TODO: use public API once NDArrayBackedExtensionArray is added to the
		# pandas.



		@pytest.fixture
		def cleanup_json_module_for_reload():

deps: Drop support for Python 3.7 and 3.8 #337

deps: Drop support for Python 3.7 and 3.8 #337

Conversation

chalmerlowe commented Apr 17, 2025 • edited Loading

Linchin commented Apr 23, 2025

chalmerlowe commented Apr 23, 2025

Linchin commented Apr 23, 2025 • edited Loading

tswast commented May 5, 2025 • edited Loading

Choose a reason for hiding this comment

chalmerlowe May 7, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Linchin May 5, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Linchin May 5, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chalmerlowe commented Apr 17, 2025 •

edited

Loading

Linchin commented Apr 23, 2025 •

edited

Loading

tswast commented May 5, 2025 •

edited

Loading

chalmerlowe May 7, 2025 •

edited

Loading

Linchin May 5, 2025 •

edited

Loading

Linchin May 5, 2025 •

edited

Loading