Skip to content

deps: Drop support for Python 3.7 and 3.8 #337

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 48 commits into from
May 7, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
48 commits
Select commit Hold shift + click to select a range
9b32164
feat: Drop support for Python 3.7 and 3.8
google-labs-jules[bot] Apr 17, 2025
dccef48
Updates python version in lint.yml
chalmerlowe Apr 18, 2025
f81c738
Updates owlbot, removing reference to 3.8
chalmerlowe Apr 18, 2025
5cb30ca
Updates CONTRIBUTING.rst
chalmerlowe Apr 18, 2025
7f44424
updates pytest warnings
chalmerlowe Apr 18, 2025
3e4e6f6
Removes test_samples-impl ref to older virtualenv package
chalmerlowe Apr 18, 2025
7beea0d
Removes references to pandas older than 1.5.0
chalmerlowe Apr 18, 2025
32d7b8f
Removes pandas older than 1.5 and misc changes
chalmerlowe Apr 18, 2025
e871d73
updates pandas in setup.py
chalmerlowe Apr 18, 2025
2bdf3ef
more updates related to pandas
chalmerlowe Apr 18, 2025
d902e31
still broken
chalmerlowe Apr 18, 2025
a0543b9
Updates FutureWarning tests to account for unittest coverage
chalmerlowe Apr 23, 2025
6a5d943
Updates json array type tests to account for unittest coverage
chalmerlowe Apr 23, 2025
1bbd5f7
updates python version checks to ensure coverage
chalmerlowe Apr 23, 2025
96a5e97
update json test for unittest coverage
chalmerlowe Apr 23, 2025
8af9bee
Update pandas_backports unittests to ensure coverage
chalmerlowe Apr 24, 2025
78574c0
Updates per review comments
chalmerlowe Apr 24, 2025
9ca34b8
Merge branch 'main' into remove-python-37-38
chalmerlowe Apr 24, 2025
de202f4
moves class from version specific compliance file to generic file
chalmerlowe Apr 30, 2025
fd9dd27
Removes weird cut and paste error
chalmerlowe Apr 30, 2025
bfe9115
fix linting errors
chalmerlowe Apr 30, 2025
91e437e
updates import statement to ensure import of JSONArrowType
chalmerlowe Apr 30, 2025
eca1ab7
Revise required github status checks
chalmerlowe Apr 30, 2025
06b4e88
update linting
chalmerlowe Apr 30, 2025
d481be4
temporarily marking a class as no cover
chalmerlowe May 1, 2025
f4a9fc6
more updates
chalmerlowe May 1, 2025
7585a86
marked several snippets as pragma no cover
chalmerlowe May 1, 2025
c777141
updates linting
chalmerlowe May 1, 2025
7d745b0
Updates constraints and setup.py
chalmerlowe May 1, 2025
9511e7a
migrates class from one time compliance file to another
chalmerlowe May 1, 2025
9638e43
updating pyarrow version
chalmerlowe May 1, 2025
d433fae
Updates linting
chalmerlowe May 1, 2025
4c522b3
removes determine all and module reload tests
chalmerlowe May 7, 2025
cbfcd1a
updates re: ndarrybackedextensionarray
chalmerlowe May 7, 2025
2f9fc09
testing blacken as part of owlbot processing using 3.8
chalmerlowe May 7, 2025
7b967e7
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] May 7, 2025
18098f0
updates blacken to 3.10
chalmerlowe May 7, 2025
20e6565
update python version in lint.yml
chalmerlowe May 7, 2025
f39b97f
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] May 7, 2025
c23d435
updates owlbot.py
chalmerlowe May 7, 2025
a52c46d
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] May 7, 2025
4ebe910
updates owlbot.py
chalmerlowe May 7, 2025
cd4942e
testing lint.yml
chalmerlowe May 7, 2025
32245de
testing linting issue
chalmerlowe May 7, 2025
695c33d
testing linting issue
chalmerlowe May 7, 2025
9ea13d3
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] May 7, 2025
2ee0533
updates lint_setup session
chalmerlowe May 7, 2025
f59ee72
Update noxfile.py
chalmerlowe May 7, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .github/sync-repo-settings.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,6 @@ branchProtectionRules:
- 'cla/google'
- 'docs'
- 'lint'
- 'unit (3.8)'
- 'unit (3.9)'
- 'unit (3.10)'
- 'unit (3.11)'
Expand Down
10 changes: 4 additions & 6 deletions .github/workflows/unittest.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,37 +5,35 @@
name: unittest
jobs:
unit:
# TODO(https://github.com/googleapis/gapic-generator-python/issues/2303): use `ubuntu-latest` once this bug is fixed.
# Use ubuntu-22.04 until Python 3.7 is removed from the test matrix
# https://docs.github.com/en/actions/using-github-hosted-runners/using-github-hosted-runners/about-github-hosted-runners#standard-github-hosted-runners-for-public-repositories
runs-on: ubuntu-22.04
# Use `ubuntu-latest` runner.
runs-on: ubuntu-latest
strategy:
matrix:
python: ['3.8', '3.9', '3.10', '3.11', '3.12', '3.13']
python: ['3.9', '3.10', '3.11', '3.12', '3.13']
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Setup Python
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python }}
- name: Install nox
run: |
python -m pip install --upgrade setuptools pip wheel
python -m pip install nox
- name: Run unit tests
env:
COVERAGE_FILE: .coverage-${{ matrix.python }}
run: |
nox -s unit-${{ matrix.python }}
- name: Upload coverage results
uses: actions/upload-artifact@v4
with:
name: coverage-artifact-${{ matrix.python }}
path: .coverage-${{ matrix.python }}
include-hidden-files: true

unit-prerelease:

Check warning

Code scanning / CodeQL

Workflow does not contain permissions Medium

Actions job or workflow does not limit the permissions of the GITHUB_TOKEN. Consider setting an explicit permissions block, using the following as a minimal starting point: {contents: read}
runs-on: ubuntu-latest
strategy:
matrix:
Expand Down Expand Up @@ -103,7 +101,7 @@
- name: Setup Python
uses: actions/setup-python@v5
with:
python-version: "3.8"
python-version: "3.9"
- name: Install coverage
run: |
python -m pip install --upgrade setuptools pip wheel
Expand Down
4 changes: 2 additions & 2 deletions CONTRIBUTING.rst
Original file line number Diff line number Diff line change
Expand Up @@ -143,12 +143,12 @@ Running System Tests
$ nox -s system

# Run a single system test
$ nox -s system-3.8 -- -k <name of test>
$ nox -s system-3.9 -- -k <name of test>


.. note::

System tests are only configured to run under Python 3.8.
System tests are only configured to run under Python 3.9.
For expediency, we do not run them in older versions of Python 3.

This alone will not run the tests. You'll need to change some local
Expand Down
4 changes: 2 additions & 2 deletions README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -34,11 +34,11 @@ dependencies.

Supported Python Versions
^^^^^^^^^^^^^^^^^^^^^^^^^
Python >= 3.7
Python >= 3.9

Unsupported Python Versions
^^^^^^^^^^^^^^^^^^^^^^^^^^^
Python <= 3.6.
Python <= 3.8.


Mac/Linux
Expand Down
42 changes: 11 additions & 31 deletions db_dtypes/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,15 +21,14 @@
import warnings

import numpy
import packaging.version
import pandas
import pandas.api.extensions
from pandas.errors import OutOfBoundsDatetime
import pyarrow
import pyarrow.compute

from db_dtypes import core
from db_dtypes.version import __version__
from db_dtypes.json import JSONArray, JSONDtype, JSONArrowType # noqa: F401

from . import _versions_helpers

Expand All @@ -47,15 +46,6 @@
_NP_BOX_DTYPE = "datetime64[us]"


# To use JSONArray and JSONDtype, you'll need Pandas 1.5.0 or later. With the removal
# of Python 3.7 compatibility, the minimum Pandas version will be updated to 1.5.0.
if packaging.version.Version(pandas.__version__) >= packaging.version.Version("1.5.0"):
Copy link
Contributor

@Linchin Linchin May 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From the comments, it seems removing Python 3.7 could ensure pandas >= 1.5.0, and thus ensure JSONArray and JSONDtype to be available. Do we still need _determine_all()?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed.

from db_dtypes.json import JSONArray, JSONArrowType, JSONDtype
else:
JSONArray = None
JSONDtype = None


@pandas.api.extensions.register_extension_dtype
class TimeDtype(core.BaseDatetimeDtype):
"""
Expand Down Expand Up @@ -364,23 +354,13 @@ def _check_python_version():

_check_python_version()


if not JSONArray or not JSONDtype:
__all__ = [
"__version__",
"DateArray",
"DateDtype",
"TimeArray",
"TimeDtype",
]
else:
__all__ = [
"__version__",
"DateArray",
"DateDtype",
"JSONDtype",
"JSONArray",
"JSONArrowType",
"TimeArray",
"TimeDtype",
]
__all__ = [
"__version__",
"DateArray",
"DateDtype",
"TimeArray",
"TimeDtype",
"JSONDtype",
"JSONArray",
"JSONArrowType",
]
8 changes: 2 additions & 6 deletions db_dtypes/core.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
import pandas
import pandas.api.extensions
from pandas.api.types import is_dtype_equal, is_list_like, is_scalar, pandas_dtype
from pandas.core.arrays import _mixins

from db_dtypes import pandas_backports

Expand All @@ -42,9 +43,7 @@ def construct_from_string(cls, name: str):
return cls()


class BaseDatetimeArray(
pandas_backports.OpsMixin, pandas_backports.NDArrayBackedExtensionArray
):
class BaseDatetimeArray(pandas_backports.OpsMixin, _mixins.NDArrayBackedExtensionArray):
# scalar used to denote NA value inside our self._ndarray, e.g. -1 for
# Categorical, iNaT for Period. Outside of object dtype, self.isna() should
# be exactly locations in self._ndarray with _internal_fill_value. See:
Expand Down Expand Up @@ -186,9 +185,6 @@ def median(
keepdims: bool = False,
skipna: bool = True,
):
if not hasattr(pandas_backports, "numpy_validate_median"):
raise NotImplementedError("Need pandas 1.3 or later to calculate median.")

pandas_backports.numpy_validate_median(
(),
{"out": out, "overwrite_input": overwrite_input, "keepdims": keepdims},
Expand Down
92 changes: 3 additions & 89 deletions db_dtypes/pandas_backports.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,18 +19,13 @@
the versions in the later versions of pandas.
"""

from typing import Any

import numpy
import packaging.version
import pandas
from pandas.api.types import is_integer
import pandas.compat.numpy.function
import pandas.core.nanops

pandas_release = packaging.version.parse(pandas.__version__).release

# Create aliases for private methods in case they move in a future version.
# # Create aliases for private methods in case they move in a future version.
nanall = pandas.core.nanops.nanall
nanany = pandas.core.nanops.nanany
nanmax = pandas.core.nanops.nanmax
Expand All @@ -40,9 +35,8 @@
numpy_validate_max = pandas.compat.numpy.function.validate_max
numpy_validate_min = pandas.compat.numpy.function.validate_min

if pandas_release >= (1, 3):
nanmedian = pandas.core.nanops.nanmedian
numpy_validate_median = pandas.compat.numpy.function.validate_median
nanmedian = pandas.core.nanops.nanmedian
numpy_validate_median = pandas.compat.numpy.function.validate_median


def import_default(module_name, force=False, default=None):
Expand Down Expand Up @@ -78,83 +72,3 @@ def import_default(module_name, force=False, default=None):
class OpsMixin:
def _cmp_method(self, other, op): # pragma: NO COVER
return NotImplemented


# TODO: use public API once pandas 1.5 / 2.x is released.
# See: https://github.com/pandas-dev/pandas/pull/45544
@import_default("pandas.core.arrays._mixins", pandas_release < (1, 3))
class NDArrayBackedExtensionArray(pandas.core.arrays.base.ExtensionArray):
def __init__(self, values, dtype):
assert isinstance(values, numpy.ndarray)
self._ndarray = values
self._dtype = dtype

@classmethod
def _from_backing_data(cls, data):
return cls(data, data.dtype)

def __getitem__(self, index):
value = self._ndarray[index]
if is_integer(index):
return self._box_func(value)
return self.__class__(value, self._dtype)

def __setitem__(self, index, value):
self._ndarray[index] = self._validate_setitem_value(value)

def __len__(self):
return len(self._ndarray)

@property
def shape(self):
return self._ndarray.shape

@property
def ndim(self) -> int:
return self._ndarray.ndim

@property
def size(self) -> int:
return self._ndarray.size

@property
def nbytes(self) -> int:
return self._ndarray.nbytes

def copy(self):
return self[:]

def repeat(self, n):
return self.__class__(self._ndarray.repeat(n), self._dtype)

def take(
self,
indices,
*,
allow_fill: bool = False,
fill_value: Any = None,
axis: int = 0,
):
from pandas.core.algorithms import take

if allow_fill:
fill_value = self._validate_scalar(fill_value)

new_data = take(
self._ndarray,
indices,
allow_fill=allow_fill,
fill_value=fill_value,
axis=axis,
)
return self._from_backing_data(new_data)

@classmethod
def _concat_same_type(cls, to_concat, axis=0):
dtypes = {str(x.dtype) for x in to_concat}
if len(dtypes) != 1:
raise ValueError("to_concat must have the same dtype (tz)", dtypes)

new_values = [x._ndarray for x in to_concat]
new_values = numpy.concatenate(new_values, axis=axis)
return to_concat[0]._from_backing_data(new_values) # type: ignore[arg-type]
2 changes: 1 addition & 1 deletion db_dtypes/version.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,4 +12,4 @@
# See the License for the specific language governing permissions and
# limitations under the License.

__version__ = "1.4.2"
__version__ = "1.4.2" # pragma: NO COVER
22 changes: 15 additions & 7 deletions noxfile.py
Original file line number Diff line number Diff line change
Expand Up @@ -32,11 +32,9 @@
ISORT_VERSION = "isort==5.11.0"
LINT_PATHS = ["docs", "db_dtypes", "tests", "noxfile.py", "setup.py"]

DEFAULT_PYTHON_VERSION = "3.8"
DEFAULT_PYTHON_VERSION = "3.9"

UNIT_TEST_PYTHON_VERSIONS: List[str] = [
"3.7",
"3.8",
"3.9",
"3.10",
"3.11",
Expand All @@ -56,7 +54,7 @@
UNIT_TEST_EXTRAS: List[str] = []
UNIT_TEST_EXTRAS_BY_PYTHON: Dict[str, List[str]] = {}

SYSTEM_TEST_PYTHON_VERSIONS: List[str] = ["3.8"]
SYSTEM_TEST_PYTHON_VERSIONS: List[str] = ["3.9"]
SYSTEM_TEST_STANDARD_DEPENDENCIES: List[str] = [
"mock",
"pytest",
Expand Down Expand Up @@ -88,7 +86,10 @@
nox.options.error_on_missing_interpreters = True


@nox.session(python=DEFAULT_PYTHON_VERSION)
# TODO: the linting process still uses python 3.8.
# As soon as that gets upgraded, we should be able to revert this session
# to using the DEFAULT_PYTHON_VERSION.
@nox.session(python="3.8")
def lint(session):
"""Run linters.

Expand All @@ -105,7 +106,11 @@ def lint(session):
session.run("flake8", "db_dtypes", "tests")


@nox.session(python=DEFAULT_PYTHON_VERSION)
# TODO: the owlbot-python docker image still has python 3.8 installed (
# and only 3.8).
# As soon as that gets upgraded, we should be able to revert this session
# to using the DEFAULT_PYTHON_VERSION.
@nox.session(python="3.8")
def blacken(session):
"""Run black. Format code to uniform standard."""
session.install(BLACK_VERSION)
Expand Down Expand Up @@ -137,7 +142,10 @@ def format(session):
)


@nox.session(python=DEFAULT_PYTHON_VERSION)
# TODO: the linting process still uses python 3.8.
# As soon as that gets upgraded, we should be able to revert this session
# to using the DEFAULT_PYTHON_VERSION.
@nox.session(python="3.8")
def lint_setup_py(session):
"""Verify that setup.py is valid (including RST check)."""
session.install("docutils", "pygments")
Expand Down
2 changes: 1 addition & 1 deletion owlbot.py
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@
# Add templated files
# ----------------------------------------------------------------------------
templated_files = common.py_library(
system_test_python_versions=["3.8"],
system_test_python_versions=["3.9"],
cov_level=100,
intersphinx_dependencies={
"pandas": "https://pandas.pydata.org/pandas-docs/stable/"
Expand Down
7 changes: 0 additions & 7 deletions pytest.ini
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,6 @@
filterwarnings =
# treat all warnings as errors
error
# Remove once support for python 3.7 and 3.8 is dropped
# Ignore warnings from older versions of pandas which still have python 3.7/3.8 support
ignore:.*distutils Version classes are deprecated:DeprecationWarning
ignore:.*resolve package from __spec__ or __package__, falling back on __name__ and __path__:ImportWarning
# Remove once https://github.com/dateutil/dateutil/issues/1314 is fixed
# dateutil is a dependency of pandas
ignore:datetime.datetime.utcfromtimestamp\(\) is deprecated:DeprecationWarning:dateutil.tz.tz
# Remove once https://github.com/googleapis/python-db-dtypes-pandas/issues/227 is fixed
ignore:.*any.*with datetime64 dtypes is deprecated and will raise in a future version:FutureWarning
ignore:.*all.*with datetime64 dtypes is deprecated and will raise in a future version:FutureWarning
3 changes: 1 addition & 2 deletions samples/snippets/requirements-test.txt
Original file line number Diff line number Diff line change
@@ -1,2 +1 @@
pytest===7.4.4; python_version == '3.7' # prevents dependabot from upgrading it
pytest==8.3.3; python_version > '3.7'
pytest==8.3.5
12 changes: 5 additions & 7 deletions setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -30,10 +30,10 @@
release_status = "Development Status :: 5 - Production/Stable"

dependencies = [
"packaging >= 17.0",
"pandas >= 1.2.0",
"pyarrow>=3.0.0",
"numpy >= 1.16.6",
"numpy >= 1.24.0",
"packaging >= 24.2.0",
"pandas >= 1.5.3",
"pyarrow >= 13.0.0",
]

package_root = os.path.abspath(os.path.dirname(__file__))
Expand Down Expand Up @@ -63,8 +63,6 @@ def readme():
"License :: OSI Approved :: Apache Software License",
"Programming Language :: Python",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.7",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
Expand All @@ -75,6 +73,6 @@ def readme():
],
platforms="Posix; MacOS X; Windows",
install_requires=dependencies,
python_requires=">=3.7",
python_requires=">=3.9",
tests_require=["pytest"],
)
Loading