initial commit to update the SQLAlchemy examples #11437

jvanasco · 2024-06-01T01:31:26Z

most examples now work on 2.x
examples are mostly typed
README files are added for more instruction
examples now have metadata for automation
a preliminary fabric tool is used for testing and reporting

Change-Id: Ib5a12cc59b2ea045c48ea90798b887e3b5d4fbbe

Description

There are quite a bit of things going on in this.

Examples are updated to run on SQLAlchemy2 - HOWEVER there are a few broken examples and I'm unsure if I did some things right.
I started to integrate typing. IMHO, the examples should not only work on 2.0 but be fully typed so they can be a reference. I tried typing one core library function, but it is likely wrong.
I had a few changes to the structure:

Some files and folders were renamed to get around shadowing issues. i added a trailing underscore when needed.
I added a _utils.py file, which includes a TypedDict for metadata about each example
I updated the __init__ for each directory to have metadata about the examples. more in a bit.
I used fabric (fabfile.org) to write a tool to help audit and test the examples. i left this in the main directory, it could be removed and refactored into another project (or never touched again). i do love using fabric, and things like /reap_dbs.py could be migrated into fabric (as well as many other build/deploy/test things). The fabric file uses env vars to control some things (like not running the intensive processes)

Most of the examples now work. There are a few that are still broken and I am unsure how to fix.

Typing is 90% of the way there. Someone will have to take over for that stuff though, as I am maxxed out on my typing skills. I know Gord is interested in modernizing the wiki and other things, tagging you in?

So I do need help to finish. It is also exposing some issues. Of note:

    examples/dogpile_caching/caching_query.py
    
        110: error: Argument 2 to "merge_frozen_result" has incompatible type "Executable"; expected "Select[Unpack[tuple[Any, ...]]] | FromStatement[Unpack[tuple[Any, ...]]] | Query[Any]"  [arg-type]
    
        this required me to add typing to `loader.merge_frozen_result`.
        i am not sure if i added the wrong typing to that (based on the docstrings and class)
        or if there are issues in the example
        
        
    examples/generic_associations/discriminator_on_association.py
        a few untyped things here
        `@declared_attr;def __tablename__(cls):` is problematic, because typing that will create issues with string __tablename__ in children
        while this can be solved with a Union within the file, it seems like `@declared_attr` should somehow address this for all users/implementations
    
        also, typing the classmethods such as `HasAddresses.address_association` is questionable
    

    examples/inheritance/single.py
        here we set:
            __table__: FromClause
        this is functional, but creates an typing override against declarative base.

    examples/vertical/dictlike-polymorphic.py
        mypy doesn't like hybrid properties, errors for a redefinition
        
    examples/versioned_history
        i can't get a few tests to pass

UNSURE
association/basic_association.py
unsure about the join change

BROKEN:

python -m examples.asyncio_.greenlet_orm
    the typing change caused an IntegrityError
    adding "nullable=True" to 1 columns let it work; I am not sure if this is the correct fix.
    
python -m examples.dogpile_caching.model
    this will generate an error if you run the model file directly
        python -m examples.dogpile_caching.model
    there should likely be a way to disable this or immediately exit

python -m examples.materialized_paths.materialized_paths

    exception:

      File "/sqlalchemy/examples/materialized_paths/materialized_paths.py", line 50, in <module>
        class Node(Base):
      File "/sqlalchemy/examples/materialized_paths/materialized_paths.py", line 70, in Node
        id.label("id"),
      File "/python-3.10.2/lib/python3.10/site-packages/sqlalchemy/sql/roles.py", line 177, in label
        raise NotImplementedError()        
    
python -m examples.space_invaders.space_invaders
    the typing change caused an IntegrityError
    adding "nullable=True" to 2 columns let it work; I am not sure if this is the correct fix.
    
python -m examples.versioned_history.test_versioning
    2 tests fail

python -m examples.versioned_rows.versioned_map
    sqlalchemy.exc.IntegrityError
    the exact exception suggests something larger is wrong

* most examples now work on 2.x * examples are mostly typed * README files are added for more instruction * examples now have metadata for automation * a preliminary fabric tool is used for testing and reporting Change-Id: Ib5a12cc59b2ea045c48ea90798b887e3b5d4fbbe

Change-Id: I0124ecd98164a404346edf8c9eeb1799dd715dd6

jvanasco · 2024-07-09T20:42:13Z

I undid the typing update against the main library, so this PR now only updates the examples.

I am not going to have time to get into the main library typing for over a month :(

zzzeek

thanks for this!

unfortunately I'm -1 on the automation steps being added here, these detract from the clarity of the examples and I'd rather keep the amount of source code presented to the user at an absolute minimum

zzzeek · 2024-08-11T13:15:29Z

examples/asyncio_/__init__.py

I understand the word "asyncio" is a python-reserved word and there's some pep somehwere telling us to not use this name as a directory name but can we please revert this within the scope of this PR ? changing that name impacts the doc rsts and real world hyperlinks and I dont see this as necessary right now

I can do a search to updates rsts and hyperlinks. The reason for the rename is filename shadowing on Python3. It didn't happen on Python2, but the imports system on Python3 keeps trying to pull system or sqlalchemy files named asyncio. I tried multiple strategies to avoid this, none worked.

zzzeek · 2024-08-11T13:16:20Z

fabfile/__README.txt

im not sure what this directory is about and I'd prefer to review this separately from the straightforward task of making the examples work, can this be done in a separate PR and removed here?

I wrote some Fabric utilities to handle automatic testing and linting of the examples directory.

If Fabric is installed (pip install fabric) the following commands will work on the main directory:

fab -l # list commands fab examples-audit # quickly audit examples to determine requirements & flake8/mypy compat fab examples-test # run each example, report if it works or not

My idea/suggestion is to leave these in-place while this PR is finalized, then remove it once everything passes.

At that point, I can repackage it into a standalone tool for the SqlAlchemy-org repository and it can leveraged into the CI pipeline so we can ensure the examples are still up to date. I think a few of the other CI/maintenance tasks could be adapted into Fabric as well.

I would do it as a separate repo now, but that would mean whomever works on helping to finalize this PR would have a harder time to run it.

zzzeek · 2024-08-11T13:18:14Z

examples/_utils.py

can we move these new top-level files to another PR so I can review the rationale etc separately

That could be done, but it might be best if I address that here - as editing them might be a better option.

_utils.py introduces two typing types: ExampleMetadata and DirectoryExamples which is a dict of {filename:ExampleMetadata}

A usecase is In asyncio_/__init__.py which lists all the examples for this directory. This notes what the requirements are for each example, in terms of packages and backing datastores. THIS IS NOT INTENDED FOR END USERS AND COULD BE DISCLAIMED AS SUCH The entire purpose is to ensure the development and CI environments have the right packages and systems installed, and notes any specialized information for the tests to run. Several of the tests require prior setup or clean databases. This is described in human language in the docs, but there is no machine readable (or parseable) way to extract this information. The only way to automate testing of the examples - which is needed to ensure they stay running across releases - is to somehow store this information.

This could be stored and tracked differently (though I'd rather not), by dropping everything into a _sqlalchemy_maintainer namespace and shadowing the directory structure. (First idea, there are likely others).

There is a necessary evil (or balance) needed to make these examples testable and plug them into the CI system. I am fully open to how this is handled, but this really does need to be handled as there were many major changes that broke the Examples without anyone noticing and we need something to protect against that.

zzzeek · 2024-08-11T13:20:13Z

examples/dogpile_caching/__init__.py

+from .._utils import DirectoryExamples
+
+
+REQUIREMENTS: DirectoryExamples = {


I'd rather just add to the README simple steps for how to install dependencies and run the examples, the automation steps here make the examples look less accessible .

the examples here are mainly intended to be read and understood; running them should be a secondary concern and someone who is running them should want to read documentation how to do so, since that's why these are examples.

Do you think this could be disclaimed with something like the following on __Init__ files and dropping the payload to the bottom of the file ?

# PLEASE IGNORE THE FOLLOWING LINES # THEY ARE FOR SQLALCHEMY'S CI SYSTEMS ONLY REQUIREMENTS: DirectoryExamples = {

This payload exists to test/prep the Examples for CI testing. Because the examples are all non-standard in design/function/requirements, there needs to be a machine readable payload somewhere that can be easily mapped to each test. They could even be in a _sqla_maintainers.py file with a big "IGNORE ME!" message in it.

if it's for testing then it should be in the test/ folder, we have an examples suite that can be expanded

Change-Id: I0e9a17dd480d60c301998cd40663eea1c8094d02

Change-Id: I63f1954219e2009d5841d8c52d6ba459afd221be

jvanasco · 2024-08-15T22:39:48Z

Ok, I've moved stuff around.

I nested all the cruft into /test/examples and did a bit more typing

/test/examples/examples_metadata - python namespace for test metadata. e.g. the pypi packages needed, how to prep/cleanup the test, etc
/test/examples/fabfile - fabric routines for managing the examples

The goal for the fabric file was twofold:

have a harness to run the examples for CI
have a harness to keep testing the base while typing is adjusted

to use:

pip install fabric
cd test/examples
fab -l
fab {command}

examples-requirements

(sqla-test) Jonathans-MacBook:examples jvanasco$ fab examples-requirements
Missing the following libraries: ['aiosqlite', 'dogpile.cache', 'psycopg2']
Has the following required libraries: ['asyncpg']

In this example, examples-requirements goes through the metadata and does a report if the packages are importable or not.

examples-audit

(sqla-test) Jonathans-MacBook:examples jvanasco$ fab examples-audit

The following examples require a PostgreSQL database:
	materialized_paths/materialized_paths.py
	asyncio_/async_orm_writeonly.py
	asyncio_/async_orm.py
	asyncio_/basic.py
	asyncio_/gather_orm_statements.py
	asyncio_/greeenlet_orm.py
The following examples require the PyPI package `dogpile.cache`:
	dogpile_caching/advanced.py
	dogpile_caching/caching_query.py
	dogpile_caching/environment.py
	dogpile_caching/fixture_data.py
	dogpile_caching/helloworld.py
	dogpile_caching/local_session_cache.py
	dogpile_caching/model.py
	dogpile_caching/relationship_caching.py
The following examples require the PyPI package `aiosqlite`:
	sharding/asyncio_.py
The following examples require the PyPI package `psycopg2`:
	materialized_paths/materialized_paths.py
The following examples require the PyPI package `asyncpg`:
	asyncio_/async_orm_writeonly.py
	asyncio_/async_orm.py
	asyncio_/basic.py
	asyncio_/gather_orm_statements.py
	asyncio_/greeenlet_orm.py
The following examples have flake8 issues:
...

The following examples have mypy issues:
...

examples-test

(sqla-test) Jonathans-MacBook:examples jvanasco$ fab examples-test
# actually tries to run the examples, prints a report afterwards

examples-ci

(sqla-test) Jonathans-MacBook:examples jvanasco$ fab examples-ci

# Currently just raises an exception
# the idea is to spec what "tests" must be run in CI, then exit with a pass/fail

jvanasco added 2 commits May 31, 2024 21:21

undo typing against main library

fc9eb7f

Change-Id: I0124ecd98164a404346edf8c9eeb1799dd715dd6

jvanasco requested a review from zzzeek July 9, 2024 20:40

CaselIT mentioned this pull request Aug 11, 2024

Fix the runtime issue related to the join operation in the associatio… #11721

Merged

3 tasks

zzzeek requested changes Aug 11, 2024

View reviewed changes

jvanasco added 2 commits August 15, 2024 18:10

organizing updates

2ab2e7c

Change-Id: I0e9a17dd480d60c301998cd40663eea1c8094d02

updates

0e0257a

Change-Id: I63f1954219e2009d5841d8c52d6ba459afd221be

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

initial commit to update the SQLAlchemy examples #11437

initial commit to update the SQLAlchemy examples #11437

jvanasco commented Jun 1, 2024

jvanasco commented Jul 9, 2024

zzzeek left a comment

zzzeek Aug 11, 2024

jvanasco Aug 12, 2024

zzzeek Aug 11, 2024

jvanasco Aug 12, 2024

zzzeek Aug 11, 2024

jvanasco Aug 12, 2024

zzzeek Aug 11, 2024

jvanasco Aug 12, 2024

zzzeek Aug 13, 2024

jvanasco commented Aug 15, 2024

		from .._utils import DirectoryExamples


		REQUIREMENTS: DirectoryExamples = {

initial commit to update the SQLAlchemy examples #11437

Are you sure you want to change the base?

initial commit to update the SQLAlchemy examples #11437

Conversation

jvanasco commented Jun 1, 2024

Description

jvanasco commented Jul 9, 2024

zzzeek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jvanasco commented Aug 15, 2024

examples-requirements

examples-audit

examples-test

examples-ci