Skip to content

typing.Union does not support attribute assignment post gh-105511 #132139

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
XuehaiPan opened this issue Apr 5, 2025 · 10 comments · Fixed by #132146
Closed

typing.Union does not support attribute assignment post gh-105511 #132139

XuehaiPan opened this issue Apr 5, 2025 · 10 comments · Fixed by #132146
Labels
3.14 new features, bugs and security fixes extension-modules C modules in the Modules dir stdlib Python modules in the Lib dir topic-typing type-bug An unexpected behavior, bug, or error

Comments

@XuehaiPan
Copy link
Contributor

XuehaiPan commented Apr 5, 2025

Bug report

Bug description:

Before #105511, typing.Union is implemented in Python, and custom attributes can be assigned to a Union variable.

ta = Union[int, str]
ta.__some_attribute__ = (int, str)

However, after #105511, typing.Union is now implemented in C, and no attributes can be assigned to it. The UnionType does not allow subclassing; users cannot bypass this using subclassing.

$ ipython
Python 3.14.0a6+ experimental free-threading build (heads/main:1755157207c, Apr  6 2025, 02:21:10) [Clang 17.0.0 (clang-1700.0.13.3)]
Type 'copyright', 'credits' or 'license' for more information
IPython 9.0.2 -- An enhanced Interactive Python. Type '?' for help.
Tip: Use `object?` to see the help on `object`, `object??` to view it's source

In [1]: import optree

In [2]: optree.PyTree[int]
/Users/PanXuehai/Projects/optree/env/lib/python3.14t/site-packages/IPython/core/interactiveshell.py:3111: SyntaxWarning: 'return' in a 'finally' block
  return result
╭────────────────────────────────────────────────────────────────────── Traceback (most recent call last) ───────────────────────────────────────────────────────────────────────╮
│ in <module>:1                                                                                                                                                                  │
│                                                                                                                                                                                │
│ /Users/PanXuehai/Projects/cpython/.pydev/lib/python3.14t/typing.py:398 in inner                                                                                                │
│                                                                                                                                                                                │
│    395 │   │   @functools.wraps(func)                                                                                                                                          │
│    396 │   │   def inner(*args, **kwds):                                                                                                                                       │
│    397 │   │   │   try:                                                                                                                                                        │
│ ❱  398 │   │   │   │   return _caches[func](*args, **kwds)                                                                                                                     │
│    399 │   │   │   except TypeError:                                                                                                                                           │
│    400 │   │   │   │   pass  # All real errors (not unhashable args) are raised below.                                                                                         │401 │   │   │   return func(*args, **kwds)                                                                                                                                  │
│                                                                                                                                                                                │
│ /Users/PanXuehai/Projects/optree/optree/typing.py:254 in __class_getitem__                                                                                                     │
│                                                                                                                                                                                │
│   251 │   │   │   Deque[recurse_ref],  # type: ignore[valid-type]                                                                                                              │252 │   │   │   CustomTreeNode[recurse_ref],  # type: ignore[valid-type]                                                                                                     │253 │   │   ]                                                                                                                                                                │
│ ❱ 254 │   │   pytree_alias.__pytree_args__ = item  # type: ignore[attr-defined]                                                                                                │255 │   │                                                                                                                                                                    │
│   256 │   │   # pylint: disable-next=no-member                                                                                                                                 │257 │   │   original_copy_with = pytree_alias.copy_with  # type: ignore[attr-defined]                                                                                        │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
AttributeError: 'typing.Union' object has no attribute '__pytree_args__' and no __dict__ for setting new attributes

In [3]: class Foo(type(int | str)):
   ...:     pass
   ...:     
╭────────────────────────────────────────────────────────────────────── Traceback (most recent call last) ───────────────────────────────────────────────────────────────────────╮
│ in <module>:1                                                                                                                                                                  │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
TypeError: type 'typing.Union' is not an acceptable base type

Repro:

./python -m pip install -v optree
./python -c 'import optree; optree.PyTree[int]'

Source: https://github.com/metaopt/optree/blob/v0.15.0/optree/typing.py#L246-L254

cc @JelleZijlstra

CPython versions tested on:

CPython main branch

Operating systems tested on:

macOS

Linked PRs

@XuehaiPan XuehaiPan added the type-bug An unexpected behavior, bug, or error label Apr 5, 2025
@XuehaiPan XuehaiPan changed the title [3.14 Regression] Union variable does not allow attribute assignment anymore [3.14 Regression] typing.Union variable does not allow attribute assignment anymore Apr 5, 2025
@tomasr8 tomasr8 added topic-typing 3.14 new features, bugs and security fixes labels Apr 5, 2025
@picnixz picnixz added stdlib Python modules in the Lib dir extension-modules C modules in the Modules dir labels Apr 5, 2025
@picnixz
Copy link
Member

picnixz commented Apr 5, 2025

Was it designed to support custom attributes? is it a documented behavior? (I don't know so I'm just asking if this was deliberate)

@JelleZijlstra
Copy link
Member

Yeah, don't do that. Setting arbitrary attributes on Union objects (or any other typing objects) was never supported and if you do that sort of thing, be prepared for your code to break.

That said, we could allow setting arbitrary attributes again by adding a managed __dict__ to union objects, at a cost of increasing memory usage for every other user of unions. I'd be curious to hear opinions on whether that's worth it.

@picnixz
Copy link
Member

picnixz commented Apr 5, 2025

at a cost of increasing memory usage for every other user of unions

How much of a cost are we talking about? I think you mentioned something about the fact that unions' size is smaller (or was it larger?) but I can't find the issue (I think I read this yesterday somewhere). If we're only talking about a few bytes when it's not used, then why not. But if we're doubling the union sizes I think it's maybe not worth.

Now, are there other projects for which this could be needed? if there is enough usage, and from large libraries, then maybe it's a possibility? By the way, for what reason do you actually need unions to support assignment?

@picnixz picnixz changed the title [3.14 Regression] typing.Union variable does not allow attribute assignment anymore typing.Union variable does not allow attribute assignment anymore post gh-105511 Apr 5, 2025
@picnixz picnixz added type-feature A feature request or enhancement and removed type-bug An unexpected behavior, bug, or error labels Apr 5, 2025
@picnixz picnixz changed the title typing.Union variable does not allow attribute assignment anymore post gh-105511 typing.Union variable does not allow attribute assignment post gh-105511 Apr 5, 2025
@picnixz picnixz changed the title typing.Union variable does not allow attribute assignment post gh-105511 typing.Union does not support attribute assignment post gh-105511 Apr 5, 2025
@picnixz picnixz removed the 3.14 new features, bugs and security fixes label Apr 6, 2025
@picnixz
Copy link
Member

picnixz commented Apr 6, 2025

(Recategorizing as a feature request since we're talking about an implementation detail that was not expected to be used)

@brianschubert
Copy link
Contributor

I think you mentioned something about the fact that unions' size is smaller (or was it larger?) but I can't find the issue

I think you're thinking of #131933 (comment)

@JelleZijlstra
Copy link
Member

Regarding size of union objects: In 3.13 and earlier, there were two kinds of unions. Both were 48 bytes:

>>> u1 = int | str
>>> u2 = Union[int, str]
>>> sys.getsizeof(u1)
48
>>> sys.getsizeof(u2)
48

In addition, typing.Union unions held a dict of 272 bytes:

>>> sys.getsizeof(u2.__dict__)
272

Though I think those are only materialized if you actually access the attribute? There's some magic involved. However, the dict has 7 entries, so (assuming 8-byte pointers) somewhere we need space for at least 7 * 8 = 56 bytes.

In 3.14, both kinds of unions were unified. The object is 72 bytes:

>>> u = int | str
>>> sys.getsizeof(u)
72

That's because relative to the earlier types.UnionType we added three pointers: one for a weakreflist and two for caching collections of hashable and unhashable arguments, which makes equality and hashing of union objects more efficient.

How much of a cost are we talking about?

I think it's another 8 bytes, so not much.

(Recategorizing as a feature request since we're talking about an implementation detail that was not expected to be used)

For what it's worth, if this was purely a feature request I'd probably reject it quickly. The only reason we still need to talk about it is that we're breaking compatibility, even if the behavior we're breaking was never meant to exist in the first place.

@picnixz
Copy link
Member

picnixz commented Apr 6, 2025

I think it's another 8 bytes, so not much.

In this case this seems okayish.

For what it's worth, if this was purely a feature request I'd probably reject it quickly. The only reason we still need to talk about it is that we're breaking compatibility, even if the behavior we're breaking was never meant to exist in the first place.

Ah I thought you were considering it as a full-fledged feature. My bad (so I'm reputting the bug+314 labels).


If we were to reinstate this, do we actually want to document the behavior? I don't really like the fact that only typing.Union would be able to set attributes (AFAIU). So if I were to choose I would either do nothing or also allow other types to support attributes assignments (but I don't know if we should document this).

@picnixz picnixz added type-bug An unexpected behavior, bug, or error 3.14 new features, bugs and security fixes and removed type-feature A feature request or enhancement labels Apr 6, 2025
@picnixz
Copy link
Member

picnixz commented Apr 6, 2025

I think you mentioned something about the fact that unions' size is smaller (or was it larger?) but I can't find the issue

I think you're thinking of #131933 (comment)

I was thinking of #131933 (comment) but yeah this was rhe discussion I had in kind. Thanks!

@AA-Turner
Copy link
Member

I think we should add this to the documentation/What's New (it is an observable change), but I don't think it's worth it to add a dict to Union or allow subclassing; these were never supported features.

Code attempting this is already brittle: it only worked for one of the two types of unions, and further required that the name was a dunder.

>>> typing.Union[int, str].__spam__ = 1
>>> typing.Union[int, str].spam = 1
Traceback (most recent call last):
  File "<...>", line 1, in <module>
    typing.Union[int, str].spam = 1
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "...\Lib\typing.py", line 1219, in __setattr__
    setattr(self.__origin__, attr, val)
AttributeError: '_SpecialForm' object has no attribute 'spam'

All dunders are explicitly reserved to the language, so I think we should close this as wontfix, and possibly consider mentioning it in the documentation.

A

@JelleZijlstra
Copy link
Member

Good point, I hadn't noticed that it worked only for dunders because of _BaseGenericAlias.__setattr__.

Another confusing aspect here is the union cache that existed on 3.13:

>>> from typing import Union
>>> u = Union[int, str]
>>> u.__some_attr__ = 42
>>> Union[int, str].__some_attr__
42

I feel that's enough reason that we should not make any change. If you want to associate arbitrary metadata with a type, you can use Annotated. I'll send a PR documenting this in What's New.

JelleZijlstra added a commit to JelleZijlstra/cpython that referenced this issue Apr 6, 2025
AlexWaygood added a commit to AlexWaygood/cpython that referenced this issue Apr 6, 2025
…set `Union` attributes

Explain in a bit more detail why we think this is an acceptable change to make without a deprecation period
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
3.14 new features, bugs and security fixes extension-modules C modules in the Modules dir stdlib Python modules in the Lib dir topic-typing type-bug An unexpected behavior, bug, or error
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants