Skip to content

Clinic signatures of HACL* hash functions are inconsistent with OpenSSL implementation #134696

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
8R0WNI3 opened this issue May 26, 2025 · 10 comments
Assignees
Labels
3.13 bugs and security fixes 3.14 bugs and security fixes 3.15 new features, bugs and security fixes extension-modules C modules in the Modules dir type-bug An unexpected behavior, bug, or error

Comments

@8R0WNI3
Copy link

8R0WNI3 commented May 26, 2025

Bug description:

According to its docstring, the function hashlib.shake_128(...).hexdigest() should accept an argument length, either as positional or as keyword argument. However, since Python 3.14, the function only accepts positional arguments and fails with TypeError: shake_128.hexdigest() takes no keyword arguments otherwise.

I narrowed the root cause for this behaviour down to the following two commits (I'm not 100% sure though as I don't have any C-background):

Reproducer:

import hashlib
hashlib.shake_128(b'').hexdigest(1) # working
hashlib.shake_128(b'').hexdigest(length=1) # not working anymore, but was working before

For example, on Python Python 3.13.3 (main, Apr 8 2025, 13:54:08) [Clang 17.0.0 (clang-1700.0.13.3)] on my mac I can still use the keyword argument, but on alpine:edge Python 3.12.10 (main, May 21 2025, 16:23:36) [GCC 14.2.0] on linux it is expecting positional arguments only.

CPython versions tested on:

CPython main branch, 3.14, 3.13, 3.12

Operating systems tested on:

Linux, macOS

Linked PRs

@8R0WNI3 8R0WNI3 added the type-bug An unexpected behavior, bug, or error label May 26, 2025
@picnixz picnixz added extension-modules C modules in the Modules dir topic-argument-clinic labels May 26, 2025
@picnixz
Copy link
Member

picnixz commented May 26, 2025

I cannot reproduce the issue. On Python 3.15, it works well and I don't think I change anything between 3.14 and 3.15 (namely I can use a keyword argument). If you cannot use a keyword argument on 3.12, then I think it's ok? I mean, the recent behavior is an extension of the old one.

@picnixz picnixz added the pending The issue will be closed if no feedback is provided label May 26, 2025
@8R0WNI3
Copy link
Author

8R0WNI3 commented May 26, 2025

On alpine:3 running Python 3.12.10 (main, May 21 2025, 20:03:03) [GCC 14.2.0], the keyword argument is allowed (this was the behaviour in the past and seems to be the case in 3.15 as well. However, on alpine:edge running Python 3.12.10 (main, May 21 2025, 16:23:36) [GCC 14.2.0], the keyword argument is not allowed anymore.

@picnixz
Copy link
Member

picnixz commented May 26, 2025

3.12 is security-only so we won't fix this IMO (it's not a security issue). In your comment, you're linking 3.12.10 twice so I don't really understand which one is 3.12 or not (only the timestamps change and I don't know which correspond to what)

@encukou
Copy link
Member

encukou commented May 26, 2025

Could you share how you're installing dependencies & building CPython on alpine:edge?

@8R0WNI3
Copy link
Author

8R0WNI3 commented May 26, 2025

Reproducer:

> docker run -it alpine:edge
  > apk add python3
  > python3 -c "import hashlib; print(hashlib.shake_128(b'').hexdigest(length=1))"

However, in the meantime I figured if I run apk upgrade libcrypto3, the keyword argument is accepted again, so this might be an issue between Python and openssl?

@picnixz
Copy link
Member

picnixz commented May 26, 2025

I may have an idea. What's the output of type(hashlib.shak_128(b''). If OpenSSl is not installed, then we may have fell back on the HACL* implementation

@picnixz
Copy link
Member

picnixz commented May 26, 2025

And indeed there is some issue here:

Image

This is in _sha3module. So the HACL* implementation doesn't allow for keyword arguments. Thanks for spotting this disparity.

@picnixz picnixz changed the title hashlib.shake_128().hexdigest() no support for keyword arguments anymore Signature for shake_{128,256}.hexdigest() is not inconsistent between OpenSSL and HACL* implementation May 26, 2025
@picnixz picnixz added 3.13 bugs and security fixes 3.14 bugs and security fixes 3.15 new features, bugs and security fixes and removed pending The issue will be closed if no feedback is provided topic-argument-clinic labels May 26, 2025
@picnixz picnixz self-assigned this May 26, 2025
@picnixz
Copy link
Member

picnixz commented May 26, 2025

Here's the plan: I'll make it possible a keyword argument for 3.13 and 3.14 (sorry but 3.12 won't be fixed as I don't consider this as a true security issue; it just allows to distinguish between HACL* and OpenSSL implementations but I don't think it's worth a security backport).

@picnixz picnixz changed the title Signature for shake_{128,256}.hexdigest() is not inconsistent between OpenSSL and HACL* implementation Inconsistent AC signatures for hash functions between OpenSSL and HACL* implementation May 26, 2025
@encukou encukou changed the title Inconsistent AC signatures for hash functions between OpenSSL and HACL* implementation Signature for shake_{128,256}.hexdigest() is inconsistent between OpenSSL and HACL* implementation May 26, 2025
@picnixz
Copy link
Member

picnixz commented May 26, 2025

Ha... actually it's not just SHAKE that is affected...

@picnixz picnixz changed the title Signature for shake_{128,256}.hexdigest() is inconsistent between OpenSSL and HACL* implementation Clinic signatures in _sha3 are inconsistent with OpenSSL implementation May 26, 2025
@picnixz picnixz changed the title Clinic signatures in _sha3 are inconsistent with OpenSSL implementation Clinic signatures in _sha3 and _blake2 are inconsistent with OpenSSL implementation May 26, 2025
@picnixz
Copy link
Member

picnixz commented May 26, 2025

Urgh, great, there is not one but two inconsistencies. In _hashlib, we have _hashlib.openssl_md5(string, *, ...) but elsewhere we have hashlib.new(name, data, *, ...).

Or, we have _sha3.sha3_224(string, *, ...) but the docs say hashlib.sha3_224(data, *, ...). So I think I'll break users that directly use the private modules, rather than those that use hashlib itself, i.e., I'll assume that the data argument for the constructors is data and not string.

Ideally, I want them to be positional-only.

@picnixz picnixz changed the title Clinic signatures in _sha3 and _blake2 are inconsistent with OpenSSL implementation Clinic signatures of HACL* hash functions are inconsistent with OpenSSL implementation May 26, 2025
picnixz added a commit that referenced this issue May 31, 2025
…AC signatures (#134713)

OpenSSL and HACL*-based hash functions constructors now support both `data` and `string` parameters.
Previously these constructor functions inconsistently supported sometimes `data` and sometimes `string`,
while the documentation expected `data` to be given in all cases.
picnixz added a commit to picnixz/cpython that referenced this issue May 31, 2025
…constructors AC signatures (pythonGH-134713)

OpenSSL and HACL*-based hash functions constructors now support both `data` and `string` parameters.
Previously these constructor functions inconsistently supported sometimes `data` and sometimes `string`,
while the documentation expected `data` to be given in all cases.
(cherry picked from commit c6e63d9)

Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
picnixz added a commit to picnixz/cpython that referenced this issue May 31, 2025
…constructors AC signatures (pythonGH-134713)

OpenSSL and HACL*-based hash functions constructors now support both `data` and `string` parameters.
Previously these constructor functions inconsistently supported sometimes `data` and sometimes `string`,
while the documentation expected `data` to be given in all cases.
(cherry picked from commit c6e63d9)

Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
picnixz added a commit to picnixz/cpython that referenced this issue May 31, 2025
…constructors AC signatures (pythonGH-134713)

OpenSSL and HACL*-based hash functions constructors now support both `data` and `string` parameters.
Previously these constructor functions inconsistently supported sometimes `data` and sometimes `string`,
while the documentation expected `data` to be given in all cases.

(cherry picked from commit c6e63d9)
(cherry picked from commit 379d0bc)

Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
picnixz added a commit to picnixz/cpython that referenced this issue May 31, 2025
…constructors AC signatures (pythonGH-134713)

OpenSSL and HACL*-based hash functions constructors now support both `data` and `string` parameters.
Previously these constructor functions inconsistently supported sometimes `data` and sometimes `string`,
while the documentation expected `data` to be given in all cases.

(cherry picked from commit c6e63d9)
(cherry picked from commit 379d0bc)

Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
picnixz added a commit that referenced this issue Jun 1, 2025
…uctors AC signatures (GH-134713) (#134961)

OpenSSL and HACL*-based hash functions constructors now support both `data` and `string` parameters.
Previously these constructor functions inconsistently supported sometimes `data` and sometimes `string`,
while the documentation expected `data` to be given in all cases.

(cherry picked from commit c6e63d9)
(cherry picked from commit 379d0bc)
picnixz added a commit that referenced this issue Jun 1, 2025
…uctors AC signatures (GH-134713) (#134962)

OpenSSL and HACL*-based hash functions constructors now support both `data` and `string` parameters.
Previously these constructor functions inconsistently supported sometimes `data` and sometimes `string`,
while the documentation expected `data` to be given in all cases.

(cherry picked from commit c6e63d9)
(cherry picked from commit 379d0bc)
@picnixz picnixz closed this as completed Jun 2, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
3.13 bugs and security fixes 3.14 bugs and security fixes 3.15 new features, bugs and security fixes extension-modules C modules in the Modules dir type-bug An unexpected behavior, bug, or error
Projects
None yet
Development

No branches or pull requests

3 participants