I added only the 3.15 label for now, but @zooba requested that @hugovk make an exception to include this for 3.14. I would be okay with that if we can agree the above implementation sounds reasonable.

In conda-package-handling we are using python-zstandard's stream_writer equivalent to compression.zstd.ZstdFile with pledged input size. This mattered because (at least python-zstandard) seemed to be eagerly allocating the maximum buffer on decompression which is too big at level 22.

I added only the 3.15

When marked with type-feature, it's implicitly targetting main

Public API

With the safeguard of raising an exception if the size is not the right one in the end, I'm +1 on including the method as public API.

Initial values

Update ZstdCompressor's struct to include two unsigned long long members current_frame_size and pledged_size, both initialized to ZSTD_CONTENTSIZE_UNKNOWN

I think you want current_frame_size initialized to 0?

Catch incorrect usage

modify ZstdCompressor's compress() and flush() to track how much data is being written to the compressor […]

I think we don't do that in pyzstd, but still get an exception (coming from libzstd):

>>> from pyzstd import ZstdCompressor
>>> c = ZstdCompressor()
>>> c._set_pledged_input_size(1)
>>> c.compress(b'aa')
Traceback (most recent call last):
  File "<python-input-3>", line 1, in <module>
    c.compress(b'aa')
    ~~~~~~~~~~^^^^^^^
pyzstd.ZstdError: Unable to compress zstd data: Src size is incorrect

>>> from pyzstd import ZstdCompressor
>>> c = ZstdCompressor()
>>> c._set_pledged_input_size(3)
>>> c.compress(b'aa')
b''
>>> c.flush()
Traceback (most recent call last):
  File "<python-input-4>", line 1, in <module>
    c.flush()
    ~~~~~~~^^
pyzstd.ZstdError: Unable to compress zstd data: Src size is incorrect

So I'm not sure if this is needed to implement it in code ourselves (if libzstd already does it).

Not enough data

An open question I have is should we check current_frame_size <= pledged_size at the end of writing when the mode isn't FLUSH_FRAME?

Based on my last point, we will get an error from libzstd in that case.

Moreover, from the standard specification, the value is expected to be the exact size:

Frame_Content_Size: This is the original (uncompressed) size.

I am personally in favor of raising an exception both if we have too much or not enough data.

Behavior and corner cases

My take on other misc points:

• An exception is raised when set_pledged_size is called but not at a start of a frame.
• The set_pledged_size must be called at the start of each frame, otherwise it goes back to default value.
• set_pledged_size(0) sets the value to 0 whereas set_pledged_size(None) sets the value to ZSTD_CONTENTSIZE_UNKNOWN
• It should be clear that users should use options={CompressionParameter.content_size_flag: False} if they don't want the size to be stored, because calling set_pledged_size(None) is not enough in some cases (e.g. comp.compress(..., ZstdCompressor.FLUSH_FRAME) called from the start of a frame)

I added only the 3.15 label for now, but @zooba requested that @hugovk make an exception to include this for 3.14. I would be okay with that if we can agree the above implementation sounds reasonable.

This makes sense to include in 3.14, please aim to merge in time for beta 3 (2025-06-17).

Great! I'll work on a PR for this a bit later today.

Thanks!