Skip to content

Built-in modules inconsistent in different systems. #132834

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
kylincodelab opened this issue Apr 23, 2025 · 5 comments
Closed

Built-in modules inconsistent in different systems. #132834

kylincodelab opened this issue Apr 23, 2025 · 5 comments
Labels
type-bug An unexpected behavior, bug, or error

Comments

@kylincodelab
Copy link

kylincodelab commented Apr 23, 2025

Bug report

Bug description:

import zlib

def zip(e):
    if isinstance(e, str):
        e = e.encode('utf-8')
    return zlib.compress(e, zlib.Z_DEFAULT_COMPRESSION, zlib.MAX_WBITS | 16)

name = 'hello kycode'
res = zip(name)
result = [x for x in res]
print(result)

The zlib module yields inconsistent results across different systems.

ubuntu 24

[31, 139, 8, 0, 0, 0, 0, 0, 0, 3, 203, 72, 205, 201, 201, 87, 200, 174, 76, 206, 79, 73, 5, 0, 133, 105, 82, 3, 12, 0, 0, 0]

win 10 arm

[31, 139, 8, 0, 0, 0, 0, 0, 0, 10, 203, 72, 205, 201, 201, 87, 200, 174, 76, 206, 79, 73, 5, 0, 133, 105, 82, 3, 12, 0, 0, 0]

CPython versions tested on:

3.11

Operating systems tested on:

No response

@kylincodelab kylincodelab added the type-bug An unexpected behavior, bug, or error label Apr 23, 2025
@StanFromIreland
Copy link
Contributor

StanFromIreland commented Apr 23, 2025

This should be closed as the zlib module is not an expected part of the stdlib, and this issue is because Windows uses a packaged version of the zlib C library whereas Linux can use others. I presume they both work in decompression?

@kylincodelab
Copy link
Author

you can test it yourself! presume what?

@emmatyping
Copy link
Member

By passing zlib.MAX_WBITS | 16 the wbits value is 31. According to the docs:

+25 to +31 = 16 + (9 to 15): Uses the low 4 bits of the value as the window size logarithm, while including a basic gzip header and trailing checksum in the output.

Emphasis mine. Per the gzip spec, the 9th byte in the header is the OS marker:

This identifies the type of file system on which compression took place

So I believe that the code is working as intended. If you want just a zlib stream, you probably want to remove the | 16. If you want a GZip stream, you need to normalize the OS byte after calling zlib.compress.

@ZeroIntensity ZeroIntensity added the pending The issue will be closed if no feedback is provided label Apr 24, 2025
@kylincodelab
Copy link
Author

By passing zlib.MAX_WBITS | 16 the wbits value is 31. According to the docs:

+25 to +31 = 16 + (9 to 15): Uses the low 4 bits of the value as the window size logarithm, while including a basic gzip header and trailing checksum in the output.

Emphasis mine. Per the gzip spec, the 9th byte in the header is the OS marker:

This identifies the type of file system on which compression took place

So I believe that the code is working as intended. If you want just a zlib stream, you probably want to remove the | 16. If you want a GZip stream, you need to normalize the OS byte after calling zlib.compress.

Thanks, but i use javascript or nodejs in in different systems, it work the same result. this make me in trouble...

@emmatyping
Copy link
Member

I'm going to close this since zlib is working as intended.

A potential solution to make a gzip output that is OS independent would be along the lines of:

def zip(e):
    if isinstance(e, str):
        e = e.encode('utf-8')
    b = bytearray(zlib.compress(e, zlib.Z_DEFAULT_COMPRESSION, zlib.MAX_WBITS | 16))
    b[9] = 255 # unknown OS, for compatibility, could also set to 3, for UNIX
    return bytes(b)

Note that the GZip format includes other metadata that is not reproducible such as the mtime field, so the above solution may not be complete.

@emmatyping emmatyping closed this as not planned Won't fix, can't repro, duplicate, stale Apr 24, 2025
@emmatyping emmatyping removed the pending The issue will be closed if no feedback is provided label Apr 24, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type-bug An unexpected behavior, bug, or error
Projects
None yet
Development

No branches or pull requests

4 participants