Thanks for the pertinent remark:

• security is the main issue of nowodays
• shrinking to a requirement.txt is the ultimate simplification goal.

So:

• removing dead wood right now:
  • adodbapi
  • db.py
  • formlayout
  • ppci
  • PySimpleGUI ..., prefers to remove it as removing old version from pypi is not good practices
  • pdfrw==0.4.post2 , is certainly from fork pdfrw2... but seeing the few downloads of the fork I didn't dare... removing and updating to pypdf to pypdf-5.1.0
• re-published:
  • baresql
  • sqlite_bro
• special:
  • winpython: that one is truly a special case... or is it ?
  • and the pypi project remains under Pierre Raybaut account, so wouldhave to change the wheel

So:

• there is the two not republished wheels, to sacrifice if needed,
• the next problem is the winpython tiny "envelop" rethinking.

And how to right now generate a requirement.txt with Sha 256 ?

I think pip is not ready for that yet, isn't it ?

And how to right now generate a requirement.txt with Sha 256 ?

I think pip is not ready for that yet, isn't it ?

I don’t believe so… however uv pip compile can generate requirements.txt containing hashes from a requirements.in with the --generate-hashes flag.

Per this issue the hashes seem to be for wheels of the selected platform.

Also somewhat related and could be useful: astral-sh/uv#1681

so, after a looong fight, published:

• baresql-0.8.0
• baresql-1.0.0
• sqlite_bro-0.13.1

last problem... down to winpython wheel or not to wheel , the complex one

last problem... down to winpython wheel or not to wheel , the complex one

I would be curious to see if it changes the size of the distribution. It might be beneficial to test it out as an alpha build (experimental)?

Let me know if you would like support with anything! I can try forking the project and testing things out if there’s guidance anywhere on how to build the executables/shims 😄

I would bet it will double the size, unless the wheel comitee decides to allow a better compression format, like zstd

... need to rethink the build system

replaying the idea, to feel the pain points:

echo %date% %time%&python -m pip freeze>req.txt
rem#comment the winpython wheel... otherwise we must publish before building...
rem#pip is not in a pip freeze
echo %date% %time% &python -m pip download -r req.txt -d wheels
echo %date% %time%

size of wheels:

• 729 Mo uncompressed
• 717 Mo zipped
• 701 Mo .7z base per windows
• .. versus 587 Mo current winpython.7z

fight non-whl format wheels, 8 wheels: filterpy, pywinusb, simplegeneric, docopt, intervaltree, MarkupSafe, msgpack, sspyrs

@stonebig

Confirmed:

• 729 MB (uncompressed)
• 592 MB (current Winpython64-3.13.2.0slimb3.7z)

I was able to get down to 687 MB using maximum compression settings with 7z.

7z a -t7z -mx9 -myx9 -mf=BCJ2 -m0=LZMA2:d3840m:fb273 -ms -mmt2 -slp wheels.7z wheels\

It is still ~100 MB larger than it was previously. This is a problem because the maximum file size for VirusTotal uploads is 650 MB 🤔

Wheels are better than sdist because user doesn't have to worry about building them. But the compression could be better and they have additional metadata bloating the wheel file size that is more valuable to the package manager than the user.

The natural bad trend on wheel size balooning is hard to fight:

• optimism: JIT, signed wheels, Microsoft providing AI infrastructure on the OS, cpython developer being more aware of the problem
• pessimism: AI let you add "features" to your wheel with ease, whatever hardware gives you, Software will take it away.

what shall be the format of a pip freeze + hashtag file ?

requirements.txt supports --hash values in it 🙂

I found out that it is still possible to validate the checksums of Python packages after they have already been installed. The hashes of all the package files in Lib/site-packages can be calculated and then compared to the hashes in Lib/site-packages/<pkg>-<ver>.dist-info/RECORD. Relevant discussion: astral-sh/uv#11723

Method A

To generate a reproducible and standalone environment with installed packages:

@echo off
echo Downloading python-build-standalone . . .
curl -LSsfo py.tar.gz https://github.com/astral-sh/python-build-standalone/releases/download/20250212/cpython-3.13.2+20250212-x86_64-pc-windows-msvc-install_only_stripped.tar.gz
tar -xf py.tar.gz
del py.tar.gz
cd python
curl -LSsfo uv.zip https://github.com/astral-sh/uv/releases/download/0.6.2/uv-x86_64-pc-windows-msvc.zip
tar -xf uv.zip
del uvx.exe
del uv.zip
echo Downloading winpython . . .
curl -LSsfo wpy.7z https://github.com/winpython/winpython/releases/download/13.1.202502222/Winpython64-3.13.2.0slimrc.7z
echo Extracting winpython . . .
tar -xf wpy.7z
uv pip freeze --python WPy64-31320/python/python.exe > requirements.in
rmdir /s /q WPy64-31320
del wpy.7z
powershell -c "(gc requirements.in) | ? { $_ -notmatch 'winpython' } | sc requirements.in"
uv pip compile --no-cache --python python.exe --generate-hashes --output-file requirements.txt requirements.in
del requirements.in
echo Reproducing winpython . . .
uv pip install --no-cache --python python.exe --require-hashes --requirements requirements.txt
del uv.exe
cd .. 
echo Zipping up winpython . . .
cmd /c "C:\Program Files\7-Zip\7z.exe" a -t7z -mx9 -myx9 -mf=BCJ2 -m0=LZMA2:d3840m:fb273 -ms -mmt2 -slp winpython.7z python

Method B

To generate a reproducible and standalone environment with wheels only (no installed packages):

@echo off
echo Downloading python-build-standalone . . .
curl -LSsfo py.tar.gz https://github.com/astral-sh/python-build-standalone/releases/download/20250212/cpython-3.13.2+20250212-x86_64-pc-windows-msvc-install_only_stripped.tar.gz
tar -xf py.tar.gz
del py.tar.gz
cd python
curl -LSsfo uv.zip https://github.com/astral-sh/uv/releases/download/0.6.2/uv-x86_64-pc-windows-msvc.zip
tar -xf uv.zip
del uvx.exe
del uv.zip
echo Downloading winpython . . .
curl -LSsfo wpy.7z https://github.com/winpython/winpython/releases/download/13.1.202502222/Winpython64-3.13.2.0slimrc.7z
echo Extracting winpython . . .
tar -xf wpy.7z
uv pip freeze --python WPy64-31320/python/python.exe > requirements.in
rmdir /s /q WPy64-31320
del wpy.7z
powershell -c "(gc requirements.in) | ? { $_ -notmatch 'winpython' } | sc requirements.in"
uv pip compile --no-cache --python python.exe --generate-hashes --output-file requirements.txt requirements.in
del requirements.in
echo Downloading packages . . .
python -m pip download --require-hashes --dest pkgs --requirement requirements.txt 
echo Validating checksums of downloaded packages . . .
powershell -c "$valid = $true; gci -path pkgs -file | % { $file = $_; $hash = (get-filehash -path $file.fullname -algorithm sha256).hash; if (-not (sls -path requirements.txt -pattern $hash)) { $valid = $false; write-host $file.name: invalid hash } }; if ($valid) { write-host All hashes valid. }"
del uv.exe
cd .. 
echo Zipping up winpython . . .
cmd /c "C:\Program Files\7-Zip\7z.exe" a -t7z -mx9 -myx9 -mf=BCJ2 -m0=LZMA2:d3840m:fb273 -ms -mmt2 -slp winpython.7z python

Alternative to the powershell one-liner... this little python script will check hashes of wheels in a directory against a requirements.txt containing hashes. Should theoretically work with any directory and any file containing valid sha256 hash strings.

from argparse import ArgumentParser
from hashlib import sha256
from pathlib import Path


def validate_hashes(file_path: str, dir_path: str) -> None:
    files = Path(dir_path).rglob("*")
    hashes = Path(file_path).read_text()
    invalid_hashes: list[Path] = []
    for file in files:
        hash = sha256(file.read_bytes()).hexdigest()
        if hash not in hashes:
            invalid_hashes.append(file)
    for file in invalid_hashes:
        print(f"{file.name}: invalid hash")
    if not invalid_hashes:
        print("All hashes valid.")


if __name__ == "__main__":
    parser = ArgumentParser(description="Check files for valid SHA-256 hashes")
    parser.add_argument("file", type=str, help="Path to file containing SHA-256 hashes")
    parser.add_argument("dir", type=str, help="Path to directory of files to check")
    args = parser.parse_args()
    validate_hashes(args.file, args.dir)

On the wheel summit, that happened March 21, 2025, 9AM-1.30PM @ META - Menlo Park, CA:

• all presentations: https://wheelnext.dev/summits/2025_03/slidedecks_and_resources/
• one presenter suggests to add the zstandard compression
• another presenter suggests to put the hash in the wheelname

There was a discussion 4 years ago, hinting to zstd-19 being 40% better : https://discuss.python.org/t/making-the-wheel-format-more-flexible-for-better-compression-speed/3810/23

Good news: PEP 751 was accepted and includes a hashes property for the wheels.

I saw, that is the breakthrough needed for the idea.
I would forsee:

• wppm "install" would be the way to tap into a local wheelhouse
• problems of wheel size and wheel unpacking-speed seem not adressed by zlib-ng, before or after compression.

experiment

echo %date% %time%&python -m pip freeze>req.txt
findstr /v "winpython" req.txt > wanted_req.txt
echo %date% %time% &python -m pip download -r wanted_req.txt -d wheels
echo %date% %time% &python compress_folder_with_zstd.py ./Wheels ./Wheels.tar.zst --level 16
echo %date% %time%

so:

• 760 327 Kb uncompressed
• 735 065 Kb in 2' with level 16
• 726 664 Kb in 4' with level 19
• 720 747 KB in 6' with level 21

echo %date% %time%
23/04/2025 12:26:56,56
python compress_folder_with_zstd.py ./Wheels ./Wheels.tar.zst --level 16&echo %date% %time%
📦 Creating TAR archive: .\Wheels.tar.tar
🗜️ Compressing to Zstandard: .\Wheels.tar.zst
✅ Done: Wheels.tar.zst (735065 KB)
23/04/2025 12:28:56,49

echo %date% %time% 
23/04/2025 13:31:00,74
python compress_folder_with_zstd.py ./Wheels ./Wheels19.tar.zst --level 19
📦 Creating TAR archive: C:\WinP\bd313\buslim\WPy64-31330b3\notebooks\Wheels19.tar.tar
🗜️ Compressing to Zstandard: C:\WinP\bd313\buslim\WPy64-31330b3\notebooks\Wheels19.tar.zst
✅ Done: Wheels19.tar.zst (726664 KB)
echo %date% %time%
23/04/2025 13:34:58,39

echo %date% %time% &python compress_folder_with_zstd.py ./Wheels ./Wheels21.tar.zst --level 21
23/04/2025 14:16:45,71
📦 Creating TAR archive: .\Wheels21.tar.tar
🗜️ Compressing to Zstandard: .\Wheels21.tar.zst
✅ Done: Wheels21.tar.zst (720747 KB)
echo %date% %time%
23/04/2025 14:22:51,96

and to restores wheels:

echo %date% %time%&python extract_tar_archive.py wheels19.tar.zst ./extracted_wheels
echo %date% %time%

gives :

echo %date% %time%
23/04/2025 13:57:04,01
🗜️ Decompressing Zstandard archive: wheels19.tar.zst
📂 Extracting TAR: wheels19
.\extract_tar_archive.py:21: DeprecationWarning: Python 3.14 will, by default, filter extracted tar archives and reject files or modify their metadata. Use the filter argument to control this behavior.
  tar.extractall(path=output_dir)
✅ Done! Extracted to: .\extracted_wheels
echo %date% %time%
23/04/2025 13:57:31,43

so, apparently:

• current wheelhouse is 23% bigger than current final WinPython
• pre-compressing Wheelhouse with zstd-19 gains 4%
• un-wheeling / re-wheeling with zopfli may gain 10%, but we loose the whole PEP-751 point
• .... so:
  • there is no Wheelhouse "size" solution, if not resolved per wheelnext (or even before in cpython bytecode way)
  • providing a lockfile will be practical

doing an extreme proof-of-concept example using zstandard-0.23/zstd-1.5.6:

• unzipping the 8 biggest wheels (weighting 295 Mo), so the top 40% weight of Wheels total weight
• that gives an unzipped directory of 852 Mo: compression factor of x2.88
• doing zstd-21 on them gives a tar.zst file of ... 202 Mo in 13', x1.46 so more than x1.22 whished to keep WinPython size
• doing zstd-16 on them gives a tar.zst file of ... 228 Mo in 3.5', x1.29 so more than x1.22 whished to keep WinPython size
• doing zstd-12 on them gives a tar.zst file of ... 239 Mo in 0.9', x1.23 so more than x1.22 whished to keep WinPython size

so... to remain equivalent in size of a "full" Winython download, having wheelnext advocating for tar.zst at compression level 12 would be the ideal solution, to make the idea "neutral" in Wheelhouse-included download size of WinPython.

Level 12 is the last "fast" level of zstd, based on old stackoverflow post here

echo %date% %time% &python compress_folder_with_zstd.py ./un_zipped_wheels ./un_zipped_wheels21.tar.zst --level 21
23/04/2025 14:27:47,57
📦 Creating TAR archive: .\un_zipped_wheels21.tar.tar
🗜️ Compressing to Zstandard: .\un_zipped_wheels21.tar.zst
✅ Done: un_zipped_wheels21.tar.zst (207122 KB)
echo %date% %time%
23/04/2025 14:40:49,10

echo %date% %time% &python compress_folder_with_zstd.py ./un_zipped_wheels ./un_zipped_wheels16.tar.zst --level 16
23/04/2025 14:46:18,45
📦 Creating TAR archive: .\un_zipped_wheels16.tar.tar
🗜️ Compressing to Zstandard: .\un_zipped_wheels16.tar.zst
✅ Done: un_zipped_wheels16.tar.zst (234463 KB)
echo %date% %time%
23/04/2025 14:49:45,50

echo %date% %time% &python compress_folder_with_zstd.py ./un_zipped_wheels ./un_zipped_wheels12.tar.zst --level 12
23/04/2025 14:53:32,33
📦 Creating TAR archive: .\un_zipped_wheels12.tar.tar
🗜️ Compressing to Zstandard: .\un_zipped_wheels12.tar.zst
✅ Done: un_zipped_wheels12.tar.zst (245180 KB)
echo %date% %time%
23/04/2025 14:54:22,15

Apparently, "pip" will be very slow to include PEP-751 lockfile.toml, like 2026.
But, for sure, your previous remark makes sense:
And how to right now generate a requirement.txt with Sha 256

to be then used as

python -m pip install --pre -r "requirements_Winpython64-3.13.3.0slim_with_hash.txt" --no-deps

Comparing wheels from internal wheelhouse and pypi, via hash-256:

• wheels from cgholke aera (who was most trusted and respected at the time, a big element of intel/Python success):
  • about 10 wheels where py3, when it's py2-py3 on pypi
  • about 7 wheels with some 4 characters difference, somthing about lineReturn
• then about 5 wheels with vast difference:
  • missing or adding, licence or test or readme
  • not sure if it's cgohlke or a re-loaded wheels per the owner, rather thinking it's a reload
  • checked one by one before switching back to pypi version.
• wheels carefully switched back to pypi version: aniso8601, appdirs, atomicwrites, binaryornot, dateutil, deprecated, defusedxml, et-xmlfile, html5lib, ipython-genutils, pep8, pexpect, ply, ptyprocess, pyls-spyder, python-pyserial, Rx, six, simpy, sortedcontainers, text-unidecode, toml, traittypes

Remains:

• intervaltree: switch from .whl local to tar.gz of pypi
• pypandoc wheel: current WinPython one is empty, while the pypi one is loaded with Pandoc.exe ... till version 1.7... so we my upgrade
• WinPython wheel.

pypandoc problem is fixed

intervaltree has no good solution:

• no alternative package except bx-python,
• has no a "deprecation warning" from recent setuptools-0.72
• ... there is a need of a fork or re-vitalization of that package.

WinPython wheel doesn't make sense yet, it's a bit a fire-and-forget with each release, WinPython way of free support.
nevertheless it has shrunk a lot in lines of code again this 2025-02 cycle, on the path to "just a lockfile".

....further brewing needed.

I did preparre a prototype of an ugly post-hash generator, to track down the not-from-pypi-default wheels:

• was helpfull for that final gap closing
• shall help to generate a legal lockfile, in waiting of practical tools

problem: how to know it's a correct lockfile generator

# generate_lockfiles and hashed-requirements
import argparse
import json
import subprocess
import sys
import hashlib
from pathlib import Path
#import tomli_w  # pip install toml
import re
import tomli_w
import sys
import platform

def normalize(name):
    return re.sub(r"[-_.]+", "-", name).lower()

def get_wheels( requirements,  from_local, wheeldir):
    added = []
    if from_local:
        added += ['--no-index' , '--trusted-host=None' , f'--find-links={from_local}']
    instruction = [sys.executable, "-m", "pip", "download", "-r",  requirements , "--no-deps", "--dest", wheeldir] + added
    print('download instruction:', instruction)
    subprocess.run(
        instruction,  # , "--only-binary=:all:"
        check=True,
        #stdout=subprocess.DEVNULL,
        #stderr=subprocess.DEVNULL
    )
def get_package_hashes(package_name, version, from_local,    wheeldir=None):
    """Retrieve SHA256 hashes for a package version by downloading its wheel."""
    added = []
    if from_local:
        added += ['--no-index' , '--trusted-host=None' , f'--find-links={from_local}']
    instruction = [sys.executable, "-m", "pip", "download", f"{package_name}=={version}", "--no-deps", "--dest", wheeldir] + added
    #print('\ninstruction:', instruction)
    #print(wheeldir)
    # subprocess.run( instruction,  check=True,  stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL  )
    wheel_files = list(Path(wheeldir).glob(f"{package_name.replace('-', '_')}-{version}*.whl"))
    wheel_files += list(list(Path(wheeldir).glob(f"{package_name.replace('-', '_')}-{version}*.tar.gz")))
    hashes = {}
    sizes = {}
    for wheel in wheel_files:
        with open(wheel, "rb") as f:
            sha256_hash = hashlib.sha256(f.read()).hexdigest()
            # md5 = hashlib.md5(f.read()).hexdigest()
            # blake2b  = hashlib.blake2b(f.read(), digest_size=32).hexdigest()
            hashes[wheel.name] = {"sha256" : sha256_hash}  #, "md5" : md5, "blake2b" : blake2b } 
            sizes[wheel.name] = Path(wheel).stat().st_size
    return hashes, sizes

def parse_requirements(requirements_file):
    """Parse a requirements.txt file into package name/version pairs."""
    with open(requirements_file, "r") as f:
        lines = f.readlines()
    packages = []
    for line in lines:
        line = line.strip()
        if not line or line.startswith("#"):
            continue
        if "==" not in line:
            print(f"Skipping unsupported requirement: {line}")
            continue
        name, version = line.split("==")
        packages.append({"name": name.strip(), "version": version.strip()})
    return packages

def generate_lockfile(packages, output_file, sha_requirement,from_local, wheeldir):
    """Generate a PEP 751-style TOML lockfile."""
    lockfile = {
        "lock-version": "1.0",
        "environments": ["sys_platform == 'win32'"],
        "requires-python": f'=={sys.version_info.major}.{sys.version_info.minor}',
        "extras": [],
        "dependency-groups": [],
        "default-groups": [],
        "created-by": "WinPython generate-lockfile",
        "packages": []
    }

    reqhashedfile = []

    for pkg in packages:
        hashes, sizes = get_package_hashes(pkg["name"], pkg["version"], from_local , wheeldir)
        for filename, sha256 in hashes.items():
            lockfile["packages"].append({
                "name": normalize(pkg["name"]),
                "version": pkg["version"],
                "wheels": [
                    {
                        "name": filename,
                        "hashes": sha256,
                        "size": f"{sizes[filename]}",
                    }
                ]
            })
            reqhashedfile += [f"{pkg["name"]}=={pkg['version']} --hash=sha256:{sha256['sha256']}  # size={sizes[filename]}"]

    with open(output_file, "wb") as f:
        tomli_w.dump(lockfile, f)
    
    # with open("requirements."+ output_file +".txt", "w", encoding="utf-8") as f:args.hashed_requirement
    with open(sha_requirement, "w", encoding="utf-8") as f:
        f.write("\n".join(reqhashedfile))

def main():
    versioningreq=f'requirement_hash.WinPythonf{platform.architecture()[0][:2]}-{sys.version_info.major}.{sys.version_info.minor}.{sys.version_info.micro}.x.txt'
    versioningtoml=f'pylock.WinPythonf{platform.architecture()[0][:2]}-{sys.version_info.major}.{sys.version_info.minor}.{sys.version_info.micro}.x.toml'
    parser = argparse.ArgumentParser(description="Generate a PEP 751-like TOML lockfile from requirements.txt.")
    parser.add_argument(
        "-r", "--requirements",
        default="wanted_req.txt",
        help="Path to the requirements.txt file (default: requirements.txt)"
    )
    parser.add_argument(
        "-l", "--from_local",
        default=None,
        help="Path to the requirements.txt file (default: requirements.txt)"
    )
    parser.add_argument(
        "-o", "--output",
        default=versioningtoml,
        help="Output file path for the lockfile (default: pylock.toml)"
    )
   
    parser.add_argument(
        "-s", "--sha_requirement",
        default=versioningreq,
        help="Output file path for the lockfile (default: requirement_hash.txt)"
    )
    args = parser.parse_args()

    wheeldir = r".\tmp_wheelsv2_local" if args.from_local else r".\tmp_wheelsv2_web"
    Path(wheeldir).mkdir(exist_ok=True)
    get_wheels(args.requirements, args.from_local, wheeldir)
    packages = parse_requirements(args.requirements)
    generate_lockfile(packages, args.output, args.sha_requirement, args.from_local, wheeldir)
    print(f"TOML Lockfile written to {args.output}")

if __name__ == "__main__":
    main()

PEP 784 is accepted; Zstandard arrives in the standard library https://peps.python.org/pep-0784/

we are at step1: STANDARD lockfiles are generated during build process, per standard pip-25.1.1