FIX update debian 32 bit CI config to avoid a SIMD related bug in numpy 1.24.2 on some GitHub Actions CI workers #29771

ogrisel · 2024-09-02T13:41:42Z

This draft PR aims at collecting debug information about the recently frequently happening failures discussed at:

⚠️ CI failed on Linux_Docker.debian_atlas_32bit (last failure: Sep 03, 2024) ⚠️ #25686 (comment)

FAILED cluster/tests/test_spectral.py::test_spectral_clustering_with_arpack_amg_solvers
FAILED feature_extraction/tests/test_image.py::test_img_to_graph - IndexError...
FAILED impute/tests/test_impute.py::test_imputation_mean_median[csc_matrix]
FAILED impute/tests/test_impute.py::test_imputation_mean_median[csc_array] - ...
FAILED preprocessing/tests/test_data.py::test_robust_scale_axis1 - IndexError...

All of these errors are IndexError that might be caused by some alternative integer arithmetic implemented by some CI workers but that cannot be reproduced locally in a docker container.

github-actions · 2024-09-02T13:42:56Z

✔️ Linting Passed

All linting checks passed. Your pull request is in excellent shape! ☀️

_{Generated for commit: 5c7751d. Link to the linter CI: here}

ogrisel · 2024-09-02T14:32:04Z

On the last run I got:

edges[0] = array([ 0,  1,  2,  4,  5,  6,  8,  9, 10, 12, 13, 14,  0,  1,  2,  3,  4,
        5,  6,  7,  8,  9, 10, 11])
n_y = 4, n_z = 1
n_y * n_z = 4
edges[0] // (n_y * n_z) = array([ 0,  1,  2,  4,  5,  6,  8,  9, 10, 12, 13, 14,  0,  1,  2,  3,  1,
        1,  1,  1,  2,  2,  2,  2], dtype=int32)

So the Euclidean division computed by NumPy is inconsistent: some results are correct (at the end of the array) but most are invalid!

Also, I don't understand why the dtype is not the same for edges[0] and edges[0] // (n_y * n_z).

…nstead of 100x

ogrisel · 2024-09-03T06:58:54Z

It seems that the problem is random on other PRs but since I added the following reproducer:

    a = np.asarray(
        [0, 1, 2, 4, 5, 6, 8, 9, 10, 12, 13, 14, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]
    )
    b = a // 4
    assert all(b <= 3)

to this PR, I have the impression that this problem is always triggered. The problem is that numpy Euclidean distance yields invalid results on a part of the array b. It could be related to a bug in a SIMD operation but numpy has not changed in a while on this build. Maybe the hardware on github actions has changed?

ogrisel · 2024-09-03T08:20:56Z

The last run did not fail. Here is the diff between a CPU that does not trigger the error (E5-2673 v4) and once that does (Platinum 8370C):

< Model name:                           Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
---
> Model name:                           Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
10c10
< Model:                                79
---
> Model:                                106
14,16c14,16
< Stepping:                             1
< BogoMIPS:                             4589.37
< Flags:                                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx smap xsaveopt md_clear
---
> Stepping:                             6
> BogoMIPS:                             5586.87
> Flags:                                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm avx512f avx512dq rdseed adx smap clflushopt avx512cd avx512bw avx512vl xsaveopt xsavec xsaves md_clear
19c19
< L1d cache:                            64 KiB (2 instances)
---
> L1d cache:                            96 KiB (2 instances)
21,22c21,34
< L2 cache:                             512 KiB (2 instances)
< L3 cache:                             50 MiB (1 instance)
---
> L2 cache:                             2.5 MiB (2 instances)
> L3 cache:                             48 MiB (1 instance)
> NUMA node(s):                         1
> NUMA node0 CPU(s):                    0,1
> Vulnerability Gather data sampling:   Unknown: Dependent on hypervisor status
> Vulnerability Itlb multihit:          KVM: Mitigation: VMX unsupported
> Vulnerability L1tf:                   Mitigation; PTE Inversion
> Vulnerability Mds:                    Mitigation; Clear CPU buffers; SMT Host state unknown
> Vulnerability Meltdown:               Mitigation; PTI
> Vulnerability Mmio stale data:        Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
> Vulnerability Reg file data sampling: Not affected
> Vulnerability Retbleed:               Not affected
> Vulnerability Spec rstack overflow:   Not affected
> Vulnerability Spec store bypass:      Vulnerabl

For reference here are the full outputs of lscpu on each run:

Incorrect integer division (test failure):

Platinum 8370C (has AVX 512)

Architecture:                         x86_64
CPU op-mode(s):                       32-bit, 64-bit
Address sizes:                        46 bits physical, 48 bits virtual
Byte Order:                           Little Endian
CPU(s):                               2
On-line CPU(s) list:                  0,1
Vendor ID:                            GenuineIntel
Model name:                           Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
CPU family:                           6
Model:                                106
Thread(s) per core:                   1
Core(s) per socket:                   2
Socket(s):                            1
Stepping:                             6
BogoMIPS:                             5586.87
Flags:                                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm avx512f avx512dq rdseed adx smap clflushopt avx512cd avx512bw avx512vl xsaveopt xsavec xsaves md_clear
Hypervisor vendor:                    Microsoft
Virtualization type:                  full
L1d cache:                            96 KiB (2 instances)
L1i cache:                            64 KiB (2 instances)
L2 cache:                             2.5 MiB (2 instances)
L3 cache:                             48 MiB (1 instance)
NUMA node(s):                         1
NUMA node0 CPU(s):                    0,1
Vulnerability Gather data sampling:   Unknown: Dependent on hypervisor status
Vulnerability Itlb multihit:          KVM: Mitigation: VMX unsupported
Vulnerability L1tf:                   Mitigation; PTE Inversion
Vulnerability Mds:                    Mitigation; Clear CPU buffers; SMT Host state unknown
Vulnerability Meltdown:               Mitigation; PTI
Vulnerability Mmio stale data:        Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed:               Not affected
Vulnerability Spec rstack overflow:   Not affected
Vulnerability Spec store bypass:      Vulnerable
Vulnerability Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:             Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Retpoline
Vulnerability Srbds:                  Not affected
Vulnerability Tsx async abort:        Mitigation; Clear CPU buffers; SMT Host state unknown

EDIT: also fails on:

Platinum 8272CL (with AVX 512) (4 runs)

Architecture:                         x86_64
CPU op-mode(s):                       32-bit, 64-bit
Address sizes:                        46 bits physical, 48 bits virtual
Byte Order:                           Little Endian
CPU(s):                               2
On-line CPU(s) list:                  0,1
Vendor ID:                            GenuineIntel
Model name:                           Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
CPU family:                           6
Model:                                85
Thread(s) per core:                   1
Core(s) per socket:                   2
Socket(s):                            1
Stepping:                             7
BogoMIPS:                             5187.81
Flags:                                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm avx512f avx512dq rdseed adx smap clflushopt avx512cd avx512bw avx512vl xsaveopt xsavec xsaves md_clear
Hypervisor vendor:                    Microsoft
Virtualization type:                  full
L1d cache:                            64 KiB (2 instances)
L1i cache:                            64 KiB (2 instances)
L2 cache:                             2 MiB (2 instances)
L3 cache:                             35.8 MiB (1 instance)
NUMA node(s):                         1
NUMA node0 CPU(s):                    0,1
Vulnerability Gather data sampling:   Unknown: Dependent on hypervisor status
Vulnerability Itlb multihit:          KVM: Mitigation: VMX unsupported
Vulnerability L1tf:                   Mitigation; PTE Inversion
Vulnerability Mds:                    Mitigation; Clear CPU buffers; SMT Host state unknown
Vulnerability Meltdown:               Mitigation; PTI
Vulnerability Mmio stale data:        Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed:               Vulnerable
Vulnerability Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:             Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Retpoline
Vulnerability Srbds:                  Not affected
Vulnerability Tsx async abort:        Mitigation; Clear CPU buffers; SMT Host state unknown

valid integer division (no test failure):

CPU E5-2673 (no AVX512) (2 runs)

Architecture:                         x86_64
CPU op-mode(s):                       32-bit, 64-bit
Address sizes:                        46 bits physical, 48 bits virtual
Byte Order:                           Little Endian
CPU(s):                               2
On-line CPU(s) list:                  0,1
Vendor ID:                            GenuineIntel
Model name:                           Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
CPU family:                           6
Model:                                79
Thread(s) per core:                   1
Core(s) per socket:                   2
Socket(s):                            1
Stepping:                             1
BogoMIPS:                             4589.37
Flags:                                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx smap xsaveopt md_clear
Hypervisor vendor:                    Microsoft
Virtualization type:                  full
L1d cache:                            64 KiB (2 instances)
L1i cache:                            64 KiB (2 instances)
L2 cache:                             512 KiB (2 instances)
L3 cache:                             50 MiB (1 instance)
Vulnerability Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:             Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Retpoline
Vulnerability Srbds:                  Not affected
Vulnerability Tsx async abort:        Mitigation; Clear CPU buffers; SMT Host state unknown

ogrisel · 2024-09-03T08:32:29Z

One noticeable difference are the AVX 512 flags on the architecture that fails. That might be a hint that the problem might be related to SIMD operations in numpy. However numpy 1.24.2 is very old and predates the highway-based optimizations I believe.

EDIT: 1.24.2 was released prior to the use of highway but there were already SIMD optimizations:

https://numpy.org/doc/1.24/reference/simd/index.html

Let me trigger a few more CI runs to check if this is reproducible.

lesteve · 2024-09-03T09:31:23Z

Inside Azure, I have seen cases where the architecture reported by OpenBLAS is SkyLakeX or Haswell. We don't have OpenBLAS in this build but I think this is somewhat similar to show the output of the following command?

head /sys/devices/cpu*/caps/pmu_name

ogrisel · 2024-09-03T09:43:46Z

@lesteve I added this command to the latest commit. However, I went trough most of the past runs in this PR (once I had enabled lscpu) and I am consolidating the number of runs for each (test success or failure) in #29771 (comment).

So far, all failures (more than 5 so far) happened with Platinum 8370C or Platinum 8272CL CPUs while the CPU E5-2673 CPU was fine twice.

So this seems to confirm that SIMD instructions are the culprit.

We would need to build a more recent version of numpy (e.g. 2.1) from source to check if this problem has been fixed with the switch to highway in numpy 2 but this seems tedious.

Alternatively we might want try to hunt for another docker image with a more recent version of python3-numpy for the 32 bit i686 platform.

lesteve · 2024-09-03T15:47:42Z

However the build time seem slower (8 min vs 3 min previously) than they used to be and I don't know why since the cache statistic look fine. However the total time is largely dominated by the test time (more than 40 min) and that does not seem to have changed.

I think we can live with this. I also thought originally that it may have been a ccache thing but like you I checked the ccache statistics which are very similar between fast and slow build so apparently not ...

ogrisel · 2024-09-03T16:01:04Z

I checked locally and trixie also ships numpy 1.26.4 so we can reasonably hope that it will fix the problem while being more stable than running apt update / install in a sid image.

ogrisel · 2024-09-03T16:17:52Z

Package libatlas3-base is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

E: Package 'libatlas3-base' has no installation candidate
E: Package 'libatlas-base-dev' has no installation candidate

f*ck. I will debug this tomorrow...

glemaitre · 2024-09-03T18:16:28Z

Indeed libatlas is not available in trixie but it is back in sid: https://packages.debian.org/sid/libatlas3-base

Edit:

Apparently it was announce that it will be removed: https://lists.debian.org/debian-science/2023/07/msg00010.html
There is additional information here regarding the status: https://tracker.debian.org/pkg/atlas

So I'm wondering if we should not stop testing against ATLAS because there is not update since 2018. Subsequently, we could modify the documentation and avoid mentioning ATLAS.

lesteve · 2024-09-04T07:37:17Z

For some reason, svm/tests/test_svm.py::test_svc_ovr_tie_breaking[NuSVC] is unstable on 32bit. I modified the test using global_random_seed, and it fails for 37 of the seeds on a Debian 32bit image. I skipped this test for now.

Full disclosure, on my machine inside the Debian 32bit image I get a segmentation fault with OpenBLAS when running pytest sklearn/metrics/tests but apparently not the CI ...

ogrisel · 2024-09-04T07:38:08Z

The test failure in test_svc_ovr_tie_breaking[NuSVC] was already reported as: #29633

sklearn/svm/tests/test_svm.py

ogrisel · 2024-09-04T07:46:25Z

I tried to use global_random_seed in test_svc_ovr_tie_breaking on a 64 bit runtime and that test always passes. So it's probably not just a problem of bad test data design unfortunately.

lesteve · 2024-09-04T07:59:36Z

Another mystery partially solved: I added ccache statistics zeroing and now this shows that there is no ccache hit, which is probably the reason why the "install" step takes longer. I don't remember for sure, but I am guessing that the cache may comes from the main branch and once this PR is merged cache reuse may be better 🤞

Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>

ogrisel

LGTM. However we need to rename the lock file and the CI config to remove "atlas" from it and avoid adding confusion about what is tested in this build. Do you want to do it @lesteve? Otherwise I can do it.

Trying to fix #29633 is low priority for me unless it affects pyodide which does not seem to be the case so far.

lesteve · 2024-09-04T08:26:48Z

However we need to rename the lock file and the CI config to remove "atlas" from it and avoid adding confusion about what is tested in this build

Good point (I thought about it at one point and forgot), done.

lesteve · 2024-09-04T09:28:00Z

OK last build is green and uses a CPU with AVX512 so let's merge this one. Thanks for debugging this one @ogrisel!

lesteve · 2024-09-04T11:25:14Z

I don't remember for sure, but I am guessing that the cache may comes from the main branch and once this PR is merged cache reuse may be better 🤞

The hypothesis seems somewhat valid as the build time is back to 2-3 minutes with a ccache hit of 100% see this PR build log.

…py 1.24.2 on some GitHub Actions CI workers (scikit-learn#29771) Co-authored-by: Loïc Estève <loic.esteve@ymail.com>

…py 1.24.2 on some GitHub Actions CI workers (#29771) Co-authored-by: Loïc Estève <loic.esteve@ymail.com>

DEBUG: print verbose info on integer ops in test_img_to_graph

a52e692

ogrisel changed the title ~~DEBUG: print verbose info on integer ops in test_img_to_graph~~ DEBUG: print verbose info on integer ops in test_img_to_graph Sep 2, 2024

github-actions bot added the module:feature_extraction label Sep 2, 2024

Do not attempt to collect coverage

8aa534d

ogrisel added 6 commits September 2, 2024 16:59

Repeat test many times

bbf38ea

Fix paren consistency

78e21de

Typo in CLI flag + speed-up collection

35f35af

Collect only in sklearn.feature_extraction and reduce repeat to 10x i…

bfb1d30

…nstead of 100x

Call lscpu before running the tests

2823e87

Start test with simplified reproducer attempt

aaa3ec5

ogrisel added the No Changelog Needed label Sep 2, 2024

ogrisel added 2 commits September 2, 2024 17:42

Trigger CI

84e2c01

Trigger CI

c21143b

ogrisel mentioned this pull request Sep 3, 2024

Fixed array api support for MinMaxScaler with clip=True #29751

Merged

ogrisel added 2 commits September 3, 2024 08:50

Trigger CI

929c71e

Trigger CI

66c7ccd

Remove the pytest-repeat thing

b77dd34

ogrisel added 3 commits September 3, 2024 10:33

Trigger CI

ecacf19

Trigger CI

acd5e85

Trigger CI

87affcc

head /sys/devices/cpu*/caps/pmu_name and trigger CI once more

b99a016

ogrisel added 2 commits September 3, 2024 11:57

Try with a more recent version of debian and python3-numpy

3b7aee2

Revert pmu_name

fd3e2ca

Try to use i386/debian:trixie instead

34dc74c

glemaitre self-requested a review September 3, 2024 18:08

glemaitre removed their request for review September 3, 2024 18:55

lesteve added 3 commits September 4, 2024 07:59

Try OpenBLAS

43a8315

[azure parallel]

fb8b849

Zero ccache statistics

3db9c21

glemaitre mentioned this pull request Sep 4, 2024

FIX accept infinite C in SVC and SVR #29780

Merged

[azure parallel] Skip NuSVC test on 32bit

83f8be0

ogrisel commented Sep 4, 2024

View reviewed changes

sklearn/svm/tests/test_svm.py Outdated Show resolved Hide resolved

lesteve mentioned this pull request Sep 4, 2024

test_svm fails on i386 with scipy 1.13 #29633

Open

[azure parallel] Update sklearn/svm/tests/test_svm.py

686f56c

Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>

lesteve mentioned this pull request Sep 4, 2024

Add _voting.py file with voting regressor implementation #29724

Merged

ogrisel commented Sep 4, 2024

View reviewed changes

[azure parallel] Remove atlas from build name

5c7751d

lesteve merged commit d0a5fdf into scikit-learn:main Sep 4, 2024
29 checks passed

lesteve mentioned this pull request Sep 4, 2024

⚠️ CI failed on Linux_Docker.debian_atlas_32bit (last failure: Sep 03, 2024) ⚠️ #25686

Closed

ogrisel deleted the debug-debian-32-indexerror branch September 4, 2024 12:15

glemaitre pushed a commit that referenced this pull request Sep 11, 2024

FIX update debian 32 bit CI config to avoid a SIMD related bug in num…

9913ed6

…py 1.24.2 on some GitHub Actions CI workers (#29771) Co-authored-by: Loïc Estève <loic.esteve@ymail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

FIX update debian 32 bit CI config to avoid a SIMD related bug in numpy 1.24.2 on some GitHub Actions CI workers #29771

FIX update debian 32 bit CI config to avoid a SIMD related bug in numpy 1.24.2 on some GitHub Actions CI workers #29771

ogrisel commented Sep 2, 2024

github-actions bot commented Sep 2, 2024 •

edited

Loading

ogrisel commented Sep 2, 2024 •

edited

Loading

ogrisel commented Sep 3, 2024

ogrisel commented Sep 3, 2024 •

edited

Loading

ogrisel commented Sep 3, 2024 •

edited

Loading

lesteve commented Sep 3, 2024 •

edited

Loading

ogrisel commented Sep 3, 2024 •

edited

Loading

lesteve commented Sep 3, 2024

ogrisel commented Sep 3, 2024

ogrisel commented Sep 3, 2024

glemaitre commented Sep 3, 2024 •

edited

Loading

lesteve commented Sep 4, 2024

ogrisel commented Sep 4, 2024

ogrisel commented Sep 4, 2024

lesteve commented Sep 4, 2024 •

edited

Loading

ogrisel left a comment

lesteve commented Sep 4, 2024

lesteve commented Sep 4, 2024

lesteve commented Sep 4, 2024 •

edited

Loading

FIX update debian 32 bit CI config to avoid a SIMD related bug in numpy 1.24.2 on some GitHub Actions CI workers #29771

FIX update debian 32 bit CI config to avoid a SIMD related bug in numpy 1.24.2 on some GitHub Actions CI workers #29771

Conversation

ogrisel commented Sep 2, 2024

github-actions bot commented Sep 2, 2024 • edited Loading

✔️ Linting Passed

ogrisel commented Sep 2, 2024 • edited Loading

ogrisel commented Sep 3, 2024

ogrisel commented Sep 3, 2024 • edited Loading

Incorrect integer division (test failure):

valid integer division (no test failure):

ogrisel commented Sep 3, 2024 • edited Loading

lesteve commented Sep 3, 2024 • edited Loading

ogrisel commented Sep 3, 2024 • edited Loading

lesteve commented Sep 3, 2024

ogrisel commented Sep 3, 2024

ogrisel commented Sep 3, 2024

glemaitre commented Sep 3, 2024 • edited Loading

lesteve commented Sep 4, 2024

ogrisel commented Sep 4, 2024

ogrisel commented Sep 4, 2024

lesteve commented Sep 4, 2024 • edited Loading

ogrisel left a comment

Choose a reason for hiding this comment

lesteve commented Sep 4, 2024

lesteve commented Sep 4, 2024

lesteve commented Sep 4, 2024 • edited Loading

github-actions bot commented Sep 2, 2024 •

edited

Loading

ogrisel commented Sep 2, 2024 •

edited

Loading

ogrisel commented Sep 3, 2024 •

edited

Loading

ogrisel commented Sep 3, 2024 •

edited

Loading

lesteve commented Sep 3, 2024 •

edited

Loading

ogrisel commented Sep 3, 2024 •

edited

Loading

glemaitre commented Sep 3, 2024 •

edited

Loading

lesteve commented Sep 4, 2024 •

edited

Loading

lesteve commented Sep 4, 2024 •

edited

Loading