st.c redefines malloc and free to be ruby_xmalloc and ruby_xfree, so when this was copied into hash.c it ended up mismatching an xmalloc with a regular free 😅, which ended up inflating oldmalloc_increase_bytes when hashes were freed by minor GC.

We found this by testing Ruby trunk in production, which showed a significant increase in major GC by cause of oldmalloc. After we were able to reproduce oldmalloc_increase_bytes increasing (semi-)consistently in a synthetic test. Finally I tracked down this location by enabling CALC_EXACT_MALLOC_SIZE and building with ASAN (which crashes consistently when trying to free an xmalloc).

Before:

$ ./miniruby -e '10.times { 1000.times { (0..10).map{[_1,_1]}.to_h }; GC.start(full_mark: false); puts GC.stat(:oldmalloc_increase_bytes) }'
169728
562752
946752
1330752
1714752
2098752
2482752
2866752
3250752
3634752

After

$ ./miniruby -e '10.times { 1000.times { (0..10).map{[_1,_1]}.to_h }; GC.start(full_mark: false); puts GC.stat(:oldmalloc_increase_bytes) }'
96
9120
9120
9120
9120
9120
9120
9120
9120
9120