Skip to content

"broken" since go 1.21 #370

New issue
New issue
Open
Open
"broken" since go 1.21#370
@asottile

Description

@asottile
asottile
opened on Apr 7, 2025

I say "broken" because according to the golang issue this was never intended to work and was merely working by accident: golang/go#65050 (comment)

here's a small reproduction demoing this breakage. I'm running on macos x86_64 which seems to be the most reliable to reproduce for some reason I don't completely understand:

setup:

$ # not shown: running the install steps in the README
$ gopy build -output=hi -vm=python3 github.com/go-python/gopy/_examples/hi
go build -v github.com/go-python/gopy/_examples/hi

--- Processing package: github.com/go-python/gopy/_examples/hi ---

--- building package ---
gopy build -output=hi -vm=python3 github.com/go-python/gopy/_examples/hi
goimports -w hi.go
go build -mod=mod -buildmode=c-shared -o hi_go.so .
/Users/asottile/workspace/z/venv/bin/python3 build.py
CGO_CFLAGS="-I/Library/Frameworks/Python.framework/Versions/3.9/include/python3.9" -fPIC -Ofast
CGO_LDFLAGS="-L/Library/Frameworks/Python.framework/Versions/3.9/lib" "-lpython3.9" -ldl -framework CoreFoundation
go build -mod=mod -buildmode=c-shared -o _hi.cpython-39-darwin.so .
$ gopy build -output=simple -vm=python3 github.com/go-python/gopy/_examples/simple
go build -v github.com/go-python/gopy/_examples/simple

--- Processing package: github.com/go-python/gopy/_examples/simple ---

--- building package ---
gopy build -output=simple -vm=python3 github.com/go-python/gopy/_examples/simple
goimports -w simple.go
go build -mod=mod -buildmode=c-shared -o simple_go.so .
/Users/asottile/workspace/z/venv/bin/python3 build.py
CGO_CFLAGS="-I/Library/Frameworks/Python.framework/Versions/3.9/include/python3.9" -fPIC -Ofast
CGO_LDFLAGS="-L/Library/Frameworks/Python.framework/Versions/3.9/lib" "-lpython3.9" -ldl -framework CoreFoundation
go build -mod=mod -buildmode=c-shared -o _simple.cpython-39-darwin.so .

reproduction

here's a small python script using the compiled output:

(all that really needs to be involved is >1 extension in the same process)

from hi.hi import Hello
from simple.simple import Add

for _ in range(5000):
    Add(2, 2)
    Hello('hi')

boom

(and sometimes the crash is in various other places -- this just seems to be the most common one)

$ python3 t.py 
fatal error: bad sweepgen in refill

goroutine 17 gp=0x1c000006700 m=1 mp=0x1c000048008 [running, locked to thread]:
runtime.throw({0x106e672a0?, 0x0?})
	/Users/asottile/opt/go/src/runtime/panic.go:1067 +0x48 fp=0x1c000056af8 sp=0x1c000056ac8 pc=0x106e12848
runtime.(*mcache).refill(0x10dc925b8, 0x0?)
	/Users/asottile/opt/go/src/runtime/mcache.go:157 +0x20d fp=0x1c000056b38 sp=0x1c000056af8 pc=0x106dbf04d
runtime.(*mcache).nextFree(0x10dc925b8, 0x34)
	/Users/asottile/opt/go/src/runtime/malloc.go:945 +0x85 fp=0x1c000056b70 sp=0x1c000056b38 pc=0x106db9d05
runtime.mallocgc(0x200, 0x106ea7020, 0x1)
	/Users/asottile/opt/go/src/runtime/malloc.go:1161 +0x4cd fp=0x1c000056c10 sp=0x1c000056b70 pc=0x106e0eb8d
runtime.makeslice(0x0?, 0x1c000056c78?, 0x106dbdc85?)
	/Users/asottile/opt/go/src/runtime/slice.go:116 +0x49 fp=0x1c000056c38 sp=0x1c000056c10 pc=0x106e14149
sync.(*Pool).pinSlow(0x106f4bb60)
	/Users/asottile/opt/go/src/sync/pool.go:241 +0x17c fp=0x1c000056cd0 sp=0x1c000056c38 pc=0x106e1ed7c
sync.(*Pool).pin(0x106f4bb60)
	/Users/asottile/opt/go/src/sync/pool.go:220 +0x46 fp=0x1c000056cf0 sp=0x1c000056cd0 pc=0x106e1ebc6
sync.(*Pool).Get(0x106f4bb60)
	/Users/asottile/opt/go/src/sync/pool.go:135 +0x1c fp=0x1c000056d28 sp=0x1c000056cf0 pc=0x106e1e91c
fmt.newPrinter()
	/Users/asottile/opt/go/src/fmt/print.go:152 +0x1e fp=0x1c000056d50 sp=0x1c000056d28 pc=0x106e4475e
fmt.Sprintf({0x106e66215, 0x11}, {0x1c000056dd8, 0x1, 0x1})
	/Users/asottile/opt/go/src/fmt/print.go:238 +0x30 fp=0x1c000056da8 sp=0x1c000056d50 pc=0x106e44b50
github.com/go-python/gopy/_examples/cpkg.Hello({0x1c00009a00a?, 0x10db58ca9?})
	/Users/asottile/workspace/z/go/pkg/mod/github.com/go-python/gopy@v0.4.10/_examples/cpkg/cpkg.go:33 +0x74 fp=0x1c000056e10 sp=0x1c000056da8 pc=0x106e4d874
github.com/go-python/gopy/_examples/hi.Hello(...)
	/Users/asottile/workspace/z/go/pkg/mod/github.com/go-python/gopy@v0.4.10/_examples/hi/hi.go:34
main.hi_Hello(0x7ffbcb6e07e0, 0x0)
	/Users/asottile/workspace/z/hi/hi.go:1708 +0xd8 fp=0x1c000056e60 sp=0x1c000056e10 pc=0x106e553d8
_cgoexp_ffdfbdb0b480_hi_Hello(0x106deb486?)
	_cgo_gotypes.go:2033 +0x1d fp=0x1c000056e80 sp=0x1c000056e60 pc=0x106e59a3d
runtime.cgocallbackg1(0x106e59a20, 0x7ffee9235c08, 0x0)
	/Users/asottile/opt/go/src/runtime/cgocall.go:442 +0x27b fp=0x1c000056f40 sp=0x1c000056e80 pc=0x106db1b7b
runtime.cgocallbackg(0x106e59a20, 0x7ffee9235c08, 0x0)
	/Users/asottile/opt/go/src/runtime/cgocall.go:361 +0x11a fp=0x1c000056f90 sp=0x1c000056f40 pc=0x106db187a
runtime.cgocallbackg(0x106e59a20, 0x7ffee9235c08, 0x0)
	<autogenerated>:1 +0x29 fp=0x1c000056fb8 sp=0x1c000056f90 pc=0x106e1c189
runtime.cgocallback(0x0, 0x0, 0x0)
	/Users/asottile/opt/go/src/runtime/asm_amd64.s:1084 +0xcc fp=0x1c000056fe0 sp=0x1c000056fb8 pc=0x106e19b6c
runtime: g 17: unexpected return pc for runtime.cgocallback called from 0x10db22381
stack: frame={sp:0x1c000056fb8, fp:0x1c000056fe0} stack=[0x1c000056000,0x1c000057000)
0x000001c000056eb8:  0x000001c000056e9f  0x020201c000056f30 
0x000001c000056ec8:  0x0000000106e12e6e <runtime.exitsyscall+0x000000000000014e>  0x000001c000006700 
0x000001c000056ed8:  0x0000000106e59a20 <_cgoexp_ffdfbdb0b480_hi_Hello+0x0000000000000000>  0x0000000000000000 
0x000001c000056ee8:  0x000001c000006700  0x000001c000056ed8 
0x000001c000056ef8:  0x0000000106db1c20 <runtime.cgocallbackg1.deferwrap2+0x0000000000000000>  0x000001c000056ec5 
0x000001c000056f08:  0x00007ffee9237000  0x0000000000000000 
0x000001c000056f18:  0x0000000000000000  0x0000000000000000 
0x000001c000056f28:  0x000001c000056ef8  0x000001c000056f80 
0x000001c000056f38:  0x0000000106db187a <runtime.cgocallbackg+0x000000000000011a>  0x0000000106e59a20 <_cgoexp_ffdfbdb0b480_hi_Hello+0x0000000000000000> 
0x000001c000056f48:  0x00007ffee9235c08  0x0000000000000000 
0x000001c000056f58:  0x000000010db22381  0x000001c000056fe0 
0x000001c000056f68:  0x0000000000000000  0x000001c000006700 
0x000001c000056f78:  0x000001c000048008  0x000001c000056fa8 
0x000001c000056f88:  0x0000000106e1c189 <runtime.cgocallbackg+0x0000000000000029>  0x0000000106e59a20 <_cgoexp_ffdfbdb0b480_hi_Hello+0x0000000000000000> 
0x000001c000056f98:  0x00007ffee9235c08  0x0000000000000000 
0x000001c000056fa8:  0x00007ffee9235b98  0x0000000106e19b6c <runtime.cgocallback+0x00000000000000cc> 
0x000001c000056fb8: <0x0000000106e59a20 <_cgoexp_ffdfbdb0b480_hi_Hello+0x0000000000000000>  0x00007ffee9235c08 
0x000001c000056fc8:  0x0000000000000000  0x0000000000000000 
0x000001c000056fd8:  0x000000010db22381 >0x0000000000000000 
0x000001c000056fe8:  0x0000000000000000  0x0000000000000000 
0x000001c000056ff8:  0x0000000000000000 

goroutine 2 gp=0xc000006c40 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/Users/asottile/opt/go/src/runtime/proc.go:424 +0xce fp=0xc000042fa8 sp=0xc000042f88 pc=0x106e1292e
runtime.goparkunlock(...)
	/Users/asottile/opt/go/src/runtime/proc.go:430
runtime.forcegchelper()
	/Users/asottile/opt/go/src/runtime/proc.go:337 +0xb3 fp=0xc000042fe0 sp=0xc000042fa8 pc=0x106de2473
runtime.goexit({})
	/Users/asottile/opt/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000042fe8 sp=0xc000042fe0 pc=0x106e19da1
created by runtime.init.7 in goroutine 1
	/Users/asottile/opt/go/src/runtime/proc.go:325 +0x1a

goroutine 3 gp=0xc000007180 m=nil [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/Users/asottile/opt/go/src/runtime/proc.go:424 +0xce fp=0xc000043780 sp=0xc000043760 pc=0x106e1292e
runtime.goparkunlock(...)
	/Users/asottile/opt/go/src/runtime/proc.go:430
runtime.bgsweep(0xc00006c000)
	/Users/asottile/opt/go/src/runtime/mgcsweep.go:277 +0x94 fp=0xc0000437c8 sp=0xc000043780 pc=0x106dce4d4
runtime.gcenable.gowrap1()
	/Users/asottile/opt/go/src/runtime/mgc.go:204 +0x25 fp=0xc0000437e0 sp=0xc0000437c8 pc=0x106dc2e25
runtime.goexit({})
	/Users/asottile/opt/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000437e8 sp=0xc0000437e0 pc=0x106e19da1
created by runtime.gcenable in goroutine 1
	/Users/asottile/opt/go/src/runtime/mgc.go:204 +0x66

goroutine 4 gp=0xc000007340 m=nil [GC scavenge wait]:
runtime.gopark(0xc00006c000?, 0x106e8b7c8?, 0x1?, 0x0?, 0xc000007340?)
	/Users/asottile/opt/go/src/runtime/proc.go:424 +0xce fp=0xc000043f78 sp=0xc000043f58 pc=0x106e1292e
runtime.goparkunlock(...)
	/Users/asottile/opt/go/src/runtime/proc.go:430
runtime.(*scavengerState).park(0x106f51400)
	/Users/asottile/opt/go/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc000043fa8 sp=0xc000043f78 pc=0x106dcbee9
runtime.bgscavenge(0xc00006c000)
	/Users/asottile/opt/go/src/runtime/mgcscavenge.go:653 +0x3c fp=0xc000043fc8 sp=0xc000043fa8 pc=0x106dcc45c
runtime.gcenable.gowrap2()
	/Users/asottile/opt/go/src/runtime/mgc.go:205 +0x25 fp=0xc000043fe0 sp=0xc000043fc8 pc=0x106dc2dc5
runtime.goexit({})
	/Users/asottile/opt/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000043fe8 sp=0xc000043fe0 pc=0x106e19da1
created by runtime.gcenable in goroutine 1
	/Users/asottile/opt/go/src/runtime/mgc.go:205 +0xa5

goroutine 5 gp=0xc000007c00 m=nil [finalizer wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/Users/asottile/opt/go/src/runtime/proc.go:424 +0xce fp=0xc000044620 sp=0xc000044600 pc=0x106e1292e
runtime.runfinq()
	/Users/asottile/opt/go/src/runtime/mfinal.go:193 +0x107 fp=0xc0000447e0 sp=0xc000044620 pc=0x106dc1ea7
runtime.goexit({})
	/Users/asottile/opt/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000447e8 sp=0xc0000447e0 pc=0x106e19da1
created by runtime.createfing in goroutine 1
	/Users/asottile/opt/go/src/runtime/mfinal.go:163 +0x3d
Abort trap: 6

aside: I decided to abandon setuptools-golang for this reason 🤷 -- and yeah a clickbaitish video with more details

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions