Skip to content

Support //go:linkname compiler directive #1006

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Mar 28, 2021

Conversation

nevkontakte
Copy link
Member

@nevkontakte nevkontakte commented Mar 27, 2021

GopherJS implementation is a bit more limited that the upsteam's, but should be sufficient for our use cases. The main limitations are:

  • The directive can't be used to "insert" an implementation into another package. It probably isn't hard to support that, but I left it out since there are very few relevant uses of that in the standard library anyway.
  • The directive can't be applied to variables. This is a harder limitation to overcome due to how variable assignments work in JavaScript. The compiler will have to be changed significantly to ensure that all the different package-level variables in Go are represented by a single JavaScript variable, otherwise assigning the variable from one package may make it invisible to another package.
  • Dead code elimination could be made more accurate. Currently any function that is referenced as an implementation of another function is considered live, even if the reference ends up eliminated.

Fixes #1000.

The test exercises linknaming to a function in an importer package, as
well as to a function from the importer package. The two cases are
distinct because they have different package initialization order and
must work both.

Currently the test is passing under upstream Go, and panics under
GopherJS.
These two types make an internal representation of a go:compile
directive.
The implementation roughly follows these steps:

  - compiler.Compile() gathers all information about go:linkname
    directives in the source code and does basic correctness checks.
  - compiler.WritePkgCode() uses this information to generate code
    that exposes referenced function implementations in the `$linknames`
    object, which serves as a global registry of sorts.
  - It also generates code that will "import" those implementations
    into appropriate packages from `$linknames` one all packages have
    been defined. This code needs to be delayed because it may be
    referencing packages which entries in `$packages` object have not
    been populated yet.

This implementation is sufficient to pass our basic test case:

```
$ gopherjs build github.com/gopherjs/gopherjs/tests/testdata/linkname                                                                                                                                148 ↵
$ node linkname.js
doing one
doing two
doing imported one: doing internal one: one secret
doing three
doing imported three: doing internal three: three secret
```

However, a few adjustments to the standard library overlays are still
needed to make it run correctly again.
Now that we support go:linkname directive, the time package expects
to find this function implementation in the `runtime` package.
@nevkontakte nevkontakte requested a review from flimzy March 27, 2021 14:05
@nevkontakte
Copy link
Member Author

@visualfc, you might be interested in taking a look at this PR. I've noticed that your implementation seems to go into infinite recursion when I build the test case from this PR with it.

Here's an example stack trace (under Ubuntu, with a limit of 2G of virtual memory to prevent my laptop from going into swapping):

 $ (ulimit -S -v $((2*1024*1024)) && gopherjs build github.com/gopherjs/gopherjs/tests/testdata/linkname)
runtime: out of memory: cannot allocate 4194304-byte block (658735104 in use)
fatal error: out of memory

goroutine 1 [running]:
runtime.throw(0x9e6045, 0xd)
        /usr/local/go/src/runtime/panic.go:1117 +0x72 fp=0xc01c612cb0 sp=0xc01c612c80 pc=0x437c12
runtime.(*mcache).refill(0x7ffb15d28f18, 0x67)
        /usr/local/go/src/runtime/mcache.go:164 +0x285 fp=0xc01c612cf8 sp=0xc01c612cb0 pc=0x419bc5
runtime.(*mcache).nextFree(0x7ffb15d28f18, 0xc02ffdb567, 0x30, 0x8, 0xc02ffdf090)
        /usr/local/go/src/runtime/malloc.go:882 +0x8d fp=0xc01c612d30 sp=0xc01c612cf8 pc=0x40ee0d
runtime.mallocgc(0x2000, 0x933e60, 0x1, 0xc01c612ee8)
        /usr/local/go/src/runtime/malloc.go:1069 +0x850 fp=0xc01c612db8 sp=0xc01c612d30 pc=0x40f810
runtime.makeslice(0x933e60, 0x2000, 0x2000, 0x10)
        /usr/local/go/src/runtime/slice.go:98 +0x6c fp=0xc01c612de8 sp=0xc01c612db8 pc=0x44f40c
os.(*File).readdir(0xc02f523cc8, 0xffffffffffffffff, 0x2, 0x0, 0x0, 0xc01c612f68, 0x4e4705, 0xc02ffdf090, 0x4d, 0x0, ...)
        /usr/local/go/src/os/dir_unix.go:35 +0xec9 fp=0xc01c612ef8 sp=0xc01c612de8 pc=0x4e1449
os.(*File).Readdir(0xc02f523cc8, 0xffffffffffffffff, 0x0, 0x0, 0xc02f523cc8, 0x0, 0x0)
        /usr/local/go/src/os/dir.go:41 +0x51 fp=0xc01c612f78 sp=0xc01c612ef8 pc=0x4e0031
io/ioutil.ReadDir(0xc02ffdf090, 0x4d, 0x2f, 0xaaaf00, 0x1, 0xc02fff4300, 0x2f)
        /usr/local/go/src/io/ioutil/ioutil.go:63 +0x90 fp=0xc01c613018 sp=0xc01c612f78 pc=0x57a970
github.com/goplusjs/gopherjs/build.NewBuildContext.func2(0xc02ffdf090, 0x4d, 0x1, 0xc01c6130d0, 0x4e7da7, 0xc02fff72e8, 0xc02ffc98c0)
        /home/aleks/Devel/golang/gopherjs-plus/build/build.go:81 +0x11c fp=0xc01c613098 sp=0xc01c613018 pc=0x8aac3c
golang.org/x/tools/go/buildutil.ReadDir(0xc01c6132e8, 0xc02ffdf090, 0x4d, 0xc02fff72e8, 0x0, 0x0, 0x0, 0x0)
        /home/aleks/go/pkg/mod/golang.org/x/tools@v0.0.0-20200131143746-097c1f2eed26/go/buildutil/util.go:183 +0x4a fp=0xc01c6130e0 sp=0xc01c613098 pc=0x89808a
github.com/goplusjs/gopherjs/build.jsFilesFromDir(0xc01c6132e8, 0xc02ffdf090, 0x4d, 0xc0000a4120, 0x0, 0x0, 0xc02ffecc00, 0x0)
        /home/aleks/Devel/golang/gopherjs-plus/build/build.go:1020 +0x5a fp=0xc01c613180 sp=0xc01c6130e0 pc=0x8a903a
github.com/goplusjs/gopherjs/build.importWithSrcDir(0x9e1730, 0x2, 0x9e2119, 0x5, 0xaab908, 0xd, 0xc00002601f, 0xe, 0x0, 0x0, ...)
        /home/aleks/Devel/golang/gopherjs-plus/build/build.go:238 +0x652 fp=0xc01c6132e8 sp=0xc01c613180 pc=0x8a14b2
github.com/goplusjs/gopherjs/build.(*Session).BuildImportPathWithPackage(0xc000212480, 0xc02ffcac01, 0x3a, 0xc02f7333b0, 0xc02ffe0360, 0x0, 0x20300b, 0x20300b)
        /home/aleks/Devel/golang/gopherjs-plus/build/build.go:660 +0x12c fp=0xc01c613530 sp=0xc01c6132e8 pc=0x8a542c
github.com/goplusjs/gopherjs/build.(*Session).buildPackage(0xc000212480, 0xc02f7333b0, 0x0, 0x0, 0x0)
        /home/aleks/Devel/golang/gopherjs-plus/build/build.go:820 +0x145c fp=0xc01c6137c8 sp=0xc01c613530 pc=0x8a801c
github.com/goplusjs/gopherjs/build.(*Session).BuildImportPathWithPackage(0xc000212480, 0xc02f21f97c, 0x38, 0x0, 0x2, 0xa7, 0xc000310270, 0xc01c613d38)
        /home/aleks/Devel/golang/gopherjs-plus/build/build.go:668 +0x18c fp=0xc01c613a10 sp=0xc01c6137c8 pc=0x8a548c

<-- some text skipped -->

github.com/goplusjs/gopherjs/build.(*Session).BuildImportPathWithPackage(0xc000212480, 0xc02fe84781, 0x38, 0xc02ea69310, 0xc02ea7b298, 0xd64a20, 0x0, 0x0)
        /home/aleks/Devel/golang/gopherjs-plus/build/build.go:668 +0x18c fp=0xc01c61eac0 sp=0xc01c61e878 pc=0x8a548c
github.com/goplusjs/gopherjs/build.(*Session).buildPackage(0xc000212480, 0xc02ea69310, 0x0, 0x0, 0x0)
        /home/aleks/Devel/golang/gopherjs-plus/build/build.go:820 +0x145c fp=0xc01c61ed58 sp=0xc01c61eac0 pc=0x8a801c
github.com/goplusjs/gopherjs/build.(*Session).BuildImportPathWithPackage(0xc000212480, 0xc02e44fd6c, 0x38, 0x0, 0x2, 0x14f, 0xc000310270, 0xc01c61f2c8)
        /home/aleks/Devel/golang/gopherjs-plus/build/build.go:668 +0x18c fp=0xc01c61efa0 sp=0xc01c61ed58 pc=0x8a548c
github.com/goplusjs/gopherjs/build.(*Session).BuildImportPath(0xc000212480, 0xc02e44fd6c, 0x38, 0x38, 0xd94ea0, 0xc02fe19400)
        /home/aleks/Devel/golang/gopherjs-plus/build/build.go:647 +0x48 fp=0xc01c61eff0 sp=0xc01c61efa0 pc=0x8a52c8
...additional frames elided...

Note that I built the gopherjs binary from your repository at goplusjs/gopherjs@0e65aed, but ran it this repository with this PR checked out. Hope that helps :)

Copy link
Member

@flimzy flimzy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Should we mention these differences from upstream here?

@nevkontakte
Copy link
Member Author

Good idea, although I've created a separate document for compiler directives.

@nevkontakte nevkontakte merged commit c6a2ff3 into gopherjs:go1.16-stdlib Mar 28, 2021
@nevkontakte nevkontakte deleted the go1.16-linkname branch March 28, 2021 18:57
@visualfc
Copy link
Contributor

visualfc commented Mar 29, 2021

@visualfc, you might be interested in taking a look at this PR. I've noticed that your implementation seems to go into infinite recursion when I build the test case from this PR with it.

Here's an example stack trace (under Ubuntu, with a limit of 2G of virtual memory to prevent my laptop from going into swapping):

goplusjs/gopherjs#35 this PR support go:linkname circle import

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants