Skip to content

Go 1.16 standard library support #989

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
nevkontakte opened this issue Feb 21, 2021 · 8 comments · Fixed by #1015
Closed

Go 1.16 standard library support #989

nevkontakte opened this issue Feb 21, 2021 · 8 comments · Fixed by #1015
Assignees
Labels
NeedsHelp Community contributions are welcome for this feature!

Comments

@nevkontakte
Copy link
Member

nevkontakte commented Feb 21, 2021

Go 1.16 has been released just a few days ago, and the latest Go version that GopherJS supported was 1.12, so we need to catch up. Note that this issue covers only the standard library and language support, Go Modules are tracked in #855.

After speaking with @flimzy and other folks we've decided that it's best to jump straight Go 1.16 version, without providing separate releases for intermediate versions. The major reasons behind this choice are:

  1. Making multiple additional releases is a lot of work that we really can't afford now.
  2. If we support Go 1.16 we should still be able to build Go code for earlier versions without any significant issues thanks to the Go backward compatibility policy.
  3. The value of these releases will also quickly diminish as the time passes.

In terms of how to proceed we have two options:

  1. Pull in changes from @visualfc's fork (https://github.com/goplusjs/gopherjs), which already pretty much supports everything we need. However, I would like to get @visualfc's permission before we do that to make sure there won't be any bad feelings left between the two projects.
    • As a side note, this won't be a very straight-forward pull since @visualfc's fork uses different import paths and whatnot, but still it is much, much easier.
  2. As a plan B, we could also reimplement everything ourselves. This will be a lot of work and create more divergence between the forks, but is still doable.
@nevkontakte nevkontakte added the NeedsHelp Community contributions are welcome for this feature! label Feb 21, 2021
@nevkontakte nevkontakte added this to the Go 1.16 support milestone Feb 21, 2021
@nevkontakte nevkontakte pinned this issue Feb 21, 2021
@visualfc
Copy link
Contributor

  1. Pull in changes from @visualfc's fork (https://github.com/goplusjs/gopherjs), which already pretty much supports everything we need. However, I would like to get @visualfc's permission before we do that to make sure there won't be any bad feelings left between the two projects.

    • As a side note, this won't be a very straight-forward pull since @visualfc's fork uses different import paths and whatnot, but still it is much, much easier.

use https://github.com/goplusjs/gopherjs any source code no problem.

https://github.com/goplusjs/gopherjs natives use tags for easy support high version. ( It's easier than you think )

compiler/natives/src/os
- os.go
- go116_os.go    // +build go1.16

@nevkontakte
Copy link
Member Author

@visualfc thanks for the permission, I appreciate that a lot.

Supporting multiple older versions of the library is nice, but it also has a cost that grows over time. If a certain library function changes its behavior between two versions, we'd have to keep two copies of it. The more versions of the standard library we support going back in time, the more redundant code we'll have to carry with us, maintain it, and fix bugs in.

In principle, I'd be happy to do that, but I'm currently very cautious of expanding the scope of the project so that that we won't be able to handle it and would let it stall again. If we manage to build up a bigger, more active community that is capable of carrying the increased complexity (and I how we eventually will), no one would object to keeping support for more versions of the standard library.

Until then, though, it is better for the project health to focus on things with the biggest value per time invested. And assuming we will be able to update GopherJS consistently for each new Go version, people still using older Go versions will be able to use a matching GopherJS version without an issue, since it all will persist in the repository. I hope that makes sense, if not — feel free to ping me or @flimzy in Gopherjs Slack and we can discuss more :)

@nevkontakte
Copy link
Member Author

I have started a new branch https://github.com/gopherjs/gopherjs/tree/go1.16-stdlib for this work. Currently it is failing to build (unsurprisingly).

Anyone interesting in helping may follow the steps below:

  1. Fork this repository.
  2. Check out go1.16-stdlib branch into $GOPATH/src/github.com/gopherjs/gopherjs.
  3. Make sure you a using Go 1.16: go version. If your system's Go version is older, you can get the up-to-date one yb running go get golang.org/dl/go1.16.
  4. Run GO111MODULE=off go install -v github.com/gopherjs/gopherjs to build gopherjs from source.
  5. Run gopherjs -v build std && gopherjs test std to try building and testing the standard library.
  6. Make one package build and/or pass tests.
  7. Send a pull request against the go1.16-stdlib branch. Even a single package fixed is a welcome contribution :)

@nevkontakte
Copy link
Member Author

I think we are getting really close to having all the tests passing. The only tricky one remaining is time/tzdata package support. The problem with tzdata is a circular dependency between time and time/tzdata:

  • When building with -tag timetzdata package time importstime/tzdata.
  • time/tzdata itself calls registerLoadFromEmbeddedTZData() function to register itself, which is implemented in the time package.

The upstream gets around it by using go:linkname compiler directive, which we currently do not support: #1000. I am now debating myself on whether to try and implement go:linkname support in GopherJS (doing so correctly may not be trivial), or trying to hack something around using the JavaScript environment. Unfortunately, the latter is also difficult to get right because depending on a particular build the order of package initialization may be different and is difficult to rely on.

@visualfc
Copy link
Contributor

visualfc commented Mar 22, 2021

I think we are getting really close to having all the tests passing. The only tricky one remaining is time/tzdata package support. The problem with tzdata is a circular dependency between time and time/tzdata:

  • When building with -tag timetzdata package time importstime/tzdata.
  • time/tzdata itself calls registerLoadFromEmbeddedTZData() function to register itself, which is implemented in the time package.

The upstream gets around it by using go:linkname compiler directive, which we currently do not support: #1000. I am now debating myself on whether to try and implement go:linkname support in GopherJS (doing so correctly may not be trivial), or trying to hack something around using the JavaScript environment. Unfortunately, the latter is also difficult to get right because depending on a particular build the order of package initialization may be different and is difficult to rely on.

goplusjs/gopherjs#2 support //go:linkname localname [importpath.name]
goplusjs/gopherjs#7
goplusjs/gopherjs@ffac742

@nevkontakte
Copy link
Member Author

Thanks, I have been actually looking at your implementation, and it gave me a bit better understanding of the problem we are dealing with 🙂 There are, unfortunately, several issues with it, which are likely to bite us at some later stage:

  1. It actually creates circular imports between packages, which is forbidden in the language spec for good reasons. Cycles in the build graph are difficult to handle correctly in general, and Go's type checking is not geared to deal with this. goplusjs/gopherjs@ffac742 is actually a symptom of that, rewriting already compiled libraries should not be necessary.
  2. Conceptually, build package is the wrong layer to handle this. The package is already very complicated thanks to the standard library augmentation logic, and having this package doing parts of compiler's job will only make it difficult to maintain. I believe that collecting linknames in compiler.Compile() stage and generating appropriate code at compiler.WritePkgCode() would be more maintainable (and more similar to Go's own implementation).
  3. Perhaps I haven't read your code correctly, but I think it only allows "importing" a function implementation, whereas as specified it should also be able to "insert" a function implementation into another package.

I posted a few more general thoughts here: #1000 (comment).

@visualfc
Copy link
Contributor

visualfc commented Mar 22, 2021

  1. It actually creates circular imports between packages, which is forbidden in the language spec for good reasons. Cycles in the build graph are difficult to handle correctly in general, and Go's type checking is not geared to deal with this. goplusjs/gopherjs@ffac742 is actually a symptom of that, rewriting already compiled libraries should not be necessary.

Because it's loaded gopherjs build pkg.a from disk. rewrite not need on rebuild all ( gopherjs build -a) .

  1. Perhaps I haven't read your code correctly, but I think it only allows "importing" a function implementation, whereas as specified it should also be able to "insert" a function implementation into another package.

Yes, insert not implement. this function no need.

@nevkontakte
Copy link
Member Author

Because it's loaded gopherjs build pkg.a from disk. rewrite not need on rebuild all ( gopherjs build -a) .

I admit I don't fully understand the problem that was fixed in goplusjs/gopherjs@ffac742, but from my understanding of how compilers work in general, a well-designed build system should not need to rewrite a build artifact if inputs didn't change. Even though GopherJS doesn't have a well-defined linker, the compiler.WritePkgCode() function logically is very close to it and it should be able to generate code that go:linkname requires without having to alter already compiled archives.

Yes, insert not implement. this function no need.

After doing some more research, I agree. The only place where the "insert"-style linkname is used is the runtime package and it looks like it doesn't work very reliably even in the upstream Go in the first place. To be honest I'm a bit puzzled why did the Go team did it this way instead of adding an "import"-style linknames in the corresponding packages, but I guess nothing stops us from doing that. Which I think is exactly what you did in your fork. See #1000 (comment) for details.

@nevkontakte nevkontakte unpinned this issue Jun 19, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
NeedsHelp Community contributions are welcome for this feature!
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants