WIP: Use cmd/go with GopherJS support. #485

dmitshur · 2016-06-27T08:59:04Z

DO NOT MERGE.

This is a very early unfinished WIP PR to use cmd/go with GopherJS support instead of own build tool (#388).

Don't try it on your computer yet! It won't work, because there are hardcoded paths specific to my setup. But that will be fixed soon, up next. I will update this when it's ready for first other people to try it.

Large parts of the code here are generated from https://github.com/gopherjs/go repository, go1.6.2-v0.1 or whatever's latest branch. That repo is used for development purposes as a cmd/go base with cherry-picked GopherJS changes on top.

Resolves #388.

Fixes #470.
Fixes #443.
Fixes #414.
Fixes #382.

TODO

Get rid of hardcoded paths in code, resolve remaining critical TODOs in code.
Figure out a way to bootstrap (install) the github.com/gopherjs/gopherjs/js and github.com/gopherjs/gopherjs/nosync packages.
GOARCH=js go get -compiler=gopherjs github.com/goxjs/websocket should work.

The issue was missing "netgo" build tag. Need to fix that.
Verify all "cmd/go would not have this issue" issues.
Implement a linker flag to control minify/not minify. Currently it always minifies by default.
Figure out a way to deal with sourcemap files. Right now they're not generated at all. Generating a 2nd file is kinda difficult and very different from all other Go compilers... maybe it should be embedded in to the same file? As a debug flag/option? Figure this out.
Implement and test .inc.js stuff.
Take care of 2 items in Setup section.
Upstream or fork required fixes/changes to bundle command:
- Part 1 - https://golang.org/cl/23785
- Part 2 - not yet done
Document the complicated source package locations (and generated package locations). Explain why it's needed.
Fix /usr/local/go/test/fixedbugs/issue10332.go which is currently failing because reflect.ValueOf(foo{}).Type().Field(0).PkgPath is being "command-line-arguments" instead of "main" for some reason.
gopherjs test fmt should work, need to fix support for external tests.

There may be an issue with _test.go files being built for normal packages.

gopherjs build -m -o /tmp/tictactoe.js github.com/shurcooL/tictactoe/cmd/tictactoe
go build encoding/json: importFromDisk: archive for path "testing" missing: open /Users/Dmitri/Dropbox/Work/2013/GoLand/pkg/gopherjs_darwin_js/testing.a: no such file or directory

This turned out to be Any package that imports "encoding/json" will needlessly import "testing" package and its deps. #481.

cmd/go would not have this issue

https://github.com/gopherjs/gopherjs/issues?q=is%3Aissue+is%3Aopen+label%3A%22cmd%2Fgo+would+not+have+this+issue%22

Setup

Need to create a GOROOT/pkg subdirectory that is writeable, if GOROOT/pkg isn't.

sudo mkdir /usr/local/go/pkg/gopherjs_darwin_js
sudo chown $USER /usr/local/go/pkg/gopherjs_darwin_js

Where darwin is your GOOS.

Otherwise you get this when installing first package (which tries to install stdlib with GOARCH=js for first time, starting with runtime/internal/sys package):

go install runtime/internal/sys: mkdir /usr/local/go/pkg/gopherjs_darwin_js: permission denied

Benchmarks seem to require System Calls (https://github.com/gopherjs/gopherjs/blob/master/doc/syscalls.md) to be installed, or otherwise you get:

BenchmarkBasic  2000000000           0.82 ns/op
fatal error: all goroutines are asleep - deadlock!
exit status 2

dmitshur · 2016-07-05T06:38:38Z

With ecc1b7a done, many tests are now passing, and many are failing.

Most of the failures are due to Examples now running, but they're still not supported due to #381. For CI to pass, either need to disable examples from running, or fix the issue causing them to be unsupported.

dmitshur · 2016-07-05T08:04:18Z

build/build_test.go

 	"go/token"
 	"strconv"
 	"strings"
 	"testing"

+	gobuild "github.com/gopherjs/gopherjs/go/build"
 	"github.com/kisielk/gotool"
 	"github.com/shurcooL/go/importgraphutil"


This doesn't fix the issue completely because github.com/shurcooL/go/importgraphutil package still uses original go/build, so it's incompatible.

Fixed completely in a49c2ae.

dmitshur · 2016-08-28T00:00:06Z

https://golang.org/cl/23785 has been reviewed and merged, which brings us ever so closely forward, since this PR depended on that change. (See "Upstream or fork required fixes/changes to bundle command" item.) I still have to make and send out another improvement to bundle that this PR depends on.

dmitshur · 2016-09-06T01:40:43Z

I'm in the process of updating/rebasing on top of cmd/go of Go 1.7.

Just finished updating the github.com/gopherjs/go repository (which is the source - most things in this PR are generated/copied from that one). There were quite a few minor changes in cmd/go and go/build packages between 1.6 and 1.7, so it took a few hours of careful work, but it's still logically straightforward.

I've pushed the result to https://github.com/gopherjs/go/commits/go1.7.0-v0.1. See individual commit messages and their changes - each commit should be logically separate and very easy to understand (that's the goal).

All of its tests are passing and things are building successfully (with noted limitations):

$ go test github.com/gopherjs/go/...
ok      github.com/gopherjs/go/cmd/go   113.478s
ok      github.com/gopherjs/go/go/build 0.072s
?       github.com/gopherjs/go/internal/race    [no test files]
ok      github.com/gopherjs/go/internal/singleflight    0.023s
?       github.com/gopherjs/go/internal/testenv [no test files]

I'm going to keep reusing this PR, and just squash, rebase, regenerate and force push on top. I'll keep a backup of the history of changes/work on the Go 1.6 branch for reference, in case we need to look something up later.

So, this PR will soon become 1 squashed commit that's rebased on latest GopherJS 1.7-1 and github.com/gopherjs/go repository on branch go1.7.0-v0.1.

Very alpha, much unfinished. Dragons be here. It won't work on your computer yet, unless you're me. But that will be fixed soon, up next. TODO: Split into logical commits, document and explain changes. Update. https://github.com/gopherjs/go/commit/524b8a70b6b558603ceca9144d36e7113f0d37b9 Use own go/build replacement. Update. https://github.com/gopherjs/go/commit/4254365d55f5a069526e73d904ba1b04f4e52709 WIP: Use embedded js, nosync packages. For now, try to use pre-built js.a and nosync.a until we can figure out a good way to build them in a normal way. Try to do this in order to make progress in CI and see what next steps need to be taken. Don't use own go/build replacement for importgraph. It's needed for GopherJS compiler, but we can't use it for importgraph package since they're 3rd party and rely on "go/build".Package. WIP: Try a fix for issue 10332. WIP: Work on disabling cgo, filling custom files during test package builds.

neelance · 2017-02-19T13:24:13Z

@shurcooL Do you have any plan on how we want to continue on this?

dmitshur · 2017-02-21T18:56:51Z

Yep, I've been meaning to post an update here.

I will be closing this PR and abandoning this effort. I may choose to revisit the effort later, but until further notice, I won't be actively working on this.

There are 2 reasons I'll close the PR:

As of early Go 1.9 development, there are many changes being done to re-organize cmd/go. It's being split up from one huge package, into many packages. See https://github.com/golang/go/issues?q=label%3AGoCommand.

GopherJS is now version 1.8-1 and supports Go 1.8. This PR was based on cmd/go from Go 1.7, and so it's out of date. It would need to be updated for Go 1.8. However, given the changes going into 1.9, it might even make sense to aim at start using that as base.

So, this PR is out of date and not helpful.
The goal of this PR was to get it into a good "mergeable" shape where we could feel comfortable about merging it into master, confident that it would work as well as the current approach, but better (fix more issues), and lead to decreased maintenance effort in the future (as outlined in proposal: Replace in-house implementation of gopherjs build tool with augmented forked cmd/go build tool. #388). This was unsuccessful.

Here's a little postmortem and explanations why this effort did not work out.

When I first prototyped this idea, I had early success. I was able to make a relatively easy-to-maintain go-gettable fork of cmd/go. I was also able to add support for a new compiler toolchain and get around all the difficulties of import paths, internal packages, and not being able to modify GOROOT for our needs.

At the time, I thought that was the hardest part, and once we'd gotten past that, we would be able to take advantage of the complicated logic that cmd/go has internally to resolve all the package build order, the logic for figuring out where to place built libraries, reusing non-stale built libraries, etc., etc. I also thought it'd be easier to simplify gopherjs itself to be just a dedicated compiler and linker, rather than everything-in-one.

However, it turned out that cmd/go internal code was actually very complicated, messy, and contained many hacks/weird fixes that are specific to gc and gccgo compilers, and various edge cases that needed to be handled. I had to both understand and re-implement those things for gopherjsToolchain (e.g., see https://github.com/golang/go/blob/go1.7/src/cmd/go/build.go#L2273-L2275). Its internal interfaces were too low level and not well documented, because they're not a part of any public API. It was also very hard to fully understand, especially when I had little spare time to dedicate to it after day job. I was hoping/wanting a clean interface to that would abstract all the messy logic, but that didn't exist (it may begin to exist with the current re-architecture of cmd/go, but I can't be sure without confirming that first; the re-architecture may have different goals and solve other problems). Many things would be easier if we could start by working on cmd/go to make it friendlier to expand towards other compilers first, but I don't know if I'd get permission to do that.

So, I gave up on it because the effort and difficulty and time needed were much higher than originally anticipated, and the benefits are not worth it.

There may be another opportunity to consider this in the future, given the cmd/go split into packages, and, for example, there were changes to make it possible to go get different versions of Go. But, that would need to evaluated. I'm happy to provide more detailed comments and participate in any future discussions on this topic, because of my experience.

dmitshur · 2017-02-21T19:12:04Z

@neelance I will also delete the https://github.com/gopherjs/go WIP repository. It was created to support this PR, and is no longer needed. Edit: Done.

neelance · 2017-02-21T21:48:36Z

All right. It was worth a try. Thanks for investigating.

dmitshur force-pushed the cmd-go branch 2 times, most recently from c7fcac1 to 8d6f9ab Compare July 5, 2016 01:06

dmitshur force-pushed the cmd-go branch from d54f081 to bc17b13 Compare July 5, 2016 08:02

dmitshur reviewed Jul 5, 2016
View reviewed changes

dmitshur mentioned this pull request Jul 18, 2016

"panic: ShowContent already declared" when build project with 'gopherjs build [packagename]' #488

Closed

dmitshur force-pushed the cmd-go branch from bc17b13 to a49c2ae Compare July 31, 2016 22:11

dmitshur mentioned this pull request Aug 6, 2016

Support for GOBIN? #494

Closed

dmitshur mentioned this pull request Sep 5, 2016

Add plumbing to support build tags in test mode. #508

Merged

dmitshur force-pushed the cmd-go branch from d4b0bb3 to 2d17f70 Compare September 6, 2016 04:41

dmitshur mentioned this pull request Sep 6, 2016

compiler/natives: Harmless one-byte difference between source and embedded reflect.go. #510

Closed

steveoc64 mentioned this pull request Sep 6, 2016

FreeBSD: gopherjs build "undeclared name: sysAF_LINK" #511

Closed

This was referenced Sep 20, 2016

Add support for source maps that use the local disk path of go source #521

Merged

Fix an issue with watching command packages. #523

Closed

flimzy mentioned this pull request Sep 25, 2016

Feature Request: gopherjs test support running and verifying example code. #239

Closed

dmitshur closed this Feb 21, 2017

dmitshur deleted the cmd-go branch February 21, 2017 18:57

dmitshur mentioned this pull request Feb 21, 2017

proposal: Replace in-house implementation of gopherjs build tool with augmented forked cmd/go build tool. #388

Closed

dmitshur mentioned this pull request May 30, 2017

Circular imports lead to stack overflow crash #414

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

WIP: Use cmd/go with GopherJS support. #485

WIP: Use cmd/go with GopherJS support. #485

Uh oh!

dmitshur commented Jun 27, 2016 •

edited

Loading

Uh oh!

dmitshur commented Jul 5, 2016

Uh oh!

dmitshur Jul 5, 2016

Uh oh!

dmitshur Jul 31, 2016

Uh oh!

dmitshur commented Aug 28, 2016 •

edited

Loading

Uh oh!

dmitshur commented Sep 6, 2016 •

edited

Loading

Uh oh!

neelance commented Feb 19, 2017

Uh oh!

dmitshur commented Feb 21, 2017 •

edited

Loading

Uh oh!

dmitshur commented Feb 21, 2017 •

edited

Loading

Uh oh!

neelance commented Feb 21, 2017

Uh oh!

Uh oh!

WIP: Use cmd/go with GopherJS support. #485

WIP: Use cmd/go with GopherJS support. #485

Uh oh!

Conversation

dmitshur commented Jun 27, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TODO

cmd/go would not have this issue

Setup

Uh oh!

dmitshur commented Jul 5, 2016

Uh oh!

dmitshur Jul 5, 2016

Choose a reason for hiding this comment

Uh oh!

dmitshur Jul 31, 2016

Choose a reason for hiding this comment

Uh oh!

dmitshur commented Aug 28, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dmitshur commented Sep 6, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

neelance commented Feb 19, 2017

Uh oh!

dmitshur commented Feb 21, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dmitshur commented Feb 21, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

neelance commented Feb 21, 2017

Uh oh!

Uh oh!

dmitshur commented Jun 27, 2016 •

edited

Loading

dmitshur commented Aug 28, 2016 •

edited

Loading

dmitshur commented Sep 6, 2016 •

edited

Loading

dmitshur commented Feb 21, 2017 •

edited

Loading

dmitshur commented Feb 21, 2017 •

edited

Loading