What version of Go are you using (go version)?

$ go version
go version devel +4a2cc73f87 Fri Oct 23 14:18:27 2020 +0000 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/myitcv/.cache/go-build"
GOENV="/home/myitcv/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/myitcv/gostuff/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/myitcv/gostuff"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/myitcv/gos"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/myitcv/gos/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/tmp/tmp.OKr5fuWb4v/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build599471778=/tmp/go-build -gno-record-gcc-switches"

What did you do?

This is a follow up to an exchange with @jayconrod in #tools on Slack:

https://gophers.slack.com/archives/C0VPK4Z5E/p1603464236390000

Consider the following:

$ cd $(mktemp -d)
$ go mod init mod.com
go: creating new go.mod: module mod.com
$ time go get cuelang.org/go
go: cuelang.org/go upgrade => v0.2.2

real    0m6.058s
user    0m0.739s
sys     0m0.225s
$ time go get cuelang.org/go

real    0m3.311s
user    0m0.436s
sys     0m0.254s
$ time go get cuelang.org/go@v0.2.2

real    0m3.240s
user    0m0.428s
sys     0m0.266s

Running the above with a local GOMODCACHE that is already "warm" (i.e. fully populated from a previous go mod download where cuelang.org/go@v0.2.2 is a dependency).

• The initial go get takes too long
• The subsequent go get's also take too long

The time on subsequent go get's appears to be taken checking version lists of transitive dependencies:

...
# get https://proxy.golang.org/cloud.google.com/go/@v/list
# get https://proxy.golang.org/cloud.google.com/go/@v/list: 200 OK (0.015s)
# get https://proxy.golang.org/dmitri.shuralyov.com/gpu/mtl/@v/list
# get https://proxy.golang.org/dmitri.shuralyov.com/gpu/mtl/@v/list: 200 OK (0.024s)
# get https://proxy.golang.org/dmitri.shuralyov.com/gpu/mtl/@latest
# get https://proxy.golang.org/dmitri.shuralyov.com/gpu/mtl/@latest: 200 OK (0.025s)
# get https://proxy.golang.org/github.com/%21burnt%21sushi/toml/@v/list
# get https://proxy.golang.org/github.com/%21burnt%21sushi/toml/@v/list: 200 OK (0.022s)
# get https://proxy.golang.org/github.com/%21burnt%21sushi/xgb/@v/list
...

It is unclear why the initial go get takes almost twice as long.

Also, if you follow, in real time, the output of go get -x cuelang.org/go@v0.2.2, assuming the output is representative of the real time execution of go get, the checking of the version lists of the transitive dependencies appears to happen in serial rather than "as concurrently as possible". (It could of course be that the proxy is doing some sort of rate limiting, but I find that harder to believe)

What did you expect to see?

Fast execution of all the above commands, because my GOMODCACHE is already warm.

What did you see instead?

The above.

Per our discussion, this feels like a fairly significant backwards step in terms of "performance". If this is all attributable to the checking of retracted versions, then I think we need to consider that check being a separate command rather than magically happening as part of other commands.

cc @jayconrod @bcmills @matloob