Thanks for the report.

If I read that stack trace right, the call to invoke on the Method is failing, not the lookup. Looks like a thread safety issue on first sight, but all my Googling so far says java.reflect.Method is thread safe :-/

My guess is sbt 1.3.x's layered class loader is unloading some necessary classes before Scala.js compilation finishes. So trying sbt 1.2.x might be necessary to identify the issue.

That's unlikely. The class loader used for linking is not managed by sbt. We manage it ourselves (see here and here). If sbt 1.3.x's layered class loader messes with our class loader, that would some kind of achievement. :p

Possible workaround: prevent the Scala.js sbt plugin from linking two things at the same time with the following setting:

Global / concurrentRestrictions += Tags.limit(ScalaJSTags.Link, 1)

I'm assigning this to v1.0.0 until we know what is going on. My main concern is that it might invalidate the way we load the linker via reflection.

Confirmed:

• Using sbt 1.2.8 caused the same error.
• Global / concurrentRestrictions += Tags.limit(ScalaJSTags.Link, 1) didn't solve the problem

Created a test branch to reproduce the issue.
wvlet/airframe#895

Note that as of now, I cannot reproduce this locally with a trivial project:

name := "Scala.js Tutorial"
scalaVersion := "2.12.8" // or any other Scala version >= 2.10.2

scalaJSUseMainModuleInitializer := true

lazy val a = project.enablePlugins(ScalaJSPlugin)

lazy val b = project.enablePlugins(ScalaJSPlugin)

lazy val c = project.enablePlugins(ScalaJSPlugin)

lazy val d = project.enablePlugins(ScalaJSPlugin)

lazy val e = project.enablePlugins(ScalaJSPlugin)

JDK8, sbt.version=1.3.6

I will try and make the code less trivial so the fastOptJS will take longer.

Even with less trivial code (the reversi example) fastOpt-ing 5 projects in parallel works just fine :-/

I can reproduce this locally with wvlet/airframe#895.

Seems that this is a long standing problem: "java.lang.reflect.Method invalid after 14 "invokes"

tl;dr: invoke starts behaving differently after being invoked 15 times (or sun.reflect.inflationThreshold times to be precise). This other method seems to fail.

I can reproduce this problem with my 5 project test project if I run
sbt -Dsun.reflect.inflationThreshold=2.

This explains why splitting the aggregate projects works. It also explains why re-running works (scalaJSLinkerImpl is a task, so a new implementation gets created on every run).

From the discussion you linked, it seems that it may be possible to fix/workaround the issue by delegating sun.reflect.* classes to the parent class loader. So basically add "scala.reflect." to our existing list at

Hmm... I really don't like this idea. It means we're opening ourselves up to having to deal with all kinds of implementation details in the parentPrefixes.

An alternative is to filter the classpath we pass to the underlying URLClassLoader: If we remove all the thing that are already provided, we do not have to know anything about package prefixes (and in fact can remove FilteringClassLoader entirely). OTOH, that will not properly isolate GCC and Guava, which is one of the major user-facing advantages of the current approach.

Not isolating GCC and Guava would be annihilate most of the benefits of our class loading approach for users, leaving them only with the disadvantages. That's not an option.

I can confirm that adding "sun." to the parentPrefixes fixes this problem in a local publish on my machine.

Direct reference to the JDK bug

@vjovanov have you ever heard of this?

@gzm0 Nice catch!

"java.lang.reflect.Method invalid after 14 "invokes"

Thank you for finding that. It makes a lot of sense about this consistent behavior.

Added a fix in #3928

I'll comment here (instead of #3928) so everything is in the same thread.

I'd like to clarify the severity of the mess I have created with this classloader thing.

The way we are loading classes makes it impossible for anything in scala.* and java.* (and some org.scalajs packages but that's fine) to depend on any other package name (even just internally). This is quite a serious breach of abstraction and this issue is only a single example of this. A Scala library that depends on sun.misc.unsafe (or similar) could trigger something similar.

As a result, I'd like us to consider the consequences on really committing to this setup and consider alternatives thoroughly. Since in theory, this means LinkerImpl can break with any minor JDK, Scala, etc. update.

That in turn means that I'm very uncomfortable releasing 1.0.0 with merely having a workaround for this issue (I'm OK with a workaround if we know a way to fix it in a compatible manner and/or are OK with accepting the current state).

I don't understand why this is such an issue. It happens sometimes that an upgrade in a dependency causes unintentional breakages (or intentional breakages that affect us in a way we thought unexpected), and we have to deal with it. Scala itself sometimes has to publish new versions following upgrades of the JDK (e.g., the issue that required us to have this workaround for a while)).

It's also worth pointing out that adding sun.* is a workaround for a JDK bug. It shouldn't be needed. The JDK should work just fine with loading everything else in sub-classloader like we are doing. We're not inherently relying on loading everything needed by the stdlib from the parent classloader. By spec, the code we wrote before ought to work everywhere.

Another way to look at it is that adding sun.* is a workaround, sure, but not a workaround for this issue in Scala.js (#3921). It is a workaround for the OpenJDK bug 6265952. We're working around someone else's bug; not our own abuse of unspecified behavior.

In that, the current comment in the PR is misleading I think. It should first foremost say that it is a workaround for that OpenJDK bug, then only reference the present issue for context.

By spec, the code we wrote before ought to work everywhere.

Does it? Then maybe I'm misunderstanding what is happening here. If this is the case, of course a workaround like #3928 is perfectly acceptable.

I don't have time right now, but I'd like to slice down what exactly is happening here (which classloader is doing what, etc.). If it turns out, we are in spec, then yes, the workaround is fine.

After some thinking, I think @sjrd is correct. This is a clearly a JDK bug only and not Scala.js abusing an implementation detail. Notably, my previous depiction of the situation is wrong: There is no coupling between dependencies of dependencies, only on direct dependencies (i.e. org.scalajs.linker.interface).

Let's make an example and take two (direct "module") dependencies of StandardImpl:

• java.util.concurrent.atomic.AtomicBoolean
• com.google.javascript.jscomp.Compiler

AtomicBoolean is loaded by the parent ClassLoader. As such, all its dependencies are immediately loaded by the parent ClassLoader (there is no reference to the private class loader anymore). If one is missing, the calling classpath is not properly set up.

Compiler (GCC) is loaded by the private ClassLoader. As such, all its dependencies are first queried on the private class loader. We have two cases here:

1. The dependency gets delegated upwards (matches parentPrefixes) or,
2. it is loaded on the private ClassLoader.

Case 2 is as intended and "recurses" into the Compiler case.

Case 1 is a bit more subtle and splits in two cases itself: The loaded dependency does not have any dependencies itself on the loading dependency (i.e. Compiler in this example). We are in the AtomicBoolean case.

However, if the loaded dependency "reverse depends" on the loading dependency (Compiler), we'll load the loading dependency again in the parent ClassLoader. This is not a problem, as long as it is not part of the interface between these two dependencies. Otherwise, doom will happen (something like "expected Foo but got Foo").

Concluding, if a class matching parentPrefixes depends on a class not matching parentPrefixes in their mutual interface our approach will fail.

While this is theoretically possible out of our control, IMHO it is highly unlikely and hence acceptable.

Coming back to this specific case here, I presume the implementation of Method.invoke simply uses the wrong ClassLoader to find MethodAccessorImpl: It seems to use the ClassLoader of the owner of the method the Method is representing and not the ClassLoader of the class Method itself.

Oh, I forgot: @sjrd does the last comment match your understanding of the situation?

Yes, that matches my understanding as well, with a more detailed analysis that what I had done :)

Actually, case 2 of the private dependency is only correct when it tries to load non-provided dependencies (i.e. stuff it declares in its POM). So technically we'd need to add all packages in the JDK to the parentPrefixes list (javax. and maybe others). But IMHO that can wait until necessary.

Oh, I just hit his while re-engineering my build file under 1.0.0-RC2. sjrd's fix for cuncurrentRestrictions did not fix it for me. I had been just compiling projects one at a time during upgrade testing.

Is there another workaround? I have 25 scala.js projects in one build. I've tried various jdks with no luck.

As an interesting sidenote, I have another unrelated project that references several of those projects via a sbt ProjectRef and it works fine when cleaning and compiling everything in that unrelated project (which cleans and rebuilds almost everything in the project with many scala.js projects).

Is there another workaround? I have 25 scala.js projects in one build. I've tried various jdks with no luck.

You can probably work around the issue by invoking sbt with

sbt -Dsun.reflect.inflationThreshold=30

You can use a higher number of necessary. With 25 projects, anything above 25 should be fine.

That works.

Fixed in a2ee859