-
Notifications
You must be signed in to change notification settings - Fork 3.1k
[indylambda] Support lambda {de}serialization #4501
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[indylambda] Support lambda {de}serialization #4501
Conversation
Builds on top of #4497. I will add a commit to update our reference in build.xml to 0.5.0 of scala-java8-compat when we merge scala/scala-java8-compat#37 and release a new version. |
97efb4f
to
5221627
Compare
High-level question: could we introduce |
a0a6efe
to
8fc192b
Compare
8fc192b
to
c76808c
Compare
|
75d8595
to
4e4affd
Compare
Needs a bit more work, will reopen early next week. I'm trying to test this out with:
|
db5301d
to
b661900
Compare
To support serialization, we use the alternative lambda metafactory that lets us specify that our anonymous functions should extend the marker interface `scala.Serializable`. They will also have a `writeObject` method added that implements the serialization proxy pattern using `j.l.invoke.SerializedLamba`. To support deserialization, we synthesize a `$deserializeLamba$` method in each class with lambdas. This will be called reflectively by `SerializedLambda#readResolve`. This method in turn delegates to `LambdaDeserializer`, currently defined [1] in `scala-java8-compat`, that uses `LambdaMetafactory` to spin up the anonymous class and instantiate it with the deserialized environment. Note: `LambdaDeserializer` can reuses the anonymous class on subsequent deserializations of a given lambda, in the same spirit as an invokedynamic call site only spins up the class on the first time it is run. But first we'll need to host a cache in a static field of each lambda hosting class. This is noted as a TODO and a failing test, and will be updated in the next commit. `LambdaDeserializer` will be moved into our standard library in the 2.12.x branch, where we can introduce dependencies on the Java 8 standard library. The enclosed test cases must be manually run with indylambda enabled. Once we enable indylambda by default on 2.12.x, the test will actually test the new feature. ``` % echo $INDYLAMBDA -Ydelambdafy:method -Ybackend:GenBCode -target:jvm-1.8 -classpath .:scala-java8-compat_2.11-0.5.0-SNAPSHOT.jar % qscala $INDYLAMBDA -e "println((() => 42).getClass)" class Main$$anon$1$$Lambda$1/1183231938 % qscala $INDYLAMBDA -e "assert(classOf[scala.Serializable].isInstance(() => 42))" % qscalac $INDYLAMBDA test/files/run/lambda-serialization.scala && qscala $INDYLAMBDA Test ``` This commit contains a few minor refactorings to the code that generates the invokedynamic instruction to use more meaningful names and to reuse Java signature generation code in ASM rather than the DIY approach. [1] scala/scala-java8-compat#37
b661900
to
d3b42d1
Compare
Thanks for the explanation why we need to use |
We add a static field to each class that defines lambdas that will hold a `ju.Map[String, MethodHandle]` to cache references to the constructors of the classes originally created by `LambdaMetafactory`. The cache is initially null, and created on the first deserialization. In case of a race between two threads deserializing the first lambda hosted by a class, the last one to finish will clobber the one-element cache of the first. This lack of strong guarantees mirrors the current policy in `LambdaDeserializer`. We should consider whether to strengthen the combinaed guarantee here. A useful benchmark would be those of the invokedynamic instruction, which allows multiple threads to call the boostrap method in parallel, but guarantees that if that happens, the results of all but one will be discarded: > If several threads simultaneously execute the bootstrap method for > the same dynamic call site, the Java Virtual Machine must choose > one returned call site object and install it visibly to all threads. We could meet this guarantee easily, albeit excessively, by synchronizing `$deserializeLambda$`. But a more fine grained approach is possible and desirable. A test is included that shows we are able to garbage collect classloaders of classes that have hosted lambda deserialization.
d3b42d1
to
1d8c632
Compare
LGTM! |
🚀 |
[indylambda] Support lambda {de}serialization
✅ |
currently only StringConcat and Lambda bootstraps are recognized and cause DynamicInvoke to be transformed/dessugared. in other cases DynamicInvoke instruction will stay in place and will cause compilation type exception: > Java.lang.ClassCastException: soot.jimple.internal.JDynamicInvokeExpr cannot be cast to soot.jimple.InstanceInvokeExpr Issue was risen in gitter channel in scope Scala/desirialize labda being inserted in all classes. All these classes were failed to compile. Even if this functionality is not used (scala/scala#4501). As a workaround changed how InvokeDynamic is being handled: - introduced single InbokeDynamicCompilerPlugin; - LabdaPlugin and StringconcatRewriter plugins are made as delegates of InbokeDynamicCompilerPlugin; - all not recognized InvokeDynamic are now translated into NoSuchMethodError exceptions
currently only StringConcat and Lambda bootstraps are recognized and cause DynamicInvoke to be transformed/dessugared. in other cases DynamicInvoke instruction will stay in place and will cause compilation type exception: > Java.lang.ClassCastException: soot.jimple.internal.JDynamicInvokeExpr cannot be cast to soot.jimple.InstanceInvokeExpr Issue was risen in gitter channel in scope Scala/desirialize labda being inserted in all classes. All these classes were failed to compile. Even if this functionality is not used (scala/scala#4501). As a workaround changed how InvokeDynamic is being handled: - introduced single InbokeDynamicCompilerPlugin; - LabdaPlugin and StringconcatRewriter plugins are made as delegates of InbokeDynamicCompilerPlugin; - all not recognized InvokeDynamic are now translated into NoSuchMethodError exceptions Co-authored-by: Tomski <tomwojciechowski@asidik.com>
currently only StringConcat and Lambda bootstraps are recognized and cause DynamicInvoke to be transformed/dessugared. in other cases DynamicInvoke instruction will stay in place and will cause compilation type exception: > Java.lang.ClassCastException: soot.jimple.internal.JDynamicInvokeExpr cannot be cast to soot.jimple.InstanceInvokeExpr Issue was risen in gitter channel in scope Scala/desirialize labda being inserted in all classes. All these classes were failed to compile. Even if this functionality is not used (scala/scala#4501). As a workaround changed how InvokeDynamic is being handled: - introduced single InbokeDynamicCompilerPlugin; - LabdaPlugin and StringconcatRewriter plugins are made as delegates of InbokeDynamicCompilerPlugin; - all not recognized InvokeDynamic are now translated into NoSuchMethodError exceptions Co-authored-by: Tomski <tomwojciechowski@asidik.com> (cherry picked from commit 8675d71)
To support serialization, we use the alternative lambda metafactory that
lets us specify that our anonymous functions should extend the marker
interface
scala.Serializable
. They will also have awriteObject
method added that implements the serialization proxy patternusing
j.l.invoke.SerializedLamba
.To support deserialization, we synthesize a
$deserializeLamba$
method ineach class. This will be called reflectively by
SerializedLambda#readResolve
. This method in turn delegates toLambdaDeserializer
, currently defined [1] inscala-java8-compat
, thatuses
LambdaMetafactory
to spin up the anonymous class and instantiate itwith the deserialized environment.
Note:
LambdaDeserializer
reuses the anonymous class on subsequentdeserializations of a given lambda, in the same spirit as an invokedynamic
call site only spins up the class on the first time it is run.
LambdaDeserializer
will be moved into our standard library in the 2.12.xbranch, where we can introduce dependencies on the Java 8 standard library.
The enclosed test cases must be manually run with indylambda enabled. Once
we enable indylambda by default on 2.12.x, the test will actually test the
new feature.
This commit contains a few minor refactorings to the code that generates
the invokedynamic instruction to use more meaningful names and to reuse
Java signature generation code in ASM rather than the DIY approach.
[1] scala/scala-java8-compat#37