Skip to content

[indylambda] Support lambda {de}serialization #4501

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
May 21, 2015

Conversation

retronym
Copy link
Member

To support serialization, we use the alternative lambda metafactory that
lets us specify that our anonymous functions should extend the marker
interface scala.Serializable. They will also have a
writeObject method added that implements the serialization proxy pattern
using j.l.invoke.SerializedLamba.

To support deserialization, we synthesize a $deserializeLamba$ method in
each class. This will be called reflectively by
SerializedLambda#readResolve. This method in turn delegates to
LambdaDeserializer, currently defined [1] in scala-java8-compat, that
uses LambdaMetafactory to spin up the anonymous class and instantiate it
with the deserialized environment.

Note: LambdaDeserializer reuses the anonymous class on subsequent
deserializations of a given lambda, in the same spirit as an invokedynamic
call site only spins up the class on the first time it is run.

LambdaDeserializer will be moved into our standard library in the 2.12.x
branch, where we can introduce dependencies on the Java 8 standard library.

The enclosed test cases must be manually run with indylambda enabled. Once
we enable indylambda by default on 2.12.x, the test will actually test the
new feature.

% echo $INDYLAMBDA
-Ydelambdafy:method -Ybackend:GenBCode -target:jvm-1.8 -classpath
.:scala-java8-compat_2.11-0.5.0-SNAPSHOT.jar
% $INDYLAMBDA -e "println((() => 42).getClass)" class
Main$$anon$1$$Lambda$1/1183231938
% qscala $INDYLAMBDA -e "assert(classOf[scala.Serializable].isInstance(()
=> 42))"
% qscalac $INDYLAMBDA test/files/run/lambda-serialization.scala && qscala
$INDYLAMBDA Test

This commit contains a few minor refactorings to the code that generates
the invokedynamic instruction to use more meaningful names and to reuse
Java signature generation code in ASM rather than the DIY approach.

[1] scala/scala-java8-compat#37

@scala-jenkins scala-jenkins added this to the 2.11.7 milestone May 15, 2015
@retronym
Copy link
Member Author

Review by @lrytz @adriaanm

Builds on top of #4497.

I will add a commit to update our reference in build.xml to 0.5.0 of scala-java8-compat when we merge scala/scala-java8-compat#37 and release a new version.

@retronym retronym force-pushed the topic/indylambda-serialization branch from 97efb4f to 5221627 Compare May 15, 2015 07:07
@lrytz
Copy link
Member

lrytz commented May 15, 2015

High-level question: could we introduce SerializableFunctionN functional interfaces instead of switching to altMetafactory?

@retronym retronym force-pushed the topic/indylambda-serialization branch 2 times, most recently from a0a6efe to 8fc192b Compare May 16, 2015 02:21
@retronym retronym closed this May 16, 2015
@retronym retronym reopened this May 16, 2015
@retronym retronym force-pushed the topic/indylambda-serialization branch from 8fc192b to c76808c Compare May 16, 2015 02:27
@retronym
Copy link
Member Author

metafactory never generates the code to implement the serialization proxy pattern, it passes isSerializable = false here.

altMetafactory lets us pass in that value via FLAG_SERIALIZABLE. If neither functional interface nor the provided marker interfaces extend j.io.Serializable, that is automatically added as an additional marker.

@retronym retronym force-pushed the topic/indylambda-serialization branch 2 times, most recently from 75d8595 to 4e4affd Compare May 16, 2015 03:07
@retronym
Copy link
Member Author

Needs a bit more work, will reopen early next week.

I'm trying to test this out with:

import java.io._
import java.{net, util}

import scala.reflect.internal.util.ScalaClassLoader.URLClassLoader
import scala.reflect.io.Streamable

class C {
  def serializeDeserialize[T <: AnyRef](obj: T) = {
    val buffer = new ByteArrayOutputStream
    val out = new ObjectOutputStream(buffer)
    out.writeObject(obj)
    val in = new ObjectInputStream(new ByteArrayInputStream(buffer.toByteArray))
    in.readObject.asInstanceOf[T]
  }

  serializeDeserialize((c: String) => c.length)

}

object Test {
  def main(args: Array[String]): Unit = {
    roundTrip
  }

  def roundTrip(): Unit = {
    val loaderCClass = classOf[C]
    val loader = getClass.getClassLoader.asInstanceOf[URLClassLoader]
    def deserializedInThrowawayClassloader = {
      val throwawayLoader: java.net.URLClassLoader = new java.net.URLClassLoader(loader.getURLs, ClassLoader.getSystemClassLoader) {
        val junk = Array.ofDim(32 * 1024 * 1024)
      }
      val clazz = throwawayLoader.loadClass("C")
      assert(clazz != loaderCClass)
      clazz.newInstance()
    }
    (1 to 100) foreach { _ =>
      deserializedInThrowawayClassloader
    }
  }

}

@retronym retronym closed this May 16, 2015
@retronym retronym reopened this May 17, 2015
@retronym retronym force-pushed the topic/indylambda-serialization branch 3 times, most recently from db5301d to b661900 Compare May 17, 2015 09:26
To support serialization, we use the alternative lambda metafactory
that lets us specify that our anonymous functions should extend the
marker interface `scala.Serializable`. They will also have a
`writeObject` method added that implements the serialization proxy
pattern using `j.l.invoke.SerializedLamba`.

To support deserialization, we synthesize a `$deserializeLamba$`
method in each class with lambdas. This will be called reflectively by
`SerializedLambda#readResolve`. This method in turn delegates to
`LambdaDeserializer`, currently defined [1] in `scala-java8-compat`,
that uses `LambdaMetafactory` to spin up the anonymous class and
instantiate it with the deserialized environment.

Note: `LambdaDeserializer` can reuses the anonymous class on subsequent
deserializations of a given lambda, in the same spirit as an
invokedynamic call site only spins up the class on the first time
it is run. But first we'll need to host a cache in a static field
of each lambda hosting class. This is noted as a TODO and a failing
test, and will be updated in the next commit.

`LambdaDeserializer` will be moved into our standard library in
the 2.12.x branch, where we can introduce dependencies on the
Java 8 standard library.

The enclosed test cases must be manually run with indylambda enabled.
Once we enable indylambda by default on 2.12.x, the test will
actually test the new feature.

```
% echo $INDYLAMBDA
-Ydelambdafy:method -Ybackend:GenBCode -target:jvm-1.8 -classpath .:scala-java8-compat_2.11-0.5.0-SNAPSHOT.jar
% qscala $INDYLAMBDA -e "println((() => 42).getClass)"
class Main$$anon$1$$Lambda$1/1183231938
% qscala $INDYLAMBDA -e "assert(classOf[scala.Serializable].isInstance(() => 42))"
% qscalac $INDYLAMBDA test/files/run/lambda-serialization.scala && qscala $INDYLAMBDA Test
```

This commit contains a few minor refactorings to the code that
generates the invokedynamic instruction to use more meaningful
names and to reuse Java signature generation code in ASM rather
than the DIY approach.

[1] scala/scala-java8-compat#37
@retronym retronym force-pushed the topic/indylambda-serialization branch from b661900 to d3b42d1 Compare May 17, 2015 09:30
@lrytz
Copy link
Member

lrytz commented May 17, 2015

Thanks for the explanation why we need to use altMetafactory, I didn't realize javac does the same: it switches to altMetafactory if the functional interface extends serializable. Looking at the code you linked, this is actually a requirement: if you use metafactory for a serializable functional interface, it generates writeObject and readObject methods that throw.

We add a static field to each class that defines lambdas that
will hold a `ju.Map[String, MethodHandle]` to cache references to
the constructors of the classes originally created by
`LambdaMetafactory`.

The cache is initially null, and created on the first deserialization.

In case of a race between two threads deserializing the first
lambda hosted by a class, the last one to finish will clobber
the one-element cache of the first.

This lack of strong guarantees mirrors the current policy in
`LambdaDeserializer`.

We should consider whether to strengthen the combinaed guarantee here.
A useful benchmark would be those of the invokedynamic instruction,
which allows multiple threads to call the boostrap method in parallel,
but guarantees that if that happens, the results of all but one will
be discarded:

> If several threads simultaneously execute the bootstrap method for
> the same dynamic call site, the Java Virtual Machine must choose
> one returned call site object and install it visibly to all threads.

We could meet this guarantee easily, albeit excessively, by
synchronizing `$deserializeLambda$`. But a more fine grained
approach is possible and desirable.

A test is included that shows we are able to garbage collect
classloaders of classes that have hosted lambda deserialization.
@retronym retronym force-pushed the topic/indylambda-serialization branch from d3b42d1 to 1d8c632 Compare May 17, 2015 23:08
@lrytz
Copy link
Member

lrytz commented May 20, 2015

LGTM!

@adriaanm
Copy link
Contributor

🚀

adriaanm added a commit that referenced this pull request May 21, 2015
[indylambda] Support lambda {de}serialization
@adriaanm adriaanm merged commit 97543db into scala:2.11.x May 21, 2015
@lrytz
Copy link
Member

lrytz commented May 22, 2015

dkimitsa added a commit to dkimitsa/robovm that referenced this pull request Dec 31, 2023
currently only StringConcat and Lambda bootstraps are recognized and cause DynamicInvoke to be transformed/dessugared.

in other cases DynamicInvoke instruction will stay in place and will cause compilation type exception:
> Java.lang.ClassCastException: soot.jimple.internal.JDynamicInvokeExpr cannot be cast to soot.jimple.InstanceInvokeExpr

Issue was risen in gitter channel in scope Scala/desirialize labda being inserted in all classes. All these classes were failed to compile. Even if this functionality is not used (scala/scala#4501).

As a workaround changed how InvokeDynamic is being handled:
- introduced single InbokeDynamicCompilerPlugin;
- LabdaPlugin and StringconcatRewriter plugins are made as delegates of InbokeDynamicCompilerPlugin;
- all not recognized InvokeDynamic are now translated into NoSuchMethodError exceptions
Tom-Ski added a commit to MobiVM/robovm that referenced this pull request Feb 28, 2024
currently only StringConcat and Lambda bootstraps are recognized and cause DynamicInvoke to be transformed/dessugared.

in other cases DynamicInvoke instruction will stay in place and will cause compilation type exception:
> Java.lang.ClassCastException: soot.jimple.internal.JDynamicInvokeExpr cannot be cast to soot.jimple.InstanceInvokeExpr

Issue was risen in gitter channel in scope Scala/desirialize labda being inserted in all classes. All these classes were failed to compile. Even if this functionality is not used (scala/scala#4501).

As a workaround changed how InvokeDynamic is being handled:
- introduced single InbokeDynamicCompilerPlugin;
- LabdaPlugin and StringconcatRewriter plugins are made as delegates of InbokeDynamicCompilerPlugin;
- all not recognized InvokeDynamic are now translated into NoSuchMethodError exceptions

Co-authored-by: Tomski <tomwojciechowski@asidik.com>
dkimitsa added a commit to dkimitsa/robovm that referenced this pull request Apr 15, 2024
currently only StringConcat and Lambda bootstraps are recognized and cause DynamicInvoke to be transformed/dessugared.

in other cases DynamicInvoke instruction will stay in place and will cause compilation type exception:
> Java.lang.ClassCastException: soot.jimple.internal.JDynamicInvokeExpr cannot be cast to soot.jimple.InstanceInvokeExpr

Issue was risen in gitter channel in scope Scala/desirialize labda being inserted in all classes. All these classes were failed to compile. Even if this functionality is not used (scala/scala#4501).

As a workaround changed how InvokeDynamic is being handled:
- introduced single InbokeDynamicCompilerPlugin;
- LabdaPlugin and StringconcatRewriter plugins are made as delegates of InbokeDynamicCompilerPlugin;
- all not recognized InvokeDynamic are now translated into NoSuchMethodError exceptions

Co-authored-by: Tomski <tomwojciechowski@asidik.com>
(cherry picked from commit 8675d71)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants