Skip to content

Changes Extend trait in order to allow streams that yield references #266

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Oct 4, 2019

Conversation

sunjay
Copy link
Contributor

@sunjay sunjay commented Oct 1, 2019

This is not ready to merge yet. I am mainly opening it so we can discuss a change I had to make to the Extend trait. cc @yoshuawuyts @stjepang (and anyone else interested)

Before this can be merged

  • Discuss/Approve changes to Extend trait
  • Change to using for_each after Adds for_each stream combinator #264 is merged
  • (optional) Wait until a copied() method is added to StreamExt so that the &char impl can be finished.
    • We can also just comment out or remove the impl that uses copied until that is added

Changes To The Extend Trait

While writing the impls of the Extend trait for the String type, I noticed that certain impls weren't possible because there is no bound on Extend that guarantees that the type A being yielded from the stream actually lives long enough. We probably didn't run into this earlier because this usually isn't a problem for owned values since the compiler doesn't have to worry about whether they will out live the stream that they come from. I ran into this because of the Extend impls that consume streams of references.

The difference between the async Extend and the standard library Extend is that the async Extend returns a value that still references the input stream. That means that if A is any reference type, the compiler needs to be able to guarantee that A will be around as long as the Future returned from the trait method is around.

To fix this, I had to add the bound shown below:

 pub trait Extend<A> {
     /// Extends a collection with the contents of a stream.
     fn stream_extend<'a, T: IntoStream<Item = A> + 'a>(
         &'a mut self,
         stream: T,
-    ) -> Pin<Box<dyn Future<Output = ()> + 'a>>;
+    ) -> Pin<Box<dyn Future<Output = ()> + 'a>> where A: 'a;
 }

This guarantees that each value of type A will last at least as long as our boxed future does. The bound had to be in a where clause on the method (and not on the declaration of A because the lifetime 'a isn't in scope at the trait level. I don't think there are any negative consequences of using a where clause like this, but that's why I wanted to bring it up for discussion.

In addition to this, I had to ensure that when writing the Extend impls for String I appropriately bounded the lifetime of the references from the stream. You can see this in the code below with where 'b: 'a.

impl<'b> Extend<&'b str> for String {
    fn stream_extend<'a, S: IntoStream<Item = &'b str> + 'a>(
        &'a mut self,
        stream: S,
    ) -> Pin<Box<dyn Future<Output = ()> + 'a>> where 'b: 'a {
        //TODO: This can just be: stream.into_stream().for_each(move |s| self.push_str(s))
        Box::pin(stream.into_stream().fold((), move |(), s| self.push_str(s)))
    }
}

I should note that initially I tried to make it work with just the impl shown above, without modifying the Extend trait. This doesn't work because it would be a stricter bound than what is found in the trait itself. Rust does not allow stricter bounds like that because it could potentially cause unsoundness when dealing with generics.

Of course, I am totally open to being completely wrong in my understanding of how to resolve this issue. I tried to solve the problem with as minimal of a change as possible. Please let me know if you have some better ideas or other suggestions.

FromStream impls for String

The purpose of adding these Extend impls is to continue my work from #129 in adding the rest of the FromStream impls. The Extend impls are used directly to add all of the FromStream impls for String. Just like with #207 and #265, this adds a new string module that is unstable just like the other modules added for FromStream.

@sunjay sunjay mentioned this pull request Oct 1, 2019
87 tasks
@yoshuawuyts
Copy link
Contributor

Please let me know if you have some better ideas or other suggestions.

@sunjay thanks for the detailed write-up! -- It definitely sounds like you've put a good deal of thought into this, and above all: this seems to be working!

I'm very excited about this patch set; this is very good! Thanks so much for doing this!

@sunjay sunjay marked this pull request as ready for review October 2, 2019 02:40
@sunjay
Copy link
Contributor Author

sunjay commented Oct 2, 2019

@yoshuawuyts This is done and ready to review/merge! I used for_each (as implemented in #264) to finish off some of the remaining TODO comments.

There is still one impl whose body is unimplemented!() until Stream gets the copied combinator. There are also some performance things that have been commented out until size_hint is added. I can't remove the unimplemented!() impl entirely because one of the FromStream impls relies on it.

I think this is in a reasonable state to merge. All of these impls are unstable, so I don't think the unimplemented is a big deal. Up to you!

@sunjay
Copy link
Contributor Author

sunjay commented Oct 2, 2019

Could you restart the build? It seems to have run into an internal compiler error.

@yoshuawuyts
Copy link
Contributor

bors r+

bors bot added a commit that referenced this pull request Oct 4, 2019
266: Changes Extend trait in order to allow streams that yield references r=yoshuawuyts a=sunjay

This is not ready to merge yet. I am mainly opening it so we can discuss a change I had to make to the `Extend` trait. cc @yoshuawuyts @stjepang (and anyone else interested)

## Before this can be merged

- [x] Discuss/Approve changes to `Extend` trait
- [x] Change to using `for_each` after #264 is merged
- [ ] (optional) Wait until a `copied()` method is added to `StreamExt` so that the `&char` impl can be finished.
    - We can also just comment out or remove the impl that uses `copied` until that is added

## Changes To The Extend Trait

While writing the impls of the `Extend` trait for the `String` type, I noticed that certain impls weren't possible because there is no bound on `Extend` that guarantees that the type `A` being yielded from the stream actually lives long enough. We probably didn't run into this earlier because this usually isn't a problem for owned values since the compiler doesn't have to worry about whether they will out live the stream that they come from. I ran into this because of the `Extend` impls that consume streams of references.

The difference between the async `Extend` and the standard library `Extend` is that the async `Extend` returns a value that still references the input stream. That means that if `A` is any reference type, the compiler needs to be able to guarantee that `A` will be around as long as the `Future` returned from the trait method is around.

To fix this, I had to add the bound shown below:

```patch
 pub trait Extend<A> {
     /// Extends a collection with the contents of a stream.
     fn stream_extend<'a, T: IntoStream<Item = A> + 'a>(
         &'a mut self,
         stream: T,
-    ) -> Pin<Box<dyn Future<Output = ()> + 'a>>;
+    ) -> Pin<Box<dyn Future<Output = ()> + 'a>> where A: 'a;
 }
```

This guarantees that each value of type `A` will last at least as long as our boxed future does. The bound had to be in a where clause on the method (and not on the declaration of `A` because the lifetime `'a` isn't in scope at the trait level. I don't think there are any negative consequences of using a where clause like this, but that's why I wanted to bring it up for discussion.

In addition to this, I had to ensure that when writing the `Extend` impls for `String` I appropriately bounded the lifetime of the references from the stream. You can see this in the code below with `where 'b: 'a`.

```rust
impl<'b> Extend<&'b str> for String {
    fn stream_extend<'a, S: IntoStream<Item = &'b str> + 'a>(
        &'a mut self,
        stream: S,
    ) -> Pin<Box<dyn Future<Output = ()> + 'a>> where 'b: 'a {
        //TODO: This can just be: stream.into_stream().for_each(move |s| self.push_str(s))
        Box::pin(stream.into_stream().fold((), move |(), s| self.push_str(s)))
    }
}
```

I should note that initially I tried to make it work with just the impl shown above, without modifying the `Extend` trait. This doesn't work because it would be a stricter bound than what is found in the trait itself. Rust does not allow stricter bounds like that because it could potentially cause unsoundness when dealing with generics.

Of course, I am totally open to being completely wrong in my understanding of how to resolve this issue. I tried to solve the problem with as minimal of a change as possible. Please let me know if you have some better ideas or other suggestions.

## `FromStream` impls for String

The purpose of adding these `Extend` impls is to continue my work from #129 in adding the rest of the `FromStream` impls. The `Extend` impls are used directly to add all of the `FromStream` impls for `String`. Just like with #207 and #265, this adds a new `string` module that is unstable just like the other modules added for `FromStream`.

Co-authored-by: Sunjay Varma <varma.sunjay@gmail.com>
@bors
Copy link
Contributor

bors bot commented Oct 4, 2019

Build succeeded

  • continuous-integration/travis-ci/push

@bors bors bot merged commit 09a15ef into async-rs:master Oct 4, 2019
@sunjay sunjay deleted the extend-string branch October 4, 2019 14:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants