Memory Leak on ActiveRecord on #to_proc #7112
Comments
|
Oh yeah, I forgot to mention this is probably related to issue #6929, but I'm pretty sure I've ironed out the GC quirks on this repro script cc @vaharoni @oscardelben @thedarkone |
|
cc/ @tenderlove |
|
What Ruby are you using? I would expect the following program to leak infinitely, but it does not: require 'my_model'
def omg
MyModel.all.group_by(&:id)
nil
end
loop do
omg
endOn my machine this stabilizes at around 70MB. Can you write a script that will leak infinitely? |
|
I was unable to repro this with rbx: $ ruby -Xversion=19 lib/leak.rb
1) No leaks: MyModel -> 0, MyAssociation -> 0 (1 is fine, 2+ is leak)
2) No leaks: MyModel -> 0, MyAssociation -> 0 (1 is fine, 2+ is leak)
3) No leaks: MyModel -> 0, MyAssociation -> 1 (1 is fine, 2+ is leak)
4) No leaks: MyModel -> 0, MyAssociation -> 1 (1 is fine, 2+ is leak)
5) No Leaks: MyModel -> 0, MyAssociation -> 1 (1 is fine, 2+ is leak)
6) Leaks memory: MyModel -> 0, MyAssociation -> 1 (1 is fine, 2+ is leak)
7) Still leaking memory: MyModel -> 0, MyAssociation -> 1 (1 is fine, 2+ is leak)
8) Doesn't leak more memory: MyModel -> 0, MyAssociation -> 1 (1 is fine, 2+ is leak)
9) Leaks more memory: MyModel -> 0, MyAssociation -> 1 (1 is fine, 2+ is leak)
10) Still leaking memory: MyModel -> 0, MyAssociation -> 1 (1 is fine, 2+ is leak)
11) Still leaking memory: MyModel -> 0, MyAssociation -> 1 (1 is fine, 2+ is leak)
Final) MyModel -> 0, MyAssociation -> 1 (1 is fine, 2+ is leak)
$ ruby -Xversion=19 ./script/rails r lib/leak.rb
1) No leaks: MyModel -> 0, MyAssociation -> 0 (1 is fine, 2+ is leak)
2) No leaks: MyModel -> 0, MyAssociation -> 0 (1 is fine, 2+ is leak)
3) No leaks: MyModel -> 0, MyAssociation -> 0 (1 is fine, 2+ is leak)
4) No leaks: MyModel -> 0, MyAssociation -> 0 (1 is fine, 2+ is leak)
5) No Leaks: MyModel -> 0, MyAssociation -> 0 (1 is fine, 2+ is leak)
6) Leaks memory: MyModel -> 0, MyAssociation -> 0 (1 is fine, 2+ is leak)
7) Still leaking memory: MyModel -> 0, MyAssociation -> 0 (1 is fine, 2+ is leak)
8) Doesn't leak more memory: MyModel -> 0, MyAssociation -> 0 (1 is fine, 2+ is leak)
9) Leaks more memory: MyModel -> 0, MyAssociation -> 0 (1 is fine, 2+ is leak)
10) Still leaking memory: MyModel -> 0, MyAssociation -> 0 (1 is fine, 2+ is leak)
11) Still leaking memory: MyModel -> 0, MyAssociation -> 0 (1 is fine, 2+ is leak)
Final) MyModel -> 0, MyAssociation -> 0 (1 is fine, 2+ is leak)nor Ruby 1.9: $ ruby lib/leak.rb
1) No leaks: MyModel -> 0, MyAssociation -> 0 (1 is fine, 2+ is leak)
2) No leaks: MyModel -> 0, MyAssociation -> 0 (1 is fine, 2+ is leak)
3) No leaks: MyModel -> 0, MyAssociation -> 0 (1 is fine, 2+ is leak)
4) No leaks: MyModel -> 0, MyAssociation -> 0 (1 is fine, 2+ is leak)
5) No Leaks: MyModel -> 0, MyAssociation -> 0 (1 is fine, 2+ is leak)
6) Leaks memory: MyModel -> 1, MyAssociation -> 0 (1 is fine, 2+ is leak)
7) Still leaking memory: MyModel -> 1, MyAssociation -> 0 (1 is fine, 2+ is leak)
8) Doesn't leak more memory: MyModel -> 1, MyAssociation -> 0 (1 is fine, 2+ is leak)
9) Leaks more memory: MyModel -> 1, MyAssociation -> 1 (1 is fine, 2+ is leak)
10) Still leaking memory: MyModel -> 1, MyAssociation -> 1 (1 is fine, 2+ is leak)
11) Still leaking memory: MyModel -> 1, MyAssociation -> 1 (1 is fine, 2+ is leak)
Final) MyModel -> 1, MyAssociation -> 1 (1 is fine, 2+ is leak)
$ ruby ./script/rails r lib/leak.rb
1) No leaks: MyModel -> 0, MyAssociation -> 0 (1 is fine, 2+ is leak)
2) No leaks: MyModel -> 0, MyAssociation -> 0 (1 is fine, 2+ is leak)
3) No leaks: MyModel -> 0, MyAssociation -> 0 (1 is fine, 2+ is leak)
4) No leaks: MyModel -> 0, MyAssociation -> 0 (1 is fine, 2+ is leak)
5) No Leaks: MyModel -> 0, MyAssociation -> 0 (1 is fine, 2+ is leak)
6) Leaks memory: MyModel -> 0, MyAssociation -> 0 (1 is fine, 2+ is leak)
7) Still leaking memory: MyModel -> 0, MyAssociation -> 0 (1 is fine, 2+ is leak)
8) Doesn't leak more memory: MyModel -> 0, MyAssociation -> 0 (1 is fine, 2+ is leak)
9) Leaks more memory: MyModel -> 0, MyAssociation -> 0 (1 is fine, 2+ is leak)
10) Still leaking memory: MyModel -> 0, MyAssociation -> 0 (1 is fine, 2+ is leak)
11) Still leaking memory: MyModel -> 0, MyAssociation -> 0 (1 is fine, 2+ is leak)
Final) MyModel -> 0, MyAssociation -> 0 (1 is fine, 2+ is leak)btw This might be a non issue, but I also see you're using a generic Rails bin stub: |
|
@tenderlove: yeah, also see my comment on the previous ticket, whereby adding some real GC pressure in the form of |
|
Well running GC.start doesn't guarantee anything is going to happen. The only way we can know for sure that there is actually a memory leak is if you can observe unbounded memory growth. Counting objects in ObjectSpace isn't going to be a reliable way to find leaks. :-/ |
|
@thedarkone I'm guessing that inspecting ObjectSpace isn't going to really give us good info. I'm going to mark this as closed since I cannot observe unbounded memory growth. If someone can provide a script that does show that problem, please comment and I'll reopen! ❤️ |
|
Hi, thanks for looking into it guys, you are awesome! It makes sense that if ObjectSpace isn't reliable to count references than the only way to assert a leak is by unbounded memory growth. I've done some further tests with the hypothesis of unbounded memory growth to confirm a leak and none of them pan out, which means you're right. This probably means #6292 can probably be closed as it apparently rests on the same premise. Thanks again! Edit: I accidentally a word |
|
I think you mean #6929. Yes, I think we can close. |
|
@mtoledo no problem. Thanks for reporting this anyway! It's good to have a record because I'm sure someone will run in to this in the future. |
|
We just recently ran into this problem, while iterating over 150,000 records, with each record having a rather large object graph. For right now, we'll try no to_proc way and see if that works for us. We can confirm, that we have the same issues in #6929. If we do this: ObjectSpace.each_object(ActiveRecord::Relation).each(&:reset)
GC.startThe |
You can see the script being used to reproduce the issue and its output in multiple ruby interpreters on this gist
The ruby garbage collector, specially in MRI, isn't reliable in regards to tracking leaks by objects living in ObjectSpace alone. Therefore, I've done my best in order to work around GC quirks to demonstrate the actual leak.
I've setup a simple rails app at https://github.com/mtoledo/memory_leak which is being used for the demonstration. Its a vanilla rails 3.2.6 app with 2 models,
MyModelandMyAssociation, connected by a relationship through amy_model_idforeign key.You can find the script used to reproduce the error at
lib/leak.rb. It specifically creates the model classes in case rails is not loaded so we can compare the behavior of ActiveRecord objects and regular objects.In short, this is basically what I've found:
There are other scenarios in which the calls cause objects to not be reclaimed by the GC on the script.
Notice the runtimes behave in strange ways, and that the amount of leaked objects is actually different in MRI and Rubinius. This could point to a different issue that's not necessarily on ActiveRecord, but the bad behavior still stands.
The text was updated successfully, but these errors were encountered: