Skip to content

Fix for Bug#4571 #14

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 3,029 commits into from
Closed

Fix for Bug#4571 #14

wants to merge 3,029 commits into from

Conversation

tinco
Copy link

@tinco tinco commented May 5, 2011

This patch fixes http://redmine.ruby-lang.org/issues/4571 by correctly parsing the microseconds into a double and passing that to the Time constructor.

nobu and others added 30 commits July 21, 2010 18:23
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@28707 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@28708 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@28709 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@28710 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
	  variable in toplevel and eval are no longer warned.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@28711 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@28712 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@28713 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@28714 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
  is ASCII-8BIT and non-ASCII character.
  The length of character should be from original byte string.
  [ruby-core:31431]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@28715 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
  must be called even when pthread_getattr_np is used.
  [ruby-core:31269]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@28716 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@28717 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
  converted.

* complex.c (nucomp_to_f, nucomp_to_r): ditto.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@28725 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@28726 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
  [ruby-dev:41872]

* include/ruby/missing.h: needs ruby/config.h.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@28727 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
  multibyte characters.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@28728 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@28729 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@28730 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
…iable.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@28731 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@28732 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* test/ruby/test_float.rb (TestFloat#test_Float): suppress
  warnings under --verbose.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@28750 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
  other.  [ruby-core:31470]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@28751 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
  nodes.  [ruby-dev:41874]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@28752 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@28753 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
  This fixes build failure on MSVC. [ruby-core:31481]

* include/ruby/ruby.h, include/ruby/missing.h:
  use BROKEN_CLOSE for replacing close(2).

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@28754 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
  iseq.  [ruby-dev:41880]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@28755 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
  This needs to merge to 1.9.2.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@28756 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
  to include only once, before including defines.h.
  including it here breaks some macro definitions.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@28757 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@28758 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* win32/Makefile.sub, bcc32/Makefile.sub (config.h): ditto.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@28759 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
  pathname.rb.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@28760 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
matzbot pushed a commit that referenced this pull request May 1, 2020
matzbot pushed a commit that referenced this pull request May 1, 2020
Patch adapted from Alessandro Minali
matzbot pushed a commit that referenced this pull request Jun 9, 2020
This reverts commit 02b216e.
This reverts commit 9b8825b.

I found that combining sweep and move is not safe.  I don't think that
we can do compaction concurrently with _anything_ unless there is a read
barrier installed.

Here is a simple example.  A class object is freed, and during it's free
step, it tries to remove itself from its parent's subclass list.
However, during the sweep step, the parent class was moved and the
"currently being freed" class didn't have references updated yet.  So we
get a segv like this:

```
(lldb) bt
* thread #1, name = 'ruby', stop reason = signal SIGSEGV
  * frame #0: 0x0000560763e344cb ruby`rb_st_lookup at st.c:320:43
    frame #1: 0x0000560763e344cb ruby`rb_st_lookup(tab=0x2f7469672f6e6f72, key=3809, value=0x0000560765bf2270) at st.c:1010
    frame #2: 0x0000560763e8f16a ruby`rb_search_class_path at variable.c:99:9
    frame #3: 0x0000560763e8f141 ruby`rb_search_class_path at variable.c:145
    frame #4: 0x0000560763e8f141 ruby`rb_search_class_path(klass=94589785585880) at variable.c:191
    frame #5: 0x0000560763ec744e ruby`rb_vm_bugreport at vm_dump.c:996:17
    frame #6: 0x0000560763f5b958 ruby`rb_bug_for_fatal_signal at error.c:675:5
    frame #7: 0x0000560763e27dad ruby`sigsegv(sig=<unavailable>, info=<unavailable>, ctx=<unavailable>) at signal.c:955:5
    frame #8: 0x00007f8b891d33c0 libpthread.so.0`___lldb_unnamed_symbol1$$libpthread.so.0 + 1
    frame #9: 0x0000560763efa8bb ruby`rb_class_remove_from_super_subclasses(klass=94589790314280) at class.c:93:56
    frame #10: 0x0000560763d10cb7 ruby`gc_sweep_step at gc.c:2674:2
    frame #11: 0x0000560763d1187b ruby`gc_sweep at gc.c:4540:2
    frame #12: 0x0000560763d101f0 ruby`gc_start at gc.c:6797:6
    frame #13: 0x0000560763d15153 ruby`rb_gc_compact at gc.c:7479:12
    frame #14: 0x0000560763eb4eb8 ruby`vm_exec_core at vm_insnhelper.c:5183:13
    frame #15: 0x0000560763ea9bae ruby`rb_vm_exec at vm.c:1953:22
    frame #16: 0x0000560763eac08d ruby`rb_yield at vm.c:1132:9
    frame #17: 0x0000560763edb4f2 ruby`rb_ary_collect at array.c:3186:9
    frame #18: 0x0000560763e9ee15 ruby`vm_call_cfunc_with_frame at vm_insnhelper.c:2575:12
    frame #19: 0x0000560763eb2e66 ruby`vm_exec_core at vm_insnhelper.c:4177:11
    frame #20: 0x0000560763ea9bae ruby`rb_vm_exec at vm.c:1953:22
    frame #21: 0x0000560763eac08d ruby`rb_yield at vm.c:1132:9
    frame #22: 0x0000560763edb4f2 ruby`rb_ary_collect at array.c:3186:9
    frame #23: 0x0000560763e9ee15 ruby`vm_call_cfunc_with_frame at vm_insnhelper.c:2575:12
    frame #24: 0x0000560763eb2e66 ruby`vm_exec_core at vm_insnhelper.c:4177:11
    frame #25: 0x0000560763ea9bae ruby`rb_vm_exec at vm.c:1953:22
    frame #26: 0x0000560763ceee01 ruby`rb_ec_exec_node(ec=0x0000560765afa530, n=0x0000560765b088e0) at eval.c:296:2
    frame #27: 0x0000560763cf3b7b ruby`ruby_run_node(n=0x0000560765b088e0) at eval.c:354:12
    frame #28: 0x0000560763cee4a3 ruby`main(argc=<unavailable>, argv=<unavailable>) at main.c:50:9
    frame #29: 0x00007f8b88e560b3 libc.so.6`__libc_start_main + 243
    frame #30: 0x0000560763cee4ee ruby`_start + 46
(lldb) f 9
frame #9: 0x0000560763efa8bb ruby`rb_class_remove_from_super_subclasses(klass=94589790314280) at class.c:93:56
   90
   91  		*RCLASS_EXT(klass)->parent_subclasses = entry->next;
   92  		if (entry->next) {
-> 93  		    RCLASS_EXT(entry->next->klass)->parent_subclasses = RCLASS_EXT(klass)->parent_subclasses;
   94  		}
   95  		xfree(entry);
   96  	    }
(lldb) command script import -r misc/lldb_cruby.py
lldb scripts for ruby has been installed.
(lldb) rp entry->next->klass
(struct RMoved) $1 = (flags = 30, destination = 94589792806680, next = 94589784369160)
(lldb)
```
tenderlove added a commit that referenced this pull request Apr 20, 2021
This commit adds a check on the ep just like in the mark function.  The
env can contain null bytes if allocation tracing is enabled.

We're seeing errors during autocompaction like this:

```
(lldb) bt 40
* thread #1, name = 'ruby', stop reason = signal SIGABRT
    frame #0: 0x00007f7d64b6018b libc.so.6`raise + 203
    frame #1: 0x00007f7d64b3f859 libc.so.6`abort + 299
    frame #2: 0x000055af5f2fefc9 ruby`die at error.c:764:5
    frame #3: 0x000055af5f2ff1ac ruby`rb_bug_for_fatal_signal(default_sighandler=0x0000000000000000, sig=11, ctx=0x000055af60bc3340, fmt="") at error.c:804:5
    frame #4: 0x000055af5f4bd08f ruby`sigsegv(sig=11, info=0x000055af60bc3470, ctx=0x000055af60bc3340) at signal.c:960:5
    frame #5: 0x00007f7d64ebe3c0 libpthread.so.0`__restore_rt
    frame #6: 0x000055af5f339b0a ruby`gc_ref_update_imemo(objspace=0x000055af60b2b040, obj=0x00007f7d5b513fd0) at gc.c:9046:13
    frame #7: 0x000055af5f339172 ruby`gc_update_object_references(objspace=0x000055af60b2b040, obj=0x00007f7d5b513fd0) at gc.c:9307:9
    frame #8: 0x000055af5f338e79 ruby`gc_ref_update(vstart=0x00007f7d5b510010, vend=0x00007f7d5b513ff8, stride=40, objspace=0x000055af60b2b040, page=0x000055af62577aa0) at gc.c:9452:21
    frame #9: 0x000055af5f337846 ruby`gc_update_references(objspace=0x000055af60b2b040, heap=0x000055af60b2b068) at gc.c:9481:9
    frame #10: 0x000055af5f336569 ruby`gc_compact_finish(objspace=0x000055af60b2b040, heap=0x000055af60b2b068) at gc.c:4840:5
    frame #11: 0x000055af5f335efb ruby`gc_page_sweep(objspace=0x000055af60b2b040, heap=0x000055af60b2b068, sweep_page=0x000055af63a1eb30) at gc.c:5046:13
    frame #12: 0x000055af5f3355c5 ruby`gc_sweep_step(objspace=0x000055af60b2b040, heap=0x000055af60b2b068) at gc.c:5214:19
    frame #13: 0x000055af5f33daf6 ruby`gc_sweep_rest(objspace=0x000055af60b2b040) at gc.c:5271:2
    frame #14: 0x000055af5f33cacd ruby`gc_sweep(objspace=0x000055af60b2b040) at gc.c:5389:2
    frame #15: 0x000055af5f33c21d ruby`gc_marks_rest(objspace=0x000055af60b2b040) at gc.c:7555:5
    frame #16: 0x000055af5f324d41 ruby`gc_rest(objspace=0x000055af60b2b040) at gc.c:8457:13
    frame #17: 0x000055af5f3297d8 ruby`garbage_collect(objspace=0x000055af60b2b040, reason=45568) at gc.c:8318:9
    frame #18: 0x000055af5f344ece ruby`garbage_collect_with_gvl(objspace=0x000055af60b2b040, reason=45568) at gc.c:8632:9
    frame #19: 0x000055af5f344e61 ruby`objspace_malloc_gc_stress(objspace=0x000055af60b2b040) at gc.c:10592:9
    frame #20: 0x000055af5f32ced1 ruby`objspace_xmalloc0(objspace=0x000055af60b2b040, size=64) at gc.c:10767:5
    frame #21: 0x000055af5f32ce11 ruby`ruby_xmalloc0(size=64) at gc.c:10988:12
    frame #22: 0x000055af5f32cdac ruby`ruby_xmalloc_body(size=64) at gc.c:10997:12
    frame #23: 0x000055af5f329415 ruby`ruby_xmalloc(size=64) at gc.c:12942:12
    frame #24: 0x00007f7d611c4fe5 objspace.so`newobj_i(tpval=0x00007f7d5b553770, data=0x000055af639031a0) at object_tracing.c:101:35
    frame #25: 0x000055af5f5b283f ruby`tp_call_trace(tpval=0x00007f7d5b553770, trace_arg=0x00007fff1016d398) at vm_trace.c:1115:2
    frame #26: 0x000055af5f5b50ec ruby`exec_hooks_body(ec=0x000055af60b2b700, list=0x000055af60b2b920, trace_arg=0x00007fff1016d398) at vm_trace.c:304:3
    frame #27: 0x000055af5f5b0f24 ruby`exec_hooks_unprotected(ec=0x000055af60b2b700, list=0x000055af60b2b920, trace_arg=0x00007fff1016d398) at vm_trace.c:333:5
    frame #28: 0x000055af5f5b0da8 ruby`rb_exec_event_hooks(trace_arg=0x00007fff1016d398, hooks=0x000055af60b2b920, pop_p=0) at vm_trace.c:378:13
    frame #29: 0x000055af5f33f8e2 ruby`rb_exec_event_hook_orig(ec=0x000055af60b2b700, hooks=0x000055af60b2b920, flag=1048576, self=0x00007f7d5b5c08c0, id=0, called_id=0, klass=0x0000000000000000, data=0x00007f7d5b513fd0, pop_p=0) at vm_core.h:1989:5
    frame #30: 0x000055af5f334975 ruby`gc_event_hook_body(ec=0x000055af60b2b700, objspace=0x000055af60b2b040, event=1048576, data=0x00007f7d5b513fd0) at gc.c:2083:5
  * frame #31: 0x000055af5f3342df ruby`newobj_slowpath_wb_protected [inlined] newobj_slowpath(klass=0x00007f7d5b9d19c8, flags=0x000000000000001a, objspace=0x000055af60b2b040, cr=0x000055af60b2b910, wb_protected=1) at gc.c:2284:9
    frame #32: 0x000055af5f33410f ruby`newobj_slowpath_wb_protected(klass=0x00007f7d5b9d19c8, flags=0x000000000000001a, objspace=0x000055af60b2b040, cr=0x000055af60b2b910) at gc.c:2299
    frame #33: 0x000055af5f333de9 ruby`newobj_of0(klass=0x00007f7d5b9d19c8, flags=0x000000000000001a, wb_protected=1, cr=0x000055af60b2b910) at gc.c:2338:11
    frame #34: 0x000055af5f3227ae ruby`newobj_of(klass=0x00007f7d5b9d19c8, flags=0x000000000000001a, v1=0x000055af657d88a0, v2=0x000055af657d8890, v3=0x0000000000000000, wb_protected=1) at gc.c:2348:17
    frame #35: 0x000055af5f322c5b ruby`rb_imemo_new(type=imemo_env, v1=0x000055af657d88a0, v2=0x000055af657d8890, v3=0x0000000000000000, v0=0x00007f7d5b9d19c8) at gc.c:2434:12
    frame #36: 0x000055af5f5a3925 ruby`vm_env_new(env_ep=0x000055af657d88a0, env_body=0x000055af657d8890, env_size=4, iseq=0x00007f7d5b9d19c8) at vm_core.h:1363:33
    frame #37: 0x000055af5f5a3808 ruby`vm_make_env_each(ec=0x000055af60b2b700, cfp=0x00007f7d6482fc90) at vm.c:801:11
    frame #38: 0x000055af5f5a368d ruby`vm_make_env_each(ec=0x000055af60b2b700, cfp=0x00007f7d6482fc20) at vm.c:752:13
    frame #39: 0x000055af5f5a368d ruby`vm_make_env_each(ec=0x000055af60b2b700, cfp=0x00007f7d6482fbb0) at vm.c:752:13
(lldb) f 31
frame #31: 0x000055af5f3342df ruby`newobj_slowpath_wb_protected [inlined] newobj_slowpath(klass=0x00007f7d5b9d19c8, flags=0x000000000000001a, objspace=0x000055af60b2b040, cr=0x000055af60b2b910, wb_protected=1) at gc.c:2284:9
   2281	        }
   2282	        GC_ASSERT(obj != 0);
   2283	        newobj_init(klass, flags, wb_protected, objspace, obj);
-> 2284	        gc_event_hook_prep(objspace, RUBY_INTERNAL_EVENT_NEWOBJ, obj, newobj_fill(obj, 0, 0, 0));
   2285	    }
   2286	    RB_VM_LOCK_LEAVE_CR_LEV(cr, &lev);
   2287
(lldb) p obj
(VALUE) $3 = 0x00007f7d5b513fd0
(lldb) f 6
frame #6: 0x000055af5f339b0a ruby`gc_ref_update_imemo(objspace=0x000055af60b2b040, obj=0x00007f7d5b513fd0) at gc.c:9046:13
   9043	        {
   9044	            rb_env_t *env = (rb_env_t *)obj;
   9045	            TYPED_UPDATE_IF_MOVED(objspace, rb_iseq_t *, env->iseq);
-> 9046	            UPDATE_IF_MOVED(objspace, env->ep[VM_ENV_DATA_INDEX_ENV]);
   9047	            gc_update_values(objspace, (long)env->env_size, (VALUE *)env->env);
   9048	        }
   9049	        break;
(lldb) p obj
(VALUE) $4 = 0x00007f7d5b513fd0
(lldb)
```
eileencodes pushed a commit to eileencodes/ruby that referenced this pull request Jul 27, 2021
)

* Fix block invalidation assertions

* Add Alan's small repro for double invalidation bug
eileencodes pushed a commit to eileencodes/ruby that referenced this pull request Jul 27, 2021
* Evaluate multiple assignment left hand side before right hand side

In regular assignment, Ruby evaluates the left hand side before
the right hand side.  For example:

```ruby
foo[0] = bar
```

Calls `foo`, then `bar`, then `[]=` on the result of `foo`.

Previously, multiple assignment didn't work this way.  If you did:

```ruby
abc.def, foo[0] = bar, baz
```

Ruby would previously call `bar`, then `baz`, then `abc`, then
`def=` on the result of `abc`, then `foo`, then `[]=` on the
result of `foo`.

This change makes multiple assignment similar to single assignment,
changing the evaluation order of the above multiple assignment code
to calling `abc`, then `foo`, then `bar`, then `baz`, then `def=` on
the result of `abc`, then `[]=` on the result of `foo`.

Implementing this is challenging with the stack-based virtual machine.
We need to keep track of all of the left hand side attribute setter
receivers and setter arguments, and then keep track of the stack level
while handling the assignment processing, so we can issue the
appropriate topn instructions to get the receiver.  Here's an example
of how the multiple assignment is executed, showing the stack and
instructions:

```
self                                      # putself
abc                                       # send
abc, self                                 # putself
abc, foo                                  # send
abc, foo, 0                               # putobject 0
abc, foo, 0, [bar, baz]                   # evaluate RHS
abc, foo, 0, [bar, baz], baz, bar         # expandarray
abc, foo, 0, [bar, baz], baz, bar, abc    # topn 5
abc, foo, 0, [bar, baz], baz, abc, bar    # swap
abc, foo, 0, [bar, baz], baz, def=        # send
abc, foo, 0, [bar, baz], baz              # pop
abc, foo, 0, [bar, baz], baz, foo         # topn 3
abc, foo, 0, [bar, baz], baz, foo, 0      # topn 3
abc, foo, 0, [bar, baz], baz, foo, 0, baz # topn 2
abc, foo, 0, [bar, baz], baz, []=         # send
abc, foo, 0, [bar, baz], baz              # pop
abc, foo, 0, [bar, baz]                   # pop
[bar, baz], foo, 0, [bar, baz]            # setn 3
[bar, baz], foo, 0                        # pop
[bar, baz], foo                           # pop
[bar, baz]                                # pop
```

As multiple assignment must deal with splats, post args, and any level
of nesting, it gets quite a bit more complex than this in non-trivial
cases. To handle this, struct masgn_state is added to keep
track of the overall state of the mass assignment, which stores a linked
list of struct masgn_attrasgn, one for each assigned attribute.

This adds a new optimization that replaces a topn 1/pop instruction
combination with a single swap instruction for multiple assignment
to non-aref attributes.

This new approach isn't compatible with one of the optimizations
previously used, in the case where the multiple assignment return value
was not needed, there was no lhs splat, and one of the left hand side
used an attribute setter.  This removes that optimization. Removing
the optimization allowed for removing the POP_ELEMENT and adjust_stack
functions.

This adds a benchmark to measure how much slower multiple
assignment is with the correct evaluation order.

This benchmark shows:

* 4-9% decrease for attribute sets
* 14-23% decrease for array member sets
* Basically same speed for local variable sets

Importantly, it shows no significant difference between the popped
(where return value of the multiple assignment is not needed) and
!popped (where return value of the multiple assignment is needed)
cases for attribute and array member sets.  This indicates the
previous optimization, which was dropped in the evaluation
order fix and only affected the popped case, is not important to
performance.

Fixes [Bug ruby#4443]

* * 2021-04-22 [ci skip]

* Remove reverse VM instruction

This was previously only used by the multiple assignment code, but
is no longer needed after the multiple assignment execution order
fix.

* fix raise in exception with jump

add_ensure_iseq() adds ensure block to the end of
jump such as next/redo/return. However, if the rescue
cause are in the body, this rescue catches the exception
in ensure clause.

  iter do
    next
  rescue
    R
  ensure
    raise
  end

In this case, R should not be executed, but executed without this patch.

Fixes [Bug #13930]
Fixes [Bug #16618]

A part of tests are written by @jeremyevans ruby#4291

* [ruby/time] Make Time friendly to Ractor

ruby/time@c784e4f166

* [ruby/cgi] handle invalid encoding

ruby/cgi@2b1c2e21a4

* [ruby/cgi] Add test for escapeHTML/unescapeHTML invalid encoding fix in pure ruby version

Also, remove pointless assert_nothing_raised(ArgumentError) while
here.

ruby/cgi@c05edf5608

* [ruby/cgi] gemspec: Explicitly empty executables list

The gem exposes no executables

ruby/cgi@cd7106ad97

* [ruby/benchmark] Add comment about terminating newline in captions; fix test method name.

ruby/benchmark@02ce298d3e

* [ruby/benchmark] gemspec: Explicitly have 0 executables

This gem exposes no executables.

ruby/benchmark@ff1ef7ae06

* Ignore JRuby files on io-console

* [ruby/io-console] Enable building the C extension on TruffleRuby.

ruby/io-console@c17b8cf3a9

* [ruby/io-console] Move FFI console under lib

Having the separate dir makes testing difficult and doesn't
reflect the structure the gem will eventually have. We can filter
these files out if necessary when building the CRuby gem.

ruby/io-console@881010447c

* Separate test used by test_ractor for Ractor in test_time.rb

* Merge net-imap-0.2.0

* [ruby/net-imap] Set timeout for IDLE responses

Fixes ruby#14

ruby/net-imap@39d39ff9bb

* [ruby/net-imap] Bump version to 0.2.1

ruby/net-imap@31f96ea884

* [ruby/uri] Upstream Java proxy property checks from JRuby

These Java properties, retrieved from JRuby's "Java env" ENV_JAVA,
allow JRuby users to use the same proxy properties the rest of the
Java platform uses.

This resolves https://bugs.ruby-lang.org/issues/11194

ruby/uri@3bd2bcc95a

* [ruby/uri] Optimize URI#hostname and URI#hostname=

ruby/uri@3b7ccfd835

* [ruby/uri] Add tests for URI::RFC{2396,3986}_Parser#inspect

ruby/uri@d47dae2f8e

* [ruby/uri] Only use UnboundMethod#bind_call if it is available

This allows tests to pass on Ruby 2.4-2.6.

Fixes ruby#19

ruby/uri@67ca99ca87

* [ruby/uri] Set required_ruby_version to 2.4 in gemspec

Tests pass on Ruby 2.4, but not on Ruby 2.3.

ruby/uri@594418079a

* [ruby/uri] remove comment about URI::escape as it is removed

ruby/uri@0f0057e1b2

* [ruby/uri] Use Regexp#match? to avoid extra allocations

`#=~` builds `MatchData`, requiring extra allocations as compared to
`#match?`, which returns a boolean w/o having to build the `MatchData`.

ruby/uri@158f58a9cc

* Update bundled_gems

* Suppress warnings for unsued variable

* * 2021-04-23 [ci skip]

* Remove unneeded comment

* test/ruby/test_assignment.rb: Avoid "assigned but unused variable"

* Fix wrong documentation

It doesn't return `nil` but raises an exception, as explained a few lines after

* * 2021-04-24 [ci skip]

* Fix setting method visibility for a refinement without an origin class

If a class has been refined but does not have an origin class,
there is a single method entry marked with VM_METHOD_TYPE_REFINED,
but it contains the original method entry.  If the original method
entry is present, we shouldn't skip the method when searching even
when skipping refined methods.

Fixes [Bug #17519]

* Remove unnecessary checks for empty kw splat

These two checks are surrounded by an if that ensures the
call site is not a kw splat call site.

* Remove part of comment that is no longer accurate

In Ruby 2.7, empty keyword splats could be added back for backwards
compatibility.  However, that stopped in Ruby 3.0.

* Add back checks for empty kw splat with tests (ruby#4405)

This reverts commit a224ce8.
Turns out the checks are needed to handle splatting an array with an
empty ruby2 keywords hash.

* [Doc] Fix a typo s/invokations/invocations/

* * 2021-04-25 [ci skip]

* [Doc] Fix a typo s/evel/eval/

* [Doc] Fix a typo s/oher/other/

* [Doc] Fix a typo s/visilibity/visibility/

* [Doc] Fix a typo s/arround/around/

* [Doc] Fix a typo s/daguten/dakuten/

* [ci skip] Fix a typo s/certificiate/certificate/

* [Doc] Fix a typo s/algorthm/algorithm/

* Fix some typos by spell checker

* * 2021-04-26 [ci skip]

* Remove test of removed reverse VM instruction

since 5512353

* spec/ruby/core/file/shared/read.rb: The behavior of FreeBSD was changed

http://rubyci.s3.amazonaws.com/freebsd12/ruby-master/log/20210426T003001Z.fail.html.gz#rubyspec

* disable shareable_constant_value for CI

To debug CI failures on FreeBSD, disable `shareable_constant_value`.

* [ruby/irb] Fix typo ture -> true [ci skip]

ruby/irb@783a0569e8

* [ruby/irb] Added assert_equal_with_term

ruby/irb@b690da96d8

* [ruby/irb] Added test_colorize

ruby/irb@10e290fc3a

* [ruby/irb] Assertions on non-tty

ruby/irb@ede12890d2

* [ruby/irb] Added `colorable` keyword option

Currently `IRB::Color.colorize` and `IRB::Color.colorize_code`
refer `$stdin.tty?` internally.
This patch adds `colorable` keyword option which overrides it.

ruby/irb@402e3f1907

* [ruby/irb] Added setup and teardown to TestIRB::TestInit

Not to be affected by existing rc files in all tests.

ruby/irb@bf434892b4

* node.c (rb_ast_new): imemo_ast is WB-unprotected

Previously imemo_ast was handled as WB-protected which caused a segfault
of the following code:

    # shareable_constant_value: literal
    M0 = {}
    M1 = {}
    ...
    M100000 = {}

My analysis is here: `shareable_constant_value: literal` creates many
Hash instances during parsing, and add them to node_buffer of imemo_ast.
However, the contents are missed because imemo_ast is incorrectly
WB-protected.

This changeset makes imemo_ast as WB-unprotected.

* Revert "disable shareable_constant_value for CI"

This reverts commit c647205.

Maybe the root issue was fixed by 7ac078e

* Document binding behavior for C call/return events for TracePoint/set_trace_func

C methods do not have bindings, so binding returns the binding of
the nearest C method.

Fixes [Bug ruby#9009]

* * 2021-04-27 [ci skip]

* Fix compiler warnings in objspace_dump.c when assertions are turned on

Example:

```
In file included from ../../../include/ruby/defines.h:72,
                 from ../../../include/ruby/ruby.h:23,
                 from ../../../gc.h:3,
                 from ../../../ext/objspace/objspace_dump.c:15:
../../../ext/objspace/objspace_dump.c: In function ‘dump_append_ld’:
../../../ext/objspace/objspace_dump.c:95:26: warning: comparison of integer expressions of different signedness: ‘long unsigned int’ and ‘int’ [-Wsign-compare]
   95 |     RUBY_ASSERT(required <= width);
      |                          ^~
```

* Fix type-o in insns.def

"redefine" -> "redefined"

* Partially revert 2c7d3b3

to make imemo_ast WB-protected again. Only the test is kept.

* Make imemo_ast WB-protected again

by firing the write barrier of imemo_ast after nd_lit is modified.
This will fix the issue of ruby#4416 more
gracefully.

* test/ruby/test_exception.rb: suppress "warning: statement not reached"

* [ruby/pathname] gemspec: Explicitly list 0 executables

This gem exposes no executables.

ruby/pathname@c401d97d58

* [ruby/gdbm] Add dependency to gdbm package on mingw

RubyInstaller2 supports metadata tags for installation of dependent
MSYS2/MINGW libraries. The openssl gem requires the mingw-openssl
package to be installed on the system, which the gem installer takes
care about, when this tag is set.

The feature is documented here:
https://github.com/oneclick/rubyinstaller2/wiki/For-gem-developers#msys2-library-dependency

Fixes oneclick/rubyinstaller2#163

ruby/gdbm@d95eed3e86

* [ruby/matrix] Use Gemfile instead of Gem::Specification#add_development_dependency.

ruby/matrix@1381fde5c1

* [ruby/matrix] v0.4.0

ruby/matrix@baea4b90d4

* [ruby/matrix] v0.4.1

ruby/matrix@f7c9981907

* [ruby/matrix] Guard for < Ruby 3.0

ruby/matrix@1ef660c627

* [ruby/net-ftp] Re-apply 827e471d438fdec1ae329afb5912b8e06d534823

ruby/net-ftp@3ca80368c4

* [ruby/net-ftp] Replace Timeout.timeout with socket timeout

Timeout.timeout is inefficient since it spins up a new thread for
each invocation, use Socket.tcp's connect_timeout option instead
when we aren't using SOCKS (we can't replace Timeout.timeout
for SOCKS yet since SOCKSSocket doesn't have a connect_timeout
option).

ruby/net-ftp@d65910132f

* [ruby/net-ftp] Close the passive connection data socket if there is an error setting up the transfer

Previously, the connection leaked in this case.  This uses
begin/ensure and checking for an error in the ensure block.

An alternative approach would be to not even perform the
connection until after the RETR (or other) command has been
sent.  However, I'm not sure all FTP servers support that.
The current behavior is:

* Send (PASV/EPSV)
* Connect to the host/port returned in 227/229 reply
* Send (RETR/other command)

Changing it to connect after the RETR could break things.
FTP servers might expect that the client has already
connected before sending the RETR.  The alternative
approach is more likely to introduce backwards compatibility
issues, compared to the begin/ensure approach taken here.

Fixes Ruby Bug 17027

ruby/net-ftp@6e8535f076

* [ruby/net-ftp] Reduce resource cosumption of Net::FTP::TIME_PARSER

Reported by Alexandr Savca as a DoS vulnerability, but Net::FTP is a
client library and the impact of the issue is low, so I have decided
to fix it as a normal issue.
Based on patch by nobu.

ruby/net-ftp@a93af636f8

* [ruby/net-ftp] Add test cases

ruby/net-ftp@865232bb2a

* [ruby/net-ftp] Replace "iff" with "if and only if"

iff means if and only if, but readers without that knowledge might
assume this to be a spelling mistake. To me, this seems like
exclusionary language that is unnecessary. Simply using "if and only if"
instead should suffice.

ruby/net-ftp@e920473618

* lldb: Add Freelist Index to dump_page output

* lldb: dump_page_rvalue - dump a heap page containing an RVALUE

rather than having to do this in a two step process:

1. heap_page obj
2. dump_page $2 (or whatever lldb variable heap_page set)

we can now just

dump_page_rvalue obj

* lldb: highlight the slot when using dump_page_rvalue

* Fix Monitor to lock per Fiber, like Mutex [Bug #17827]

* * 2021-04-28 [ci skip]

* test/ruby/test_fiber.rb: reduce the count of object creation to cause GC

... on Solaris. This is the same as 5478871.

http://rubyci.s3.amazonaws.com/solaris10-gcc/ruby-master/log/20210427T160003Z.fail.html.gz
```
[ 7667/20965] TestFiber#test_fork_from_fiber/export/home/users/chkbuild/cb-gcc/tmp/build/20210427T160003Z/ruby/test/ruby/test_fiber.rb:397:in `transfer': can't alloc machine stack to fiber (1 x 139264 bytes): Not enough space (FiberError)
	from /export/home/users/chkbuild/cb-gcc/tmp/build/20210427T160003Z/ruby/test/ruby/test_fiber.rb:397:in `block (6 levels) in test_fork_from_fiber'
	from /export/home/users/chkbuild/cb-gcc/tmp/build/20210427T160003Z/ruby/test/ruby/test_fiber.rb:396:in `times'
	from /export/home/users/chkbuild/cb-gcc/tmp/build/20210427T160003Z/ruby/test/ruby/test_fiber.rb:396:in `block (5 levels) in test_fork_from_fiber'
	from /export/home/users/chkbuild/cb-gcc/tmp/build/20210427T160003Z/ruby/test/ruby/test_fiber.rb:392:in `fork'
	from /export/home/users/chkbuild/cb-gcc/tmp/build/20210427T160003Z/ruby/test/ruby/test_fiber.rb:392:in `block (4 levels) in test_fork_from_fiber'
 = 0.88 s

...

  1) Failure:
TestFiber#test_fork_from_fiber [/export/home/users/chkbuild/cb-gcc/tmp/build/20210427T160003Z/ruby/test/ruby/test_fiber.rb:409]:
[ruby-core:41456].
<0> expected but was
<1>.
```

* test/net/ftp/test_ftp.rb: remove unused variable

* test/net/ftp/test_ftp.rb: reduce the size of a long response

"9" * 999999999 (about 1 GB) was too large for some CI servers.
This commit changes the size to 999999 (about 1 MB).

http://rubyci.s3.amazonaws.com/scw-9d6766/ruby-master/log/20210427T141707Z.fail.html.gz
http://rubyci.s3.amazonaws.com/raspbian10-aarch64/ruby-master/log/20210427T145408Z.fail.html.gz

* test/net/ftp/test_ftp.rb: Use RubyVM::JIT instead of RubyVM::MJIT

* [ruby/net-smtp] Net::SMTP.start() and #start() accepts ssl_context_params keyword argument

	Additional params are passed to OpenSSL::SSL::SSLContext#set_params.

	For example, `Net::SMTP#start(ssl_context_params: { cert_store: my_store, timeout: 123 })`
	calls `set_params({ cert_store: my_store, timeout: 123 })`.

ruby/net-smtp@4213389c21

* [ruby/net-smtp] Replace Timeout.timeout with socket timeout

Timeout.timeout is inefficient since it spins up a new thread for
each invocation, use Socket.tcp's connect_timeout option instead

ruby/net-smtp@6ae4a59f05

* [ruby/net-smtp] Removed needless files from Gem::Specification#files

ruby/net-smtp@69bba6b125

* [ruby/net-smtp] mod: bump to a new VERSION that could be checked for testings >0.2.1

ruby/net-smtp@8f2c9323e2

Co-authored-by: Jeremy Evans <code@jeremyevans.net>
Co-authored-by: git <svn-admin@ruby-lang.org>
Co-authored-by: Koichi Sasada <ko1@atdot.net>
Co-authored-by: Kir Shatrov <shatrov@me.com>
Co-authored-by: pavel <pavel.rosicky@easy.cz>
Co-authored-by: Olle Jonsson <olle.jonsson@gmail.com>
Co-authored-by: Keith Bennett <keithrbennett@gmail.com>
Co-authored-by: Hiroshi SHIBATA <hsbt@ruby-lang.org>
Co-authored-by: Duncan MacGregor <duncan.macgregor@oracle.com>
Co-authored-by: Charles Oliver Nutter <headius@headius.com>
Co-authored-by: Shugo Maeda <shugo@ruby-lang.org>
Co-authored-by: Lukas Zapletal <lzap+git@redhat.com>
Co-authored-by: Felix Wong <felix@waf.hk>
Co-authored-by: Steven Harman <steven@harmanly.com>
Co-authored-by: Kazuhiro NISHIYAMA <zn@mbf.nifty.com>
Co-authored-by: S-H-GAMELINKS <gamelinks007@gmail.com>
Co-authored-by: Yusuke Endoh <mame@ruby-lang.org>
Co-authored-by: romainsalles <romainsalles@users.noreply.github.com>
Co-authored-by: Alan Wu <XrXr@users.noreply.github.com>
Co-authored-by: wonda-tea-coffee <rikita.ishikawa@crowdworks.co.jp>
Co-authored-by: wonda-tea-coffee <lagrange.resolvent@gmail.com>
Co-authored-by: Ryuta Kamizono <kamipo@gmail.com>
Co-authored-by: Nobuyoshi Nakada <nobu@ruby-lang.org>
Co-authored-by: Peter Zhu <peter@peterzhu.ca>
Co-authored-by: ebrohman <ericbrohman@gmail.com>
Co-authored-by: Lars Kanis <kanis@comcard.de>
Co-authored-by: Marc-Andre Lafortune <github@marc-andre.ca>
Co-authored-by: mohamed <mohamed.m.m.hafez@gmail.com>
Co-authored-by: Gannon McGibbon <gannon.mcgibbon@gmail.com>
Co-authored-by: Matt Valentine-House <matt@eightbitraptor.com>
Co-authored-by: Benoit Daloze <eregontp@gmail.com>
Co-authored-by: Tom Freudenberg <tom.freudenberg@4commerce.de>
eileencodes pushed a commit to eileencodes/ruby that referenced this pull request Jul 27, 2021
* Evaluate multiple assignment left hand side before right hand side

In regular assignment, Ruby evaluates the left hand side before
the right hand side.  For example:

```ruby
foo[0] = bar
```

Calls `foo`, then `bar`, then `[]=` on the result of `foo`.

Previously, multiple assignment didn't work this way.  If you did:

```ruby
abc.def, foo[0] = bar, baz
```

Ruby would previously call `bar`, then `baz`, then `abc`, then
`def=` on the result of `abc`, then `foo`, then `[]=` on the
result of `foo`.

This change makes multiple assignment similar to single assignment,
changing the evaluation order of the above multiple assignment code
to calling `abc`, then `foo`, then `bar`, then `baz`, then `def=` on
the result of `abc`, then `[]=` on the result of `foo`.

Implementing this is challenging with the stack-based virtual machine.
We need to keep track of all of the left hand side attribute setter
receivers and setter arguments, and then keep track of the stack level
while handling the assignment processing, so we can issue the
appropriate topn instructions to get the receiver.  Here's an example
of how the multiple assignment is executed, showing the stack and
instructions:

```
self                                      # putself
abc                                       # send
abc, self                                 # putself
abc, foo                                  # send
abc, foo, 0                               # putobject 0
abc, foo, 0, [bar, baz]                   # evaluate RHS
abc, foo, 0, [bar, baz], baz, bar         # expandarray
abc, foo, 0, [bar, baz], baz, bar, abc    # topn 5
abc, foo, 0, [bar, baz], baz, abc, bar    # swap
abc, foo, 0, [bar, baz], baz, def=        # send
abc, foo, 0, [bar, baz], baz              # pop
abc, foo, 0, [bar, baz], baz, foo         # topn 3
abc, foo, 0, [bar, baz], baz, foo, 0      # topn 3
abc, foo, 0, [bar, baz], baz, foo, 0, baz # topn 2
abc, foo, 0, [bar, baz], baz, []=         # send
abc, foo, 0, [bar, baz], baz              # pop
abc, foo, 0, [bar, baz]                   # pop
[bar, baz], foo, 0, [bar, baz]            # setn 3
[bar, baz], foo, 0                        # pop
[bar, baz], foo                           # pop
[bar, baz]                                # pop
```

As multiple assignment must deal with splats, post args, and any level
of nesting, it gets quite a bit more complex than this in non-trivial
cases. To handle this, struct masgn_state is added to keep
track of the overall state of the mass assignment, which stores a linked
list of struct masgn_attrasgn, one for each assigned attribute.

This adds a new optimization that replaces a topn 1/pop instruction
combination with a single swap instruction for multiple assignment
to non-aref attributes.

This new approach isn't compatible with one of the optimizations
previously used, in the case where the multiple assignment return value
was not needed, there was no lhs splat, and one of the left hand side
used an attribute setter.  This removes that optimization. Removing
the optimization allowed for removing the POP_ELEMENT and adjust_stack
functions.

This adds a benchmark to measure how much slower multiple
assignment is with the correct evaluation order.

This benchmark shows:

* 4-9% decrease for attribute sets
* 14-23% decrease for array member sets
* Basically same speed for local variable sets

Importantly, it shows no significant difference between the popped
(where return value of the multiple assignment is not needed) and
!popped (where return value of the multiple assignment is needed)
cases for attribute and array member sets.  This indicates the
previous optimization, which was dropped in the evaluation
order fix and only affected the popped case, is not important to
performance.

Fixes [Bug ruby#4443]

* * 2021-04-22 [ci skip]

* Remove reverse VM instruction

This was previously only used by the multiple assignment code, but
is no longer needed after the multiple assignment execution order
fix.

* fix raise in exception with jump

add_ensure_iseq() adds ensure block to the end of
jump such as next/redo/return. However, if the rescue
cause are in the body, this rescue catches the exception
in ensure clause.

  iter do
    next
  rescue
    R
  ensure
    raise
  end

In this case, R should not be executed, but executed without this patch.

Fixes [Bug #13930]
Fixes [Bug #16618]

A part of tests are written by @jeremyevans ruby#4291

* [ruby/time] Make Time friendly to Ractor

ruby/time@c784e4f166

* [ruby/cgi] handle invalid encoding

ruby/cgi@2b1c2e21a4

* [ruby/cgi] Add test for escapeHTML/unescapeHTML invalid encoding fix in pure ruby version

Also, remove pointless assert_nothing_raised(ArgumentError) while
here.

ruby/cgi@c05edf5608

* [ruby/cgi] gemspec: Explicitly empty executables list

The gem exposes no executables

ruby/cgi@cd7106ad97

* [ruby/benchmark] Add comment about terminating newline in captions; fix test method name.

ruby/benchmark@02ce298d3e

* [ruby/benchmark] gemspec: Explicitly have 0 executables

This gem exposes no executables.

ruby/benchmark@ff1ef7ae06

* Ignore JRuby files on io-console

* [ruby/io-console] Enable building the C extension on TruffleRuby.

ruby/io-console@c17b8cf3a9

* [ruby/io-console] Move FFI console under lib

Having the separate dir makes testing difficult and doesn't
reflect the structure the gem will eventually have. We can filter
these files out if necessary when building the CRuby gem.

ruby/io-console@881010447c

* Separate test used by test_ractor for Ractor in test_time.rb

* Merge net-imap-0.2.0

* [ruby/net-imap] Set timeout for IDLE responses

Fixes ruby#14

ruby/net-imap@39d39ff9bb

* [ruby/net-imap] Bump version to 0.2.1

ruby/net-imap@31f96ea884

* [ruby/uri] Upstream Java proxy property checks from JRuby

These Java properties, retrieved from JRuby's "Java env" ENV_JAVA,
allow JRuby users to use the same proxy properties the rest of the
Java platform uses.

This resolves https://bugs.ruby-lang.org/issues/11194

ruby/uri@3bd2bcc95a

* [ruby/uri] Optimize URI#hostname and URI#hostname=

ruby/uri@3b7ccfd835

* [ruby/uri] Add tests for URI::RFC{2396,3986}_Parser#inspect

ruby/uri@d47dae2f8e

* [ruby/uri] Only use UnboundMethod#bind_call if it is available

This allows tests to pass on Ruby 2.4-2.6.

Fixes ruby#19

ruby/uri@67ca99ca87

* [ruby/uri] Set required_ruby_version to 2.4 in gemspec

Tests pass on Ruby 2.4, but not on Ruby 2.3.

ruby/uri@594418079a

* [ruby/uri] remove comment about URI::escape as it is removed

ruby/uri@0f0057e1b2

* [ruby/uri] Use Regexp#match? to avoid extra allocations

`#=~` builds `MatchData`, requiring extra allocations as compared to
`#match?`, which returns a boolean w/o having to build the `MatchData`.

ruby/uri@158f58a9cc

* Update bundled_gems

* Suppress warnings for unsued variable

* * 2021-04-23 [ci skip]

* Remove unneeded comment

* test/ruby/test_assignment.rb: Avoid "assigned but unused variable"

* Fix wrong documentation

It doesn't return `nil` but raises an exception, as explained a few lines after

* * 2021-04-24 [ci skip]

* Fix setting method visibility for a refinement without an origin class

If a class has been refined but does not have an origin class,
there is a single method entry marked with VM_METHOD_TYPE_REFINED,
but it contains the original method entry.  If the original method
entry is present, we shouldn't skip the method when searching even
when skipping refined methods.

Fixes [Bug #17519]

* Remove unnecessary checks for empty kw splat

These two checks are surrounded by an if that ensures the
call site is not a kw splat call site.

* Remove part of comment that is no longer accurate

In Ruby 2.7, empty keyword splats could be added back for backwards
compatibility.  However, that stopped in Ruby 3.0.

* Add back checks for empty kw splat with tests (ruby#4405)

This reverts commit a224ce8.
Turns out the checks are needed to handle splatting an array with an
empty ruby2 keywords hash.

* [Doc] Fix a typo s/invokations/invocations/

* * 2021-04-25 [ci skip]

* [Doc] Fix a typo s/evel/eval/

* [Doc] Fix a typo s/oher/other/

* [Doc] Fix a typo s/visilibity/visibility/

* [Doc] Fix a typo s/arround/around/

* [Doc] Fix a typo s/daguten/dakuten/

* [ci skip] Fix a typo s/certificiate/certificate/

* [Doc] Fix a typo s/algorthm/algorithm/

* Fix some typos by spell checker

* * 2021-04-26 [ci skip]

* Remove test of removed reverse VM instruction

since 5512353

* spec/ruby/core/file/shared/read.rb: The behavior of FreeBSD was changed

http://rubyci.s3.amazonaws.com/freebsd12/ruby-master/log/20210426T003001Z.fail.html.gz#rubyspec

* disable shareable_constant_value for CI

To debug CI failures on FreeBSD, disable `shareable_constant_value`.

* [ruby/irb] Fix typo ture -> true [ci skip]

ruby/irb@783a0569e8

* [ruby/irb] Added assert_equal_with_term

ruby/irb@b690da96d8

* [ruby/irb] Added test_colorize

ruby/irb@10e290fc3a

* [ruby/irb] Assertions on non-tty

ruby/irb@ede12890d2

* [ruby/irb] Added `colorable` keyword option

Currently `IRB::Color.colorize` and `IRB::Color.colorize_code`
refer `$stdin.tty?` internally.
This patch adds `colorable` keyword option which overrides it.

ruby/irb@402e3f1907

* [ruby/irb] Added setup and teardown to TestIRB::TestInit

Not to be affected by existing rc files in all tests.

ruby/irb@bf434892b4

* node.c (rb_ast_new): imemo_ast is WB-unprotected

Previously imemo_ast was handled as WB-protected which caused a segfault
of the following code:

    # shareable_constant_value: literal
    M0 = {}
    M1 = {}
    ...
    M100000 = {}

My analysis is here: `shareable_constant_value: literal` creates many
Hash instances during parsing, and add them to node_buffer of imemo_ast.
However, the contents are missed because imemo_ast is incorrectly
WB-protected.

This changeset makes imemo_ast as WB-unprotected.

* Revert "disable shareable_constant_value for CI"

This reverts commit c647205.

Maybe the root issue was fixed by 7ac078e

* Document binding behavior for C call/return events for TracePoint/set_trace_func

C methods do not have bindings, so binding returns the binding of
the nearest C method.

Fixes [Bug ruby#9009]

* * 2021-04-27 [ci skip]

* Fix compiler warnings in objspace_dump.c when assertions are turned on

Example:

```
In file included from ../../../include/ruby/defines.h:72,
                 from ../../../include/ruby/ruby.h:23,
                 from ../../../gc.h:3,
                 from ../../../ext/objspace/objspace_dump.c:15:
../../../ext/objspace/objspace_dump.c: In function ‘dump_append_ld’:
../../../ext/objspace/objspace_dump.c:95:26: warning: comparison of integer expressions of different signedness: ‘long unsigned int’ and ‘int’ [-Wsign-compare]
   95 |     RUBY_ASSERT(required <= width);
      |                          ^~
```

* Fix type-o in insns.def

"redefine" -> "redefined"

* Partially revert 2c7d3b3

to make imemo_ast WB-protected again. Only the test is kept.

* Make imemo_ast WB-protected again

by firing the write barrier of imemo_ast after nd_lit is modified.
This will fix the issue of ruby#4416 more
gracefully.

* test/ruby/test_exception.rb: suppress "warning: statement not reached"

* [ruby/pathname] gemspec: Explicitly list 0 executables

This gem exposes no executables.

ruby/pathname@c401d97d58

* [ruby/gdbm] Add dependency to gdbm package on mingw

RubyInstaller2 supports metadata tags for installation of dependent
MSYS2/MINGW libraries. The openssl gem requires the mingw-openssl
package to be installed on the system, which the gem installer takes
care about, when this tag is set.

The feature is documented here:
https://github.com/oneclick/rubyinstaller2/wiki/For-gem-developers#msys2-library-dependency

Fixes oneclick/rubyinstaller2#163

ruby/gdbm@d95eed3e86

* [ruby/matrix] Use Gemfile instead of Gem::Specification#add_development_dependency.

ruby/matrix@1381fde5c1

* [ruby/matrix] v0.4.0

ruby/matrix@baea4b90d4

* [ruby/matrix] v0.4.1

ruby/matrix@f7c9981907

* [ruby/matrix] Guard for < Ruby 3.0

ruby/matrix@1ef660c627

* [ruby/net-ftp] Re-apply 827e471d438fdec1ae329afb5912b8e06d534823

ruby/net-ftp@3ca80368c4

* [ruby/net-ftp] Replace Timeout.timeout with socket timeout

Timeout.timeout is inefficient since it spins up a new thread for
each invocation, use Socket.tcp's connect_timeout option instead
when we aren't using SOCKS (we can't replace Timeout.timeout
for SOCKS yet since SOCKSSocket doesn't have a connect_timeout
option).

ruby/net-ftp@d65910132f

* [ruby/net-ftp] Close the passive connection data socket if there is an error setting up the transfer

Previously, the connection leaked in this case.  This uses
begin/ensure and checking for an error in the ensure block.

An alternative approach would be to not even perform the
connection until after the RETR (or other) command has been
sent.  However, I'm not sure all FTP servers support that.
The current behavior is:

* Send (PASV/EPSV)
* Connect to the host/port returned in 227/229 reply
* Send (RETR/other command)

Changing it to connect after the RETR could break things.
FTP servers might expect that the client has already
connected before sending the RETR.  The alternative
approach is more likely to introduce backwards compatibility
issues, compared to the begin/ensure approach taken here.

Fixes Ruby Bug 17027

ruby/net-ftp@6e8535f076

* [ruby/net-ftp] Reduce resource cosumption of Net::FTP::TIME_PARSER

Reported by Alexandr Savca as a DoS vulnerability, but Net::FTP is a
client library and the impact of the issue is low, so I have decided
to fix it as a normal issue.
Based on patch by nobu.

ruby/net-ftp@a93af636f8

* [ruby/net-ftp] Add test cases

ruby/net-ftp@865232bb2a

* [ruby/net-ftp] Replace "iff" with "if and only if"

iff means if and only if, but readers without that knowledge might
assume this to be a spelling mistake. To me, this seems like
exclusionary language that is unnecessary. Simply using "if and only if"
instead should suffice.

ruby/net-ftp@e920473618

* lldb: Add Freelist Index to dump_page output

* lldb: dump_page_rvalue - dump a heap page containing an RVALUE

rather than having to do this in a two step process:

1. heap_page obj
2. dump_page $2 (or whatever lldb variable heap_page set)

we can now just

dump_page_rvalue obj

* lldb: highlight the slot when using dump_page_rvalue

* Fix Monitor to lock per Fiber, like Mutex [Bug #17827]

* * 2021-04-28 [ci skip]

* test/ruby/test_fiber.rb: reduce the count of object creation to cause GC

... on Solaris. This is the same as 5478871.

http://rubyci.s3.amazonaws.com/solaris10-gcc/ruby-master/log/20210427T160003Z.fail.html.gz
```
[ 7667/20965] TestFiber#test_fork_from_fiber/export/home/users/chkbuild/cb-gcc/tmp/build/20210427T160003Z/ruby/test/ruby/test_fiber.rb:397:in `transfer': can't alloc machine stack to fiber (1 x 139264 bytes): Not enough space (FiberError)
	from /export/home/users/chkbuild/cb-gcc/tmp/build/20210427T160003Z/ruby/test/ruby/test_fiber.rb:397:in `block (6 levels) in test_fork_from_fiber'
	from /export/home/users/chkbuild/cb-gcc/tmp/build/20210427T160003Z/ruby/test/ruby/test_fiber.rb:396:in `times'
	from /export/home/users/chkbuild/cb-gcc/tmp/build/20210427T160003Z/ruby/test/ruby/test_fiber.rb:396:in `block (5 levels) in test_fork_from_fiber'
	from /export/home/users/chkbuild/cb-gcc/tmp/build/20210427T160003Z/ruby/test/ruby/test_fiber.rb:392:in `fork'
	from /export/home/users/chkbuild/cb-gcc/tmp/build/20210427T160003Z/ruby/test/ruby/test_fiber.rb:392:in `block (4 levels) in test_fork_from_fiber'
 = 0.88 s

...

  1) Failure:
TestFiber#test_fork_from_fiber [/export/home/users/chkbuild/cb-gcc/tmp/build/20210427T160003Z/ruby/test/ruby/test_fiber.rb:409]:
[ruby-core:41456].
<0> expected but was
<1>.
```

* test/net/ftp/test_ftp.rb: remove unused variable

* test/net/ftp/test_ftp.rb: reduce the size of a long response

"9" * 999999999 (about 1 GB) was too large for some CI servers.
This commit changes the size to 999999 (about 1 MB).

http://rubyci.s3.amazonaws.com/scw-9d6766/ruby-master/log/20210427T141707Z.fail.html.gz
http://rubyci.s3.amazonaws.com/raspbian10-aarch64/ruby-master/log/20210427T145408Z.fail.html.gz

* test/net/ftp/test_ftp.rb: Use RubyVM::JIT instead of RubyVM::MJIT

* [ruby/net-smtp] Net::SMTP.start() and #start() accepts ssl_context_params keyword argument

	Additional params are passed to OpenSSL::SSL::SSLContext#set_params.

	For example, `Net::SMTP#start(ssl_context_params: { cert_store: my_store, timeout: 123 })`
	calls `set_params({ cert_store: my_store, timeout: 123 })`.

ruby/net-smtp@4213389c21

* [ruby/net-smtp] Replace Timeout.timeout with socket timeout

Timeout.timeout is inefficient since it spins up a new thread for
each invocation, use Socket.tcp's connect_timeout option instead

ruby/net-smtp@6ae4a59f05

* [ruby/net-smtp] Removed needless files from Gem::Specification#files

ruby/net-smtp@69bba6b125

* [ruby/net-smtp] mod: bump to a new VERSION that could be checked for testings >0.2.1

ruby/net-smtp@8f2c9323e2

* [ruby/net-http] Replace Timeout.timeout in Net:HTTP#connect

Use Socket.tcp's connect_timeout option instead

ruby/net-http@753cae3bbc

* [ruby/net-http] Decode user and password from env configured proxy

If someone sets an env variable defining a http_proxy, containing a
username / password with percent-encoded characters, then the resulting
base64 encoded auth header will be wrong.

For example, suppose a username is `Y\X` and the password is `R%S] ?X`.
Properly URL encoded the proxy url would be:

    http://Y%5CX:R%25S%5D%20%3FX@proxy.example:8000

The resulting proxy auth header should be: `WVxYOlIlU10gP1g=`, but the
getters defined by ruby StdLib `URI` return a username `Y%5CX` and
password `R%25S%5D%20%3FX`, resulting in `WSU1Q1g6UiUyNVMlNUQlMjAlM0ZY`.
As a result the proxy will deny the request.

Please note that this is my first contribution to the ruby ecosystem, to
standard lib especially and I am not a ruby developer.

References:

- https://gitlab.com/gitlab-org/gitlab/-/issues/289836
- https://bugs.ruby-lang.org/projects/ruby-master/repository/trunk/revisions/58461
- https://bugs.ruby-lang.org/issues/17542

ruby/net-http@e57d4f38aa

* [ruby/net-http] Fix the regexp used to clean the host

Introduced in ruby@c1652035644

`/s` marks the regexp as encoded with Windows-31J which makes little
sense.

Nurse thinks the intent was to use `/m` for a multi-line regexp.

ruby/net-http@6c15342cdf

* [ruby/net-http] Initialize OpenSSL early before creating TCPSocket

OpenSSL make take some time to initialize, and it would be best
to take that time before connecting instead of after.

From joshc on Redmine.

Fixes Ruby Bug ruby#9459

ruby/net-http@14e09fba24

* [ruby/net-imap] Fix typo intentionaly -> intentionally [ci skip]

ruby/net-imap@4057c662e7

* [ruby/optparse] Add EditorConfig file

More info here: https://editorconfig.org/

For example, `ruby/ruby` has it: https://github.com/ruby/ruby/blob/05ebaee/.editorconfig

Also fix some offenses.

ruby/optparse@29402e7e0e

* [ruby/ostruct] Compatibility with Ruby 2.5

ruby/ostruct@ecd9fafdf8

* [ruby/ostruct] Add compatibility for to_h with block in Ruby 2.5

ruby/ostruct@da45de5068

* Guard for < Ruby 3.0

* NDEBUG is ignored since Ruby 3.0

* Removed unused macro HAVE_CONFIG_H

It seems like a vestige of ext/md5.

* Specify -c to emit pch with clang (ruby#4423)

[Bug #17836]

* * 2021-04-29 [ci skip]

* test/net/smtp/test_smtp.rb: wait a moment before socket is closed

On Solaris, Socket.tcp seems to fail with EINVAL if the server closes
the connection immediately after accpeted. I think this is a bug of
Socket.tcp, but seems difficult to fix soon.

http://rubyci.s3.amazonaws.com/solaris11-sunc/ruby-master/log/20210429T100007Z.fail.html.gz
```
  1) Failure:
Net::TestSMTP#test_eof_error_backtrace [/export/home/chkbuild/chkbuild-sunc/tmp/build/20210429T100007Z/ruby/test/net/smtp/test_smtp.rb:193]:
[ruby-core:78550] [Bug ruby#13018].
[EOFError] exception expected, not #<Net::ReadTimeout: Net::ReadTimeout>.
```

* * 2021-04-30 [ci skip]

* lldb: Warn when attempting to dump invalid pages

* Update Time documentation

* [ruby/irb] Need reline >= 0.1.6

irb 1.3.5 need reline >= 0.1.6 because irb use `Reline::IOGate.in_pasting?`.
This method defined after reline 0.1.6.

fix ruby#228.

ruby/irb@6b7b8fc324

* Silence GCC 11 warnings

```
../strftime.c: In function 'rb_strftime_with_timespec':
../strftime.c:392:39: warning: comparison is always false due to limited range of data type [-Wtype-limits]
  392 |                         if (vtm->wday < 0 || vtm->wday > 6)
      |                                       ^
../strftime.c:403:39: warning: comparison is always false due to limited range of data type [-Wtype-limits]
  403 |                         if (vtm->wday < 0 || vtm->wday > 6)
      |                                       ^
```

* Correct documentation example on Hash#dig

Fixes [Misc #17842]. The current documentation suggests that:

        {foo: {bar: {baz: 2}}}.dig(:foo, :bar) # => {:bar=>{:baz=>2}}

when it should be:

        {foo: {bar: {baz: 2}}}.dig(:foo, :bar) # => {:baz=>2}

* * 2021-05-01 [ci skip]

* Fix example for custom warn method

Regexp has a match? method.

[ci skip]

* Adjust struct member offset for i386 Cygwin

Fixes [Bug #17606]

* Get rid of misleading indentation

* * 2021-05-03 [ci skip]

* Workaround failures on Windows

* * 2021-05-04 [ci skip]

* Eagerly allocate instance variable tables along with object

This allows us to allocate the right size for the object in advance,
meaning that we don't have to pay the cost of ivar table extension
later.  The idea is that if an object type ever became "extended" at
some point, then it is very likely it will become extended again.  So we
may as well allocate the ivar table up front.

* Fix test/net/http/test_https.rb host naming for Windows

* spec/ruby/library/net/http/http/fixtures/http_server.rb host naming for Windows

* Suppress maybe-uninitialized warning by mingw gcc 11

* test/net/imap/test_imap.rb: wait a moment before socket is closed

to try to suppress a failure on Solaris. This is the same as
19504d1

http://rubyci.s3.amazonaws.com/solaris11-sunc/ruby-master/log/20210504T070007Z.fail.html.gz
```
  1) Error:
IMAPTest#test_idle_done_not_during_idle:
Errno::EINVAL: Invalid argument - connect(2) for [::1]:33839
    /export/home/chkbuild/chkbuild-sunc/tmp/build/20210504T070007Z/ruby/.ext/common/socket.rb:1214:in `__connect_nonblock'
    /export/home/chkbuild/chkbuild-sunc/tmp/build/20210504T070007Z/ruby/.ext/common/socket.rb:1214:in `connect_nonblock'
    /export/home/chkbuild/chkbuild-sunc/tmp/build/20210504T070007Z/ruby/.ext/common/socket.rb:56:in `connect_internal'
    /export/home/chkbuild/chkbuild-sunc/tmp/build/20210504T070007Z/ruby/.ext/common/socket.rb:137:in `connect'
    /export/home/chkbuild/chkbuild-sunc/tmp/build/20210504T070007Z/ruby/.ext/common/socket.rb:642:in `block in tcp'
    /export/home/chkbuild/chkbuild-sunc/tmp/build/20210504T070007Z/ruby/.ext/common/socket.rb:227:in `each'
    /export/home/chkbuild/chkbuild-sunc/tmp/build/20210504T070007Z/ruby/.ext/common/socket.rb:227:in `foreach'
    /export/home/chkbuild/chkbuild-sunc/tmp/build/20210504T070007Z/ruby/.ext/common/socket.rb:632:in `tcp'
    /export/home/chkbuild/chkbuild-sunc/tmp/build/20210504T070007Z/ruby/lib/net/imap.rb:1223:in `tcp_socket'
    /export/home/chkbuild/chkbuild-sunc/tmp/build/20210504T070007Z/ruby/lib/net/imap.rb:1180:in `initialize'
    /export/home/chkbuild/chkbuild-sunc/tmp/build/20210504T070007Z/ruby/test/net/imap/test_imap.rb:289:in `new'
    /export/home/chkbuild/chkbuild-sunc/tmp/build/20210504T070007Z/ruby/test/net/imap/test_imap.rb:289:in `test_idle_done_not_during_idle'
```

* Add -Werror=undef to default warnflags for core

* See [Feature #17752]
* For external extensions it's transformed to just warn and not error (-Wundef)
  like other other -Werror in warnflags.

* vm_dump.c: rename HAVE_BACKTRACE to USE_BACKTRACE

* HAVE_ macros should only be defined or undefined, not used for their value.
* See [Feature #17752]

Co-authored-by: xtkoba (Tee KOBAYASHI) <xtkoba+ruby@gmail.com>

* Fix -Wundef warnings in coroutine/*/Context.h

* See [Feature #17752]

Co-authored-by: xtkoba (Tee KOBAYASHI) <xtkoba+ruby@gmail.com>

* Fix trivial -Wundef warnings

* See [Feature #17752]

Co-authored-by: xtkoba (Tee KOBAYASHI) <xtkoba+ruby@gmail.com>

* Add RBIMPL_RVALUE_EMBED_LEN_MAX neeeded by internal/bignum.h

* It evaluated to 0 before, revealed by -Wundef
* See [Feature #17752]

Co-authored-by: xtkoba (Tee KOBAYASHI) <xtkoba+ruby@gmail.com>

* Fix -Wundef warnings in core extensions

* See [Feature #17752]

* Fix -Wundef warnings for HAVE_RB_EXT_RACTOR_SAFE

* See [Feature #17752]

* Fix -Wundef warnings for patterns `#if HAVE`

* See [Feature #17752]
* Using this to detect them:
  git grep -P 'if\s+HAVE' | grep -Pv 'HAVE_LONG_LONG|/ChangeLog|HAVE_TYPEOF'

* HAVE_* macros should not be defined with value 0

* See [Feature #17752]

* Fix -Wundef warnings for RBIMPL_HAS_BUILTIN

* See [Feature #17752]
* Defining explicitly to 0 seems the best solution, see ruby#4428
* For example:
./include/ruby/internal/has/builtin.h:49:33: error: "RBIMPL_HAS_BUILTIN___builtin_assume" is not defined, evaluates to 0 [-Werror=undef]
   49 | # define RBIMPL_HAS_BUILTIN(_) (RBIMPL_HAS_BUILTIN_ ## _)
      |                                 ^~~~~~~~~~~~~~~~~~~
./include/ruby/internal/assume.h:75:7: note: in expansion of macro ‘RBIMPL_HAS_BUILTIN’
   75 | #elif RBIMPL_HAS_BUILTIN(__builtin_assume)
      |       ^~~~~~~~~~~~~~~~~~

* Correctly update array capacity after realloc

Reallocating to a smaller size in the transient heap may result in no
change in the actual capacity but the capacity of the array is still
updated to the smaller value.

This commit changes `ary_heap_realloc` to return the new capacity which
can be used by the caller to correctly update the capacity.

* * 2021-05-05 [ci skip]

* Fix documentation for IO#unget{byte,c}

Fixes [Bug #14400]

* What's Here for class IO (ruby#4440)

* What's Here for class IO

* test/net/ftp/test_ftp.rb - fix intermittent MinGW failure

Fixes intermittent error as below:

[242/838] 5316=test_ftp
#<Thread:0x0000020aa8733f20 D:/a/ruby/ruby/src/test/net/ftp/test_ftp.rb:2532 run> terminated with exception (report_on_exception is true):
D:/a/ruby/ruby/src/tool/lib/minitest/unit.rb:199:in `assert': Expected #<Errno::ECONNRESET: An existing connection was forcibly closed by the remote host.> to be nil. (MiniTest::Assertion)
	from D:/a/ruby/ruby/src/tool/lib/test/unit/core_assertions.rb:504:in `assert'
	from D:/a/ruby/ruby/src/tool/lib/minitest/unit.rb:299:in `assert_nil'
	from D:/a/ruby/ruby/src/test/net/ftp/test_ftp.rb:430:in `ensure in block in test_list_read_timeout_exceeded'
	from D:/a/ruby/ruby/src/test/net/ftp/test_ftp.rb:431:in `block in test_list_read_timeout_exceeded'
	from D:/a/ruby/ruby/src/test/net/ftp/test_ftp.rb:2539:in `block in create_ftp_server'
D:/a/ruby/ruby/src/test/net/ftp/test_ftp.rb:426:in `write': An existing connection was forcibly closed by the remote host. (Errno::ECONNRESET)
	from D:/a/ruby/ruby/src/test/net/ftp/test_ftp.rb:426:in `print'
	from D:/a/ruby/ruby/src/test/net/ftp/test_ftp.rb:426:in `block (2 levels) in test_list_read_timeout_exceeded'
	from D:/a/ruby/ruby/src/test/net/ftp/test_ftp.rb:420:in `each'
	from D:/a/ruby/ruby/src/test/net/ftp/test_ftp.rb:420:in `each_with_index'
	from D:/a/ruby/ruby/src/test/net/ftp/test_ftp.rb:420:in `block in test_list_read_timeout_exceeded'
	from D:/a/ruby/ruby/src/test/net/ftp/test_ftp.rb:2539:in `block in create_ftp_server'

* Fix compilation errors for c99

ENUM_OVER_INT is sometimes not defined. Use #ifdef instead if #if.

* Fix compilation error in mingw

__LITTLE_ENDIAN is not defined.

* Fix compilation errors in FreeBSD

__FreeBSD_version is defined in sys/param.h.

* Fix compilation error in thread_win32.c

USE_WIN32_MUTEX flag may not be defined.

* `_MSC_VER` may not be defined

* Workaround for gcc-4 bug

False positive `-Wundef` in `#elif` after `#if defined`.

* configure.ac: check if __builtin_expect is available or not

include/ruby/internal/has/builtin.h uses HAVE_BUILTIN___BUILTIN_EXPECT
for icc but previously it was not defined.

This is a follow up of 8b32de2 and this
will fix the following failures:

http://rubyci.s3.amazonaws.com/icc-x64/ruby-master/log/20210505T030003Z.fail.html.gz
```
  1) Failure:
TestMkmf::TestConvertible#test_typeof_builtin [/home/chkbuild/chkbuild/tmp/build/20210505T030003Z/ruby/test/mkmf/test_convertible.rb:9]:
convertible_int: checking for convertible type of short... -------------------- short

--------------------

convertible_int: checking for convertible type of int... -------------------- int

--------------------

convertible_int: checking for convertible type of long... -------------------- long

--------------------

convertible_int: checking for convertible type of signed short... -------------------- failed

"icc -std=gnu99 -o conftest -I. -I/home/chkbuild/chkbuild/tmp/build/20210505T030003Z/ruby/.ext/include/x86_64-linux -I/home/chkbuild/chkbuild/tmp/build/20210505T030003Z/ruby/include -I./test    -O3 -ggdb -Wall -Wextra -Wdeprecated-declarations -Wimplicit-function-declaration -Wimplicit-int -Wpointer-arith -Wshorten-64-to-32 -Wwrite-strings -Wno-long-long -Wno-missing-field-initializers -Wno-overlength-strings -Wno-unused-parameter -Wunused-variable -diag-disable=175,188,1684,2259,2312 -Wextra-tokens -Wundef conftest.c  -L. -L/home/chkbuild/chkbuild/tmp/build/20210505T030003Z/ruby -Wl,-rpath,/home/chkbuild/chkbuild/tmp/build/20210505T030003Z/ruby -L. -fstack-protector-strong -rdynamic -Wl,-export-dynamic     -Wl,-rpath,/home/chkbuild/chkbuild/tmp/build/20210505T030003Z/lib -L/home/chkbuild/chkbuild/tmp/build/20210505T030003Z/lib -lruby-static -lz -lpthread -lrt -lrt -ldl -lcrypt -lm   -lm   -lc"
In file included from /home/chkbuild/chkbuild/tmp/build/20210505T030003Z/ruby/include/ruby/defines.h(72),
                 from /home/chkbuild/chkbuild/tmp/build/20210505T030003Z/ruby/include/ruby/ruby.h(23),
                 from /home/chkbuild/chkbuild/tmp/build/20210505T030003Z/ruby/include/ruby.h(39),
                 from conftest.c(1):
/home/chkbuild/chkbuild/tmp/build/20210505T030003Z/ruby/include/ruby/backward/2/assume.h(34): warning ruby#193: zero used for undefined preprocessing identifier "HAVE_BUILTIN___BUILTIN_EXPECT"
  #if RBIMPL_HAS_BUILTIN(__builtin_expect)

...
```

* Fix compilation on M1 Mac

As PAGE_SIZE may not be a preprocessor constant, dispatch at
runtime in that case.

* * 2021-05-06 [ci skip]

* PAGE_SIZE is used only when mmap is available

* Fix PAGE_SIZE macro detection in autoconf

The current fix for PAGE_SIZE macro detection in autoconf does not work
correctly. I see the following output with running configure on Linux:

```
checking PAGE_SIZE is defined... no
```

Linux has PAGE_SIZE macro. This is happening because the macro exists in
sys/user.h and not in the malloc headers.

* Fall back to sysconf to determine page size during runtime

On some platforms the PAGE_SIZE macro does not exist so we can fall back
to `sysconf` to determine the page size at runtime.

* Get rid of including sys/user.h on macOS

LIST_HEAD in ccan/list conflicts with sys/queue.h.

```
./ccan/list/list.h:75:9: warning: 'LIST_HEAD' macro redefined [-Wmacro-redefined]
        ^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/queue.h:465:9: note: previous definition is here
        ^
```

* Check only whether PAGE_SIZE is compile-time const

* Revised PAGE_MAX_SIZE case

* Reuse sysconf result

* [ruby/erb] Document that `<% #` doesn't work

[Bug #17846]

ruby/erb@b58b188

* [ruby/net-imap] Move each authenticator to its own file

Also updates rdoc with SASL specifications and deprecations.  Of these
four, only `PLAIN` isn't deprecated!

+@@Authenticators+ was changed to a class instance var
+@Authenticators+.  No one should have been using the class variable
directly, so that should be fine.

ruby/net-imap@23f241b081

* [ruby/net-imap] Update AUTH=PLAIN to be a little closer to RFC4616

* Add authzid support
* must not contain NULL chars
* improve rdoc

ruby/net-imap@a587fc71b7

* [ruby/net-imap] Clean up authenticators rdoc

Added RFC links to all SASL mechanism specifications.

ruby/net-imap@53ff4b0c09

* [ruby/net-imap] move ResponseParser to lib/net/imap/response_parser

Partially implements ruby#10.

ruby/net-imap@c2408aac9a

* [ruby/net-imap] move response data structs to net/imap/response_data

Partially implements ruby#10.

ruby/net-imap@746757b936

* [ruby/net-imap] move command data formatters to net/imap/command_data

Partially implements ruby#10.

ruby/net-imap@24e929fdd2

* [ruby/net-imap] Move UTF7 & datetime formatting to net/imap/data_encoding

Partially implements ruby#10.

ruby/net-imap@0d43c5e856

* [ruby/net-imap] Move flags to net/imap/flags

Partially implements ruby#10.

ruby/net-imap@2a9afa83bf

* [ruby/net-imap] Move send_*_data into net/imap/command_data

Partially implements ruby#10.

ruby/net-imap@64d1080d63

* [ruby/net-imap] Many documentation improvements

* updated obsoleted RFCs to current versions
* linked most references to their RFCs
* linked extension commands to their RFCs
* removed unidiomatic `()` from instance method links
* escaped `IMAP` in a few places
* converted all response structs to explicit classes: this makes much
  nicer rdoc output than listing them all under "constants"
* grouped flags constants into their own sections

ruby/net-imap@9cd562ac84

* Move net-imap.gemspec to under the lib/net/imap directory.

* Fixed the file path for net-imap.gemspec

* [ruby/net-http] Do not require stringio

It is not used in net/http library code since commit 15ccd0118c13
(r36473 in ruby svn trunk, 2012).

require's in test suite are also cleaned up.

ruby/net-http@996d18a43f

* Import from ruby/strscan#19

* Use Gemfile instead of Gem::Specification#add_development_dependency.

* Use pend instead of skip for test-unit.

* [ruby/strscan] Fix segmentation fault of `StringScanner#charpos` when `String#byteslice` returns non string value [Bug #17756] (#20)

ruby/strscan@92961cde2b

* [ruby/strscan] Replace "iff" with "if and only if" (ruby#18)

iff means if and only if, but readers without that knowledge might
assume this to be a spelling mistake. To me, this seems like
exclusionary language that is unnecessary. Simply using "if and only if"
instead should suffice.

ruby/strscan@066451c11e

* [ruby/timeout] Make Timeout::Error#exception with multiple arguments not ignore arguments

This makes:

  raise(Timeout::Error.new("hello"), "world")

raise a TimeoutError instance with "world" as the message instead
of "hello", for consistency with other Ruby exception classes.

This required some internal changes to keep the tests passing.

Fixes [Bug #17812]

ruby/timeout@952154dbf9

* [ruby/timeout] Avoid unnecessary object allocation

Idea from nobu.

ruby/timeout@aecdaa23b3

* [ruby/timeout] Only run timeout_after hook on fiber scheduler if scheduler exists

ruby/timeout@4893cde0ed

* Use assert_ractor for separating test processes

* net-http no longer requires stringio

* Allow newobj_of0 and newobj_slowpath to allocate into multiple heap slots

* Store rb_classext_t next to RClass slots on the heap

* lldb: teach rp about T_PAYLOAD

* Fix preprocessor warning/error

* Try to fix #if issue in vm.inc.erb

Co-authored-by: Jeremy Evans <code@jeremyevans.net>
Co-authored-by: git <svn-admin@ruby-lang.org>
Co-authored-by: Koichi Sasada <ko1@atdot.net>
Co-authored-by: Kir Shatrov <shatrov@me.com>
Co-authored-by: pavel <pavel.rosicky@easy.cz>
Co-authored-by: Olle Jonsson <olle.jonsson@gmail.com>
Co-authored-by: Keith Bennett <keithrbennett@gmail.com>
Co-authored-by: Hiroshi SHIBATA <hsbt@ruby-lang.org>
Co-authored-by: Duncan MacGregor <duncan.macgregor@oracle.com>
Co-authored-by: Charles Oliver Nutter <headius@headius.com>
Co-authored-by: Shugo Maeda <shugo@ruby-lang.org>
Co-authored-by: Lukas Zapletal <lzap+git@redhat.com>
Co-authored-by: Felix Wong <felix@waf.hk>
Co-authored-by: Steven Harman <steven@harmanly.com>
Co-authored-by: Kazuhiro NISHIYAMA <zn@mbf.nifty.com>
Co-authored-by: S-H-GAMELINKS <gamelinks007@gmail.com>
Co-authored-by: Yusuke Endoh <mame@ruby-lang.org>
Co-authored-by: romainsalles <romainsalles@users.noreply.github.com>
Co-authored-by: Alan Wu <XrXr@users.noreply.github.com>
Co-authored-by: wonda-tea-coffee <rikita.ishikawa@crowdworks.co.jp>
Co-authored-by: wonda-tea-coffee <lagrange.resolvent@gmail.com>
Co-authored-by: Ryuta Kamizono <kamipo@gmail.com>
Co-authored-by: Nobuyoshi Nakada <nobu@ruby-lang.org>
Co-authored-by: Peter Zhu <peter@peterzhu.ca>
Co-authored-by: ebrohman <ericbrohman@gmail.com>
Co-authored-by: Lars Kanis <kanis@comcard.de>
Co-authored-by: Marc-Andre Lafortune <github@marc-andre.ca>
Co-authored-by: mohamed <mohamed.m.m.hafez@gmail.com>
Co-authored-by: Gannon McGibbon <gannon.mcgibbon@gmail.com>
Co-authored-by: Matt Valentine-House <matt@eightbitraptor.com>
Co-authored-by: Benoit Daloze <eregontp@gmail.com>
Co-authored-by: Tom Freudenberg <tom.freudenberg@4commerce.de>
Co-authored-by: Lukas Eipert <leipert@gitlab.com>
Co-authored-by: Jean Boussier <jean.boussier@gmail.com>
Co-authored-by: Alexander Popov <alex.wayfer@gmail.com>
Co-authored-by: Takashi Kokubun <takashikkbn@gmail.com>
Co-authored-by: Burdette Lamar <BurdetteLamar@Yahoo.com>
Co-authored-by: ima1zumi <mariimaizumi5@gmail.com>
Co-authored-by: xtkoba <69125751+xtkoba@users.noreply.github.com>
Co-authored-by: Nick Kelley <nick@nickkelley.dev>
Co-authored-by: Adam Daniels <adam@mediadrive.ca>
Co-authored-by: Aaron Patterson <tenderlove@ruby-lang.org>
Co-authored-by: MSP-Greg <Greg.mpls@gmail.com>
Co-authored-by: xtkoba (Tee KOBAYASHI) <xtkoba+ruby@gmail.com>
Co-authored-by: nicholas a. evans <nicholas.evans@gmail.com>
Co-authored-by: Kazuki Yamaguchi <k@rhe.jp>
Co-authored-by: Kenichi Kamiya <kachick1@gmail.com>
XrXr pushed a commit that referenced this pull request Oct 20, 2021
* Fix block invalidation assertions

* Add Alan's small repro for double invalidation bug
chrisseaton added a commit to chrisseaton/ruby that referenced this pull request Aug 3, 2022
This commit makes MMTk a runtime option.

Code related to MMTk is still by a macro (USE_MMTK, previously USE_THIRD_PARTY_HEAP), but MMTk also needs to be enabled at run time using command line options (`--mmtk` or `--enable-mmtk`), or environment variables (`RUBYOPT=--mmtk` or `MMTK_PLAN=...`)
tenderlove added a commit to tenderlove/ruby that referenced this pull request Oct 31, 2022
Always look up instance variable buffers when iterating.  It is possible
for the instance variable buffer to change out from under the object
during iteration, so we cannot cache the buffer on the stack.

In the case of Bug #19095, the transient heap moved the buffer during
iteration:

```
;
-> 1361	    }
   1362	}
   1363	#endif
   1364
miniruby`rb_obj_transient_heap_evacuate:
->  0x1006e5178 <+328>: b      0x1006e517c               ; <+332> at variable.c:1362:1
    0x1006e517c <+332>: ldp    x29, x30, [sp, #0x50]
    0x1006e5180 <+336>: add    sp, sp, #0x60
    0x1006e5184 <+340>: ret
Target 0: (miniruby) stopped.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = watchpoint 1
  * frame #0: 0x00000001006e5178 miniruby`rb_obj_transient_heap_evacuate(obj=0x000000010d6b94b0, promote=1) at variable.c:1361:5
    frame #1: 0x00000001006cb150 miniruby`transient_heap_block_evacuate(theap=0x0000000100b196c0, block=0x0000000107c00000) at transient_heap.c:734:17
    frame #2: 0x00000001006c854c miniruby`transient_heap_evacuate(dmy=0x0000000000000000) at transient_heap.c:808:17
    frame #3: 0x00000001007fe6c0 miniruby`rb_postponed_job_flush(vm=0x0000000104402900) at vm_trace.c:1773:21
    frame #4: 0x0000000100637a84 miniruby`rb_threadptr_execute_interrupts(th=0x0000000103803bc0, blocking_timing=0) at thread.c:2316:13
    frame #5: 0x000000010078b730 miniruby`rb_vm_check_ints(ec=0x00000001048038d0) at vm_core.h:2025:9
    frame #6: 0x00000001006fbd10 miniruby`vm_pop_frame(ec=0x00000001048038d0, cfp=0x0000000104a04440, ep=0x0000000104904a28) at vm_insnhelper.c:422:5
    frame ruby#7: 0x00000001006fbca0 miniruby`rb_vm_pop_frame(ec=0x00000001048038d0) at vm_insnhelper.c:431:5
    frame ruby#8: 0x00000001007d6420 miniruby`vm_call0_cfunc_with_frame(ec=0x00000001048038d0, calling=0x000000016fdcc6a0, argv=0x0000000000000000) at vm_eval.c:153:9
    frame ruby#9: 0x00000001007d44cc miniruby`vm_call0_cfunc(ec=0x00000001048038d0, calling=0x000000016fdcc6a0, argv=0x0000000000000000) at vm_eval.c:164:12
    frame ruby#10: 0x0000000100766e80 miniruby`vm_call0_body(ec=0x00000001048038d0, calling=0x000000016fdcc6a0, argv=0x0000000000000000) at vm_eval.c:210:15
    frame ruby#11: 0x00000001007d76f0 miniruby`vm_call0_cc(ec=0x00000001048038d0, recv=0x000000010d6b49d8, id=2769, argc=0, argv=0x0000000000000000, cc=0x000000010d6b2e58, kw_splat=0) at vm_eval.c:87:12
    frame ruby#12: 0x0000000100769e48 miniruby`rb_funcallv_scope(recv=0x000000010d6b49d8, mid=2769, argc=0, argv=0x0000000000000000, scope=CALL_FCALL) at vm_eval.c:1051:16
    frame ruby#13: 0x0000000100760a54 miniruby`rb_funcallv(recv=0x000000010d6b49d8, mid=2769, argc=0, argv=0x0000000000000000) at vm_eval.c:1066:12
    frame ruby#14: 0x000000010037513c miniruby`rb_inspect(obj=0x000000010d6b49d8) at object.c:633:34
    frame ruby#15: 0x000000010002c950 miniruby`inspect_ary(ary=0x000000010d6b4938, dummy=0x0000000000000000, recur=0) at array.c:3091:13
```

In general though, any calls back out to the interpreter could change
the IV buffer, so it's not safe to cache.

[Bug #19095]
tenderlove added a commit to tenderlove/ruby that referenced this pull request Oct 31, 2022
Always look up instance variable buffers when iterating.  It is possible
for the instance variable buffer to change out from under the object
during iteration, so we cannot cache the buffer on the stack.

In the case of Bug #19095, the transient heap moved the buffer during
iteration:

```
Watchpoint 1 hit:
old value: 0x0000000107c00df8
new value: 0x00000001032743c0
Process 31720 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = watchpoint 1
    frame #0: 0x00000001006e5178 miniruby`rb_obj_transient_heap_evacuate(obj=0x000000010d6b94b0, promote=1) at variable.c:1361:5
   1358	        }
   1359	        MEMCPY(new_ptr, old_ptr, VALUE, len);
   1360	        ROBJECT(obj)->as.heap.ivptr = new_ptr;
-> 1361	    }
   1362	}
   1363	#endif
   1364
miniruby`rb_obj_transient_heap_evacuate:
->  0x1006e5178 <+328>: b      0x1006e517c               ; <+332> at variable.c:1362:1
    0x1006e517c <+332>: ldp    x29, x30, [sp, #0x50]
    0x1006e5180 <+336>: add    sp, sp, #0x60
    0x1006e5184 <+340>: ret
Target 0: (miniruby) stopped.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = watchpoint 1
  * frame #0: 0x00000001006e5178 miniruby`rb_obj_transient_heap_evacuate(obj=0x000000010d6b94b0, promote=1) at variable.c:1361:5
    frame #1: 0x00000001006cb150 miniruby`transient_heap_block_evacuate(theap=0x0000000100b196c0, block=0x0000000107c00000) at transient_heap.c:734:17
    frame #2: 0x00000001006c854c miniruby`transient_heap_evacuate(dmy=0x0000000000000000) at transient_heap.c:808:17
    frame #3: 0x00000001007fe6c0 miniruby`rb_postponed_job_flush(vm=0x0000000104402900) at vm_trace.c:1773:21
    frame #4: 0x0000000100637a84 miniruby`rb_threadptr_execute_interrupts(th=0x0000000103803bc0, blocking_timing=0) at thread.c:2316:13
    frame #5: 0x000000010078b730 miniruby`rb_vm_check_ints(ec=0x00000001048038d0) at vm_core.h:2025:9
    frame #6: 0x00000001006fbd10 miniruby`vm_pop_frame(ec=0x00000001048038d0, cfp=0x0000000104a04440, ep=0x0000000104904a28) at vm_insnhelper.c:422:5
    frame ruby#7: 0x00000001006fbca0 miniruby`rb_vm_pop_frame(ec=0x00000001048038d0) at vm_insnhelper.c:431:5
    frame ruby#8: 0x00000001007d6420 miniruby`vm_call0_cfunc_with_frame(ec=0x00000001048038d0, calling=0x000000016fdcc6a0, argv=0x0000000000000000) at vm_eval.c:153:9
    frame ruby#9: 0x00000001007d44cc miniruby`vm_call0_cfunc(ec=0x00000001048038d0, calling=0x000000016fdcc6a0, argv=0x0000000000000000) at vm_eval.c:164:12
    frame ruby#10: 0x0000000100766e80 miniruby`vm_call0_body(ec=0x00000001048038d0, calling=0x000000016fdcc6a0, argv=0x0000000000000000) at vm_eval.c:210:15
    frame ruby#11: 0x00000001007d76f0 miniruby`vm_call0_cc(ec=0x00000001048038d0, recv=0x000000010d6b49d8, id=2769, argc=0, argv=0x0000000000000000, cc=0x000000010d6b2e58, kw_splat=0) at vm_eval.c:87:12
    frame ruby#12: 0x0000000100769e48 miniruby`rb_funcallv_scope(recv=0x000000010d6b49d8, mid=2769, argc=0, argv=0x0000000000000000, scope=CALL_FCALL) at vm_eval.c:1051:16
    frame ruby#13: 0x0000000100760a54 miniruby`rb_funcallv(recv=0x000000010d6b49d8, mid=2769, argc=0, argv=0x0000000000000000) at vm_eval.c:1066:12
    frame ruby#14: 0x000000010037513c miniruby`rb_inspect(obj=0x000000010d6b49d8) at object.c:633:34
    frame ruby#15: 0x000000010002c950 miniruby`inspect_ary(ary=0x000000010d6b4938, dummy=0x0000000000000000, recur=0) at array.c:3091:13
    frame ruby#16: 0x0000000100642020 miniruby`exec_recursive(func=(miniruby`inspect_ary at array.c:3084), obj=0x000000010d6b4938, pairid=0x0000000000000000, arg=0x0000000000000000, outer=0, mid=2769) at thread.c:5177:23
    frame ruby#17: 0x00000001006412fc miniruby`rb_exec_recursive(func=(miniruby`inspect_ary at array.c:3084), obj=0x000000010d6b4938, arg=0x0000000000000000) at thread.c:5205:12
    frame ruby#18: 0x00000001000127f0 miniruby`rb_ary_inspect(ary=0x000000010d6b4938) at array.c:3117:12
```

In general though, any calls back out to the interpreter could change
the IV buffer, so it's not safe to cache.

[Bug #19095]
tenderlove added a commit to tenderlove/ruby that referenced this pull request Oct 31, 2022
Always look up instance variable buffers when iterating.  It is possible
for the instance variable buffer to change out from under the object
during iteration, so we cannot cache the buffer on the stack.

In the case of Bug #19095, the transient heap moved the buffer during
iteration:

```
Watchpoint 1 hit:
old value: 0x0000000107c00df8
new value: 0x00000001032743c0
Process 31720 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = watchpoint 1
    frame #0: 0x00000001006e5178 miniruby`rb_obj_transient_heap_evacuate(obj=0x000000010d6b94b0, promote=1) at variable.c:1361:5
   1358	        }
   1359	        MEMCPY(new_ptr, old_ptr, VALUE, len);
   1360	        ROBJECT(obj)->as.heap.ivptr = new_ptr;
-> 1361	    }
   1362	}
   1363	#endif
   1364
miniruby`rb_obj_transient_heap_evacuate:
->  0x1006e5178 <+328>: b      0x1006e517c               ; <+332> at variable.c:1362:1
    0x1006e517c <+332>: ldp    x29, x30, [sp, #0x50]
    0x1006e5180 <+336>: add    sp, sp, #0x60
    0x1006e5184 <+340>: ret
Target 0: (miniruby) stopped.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = watchpoint 1
  * frame #0: 0x00000001006e5178 miniruby`rb_obj_transient_heap_evacuate(obj=0x000000010d6b94b0, promote=1) at variable.c:1361:5
    frame #1: 0x00000001006cb150 miniruby`transient_heap_block_evacuate(theap=0x0000000100b196c0, block=0x0000000107c00000) at transient_heap.c:734:17
    frame #2: 0x00000001006c854c miniruby`transient_heap_evacuate(dmy=0x0000000000000000) at transient_heap.c:808:17
    frame #3: 0x00000001007fe6c0 miniruby`rb_postponed_job_flush(vm=0x0000000104402900) at vm_trace.c:1773:21
    frame #4: 0x0000000100637a84 miniruby`rb_threadptr_execute_interrupts(th=0x0000000103803bc0, blocking_timing=0) at thread.c:2316:13
    frame #5: 0x000000010078b730 miniruby`rb_vm_check_ints(ec=0x00000001048038d0) at vm_core.h:2025:9
    frame #6: 0x00000001006fbd10 miniruby`vm_pop_frame(ec=0x00000001048038d0, cfp=0x0000000104a04440, ep=0x0000000104904a28) at vm_insnhelper.c:422:5
    frame ruby#7: 0x00000001006fbca0 miniruby`rb_vm_pop_frame(ec=0x00000001048038d0) at vm_insnhelper.c:431:5
    frame ruby#8: 0x00000001007d6420 miniruby`vm_call0_cfunc_with_frame(ec=0x00000001048038d0, calling=0x000000016fdcc6a0, argv=0x0000000000000000) at vm_eval.c:153:9
    frame ruby#9: 0x00000001007d44cc miniruby`vm_call0_cfunc(ec=0x00000001048038d0, calling=0x000000016fdcc6a0, argv=0x0000000000000000) at vm_eval.c:164:12
    frame ruby#10: 0x0000000100766e80 miniruby`vm_call0_body(ec=0x00000001048038d0, calling=0x000000016fdcc6a0, argv=0x0000000000000000) at vm_eval.c:210:15
    frame ruby#11: 0x00000001007d76f0 miniruby`vm_call0_cc(ec=0x00000001048038d0, recv=0x000000010d6b49d8, id=2769, argc=0, argv=0x0000000000000000, cc=0x000000010d6b2e58, kw_splat=0) at vm_eval.c:87:12
    frame ruby#12: 0x0000000100769e48 miniruby`rb_funcallv_scope(recv=0x000000010d6b49d8, mid=2769, argc=0, argv=0x0000000000000000, scope=CALL_FCALL) at vm_eval.c:1051:16
    frame ruby#13: 0x0000000100760a54 miniruby`rb_funcallv(recv=0x000000010d6b49d8, mid=2769, argc=0, argv=0x0000000000000000) at vm_eval.c:1066:12
    frame ruby#14: 0x000000010037513c miniruby`rb_inspect(obj=0x000000010d6b49d8) at object.c:633:34
    frame ruby#15: 0x000000010002c950 miniruby`inspect_ary(ary=0x000000010d6b4938, dummy=0x0000000000000000, recur=0) at array.c:3091:13
    frame ruby#16: 0x0000000100642020 miniruby`exec_recursive(func=(miniruby`inspect_ary at array.c:3084), obj=0x000000010d6b4938, pairid=0x0000000000000000, arg=0x0000000000000000, outer=0, mid=2769) at thread.c:5177:23
    frame ruby#17: 0x00000001006412fc miniruby`rb_exec_recursive(func=(miniruby`inspect_ary at array.c:3084), obj=0x000000010d6b4938, arg=0x0000000000000000) at thread.c:5205:12
    frame ruby#18: 0x00000001000127f0 miniruby`rb_ary_inspect(ary=0x000000010d6b4938) at array.c:3117:12
```

In general though, any calls back out to the interpreter could change
the IV buffer, so it's not safe to cache.

[Bug #19095]
tenderlove added a commit that referenced this pull request Nov 1, 2022
Always look up instance variable buffers when iterating.  It is possible
for the instance variable buffer to change out from under the object
during iteration, so we cannot cache the buffer on the stack.

In the case of Bug #19095, the transient heap moved the buffer during
iteration:

```
Watchpoint 1 hit:
old value: 0x0000000107c00df8
new value: 0x00000001032743c0
Process 31720 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = watchpoint 1
    frame #0: 0x00000001006e5178 miniruby`rb_obj_transient_heap_evacuate(obj=0x000000010d6b94b0, promote=1) at variable.c:1361:5
   1358	        }
   1359	        MEMCPY(new_ptr, old_ptr, VALUE, len);
   1360	        ROBJECT(obj)->as.heap.ivptr = new_ptr;
-> 1361	    }
   1362	}
   1363	#endif
   1364
miniruby`rb_obj_transient_heap_evacuate:
->  0x1006e5178 <+328>: b      0x1006e517c               ; <+332> at variable.c:1362:1
    0x1006e517c <+332>: ldp    x29, x30, [sp, #0x50]
    0x1006e5180 <+336>: add    sp, sp, #0x60
    0x1006e5184 <+340>: ret
Target 0: (miniruby) stopped.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = watchpoint 1
  * frame #0: 0x00000001006e5178 miniruby`rb_obj_transient_heap_evacuate(obj=0x000000010d6b94b0, promote=1) at variable.c:1361:5
    frame #1: 0x00000001006cb150 miniruby`transient_heap_block_evacuate(theap=0x0000000100b196c0, block=0x0000000107c00000) at transient_heap.c:734:17
    frame #2: 0x00000001006c854c miniruby`transient_heap_evacuate(dmy=0x0000000000000000) at transient_heap.c:808:17
    frame #3: 0x00000001007fe6c0 miniruby`rb_postponed_job_flush(vm=0x0000000104402900) at vm_trace.c:1773:21
    frame #4: 0x0000000100637a84 miniruby`rb_threadptr_execute_interrupts(th=0x0000000103803bc0, blocking_timing=0) at thread.c:2316:13
    frame #5: 0x000000010078b730 miniruby`rb_vm_check_ints(ec=0x00000001048038d0) at vm_core.h:2025:9
    frame #6: 0x00000001006fbd10 miniruby`vm_pop_frame(ec=0x00000001048038d0, cfp=0x0000000104a04440, ep=0x0000000104904a28) at vm_insnhelper.c:422:5
    frame #7: 0x00000001006fbca0 miniruby`rb_vm_pop_frame(ec=0x00000001048038d0) at vm_insnhelper.c:431:5
    frame #8: 0x00000001007d6420 miniruby`vm_call0_cfunc_with_frame(ec=0x00000001048038d0, calling=0x000000016fdcc6a0, argv=0x0000000000000000) at vm_eval.c:153:9
    frame #9: 0x00000001007d44cc miniruby`vm_call0_cfunc(ec=0x00000001048038d0, calling=0x000000016fdcc6a0, argv=0x0000000000000000) at vm_eval.c:164:12
    frame #10: 0x0000000100766e80 miniruby`vm_call0_body(ec=0x00000001048038d0, calling=0x000000016fdcc6a0, argv=0x0000000000000000) at vm_eval.c:210:15
    frame #11: 0x00000001007d76f0 miniruby`vm_call0_cc(ec=0x00000001048038d0, recv=0x000000010d6b49d8, id=2769, argc=0, argv=0x0000000000000000, cc=0x000000010d6b2e58, kw_splat=0) at vm_eval.c:87:12
    frame #12: 0x0000000100769e48 miniruby`rb_funcallv_scope(recv=0x000000010d6b49d8, mid=2769, argc=0, argv=0x0000000000000000, scope=CALL_FCALL) at vm_eval.c:1051:16
    frame #13: 0x0000000100760a54 miniruby`rb_funcallv(recv=0x000000010d6b49d8, mid=2769, argc=0, argv=0x0000000000000000) at vm_eval.c:1066:12
    frame #14: 0x000000010037513c miniruby`rb_inspect(obj=0x000000010d6b49d8) at object.c:633:34
    frame #15: 0x000000010002c950 miniruby`inspect_ary(ary=0x000000010d6b4938, dummy=0x0000000000000000, recur=0) at array.c:3091:13
    frame #16: 0x0000000100642020 miniruby`exec_recursive(func=(miniruby`inspect_ary at array.c:3084), obj=0x000000010d6b4938, pairid=0x0000000000000000, arg=0x0000000000000000, outer=0, mid=2769) at thread.c:5177:23
    frame #17: 0x00000001006412fc miniruby`rb_exec_recursive(func=(miniruby`inspect_ary at array.c:3084), obj=0x000000010d6b4938, arg=0x0000000000000000) at thread.c:5205:12
    frame #18: 0x00000001000127f0 miniruby`rb_ary_inspect(ary=0x000000010d6b4938) at array.c:3117:12
```

In general though, any calls back out to the interpreter could change
the IV buffer, so it's not safe to cache.

[Bug #19095]
ko1 added a commit that referenced this pull request Oct 14, 2023
to avoid deadlock

```ruby
r = Ractor.new do
  obj = Thread.new{}
  Ractor.yield obj
rescue => e
  e.message
end
p r.take
```

```
(lldb) bt
* thread #1, name = 'miniruby', stop reason = signal SIGSTOP
  * frame #0: 0x0000ffff44881410 libpthread.so.0`__lll_lock_wait + 88
    frame #1: 0x0000ffff4487a078 libpthread.so.0`__pthread_mutex_lock + 232
    frame #2: 0x0000aaab617c0980 miniruby`rb_native_mutex_lock(lock=<unavailable>) at thread_pthread.c:109:14
    frame #3: 0x0000aaab617c1d58 miniruby`ubf_event_waiting [inlined] thread_sched_lock_(th=0x0000aaab9df82980, file=<unavailable>, line=46, sched=0x0000aaab9dec79b8) at thread_pthread.c:351:5
    frame #4: 0x0000aaab617c1d50 miniruby`ubf_event_waiting(ptr=0x0000aaab9df82980) at thread_pthread_mn.c:46:5
    frame #5: 0x0000aaab617c6020 miniruby`rb_threadptr_interrupt [inlined] rb_threadptr_interrupt_common(trap=0, th=0x0000aaab9df82980) at thread.c:352:25
    frame #6: 0x0000aaab617c5fec miniruby`rb_threadptr_interrupt(th=0x0000aaab9df82980) at thread.c:365:5
    frame #7: 0x0000aaab617379b0 miniruby`rb_ractor_terminate_all at ractor.c:2364:13
    frame #8: 0x0000aaab6173797c miniruby`rb_ractor_terminate_all at ractor.c:2383:17
    frame #9: 0x0000aaab61737958 miniruby`rb_ractor_terminate_all [inlined] ractor_terminal_interrupt_all(vm=0x0000aaab9dea3320) at ractor.c:2375:1
    frame #10: 0x0000aaab61737950 miniruby`rb_ractor_terminate_all at ractor.c:2424:13
    frame #11: 0x0000aaab6164f108 miniruby`rb_ec_cleanup(ec=0x0000aaab9dea5900, ex=RUBY_TAG_NONE) at eval.c:239:9
    frame #12: 0x0000aaab6164fa3c miniruby`ruby_run_node(n=0x0000ffff417ed178) at eval.c:328:12
    frame #13: 0x0000aaab615a5ab0 miniruby`main at main.c:39:12
    frame #14: 0x0000aaab615a5a98 miniruby`main(argc=<unavailable>, argv=<unavailable>) at main.c:58:12
    frame #15: 0x0000ffff44714b2c libc.so.6`__libc_start_main + 228
    frame #16: 0x0000aaab615a5b0c miniruby`_start + 52
(lldb) thread select 3
* thread #3, name = 'bootstraptest.*', stop reason = signal SIGSTOP
    frame #0: 0x0000ffff448813ec libpthread.so.0`__lll_lock_wait + 52
libpthread.so.0`__lll_lock_wait:
->  0xffff448813ec <+52>: svc    #0
    0xffff448813f0 <+56>: eor    w20, w20, #0x80
    0xffff448813f4 <+60>: sxtw   x20, w20
    0xffff448813f8 <+64>: b      0xffff44881414            ; <+92>
(lldb) bt
* thread #3, name = 'bootstraptest.*', stop reason = signal SIGSTOP
  * frame #0: 0x0000ffff448813ec libpthread.so.0`__lll_lock_wait + 52
    frame #1: 0x0000ffff4487a078 libpthread.so.0`__pthread_mutex_lock + 232
    frame #2: 0x0000aaab617c0980 miniruby`rb_native_mutex_lock(lock=<unavailable>) at thread_pthread.c:109:14
    frame #3: 0x0000aaab61823d68 miniruby`rb_vm_lock_enter_body [inlined] vm_lock_enter(no_barrier=false, lev=0x0000ffff215bfbe4, locked=false, vm=0x0000aaab9dea3320, cr=0x0000aaab9dec7890) at vm_sync.c:57:9
    frame #4: 0x0000aaab61823d60 miniruby`rb_vm_lock_enter_body(lev=0x0000ffff215bfbe4) at vm_sync.c:119:9
    frame #5: 0x0000aaab617c1b30 miniruby`thread_sched_setup_running_threads [inlined] rb_vm_lock_enter(file=<unavailable>, line=597, lev=0x0000ffff215bfbe4) at vm_sync.h:75:9
    frame #6: 0x0000aaab617c1b14 miniruby`thread_sched_setup_running_threads(vm=0x0000aaab9dea3320, add_th=0x0000aaab9df82980, del_th=<unavailable>, add_timeslice_th=0x0000000000000000, cr=<unavailable>, sched=<unavailable>, sched=<unavailable>) at thread_pthread.c:597:9
    frame #7: 0x0000aaab617c29b4 miniruby`thread_sched_wait_running_turn at thread_pthread.c:614:5
    frame #8: 0x0000aaab617c298c miniruby`thread_sched_wait_running_turn(sched=0x0000aaab9dec79b8, th=0x0000aaab9df82980, can_direct_transfer=true) at thread_pthread.c:868:9
    frame #9: 0x0000aaab617c6f0c miniruby`thread_sched_wait_events(sched=0x0000aaab9dec79b8, th=0x0000aaab9df82980, fd=<unavailable>, events=<unavailable>, rel=<unavailable>) at thread_pthread_mn.c:90:17
    frame #10: 0x0000aaab617c7354 miniruby`rb_thread_terminate_all at thread_pthread.c:3248:13
    frame #11: 0x0000aaab617c733c miniruby`rb_thread_terminate_all(th=0x0000aaab9df82980) at thread.c:466:13
    frame #12: 0x0000aaab617c7a64 miniruby`thread_start_func_2(th=0x0000aaab9df82980, stack_start=<unavailable>) at thread.c:713:9
    frame #13: 0x0000aaab617c7d1c miniruby`co_start [inlined] call_thread_start_func_2(th=0x0000aaab9df82980) at thread_pthread.c:2165:5
    frame #14: 0x0000aaab617c7cd0 miniruby`co_start(from=<unavailable>, self=0x0000aaab9df0f760) at thread_pthread_mn.c:421:9
```
KJTsanaktsidis added a commit to KJTsanaktsidis/ruby that referenced this pull request Feb 18, 2024
It appears that tok(p) is not NULL terminated here, so we need to use
strndup to copy only the correct number of bytes.

[1/1] TestRubyLiteral#test_integer=================================================================
==484771==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x5060001ab1fc at pc 0x5597fe21d8e1 bp 0x7ffdc6fb0a50 sp 0x7ffdc6fb0210
READ of size 61 at 0x5060001ab1fc thread T0
    #0 0x5597fe21d8e0 in strlen.part.0 /home/kj/llvm-project/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:391:5
    #1 0x5597fe6b2feb in ruby_strdup /home/kj/ruby/build/../util.c:538:18
    ruby#2 0x5597fe4cb1c5 in set_number_literal /home/kj/ruby/build/parse.y:9694:9
    ruby#3 0x5597fe4cab3d in no_digits /home/kj/ruby/build/parse.y:10409:12
    ruby#4 0x5597fe4b9de9 in parse_numeric /home/kj/ruby/build/parse.y
    ruby#5 0x5597fe4a8adf in parser_yylex /home/kj/ruby/build/parse.y
    ruby#6 0x5597fe45c5cd in yylex /home/kj/ruby/build/parse.y:11916:9
    ruby#7 0x5597fe45c5cd in ruby_yyparse /home/kj/ruby/build/parse.c:11200:16
    ruby#8 0x5597fe49dc00 in yycompile0 /home/kj/ruby/build/parse.y:8121:9
    ruby#9 0x5597fe76db1b in rb_suppress_tracing /home/kj/ruby/build/../vm_trace.c:487:18
    ruby#10 0x5597fe494416 in yycompile /home/kj/ruby/build/parse.y:8177:5
    ruby#11 0x5597fe494416 in parser_compile_string /home/kj/ruby/build/parse.y:8240:12
    ruby#12 0x5597fe494416 in rb_ruby_parser_compile_string_path /home/kj/ruby/build/parse.y:8247:12
    ruby#13 0x5597fe498858 in rb_parser_compile_string_path /home/kj/ruby/build/parse.y:16663:12
    ruby#14 0x5597fe75688c in eval_make_iseq /home/kj/ruby/build/../vm_eval.c:1799:11
    ruby#15 0x5597fe70c8fa in eval_string_with_cref /home/kj/ruby/build/../vm_eval.c:1837:12
    ruby#16 0x5597fe70c396 in rb_f_eval /home/kj/ruby/build/../vm_eval.c:1912:16
    ruby#17 0x5597fe73f5e2 in vm_call_cfunc_with_frame_ /home/kj/ruby/build/../vm_insnhelper.c:3492:11
    ruby#18 0x5597fe6dca64 in vm_sendish /home/kj/ruby/build/../vm_callinfo.h
    ruby#19 0x5597fe6e64fa in vm_exec_core /home/kj/ruby/build/../insns.def:867:11
    #20 0x5597fe6dde00 in vm_exec_loop /home/kj/ruby/build/../vm.c:2578:22
    ruby#21 0x5597fe6dde00 in rb_vm_exec /home/kj/ruby/build/../vm.c:2557:18
    ruby#22 0x5597fe758bc4 in invoke_block /home/kj/ruby/build/../vm.c:1515:12
    ruby#23 0x5597fe758bc4 in invoke_iseq_block_from_c /home/kj/ruby/build/../vm.c:1585:16
    ruby#24 0x5597fe758bc4 in invoke_block_from_c_bh /home/kj/ruby/build/../vm.c:1603:20
    ruby#25 0x5597fe70e4b7 in vm_yield_with_cref /home/kj/ruby/build/../vm.c:1640:12
    ruby#26 0x5597fe709861 in vm_yield /home/kj/ruby/build/../vm.c:1648:12
    ruby#27 0x5597fe709861 in rb_yield_0 /home/kj/ruby/build/../vm_eval.c:1366:12
    ruby#28 0x5597fe709861 in rb_yield /home/kj/ruby/build/../vm_eval.c
    ruby#29 0x5597fec0eff9 in rb_ary_collect /home/kj/ruby/build/../array.c:3601:30
    ruby#30 0x5597fe73f5e2 in vm_call_cfunc_with_frame_ /home/kj/ruby/build/../vm_insnhelper.c:3492:11
    ruby#31 0x5597fe6dca64 in vm_sendish /home/kj/ruby/build/../vm_callinfo.h
    ruby#32 0x5597fe6e2d8f in vm_exec_core /home/kj/ruby/build/../insns.def:847:11
    ruby#33 0x5597fe6dde00 in vm_exec_loop /home/kj/ruby/build/../vm.c:2578:22
    ruby#34 0x5597fe6dde00 in rb_vm_exec /home/kj/ruby/build/../vm.c:2557:18
    ruby#35 0x5597fe3ffe9e in load_iseq_eval /home/kj/ruby/build/../load.c:778:5
    ruby#36 0x5597fe3fb498 in require_internal /home/kj/ruby/build/../load.c:1284:21
    ruby#37 0x5597fe3f9bf3 in rb_require_string_internal /home/kj/ruby/build/../load.c:1383:18
    ruby#38 0x5597fe73f5e2 in vm_call_cfunc_with_frame_ /home/kj/ruby/build/../vm_insnhelper.c:3492:11
    ruby#39 0x5597fe6dca64 in vm_sendish /home/kj/ruby/build/../vm_callinfo.h
    ruby#40 0x5597fe6e64fa in vm_exec_core /home/kj/ruby/build/../insns.def:867:11
    ruby#41 0x5597fe6dda82 in rb_vm_exec /home/kj/ruby/build/../vm.c:2551:22
    ruby#42 0x5597fe30a753 in rb_ec_exec_node /home/kj/ruby/build/../eval.c:283:9
    ruby#43 0x5597fe30a43d in ruby_run_node /home/kj/ruby/build/../eval.c:323:30
    ruby#44 0x5597fe3059b0 in rb_main /home/kj/ruby/build/../main.c:40:12
    ruby#45 0x5597fe3059b0 in main /home/kj/ruby/build/../main.c:59:12
    ruby#46 0x7f1a93141149 in __libc_start_call_main /usr/src/debug/glibc-2.38-16.fc39.x86_64/csu/../sysdeps/nptl/libc_start_call_main.h:58:16
    ruby#47 0x7f1a9314120a in __libc_start_main@GLIBC_2.2.5 /usr/src/debug/glibc-2.38-16.fc39.x86_64/csu/../csu/libc-start.c:360:3
    ruby#48 0x5597fe1d3e34 in _start (/home/kj/ruby/build/ruby+0x38ae34)

0x5060001ab1fc is located 0 bytes after 60-byte region [0x5060001ab1c0,0x5060001ab1fc)
allocated by thread T0 here:
    #0 0x5597fe2bde4f in malloc /home/kj/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cpp:68:3
    #1 0x5597fe3491a9 in objspace_xmalloc0 /home/kj/ruby/build/../gc.c:12605:5
    ruby#2 0x5597fe4a8adf in parser_yylex /home/kj/ruby/build/parse.y
    ruby#3 0x5597fe45c5cd in yylex /home/kj/ruby/build/parse.y:11916:9
    ruby#4 0x5597fe45c5cd in ruby_yyparse /home/kj/ruby/build/parse.c:11200:16
    ruby#5 0x5597fe49dc00 in yycompile0 /home/kj/ruby/build/parse.y:8121:9

SUMMARY: AddressSanitizer: heap-buffer-overflow /home/kj/ruby/build/../util.c:538:18 in ruby_strdup
Shadow bytes around the buggy address:
  0x5060001aaf00: fa fa fa fa fd fd fd fd fd fd fd fd fa fa fa fa
  0x5060001aaf80: 00 00 00 00 00 00 00 04 fa fa fa fa 00 00 00 00
  0x5060001ab000: 00 00 00 fa fa fa fa fa 00 00 00 00 00 00 00 fa
  0x5060001ab080: fa fa fa fa 00 00 00 00 00 00 00 00 fa fa fa fa
  0x5060001ab100: 00 00 00 00 00 00 00 00 fa fa fa fa 00 00 00 00
=>0x5060001ab180: 00 00 00 00 fa fa fa fa 00 00 00 00 00 00 00[04]
  0x5060001ab200: fa fa fa fa fd fd fd fd fd fd fd fd fa fa fa fa
  0x5060001ab280: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x5060001ab300: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x5060001ab380: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x5060001ab400: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==484771==ABORTING
KJTsanaktsidis added a commit to KJTsanaktsidis/ruby that referenced this pull request Feb 18, 2024
The string passed to openssl seems to be poisoned, and adding a guard
somewhere where the compiler can't optimize it away fixes it.

I'm honestly not 100% sure why this is nescessary in the write case;
the str str is clearly still used. But, shrug.
[3/76] OpenSSL::TestSSL#test_parallel=================================================================
==552674==ERROR: AddressSanitizer: use-after-poison on address 0x7fbba354def0 at pc 0x7fbba3694bb1 bp 0x7fbba29fe830 sp 0x7fbba29fe828
READ of size 8 at 0x7fbba354def0 thread T13
    #0 0x7fbba3694bb0 in RB_BUILTIN_TYPE /home/kj/ruby/build/ext/openssl/../../../include/ruby/internal/value_type.h:190:30
    #1 0x7fbba3694bb0 in rbimpl_RB_TYPE_P_fastpath /home/kj/ruby/build/ext/openssl/../../../include/ruby/internal/value_type.h:351:19
    ruby#2 0x7fbba3694bb0 in Check_Type /home/kj/ruby/build/ext/openssl/../../../include/ruby/internal/value_type.h:434:9
    ruby#3 0x7fbba3694bb0 in rbimpl_rstring_getmem /home/kj/ruby/build/ext/openssl/../../../include/ruby/internal/core/rstring.h:391:5
    ruby#4 0x7fbba3694bb0 in RSTRING_PTR /home/kj/ruby/build/ext/openssl/../../../include/ruby/internal/core/rstring.h:418:17
    ruby#5 0x7fbba3694bb0 in ossl_ssl_read_internal /home/kj/ruby/build/ext/openssl/../../../ext/openssl/ossl_ssl.c:1969:35
    ruby#6 0x55ea9dfe76d2 in vm_call_cfunc_with_frame_ /home/kj/ruby/build/../vm_insnhelper.c:3492:11
    ruby#7 0x55ea9df84b54 in vm_sendish /home/kj/ruby/build/../vm_callinfo.h
    ruby#8 0x55ea9df8e5ea in vm_exec_core /home/kj/ruby/build/../insns.def:867:11
    ruby#9 0x55ea9df85b72 in rb_vm_exec /home/kj/ruby/build/../vm.c:2551:22
    ruby#10 0x55ea9dffa326 in vm_call0_body /home/kj/ruby/build/../vm_eval.c:229:20
    ruby#11 0x55ea9dfad48e in vm_call0_cc /home/kj/ruby/build/../vm_eval.c:110:12
    ruby#12 0x55ea9dfadae0 in rb_vm_call0 /home/kj/ruby/build/../vm_eval.c:70:12
    ruby#13 0x55ea9dfadae0 in rb_vm_call_kw /home/kj/ruby/build/../vm_eval.c:330:12
    ruby#14 0x55ea9dd8a22c in call_method_data /home/kj/ruby/build/../proc.c:2469:12
    ruby#15 0x55ea9dd8a22c in rb_method_call_with_block_kw /home/kj/ruby/build/../proc.c:2483:12
    ruby#16 0x55ea9dfe76d2 in vm_call_cfunc_with_frame_ /home/kj/ruby/build/../vm_insnhelper.c:3492:11
    ruby#17 0x55ea9df84b54 in vm_sendish /home/kj/ruby/build/../vm_callinfo.h
    ruby#18 0x55ea9df8e5ea in vm_exec_core /home/kj/ruby/build/../insns.def:867:11
    ruby#19 0x55ea9df85b72 in rb_vm_exec /home/kj/ruby/build/../vm.c:2551:22
    #20 0x55ea9dfbfb3f in invoke_block /home/kj/ruby/build/../vm.c:1515:12
    ruby#21 0x55ea9dfbfb3f in invoke_iseq_block_from_c /home/kj/ruby/build/../vm.c:1585:16
    ruby#22 0x55ea9dfbfb3f in invoke_block_from_c_proc /home/kj/ruby/build/../vm.c:1683:16
    ruby#23 0x55ea9dfbfb3f in vm_invoke_proc /home/kj/ruby/build/../vm.c:1713:12
    ruby#24 0x55ea9dfbf4fd in rb_vm_invoke_proc /home/kj/ruby/build/../vm.c:1734:16
    ruby#25 0x55ea9df21e5b in thread_do_start_proc /home/kj/ruby/build/../thread.c:595:16
    ruby#26 0x55ea9df20163 in thread_do_start /home/kj/ruby/build/../thread.c:614:18
    ruby#27 0x55ea9df20163 in thread_start_func_2 /home/kj/ruby/build/../thread.c:668:9
    ruby#28 0x55ea9df1f62a in call_thread_start_func_2 /home/kj/ruby/build/../thread_pthread.c:2234:5
    ruby#29 0x55ea9df1f62a in nt_start /home/kj/ruby/build/../thread_pthread.c:2279:13
    ruby#30 0x55ea9daa32a4 in asan_thread_start(void*) /home/kj/llvm-project/compiler-rt/lib/asan/asan_interceptors.cpp:239:43
    ruby#31 0x7fbbbac8d896 in start_thread /usr/src/debug/glibc-2.38-16.fc39.x86_64/nptl/pthread_create.c:444:8
    ruby#32 0x7fbbbad1480b in __GI___clone3 ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Address 0x7fbba354def0 is a wild pointer inside of access range of size 0x000000000008.
SUMMARY: AddressSanitizer: use-after-poison /home/kj/ruby/build/ext/openssl/../../../include/ruby/internal/value_type.h:190:30 in RB_BUILTIN_TYPE
Shadow bytes around the buggy address:
  0x7fbba354dc00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x7fbba354dc80: 00 00 00 00 00 00 00 00 00 00 00 00 f7 00 00 00
  0x7fbba354dd00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x7fbba354dd80: 00 00 00 00 f7 00 00 00 00 00 00 00 00 00 00 00
  0x7fbba354de00: 00 00 00 00 00 00 f7 00 00 00 00 00 f7 00 00 00
=>0x7fbba354de80: 00 00 00 00 00 00 00 00 f7 00 00 00 00 00[f7]00
  0x7fbba354df00: 00 00 00 00 00 00 00 00 00 00 f7 00 00 00 00 00
  0x7fbba354df80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x7fbba354e000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x7fbba354e080: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x7fbba354e100: 00 00 00 00 00 00 f7 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
Thread T13 created by T11 here:
    #0 0x55ea9db5d8fd in pthread_create /home/kj/llvm-project/compiler-rt/lib/asan/asan_interceptors.cpp:250:3
    #1 0x55ea9df037b8 in native_thread_create0 /home/kj/ruby/build/../thread_pthread.c:2151:11
    ruby#2 0x55ea9df037b8 in native_thread_create_dedicated /home/kj/ruby/build/../thread_pthread.c:2218:12
    ruby#3 0x55ea9df037b8 in native_thread_create /home/kj/ruby/build/../thread_pthread.c:2392:16
    ruby#4 0x55ea9df037b8 in thread_create_core /home/kj/ruby/build/../thread.c:847:11
    ruby#5 0x55ea9df15a39 in thread_initialize /home/kj/ruby/build/../thread.c:955:16
    ruby#6 0x55ea9dfad48e in vm_call0_cc /home/kj/ruby/build/../vm_eval.c:110:12
    ruby#7 0x55ea9dffd34e in rb_call0 /home/kj/ruby/build/../vm_eval.c:573:12
    ruby#8 0x55ea9dfb0362 in rb_call /home/kj/ruby/build/../vm_eval.c:899:12
    ruby#9 0x55ea9dfb0362 in rb_funcallv_kw /home/kj/ruby/build/../vm_eval.c:1092:12

Thread T11 created by T0 here:
    #0 0x55ea9db5d8fd in pthread_create /home/kj/llvm-project/compiler-rt/lib/asan/asan_interceptors.cpp:250:3
    #1 0x55ea9df037b8 in native_thread_create0 /home/kj/ruby/build/../thread_pthread.c:2151:11
    ruby#2 0x55ea9df037b8 in native_thread_create_dedicated /home/kj/ruby/build/../thread_pthread.c:2218:12
    ruby#3 0x55ea9df037b8 in native_thread_create /home/kj/ruby/build/../thread_pthread.c:2392:16
    ruby#4 0x55ea9df037b8 in thread_create_core /home/kj/ruby/build/../thread.c:847:11
    ruby#5 0x55ea9df15a39 in thread_initialize /home/kj/ruby/build/../thread.c:955:16
    ruby#6 0x55ea9dfad48e in vm_call0_cc /home/kj/ruby/build/../vm_eval.c:110:12
    ruby#7 0x55ea9dffd34e in rb_call0 /home/kj/ruby/build/../vm_eval.c:573:12
    ruby#8 0x55ea9dfb0362 in rb_call /home/kj/ruby/build/../vm_eval.c:899:12
    ruby#9 0x55ea9dfb0362 in rb_funcallv_kw /home/kj/ruby/build/../vm_eval.c:1092:12

==552674==ABORTING
KJTsanaktsidis added a commit to KJTsanaktsidis/ruby that referenced this pull request Feb 28, 2024
It appears that tok(p) is not NULL terminated here, so we need to use
strndup to copy only the correct number of bytes.

[1/1] TestRubyLiteral#test_integer=================================================================
==484771==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x5060001ab1fc at pc 0x5597fe21d8e1 bp 0x7ffdc6fb0a50 sp 0x7ffdc6fb0210
READ of size 61 at 0x5060001ab1fc thread T0
    #0 0x5597fe21d8e0 in strlen.part.0 /home/kj/llvm-project/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:391:5
    #1 0x5597fe6b2feb in ruby_strdup /home/kj/ruby/build/../util.c:538:18
    ruby#2 0x5597fe4cb1c5 in set_number_literal /home/kj/ruby/build/parse.y:9694:9
    ruby#3 0x5597fe4cab3d in no_digits /home/kj/ruby/build/parse.y:10409:12
    ruby#4 0x5597fe4b9de9 in parse_numeric /home/kj/ruby/build/parse.y
    ruby#5 0x5597fe4a8adf in parser_yylex /home/kj/ruby/build/parse.y
    ruby#6 0x5597fe45c5cd in yylex /home/kj/ruby/build/parse.y:11916:9
    ruby#7 0x5597fe45c5cd in ruby_yyparse /home/kj/ruby/build/parse.c:11200:16
    ruby#8 0x5597fe49dc00 in yycompile0 /home/kj/ruby/build/parse.y:8121:9
    ruby#9 0x5597fe76db1b in rb_suppress_tracing /home/kj/ruby/build/../vm_trace.c:487:18
    ruby#10 0x5597fe494416 in yycompile /home/kj/ruby/build/parse.y:8177:5
    ruby#11 0x5597fe494416 in parser_compile_string /home/kj/ruby/build/parse.y:8240:12
    ruby#12 0x5597fe494416 in rb_ruby_parser_compile_string_path /home/kj/ruby/build/parse.y:8247:12
    ruby#13 0x5597fe498858 in rb_parser_compile_string_path /home/kj/ruby/build/parse.y:16663:12
    ruby#14 0x5597fe75688c in eval_make_iseq /home/kj/ruby/build/../vm_eval.c:1799:11
    ruby#15 0x5597fe70c8fa in eval_string_with_cref /home/kj/ruby/build/../vm_eval.c:1837:12
    ruby#16 0x5597fe70c396 in rb_f_eval /home/kj/ruby/build/../vm_eval.c:1912:16
    ruby#17 0x5597fe73f5e2 in vm_call_cfunc_with_frame_ /home/kj/ruby/build/../vm_insnhelper.c:3492:11
    ruby#18 0x5597fe6dca64 in vm_sendish /home/kj/ruby/build/../vm_callinfo.h
    ruby#19 0x5597fe6e64fa in vm_exec_core /home/kj/ruby/build/../insns.def:867:11
    #20 0x5597fe6dde00 in vm_exec_loop /home/kj/ruby/build/../vm.c:2578:22
    ruby#21 0x5597fe6dde00 in rb_vm_exec /home/kj/ruby/build/../vm.c:2557:18
    ruby#22 0x5597fe758bc4 in invoke_block /home/kj/ruby/build/../vm.c:1515:12
    ruby#23 0x5597fe758bc4 in invoke_iseq_block_from_c /home/kj/ruby/build/../vm.c:1585:16
    ruby#24 0x5597fe758bc4 in invoke_block_from_c_bh /home/kj/ruby/build/../vm.c:1603:20
    ruby#25 0x5597fe70e4b7 in vm_yield_with_cref /home/kj/ruby/build/../vm.c:1640:12
    ruby#26 0x5597fe709861 in vm_yield /home/kj/ruby/build/../vm.c:1648:12
    ruby#27 0x5597fe709861 in rb_yield_0 /home/kj/ruby/build/../vm_eval.c:1366:12
    ruby#28 0x5597fe709861 in rb_yield /home/kj/ruby/build/../vm_eval.c
    ruby#29 0x5597fec0eff9 in rb_ary_collect /home/kj/ruby/build/../array.c:3601:30
    ruby#30 0x5597fe73f5e2 in vm_call_cfunc_with_frame_ /home/kj/ruby/build/../vm_insnhelper.c:3492:11
    ruby#31 0x5597fe6dca64 in vm_sendish /home/kj/ruby/build/../vm_callinfo.h
    ruby#32 0x5597fe6e2d8f in vm_exec_core /home/kj/ruby/build/../insns.def:847:11
    ruby#33 0x5597fe6dde00 in vm_exec_loop /home/kj/ruby/build/../vm.c:2578:22
    ruby#34 0x5597fe6dde00 in rb_vm_exec /home/kj/ruby/build/../vm.c:2557:18
    ruby#35 0x5597fe3ffe9e in load_iseq_eval /home/kj/ruby/build/../load.c:778:5
    ruby#36 0x5597fe3fb498 in require_internal /home/kj/ruby/build/../load.c:1284:21
    ruby#37 0x5597fe3f9bf3 in rb_require_string_internal /home/kj/ruby/build/../load.c:1383:18
    ruby#38 0x5597fe73f5e2 in vm_call_cfunc_with_frame_ /home/kj/ruby/build/../vm_insnhelper.c:3492:11
    ruby#39 0x5597fe6dca64 in vm_sendish /home/kj/ruby/build/../vm_callinfo.h
    ruby#40 0x5597fe6e64fa in vm_exec_core /home/kj/ruby/build/../insns.def:867:11
    ruby#41 0x5597fe6dda82 in rb_vm_exec /home/kj/ruby/build/../vm.c:2551:22
    ruby#42 0x5597fe30a753 in rb_ec_exec_node /home/kj/ruby/build/../eval.c:283:9
    ruby#43 0x5597fe30a43d in ruby_run_node /home/kj/ruby/build/../eval.c:323:30
    ruby#44 0x5597fe3059b0 in rb_main /home/kj/ruby/build/../main.c:40:12
    ruby#45 0x5597fe3059b0 in main /home/kj/ruby/build/../main.c:59:12
    ruby#46 0x7f1a93141149 in __libc_start_call_main /usr/src/debug/glibc-2.38-16.fc39.x86_64/csu/../sysdeps/nptl/libc_start_call_main.h:58:16
    ruby#47 0x7f1a9314120a in __libc_start_main@GLIBC_2.2.5 /usr/src/debug/glibc-2.38-16.fc39.x86_64/csu/../csu/libc-start.c:360:3
    ruby#48 0x5597fe1d3e34 in _start (/home/kj/ruby/build/ruby+0x38ae34)

0x5060001ab1fc is located 0 bytes after 60-byte region [0x5060001ab1c0,0x5060001ab1fc)
allocated by thread T0 here:
    #0 0x5597fe2bde4f in malloc /home/kj/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cpp:68:3
    #1 0x5597fe3491a9 in objspace_xmalloc0 /home/kj/ruby/build/../gc.c:12605:5
    ruby#2 0x5597fe4a8adf in parser_yylex /home/kj/ruby/build/parse.y
    ruby#3 0x5597fe45c5cd in yylex /home/kj/ruby/build/parse.y:11916:9
    ruby#4 0x5597fe45c5cd in ruby_yyparse /home/kj/ruby/build/parse.c:11200:16
    ruby#5 0x5597fe49dc00 in yycompile0 /home/kj/ruby/build/parse.y:8121:9

SUMMARY: AddressSanitizer: heap-buffer-overflow /home/kj/ruby/build/../util.c:538:18 in ruby_strdup
Shadow bytes around the buggy address:
  0x5060001aaf00: fa fa fa fa fd fd fd fd fd fd fd fd fa fa fa fa
  0x5060001aaf80: 00 00 00 00 00 00 00 04 fa fa fa fa 00 00 00 00
  0x5060001ab000: 00 00 00 fa fa fa fa fa 00 00 00 00 00 00 00 fa
  0x5060001ab080: fa fa fa fa 00 00 00 00 00 00 00 00 fa fa fa fa
  0x5060001ab100: 00 00 00 00 00 00 00 00 fa fa fa fa 00 00 00 00
=>0x5060001ab180: 00 00 00 00 fa fa fa fa 00 00 00 00 00 00 00[04]
  0x5060001ab200: fa fa fa fa fd fd fd fd fd fd fd fd fa fa fa fa
  0x5060001ab280: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x5060001ab300: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x5060001ab380: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x5060001ab400: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==484771==ABORTING
KJTsanaktsidis added a commit to KJTsanaktsidis/ruby that referenced this pull request Mar 28, 2024
It appears that tok(p) is not NULL terminated here, so we need to use
strndup to copy only the correct number of bytes.

[1/1] TestRubyLiteral#test_integer=================================================================
==484771==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x5060001ab1fc at pc 0x5597fe21d8e1 bp 0x7ffdc6fb0a50 sp 0x7ffdc6fb0210
READ of size 61 at 0x5060001ab1fc thread T0
    #0 0x5597fe21d8e0 in strlen.part.0 /home/kj/llvm-project/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:391:5
    #1 0x5597fe6b2feb in ruby_strdup /home/kj/ruby/build/../util.c:538:18
    ruby#2 0x5597fe4cb1c5 in set_number_literal /home/kj/ruby/build/parse.y:9694:9
    ruby#3 0x5597fe4cab3d in no_digits /home/kj/ruby/build/parse.y:10409:12
    ruby#4 0x5597fe4b9de9 in parse_numeric /home/kj/ruby/build/parse.y
    ruby#5 0x5597fe4a8adf in parser_yylex /home/kj/ruby/build/parse.y
    ruby#6 0x5597fe45c5cd in yylex /home/kj/ruby/build/parse.y:11916:9
    ruby#7 0x5597fe45c5cd in ruby_yyparse /home/kj/ruby/build/parse.c:11200:16
    ruby#8 0x5597fe49dc00 in yycompile0 /home/kj/ruby/build/parse.y:8121:9
    ruby#9 0x5597fe76db1b in rb_suppress_tracing /home/kj/ruby/build/../vm_trace.c:487:18
    ruby#10 0x5597fe494416 in yycompile /home/kj/ruby/build/parse.y:8177:5
    ruby#11 0x5597fe494416 in parser_compile_string /home/kj/ruby/build/parse.y:8240:12
    ruby#12 0x5597fe494416 in rb_ruby_parser_compile_string_path /home/kj/ruby/build/parse.y:8247:12
    ruby#13 0x5597fe498858 in rb_parser_compile_string_path /home/kj/ruby/build/parse.y:16663:12
    ruby#14 0x5597fe75688c in eval_make_iseq /home/kj/ruby/build/../vm_eval.c:1799:11
    ruby#15 0x5597fe70c8fa in eval_string_with_cref /home/kj/ruby/build/../vm_eval.c:1837:12
    ruby#16 0x5597fe70c396 in rb_f_eval /home/kj/ruby/build/../vm_eval.c:1912:16
    ruby#17 0x5597fe73f5e2 in vm_call_cfunc_with_frame_ /home/kj/ruby/build/../vm_insnhelper.c:3492:11
    ruby#18 0x5597fe6dca64 in vm_sendish /home/kj/ruby/build/../vm_callinfo.h
    ruby#19 0x5597fe6e64fa in vm_exec_core /home/kj/ruby/build/../insns.def:867:11
    #20 0x5597fe6dde00 in vm_exec_loop /home/kj/ruby/build/../vm.c:2578:22
    ruby#21 0x5597fe6dde00 in rb_vm_exec /home/kj/ruby/build/../vm.c:2557:18
    ruby#22 0x5597fe758bc4 in invoke_block /home/kj/ruby/build/../vm.c:1515:12
    ruby#23 0x5597fe758bc4 in invoke_iseq_block_from_c /home/kj/ruby/build/../vm.c:1585:16
    ruby#24 0x5597fe758bc4 in invoke_block_from_c_bh /home/kj/ruby/build/../vm.c:1603:20
    ruby#25 0x5597fe70e4b7 in vm_yield_with_cref /home/kj/ruby/build/../vm.c:1640:12
    ruby#26 0x5597fe709861 in vm_yield /home/kj/ruby/build/../vm.c:1648:12
    ruby#27 0x5597fe709861 in rb_yield_0 /home/kj/ruby/build/../vm_eval.c:1366:12
    ruby#28 0x5597fe709861 in rb_yield /home/kj/ruby/build/../vm_eval.c
    ruby#29 0x5597fec0eff9 in rb_ary_collect /home/kj/ruby/build/../array.c:3601:30
    ruby#30 0x5597fe73f5e2 in vm_call_cfunc_with_frame_ /home/kj/ruby/build/../vm_insnhelper.c:3492:11
    ruby#31 0x5597fe6dca64 in vm_sendish /home/kj/ruby/build/../vm_callinfo.h
    ruby#32 0x5597fe6e2d8f in vm_exec_core /home/kj/ruby/build/../insns.def:847:11
    ruby#33 0x5597fe6dde00 in vm_exec_loop /home/kj/ruby/build/../vm.c:2578:22
    ruby#34 0x5597fe6dde00 in rb_vm_exec /home/kj/ruby/build/../vm.c:2557:18
    ruby#35 0x5597fe3ffe9e in load_iseq_eval /home/kj/ruby/build/../load.c:778:5
    ruby#36 0x5597fe3fb498 in require_internal /home/kj/ruby/build/../load.c:1284:21
    ruby#37 0x5597fe3f9bf3 in rb_require_string_internal /home/kj/ruby/build/../load.c:1383:18
    ruby#38 0x5597fe73f5e2 in vm_call_cfunc_with_frame_ /home/kj/ruby/build/../vm_insnhelper.c:3492:11
    ruby#39 0x5597fe6dca64 in vm_sendish /home/kj/ruby/build/../vm_callinfo.h
    ruby#40 0x5597fe6e64fa in vm_exec_core /home/kj/ruby/build/../insns.def:867:11
    ruby#41 0x5597fe6dda82 in rb_vm_exec /home/kj/ruby/build/../vm.c:2551:22
    ruby#42 0x5597fe30a753 in rb_ec_exec_node /home/kj/ruby/build/../eval.c:283:9
    ruby#43 0x5597fe30a43d in ruby_run_node /home/kj/ruby/build/../eval.c:323:30
    ruby#44 0x5597fe3059b0 in rb_main /home/kj/ruby/build/../main.c:40:12
    ruby#45 0x5597fe3059b0 in main /home/kj/ruby/build/../main.c:59:12
    ruby#46 0x7f1a93141149 in __libc_start_call_main /usr/src/debug/glibc-2.38-16.fc39.x86_64/csu/../sysdeps/nptl/libc_start_call_main.h:58:16
    ruby#47 0x7f1a9314120a in __libc_start_main@GLIBC_2.2.5 /usr/src/debug/glibc-2.38-16.fc39.x86_64/csu/../csu/libc-start.c:360:3
    ruby#48 0x5597fe1d3e34 in _start (/home/kj/ruby/build/ruby+0x38ae34)

0x5060001ab1fc is located 0 bytes after 60-byte region [0x5060001ab1c0,0x5060001ab1fc)
allocated by thread T0 here:
    #0 0x5597fe2bde4f in malloc /home/kj/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cpp:68:3
    #1 0x5597fe3491a9 in objspace_xmalloc0 /home/kj/ruby/build/../gc.c:12605:5
    ruby#2 0x5597fe4a8adf in parser_yylex /home/kj/ruby/build/parse.y
    ruby#3 0x5597fe45c5cd in yylex /home/kj/ruby/build/parse.y:11916:9
    ruby#4 0x5597fe45c5cd in ruby_yyparse /home/kj/ruby/build/parse.c:11200:16
    ruby#5 0x5597fe49dc00 in yycompile0 /home/kj/ruby/build/parse.y:8121:9

SUMMARY: AddressSanitizer: heap-buffer-overflow /home/kj/ruby/build/../util.c:538:18 in ruby_strdup
Shadow bytes around the buggy address:
  0x5060001aaf00: fa fa fa fa fd fd fd fd fd fd fd fd fa fa fa fa
  0x5060001aaf80: 00 00 00 00 00 00 00 04 fa fa fa fa 00 00 00 00
  0x5060001ab000: 00 00 00 fa fa fa fa fa 00 00 00 00 00 00 00 fa
  0x5060001ab080: fa fa fa fa 00 00 00 00 00 00 00 00 fa fa fa fa
  0x5060001ab100: 00 00 00 00 00 00 00 00 fa fa fa fa 00 00 00 00
=>0x5060001ab180: 00 00 00 00 fa fa fa fa 00 00 00 00 00 00 00[04]
  0x5060001ab200: fa fa fa fa fd fd fd fd fd fd fd fd fa fa fa fa
  0x5060001ab280: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x5060001ab300: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x5060001ab380: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x5060001ab400: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==484771==ABORTING

[Bug #20398]
KJTsanaktsidis added a commit to KJTsanaktsidis/ruby that referenced this pull request Mar 28, 2024
It appears that tok(p) is not NULL terminated here, so we need to use
strndup to copy only the correct number of bytes.

[1/1] TestRubyLiteral#test_integer=================================================================
==484771==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x5060001ab1fc at pc 0x5597fe21d8e1 bp 0x7ffdc6fb0a50 sp 0x7ffdc6fb0210
READ of size 61 at 0x5060001ab1fc thread T0
    #0 0x5597fe21d8e0 in strlen.part.0 /home/kj/llvm-project/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:391:5
    #1 0x5597fe6b2feb in ruby_strdup /home/kj/ruby/build/../util.c:538:18
    ruby#2 0x5597fe4cb1c5 in set_number_literal /home/kj/ruby/build/parse.y:9694:9
    ruby#3 0x5597fe4cab3d in no_digits /home/kj/ruby/build/parse.y:10409:12
    ruby#4 0x5597fe4b9de9 in parse_numeric /home/kj/ruby/build/parse.y
    ruby#5 0x5597fe4a8adf in parser_yylex /home/kj/ruby/build/parse.y
    ruby#6 0x5597fe45c5cd in yylex /home/kj/ruby/build/parse.y:11916:9
    ruby#7 0x5597fe45c5cd in ruby_yyparse /home/kj/ruby/build/parse.c:11200:16
    ruby#8 0x5597fe49dc00 in yycompile0 /home/kj/ruby/build/parse.y:8121:9
    ruby#9 0x5597fe76db1b in rb_suppress_tracing /home/kj/ruby/build/../vm_trace.c:487:18
    ruby#10 0x5597fe494416 in yycompile /home/kj/ruby/build/parse.y:8177:5
    ruby#11 0x5597fe494416 in parser_compile_string /home/kj/ruby/build/parse.y:8240:12
    ruby#12 0x5597fe494416 in rb_ruby_parser_compile_string_path /home/kj/ruby/build/parse.y:8247:12
    ruby#13 0x5597fe498858 in rb_parser_compile_string_path /home/kj/ruby/build/parse.y:16663:12
    ruby#14 0x5597fe75688c in eval_make_iseq /home/kj/ruby/build/../vm_eval.c:1799:11
    ruby#15 0x5597fe70c8fa in eval_string_with_cref /home/kj/ruby/build/../vm_eval.c:1837:12
    ruby#16 0x5597fe70c396 in rb_f_eval /home/kj/ruby/build/../vm_eval.c:1912:16
    ruby#17 0x5597fe73f5e2 in vm_call_cfunc_with_frame_ /home/kj/ruby/build/../vm_insnhelper.c:3492:11
    ruby#18 0x5597fe6dca64 in vm_sendish /home/kj/ruby/build/../vm_callinfo.h
    ruby#19 0x5597fe6e64fa in vm_exec_core /home/kj/ruby/build/../insns.def:867:11
    #20 0x5597fe6dde00 in vm_exec_loop /home/kj/ruby/build/../vm.c:2578:22
    ruby#21 0x5597fe6dde00 in rb_vm_exec /home/kj/ruby/build/../vm.c:2557:18
    ruby#22 0x5597fe758bc4 in invoke_block /home/kj/ruby/build/../vm.c:1515:12
    ruby#23 0x5597fe758bc4 in invoke_iseq_block_from_c /home/kj/ruby/build/../vm.c:1585:16
    ruby#24 0x5597fe758bc4 in invoke_block_from_c_bh /home/kj/ruby/build/../vm.c:1603:20
    ruby#25 0x5597fe70e4b7 in vm_yield_with_cref /home/kj/ruby/build/../vm.c:1640:12
    ruby#26 0x5597fe709861 in vm_yield /home/kj/ruby/build/../vm.c:1648:12
    ruby#27 0x5597fe709861 in rb_yield_0 /home/kj/ruby/build/../vm_eval.c:1366:12
    ruby#28 0x5597fe709861 in rb_yield /home/kj/ruby/build/../vm_eval.c
    ruby#29 0x5597fec0eff9 in rb_ary_collect /home/kj/ruby/build/../array.c:3601:30
    ruby#30 0x5597fe73f5e2 in vm_call_cfunc_with_frame_ /home/kj/ruby/build/../vm_insnhelper.c:3492:11
    ruby#31 0x5597fe6dca64 in vm_sendish /home/kj/ruby/build/../vm_callinfo.h
    ruby#32 0x5597fe6e2d8f in vm_exec_core /home/kj/ruby/build/../insns.def:847:11
    ruby#33 0x5597fe6dde00 in vm_exec_loop /home/kj/ruby/build/../vm.c:2578:22
    ruby#34 0x5597fe6dde00 in rb_vm_exec /home/kj/ruby/build/../vm.c:2557:18
    ruby#35 0x5597fe3ffe9e in load_iseq_eval /home/kj/ruby/build/../load.c:778:5
    ruby#36 0x5597fe3fb498 in require_internal /home/kj/ruby/build/../load.c:1284:21
    ruby#37 0x5597fe3f9bf3 in rb_require_string_internal /home/kj/ruby/build/../load.c:1383:18
    ruby#38 0x5597fe73f5e2 in vm_call_cfunc_with_frame_ /home/kj/ruby/build/../vm_insnhelper.c:3492:11
    ruby#39 0x5597fe6dca64 in vm_sendish /home/kj/ruby/build/../vm_callinfo.h
    ruby#40 0x5597fe6e64fa in vm_exec_core /home/kj/ruby/build/../insns.def:867:11
    ruby#41 0x5597fe6dda82 in rb_vm_exec /home/kj/ruby/build/../vm.c:2551:22
    ruby#42 0x5597fe30a753 in rb_ec_exec_node /home/kj/ruby/build/../eval.c:283:9
    ruby#43 0x5597fe30a43d in ruby_run_node /home/kj/ruby/build/../eval.c:323:30
    ruby#44 0x5597fe3059b0 in rb_main /home/kj/ruby/build/../main.c:40:12
    ruby#45 0x5597fe3059b0 in main /home/kj/ruby/build/../main.c:59:12
    ruby#46 0x7f1a93141149 in __libc_start_call_main /usr/src/debug/glibc-2.38-16.fc39.x86_64/csu/../sysdeps/nptl/libc_start_call_main.h:58:16
    ruby#47 0x7f1a9314120a in __libc_start_main@GLIBC_2.2.5 /usr/src/debug/glibc-2.38-16.fc39.x86_64/csu/../csu/libc-start.c:360:3
    ruby#48 0x5597fe1d3e34 in _start (/home/kj/ruby/build/ruby+0x38ae34)

0x5060001ab1fc is located 0 bytes after 60-byte region [0x5060001ab1c0,0x5060001ab1fc)
allocated by thread T0 here:
    #0 0x5597fe2bde4f in malloc /home/kj/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cpp:68:3
    #1 0x5597fe3491a9 in objspace_xmalloc0 /home/kj/ruby/build/../gc.c:12605:5
    ruby#2 0x5597fe4a8adf in parser_yylex /home/kj/ruby/build/parse.y
    ruby#3 0x5597fe45c5cd in yylex /home/kj/ruby/build/parse.y:11916:9
    ruby#4 0x5597fe45c5cd in ruby_yyparse /home/kj/ruby/build/parse.c:11200:16
    ruby#5 0x5597fe49dc00 in yycompile0 /home/kj/ruby/build/parse.y:8121:9

SUMMARY: AddressSanitizer: heap-buffer-overflow /home/kj/ruby/build/../util.c:538:18 in ruby_strdup
Shadow bytes around the buggy address:
  0x5060001aaf00: fa fa fa fa fd fd fd fd fd fd fd fd fa fa fa fa
  0x5060001aaf80: 00 00 00 00 00 00 00 04 fa fa fa fa 00 00 00 00
  0x5060001ab000: 00 00 00 fa fa fa fa fa 00 00 00 00 00 00 00 fa
  0x5060001ab080: fa fa fa fa 00 00 00 00 00 00 00 00 fa fa fa fa
  0x5060001ab100: 00 00 00 00 00 00 00 00 fa fa fa fa 00 00 00 00
=>0x5060001ab180: 00 00 00 00 fa fa fa fa 00 00 00 00 00 00 00[04]
  0x5060001ab200: fa fa fa fa fd fd fd fd fd fd fd fd fa fa fa fa
  0x5060001ab280: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x5060001ab300: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x5060001ab380: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x5060001ab400: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==484771==ABORTING

[Bug #20398]
peterzhu2118 added a commit to peterzhu2118/ruby that referenced this pull request Nov 28, 2024
[Bug #20921]

When we create a cache entry for a constant, the following sequence of
events could happen:

- vm_track_constant_cache is called to insert a constant cache.
- In vm_track_constant_cache, we first look up the ST table for the ID
  of the constant. Assume the ST table exists because another iseq also
  holds a cache entry for this ID.
- We then insert into this ST table with the iseq_inline_constant_cache.
- However, while inserting into this ST table, it allocates memory, which
  could trigger a GC. Assume that it does trigger a GC.
- The GC frees the one and only other iseq that holds a cache entry for
  this ID.
- In remove_from_constant_cache, it will appear that the ST table is now
  empty because there are no more iseq with cache entries for this ID, so
  we free the ST table.
- We complete GC and continue our st_insert. However, this ST table has
  been freed so we now have a use-after-free.

This issue is very hard to reproduce, because it requires that the GC runs
at a very specific time. However, we can make it show up by applying this
patch which runs GC right before the st_insert to mimic the st_insert
triggering a GC:

    diff --git a/vm_insnhelper.c b/vm_insnhelper.c
    index 3cb23f06f0..a93998136a 100644
    --- a/vm_insnhelper.c
    +++ b/vm_insnhelper.c
    @@ -6338,6 +6338,10 @@ vm_track_constant_cache(ID id, void *ic)
            rb_id_table_insert(const_cache, id, (VALUE)ics);
        }

    +    if (id == rb_intern("MyConstant")) rb_gc();
    +
        st_insert(ics, (st_data_t) ic, (st_data_t) Qtrue);
    }

And if we run this script:

    Object.const_set("MyConstant", "Hello!")

    my_proc = eval("-> { MyConstant }")
    my_proc.call

    my_proc = eval("-> { MyConstant }")
    my_proc.call

We can see that ASAN outputs a use-after-free error:

    ==36540==ERROR: AddressSanitizer: heap-use-after-free on address 0x606000049528 at pc 0x000102f3ceac bp 0x00016d607a70 sp 0x00016d607a68
    READ of size 8 at 0x606000049528 thread T0
        #0 0x102f3cea8 in do_hash st.c:321
        #1 0x102f3ddd0 in rb_st_insert st.c:1132
        ruby#2 0x103140700 in vm_track_constant_cache vm_insnhelper.c:6345
        ruby#3 0x1030b91d8 in vm_ic_track_const_chain vm_insnhelper.c:6356
        ruby#4 0x1030b8cf8 in rb_vm_opt_getconstant_path vm_insnhelper.c:6424
        ruby#5 0x1030bc1e0 in vm_exec_core insns.def:263
        ruby#6 0x1030b55fc in rb_vm_exec vm.c:2585
        ruby#7 0x1030fe0ac in rb_iseq_eval_main vm.c:2851
        ruby#8 0x102a82588 in rb_ec_exec_node eval.c:281
        ruby#9 0x102a81fe0 in ruby_run_node eval.c:319
        ruby#10 0x1027f3db4 in rb_main main.c:43
        ruby#11 0x1027f3bd4 in main main.c:68
        ruby#12 0x183900270  (<unknown module>)

    0x606000049528 is located 8 bytes inside of 56-byte region [0x606000049520,0x606000049558)
    freed by thread T0 here:
        #0 0x104174d40 in free+0x98 (libclang_rt.asan_osx_dynamic.dylib:arm64e+0x54d40)
        #1 0x102ada89c in rb_gc_impl_free default.c:8183
        ruby#2 0x102ada7dc in ruby_sized_xfree gc.c:4507
        ruby#3 0x102ac4d34 in ruby_xfree gc.c:4518
        ruby#4 0x102f3cb34 in rb_st_free_table st.c:663
        ruby#5 0x102bd52d8 in remove_from_constant_cache iseq.c:119
        ruby#6 0x102bbe2cc in iseq_clear_ic_references iseq.c:153
        ruby#7 0x102bbd2a0 in rb_iseq_free iseq.c:166
        ruby#8 0x102b32ed0 in rb_imemo_free imemo.c:564
        ruby#9 0x102ac4b44 in rb_gc_obj_free gc.c:1407
        ruby#10 0x102af4290 in gc_sweep_plane default.c:3546
        ruby#11 0x102af3bdc in gc_sweep_page default.c:3634
        ruby#12 0x102aeb140 in gc_sweep_step default.c:3906
        ruby#13 0x102aeadf0 in gc_sweep_rest default.c:3978
        ruby#14 0x102ae4714 in gc_sweep default.c:4155
        ruby#15 0x102af8474 in gc_start default.c:6484
        ruby#16 0x102afbe30 in garbage_collect default.c:6363
        ruby#17 0x102ad37f0 in rb_gc_impl_start default.c:6816
        ruby#18 0x102ad3634 in rb_gc gc.c:3624
        ruby#19 0x1031406ec in vm_track_constant_cache vm_insnhelper.c:6342
        #20 0x1030b91d8 in vm_ic_track_const_chain vm_insnhelper.c:6356
        ruby#21 0x1030b8cf8 in rb_vm_opt_getconstant_path vm_insnhelper.c:6424
        ruby#22 0x1030bc1e0 in vm_exec_core insns.def:263
        ruby#23 0x1030b55fc in rb_vm_exec vm.c:2585
        ruby#24 0x1030fe0ac in rb_iseq_eval_main vm.c:2851
        ruby#25 0x102a82588 in rb_ec_exec_node eval.c:281
        ruby#26 0x102a81fe0 in ruby_run_node eval.c:319
        ruby#27 0x1027f3db4 in rb_main main.c:43
        ruby#28 0x1027f3bd4 in main main.c:68
        ruby#29 0x183900270  (<unknown module>)

    previously allocated by thread T0 here:
        #0 0x104174c04 in malloc+0x94 (libclang_rt.asan_osx_dynamic.dylib:arm64e+0x54c04)
        #1 0x102ada0ec in rb_gc_impl_malloc default.c:8198
        ruby#2 0x102acee44 in ruby_xmalloc gc.c:4438
        ruby#3 0x102f3c85c in rb_st_init_table_with_size st.c:571
        ruby#4 0x102f3c900 in rb_st_init_table st.c:600
        ruby#5 0x102f3c920 in rb_st_init_numtable st.c:608
        ruby#6 0x103140698 in vm_track_constant_cache vm_insnhelper.c:6337
        ruby#7 0x1030b91d8 in vm_ic_track_const_chain vm_insnhelper.c:6356
        ruby#8 0x1030b8cf8 in rb_vm_opt_getconstant_path vm_insnhelper.c:6424
        ruby#9 0x1030bc1e0 in vm_exec_core insns.def:263
        ruby#10 0x1030b55fc in rb_vm_exec vm.c:2585
        ruby#11 0x1030fe0ac in rb_iseq_eval_main vm.c:2851
        ruby#12 0x102a82588 in rb_ec_exec_node eval.c:281
        ruby#13 0x102a81fe0 in ruby_run_node eval.c:319
        ruby#14 0x1027f3db4 in rb_main main.c:43
        ruby#15 0x1027f3bd4 in main main.c:68
        ruby#16 0x183900270  (<unknown module>)

This commit fixes this bug by adding a inserting_constant_cache_id field
to the VM, which stores the ID that is currently being inserted and, in
remove_from_constant_cache, we don't free the ST table for ID equal to
this one.

Co-Authored-By: Alan Wu <alanwu@ruby-lang.org>
peterzhu2118 added a commit to peterzhu2118/ruby that referenced this pull request Nov 28, 2024
[Bug #20921]

When we create a cache entry for a constant, the following sequence of
events could happen:

- vm_track_constant_cache is called to insert a constant cache.
- In vm_track_constant_cache, we first look up the ST table for the ID
  of the constant. Assume the ST table exists because another iseq also
  holds a cache entry for this ID.
- We then insert into this ST table with the iseq_inline_constant_cache.
- However, while inserting into this ST table, it allocates memory, which
  could trigger a GC. Assume that it does trigger a GC.
- The GC frees the one and only other iseq that holds a cache entry for
  this ID.
- In remove_from_constant_cache, it will appear that the ST table is now
  empty because there are no more iseq with cache entries for this ID, so
  we free the ST table.
- We complete GC and continue our st_insert. However, this ST table has
  been freed so we now have a use-after-free.

This issue is very hard to reproduce, because it requires that the GC runs
at a very specific time. However, we can make it show up by applying this
patch which runs GC right before the st_insert to mimic the st_insert
triggering a GC:

    diff --git a/vm_insnhelper.c b/vm_insnhelper.c
    index 3cb23f06f0..a93998136a 100644
    --- a/vm_insnhelper.c
    +++ b/vm_insnhelper.c
    @@ -6338,6 +6338,10 @@ vm_track_constant_cache(ID id, void *ic)
            rb_id_table_insert(const_cache, id, (VALUE)ics);
        }

    +    if (id == rb_intern("MyConstant")) rb_gc();
    +
        st_insert(ics, (st_data_t) ic, (st_data_t) Qtrue);
    }

And if we run this script:

    Object.const_set("MyConstant", "Hello!")

    my_proc = eval("-> { MyConstant }")
    my_proc.call

    my_proc = eval("-> { MyConstant }")
    my_proc.call

We can see that ASAN outputs a use-after-free error:

    ==36540==ERROR: AddressSanitizer: heap-use-after-free on address 0x606000049528 at pc 0x000102f3ceac bp 0x00016d607a70 sp 0x00016d607a68
    READ of size 8 at 0x606000049528 thread T0
        #0 0x102f3cea8 in do_hash st.c:321
        #1 0x102f3ddd0 in rb_st_insert st.c:1132
        ruby#2 0x103140700 in vm_track_constant_cache vm_insnhelper.c:6345
        ruby#3 0x1030b91d8 in vm_ic_track_const_chain vm_insnhelper.c:6356
        ruby#4 0x1030b8cf8 in rb_vm_opt_getconstant_path vm_insnhelper.c:6424
        ruby#5 0x1030bc1e0 in vm_exec_core insns.def:263
        ruby#6 0x1030b55fc in rb_vm_exec vm.c:2585
        ruby#7 0x1030fe0ac in rb_iseq_eval_main vm.c:2851
        ruby#8 0x102a82588 in rb_ec_exec_node eval.c:281
        ruby#9 0x102a81fe0 in ruby_run_node eval.c:319
        ruby#10 0x1027f3db4 in rb_main main.c:43
        ruby#11 0x1027f3bd4 in main main.c:68
        ruby#12 0x183900270  (<unknown module>)

    0x606000049528 is located 8 bytes inside of 56-byte region [0x606000049520,0x606000049558)
    freed by thread T0 here:
        #0 0x104174d40 in free+0x98 (libclang_rt.asan_osx_dynamic.dylib:arm64e+0x54d40)
        #1 0x102ada89c in rb_gc_impl_free default.c:8183
        ruby#2 0x102ada7dc in ruby_sized_xfree gc.c:4507
        ruby#3 0x102ac4d34 in ruby_xfree gc.c:4518
        ruby#4 0x102f3cb34 in rb_st_free_table st.c:663
        ruby#5 0x102bd52d8 in remove_from_constant_cache iseq.c:119
        ruby#6 0x102bbe2cc in iseq_clear_ic_references iseq.c:153
        ruby#7 0x102bbd2a0 in rb_iseq_free iseq.c:166
        ruby#8 0x102b32ed0 in rb_imemo_free imemo.c:564
        ruby#9 0x102ac4b44 in rb_gc_obj_free gc.c:1407
        ruby#10 0x102af4290 in gc_sweep_plane default.c:3546
        ruby#11 0x102af3bdc in gc_sweep_page default.c:3634
        ruby#12 0x102aeb140 in gc_sweep_step default.c:3906
        ruby#13 0x102aeadf0 in gc_sweep_rest default.c:3978
        ruby#14 0x102ae4714 in gc_sweep default.c:4155
        ruby#15 0x102af8474 in gc_start default.c:6484
        ruby#16 0x102afbe30 in garbage_collect default.c:6363
        ruby#17 0x102ad37f0 in rb_gc_impl_start default.c:6816
        ruby#18 0x102ad3634 in rb_gc gc.c:3624
        ruby#19 0x1031406ec in vm_track_constant_cache vm_insnhelper.c:6342
        #20 0x1030b91d8 in vm_ic_track_const_chain vm_insnhelper.c:6356
        ruby#21 0x1030b8cf8 in rb_vm_opt_getconstant_path vm_insnhelper.c:6424
        ruby#22 0x1030bc1e0 in vm_exec_core insns.def:263
        ruby#23 0x1030b55fc in rb_vm_exec vm.c:2585
        ruby#24 0x1030fe0ac in rb_iseq_eval_main vm.c:2851
        ruby#25 0x102a82588 in rb_ec_exec_node eval.c:281
        ruby#26 0x102a81fe0 in ruby_run_node eval.c:319
        ruby#27 0x1027f3db4 in rb_main main.c:43
        ruby#28 0x1027f3bd4 in main main.c:68
        ruby#29 0x183900270  (<unknown module>)

    previously allocated by thread T0 here:
        #0 0x104174c04 in malloc+0x94 (libclang_rt.asan_osx_dynamic.dylib:arm64e+0x54c04)
        #1 0x102ada0ec in rb_gc_impl_malloc default.c:8198
        ruby#2 0x102acee44 in ruby_xmalloc gc.c:4438
        ruby#3 0x102f3c85c in rb_st_init_table_with_size st.c:571
        ruby#4 0x102f3c900 in rb_st_init_table st.c:600
        ruby#5 0x102f3c920 in rb_st_init_numtable st.c:608
        ruby#6 0x103140698 in vm_track_constant_cache vm_insnhelper.c:6337
        ruby#7 0x1030b91d8 in vm_ic_track_const_chain vm_insnhelper.c:6356
        ruby#8 0x1030b8cf8 in rb_vm_opt_getconstant_path vm_insnhelper.c:6424
        ruby#9 0x1030bc1e0 in vm_exec_core insns.def:263
        ruby#10 0x1030b55fc in rb_vm_exec vm.c:2585
        ruby#11 0x1030fe0ac in rb_iseq_eval_main vm.c:2851
        ruby#12 0x102a82588 in rb_ec_exec_node eval.c:281
        ruby#13 0x102a81fe0 in ruby_run_node eval.c:319
        ruby#14 0x1027f3db4 in rb_main main.c:43
        ruby#15 0x1027f3bd4 in main main.c:68
        ruby#16 0x183900270  (<unknown module>)

This commit fixes this bug by adding a inserting_constant_cache_id field
to the VM, which stores the ID that is currently being inserted and, in
remove_from_constant_cache, we don't free the ST table for ID equal to
this one.

Co-Authored-By: Alan Wu <alanwu@ruby-lang.org>
peterzhu2118 added a commit to peterzhu2118/ruby that referenced this pull request Nov 29, 2024
[Bug #20921]

When we create a cache entry for a constant, the following sequence of
events could happen:

- vm_track_constant_cache is called to insert a constant cache.
- In vm_track_constant_cache, we first look up the ST table for the ID
  of the constant. Assume the ST table exists because another iseq also
  holds a cache entry for this ID.
- We then insert into this ST table with the iseq_inline_constant_cache.
- However, while inserting into this ST table, it allocates memory, which
  could trigger a GC. Assume that it does trigger a GC.
- The GC frees the one and only other iseq that holds a cache entry for
  this ID.
- In remove_from_constant_cache, it will appear that the ST table is now
  empty because there are no more iseq with cache entries for this ID, so
  we free the ST table.
- We complete GC and continue our st_insert. However, this ST table has
  been freed so we now have a use-after-free.

This issue is very hard to reproduce, because it requires that the GC runs
at a very specific time. However, we can make it show up by applying this
patch which runs GC right before the st_insert to mimic the st_insert
triggering a GC:

    diff --git a/vm_insnhelper.c b/vm_insnhelper.c
    index 3cb23f06f0..a93998136a 100644
    --- a/vm_insnhelper.c
    +++ b/vm_insnhelper.c
    @@ -6338,6 +6338,10 @@ vm_track_constant_cache(ID id, void *ic)
            rb_id_table_insert(const_cache, id, (VALUE)ics);
        }

    +    if (id == rb_intern("MyConstant")) rb_gc();
    +
        st_insert(ics, (st_data_t) ic, (st_data_t) Qtrue);
    }

And if we run this script:

    Object.const_set("MyConstant", "Hello!")

    my_proc = eval("-> { MyConstant }")
    my_proc.call

    my_proc = eval("-> { MyConstant }")
    my_proc.call

We can see that ASAN outputs a use-after-free error:

    ==36540==ERROR: AddressSanitizer: heap-use-after-free on address 0x606000049528 at pc 0x000102f3ceac bp 0x00016d607a70 sp 0x00016d607a68
    READ of size 8 at 0x606000049528 thread T0
        #0 0x102f3cea8 in do_hash st.c:321
        #1 0x102f3ddd0 in rb_st_insert st.c:1132
        ruby#2 0x103140700 in vm_track_constant_cache vm_insnhelper.c:6345
        ruby#3 0x1030b91d8 in vm_ic_track_const_chain vm_insnhelper.c:6356
        ruby#4 0x1030b8cf8 in rb_vm_opt_getconstant_path vm_insnhelper.c:6424
        ruby#5 0x1030bc1e0 in vm_exec_core insns.def:263
        ruby#6 0x1030b55fc in rb_vm_exec vm.c:2585
        ruby#7 0x1030fe0ac in rb_iseq_eval_main vm.c:2851
        ruby#8 0x102a82588 in rb_ec_exec_node eval.c:281
        ruby#9 0x102a81fe0 in ruby_run_node eval.c:319
        ruby#10 0x1027f3db4 in rb_main main.c:43
        ruby#11 0x1027f3bd4 in main main.c:68
        ruby#12 0x183900270  (<unknown module>)

    0x606000049528 is located 8 bytes inside of 56-byte region [0x606000049520,0x606000049558)
    freed by thread T0 here:
        #0 0x104174d40 in free+0x98 (libclang_rt.asan_osx_dynamic.dylib:arm64e+0x54d40)
        #1 0x102ada89c in rb_gc_impl_free default.c:8183
        ruby#2 0x102ada7dc in ruby_sized_xfree gc.c:4507
        ruby#3 0x102ac4d34 in ruby_xfree gc.c:4518
        ruby#4 0x102f3cb34 in rb_st_free_table st.c:663
        ruby#5 0x102bd52d8 in remove_from_constant_cache iseq.c:119
        ruby#6 0x102bbe2cc in iseq_clear_ic_references iseq.c:153
        ruby#7 0x102bbd2a0 in rb_iseq_free iseq.c:166
        ruby#8 0x102b32ed0 in rb_imemo_free imemo.c:564
        ruby#9 0x102ac4b44 in rb_gc_obj_free gc.c:1407
        ruby#10 0x102af4290 in gc_sweep_plane default.c:3546
        ruby#11 0x102af3bdc in gc_sweep_page default.c:3634
        ruby#12 0x102aeb140 in gc_sweep_step default.c:3906
        ruby#13 0x102aeadf0 in gc_sweep_rest default.c:3978
        ruby#14 0x102ae4714 in gc_sweep default.c:4155
        ruby#15 0x102af8474 in gc_start default.c:6484
        ruby#16 0x102afbe30 in garbage_collect default.c:6363
        ruby#17 0x102ad37f0 in rb_gc_impl_start default.c:6816
        ruby#18 0x102ad3634 in rb_gc gc.c:3624
        ruby#19 0x1031406ec in vm_track_constant_cache vm_insnhelper.c:6342
        #20 0x1030b91d8 in vm_ic_track_const_chain vm_insnhelper.c:6356
        ruby#21 0x1030b8cf8 in rb_vm_opt_getconstant_path vm_insnhelper.c:6424
        ruby#22 0x1030bc1e0 in vm_exec_core insns.def:263
        ruby#23 0x1030b55fc in rb_vm_exec vm.c:2585
        ruby#24 0x1030fe0ac in rb_iseq_eval_main vm.c:2851
        ruby#25 0x102a82588 in rb_ec_exec_node eval.c:281
        ruby#26 0x102a81fe0 in ruby_run_node eval.c:319
        ruby#27 0x1027f3db4 in rb_main main.c:43
        ruby#28 0x1027f3bd4 in main main.c:68
        ruby#29 0x183900270  (<unknown module>)

    previously allocated by thread T0 here:
        #0 0x104174c04 in malloc+0x94 (libclang_rt.asan_osx_dynamic.dylib:arm64e+0x54c04)
        #1 0x102ada0ec in rb_gc_impl_malloc default.c:8198
        ruby#2 0x102acee44 in ruby_xmalloc gc.c:4438
        ruby#3 0x102f3c85c in rb_st_init_table_with_size st.c:571
        ruby#4 0x102f3c900 in rb_st_init_table st.c:600
        ruby#5 0x102f3c920 in rb_st_init_numtable st.c:608
        ruby#6 0x103140698 in vm_track_constant_cache vm_insnhelper.c:6337
        ruby#7 0x1030b91d8 in vm_ic_track_const_chain vm_insnhelper.c:6356
        ruby#8 0x1030b8cf8 in rb_vm_opt_getconstant_path vm_insnhelper.c:6424
        ruby#9 0x1030bc1e0 in vm_exec_core insns.def:263
        ruby#10 0x1030b55fc in rb_vm_exec vm.c:2585
        ruby#11 0x1030fe0ac in rb_iseq_eval_main vm.c:2851
        ruby#12 0x102a82588 in rb_ec_exec_node eval.c:281
        ruby#13 0x102a81fe0 in ruby_run_node eval.c:319
        ruby#14 0x1027f3db4 in rb_main main.c:43
        ruby#15 0x1027f3bd4 in main main.c:68
        ruby#16 0x183900270  (<unknown module>)

This commit fixes this bug by adding a inserting_constant_cache_id field
to the VM, which stores the ID that is currently being inserted and, in
remove_from_constant_cache, we don't free the ST table for ID equal to
this one.

Co-Authored-By: Alan Wu <alanwu@ruby-lang.org>
peterzhu2118 added a commit that referenced this pull request Nov 29, 2024
[Bug #20921]

When we create a cache entry for a constant, the following sequence of
events could happen:

- vm_track_constant_cache is called to insert a constant cache.
- In vm_track_constant_cache, we first look up the ST table for the ID
  of the constant. Assume the ST table exists because another iseq also
  holds a cache entry for this ID.
- We then insert into this ST table with the iseq_inline_constant_cache.
- However, while inserting into this ST table, it allocates memory, which
  could trigger a GC. Assume that it does trigger a GC.
- The GC frees the one and only other iseq that holds a cache entry for
  this ID.
- In remove_from_constant_cache, it will appear that the ST table is now
  empty because there are no more iseq with cache entries for this ID, so
  we free the ST table.
- We complete GC and continue our st_insert. However, this ST table has
  been freed so we now have a use-after-free.

This issue is very hard to reproduce, because it requires that the GC runs
at a very specific time. However, we can make it show up by applying this
patch which runs GC right before the st_insert to mimic the st_insert
triggering a GC:

    diff --git a/vm_insnhelper.c b/vm_insnhelper.c
    index 3cb23f06f0..a93998136a 100644
    --- a/vm_insnhelper.c
    +++ b/vm_insnhelper.c
    @@ -6338,6 +6338,10 @@ vm_track_constant_cache(ID id, void *ic)
            rb_id_table_insert(const_cache, id, (VALUE)ics);
        }

    +    if (id == rb_intern("MyConstant")) rb_gc();
    +
        st_insert(ics, (st_data_t) ic, (st_data_t) Qtrue);
    }

And if we run this script:

    Object.const_set("MyConstant", "Hello!")

    my_proc = eval("-> { MyConstant }")
    my_proc.call

    my_proc = eval("-> { MyConstant }")
    my_proc.call

We can see that ASAN outputs a use-after-free error:

    ==36540==ERROR: AddressSanitizer: heap-use-after-free on address 0x606000049528 at pc 0x000102f3ceac bp 0x00016d607a70 sp 0x00016d607a68
    READ of size 8 at 0x606000049528 thread T0
        #0 0x102f3cea8 in do_hash st.c:321
        #1 0x102f3ddd0 in rb_st_insert st.c:1132
        #2 0x103140700 in vm_track_constant_cache vm_insnhelper.c:6345
        #3 0x1030b91d8 in vm_ic_track_const_chain vm_insnhelper.c:6356
        #4 0x1030b8cf8 in rb_vm_opt_getconstant_path vm_insnhelper.c:6424
        #5 0x1030bc1e0 in vm_exec_core insns.def:263
        #6 0x1030b55fc in rb_vm_exec vm.c:2585
        #7 0x1030fe0ac in rb_iseq_eval_main vm.c:2851
        #8 0x102a82588 in rb_ec_exec_node eval.c:281
        #9 0x102a81fe0 in ruby_run_node eval.c:319
        #10 0x1027f3db4 in rb_main main.c:43
        #11 0x1027f3bd4 in main main.c:68
        #12 0x183900270  (<unknown module>)

    0x606000049528 is located 8 bytes inside of 56-byte region [0x606000049520,0x606000049558)
    freed by thread T0 here:
        #0 0x104174d40 in free+0x98 (libclang_rt.asan_osx_dynamic.dylib:arm64e+0x54d40)
        #1 0x102ada89c in rb_gc_impl_free default.c:8183
        #2 0x102ada7dc in ruby_sized_xfree gc.c:4507
        #3 0x102ac4d34 in ruby_xfree gc.c:4518
        #4 0x102f3cb34 in rb_st_free_table st.c:663
        #5 0x102bd52d8 in remove_from_constant_cache iseq.c:119
        #6 0x102bbe2cc in iseq_clear_ic_references iseq.c:153
        #7 0x102bbd2a0 in rb_iseq_free iseq.c:166
        #8 0x102b32ed0 in rb_imemo_free imemo.c:564
        #9 0x102ac4b44 in rb_gc_obj_free gc.c:1407
        #10 0x102af4290 in gc_sweep_plane default.c:3546
        #11 0x102af3bdc in gc_sweep_page default.c:3634
        #12 0x102aeb140 in gc_sweep_step default.c:3906
        #13 0x102aeadf0 in gc_sweep_rest default.c:3978
        #14 0x102ae4714 in gc_sweep default.c:4155
        #15 0x102af8474 in gc_start default.c:6484
        #16 0x102afbe30 in garbage_collect default.c:6363
        #17 0x102ad37f0 in rb_gc_impl_start default.c:6816
        #18 0x102ad3634 in rb_gc gc.c:3624
        #19 0x1031406ec in vm_track_constant_cache vm_insnhelper.c:6342
        #20 0x1030b91d8 in vm_ic_track_const_chain vm_insnhelper.c:6356
        #21 0x1030b8cf8 in rb_vm_opt_getconstant_path vm_insnhelper.c:6424
        #22 0x1030bc1e0 in vm_exec_core insns.def:263
        #23 0x1030b55fc in rb_vm_exec vm.c:2585
        #24 0x1030fe0ac in rb_iseq_eval_main vm.c:2851
        #25 0x102a82588 in rb_ec_exec_node eval.c:281
        #26 0x102a81fe0 in ruby_run_node eval.c:319
        #27 0x1027f3db4 in rb_main main.c:43
        #28 0x1027f3bd4 in main main.c:68
        #29 0x183900270  (<unknown module>)

    previously allocated by thread T0 here:
        #0 0x104174c04 in malloc+0x94 (libclang_rt.asan_osx_dynamic.dylib:arm64e+0x54c04)
        #1 0x102ada0ec in rb_gc_impl_malloc default.c:8198
        #2 0x102acee44 in ruby_xmalloc gc.c:4438
        #3 0x102f3c85c in rb_st_init_table_with_size st.c:571
        #4 0x102f3c900 in rb_st_init_table st.c:600
        #5 0x102f3c920 in rb_st_init_numtable st.c:608
        #6 0x103140698 in vm_track_constant_cache vm_insnhelper.c:6337
        #7 0x1030b91d8 in vm_ic_track_const_chain vm_insnhelper.c:6356
        #8 0x1030b8cf8 in rb_vm_opt_getconstant_path vm_insnhelper.c:6424
        #9 0x1030bc1e0 in vm_exec_core insns.def:263
        #10 0x1030b55fc in rb_vm_exec vm.c:2585
        #11 0x1030fe0ac in rb_iseq_eval_main vm.c:2851
        #12 0x102a82588 in rb_ec_exec_node eval.c:281
        #13 0x102a81fe0 in ruby_run_node eval.c:319
        #14 0x1027f3db4 in rb_main main.c:43
        #15 0x1027f3bd4 in main main.c:68
        #16 0x183900270  (<unknown module>)

This commit fixes this bug by adding a inserting_constant_cache_id field
to the VM, which stores the ID that is currently being inserted and, in
remove_from_constant_cache, we don't free the ST table for ID equal to
this one.

Co-Authored-By: Alan Wu <alanwu@ruby-lang.org>
peterzhu2118 added a commit to peterzhu2118/ruby that referenced this pull request Dec 3, 2024
[Bug #20921]

When we create a cache entry for a constant, the following sequence of
events could happen:

- vm_track_constant_cache is called to insert a constant cache.
- In vm_track_constant_cache, we first look up the ST table for the ID
  of the constant. Assume the ST table exists because another iseq also
  holds a cache entry for this ID.
- We then insert into this ST table with the iseq_inline_constant_cache.
- However, while inserting into this ST table, it allocates memory, which
  could trigger a GC. Assume that it does trigger a GC.
- The GC frees the one and only other iseq that holds a cache entry for
  this ID.
- In remove_from_constant_cache, it will appear that the ST table is now
  empty because there are no more iseq with cache entries for this ID, so
  we free the ST table.
- We complete GC and continue our st_insert. However, this ST table has
  been freed so we now have a use-after-free.

This issue is very hard to reproduce, because it requires that the GC runs
at a very specific time. However, we can make it show up by applying this
patch which runs GC right before the st_insert to mimic the st_insert
triggering a GC:

    diff --git a/vm_insnhelper.c b/vm_insnhelper.c
    index 3cb23f06f0..a93998136a 100644
    --- a/vm_insnhelper.c
    +++ b/vm_insnhelper.c
    @@ -6338,6 +6338,10 @@ vm_track_constant_cache(ID id, void *ic)
            rb_id_table_insert(const_cache, id, (VALUE)ics);
        }

    +    if (id == rb_intern("MyConstant")) rb_gc();
    +
        st_insert(ics, (st_data_t) ic, (st_data_t) Qtrue);
    }

And if we run this script:

    Object.const_set("MyConstant", "Hello!")

    my_proc = eval("-> { MyConstant }")
    my_proc.call

    my_proc = eval("-> { MyConstant }")
    my_proc.call

We can see that ASAN outputs a use-after-free error:

    ==36540==ERROR: AddressSanitizer: heap-use-after-free on address 0x606000049528 at pc 0x000102f3ceac bp 0x00016d607a70 sp 0x00016d607a68
    READ of size 8 at 0x606000049528 thread T0
        #0 0x102f3cea8 in do_hash st.c:321
        #1 0x102f3ddd0 in rb_st_insert st.c:1132
        ruby#2 0x103140700 in vm_track_constant_cache vm_insnhelper.c:6345
        ruby#3 0x1030b91d8 in vm_ic_track_const_chain vm_insnhelper.c:6356
        ruby#4 0x1030b8cf8 in rb_vm_opt_getconstant_path vm_insnhelper.c:6424
        ruby#5 0x1030bc1e0 in vm_exec_core insns.def:263
        ruby#6 0x1030b55fc in rb_vm_exec vm.c:2585
        ruby#7 0x1030fe0ac in rb_iseq_eval_main vm.c:2851
        ruby#8 0x102a82588 in rb_ec_exec_node eval.c:281
        ruby#9 0x102a81fe0 in ruby_run_node eval.c:319
        ruby#10 0x1027f3db4 in rb_main main.c:43
        ruby#11 0x1027f3bd4 in main main.c:68
        ruby#12 0x183900270  (<unknown module>)

    0x606000049528 is located 8 bytes inside of 56-byte region [0x606000049520,0x606000049558)
    freed by thread T0 here:
        #0 0x104174d40 in free+0x98 (libclang_rt.asan_osx_dynamic.dylib:arm64e+0x54d40)
        #1 0x102ada89c in rb_gc_impl_free default.c:8183
        ruby#2 0x102ada7dc in ruby_sized_xfree gc.c:4507
        ruby#3 0x102ac4d34 in ruby_xfree gc.c:4518
        ruby#4 0x102f3cb34 in rb_st_free_table st.c:663
        ruby#5 0x102bd52d8 in remove_from_constant_cache iseq.c:119
        ruby#6 0x102bbe2cc in iseq_clear_ic_references iseq.c:153
        ruby#7 0x102bbd2a0 in rb_iseq_free iseq.c:166
        ruby#8 0x102b32ed0 in rb_imemo_free imemo.c:564
        ruby#9 0x102ac4b44 in rb_gc_obj_free gc.c:1407
        ruby#10 0x102af4290 in gc_sweep_plane default.c:3546
        ruby#11 0x102af3bdc in gc_sweep_page default.c:3634
        ruby#12 0x102aeb140 in gc_sweep_step default.c:3906
        ruby#13 0x102aeadf0 in gc_sweep_rest default.c:3978
        ruby#14 0x102ae4714 in gc_sweep default.c:4155
        ruby#15 0x102af8474 in gc_start default.c:6484
        ruby#16 0x102afbe30 in garbage_collect default.c:6363
        ruby#17 0x102ad37f0 in rb_gc_impl_start default.c:6816
        ruby#18 0x102ad3634 in rb_gc gc.c:3624
        ruby#19 0x1031406ec in vm_track_constant_cache vm_insnhelper.c:6342
        #20 0x1030b91d8 in vm_ic_track_const_chain vm_insnhelper.c:6356
        ruby#21 0x1030b8cf8 in rb_vm_opt_getconstant_path vm_insnhelper.c:6424
        ruby#22 0x1030bc1e0 in vm_exec_core insns.def:263
        ruby#23 0x1030b55fc in rb_vm_exec vm.c:2585
        ruby#24 0x1030fe0ac in rb_iseq_eval_main vm.c:2851
        ruby#25 0x102a82588 in rb_ec_exec_node eval.c:281
        ruby#26 0x102a81fe0 in ruby_run_node eval.c:319
        ruby#27 0x1027f3db4 in rb_main main.c:43
        ruby#28 0x1027f3bd4 in main main.c:68
        ruby#29 0x183900270  (<unknown module>)

    previously allocated by thread T0 here:
        #0 0x104174c04 in malloc+0x94 (libclang_rt.asan_osx_dynamic.dylib:arm64e+0x54c04)
        #1 0x102ada0ec in rb_gc_impl_malloc default.c:8198
        ruby#2 0x102acee44 in ruby_xmalloc gc.c:4438
        ruby#3 0x102f3c85c in rb_st_init_table_with_size st.c:571
        ruby#4 0x102f3c900 in rb_st_init_table st.c:600
        ruby#5 0x102f3c920 in rb_st_init_numtable st.c:608
        ruby#6 0x103140698 in vm_track_constant_cache vm_insnhelper.c:6337
        ruby#7 0x1030b91d8 in vm_ic_track_const_chain vm_insnhelper.c:6356
        ruby#8 0x1030b8cf8 in rb_vm_opt_getconstant_path vm_insnhelper.c:6424
        ruby#9 0x1030bc1e0 in vm_exec_core insns.def:263
        ruby#10 0x1030b55fc in rb_vm_exec vm.c:2585
        ruby#11 0x1030fe0ac in rb_iseq_eval_main vm.c:2851
        ruby#12 0x102a82588 in rb_ec_exec_node eval.c:281
        ruby#13 0x102a81fe0 in ruby_run_node eval.c:319
        ruby#14 0x1027f3db4 in rb_main main.c:43
        ruby#15 0x1027f3bd4 in main main.c:68
        ruby#16 0x183900270  (<unknown module>)

This commit fixes this bug by adding a inserting_constant_cache_id field
to the VM, which stores the ID that is currently being inserted and, in
remove_from_constant_cache, we don't free the ST table for ID equal to
this one.

Co-Authored-By: Alan Wu <alanwu@ruby-lang.org>
matzbot pushed a commit that referenced this pull request Dec 15, 2024
	Fix use-after-free in constant cache

	[Bug #20921]

	When we create a cache entry for a constant, the following sequence of
	events could happen:

	- vm_track_constant_cache is called to insert a constant cache.
	- In vm_track_constant_cache, we first look up the ST table for the ID
	  of the constant. Assume the ST table exists because another iseq also
	  holds a cache entry for this ID.
	- We then insert into this ST table with the iseq_inline_constant_cache.
	- However, while inserting into this ST table, it allocates memory, which
	  could trigger a GC. Assume that it does trigger a GC.
	- The GC frees the one and only other iseq that holds a cache entry for
	  this ID.
	- In remove_from_constant_cache, it will appear that the ST table is now
	  empty because there are no more iseq with cache entries for this ID, so
	  we free the ST table.
	- We complete GC and continue our st_insert. However, this ST table has
	  been freed so we now have a use-after-free.

	This issue is very hard to reproduce, because it requires that the GC runs
	at a very specific time. However, we can make it show up by applying this
	patch which runs GC right before the st_insert to mimic the st_insert
	triggering a GC:

	    diff --git a/vm_insnhelper.c b/vm_insnhelper.c
	    index 3cb23f06f0..a93998136a 100644
	    --- a/vm_insnhelper.c
	    +++ b/vm_insnhelper.c
	    @@ -6338,6 +6338,10 @@ vm_track_constant_cache(ID id, void *ic)
	            rb_id_table_insert(const_cache, id, (VALUE)ics);
	        }

	    +    if (id == rb_intern("MyConstant")) rb_gc();
	    +
	        st_insert(ics, (st_data_t) ic, (st_data_t) Qtrue);
	    }

	And if we run this script:

	    Object.const_set("MyConstant", "Hello!")

	    my_proc = eval("-> { MyConstant }")
	    my_proc.call

	    my_proc = eval("-> { MyConstant }")
	    my_proc.call

	We can see that ASAN outputs a use-after-free error:

	    ==36540==ERROR: AddressSanitizer: heap-use-after-free on address 0x606000049528 at pc 0x000102f3ceac bp 0x00016d607a70 sp 0x00016d607a68
	    READ of size 8 at 0x606000049528 thread T0
	        #0 0x102f3cea8 in do_hash st.c:321
	        #1 0x102f3ddd0 in rb_st_insert st.c:1132
	        #2 0x103140700 in vm_track_constant_cache vm_insnhelper.c:6345
	        #3 0x1030b91d8 in vm_ic_track_const_chain vm_insnhelper.c:6356
	        #4 0x1030b8cf8 in rb_vm_opt_getconstant_path vm_insnhelper.c:6424
	        #5 0x1030bc1e0 in vm_exec_core insns.def:263
	        #6 0x1030b55fc in rb_vm_exec vm.c:2585
	        #7 0x1030fe0ac in rb_iseq_eval_main vm.c:2851
	        #8 0x102a82588 in rb_ec_exec_node eval.c:281
	        #9 0x102a81fe0 in ruby_run_node eval.c:319
	        #10 0x1027f3db4 in rb_main main.c:43
	        #11 0x1027f3bd4 in main main.c:68
	        #12 0x183900270  (<unknown module>)

	    0x606000049528 is located 8 bytes inside of 56-byte region [0x606000049520,0x606000049558)
	    freed by thread T0 here:
	        #0 0x104174d40 in free+0x98 (libclang_rt.asan_osx_dynamic.dylib:arm64e+0x54d40)
	        #1 0x102ada89c in rb_gc_impl_free default.c:8183
	        #2 0x102ada7dc in ruby_sized_xfree gc.c:4507
	        #3 0x102ac4d34 in ruby_xfree gc.c:4518
	        #4 0x102f3cb34 in rb_st_free_table st.c:663
	        #5 0x102bd52d8 in remove_from_constant_cache iseq.c:119
	        #6 0x102bbe2cc in iseq_clear_ic_references iseq.c:153
	        #7 0x102bbd2a0 in rb_iseq_free iseq.c:166
	        #8 0x102b32ed0 in rb_imemo_free imemo.c:564
	        #9 0x102ac4b44 in rb_gc_obj_free gc.c:1407
	        #10 0x102af4290 in gc_sweep_plane default.c:3546
	        #11 0x102af3bdc in gc_sweep_page default.c:3634
	        #12 0x102aeb140 in gc_sweep_step default.c:3906
	        #13 0x102aeadf0 in gc_sweep_rest default.c:3978
	        #14 0x102ae4714 in gc_sweep default.c:4155
	        #15 0x102af8474 in gc_start default.c:6484
	        #16 0x102afbe30 in garbage_collect default.c:6363
	        #17 0x102ad37f0 in rb_gc_impl_start default.c:6816
	        #18 0x102ad3634 in rb_gc gc.c:3624
	        #19 0x1031406ec in vm_track_constant_cache vm_insnhelper.c:6342
	        #20 0x1030b91d8 in vm_ic_track_const_chain vm_insnhelper.c:6356
	        #21 0x1030b8cf8 in rb_vm_opt_getconstant_path vm_insnhelper.c:6424
	        #22 0x1030bc1e0 in vm_exec_core insns.def:263
	        #23 0x1030b55fc in rb_vm_exec vm.c:2585
	        #24 0x1030fe0ac in rb_iseq_eval_main vm.c:2851
	        #25 0x102a82588 in rb_ec_exec_node eval.c:281
	        #26 0x102a81fe0 in ruby_run_node eval.c:319
	        #27 0x1027f3db4 in rb_main main.c:43
	        #28 0x1027f3bd4 in main main.c:68
	        #29 0x183900270  (<unknown module>)

	    previously allocated by thread T0 here:
	        #0 0x104174c04 in malloc+0x94 (libclang_rt.asan_osx_dynamic.dylib:arm64e+0x54c04)
	        #1 0x102ada0ec in rb_gc_impl_malloc default.c:8198
	        #2 0x102acee44 in ruby_xmalloc gc.c:4438
	        #3 0x102f3c85c in rb_st_init_table_with_size st.c:571
	        #4 0x102f3c900 in rb_st_init_table st.c:600
	        #5 0x102f3c920 in rb_st_init_numtable st.c:608
	        #6 0x103140698 in vm_track_constant_cache vm_insnhelper.c:6337
	        #7 0x1030b91d8 in vm_ic_track_const_chain vm_insnhelper.c:6356
	        #8 0x1030b8cf8 in rb_vm_opt_getconstant_path vm_insnhelper.c:6424
	        #9 0x1030bc1e0 in vm_exec_core insns.def:263
	        #10 0x1030b55fc in rb_vm_exec vm.c:2585
	        #11 0x1030fe0ac in rb_iseq_eval_main vm.c:2851
	        #12 0x102a82588 in rb_ec_exec_node eval.c:281
	        #13 0x102a81fe0 in ruby_run_node eval.c:319
	        #14 0x1027f3db4 in rb_main main.c:43
	        #15 0x1027f3bd4 in main main.c:68
	        #16 0x183900270  (<unknown module>)

	This commit fixes this bug by adding a inserting_constant_cache_id field
	to the VM, which stores the ID that is currently being inserted and, in
	remove_from_constant_cache, we don't free the ST table for ID equal to
	this one.

	Co-Authored-By: Alan Wu <alanwu@ruby-lang.org>
k0kubun added a commit that referenced this pull request Jan 15, 2025
	Fix use-after-free in constant cache

	[Bug #20921]

	When we create a cache entry for a constant, the following sequence of
	events could happen:

	- vm_track_constant_cache is called to insert a constant cache.
	- In vm_track_constant_cache, we first look up the ST table for the ID
	  of the constant. Assume the ST table exists because another iseq also
	  holds a cache entry for this ID.
	- We then insert into this ST table with the iseq_inline_constant_cache.
	- However, while inserting into this ST table, it allocates memory, which
	  could trigger a GC. Assume that it does trigger a GC.
	- The GC frees the one and only other iseq that holds a cache entry for
	  this ID.
	- In remove_from_constant_cache, it will appear that the ST table is now
	  empty because there are no more iseq with cache entries for this ID, so
	  we free the ST table.
	- We complete GC and continue our st_insert. However, this ST table has
	  been freed so we now have a use-after-free.

	This issue is very hard to reproduce, because it requires that the GC runs
	at a very specific time. However, we can make it show up by applying this
	patch which runs GC right before the st_insert to mimic the st_insert
	triggering a GC:

	    diff --git a/vm_insnhelper.c b/vm_insnhelper.c
	    index 3cb23f06f0..a93998136a 100644
	    --- a/vm_insnhelper.c
	    +++ b/vm_insnhelper.c
	    @@ -6338,6 +6338,10 @@ vm_track_constant_cache(ID id, void *ic)
	            rb_id_table_insert(const_cache, id, (VALUE)ics);
	        }

	    +    if (id == rb_intern("MyConstant")) rb_gc();
	    +
	        st_insert(ics, (st_data_t) ic, (st_data_t) Qtrue);
	    }

	And if we run this script:

	    Object.const_set("MyConstant", "Hello!")

	    my_proc = eval("-> { MyConstant }")
	    my_proc.call

	    my_proc = eval("-> { MyConstant }")
	    my_proc.call

	We can see that ASAN outputs a use-after-free error:

	    ==36540==ERROR: AddressSanitizer: heap-use-after-free on address 0x606000049528 at pc 0x000102f3ceac bp 0x00016d607a70 sp 0x00016d607a68
	    READ of size 8 at 0x606000049528 thread T0
	        #0 0x102f3cea8 in do_hash st.c:321
	        #1 0x102f3ddd0 in rb_st_insert st.c:1132
	        #2 0x103140700 in vm_track_constant_cache vm_insnhelper.c:6345
	        #3 0x1030b91d8 in vm_ic_track_const_chain vm_insnhelper.c:6356
	        #4 0x1030b8cf8 in rb_vm_opt_getconstant_path vm_insnhelper.c:6424
	        #5 0x1030bc1e0 in vm_exec_core insns.def:263
	        #6 0x1030b55fc in rb_vm_exec vm.c:2585
	        #7 0x1030fe0ac in rb_iseq_eval_main vm.c:2851
	        #8 0x102a82588 in rb_ec_exec_node eval.c:281
	        #9 0x102a81fe0 in ruby_run_node eval.c:319
	        #10 0x1027f3db4 in rb_main main.c:43
	        #11 0x1027f3bd4 in main main.c:68
	        #12 0x183900270  (<unknown module>)

	    0x606000049528 is located 8 bytes inside of 56-byte region [0x606000049520,0x606000049558)
	    freed by thread T0 here:
	        #0 0x104174d40 in free+0x98 (libclang_rt.asan_osx_dynamic.dylib:arm64e+0x54d40)
	        #1 0x102ada89c in rb_gc_impl_free default.c:8183
	        #2 0x102ada7dc in ruby_sized_xfree gc.c:4507
	        #3 0x102ac4d34 in ruby_xfree gc.c:4518
	        #4 0x102f3cb34 in rb_st_free_table st.c:663
	        #5 0x102bd52d8 in remove_from_constant_cache iseq.c:119
	        #6 0x102bbe2cc in iseq_clear_ic_references iseq.c:153
	        #7 0x102bbd2a0 in rb_iseq_free iseq.c:166
	        #8 0x102b32ed0 in rb_imemo_free imemo.c:564
	        #9 0x102ac4b44 in rb_gc_obj_free gc.c:1407
	        #10 0x102af4290 in gc_sweep_plane default.c:3546
	        #11 0x102af3bdc in gc_sweep_page default.c:3634
	        #12 0x102aeb140 in gc_sweep_step default.c:3906
	        #13 0x102aeadf0 in gc_sweep_rest default.c:3978
	        #14 0x102ae4714 in gc_sweep default.c:4155
	        #15 0x102af8474 in gc_start default.c:6484
	        #16 0x102afbe30 in garbage_collect default.c:6363
	        #17 0x102ad37f0 in rb_gc_impl_start default.c:6816
	        #18 0x102ad3634 in rb_gc gc.c:3624
	        #19 0x1031406ec in vm_track_constant_cache vm_insnhelper.c:6342
	        #20 0x1030b91d8 in vm_ic_track_const_chain vm_insnhelper.c:6356
	        #21 0x1030b8cf8 in rb_vm_opt_getconstant_path vm_insnhelper.c:6424
	        #22 0x1030bc1e0 in vm_exec_core insns.def:263
	        #23 0x1030b55fc in rb_vm_exec vm.c:2585
	        #24 0x1030fe0ac in rb_iseq_eval_main vm.c:2851
	        #25 0x102a82588 in rb_ec_exec_node eval.c:281
	        #26 0x102a81fe0 in ruby_run_node eval.c:319
	        #27 0x1027f3db4 in rb_main main.c:43
	        #28 0x1027f3bd4 in main main.c:68
	        #29 0x183900270  (<unknown module>)

	    previously allocated by thread T0 here:
	        #0 0x104174c04 in malloc+0x94 (libclang_rt.asan_osx_dynamic.dylib:arm64e+0x54c04)
	        #1 0x102ada0ec in rb_gc_impl_malloc default.c:8198
	        #2 0x102acee44 in ruby_xmalloc gc.c:4438
	        #3 0x102f3c85c in rb_st_init_table_with_size st.c:571
	        #4 0x102f3c900 in rb_st_init_table st.c:600
	        #5 0x102f3c920 in rb_st_init_numtable st.c:608
	        #6 0x103140698 in vm_track_constant_cache vm_insnhelper.c:6337
	        #7 0x1030b91d8 in vm_ic_track_const_chain vm_insnhelper.c:6356
	        #8 0x1030b8cf8 in rb_vm_opt_getconstant_path vm_insnhelper.c:6424
	        #9 0x1030bc1e0 in vm_exec_core insns.def:263
	        #10 0x1030b55fc in rb_vm_exec vm.c:2585
	        #11 0x1030fe0ac in rb_iseq_eval_main vm.c:2851
	        #12 0x102a82588 in rb_ec_exec_node eval.c:281
	        #13 0x102a81fe0 in ruby_run_node eval.c:319
	        #14 0x1027f3db4 in rb_main main.c:43
	        #15 0x1027f3bd4 in main main.c:68
	        #16 0x183900270  (<unknown module>)

	This commit fixes this bug by adding a inserting_constant_cache_id field
	to the VM, which stores the ID that is currently being inserted and, in
	remove_from_constant_cache, we don't free the ST table for ID equal to
	this one.

	Co-Authored-By: Alan Wu <alanwu@ruby-lang.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.