Is this extra complexity justified to fix the problem?

I believe "NO".

I understand:

It stops all other ractors from executing (using the existing barrier interrupt mechanism)

But I couldn't understand why they are needed:

It stops the current thread from descheduling and switching to another thread (by adding a new dont_switch flag to the thread, and consulting that before marking the thread as performing a "blocking operation")
It stops the current thread from switching to a new fiber with the fiber scheduler (by using the thread->blocking counter)
It stops signal trap interrupts from executing (by changing the interrupt mask).

dump_all is C method and no switching, not?
If IO is used, I believe we should store data without Ruby's IO but use system IO (and sendfile to the Ruby's IO).

If IO is used, I believe we should store data without Ruby's IO but use system IO (and sendfile to the Ruby's IO).

Are there any portability problems with doing this, do you know? Can we extract the FD from the IO object, set it to blocking, and then just call write into it directly?

@ko1 I did some more research into this, and I think it's less complex to add this RB_VM_LOCK_ENTER_ISOLATE abstraction than the alternative.

The problem is, if we're given an IO to dump to, we don't know if that IO's file descriptor is set as blocking or nonblocking at the OS level. That means we have two options:

1. Check if the FD is blocking, and if so, switch it to blocking and then back to nonblocking when we're done.
2. Call write(2) on the FD, and if we get an EEAGAIN, call into poll(2) to wait for it to be writable again.

The problem with option 1, is that although we have a rb_io_set_nonblock method in io.h to put a FD into nonblocking mode, Ruby doesn't provide a method to set it back to blocking. There's the io/nonblock extension (which reimplements the fcntl calls itself), but that's now a (bundled? default?) gem. If we wanted to do this properly, we need to add two new APIs to io.h; one to set the FD to blocking mode, and a way to get the current mode (so we know whether to set it back). There will also need to be changes in win32.c to support this.

I guess the signature of what we want to add is...

void rb_io_set_nonblock2(VALUE io, bool nonblocking, bool *old_value)

or something along those lines.

The problem with option 2 is that we would need to replicate the machinery about whether to use select or poll etc which is wrapped up in rb_thread_wait_for_single_fd.

The reason I think my suggested approach in this PR is better is that it doesn't leak platform-specific stuff like FD O_NONBLOCK state, select vs poll, etc into objspace. Instead, we tell Ruby what we want to achieve ("I want this block to be the only possible code that can run in the VM until it finishes"), and then keep using all the ordinary I/O machinery just like we always would.

@ko1 I did some more research into this, and I think it's less complex to add this RB_VM_LOCK_ENTER_ISOLATE abstraction than the alternative.

Do not introduce general Thread control mechanism for the specific purpose like this. It is too much only for this purpose.

Can we extract the FD from the IO object, set it to blocking, and then just call write into it directly?

For example, we can dump all output to a temp file with C manner (no-Ruby related). After that we can put Ruby's IO from that file. Of course we can store all dumps to the memory but I'm not sure it is feasible.

we can dump all output to a temp file with C manner (no-Ruby related)

Oh that's not a bad idea actually. Let me try that. Thanks for the suggestion.

Well this almost works, except I need to exclude other Ractors still, and RB_VM_LOCK_ENTER() won't work from objspace because it's an extension module and the symbols needed by that macro aren't exported.

@ko1 would you mind if I added an exported version of that macro to vm_sync.c so it can be called from objspace_dump?

(alternatively: I wonder if objspace dump should be in an extension module at all - maybe it should just be linked with directly? it requires intimate knowledge about how objects in cruby work anyway)

@ko1 would you mind if I added an exported version of that macro to vm_sync.c so it can be called from objspace_dump?

• In general normal C extensions should not touch this kind of feature so I'm against to expose this feature.
• When Ractor local GC (heap) will be introduced I'm not sure what should be done for it.
• Exporting current macro (sync begin/end respectively) is bad idea.

objspace extension is tightly coupled with MRI internal so I suggest:

• make some sophisticated APIs
• put it on src/internal/xxx.h (racotr.h?)
• use it from src/ext/objspace

OK, I added

RUBY_SYMBOL_EXPORT_BEGIN
void rb_vm_lock_enter_ext(unsigned int *lev, const char *file, int line);
void rb_vm_lock_leave_ext(unsigned int *lev, const char *file, int line);
RUBY_SYMBOL_EXPORT_END

to vm_sync.h, which is an internal header, so this hasn't added to the actual Ruby API at all.

Is this what you had in mind? Or did you more have in mind something like

rb_ractor_isolate_begin(unsigned int *lev);
rb_ractor_isolate_end(unsigned int *lev);

to be added to ractor.h instead? The end result will be pretty much the same either way.

@ko1 Would appreciate your opinion on ☝️ when you have time 🙏 thank you.