Tail call VM [2] #18720

arnaud-lb · 2025-05-31T12:13:00Z

This is the second part of Tail call VM #17849
First part was Pass opline as argument to opcode handlers in CALL VM #17952

This part takes tail-calling and preserve_none from #17849:

Opcode handlers dispatch directly to the next handler by tail-calling, which reduces function call overhead and avoids returning to the executor loop
preserve_none reduces register saving overhead in opcode handlers

This also implements JIT support.

Non-dispatching opcode handlers

JIT needs non-dispatching opcode handlers (opcode handlers that return instead of calling the next one). I've tried two approaches for this:

Generate a second, non-dispatching, variant of each handler. Use the variant as call_handler
Use indirect dispatch similarly to the hybrid VM: zend_op->handler is a function that calls the real handler and dispatches.

I've tried both approach (the first one in this branch, and the second one in master...arnaud-lb:php-src:hybrid-tailcall.

The second approach resulted in a slightly slower VM due to indirect dispatching, and JIT generated more spilling when calling handlers as they clobber all registers.

Therefore I've taken the first approach in this PR.

A 3rd approach would be to control dispatching via an additional handler parameter, or to pass a dispatch function to handlers, but I suspect this would have been slower.

Fixed regs and preserved regs

Thanks to the preserve_none convention, JIT'ed code only has to preserve rbp, which reduces the size of prologue/epilogue. Instead of preserving it, I add it to the set of fixed registers, so it's not used. This results in faster code.

Also, quite conveniently, preserve_none receives its first arguments via registers that are callee-saved in sysv. Therefore we can use the arg1 and arg2 regs as our fixed registers. This avoids moving arg1 and arg2 to SP/IP in prologue, or setting arg1/arg2 when tail-calling other handlers.

Benchmarks:

Benchmark	Mode	vs base	vs gcc	vs valgrind
bench.php	JIT	-4%	-0%	-5%
bench.php	Non-JIT	-44%	+3%
symfony demo	JIT	-0%	-0%
symfony demo	Non-JIT	-2.8%	-0%

base: Clang build of master, wall time
gcc: GCC build of master, wall time
valgrind: Clang build of master, valgrind instructions

Conclusion: Clang builds are now as fast GCC builds on the Symfony Demo benchmark in both JIT and non-JIT modes.

Issues

The preserve_none calling convention is documented as unstable. JIT would break if it changed. I suggest checking this at build time, and disabling this optimization (tailcalling + preserve_none) if preserve_none changed. Edit: added a configure check.
Clang currently fails to build PHP when using preserve_none and ASAN, therefore we disable this if ASAN is enabled. Edit: This seems to be fixed in recent Clang versions.

UPGRADING

  . PHP binaries built with Clang>=19 are now as fast as binaries built with
    GCC. The performance of binaries built with other compilers has also
    improved considerably.

TODO

x86
aarch64
IR PRs
preserve_none configure check

ext/opcache/jit/ir/ir.h

ext/opcache/jit/ir/ir_x86.dasc

arnaud-lb · 2025-05-31T12:27:22Z

Zend/zend_compile.h

@@ -135,7 +136,7 @@ void zend_const_expr_to_zval(zval *result, zend_ast **ast_ptr, bool allow_dynami
 typedef int (*user_opcode_handler_t) (zend_execute_data *execute_data);

 struct _zend_op {
-	const void *handler;
+	zend_vm_opcode_handler_t handler;


Typed handler pointers in a few places to prevent confusion between orig handler and call handlers (zend_vm_opcode_handler_t / zend_vm_opcode_handler_func_t).

arnaud-lb · 2025-05-31T12:31:16Z

ext/opcache/jit/zend_jit_vm_helpers.c

-ZEND_OPCODE_HANDLER_RET ZEND_FASTCALL zend_jit_func_trace_helper(ZEND_OPCODE_HANDLER_ARGS)
+ZEND_OPCODE_HANDLER_RET ZEND_OPCODE_HANDLER_CCONV zend_jit_func_trace_helper(ZEND_OPCODE_HANDLER_ARGS)
 {
-	ZEND_OPCODE_TAIL_CALL_EX(zend_jit_trace_counter_helper,
-		((ZEND_JIT_COUNTER_INIT + JIT_G(hot_func) - 1) / JIT_G(hot_func)));
+	zend_jit_op_array_trace_extension *jit_extension =
+		(zend_jit_op_array_trace_extension*)ZEND_FUNC_INFO(&EX(func)->op_array);
+	size_t offset = jit_extension->offset;
+	uint32_t cost = ((ZEND_JIT_COUNTER_INIT + JIT_G(hot_func) - 1) / JIT_G(hot_func));
+
+	*(ZEND_OP_TRACE_INFO(opline, offset)->counter) -= cost;
+
+	ZEND_OPCODE_TAIL_CALL(zend_jit_trace_counter_helper);


Tail-calling was not possible due to the extra arg in zend_jit_trace_counter_helper, so I removed it.

Alternative would be to define these handlers in IR, as we do for the hybrid JIT.

I didn't understand why extra_arg became a problem

Clang ˋmusttailˋ disallows calling functions with a different signature. Presumably this is to garantee portability. This is discussed here: https://blog.reverberate.org/2025/02/10/tail-call-updates.html

arnaud-lb · 2025-06-02T08:41:51Z

@dstogov this is still a WIP, but I would like to have your opinion on this. This is the same ideas as #17849, but with JIT support which implies a few things described above. I would like your opinion before handling the items in the todo list.

dstogov

The idea makes sense and the benchmark results look promising.
It's also interesting how it affects the VM code size.

Unfortunately at this moment I can't review a huge path like this in all details.
Anyway, I think it makes sense to finalize and land it.

Zend/zend_vm.h

dstogov · 2025-06-09T09:49:12Z

Zend/zend_vm_gen.php

+                            out($f,"typedef struct _zend_vm_trampoline {\n");
+                            out($f,"    const zend_op            *opline;\n");
+                            out($f,"    zend_vm_opcode_handler_t  handler;\n");
+                            out($f,"} zend_vm_trampoline;\n");


C standard doesn't require returning content of struct in a pair of registers.
x86_64 ABI does this, but all other ABIs have to be checked (x86, Windows64, ...).

Indeed.

SysV amd64 and aarch64 return the struct via registers, but SysV i386 and MS ABI do not.

Tail calling is effectively only enabled on clang, so this should not be an issue on Windows, at least currently.

It is an issue on x86 however.

I've made the following changes:

Added __attribute__((sysv_abi)) on handlers returning zend_vm_trampoline, when compiling with Clang on Windows, so the struct is returned via registers.

Enable tail calling only on x86_64 and arm64 (Edit: preserve_none is not supported on x86 anyway)

ext/opcache/jit/ir/ir_emit.c

ext/opcache/jit/ir/ir_x86.dasc

dstogov · 2025-06-09T10:12:58Z

ext/opcache/jit/zend_jit_ir.c

+				 * https://github.com/llvm/llvm-project/blob/a414877a7a5f000d01370acb1162eb1dea87f48c/llvm/lib/Target/X86/X86RegisterInfo.cpp#L319
+				 * https://github.com/llvm/llvm-project/blob/68bfe91b5a34f80dbcc4f0a7fa5d7aa1cdf959c2/llvm/lib/Target/X86/X86CallingConv.td#L1183
+				 */
+				jit->ctx.fixed_regset |= (1<<5);


fixed_regs are the registers reserved for the register variables.
I remember there were some problems with RBP usage, but I don't remember the exact problem.

In this case I add the register only to prevent the RA from using it. This is less expensive than adding it to the set of preserved registers. The same thing is done for the HYBRID VM a few lines below.

dstogov · 2025-06-09T10:24:10Z

ext/opcache/jit/zend_jit_vm_helpers.c

-ZEND_OPCODE_HANDLER_RET ZEND_FASTCALL zend_jit_func_trace_helper(ZEND_OPCODE_HANDLER_ARGS)
+ZEND_OPCODE_HANDLER_RET ZEND_OPCODE_HANDLER_CCONV zend_jit_func_trace_helper(ZEND_OPCODE_HANDLER_ARGS)
 {
-	ZEND_OPCODE_TAIL_CALL_EX(zend_jit_trace_counter_helper,
-		((ZEND_JIT_COUNTER_INIT + JIT_G(hot_func) - 1) / JIT_G(hot_func)));
+	zend_jit_op_array_trace_extension *jit_extension =
+		(zend_jit_op_array_trace_extension*)ZEND_FUNC_INFO(&EX(func)->op_array);
+	size_t offset = jit_extension->offset;
+	uint32_t cost = ((ZEND_JIT_COUNTER_INIT + JIT_G(hot_func) - 1) / JIT_G(hot_func));
+
+	*(ZEND_OP_TRACE_INFO(opline, offset)->counter) -= cost;
+
+	ZEND_OPCODE_TAIL_CALL(zend_jit_trace_counter_helper);


I didn't understand why extra_arg became a problem

ext/opcache/jit/ir/ir_x86.dasc

arnaud-lb · 2025-07-26T13:11:04Z

@dstogov I've addressed all issues, and all necessary IR changes have been merged, so this branch is now ready to be reviewed again, and hopefully merged :)

Since your last review I've made the following notable change: As adding preserve_none to IR requires longer term changes, I took the decision to not rely on IR for calling preserve_none functions. Instead I call these functions with the default convention in JIT. Arguments are passed via fixed regs, and the return register is the same as the default convention. This is safe because we always tailcall these functions, and only from other preserve_none functions, therefore we don't need to care about preserving regs around these calls.

dstogov

Please see my questions...

dstogov · 2025-08-12T10:21:18Z

Zend/Zend.m4

+uintptr_t __attribute__((preserve_none)) test(void) {
+	uintptr_t ret;
+
+#if defined(__x86_64__)
+	__asm__ __volatile__(
+		/* XORing to make it unlikely the value exists in any other register */
+		"movq %1, %%r12\n"
+		"xorq %3, %%r12\n"
+		"movq %2, %%r13\n"
+		"xorq %3, %%r13\n"
+		"xorq %%rax, %%rax\n"
+		"call fun\n"
+		: "=a" (ret)
+		: "r" (const1), "r" (const2), "r" (key)
+		: "r12", "r13"
+	);


Why do we need these asm tests?
preserve_none is an unstable CLANG extension, but can't we just test CLANG version?

These tests ensure that the tailcall VM will be enabled in all future versions of Clang as long as these expectations remain valid.

This test will just hide the problem in case of possible incompatibility with future versions.
I wouldn't care about compatibility with future versions. They always may break something.

dstogov · 2025-08-12T10:35:41Z

Zend/zend_vm_execute.skl

-#if ZEND_VM_KIND == ZEND_VM_KIND_CALL
-	return op->handler;
-#elif ZEND_VM_KIND == ZEND_VM_KIND_HYBRID
+#if ZEND_VM_KIND == ZEND_VM_KIND_HYBRID || ZEND_VM_TAIL_CALL_DISPATCH


Why do you use a separate define ZEND_VM_TAIL_CALL_DISPATCH?
May be it's better to use ZEND_VM_KIND_TAIL_CALL?
This would look more consistent.

Good point, I will check

dstogov · 2025-08-12T11:33:43Z

Zend/zend_portability.h

+#if __has_attribute(sysv_abi)
+# define HAVE_SYSV_ABI
+# define ZEND_SYSV_ABI __attribute__((sysv_abi))
+#endif


The fact that we have sysv_abi attribute, doesn't mean that SYSV_ABI is the default calling convention.
According to CLANG manual it's available on Windows.

According to CLANG manual it's available on Windows.

Yes that's the point :) This is related to #18720 (comment), but this is not absolutely required as we don't support building with Clang on Windows yet, so I can remove this.

No problem. I saw that you use it especially for Windows.

dstogov · 2025-08-12T11:44:47Z

Zend/zend_vm_gen.php

+                // Helper with parameter. Must use trampoline dispatch due to
+                // incompatible signature for tailcall.
+                out($f, "#undef  ZEND_VM_DISPATCH\n");
+                out($f, "#define ZEND_VM_DISPATCH(handler) ZEND_VM_DISPATCH_NOTAIL(handler)\n");


Repeatable redefinition of the same macro looks very weird.
Can't we redefine it once - between generating code for TAIL_CALL and regular CALL handlers?

We need this before VM helpers that take extra arguments, because __attribute__((musttail)) doesn't allow them to tailcall to helpers or handlers with a different number of args.

Currently, VM helpers with extra arguments are interleaved with normal VM helpers and opcode handlers.

Avoiding the repeated redefinition would require to group VM helpers with extra args together, however this may change code locality.

WDYT?

I think that all TAIL_CALL handlers and helpers should be generated after/before all CALL handlers and helpers.

dstogov · 2025-08-12T12:25:54Z

Zend/zend_vm_gen.php

@@ -1746,7 +1774,8 @@ function gen_executor_code($f, $spec, $kind, $prolog, &$switch_labels = array())
    // Generate handler for undefined opcodes
    switch ($kind) {
        case ZEND_VM_KIND_CALL:
-            gen_null_handler($f);
+            gen_null_handler($f, $variant);
+            gen_halt_handler($f, $variant);


I suppose halt_handler is necessary for tail_call variant only.

dstogov · 2025-08-12T13:59:47Z

ext/opcache/jit/zend_jit_ir.c

+# define IR_OPCODE_HANDLER_RET IR_ADDR
+#endif
+
+#define IR_OPCODE_HANDLER_FUNC IR_FASTCALL_FUNC


Why do we need this new name?

dstogov · 2025-08-12T14:05:35Z

ext/opcache/jit/zend_jit_ir.c

+# define ir_CONST_OPCODE_HANDLER_FUNC(_addr) \
+	jit_CONST_OPCODE_HANDLER_FUNC(jit, _addr)
+# define ir_CAST_OPCODE_HANDLER_FUNC(_addr)  ir_fold2(_ir_CTX, IR_OPT(IR_PROTO, IR_ADDR), (_addr), \
+	ir_proto_0(_ir_CTX, IR_OPCODE_HANDLER_FUNC, IR_I32))


I'm not sure about IR_I32.

dstogov · 2025-08-12T14:28:01Z

ext/opcache/jit/zend_jit_ir.c

+				jit->ctx.fixed_regset |= (1<<IR_REG_FP) | (1<<IR_REG_LR);
+#else
+				ZEND_UNREACHABLE();
+#endif


jit->ctx.fixed_stack_frame_size assigned above may be reduced as we don't need space to save/restore any register now.

dstogov · 2025-08-12T14:42:42Z

ext/opcache/jit/zend_jit_ir.c

+#  define ZREG_FP             12 /* IR_REG_R12 */
+#  define ZREG_IP             13 /* IR_REG_R13 */


These registers are not marked as "fixed" any more, so IR may reuse them for something else.
At least it should be used for passing arguments to helper functions.
On the other hand, they are still used in RLOAD/RSTORE instructions.
I'm surprised how this can work. What do I miss?

dstogov · 2025-08-12T14:45:48Z

ext/opcache/jit/zend_jit_ir.c

-		    opline->opcode == ZEND_GENERATOR_CREATE) {
+		    opline->opcode == ZEND_GENERATOR_CREATE ||
+		    opline->opcode == ZEND_INCLUDE_OR_EVAL) {


Is this an independent fix?

github-actions bot added Category: Engine Extension: opcache ABI break labels May 31, 2025

arnaud-lb commented May 31, 2025

View reviewed changes

arnaud-lb force-pushed the tailcall branch from 07624db to 9118090 Compare May 31, 2025 20:01

dstogov reviewed Jun 9, 2025

View reviewed changes

arnaud-lb force-pushed the tailcall branch 3 times, most recently from c8cb9d5 to e334ae4 Compare June 27, 2025 16:42

dstogov reviewed Jun 30, 2025

View reviewed changes

ext/opcache/jit/ir/ir_x86.dasc Outdated Show resolved Hide resolved

dstogov reviewed Jun 30, 2025

View reviewed changes

ext/opcache/jit/ir/ir_x86.dasc Outdated Show resolved Hide resolved

arnaud-lb force-pushed the tailcall branch from e334ae4 to ad7d2aa Compare July 1, 2025 17:29

arnaud-lb mentioned this pull request Jul 2, 2025

Introduce zend_vm_opcode_handler_t / zend_vm_opcode_handler_func_t #19006

Closed

arnaud-lb force-pushed the tailcall branch from ad7d2aa to 04d3408 Compare July 25, 2025 18:09

github-actions bot added the Category: Build System label Jul 25, 2025

Tail call VM

46368eb

arnaud-lb force-pushed the tailcall branch from 04d3408 to 46368eb Compare July 26, 2025 12:01

arnaud-lb marked this pull request as ready for review July 26, 2025 13:11

arnaud-lb requested a review from dstogov July 26, 2025 13:11

dstogov reviewed Aug 12, 2025

View reviewed changes

		# define ZREG_FP 12 /* IR_REG_R12 */
		# define ZREG_IP 13 /* IR_REG_R13 */

Tail call VM [2] #18720

Are you sure you want to change the base?

Tail call VM [2] #18720

Uh oh!

Conversation

arnaud-lb commented May 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Non-dispatching opcode handlers

Fixed regs and preserved regs

Benchmarks:

Issues

UPGRADING

TODO

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

arnaud-lb commented Jun 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dstogov left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

arnaud-lb Jun 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

arnaud-lb commented Jul 26, 2025

Uh oh!

dstogov left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

arnaud-lb commented May 31, 2025 •

edited

Loading

arnaud-lb commented Jun 2, 2025 •

edited

Loading

arnaud-lb Jun 25, 2025 •

edited

Loading