Tail call VM [2] #18720

arnaud-lb · 2025-05-31T12:13:00Z

This is the second part of Tail call VM #17849
First part was Pass opline as argument to opcode handlers in CALL VM #17952

This part takes tail-calling and preserve_none from #17849:

Opcode handlers dispatch directly to the next handler by tail-calling, which reduces function call overhead and avoids returning to the executor loop
preserve_none reduces register saving overhead in opcode handlers

This also implements JIT support.

Non-dispatching opcode handlers

JIT needs non-dispatching opcode handlers (opcode handlers that return instead of calling the next one). I've tried two approaches for this:

Generate a second, non-dispatching, variant of each handler. Use the variant as call_handler
Use indirect dispatch similarly to the hybrid VM: zend_op->handler is a function that calls the real handler and dispatches.

I've tried both approach (the first one in this branch, and the second one in master...arnaud-lb:php-src:hybrid-tailcall.

The second approach resulted in a slightly slower VM due to indirect dispatching, and JIT generated more spilling when calling handlers as they clobber all registers.

Therefore I've taken the first approach in this PR.

A 3rd approach would be to control dispatching via an additional handler parameter, or to pass a dispatch function to handlers, but I suspect this would have been slower.

Fixed regs and preserved regs

Thanks to the preserve_none convention, JIT'ed code only has to preserve rbp, which reduces the size of prologue/epilogue. Instead of preserving it, I add it to the set of fixed registers, so it's not used. This results in faster code.

Also, quite conveniently, preserve_none receives its first arguments via registers that are callee-saved in sysv. Therefore we can use the arg1 and arg2 regs as our fixed registers. This avoids moving arg1 and arg2 to SP/IP in prologue, or setting arg1/arg2 when tail-calling other handlers.

Benchmarks:

Benchmark	Mode	vs base	vs gcc	vs valgrind
bench.php	JIT	-4%	-0%	-5%
bench.php	Non-JIT	-44%	+3%
symfony demo	JIT	-0%	-0%
symfony demo	Non-JIT	-2.8%	-0%

base: Clang build of master, wall time
gcc: GCC build of master, wall time
valgrind: Clang build of master, valgrind instructions

Conclusion: Clang builds are now as fast GCC builds on the Symfony Demo benchmark in both JIT and non-JIT modes.

Issues

The preserve_none calling convention is documented as unstable. JIT would break if it changed. I suggest checking this at build time, and disabling this optimization (tailcalling + preserve_none) if preserve_none changed. Edit: added a configure check.
Clang currently fails to build PHP when using preserve_none and ASAN, therefore we disable this if ASAN is enabled. Edit: This seems to be fixed in recent Clang versions.

UPGRADING

  . PHP binaries built with Clang>=19 are now as fast as binaries built with
    GCC. The performance of binaries built with other compilers has also
    improved considerably.

TODO

x86
aarch64
IR PRs
preserve_none configure check

ext/opcache/jit/ir/ir.h

ext/opcache/jit/ir/ir_x86.dasc

Zend/zend_compile.h

arnaud-lb · 2025-05-31T12:31:16Z

ext/opcache/jit/zend_jit_vm_helpers.c

-ZEND_OPCODE_HANDLER_RET ZEND_FASTCALL zend_jit_func_trace_helper(ZEND_OPCODE_HANDLER_ARGS)
+ZEND_OPCODE_HANDLER_RET ZEND_OPCODE_HANDLER_CCONV zend_jit_func_trace_helper(ZEND_OPCODE_HANDLER_ARGS)
 {
-	ZEND_OPCODE_TAIL_CALL_EX(zend_jit_trace_counter_helper,
-		((ZEND_JIT_COUNTER_INIT + JIT_G(hot_func) - 1) / JIT_G(hot_func)));
+	zend_jit_op_array_trace_extension *jit_extension =
+		(zend_jit_op_array_trace_extension*)ZEND_FUNC_INFO(&EX(func)->op_array);
+	size_t offset = jit_extension->offset;
+	uint32_t cost = ((ZEND_JIT_COUNTER_INIT + JIT_G(hot_func) - 1) / JIT_G(hot_func));
+
+	*(ZEND_OP_TRACE_INFO(opline, offset)->counter) -= cost;
+
+	ZEND_OPCODE_TAIL_CALL(zend_jit_trace_counter_helper);


Tail-calling was not possible due to the extra arg in zend_jit_trace_counter_helper, so I removed it.

Alternative would be to define these handlers in IR, as we do for the hybrid JIT.

I didn't understand why extra_arg became a problem

Clang ˋmusttailˋ disallows calling functions with a different signature. Presumably this is to garantee portability. This is discussed here: https://blog.reverberate.org/2025/02/10/tail-call-updates.html

arnaud-lb · 2025-06-02T08:41:51Z

@dstogov this is still a WIP, but I would like to have your opinion on this. This is the same ideas as #17849, but with JIT support which implies a few things described above. I would like your opinion before handling the items in the todo list.

dstogov

The idea makes sense and the benchmark results look promising.
It's also interesting how it affects the VM code size.

Unfortunately at this moment I can't review a huge path like this in all details.
Anyway, I think it makes sense to finalize and land it.

Zend/zend_vm.h

Zend/zend_vm_gen.php

ext/opcache/jit/ir/ir_emit.c

ext/opcache/jit/ir/ir_x86.dasc

dstogov · 2025-06-09T10:12:58Z

ext/opcache/jit/zend_jit_ir.c

+				 * https://github.com/llvm/llvm-project/blob/a414877a7a5f000d01370acb1162eb1dea87f48c/llvm/lib/Target/X86/X86RegisterInfo.cpp#L319
+				 * https://github.com/llvm/llvm-project/blob/68bfe91b5a34f80dbcc4f0a7fa5d7aa1cdf959c2/llvm/lib/Target/X86/X86CallingConv.td#L1183
+				 */
+				jit->ctx.fixed_regset |= (1<<5);


fixed_regs are the registers reserved for the register variables.
I remember there were some problems with RBP usage, but I don't remember the exact problem.

In this case I add the register only to prevent the RA from using it. This is less expensive than adding it to the set of preserved registers. The same thing is done for the HYBRID VM a few lines below.

dstogov · 2025-06-09T10:24:10Z

ext/opcache/jit/zend_jit_vm_helpers.c

-ZEND_OPCODE_HANDLER_RET ZEND_FASTCALL zend_jit_func_trace_helper(ZEND_OPCODE_HANDLER_ARGS)
+ZEND_OPCODE_HANDLER_RET ZEND_OPCODE_HANDLER_CCONV zend_jit_func_trace_helper(ZEND_OPCODE_HANDLER_ARGS)
 {
-	ZEND_OPCODE_TAIL_CALL_EX(zend_jit_trace_counter_helper,
-		((ZEND_JIT_COUNTER_INIT + JIT_G(hot_func) - 1) / JIT_G(hot_func)));
+	zend_jit_op_array_trace_extension *jit_extension =
+		(zend_jit_op_array_trace_extension*)ZEND_FUNC_INFO(&EX(func)->op_array);
+	size_t offset = jit_extension->offset;
+	uint32_t cost = ((ZEND_JIT_COUNTER_INIT + JIT_G(hot_func) - 1) / JIT_G(hot_func));
+
+	*(ZEND_OP_TRACE_INFO(opline, offset)->counter) -= cost;
+
+	ZEND_OPCODE_TAIL_CALL(zend_jit_trace_counter_helper);


I didn't understand why extra_arg became a problem

ext/opcache/jit/ir/ir_x86.dasc

arnaud-lb · 2025-07-26T13:11:04Z

@dstogov I've addressed all issues, and all necessary IR changes have been merged, so this branch is now ready to be reviewed again, and hopefully merged :)

Since your last review I've made the following notable change: As adding preserve_none to IR requires longer term changes, I took the decision to not rely on IR for calling preserve_none functions. Instead I call these functions with the default convention in JIT. Arguments are passed via fixed regs, and the return register is the same as the default convention. This is safe because we always tailcall these functions, and only from other preserve_none functions, therefore we don't need to care about preserving regs around these calls.

dstogov

Please see my questions...

dstogov · 2025-08-12T10:21:18Z

Zend/Zend.m4

+uintptr_t __attribute__((preserve_none)) test(void) {
+	uintptr_t ret;
+
+#if defined(__x86_64__)
+	__asm__ __volatile__(
+		/* XORing to make it unlikely the value exists in any other register */
+		"movq %1, %%r12\n"
+		"xorq %3, %%r12\n"
+		"movq %2, %%r13\n"
+		"xorq %3, %%r13\n"
+		"xorq %%rax, %%rax\n"
+		"call fun\n"
+		: "=a" (ret)
+		: "r" (const1), "r" (const2), "r" (key)
+		: "r12", "r13"
+	);


Why do we need these asm tests?
preserve_none is an unstable CLANG extension, but can't we just test CLANG version?

These tests ensure that the tailcall VM will be enabled in all future versions of Clang as long as these expectations remain valid.

This test will just hide the problem in case of possible incompatibility with future versions.
I wouldn't care about compatibility with future versions. They always may break something.

By hard coding clang versions, we would have to regularly update the versions in the future, including in release branches.

The only thing we rely on is that arg1/arg2 are passed via r12/r13, that the return register is rax, and that callers don't expect any register to be saved. The last point is unlikely to change as that's the purpose of this function. This test would detect if arg1/arg2 or ret changes in a future version. It seemms to me that any other change to this calling convention would be unlikely to break the JIT?

Zend/zend_vm_execute.skl

Zend/zend_portability.h

dstogov · 2025-08-12T11:44:47Z

Zend/zend_vm_gen.php

+                // Helper with parameter. Must use trampoline dispatch due to
+                // incompatible signature for tailcall.
+                out($f, "#undef  ZEND_VM_DISPATCH\n");
+                out($f, "#define ZEND_VM_DISPATCH(handler) ZEND_VM_DISPATCH_NOTAIL(handler)\n");


Repeatable redefinition of the same macro looks very weird.
Can't we redefine it once - between generating code for TAIL_CALL and regular CALL handlers?

We need this before VM helpers that take extra arguments, because __attribute__((musttail)) doesn't allow them to tailcall to helpers or handlers with a different number of args.

Currently, VM helpers with extra arguments are interleaved with normal VM helpers and opcode handlers.

Avoiding the repeated redefinition would require to group VM helpers with extra args together, however this may change code locality.

WDYT?

I think that all TAIL_CALL handlers and helpers should be generated after/before all CALL handlers and helpers.

I've made changes so that we generate handlers in this order:

CALL handlers

TAILCALL handlers and helpers without extra args

TAILCALL helpers with extra args

The undef/define is used only once before the last group.

I feared this would affect the location of these function in memory, but it doesn't, and doesn't affect benchmarks.

Zend/zend_vm_gen.php

ext/opcache/jit/zend_jit_ir.c

dstogov · 2025-08-12T14:42:42Z

ext/opcache/jit/zend_jit_ir.c

+#  define ZREG_FP             12 /* IR_REG_R12 */
+#  define ZREG_IP             13 /* IR_REG_R13 */


These registers are not marked as "fixed" any more, so IR may reuse them for something else.
At least it should be used for passing arguments to helper functions.
On the other hand, they are still used in RLOAD/RSTORE instructions.
I'm surprised how this can work. What do I miss?

They are still marked as fixed :)

dstogov · 2025-08-12T14:45:48Z

ext/opcache/jit/zend_jit_ir.c

-		    opline->opcode == ZEND_GENERATOR_CREATE) {
+		    opline->opcode == ZEND_GENERATOR_CREATE ||
+		    opline->opcode == ZEND_INCLUDE_OR_EVAL) {


Is this an independent fix?

Not really. This was not needed before because the trace returns IP|ZEND_VM_ENTER_BIT immediately after ZEND_INCLUDE_OR_EVAL (ZEND_INCLUDE_OR_EVAL ends tracing with ZEND_JIT_TRACE_STOP_INTERPRETER). But in TAILCALL VM we call IP->handler instead, so we need to decode IP first.

…rampoline)

…-define it

Was a left over of relying on IR for preserve_none support

This reverts commit f0d41731921711cddea8dcc1d56ed16af5c2098b.

…n is not necessary anymore

github-actions bot added Category: Engine Extension: opcache ABI break labels May 31, 2025

arnaud-lb commented May 31, 2025

View reviewed changes

arnaud-lb force-pushed the tailcall branch from 07624db to 9118090 Compare May 31, 2025 20:01

dstogov reviewed Jun 9, 2025

View reviewed changes

arnaud-lb force-pushed the tailcall branch 3 times, most recently from c8cb9d5 to e334ae4 Compare June 27, 2025 16:42

dstogov reviewed Jun 30, 2025

View reviewed changes

ext/opcache/jit/ir/ir_x86.dasc Outdated Show resolved Hide resolved

dstogov reviewed Jun 30, 2025

View reviewed changes

ext/opcache/jit/ir/ir_x86.dasc Outdated Show resolved Hide resolved

arnaud-lb force-pushed the tailcall branch from e334ae4 to ad7d2aa Compare July 1, 2025 17:29

arnaud-lb mentioned this pull request Jul 2, 2025

Introduce zend_vm_opcode_handler_t / zend_vm_opcode_handler_func_t #19006

Closed

arnaud-lb force-pushed the tailcall branch from ad7d2aa to 04d3408 Compare July 25, 2025 18:09

github-actions bot added the Category: Build System label Jul 25, 2025

arnaud-lb force-pushed the tailcall branch from 04d3408 to 46368eb Compare July 26, 2025 12:01

arnaud-lb marked this pull request as ready for review July 26, 2025 13:11

arnaud-lb requested a review from dstogov July 26, 2025 13:11

dstogov reviewed Aug 12, 2025

View reviewed changes

github-actions bot added the SAPI: phpdbg label Aug 14, 2025

arnaud-lb added 8 commits August 14, 2025 10:56

Tail call VM

3ca87ce

Return opline directly from helpers with extra args (remove zend_vm_t…

140a3cd

…rampoline)

ZEND_VM_TAIL_CALL_DISPATCH -> ZEND_VM_KIND_TAILCALL

0563418

Split ZEND_OPCODE_HANDLER_CCONV in two macros so we do not have to re…

09032b6

…-define it

Drop $variant, use $kind instead. Name tailcall handlers with _TAILCALL.

a7ebc3c

Generate CALL handlers first, then TAILCALL handlers

f11f508

Simplify dispatching from helpers with extra args

cc294ed

Remove IR_OPCODE_HANDLER_FUNC

3ff3588

Was a left over of relying on IR for preserve_none support

arnaud-lb added 14 commits August 14, 2025 10:57

Remove unnecessary case

169f6f1

Fix return type

8c780d1

fixup

79086d1

Revert "Simplify dispatching from helpers with extra args"

67c58ad

This reverts commit f0d41731921711cddea8dcc1d56ed16af5c2098b.

fixup phpdbg

c1e0d52

Clean generated code wrt ZEND_VM_DISPATCH_NOTAIL

7305116

Generate helpers with extra args, last

0c3515a

Remove ZEND_VM_DISPATCH_TO_* macros

707d942

Rework special casing of halt and leave helpers

764410c

Cleanup gen_null_handler/gen_halt_handler: The generate-once conditio…

780ac6f

…n is not necessary anymore

WS

ad283c4

Comment

1234e5c

Generated file

c5823ae

Lower fixed_stack_frame_size

e77d83d

arnaud-lb force-pushed the tailcall branch from 9139679 to e77d83d Compare August 14, 2025 10:19

		# define ZREG_FP 12 /* IR_REG_R12 */
		# define ZREG_IP 13 /* IR_REG_R13 */

Tail call VM [2] #18720

Are you sure you want to change the base?

Tail call VM [2] #18720

Uh oh!

Conversation

arnaud-lb commented May 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Non-dispatching opcode handlers

Fixed regs and preserved regs

Benchmarks:

Issues

UPGRADING

TODO

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

arnaud-lb commented Jun 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dstogov left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

arnaud-lb commented Jul 26, 2025

Uh oh!

dstogov left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

arnaud-lb commented May 31, 2025 •

edited

Loading

arnaud-lb commented Jun 2, 2025 •

edited

Loading