I have wrapped the shared texture handle to VideoFrame object successfully.

However, several major issue blocks:

1. The API needs to be in main world of renderer process (which I don't see a working sample in Electron), and VideoFrame doesn't works well with contextBridge. Currently I disabled contextIsolation and enabled nodeIntegration for testing.

2. The VideoFrame wants to take ownership of the handles we provided, on macOS and Linux this is just fine, macOS can retain the IOSurface (ref +1) and Linux calls dup on dma buf fd. But on Windows is can't DuplicateHandle because there's no handle to source process, we don't even know the pid either. Currently I slightly modified the DXGIHandle to prevent it takes ownership.

3. The generated VideoFrame is a GPU frame, when I tried to render to canvas, import to WebGPU, currently Chromium always tries to map to CPU memory to read pixels, and fail.

4. There's a zero copy path, which needs a SharedImage and now only supports NV12 for unknown reason. The SharedImage needs to create with a GPU Service, which seems hard to get one from anyehere.

5. I expect more Chromium patches, and these might not be able to merge to upstream.

Update:

I now got the SharedImageInterface from SharedGpuContext, but when I'm trying to call CreateSharedImage, the Renderer Process crashed without any debug message. Trying to use a debug build with symbol_level 2 to check again.

😞 Bad news:

SharedImage doesn't work well with this hack. It requires taking ownership of handles, and the renderer process seems having some issue with duplication.

😄 Good news:

Works with WebGPU (with hack)

It becomes more tricky, because the HANDLE on Windows has some internal design at Chromium:

When passing through ipcz, it DuplicateHandle automatically for target process. However it seems causing some error for D3D12 to import (mojo\core\ipcz_driver\transport.cc:221)

I love the good news 😄 ! What do you mean by 'hack'?
And... thank you for working on this!

😄 Good news:

I found hacking WebGPU also very ugly, so I truned my route back to VideoFrame.

After days of debugging, I found managing the handle on Windows very tricky, for example, when an FrameSinkVideoCapturer's output is first CreateSharedHandle on GPU process, and when it pass through IPC, the ipcz automatically DuplicateHandle for target process. Now we are at Main process, we need to pass the handle to Renderer process, but this time, we have no ipcz does the DuplicateHandle thing for us.

I then add a extra param to let user tell the handle's owner process id, and DuplicateHandle it internally to Renderer process. Later, the ipcz will clone the handle for us when passing it back to GPU Process. That's also the reason why Renderer process was crashing before (without any debugging message!), the handle belongs to Main process and it failed to duplicate the handle, making ipc serialization failure.

For other shared handle, user needs either use deprecated GetSharedHandle to create a non-NT handle, or pass the handle owner's pid, in order to successfully import handle on Windows.

At this step, I successfully imported as a VideoFrame, and successfully call WebGPU's importExternalTexture and get a valid object. But I found the rendering has some problem (full black). After some debugging, I found media::VideoFrame::WrapSharedImage will not take own of SharedImage (frame->shared_image_ = shared_image->MakeUnowned();), and cause it destroy immediately when scoped_refptr release. I now manually bind to ReleaseCallback and it works~!

If you're interested, here's a working sample: https://github.com/reitowo/electron-test-import-texture

That's huge! Thank you @reitowo

This now works like a charm in Windows.

The macOS's IOSurface also needs to pass by mach_port across processes. https://source.chromium.org/chromium/chromium/src/+/main:mojo/core/channel_mac.cc;drc=c7beb80e1c9ed11289a501218de27681143b9ba9;l=388

Passing mach_port is much more complicated than I thought. Maybe still need to think twice about the API.

I think the best option is to import at Main process, and use Mailbox to pass between processes in Chromium, which is same pattern in Chromium anyway.

I've done everything to make it work on Windows and macOS! Due to lack of development environment I choose drop Linux support for now.

Test also included, run npm run test -- -g sharedTexture to see. (It will fail (skipped) on CI for lack of GPU, except mac-arm64 which I think have a M series chip)

I'm also happy about current API design, especially zero upstream patch.

Looking forward to hear your advice! @codebytere @nikwen @MarshallOfSound

Binary for testing: https://github.com/reitowo/electron/releases/download/import-texture-v1/dist.zip

Working sample: https://github.com/reitowo/electron-test-import-texture

I found a bug when writing test on macOS that nativeImage was not able to compare pixel data correctly #46949. So I changed the compare method in test.

This is extremely cool @reitowo! The API looks great and should match my use case perfectly - I'll try this out soon

We successfully implemented the new feature of this PR with import of external resource on MacOS using Syphon benoitlahoz/node-syphon#43. I also attempted to replicate the same functionality on Windows with Spout, but ran into issues. While it initially works for a few seconds, it quickly fails. likely due to a problem on the Spout side. I'm seeing the following error: Detected dangling raw_ptr in unretained with.... (with the same example and proper cleanup and release of resources on the electron side)

Out of curiosity, have you tried implementing external read access via Spout, similar to your write approach in your repo https://github.com/reitowo/electron-spout @reitowo?

I'd love to assist with testing edge cases, as I’m very invested in the success of this PR.

@gase12 As a contributor to Three.js, I can confirm that unless you're using WebGPU and VideoFrame with importExternalTexture, you’ll likely see significant CPU overhead.

It will be helpful if you share more logs.

I tried to reproduce the issue using https://github.com/reitowo/electron-test-import-texture and everything seems to work fine there. So it looks like the problem is specific to my application setup. Just to reiterate, there's nothing to report against your PR. Everything is functioning well on both Windows and macOS after nearly two weeks of testing. Great work! 👍

On Windows the maximum amount of pixel is limited to 2 millions, which automatically resize any shared texture that would have more than that amount of pixel which is very low:

// init with 2560x1440 for example:
2560 1440 { x: 0, y: 0, width: 2560, height: 1440 }
{
  release: [Function (anonymous)],
  textureInfo: {
    pixelFormat: 'bgra',
    codedSize: { width: 1920, height: 1032 }, // here size to 1920x1032
    visibleRect: { x: 0, y: 0, width: 1920, height: 1032 },
    contentRect: { x: 0, y: 0, width: 1920, height: 1032 },
    timestamp: 1383278,
    widgetType: 'frame',
    metadata: {
      captureUpdateRect: [Object],
      regionCaptureRect: null,
      sourceSize: [Object],
      frameCount: 1
    },
    sharedTextureHandle: <Buffer 1c 19 00 00 00 00 00 00>
  }
} {
  pixelFormat: 'bgra',
  codedSize: { width: 1920, height: 1032 },
  visibleRect: { x: 0, y: 0, width: 1920, height: 1032 },
  contentRect: { x: 0, y: 0, width: 1920, height: 1032 },
  timestamp: 1383278,
  widgetType: 'frame',
  metadata: {
    captureUpdateRect: { x: 0, y: 0, width: 1920, height: 1032 },
    regionCaptureRect: null,
    sourceSize: { width: 1920, height: 1032 },
    frameCount: 1
  },
  sharedTextureHandle: <Buffer 1c 19 00 00 00 00 00 00>
}

The issue is easily testable on https://github.com/reitowo/electron-test-import-texture by simplying making a texture bigger than 1920x1032.

I believe we could bypass that limitation with something like this in osr_video_consumer.cc

void OffScreenVideoConsumer::OnFrameCaptured(
    ::media::mojom::VideoBufferHandlePtr data,
    ::media::mojom::VideoFrameInfoPtr info,
    const gfx::Rect& content_rect,
    mojo::PendingRemote<viz::mojom::FrameSinkVideoConsumerFrameCallbacks>
        callbacks) {
  // Explicitly provide feedback to indicate maximum capability.
  // This is done before 'callbacks' is moved further along.
  if (callbacks) {
    mojo::Remote<viz::mojom::FrameSinkVideoConsumerFrameCallbacks> feedback_sender(
        std::move(callbacks));

    media::VideoCaptureFeedback feedback;
    feedback.max_pixels = std::numeric_limits<int>::max();
    feedback.resource_utilization = 0.0;  // Indicate N/A or no constraint
    feedback_sender->ProvideFeedback(feedback);

    // Get the PendingRemote back to be used by the rest of the function.
    callbacks = feedback_sender.Unbind();
  }

What do you think @reitowo?

Can't reproduce your problem. I could easily get 1920x1080 (150% scale to 2880x1620)

{
  release: [Function (anonymous)],
  textureInfo: {
    pixelFormat: 'bgra',
    codedSize: { width: 2880, height: 1620 },
    visibleRect: { x: 0, y: 0, width: 2880, height: 1620 },
    contentRect: { x: 0, y: 0, width: 2880, height: 1620 },
    timestamp: 116518854,
    colorSpace: {
      primaries: 'bt709',
      transfer: 'srgb',
      matrix: 'rgb',
      range: 'full'
    },
    widgetType: 'frame',
    metadata: {
      captureUpdateRect: [Object],
      regionCaptureRect: null,
      sourceSize: [Object],
      frameCount: 2589
    },
    sharedTextureData: { ntHandle: <Buffer 9c 11 00 00 00 00 00 00> }
  }
}

And the feedback should be useless as it will make a call to VideoCaptureOracle::RecordConsumerFeedback if you explicitly call it. If no call is made the oracle should have no limit. We already call VideoCaptureOracle::SetAutoThrottlingEnabled(bool enabled) to disable the capture limit.

It seems to be a GPU limitation. I got one of the latest AMD (Radeon RX 7900 XTX) and these GPU often seems to be very limited on Shared Texture.

• NVIDIA GPUs generally allow for large shared textures (up to 16K x 16K or more), making inter-process shared rendering or GPU-based IPC (inter-process communication) more flexible.
• AMD GPUs on Windows often hit a limit around 2 million pixels for shared resources. This translates roughly to a 1920x1080 texture or similar.

This limitation is not documented by Microsoft, but widely observed and reported by developers working with Direct3D interop or real-time rendering systems like game engines, media pipelines, and remote desktop frameworks.

About performance: I created 16 browser 720P OSR send to 1 renderer process, and tested both use 16 canvas or 1 combined canvas render at and the results are same (16x 50fps). On my 3070 we can achieve:

The GPU process's CPU usage is high, I don't know how to optimize further, the GPU usage is not high.

@RenaudRohlinger Could you share more links to the reports by community? Sounds interesting.

Is it possible the video frame is going through the same path people are having issues with in https://issues.chromium.org/issues/406357270 for recent AMD GPUs? You could try passing the flags mentioned in that issue to check if it helps.

Thanks for the reference.

Yes, here those VideoProcessorGetOutputExtension … 0x80070057 errors seems to be
exactly what Chromium issue 406357270 tracks. They mean the driver can’t
open a hardware-secure video processor, so Chromium falls back to the
“clear” path and clamps every shared texture to 2 097 152 px (~1080p).

I tried the combo:

--disable-gpu-driver-bug-workarounds
--ignore-gpu-blocklist
--enable-features=HardwareSecureDecryption:force_support_clear_lead/true

In Chrome this lifts the limit only if the driver is actually able to
create a secure surface. On the affected RX 7900 / 6900 drivers it still
fails, the same 0x80070057 shows up, and the 2-MP cap remains.
So the flags are a good diagnostic step, but they don’t cure the bug.

The OSR relies on the VideoCapture path, so this bug affected us. I think since we can import external texture as a SharedImage we may someday no more rely on VideoConsumer and make our own OSR host (in the future)

I see. In the meantime, do you have a patch in mind for AMD, or is there nothing Electron can do here?

No, I have no AMD GPU on hand. And I think FrameSinkVideoCapturer doesn't involve video encode/decode, if it does, probably it is the oracle's bad again. It will be helpful if you can run with --no-sandbox and get some detailed log files to see if there's more error message.

Only error I have when logging with --no-sandbox:

source: node:electron/js2c/sandbox_bundle (8414)
[9196:0522/145414.813:VERBOSE1:media\gpu\windows\mf_video_encoder_shared_state.cc:145] Hardware encode acceleration is not available for vp9
[9196:0522/145414.969:ERROR:media\gpu\windows\mf_video_encoder_util.cc:552] Set output type failed, The parameter is incorrect. (0x80070057) 
{
  release: [Function (anonymous)],
  textureInfo: {
    pixelFormat: 'bgra',
    codedSize: { width: 1920, height: 1032 },
    visibleRect: { x: 0, y: 0, width: 1920, height: 1032 },
    contentRect: { x: 0, y: 0, width: 1920, height: 1032 },
    timestamp: 0,
    colorSpace: {
      primaries: 'bt709',
      transfer: 'srgb',
      matrix: 'rgb',
      range: 'full'
    },
    widgetType: 'frame',
    metadata: {
      captureUpdateRect: [Object],
      regionCaptureRect: null,
      sourceSize: [Object],
      frameCount: 0,
      isWebGpuCompatible: false
    },
    sharedTextureHandle: <Buffer 18 0a 00 00 00 00 00 00>
  }

Related source code:
https://github.com/chromium/chromium/blob/main/media/gpu/windows/mf_video_encoder_shared_state.cc#L144-L146
https://github.com/chromium/chromium/blob/main/media/gpu/windows/mf_video_encoder_util.cc#L546-L547

Hello @reitowo,

I did some extensive investigation on this issue, I even went as far as purchasing an RTX GPU to confirm whether the problem was related to AMD. To my surprise, it turns out the issue wasn't GPU-related at all, but display-related.

My current physical display is limited to a resolution of 1920x1080 with a device pixel ratio (DPR) of 1. Because of this, the capture API appears to be constrained by the physical resolution of the display, meaning it cannot exceed what the screen supports. This limitation seems to be the root cause of the problem.

I believe you haven’t encountered this issue because your display likely supports a higher resolution than mine.

Here are some improved logs that highlight the limitation:

[42300:0529/113219.677:VERBOSE1:components\device_event_log\device_event_log_impl.cc:200] [11:32:19.675] Display: EVENT: screen_win.cc:989 Displays updated, count: 1
[42300:0529/113219.677:VERBOSE1:components\device_event_log\device_event_log_impl.cc:200] [11:32:19.677] Display: EVENT: screen_win.cc:991 Display[4028745484] bounds=[0,0 1920x1080], workarea=[0,0 1920x1032], 

[46736:0529/113220.363:VERBOSE1:media\capture\content\capture_resolution_chooser.cc:205] Recomputed snapped frame sizes: 78x42 <--> 245x132 <--> 413x222 <--> 580x312 <--> 747x402 <--> 915x492 <--> 1082x582 <--> 1250x672 <--> 1417x762 <--> 1585x852 <--> 1752x942 <--> 1920x1032
[46736:0529/113220.363:VERBOSE1:media\capture\content\capture_resolution_chooser.cc:157] Recomputed capture size from 640x360 to 1920x1032 (100% of ideal size)
{
  release: [Function (anonymous)],
  textureInfo: {
    pixelFormat: 'bgra',
    codedSize: { width: 1920, height: 1032 },
    visibleRect: { x: 0, y: 0, width: 1920, height: 1032 },
    contentRect: { x: 0, y: 0, width: 1920, height: 1032 },
    timestamp: 0,
    colorSpace: {
      primaries: 'bt709',
      transfer: 'srgb',
      matrix: 'rgb',
      range: 'full'
    },
    widgetType: 'frame',
    metadata: {
      captureUpdateRect: [Object],
      regionCaptureRect: null,
      sourceSize: [Object],
      frameCount: 0
    },
    sharedTextureData: { ntHandle: <Buffer d4 0c 00 00 00 00 00 00> }
 
  }
}

Wow, nice catch! I do using a 4K monitor, and I successfully reproduced your problem. I'll see how to bypass that.

I think by design this should take effect and no constraint should take effect. I'll take a look tomorrow as my Chromium build went invalid :(

Oh, it suddenly become reasonable if electron somehow limits the width and height to the working area, and the resolution_chooser->SetSourceSize will be the final decision as the source frame is just 1920*1032, there's no point to be larger than that.

Yes, can be verified if you try calling osr.setSize(1920, 1080). You'll see the output is now correct. For me, it's 4K with 150% pixel ratio.

Indeed it now works! Thank you for finding this. I think it would be worth it mentioning this detail in the documentation.

Hi, guys.

This PR will be seperated for several minor PRs, and the PR is converted to an RFC for better outcome. electron/rfcs#17. I'll link all minor PRs in this PR for reference. Let's continue keep discussion here!

I've done everything to make it work on Windows and macOS! Due to lack of development environment I choose drop Linux support for now.

Test also included, run npm run test -- -g sharedTexture to see. (It will fail (skipped) on CI for lack of GPU, except mac-arm64 which I think have a M series chip)

I'm also happy about current API design, especially zero upstream patch.

Looking forward to hear your advice! @codebytere @nikwen @MarshallOfSound

Binary for testing: https://github.com/reitowo/electron/releases/download/import-texture-v1/dist.zip

Working sample: https://github.com/reitowo/electron-test-import-texture

output.mp4

Hi @reitowo

Thank you for your work. It’s exactly what we need! I would like to try your test project, but I encountered a 'not found' error with the download link. Could you please provide the latest compiled binary so that we can run the example?

Thank you!

Close this in favor of the last piece of the PR.