Hi @JohannesGaessler .Based on your helpful suggestions from our last discussion, I've updated the logic. Would you mind reviewing this revised version to confirm if it meets the requirements? Thanks for your time and advice.

Did you confirm that the code works correctly both when GGML_CUDA_ENABLE_UNIFIED_MEMORY is and isn't set?

Should we check something like canMapHostMemory / canUseHostPointerForRegisteredMem in addition to integrated before letting the GPU directly access buffer allocated by cudaMallocHost? The integrated flag doesn't seem to cover all the prerequisites of such access.

Thank you very much for your professional modification of my code.As this is my first contribution, I apologize for any oversight on my part. @JohannesGaessler

Did you confirm that the code works correctly both when GGML_CUDA_ENABLE_UNIFIED_MEMORY is and isn't set?

Regarding the point you raised, I believe toggling GGML_CUDA_ENABLE_UNIFIED_MEMORY on/off shouldn't impact this particular change, as it appears to only affect the device_memory allocation API for 'ggml_cuda_device_malloc', which remains unrelated to host memory(pinned memory) logical reasoning,as show in: related_source_code. If my understanding is flawed here, I'd greatly appreciate your correction. Thank you for your guidance.

However, I've observed that enabling CUDA Graph on Jetson devices triggers an assertion failure related to 'buft'. I'm currently diagnosing and addressing this issue, so this branch may not yet ready for integration

Should we check something like canMapHostMemory / canUseHostPointerForRegisteredMem in addition to integrated before letting the GPU directly access buffer allocated by cudaMallocHost? The integrated flag doesn't seem to cover all the prerequisites of such access.

I did consider this aspect previously. Based on my understanding, mapping or register 'host memory' would fall under the purview of 'buffer_from_host_ptr' functionality. While it intersects conceptually with 'host_buffer' ,but the two represent distinct mechanismsas. as we can see:

struct ggml_backend_dev_caps {
        // asynchronous operations
        bool async;
        // pinned host buffer
        bool host_buffer;
        // creating buffers from host ptr
        bool buffer_from_host_ptr;
        // event synchronization
        bool events;
    };

And one thing should note: The device's ability to map/register host pointers (pageable memory) for cuda_device may require 'I/O Coherency' support consideration, whereas accessing pinned host buffers doesn't share this requirement.

The current implementation appears to lack support for 'buffer_from_host_ptr' on all CUDA devices. Enabling this on integrated GPUs would likely necessitate significant code modifications.I don't have a good idea to add this support yet.

Of course, my understanding of the above concepts may not be very accurate. If I have fallen into some misunderstanding, please discuss it with me.

Hello, @JohannesGaessler .I apologize for bothering you again. I've pushed a new commit (bd21613) to this PR to fix the assert issue encountered when running CUDA Graph on the integrateGPU device.

For this latest commit, I have tested and verified it on both device types: dGPU (NVIDIA RTX 4060) and iGPU (Jetson Orin). I also tested with both the 'cuda_Graph' and 'GGML_CUDA_ENABLE_UNIFIED_MEMORY' flags enabled and disabled. All tests now pass. Could you please check the PR now and see if anything else needs attention?

Should we check something like canMapHostMemory / canUseHostPointerForRegisteredMem in addition to integrated before letting the GPU directly access buffer allocated by cudaMallocHost? The integrated flag doesn't seem to cover all the prerequisites of such access.

In the context of ggml, a "CUDA host buffer" is always allocated as pinned memory. To my understanding, if canMapHostMemory is false, then the allocation of pinned memory will fail and integrated && ggml_backend_buft_is_cuda_host(buft)) will evaluate to false.

hi, @JohannesGaessler .After testing the latest suggest commit on my jetson device, I found that assert would be triggered.as we can see:

so i revoke this change: GGML_ASSERT(!integrated || prop.canUseHostPointerForRegisteredMem);

I seem to have found the answer based on this post：link.

So in dGPU, this flag is 1, and in arm Soc such as jetson is 0. It seems that must use 'cudaHostGetDevicePointer' to get the pointer

For the latest commit 63db683 , I have tested it on my jetson, and both the opening and closing of flag 'GGML_CUDA_ENABLE_UNIFIED_MEMORY' has passed the test