Skip to content

threading: support for GGML_SCHED_PRIO_LOW, update thread info on Windows to avoid throttling #12995

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
May 31, 2025
Merged

threading: support for GGML_SCHED_PRIO_LOW, update thread info on Windows to avoid throttling #12995

merged 3 commits into from
May 31, 2025

Conversation

max-krasnyansky
Copy link
Collaborator

We talked about adding LOW priority for GGML threads in the original threadpool PR. It might be useful for some cases to avoid contention.

Latest Windows ARM64 releases started parking (offlining) the CPU cores more aggresively which results in suboptimal performance with n_threads > 4. To deal with that we now disable Power Throttling for our threads for the NORMAL and higher priorities.

Make sure to read the contributing guidelines before submitting a PR

@github-actions github-actions bot added examples ggml changes relating to the ggml tensor library for machine learning labels Apr 17, 2025
@max-krasnyansky max-krasnyansky mentioned this pull request Apr 17, 2025
Open
@max-krasnyansky max-krasnyansky force-pushed the maxk/sched-prio-updates branch 2 times, most recently from bced6ad to b4458d4 Compare April 17, 2025 16:45
slaren
slaren reviewed Apr 17, 2025
View reviewed changes
@max-krasnyansky max-krasnyansky force-pushed the maxk/sched-prio-updates branch from a157d1b to 854252f Compare April 17, 2025 21:17
@max-krasnyansky max-krasnyansky marked this pull request as ready for review April 17, 2025 21:34
@max-krasnyansky max-krasnyansky force-pushed the maxk/sched-prio-updates branch 2 times, most recently from 9d4b917 to f155d23 Compare April 18, 2025 20:44
@slaren
Copy link
Member

slaren commented Apr 22, 2025

The CI failure here is a bit weird.. it is actually using mingw rather than MSVC, and the problem seems to be that this version of mingw does not support this API at all. We should just remove the msvc-arm64 build from the CI, along with the arm64-windows-msvc.cmake toolchain file, since it doesn't work, and this configuration is no longer supported.

max-krasnyansky and others added 2 commits May 30, 2025 17:15
@max-krasnyansky @slaren
threading: support for GGML_SCHED_PRIO_LOW, update thread info on Win…
199a838 
…dows to avoid throttling

We talked about adding LOW priority for GGML threads in the original threadpool PR.
It might be useful for some cases to avoid contention.

Latest Windows ARM64 releases started parking (offlining) the CPU cores
more aggresively which results in suboptimal performance with n_threads > 4.
To deal with that we now disable Power Throttling for our threads for the NORMAL
and higher priorities.

Co-authored-by: Diego Devesa <slarengh@gmail.com>
@max-krasnyansky
threading: disable SetThreadInfo() calls for older Windows versions
9087dd2
@max-krasnyansky max-krasnyansky force-pushed the maxk/sched-prio-updates branch from f155d23 to 9087dd2 Compare May 31, 2025 00:47
@max-krasnyansky
Copy link
Collaborator Author

max-krasnyansky commented May 31, 2025 •
edited
Loading

The CI failure here is a bit weird.. it is actually using mingw rather than MSVC, and the problem seems to be that this version of mingw does not support this API at all. We should just remove the msvc-arm64 build from the CI, along with the arm64-windows-msvc.cmake toolchain file, since it doesn't work, and this configuration is no longer supported.

Looks like CI is good now after you guys nuked msvc builds.
@slaren please approve when you get the chance

@max-krasnyansky max-krasnyansky self-assigned this May 31, 2025
slaren
slaren approved these changes May 31, 2025
View reviewed changes
@max-krasnyansky @slaren
Update tools/llama-bench/llama-bench.cpp
ac35e50 
Co-authored-by: Diego Devesa <slarengh@gmail.com>
@max-krasnyansky max-krasnyansky merged commit 053b153 into master May 31, 2025
46 checks passed
@MaggotHATE
Copy link
Contributor

MaggotHATE commented Jun 1, 2025 •
edited
Loading

I'm getting compiling error after this PR. Windows 10, i7 8700, MinGW (w64devkit):

ggml-cpu.c:2434:9: error: unknown type name 'THREAD_POWER_THROTTLING_STATE'; did you mean 'PPROCESS_POWER_THROTTLING_STATE'?

Is there a check missing?

UPD: yes, it's missing a simple check, like:

#if defined(__GNUC__)
// MinGW doesn't support THREAD_POWER_THROTTLING_CURRENT_VERSION and THREAD_POWER_THROTTLING_EXECUTION_SPEED
#else
...

@LostRuins
Copy link
Collaborator

LostRuins commented Jun 2, 2025 •
edited
Loading

#if _WIN32_WINNT >= 0x0602

If i am not wrong, this is set at compile time, which means that this will break windows 7 compatibility. Could it be changed to a runtime check instead?

Edit: Quick search says that SetThreadInformation fails gracefully on unsupported systems, not entirely sure though.

Also, why not use PROCESS_POWER_THROTTLING_CURRENT_VERSION which seems more widely available.

@ggerganov ggerganov mentioned this pull request Jun 5, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@slaren slaren slaren approved these changes

Assignees

@max-krasnyansky max-krasnyansky

Labels
examples ggml changes relating to the ggml tensor library for machine learning
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@max-krasnyansky @slaren @MaggotHATE @LostRuins