[Offload][Conformance] Add README file #155190
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@llvm/pr-subscribers-offload Author: Leandro Lacerda (leandrolcampos) ChangesThis patch introduces a The goal of this document is to provide clear and thorough instructions for new users and future contributors. It covers the project's purpose, system requirements, build and execution steps, testing methodology, and overall architecture. Patch is 31.54 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/155190.diff 1 Files Affected:
diff --git a/offload/unittests/Conformance/README.md b/offload/unittests/Conformance/README.md
new file mode 100644
index 0000000000000..91010b420cf96
--- /dev/null
+++ b/offload/unittests/Conformance/README.md
@@ -0,0 +1,810 @@
+# GPU Math Conformance Tests
+
+## Overview
+
+This test suite provides a framework to systematically measure the accuracy of math functions on GPUs and verify their conformance with standards like OpenCL.
+
+While the primary focus is validating the implementations in the C standard math library (LLVM-libm), these tests can also be executed against other math library providers, such as CUDA Math and HIP Math, for comparison.
+
+The goals of this project are to empower LLVM-libm contributors with a robust tool for validating their implementations and to build trust with end-users by providing transparent accuracy data.
+
+### Table of Contents
+
+- [Getting Started](#getting-started)
+- [Running the Tests](#running-the-tests)
+- [Test Results](#test-results)
+- [Adding New Tests](#adding-new-tests)
+- [Architecture Overview](#architecture-overview)
+
+## Getting Started
+
+This guide covers how to build the necessary dependencies, which include the new Offload API and the C standard library for both host and GPU targets.
+
+### System Requirements
+
+Before you begin, ensure your system meets the following requirements:
+
+* A system with an AMD or NVIDIA GPU.
+* The latest proprietary GPU drivers installed.
+* The corresponding development SDK for your hardware:
+ * **AMD:** [ROCm SDK](https://rocm.docs.amd.com)
+ * **NVIDIA:** [CUDA Toolkit](https://developer.nvidia.com/cuda-toolkit)
+
+
+### 1. Check out the LLVM project
+
+First, clone the LLVM project repository. For users who only need to build and run the tests, a shallow clone is sufficient and much faster.
+
+```bash
+git clone --depth=1 https://github.com/llvm/llvm-project.git
+```
+
+### 2. Configure the Build
+
+The recommended way to build the dependencies is to use the LLVM runtimes support. This approach automatically handles bootstrapping an up-to-date `clang` compiler and then uses it to build the `offload` and `libc` components.
+
+First, define an environment variable, `$LLVM_HOME`, to specify the desired installation directory for the toolchain and libraries.
+
+```bash
+export LLVM_HOME=/path/to/your/llvm/install
+```
+
+The command below provides an example configuration for a `Release` build. It specifically instructs the build system to compile `libc` for three distinct targets: the native host (`default`), AMD GPUs (`amdgcn-amd-amdhsa`), and NVIDIA GPUs (`nvptx64-nvidia-cuda`). You can customize options like `-DCMAKE_BUILD_TYPE` as needed.
+
+Execute this command from the root of the `llvm-project` directory:
+
+```bash
+cmake -S llvm -B build -G Ninja \
+ -DLLVM_ENABLE_PROJECTS="clang;lld" \
+ -DLLVM_ENABLE_RUNTIMES="openmp;offload;libc" \
+ -DCMAKE_BUILD_TYPE=Release \
+ -DLLVM_ENABLE_ASSERTIONS=ON \
+ -DLLVM_PARALLEL_LINK_JOBS=1 \
+ -DCMAKE_INSTALL_PREFIX="$LLVM_HOME" \
+ -DRUNTIMES_amdgcn-amd-amdhsa_LLVM_ENABLE_RUNTIMES=libc \
+ -DRUNTIMES_nvptx64-nvidia-cuda_LLVM_ENABLE_RUNTIMES=libc \
+ -DLLVM_RUNTIME_TARGETS="default;amdgcn-amd-amdhsa;nvptx64-nvidia-cuda"
+```
+
+### 3. Build and Install
+
+After configuring the build, compile and install the necessary dependencies using Ninja.
+
+```bash
+ninja -C build install -j8
+```
+
+> [!NOTE]
+> Running Ninja with high parallelism can cause spurious failures, out-of-resource errors, or indefinite hangs. Limit the number of jobs with `-j<N>` if you hit such issues.
+
+## Running the Tests
+
+### Default Test
+
+To build and run the conformance test for a given function (e.g., `logf`) against the default C standard math library `llvm-libm` provider, use the following command. This will execute the test on all available and supported platforms.
+
+```bash
+ninja -C build/runtimes/runtimes-bins offload.conformance.logf
+```
+
+### Testing Other Providers
+
+Once the test binary has been built, you can run it against other math library providers using the `--test-configs` flag.
+
+* **For `cuda-math` on an NVIDIA GPU:**
+
+ ```bash
+ ./build/runtimes/runtimes-bins/offload/logf.conformance --test-configs=cuda-math:cuda
+ ```
+
+* **For `hip-math` on an AMD GPU:**
+
+ ```bash
+ ./build/runtimes/runtimes-bins/offload/logf.conformance --test-configs=hip-math:amdgpu
+ ```
+
+You can also run all available configurations for a test with:
+
+```bash
+./build/runtimes/runtimes-bins/offload/logf.conformance --test-configs=all
+```
+
+## Test Results
+
+The results from the GPU are validated against reference values computed on the host CPU. These references are generated by the host's LLVM-libm, and the comparison is done by measuring the ULP (Units in the Last Place) distance. The pass/fail tolerances are typically based on the OpenCL C specification.
+
+All tests assume the host machine operates in the default floating-point environment with the round-to-nearest-even rounding mode.
+
+The results are generated using two distinct methodologies, chosen based on the size of the function's input space:
+
+* **Exhaustive Testing**: This method iterates over every representable point in the input space. It is used for half-precision functions and single-precision univariate functions, ensuring complete coverage of the input space.
+
+* **Randomized Testing**: This method is used for functions with larger input spaces, such as single-precision bivariate and double-precision functions, where exhaustive testing is computationally infeasible. Although not exhaustive, the tests are deterministic, using a fixed seed to sample a large, reproducible subset of points from the input space. The specific seed and sample size are configured in each test's source file, typically using 2<sup>32</sup> samples.
+
+---
+
+The following tables show the maximum observed ULP distance for each function compared against its host-side reference.
+
+### Exhaustive Test Results for Half-Precision Math Functions
+
+<table style="table-layout: fixed;">
+ <thead>
+ <tr>
+ <th rowspan="2" style="text-align: left; width: 150px;">Function</th>
+ <th rowspan="2" style="text-align: center;">ULP Tolerance</th>
+ <th colspan="4" style="text-align: center;">Max ULP Distance</th>
+ </tr>
+ <tr>
+ <th style="text-align: center;">llvm-libm<br>(AMDGPU)</th>
+ <th style="text-align: center;">llvm-libm<br>(CUDA)</th>
+ <th style="text-align: center;">cuda-math<br>(CUDA)</th>
+ <th style="text-align: center;">hip-math<br>(AMDGPU)</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left;">acosf16</td>
+ <td style="text-align: center;">2</td>
+ <td style="text-align: center;">1</td>
+ <td style="text-align: center;">1</td>
+ <td style="text-align: center;"></td>
+ <td style="text-align: center;">1</td>
+ </tr>
+ <tr>
+ <td style="text-align: left;">acoshf16</td>
+ <td style="text-align: center;">2</td>
+ <td style="text-align: center;">0</td>
+ <td style="text-align: center;">0</td>
+ <td style="text-align: center;"></td>
+ <td style="text-align: center;">0</td>
+ </tr>
+ <tr>
+ <td style="text-align: left;">acospif16</td>
+ <td style="text-align: center;">2</td>
+ <td style="text-align: center;">0</td>
+ <td style="text-align: center;">0</td>
+ <td style="text-align: center;"></td>
+ <td style="text-align: center;"></td>
+ </tr>
+ <tr>
+ <td style="text-align: left;">asinf16</td>
+ <td style="text-align: center;">2</td>
+ <td style="text-align: center;">0</td>
+ <td style="text-align: center;">0</td>
+ <td style="text-align: center;"></td>
+ <td style="text-align: center;">2</td>
+ </tr>
+ <tr>
+ <td style="text-align: left;">asinhf16</td>
+ <td style="text-align: center;">2</td>
+ <td style="text-align: center;">1</td>
+ <td style="text-align: center;">1</td>
+ <td style="text-align: center;"></td>
+ <td style="text-align: center;">1</td>
+ </tr>
+ <tr>
+ <td style="text-align: left;">atanf16</td>
+ <td style="text-align: center;">2</td>
+ <td style="text-align: center;">1</td>
+ <td style="text-align: center;">1</td>
+ <td style="text-align: center;"></td>
+ <td style="text-align: center;">1</td>
+ </tr>
+ <tr>
+ <td style="text-align: left;">atanhf16</td>
+ <td style="text-align: center;">2</td>
+ <td style="text-align: center;">0</td>
+ <td style="text-align: center;">0</td>
+ <td style="text-align: center;"></td>
+ <td style="text-align: center;">1</td>
+ </tr>
+ <tr>
+ <td style="text-align: left;">cosf16</td>
+ <td style="text-align: center;">2</td>
+ <td style="text-align: center;">1</td>
+ <td style="text-align: center;">1</td>
+ <td style="text-align: center;"></td>
+ <td style="text-align: center;">1</td>
+ </tr>
+ <tr>
+ <td style="text-align: left;">coshf16</td>
+ <td style="text-align: center;">2</td>
+ <td style="text-align: center;">1</td>
+ <td style="text-align: center;">0</td>
+ <td style="text-align: center;"></td>
+ <td style="text-align: center;">1</td>
+ </tr>
+ <tr>
+ <td style="text-align: left;">cospif16</td>
+ <td style="text-align: center;">2</td>
+ <td style="text-align: center;">0</td>
+ <td style="text-align: center;">0</td>
+ <td style="text-align: center;"></td>
+ <td style="text-align: center;"></td>
+ </tr>
+ <tr>
+ <td style="text-align: left;">expf16</td>
+ <td style="text-align: center;">2</td>
+ <td style="text-align: center;">1</td>
+ <td style="text-align: center;">1</td>
+ <td style="text-align: center;"></td>
+ <td style="text-align: center;">1</td>
+ </tr>
+ <tr>
+ <td style="text-align: left;">exp10f16</td>
+ <td style="text-align: center;">2</td>
+ <td style="text-align: center;">1</td>
+ <td style="text-align: center;">1</td>
+ <td style="text-align: center;"></td>
+ <td style="text-align: center;">1</td>
+ </tr>
+ <tr>
+ <td style="text-align: left;">exp2f16</td>
+ <td style="text-align: center;">2</td>
+ <td style="text-align: center;">1</td>
+ <td style="text-align: center;">1</td>
+ <td style="text-align: center;"></td>
+ <td style="text-align: center;">0</td>
+ </tr>
+ <tr>
+ <td style="text-align: left;">expm1f16</td>
+ <td style="text-align: center;">2</td>
+ <td style="text-align: center;">1</td>
+ <td style="text-align: center;">1</td>
+ <td style="text-align: center;"></td>
+ <td style="text-align: center;">1</td>
+ </tr>
+ <tr>
+ <td style="text-align: left;">hypotf16</td>
+ <td style="text-align: center;">2</td>
+ <td style="text-align: center;">0</td>
+ <td style="text-align: center;">0</td>
+ <td style="text-align: center;"></td>
+ <td style="text-align: center;"></td>
+ </tr>
+ <tr>
+ <td style="text-align: left;">logf16</td>
+ <td style="text-align: center;">2</td>
+ <td style="text-align: center;">1</td>
+ <td style="text-align: center;">1</td>
+ <td style="text-align: center;"></td>
+ <td style="text-align: center;">1</td>
+ </tr>
+ <tr>
+ <td style="text-align: left;">log10f16</td>
+ <td style="text-align: center;">2</td>
+ <td style="text-align: center;">1</td>
+ <td style="text-align: center;">1</td>
+ <td style="text-align: center;"></td>
+ <td style="text-align: center;">1</td>
+ </tr>
+ <tr>
+ <td style="text-align: left;">log2f16</td>
+ <td style="text-align: center;">2</td>
+ <td style="text-align: center;">1</td>
+ <td style="text-align: center;">1</td>
+ <td style="text-align: center;"></td>
+ <td style="text-align: center;">0</td>
+ </tr>
+ <tr>
+ <td style="text-align: left;">sinf16</td>
+ <td style="text-align: center;">2</td>
+ <td style="text-align: center;">1</td>
+ <td style="text-align: center;">1</td>
+ <td style="text-align: center;"></td>
+ <td style="text-align: center;">1</td>
+ </tr>
+ <tr>
+ <td style="text-align: left;">sinhf16</td>
+ <td style="text-align: center;">2</td>
+ <td style="text-align: center;">1</td>
+ <td style="text-align: center;">1</td>
+ <td style="text-align: center;"></td>
+ <td style="text-align: center;">1</td>
+ </tr>
+ <tr>
+ <td style="text-align: left;">sinpif16</td>
+ <td style="text-align: center;">2</td>
+ <td style="text-align: center;">0</td>
+ <td style="text-align: center;">0</td>
+ <td style="text-align: center;"></td>
+ <td style="text-align: center;"></td>
+ </tr>
+ <tr>
+ <td style="text-align: left;">tanf16</td>
+ <td style="text-align: center;">2</td>
+ <td style="text-align: center;">1</td>
+ <td style="text-align: center;">1</td>
+ <td style="text-align: center;"></td>
+ <td style="text-align: center;">2</td>
+ </tr>
+ <tr>
+ <td style="text-align: left;">tanhf16</td>
+ <td style="text-align: center;">2</td>
+ <td style="text-align: center;">0</td>
+ <td style="text-align: center;">0</td>
+ <td style="text-align: center;"></td>
+ <td style="text-align: center;">1</td>
+ </tr>
+ <tr>
+ <td style="text-align: left;">tanpif16</td>
+ <td style="text-align: center;">2</td>
+ <td style="text-align: center;">1</td>
+ <td style="text-align: center;">1</td>
+ <td style="text-align: center;"></td>
+ <td style="text-align: center;"></td>
+ </tr>
+ </tbody>
+</table>
+
+### Exhaustive Test Results for Single-Precision Univariate Math Functions
+
+<table style="table-layout: fixed;">
+ <thead>
+ <tr>
+ <th rowspan="2" style="text-align: left; width: 150px;">Function</th>
+ <th rowspan="2" style="text-align: center;">ULP Tolerance</th>
+ <th colspan="4" style="text-align: center;">Max ULP Distance</th>
+ </tr>
+ <tr>
+ <th style="text-align: center;">llvm-libm<br>(AMDGPU)</th>
+ <th style="text-align: center;">llvm-libm<br>(CUDA)</th>
+ <th style="text-align: center;">cuda-math<br>(CUDA)</th>
+ <th style="text-align: center;">hip-math<br>(AMDGPU)</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left;">acosf</td>
+ <td style="text-align: center;">4</td>
+ <td style="text-align: center;">1</td>
+ <td style="text-align: center;">1</td>
+ <td style="text-align: center;">1</td>
+ <td style="text-align: center;">1</td>
+ </tr>
+ <tr>
+ <td style="text-align: left;">acoshf</td>
+ <td style="text-align: center;">4</td>
+ <td style="text-align: center;">1</td>
+ <td style="text-align: center;">1</td>
+ <td style="text-align: center;">2</td>
+ <td style="text-align: center;">1</td>
+ </tr>
+ <tr>
+ <td style="text-align: left;">asinf</td>
+ <td style="text-align: center;">4</td>
+ <td style="text-align: center;">1</td>
+ <td style="text-align: center;">1</td>
+ <td style="text-align: center;">1</td>
+ <td style="text-align: center;">3</td>
+ </tr>
+ <tr>
+ <td style="text-align: left;">asinhf</td>
+ <td style="text-align: center;">4</td>
+ <td style="text-align: center;">1</td>
+ <td style="text-align: center;">1</td>
+ <td style="text-align: center;">2</td>
+ <td style="text-align: center;">1</td>
+ </tr>
+ <tr>
+ <td style="text-align: left;">atanf</td>
+ <td style="text-align: center;">5</td>
+ <td style="text-align: center;">0</td>
+ <td style="text-align: center;">0</td>
+ <td style="text-align: center;">1</td>
+ <td style="text-align: center;">2</td>
+ </tr>
+ <tr>
+ <td style="text-align: left;">atanhf</td>
+ <td style="text-align: center;">5</td>
+ <td style="text-align: center;">0</td>
+ <td style="text-align: center;">0</td>
+ <td style="text-align: center;">3</td>
+ <td style="text-align: center;">1</td>
+ </tr>
+ <tr>
+ <td style="text-align: left;">cbrtf</td>
+ <td style="text-align: center;">2</td>
+ <td style="text-align: center;">0</td>
+ <td style="text-align: center;">0</td>
+ <td style="text-align: center;">1</td>
+ <td style="text-align: center;">1</td>
+ </tr>
+ <tr>
+ <td style="text-align: left;">cosf</td>
+ <td style="text-align: center;">4</td>
+ <td style="text-align: center;">1</td>
+ <td style="text-align: center;">1</td>
+ <td style="text-align: center;">2</td>
+ <td style="text-align: center;">2</td>
+ </tr>
+ <tr>
+ <td style="text-align: left;">coshf</td>
+ <td style="text-align: center;">4</td>
+ <td style="text-align: center;">0</td>
+ <td style="text-align: center;">0</td>
+ <td style="text-align: center;">2</td>
+ <td style="text-align: center;">1</td>
+ </tr>
+ <tr>
+ <td style="text-align: left;">cospif</td>
+ <td style="text-align: center;">4</td>
+ <td style="text-align: center;">0</td>
+ <td style="text-align: center;">0</td>
+ <td style="text-align: center;">1</td>
+ <td style="text-align: center;">1</td>
+ </tr>
+ <tr>
+ <td style="text-align: left;">erff</td>
+ <td style="text-align: center;">16</td>
+ <td style="text-align: center;">0</td>
+ <td style="text-align: center;">0</td>
+ <td style="text-align: center;">1</td>
+ <td style="text-align: center;">2</td>
+ </tr>
+ <tr>
+ <td style="text-align: left;">expf</td>
+ <td style="text-align: center;">3</td>
+ <td style="text-align: center;">0</td>
+ <td style="text-align: center;">0</td>
+ <td style="text-align: center;">2</td>
+ <td style="text-align: center;">1</td>
+ </tr>
+ <tr>
+ <td style="text-align: left;">exp10f</td>
+ <td style="text-align: center;">3</td>
+ <td style="text-align: center;">0</td>
+ <td style="text-align: center;">0</td>
+ <td style="text-align: center;">2</td>
+ <td style="text-align: center;">1</td>
+ </tr>
+ <tr>
+ <td style="text-align: left;">exp2f</td>
+ <td style="text-align: center;">3</td>
+ <td style="text-align: center;">1</td>
+ <td style="text-align: center;">1</td>
+ <td style="text-align: center;">2</td>
+ <td style="text-align: center;">1</td>
+ </tr>
+ <tr>
+ <td style="text-align: left;">expm1f</td>
+ <td style="text-align: center;">3</td>
+ <td style="text-align: center;">1</td>
+ <td style="text-align: center;">1</td>
+ <td style="text-align: center;">1</td>
+ <td style="text-align: center;">1</td>
+ </tr>
+ <tr>
+ <td style="text-align: left;">logf</td>
+ <td style="text-align: center;">3</td>
+ <td style="text-align: center;">1</td>
+ <td style="text-align: center;">1</td>
+ <td style="text-align: center;">1</td>
+ <td style="text-align: center;">2</td>
+ </tr>
+ <tr>
+ <td style="text-align: left;">log10f</td>
+ <td style="text-align: center;">3</td>
+ <td style="text-align: center;">1</td>
+ <td style="text-align: center;">1</td>
+ <td style="text-align: center;">2</td>
+ <td style="text-align: center;">2</td>
+ </tr>
+ <tr>
+ <td style="text-align: left;">log1pf</td>
+ <td style="text-align: center;">2</td>
+ <td style="text-align: center;">1</td>
+ <td style="text-align: center;">1</td>
+ <td style="text-align: center;">1</td>
+ <td style="text-align: center;">1</td>
+ </tr>
+ <tr>
+ <td style="text-align: left;">log2f</td>
+ <td style="text-align: center;">3</td>
+ <td style="text-align: center;">0</td>
+ <td style="text-align: center;">0</td>
+ <td style="text-align: center;">1</td>
+ <td style="text-align: center;">1</td>
+ </tr>
+ <tr>
+ <td style="text-align: left;">sinf</td>
+ <td style="text-align: center;">4</td>
+ <td style="text-align: center;">1</td>
+ <td style="text-align: center;">1</td>
+ <td style="text-align: center;">1</td>
+ <td...
[truncated]
|
jhuber6
reviewed
Aug 25, 2025
There was a problem hiding this comment.
Most of this is documentation, I'd prefer that go with the libc documentation somewhere. Maybe @lntue knows a good place for that to live. Building instructions should link to offload / libc. This should probably just be a description of how to run the tests.
Thanks for the feedback! I've already pushed an update to address your point about the build instructions. The "Getting Started" section now links to the official libc for GPUs documentation. |
Yes, I think especially the test results section should be moved to https://libc.llvm.org/gpu/index.html under a new |
|
I've pushed a commit that removes the "Test Results" section. Per our discussion, these results will be moved to a new I have also removed the "Architecture Overview" section to keep the document more focused. |
jhuber6
approved these changes
Aug 26, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This patch introduces a
README.mdfile for the GPU math conformance test suite located inoffload/unittests/Conformance.The goal of this document is to provide clear and thorough instructions for new users and future contributors. It covers the project's purpose, system requirements, build and execution steps, testing methodology, and overall architecture.