github · ethanpalm · Sep 16, 2021 · Sep 10, 2021 · Sep 13, 2021 · Sep 13, 2021
@@ -228,6 +228,123 @@ commands that you can specify for compiled languages.
   This command runs a custom script that contains all of the commands required
   to build the project.
 
+Using indirect build tracing
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If the CodeQL CLI autobuilders for compiled languages do not work with your CI workflow and you cannot wrap invocations of build commands with ``codeql database trace-command``, you can use indirect build tracing to create a CodeQL database. To use indirect build tracing, your CI system must be able to set custom environment variables for each build action.
-If the CodeQL CLI autobuilders for compiled languages do not work with your CI workflow and you cannot wrap invocations of build commands with ``codeql database trace-command``, you can use indirect build tracing to create a CodeQL database. To use indirect build tracing, your CI system must be able to set custom environment variables for each build action.
+If the CodeQL CLI autobuilders for compiled languages do not work with your CI workflow and you cannot specify build commands with using the ``--command`` option, you can use indirect build tracing to create a CodeQL database. To use indirect build tracing, your CI system must be able to set custom environment variables for each build action.
-If the CodeQL CLI autobuilders for compiled languages do not work with your CI workflow and you cannot wrap invocations of build commands with ``codeql database trace-command``, you can use indirect build tracing to create a CodeQL database. To use indirect build tracing, your CI system must be able to set custom environment variables for each build action.
+If the CodeQL CLI autobuilders for compiled languages do not work with your CI workflow and you cannot specify build commands with using the ``--command`` option, you can use indirect build tracing to create a CodeQL database. To use indirect build tracing, your CI system must be able to set custom environment variables for each build action.
+
+To create a CodeQL database with indirect build tracing, run the following command from the checkout root of your project:
+
+::
+
+  codeql database init ... --begin-tracing <database>
+
+You must specify:
+
+- ``<database>``: a path to the new database to be created. This directory will
+  be created when you execute the command---you cannot specify an existing
+  directory. 
+- ``--begin-tracing``: creates scripts that can be used to set up an environment in which build commands will be traced.
+
+You may specify other options for the ``codeql database init`` command as normal.
+
+.. pull-quote:: Note
+
+    If the build runs on Windows, you must set either ``--trace-process-level <number>`` or ``--trace-process-name <parent process name>`` so that the option points to a parent CI process that will observe all build steps for the code being analyzed.
+
+
+The ``codeql database init`` command will output a message::
+
+  Created skeleton <database>. This in-progress database is ready to be populated by an extractor.
+  In order to initialise tracing, some environment variables need to be set in the shell your build will run in.
+  A number of scripts to do this have been created in <database>/temp/tracingEnvironment.
+  Please run one of these scripts before invoking your build command.
+
+  Based on your operating system, we recommend you run: ...
+
+The ``codeql database init`` command creates ``<database>/temp/tracingEnvironment`` with files that contain environment variables and values that will enable CodeQL to trace a sequence of build steps. These files are named ``start-tracing.{json,sh,bat,ps1}``. Use one of these files with your CI system's mechanism for setting environment variables for future steps. You can:
+
+* Read the JSON file, process it, and print out environment variables in the format expected by your CI system. For example, Azure DevOps expects ``echo "##vso[task.setvariable variable=NAME]VALUE"``.
+* Or, if your CI system persists the environment,  source the appropriate ``start-tracing`` script to set the CodeQL variables in the shell environment of the CI system.
+
+Build your code; optionally, unset the environment variables using an ``end-tracing.{json,sh,bat,ps1}`` script from the directory where the ``start-tracing`` scripts are stored; and then run the command ``codeql database finalize <database>``.
+
+Once you have created a CodeQL database using indirect build tracing, you can work with it like any other CodeQL database. For example, analyze the database, and upload the results to GitHub if you use code scanning.
+
+Example of creating a CodeQL database using indirect build tracing
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The following example shows how you could use indirect build tracing in an Azure DevOps pipeline to create a CodeQL database::
+
+   steps:
+       # Download the CodeQL CLI and query packs...
+       # Check out the repository ...
+
+       # Run any pre-build tasks, for example, restore NuGet dependencies...
+
+       # Initialize the CodeQL database.
+       # In this example, the CodeQL CLI has been downloaded and placed on the PATH.
+       - task: CmdLine@1
+          displayName: Initialize CodeQL database
+         inputs:
+             # Assumes the source code is checked out to the current working directory.
+             # Creates a database at `<current working directory>/db`.
+             # Running on Windows, so specifies a trace process level.
+             script: "codeql database init --language csharp --trace-process-name Agent.Worker.exe --source-root . --begin-tracing db"
+
+       # Read the generated environment variables and values,
+       # and set them so they are available for subsequent commands
+       # in the build pipeline. This is done in PowerShell in this example.
+       - task: PowerShell@1
+          displayName: Set CodeQL environment variables
+          inputs:
+             targetType: inline
+             script: >
+                $json = Get-Content $(System.DefaultWorkingDirectory)/db/temp/tracingEnvironment/start-tracing.json | ConvertFrom-Json
+                $json.PSObject.Properties | ForEach-Object {
+                    $template = "##vso[task.setvariable variable="
+                    $template += $_.Name
+                    $template += "]"
+                    $template += $_.Value
+                    echo "$template"
+                }
+
+       # Execute the pre-defined build step. Note the `msbuildArgs` variable.
+       - task: VSBuild@1
+           inputs:
+             solution: '**/*.sln'
+             # Disable MSBuild shared compilation for C# builds.
+             msbuildArgs: /p:OutDir=$(Build.ArtifactStagingDirectory) /p:UseSharedCompilation=false
+             platform: Any CPU
+             configuration: Release
+             # Execute a clean build, in order to remove any existing build artifacts prior to the build.
+             clean: True
+          displayName: Visual Studio Build
+
+       # Read and set the generated environment variables to end build tracing. This is done in PowerShell in this example.
+       - task: PowerShell@1
+          displayName: Clear CodeQL environment variables
+          inputs:
+             targetType: inline
+             script: >
+                $json = Get-Content $(System.DefaultWorkingDirectory)/db/temp/tracingEnvironment/end-tracing.json | ConvertFrom-Json
+                $json.PSObject.Properties | ForEach-Object {
+                    $template = "##vso[task.setvariable variable="
+                    $template += $_.Name
+                    $template += "]"
+                    $template += $_.Value
+                    echo "$template"
+                }
+
+       - task: CmdLine@2
+          displayName: Finalize CodeQL database
+          inputs:
+             script: 'codeql database finalize db'
+
+       # Other tasks go here, for example:
+       # `codeql database analyze`
+       # then `codeql github upload-results` ...
+
 Obtaining databases from LGTM.com
 ---------------------------------