Skip to content

Commit b73a2f7

Browse files
ethanpalmethanpalm
authored
Merge pull request github#6667 from ethanpalm/indirect-build-tracing-docs
Add indirect build tracing docs
2 parents a67db45 + 4d7aa5c commit b73a2f7

File tree

1 file changed

+117
-0
lines changed

1 file changed

+117
-0
lines changed

‎docs/codeql/codeql-cli/creating-codeql-databases.rst

+117
Original file line numberDiff line numberDiff line change
@@ -228,6 +228,123 @@ commands that you can specify for compiled languages.
228228
This command runs a custom script that contains all of the commands required
229229
to build the project.
230230

231+
Using indirect build tracing
232+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
233+
234+
If the CodeQL CLI autobuilders for compiled languages do not work with your CI workflow and you cannot wrap invocations of build commands with ``codeql database trace-command``, you can use indirect build tracing to create a CodeQL database. To use indirect build tracing, your CI system must be able to set custom environment variables for each build action.
235+
236+
To create a CodeQL database with indirect build tracing, run the following command from the checkout root of your project:
237+
238+
::
239+
240+
codeql database init ... --begin-tracing <database>
241+
242+
You must specify:
243+
244+
- ``<database>``: a path to the new database to be created. This directory will
245+
be created when you execute the command---you cannot specify an existing
246+
directory.
247+
- ``--begin-tracing``: creates scripts that can be used to set up an environment in which build commands will be traced.
248+
249+
You may specify other options for the ``codeql database init`` command as normal.
250+
251+
.. pull-quote:: Note
252+
253+
If the build runs on Windows, you must set either ``--trace-process-level <number>`` or ``--trace-process-name <parent process name>`` so that the option points to a parent CI process that will observe all build steps for the code being analyzed.
254+
255+
256+
The ``codeql database init`` command will output a message::
257+
258+
Created skeleton <database>. This in-progress database is ready to be populated by an extractor.
259+
In order to initialise tracing, some environment variables need to be set in the shell your build will run in.
260+
A number of scripts to do this have been created in <database>/temp/tracingEnvironment.
261+
Please run one of these scripts before invoking your build command.
262+
263+
Based on your operating system, we recommend you run: ...
264+
265+
The ``codeql database init`` command creates ``<database>/temp/tracingEnvironment`` with files that contain environment variables and values that will enable CodeQL to trace a sequence of build steps. These files are named ``start-tracing.{json,sh,bat,ps1}``. Use one of these files with your CI system's mechanism for setting environment variables for future steps. You can:
266+
267+
* Read the JSON file, process it, and print out environment variables in the format expected by your CI system. For example, Azure DevOps expects ``echo "##vso[task.setvariable variable=NAME]VALUE"``.
268+
* Or, if your CI system persists the environment, source the appropriate ``start-tracing`` script to set the CodeQL variables in the shell environment of the CI system.
269+
270+
Build your code; optionally, unset the environment variables using an ``end-tracing.{json,sh,bat,ps1}`` script from the directory where the ``start-tracing`` scripts are stored; and then run the command ``codeql database finalize <database>``.
271+
272+
Once you have created a CodeQL database using indirect build tracing, you can work with it like any other CodeQL database. For example, analyze the database, and upload the results to GitHub if you use code scanning.
273+
274+
Example of creating a CodeQL database using indirect build tracing
275+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
276+
277+
The following example shows how you could use indirect build tracing in an Azure DevOps pipeline to create a CodeQL database::
278+
279+
steps:
280+
# Download the CodeQL CLI and query packs...
281+
# Check out the repository ...
282+
283+
# Run any pre-build tasks, for example, restore NuGet dependencies...
284+
285+
# Initialize the CodeQL database.
286+
# In this example, the CodeQL CLI has been downloaded and placed on the PATH.
287+
- task: CmdLine@1
288+
displayName: Initialize CodeQL database
289+
inputs:
290+
# Assumes the source code is checked out to the current working directory.
291+
# Creates a database at `<current working directory>/db`.
292+
# Running on Windows, so specifies a trace process level.
293+
script: "codeql database init --language csharp --trace-process-name Agent.Worker.exe --source-root . --begin-tracing db"
294+
295+
# Read the generated environment variables and values,
296+
# and set them so they are available for subsequent commands
297+
# in the build pipeline. This is done in PowerShell in this example.
298+
- task: PowerShell@1
299+
displayName: Set CodeQL environment variables
300+
inputs:
301+
targetType: inline
302+
script: >
303+
$json = Get-Content $(System.DefaultWorkingDirectory)/db/temp/tracingEnvironment/start-tracing.json | ConvertFrom-Json
304+
$json.PSObject.Properties | ForEach-Object {
305+
$template = "##vso[task.setvariable variable="
306+
$template += $_.Name
307+
$template += "]"
308+
$template += $_.Value
309+
echo "$template"
310+
}
311+
312+
# Execute the pre-defined build step. Note the `msbuildArgs` variable.
313+
- task: VSBuild@1
314+
inputs:
315+
solution: '**/*.sln'
316+
# Disable MSBuild shared compilation for C# builds.
317+
msbuildArgs: /p:OutDir=$(Build.ArtifactStagingDirectory) /p:UseSharedCompilation=false
318+
platform: Any CPU
319+
configuration: Release
320+
# Execute a clean build, in order to remove any existing build artifacts prior to the build.
321+
clean: True
322+
displayName: Visual Studio Build
323+
324+
# Read and set the generated environment variables to end build tracing. This is done in PowerShell in this example.
325+
- task: PowerShell@1
326+
displayName: Clear CodeQL environment variables
327+
inputs:
328+
targetType: inline
329+
script: >
330+
$json = Get-Content $(System.DefaultWorkingDirectory)/db/temp/tracingEnvironment/end-tracing.json | ConvertFrom-Json
331+
$json.PSObject.Properties | ForEach-Object {
332+
$template = "##vso[task.setvariable variable="
333+
$template += $_.Name
334+
$template += "]"
335+
$template += $_.Value
336+
echo "$template"
337+
}
338+
339+
- task: CmdLine@2
340+
displayName: Finalize CodeQL database
341+
inputs:
342+
script: 'codeql database finalize db'
343+
344+
# Other tasks go here, for example:
345+
# `codeql database analyze`
346+
# then `codeql github upload-results` ...
347+
231348
Obtaining databases from LGTM.com
232349
---------------------------------
233350

0 commit comments

Comments
 (0)