Hey @davewichers, thanks a lot for reaching out with this! I'll circle this back to our internal team responsible for Java analysis 👍

Hey @davewichers, as reported in this change note, we recently removed reporting of MD5 and SHA1 hashing from the java/weak-cryptographic-algorithm to the less precise java/potentially-weak-cryptographic-algorithm, as the former was alerting on too many cases of legitimate non cryptographic usages of those hashes. Maybe you can switch to using that query instead in your benchmarking?

@redsun82 - the codeQL documentation related to finding out which rulepacks exist and how to use them is super confusing. Where is the list of ALL the published codeql rulepacks? I only found: codeql\java-queries and codeql\java-all, but when I try to use java-all, it says 'this is a library and does not contain queries to run'. I want to use ALL the codeQL java rules in my run. How do I do that?

Hi @davewichers, the concept of a set of queries is captured by "query suites":

• docs for running with the CLI (see in particular the explanation of the <packs,queries> positional argument). What I also found indeed a bit confusing here, is that we don't stress enough that this documentation relates to full codeQL bundles published as releases of codeql-action. Please find more instructions below.
• docs for codescanning
• files defining query suites for java here)

I heard back from the team, and they advise to use the java-security-extended suite for the benchmark. To do so, you should provide java-security-extended or java-security-extended.qls as second positional argument to codeql database analyze. If you download the latest release for the full codeql bundle, that will be included together with precompiled queries which will make the command run faster. You can however also run the queries from a checkout of codeql, provided you give the right --search-path.

Where is the list of ALL the published codeql rulepacks?

Probably the best way to get that list is to run

codeql resolve queries

This will list all query suites with their .qls extension. The query suite argument to codeql database analyze can be provided either with or without that extension.

I'm sorry if you found the documentation lacking, we will try to make it better! 🙌

Hi @davewichers

I'm very sorry for answering so late to this, and that you encountered these problems!

I think the source of confusion here is that java-security-extended is not a query pack, but rather a subset thereof (which we call query suites). They are included in the codeql/java-all pack you already download, and then they can be run by providing them as is to codeql database analyze, i.e. without the codeql/ prefix. See my comment above:

To do so, you should provide java-security-extended or java-security-extended.qls as second positional argument to codeql database analyze

So in your case the correct analyze command should be

codeql database analyze owasp-benchmark java-security-extended --format=sarifv2.1.0 --output=results/Benchmark-v1.2_CodeQL.sarif

Before running that you can double check the query suite is included by running

codeql resolve queries

This should contain the lines

  java-code-scanning.qls - Standard Code Scanning queries for Java
  java-lgtm-full.qls - Standard LGTM queries for Java, including ones not displayed by default
  java-lgtm.qls - Standard LGTM queries for Java
  java-security-and-quality.qls - Security-and-quality queries for Java
  java-security-experimental.qls - Extended and experimental security queries for Java and Kotlin
  java-security-extended.qls - Security-extended queries for Java

Those are all possible positional arguments to codeql database analyze, with the .qls extension being optional, no further prefixes required.

I will try out a PR to clarify the documentation to the <packs,queries> argument to database analyze.

This issue is stale because it has been open 14 days with no activity. Comment or remove the Stale label in order to avoid having this issue closed in 7 days.

I'm still waiting for an answer to my last comment from @redsun82. Can you answer my question/help me to get this to actually work?

Hi @davewichers,

Sorry for the late response and confusion around using the security-extended query suite.

All our queries live in a query pack. For Java that pack is codeql/java-queries.
The codeql/java-all pack is the library pack the queries rely on and you shouldn't be concerned with that one.

To run a suite you can directly reference the query pack and the suite therein.

codeql database analyze \
<path-to-db> \
codeql/java-queries:codeql-suites/java-security-extended.qls \
--format=sarifv2.1.0 \
--output=results.sarif \
-j0 \
--download

Note this adds to additional flags -j0 to use all available threads and --download to download any missing query packs automatically (i.e., you can skip the codeql pack download step).

To resolve the queries in a suite you can use the codeql resolve queries command like:

codeql resolve queries codeql/java-queries:java-security-extended

Note that this doesn't support a --download flag so if you run the before the packs has been downloaded (with for example the above analyze command) you need to run codeql pack download codeql/java-queries beforehand.

If you want to resolve all the queries in a pack you can use the command:

codeql resolve queries codeql/java-queries:.

Not that this will also contain internal queries we use for telemetry and diagnostics, so it is advised to use the query suites to select the useful queries to run.

Let me know if this resolves your issue.

@rvermeulen - it appears this worked. Thanks for the detailed instructions. It would be nice if your documentation described how to do all this properly so users wouldn't have to open issues to figure out how to get this to work.

@davewichers

Good to hear it worked and thanks for the feedback. We have this documented at https://docs.github.com/en/code-security/codeql-cli/getting-started-with-the-codeql-cli/analyzing-your-code-with-codeql-queries#running-query-suites, but it sounds like we need to work on the discoverability of our documentation.

Yes. I couldn't find these details when I followed the normal path to the instructions on how to run queries. So, something should be improved to make these details/scenarios much easier to find.

Hi @davewichers,

Could you share your starting point in the docs so I can provide this as feedback to our docs team?

This issue is stale because it has been open 14 days with no activity. Comment or remove the Stale label in order to avoid having this issue closed in 7 days.

This issue was closed because it has been inactive for 7 days.