Skip to content

Some files missing after database creation #14

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
mvanotti opened this issue Jan 25, 2020 · 12 comments
Open

Some files missing after database creation #14

mvanotti opened this issue Jan 25, 2020 · 12 comments
Labels

Comments

@mvanotti
Copy link

While trying to create a database for the fuchsia operating system, it seems to leave out some of the files that were built.

Note that the steps to build fuchsia and a codeql database require ~200GB of disk space. I just want to get a better understanding of why it might be skipping some of the files.


Steps to reproduce:

  1. Follow steps on the fuchsia site to download an build fuchsia
  2. Inside the fuchsia directory, run: fx set workstation.x64 --with-base //bundles:kitchen_sink --no-goma
  3. Build the database: codeql database create fuchsia-ql --language=cpp --source-root=${FUCHSIA_DIR} --command="./scripts/fx clean-build"
  4. List the files with a script similar to this one
  5. List all the fuchsia cc, ignoring the third_party files: find ${FUCHSIA_DIR} -iname "*.cc" | grep -v "third_party"
  6. Compare the files in the database with the cc files. Some of the files are missing.

This is the list that I have in my system. In particular, files like /zircon/tools/zbi/zbi.cc should always be built.

To get a list of all the compiled files in fuchsia, run: fx compdb and then look at out/default/compile_commands.json and out/default.zircon/compile_commands.json

@alexet
Copy link
Contributor

alexet commented Jan 27, 2020

We crash while extracting that file so it doesn't end up in the database. We can reproduce it and we are working on a fix.

@matt-gretton-dann
Copy link

Can you please give the output of codeql --version? There are several different issues going on here and I want to check that I am seeing the same issues you are.

@mvanotti
Copy link
Author

Hi Matt,

$ ./codeql/codeql --version
CodeQL command-line toolchain.
Version: 2.0.1.
Copyright (C) 2019 GitHub, Inc.

I'm using the latest released version downloaded from this github repo

@matt-gretton-dann
Copy link

Thank you for that info. I have identified several different issues which in our code base which are being triggered by the fuchsia build. I am working on fixes - but unfortunately they are unlikely to land in the next release of CodeQL. I will keep you updated on how the work is going through this ticket.

@mvanotti
Copy link
Author

mvanotti commented Feb 2, 2020

Cool! Let me know if there's anything I can do to help.

@matt-gretton-dann
Copy link

We have fixes internally now for a couple of the main issues (lld not recognised as a linker, and the vast majority of extraction failures). These should make it into the next release of the CodeQL CLI Binaries (v2.0.3). I do not have a date for that release yet.

@hmakholm
Copy link
Contributor

(@matt-gretton-dann actually 2.0.3 will probably be a point release after 2.0.2 to fix just a single brown-paper-bag bug, and probably out tomorrow. The one that contains your fixes this will then be 2.0.4).

@mvanotti
Copy link
Author

Do you know if the change made it a release? We are at 2.1.0

@mvanotti
Copy link
Author

/zircon/tools/zbi/zbi.cc is now being included in my database. Will continue testing to see if there's something else missing.

@mvanotti
Copy link
Author

I ran the following query to list all the files:

import cpp

from File f
select f.getAbsolutePath(), ""

I see 8046 different files.

And also gathered all the files from compile_commands.json used during compilation in fuchsia (both in out/default and out/default.zircon). Doing a sort | uniq, I get 9320 different cc files.

This means that there are ~1.3k files that are missing. This gist contains the list of files in the db and in compile_commands.json.

Some examples of missing files (by eye) are some of the autogenerated fidl files, some unit tests, and some random files that I have no idea why are missing.

@mvanotti
Copy link
Author

Hi! I've re-ran this test with CodeQL CLI version 2.2.4, and found similar results.

By looking only at c and cc files, I see that there are 1140 missing files from the database, albeit ~700 of them are test files, but they do appear in compile_commands.json.

@matt-gretton-dann matt-gretton-dann removed their assignment Dec 22, 2020
@Manouchehri
Copy link

@mvanotti Are you still running into this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

6 participants