AST Viewing unable to use extracted database bundle's src.zip

[Manouchehri]

Describe the bug
When I try to open a large database.zip bundle, I get:

The "start" argument must be of type number. Received type object

Full log:

Starting CodeQL extension
Checking CodeQL version using CodeQL CLI: /home/dave/.config/Code - Insiders/User/globalStorage/github.vscode-codeql/distribution7/codeql/codeql version -v --log-to-stderr --format=terse...
[2020-12-06 16:43:52] This is codeql version -v --log-to-stderr --format=terse
[2020-12-06 16:43:52] Terminating normally.

CLI command succeeded.
Didn't perform CodeQL CLI update check since a check was already performed within the previous 86400 seconds.
Checking CodeQL version using CodeQL CLI: /home/dave/.config/Code - Insiders/User/globalStorage/github.vscode-codeql/distribution7/codeql/codeql version -v --log-to-stderr --format=terse...
[2020-12-06 16:43:53] This is codeql version -v --log-to-stderr --format=terse
[2020-12-06 16:43:53] Terminating normally.

CLI command succeeded.
Found compatible version of CodeQL CLI (version 2.4.0)
Initializing configuration listener...
Initializing CodeQL cli server...
Initializing query server client.
Starting CodeQL CLI Server using CodeQL CLI: /home/dave/.config/Code - Insiders/User/globalStorage/github.vscode-codeql/distribution7/codeql/codeql execute cli-server -v --log-to-stderr
CodeQL CLI Server started on PID: 15256
Resolving RAM settings using CodeQL CLI: resolve ram -v --log-to-stderr --format json...
CLI command succeeded.
[2020-12-06 16:43:54] This is codeql execute cli-server -v --log-to-stderr

Initializing database manager.
Initializing database panel.
Registering database panel commands.
Initializing query history manager.
Registering query history panel commands.
Initializing results panel interface.
Initializing compare panel interface.
Initializing source archive filesystem provider.
Initializing CodeQL language server.
Initializing QLTest interface.
Resolving tests using CodeQL CLI: resolve tests --strict-test-discovery -v --log-to-stderr --format json /mnt/space/mono/master/src...
Registering CodeQL test panel commands.
Registering top-level command palette commands.
Starting language server.
Registering jump-to-definition handlers.
Removing orphaned databases from workspace storage.
Successfully finished extension initialization.
No orphaned databases found.
CLI command succeeded.

The "start" argument must be of type number. Received type object

Additional context

Version: 1.52.0-insider
Commit: 5e350b1b79675cecdff224eb00f7bf62ae8789fc
Date: 2020-12-04T10:15:13.654Z
Electron: 9.3.5
Chrome: 83.0.4103.122
Node.js: 12.14.1
V8: 8.3.110.13-electron.0
OS: Linux x64 5.4.0-52-generic

[Manouchehri]

I also downgraded VS Code, same issue.

Version: 1.51.1
Commit: e5a624b788d92b8d34d1392e4c4d9789406efe8f
Date: 2020-11-10T23:31:29.624Z
Electron: 9.3.3
Chrome: 83.0.4103.122
Node.js: 12.14.1
V8: 8.3.110.13-electron.0
OS: Linux x64 5.4.0-52-generic

[aeisenberg]

Thanks for reporting. It's certainly possible that #621 is related.

Are you able to work around this by unzip manually and importing the folder?

Also, would you be willing to share the database, so I can try this?

[Manouchehri]

Unzipping and importing the folder works, but it's a bit of a bummer that viewing the AST only works if it's a zip file.

It's a relatively big database at almost 20GB, might not be the easiest to share.

[aeisenberg]

The AST viewer only requires the src.zip. The rest of the database can be unzipped. Is this not what you are seeing?

[Manouchehri]

That's correct, the AST viewer doesn't work with my unzipped database bundle folder (which has the required src.zip file as well). I'm able to browser the files in the database just fine in VS Code (which I assume are being read from src.zip, because they're not extracted on my filesystem), but yeah, AST viewer doesn't like it anyway.

I believe it's because when I open a folder, uri.scheme isn't set to zipArchiveScheme. This is just a guess though, I'm not familiar with the code here.

vscode-codeql/extensions/ql-vscode/src/contextual/templateProvider.ts

Lines 146 to 148 in fe1476f

	if (uri.scheme !== zipArchiveScheme) {
	throw new Error('AST Viewing is only available for databases with zipped source archives.');
	}

[aeisenberg]

That's surprising to me. Internally, the extension using zipArchiveSchemes for source folders coming from zipped archives. For example, if you open your *.code-workspace file for this project, you should see an entry like this for your database:

                {
                        "name": "[cert_git source archive]",
                        "uri": "codeql-zip-archive://0-70/Users/andrew.eisenberg/codeql-home/databases/cert_git/src.zip"
                }

Notice the codeql-zip-archive part of the uri. This means that the extension can interpret the zip archive as sources for that database.

Anyway, I think I know a fix for the problem of downloading and installing large databases and I will work on it today. The problem is that the extension downloads and unzips in a single stream without saving the original zip file to disk. This saves time and disk space. However, if the zip is too large, or if the zip central directory is incorrect, then this approach may not work. See ZJONSSON/node-unzipper#149 also, #622.

The solution is to first save to disk, and then unzip from disk using a slightly different API. I don't know if this will solve the problem you are having, but it seems likely it is related.

[Manouchehri]

Odd, mine looks like that already.

{
	"folders": [
		{
			"path": "src"
		},
		{
			"path": "../../codeql"
		},
		{
			"name": "[codeql source archive]",
			"uri": "codeql-zip-archive://0-45/mnt/space/mono/master/src/codeql/src.zip"
		}
	]
}

I think I should rename this ticket to just handle the current AST Viewer issue, and stick to #622 for the large unzipping issues that you're already working on. Thanks for the fast responses!

[aeisenberg]

Hmmm...I see that this is on a mounted drive. Is it over the network? I wonder if that is related.

More precisely, what filesystem is that mount using?

[Manouchehri]

Er so /mnt/space isn't actually a different filesystem than / on my laptop, I just use the same folder naming conventions across all my machines to keep my sanity. The database bundle zip was generated from my headless workstation.

My / on my laptop is ext4.

[aeisenberg]

PR for unzipping issue is #700.

[aeisenberg]

The only way that you can receive that error for the AST viewer is if you are trying to view a file that is not from a codeql source archive. The extension determines which file to parse by looking at the active editor (or the selected file if invoking from the file explorer).

I can see two possibilities:

1. Your active editor is pointing to an unzipped source file that you somehow have added to your workspace.
2. You have some other extension in your vs code instance that is able to view zip file contents and it is somehow hijacking opening this editor.

Make sure that your editor is opening a file from the zip archive using the extension's support for zipped archives. If this doesn't help, then I am truly befuddled and will need some more information.

[Manouchehri]

You're right, it was another extension! I believe it was ccls.

[aeisenberg]

That's good to know. I'll try installing the other extension and see if there is a way to get both to work.

[Manouchehri]

Bit off-topic, but is it possible to calculate the AST from the codeql CLI? It's pretty slow on my laptop.

[Manouchehri]

Somewhat related, I opened a ticket regarding using ccls with CodeQL. MaskRay/ccls#776