Explore using tree sitter for syntax highlighting

[alexr00]

Goals of the explorations:

• Get Aiday's exploration up and running again: [Exploration] Tree-sitter tokenization exploration (Fixes #161256) #161479
• Come up with ideas for a contribution model
• Confirm performance findings from [Exploration] Tree-sitter tokenization exploration (Fixes #161256) #161479 and understand incremental highlighting

[alexr00]

Some notes on incremental parsing:

• Per Can web-tree-sitter actually do async parsing? tree-sitter/tree-sitter#2118, it looks like we can set a timeout so that each call to parse times-out, with the parsing progress preserved for the next call to parse.
• I've confirmed that this is the case; however, it appears to be broken in the latest version of web-tree-sitter. Going back to 0.22.2 works (latest is 0.22.5).
• There does not appear to be any significant time penalty for setting a timeout.

@hediet FYI.

[alexr00]

Opened tree-sitter/tree-sitter#3341 for the above issue.

[alexr00]

After some discussion, we've decided to first understand the performance of using tree-sitter to parse files. #213565 contains that work.

[alexr00]

We also need new internal editor API to write a more optimal parse function, which @rebornix may help with.

[alexr00]

Progress in July:

• Created a build pipeline for the relevant wasm files (tree-sitter and typescript right now): https://github.com/microsoft/vscode-tree-sitter-wasm
• Published first npm package with said wasm files: https://www.npmjs.com/package/@vscode/tree-sitter-wasm
• Incorporated that package in Add telemtry for how long it takes to parse files with tree-sitter #213565, which is almost ready to merge, but we'll hold off until after endgame as there's no need to rush it in before release.

[alexr00]

Updates (though I didn't get to work on this October):

• We have syntax highlighting tests for tree sitter
• Typescript is still the only supported language
• Lots of small typescript improvements

For November, the main plan is performance.

[ghost]

That's along of junk

[Plixo2]

Can we get an update on this?

[ardokirsipuu]

Is this going to affect also Markdown? Asking because this task #180246 was closed without a fix (referencing to the abandoned TextMate's library) and this work here seems to be the only light at the end of the tunnel for resolving this.

@alexr00 Markdown wasn't part of your response here but there is a tree-sitter grammar for markdown existing, although it has some "interesting" comment about WASM, so I'm not sure if it will suite VS Code or not: https://github.com/tree-sitter-grammars/tree-sitter-markdown?tab=readme-ov-file#usage-with-wasm

Either way, should there be an open ticket to track this Markdown highlighting bug? So it would appear in this list.

[savetheclocktower]

although it has some "interesting" comment about WASM, so I'm not sure if it will suite VS Code or not: https://github.com/tree-sitter-grammars/tree-sitter-markdown?tab=readme-ov-file#usage-with-wasm

Pulsar got around this by building a custom copy of web-tree-sitter as described here. It's annoying, but not a deal-breaker.

[SeniorMars]

I'm putting this as a side note, but I'm sure solving the custom copy of web-tree-sitter problem would help (and be of interest) to Zed since I believe one of their goals is to be available through the web, too.

[alexr00]

Opened #237316 for markdown to keep it on my radar for either finding a new textmate grammar or considering for tree-sitter (depending on the situation with wasm).

[alexr00]

Actually, just re-opened #180246.