Tree-sitter syntaxes

An aggregator of tree-sitter syntaxes for Neovim's built-in vim.treesitter and compatible consumers. Produces tested shared libraries and queries via Nix.

Trust model

Tree-sitter parsers run as compiled libraries. This means that compiling an arbitrary Tree-sitter parser repository may allow arbitrary code execution in the environment executing the parser. This can be fine depending on purpose and if you trust the upstream developers. However, editors often desire support for a large number of languages, which means that a very large set of authors may be involved in producing the final parsers. This creates a large dependency attack surface.

The Tree-sitter parser generator normally derives the generated parser.c file from a grammar.js file, which outputs grammar.json as an intermediate representation. Additionally, external scanners may be linked into the generated parser by a grammar author to parse tokens which are otherwise inconvenient or impossible to describe using the intermediate representation.

To remedy the attack surface, this repository builds all parsers in two steps. The generation step, which is only allowed to create the grammar.json file, is the first step. The output is of course inert, but upstream Tree-sitter does not sufficiently protect against attempts to interpolate malicious strings into the generated C files. Specifically, the .name field could otherwise be exploited, but we pin it to the expected value.

All external scanner files are reviewed and verified across parser revisions by hash. Issues found in the review are collected in the grammars file. This is done on a best-effort basis. Issues are not limited to intentionally malicious constructions, but also latent security issues. Issues which are purely correctness-related, like potential crashes, will only be commented.

Initiatives to compile Tree-sitter parsers to Wasm have yielded the ability to load such parsers via Wasmtime into Neovim. This feature is available when Neovim is built with ENABLE_WASMTIME against a libtree-sitter of at least 0.26.10. When using Wasm parsers, the runtime is fully sandboxed, which closes the ability to exploit memory vulnerabilities or otherwise run malicious code.

Tests

Parsers and queries are verified to not regress accross updates upstream via tests in this repository. To test, run:

nix flake check

Or:

just check

Attribution

An improper subset of queries are vendored from nvim-treesitter under Apache-2.0. We may however update and maintain these querries progressively.

This work is licensed under the Apache License 2.0.

Consuming

Append this derivation to your Neovim's runtimepath:

syntaxes.lib.mkSyntaxes {
  inherit pkgs;
}

You can also just include the languages you need:

syntaxes.lib.mkSyntaxes {
  inherit pkgs;
  langs = ["ocaml"];
}

syntaxes.lib.supportedLangs is the full list of registry languages, which can be used to exclude languages:

syntaxes.lib.mkSyntaxes {
  inherit pkgs;
  langs = builtins.filter (l: l != "ocaml") syntaxes.lib.supportedLangs;
}

By default the bundle contains native parsers. The formats option selects between native (so), Wasm (wasm) and precompiled Wasm (cwasm) parsers, or bundles several side by side:

syntaxes.lib.mkSyntaxes {
  inherit pkgs;
  formats = ["wasm"];
}

When one runtimepath entry carries several formats of a parser, Neovim picks the native parser first, then a precompiled Wasm parser, then a plain Wasm parser. Parsers earlier on the runtimepath take precedence later ones regardless of format. As such, you should omit so if you want to use Wasm-based parsers in Neovim.

The pkgs passed in these examples specifies the package set used by the C compiler. The generation step uses a patched Tree-sitter package pinned by this repository. Holding it fixed preserves the security of the generation step across Tree-sitter versions. Additionally, the patches remedy a plethora of performance, memory and generation issues without which this project would not be viable. If you really want to use your own version, it can be overriden via the flake input.

Per-grammar packages are also exposed as packages.<system>.grammar-<lang>-so, packages.<system>.grammar-<lang>-wasm and packages.<system>.grammar-<lang>-cwasm, single-format bundles of every grammar as packages.<system>.grammars-so, packages.<system>.grammars-wasm and packages.<system>.grammars-cwasm, and the full bundle with all formats as packages.<system>.default. These cwasm artifacts serialize with the nixpkgs wasmtime, so outside of testing you will usually want mkSyntaxes with your editor's wasmtime instead.

Precompiled Wasm parsers

Wasmtime, the Wasm runtime used by Neovim, can compile Wasm files ahead-of-time to so called compiled Wasm (cwasm) files. This can considerably reduce the cold-start time to load a parser, down to below one frame for the slowest parsers. Neovim however does not expose a way to use these files natively.

This project maintains two patches that make cwasm a first-class parser format on the runtimepath:

• load-precompiled-language adds ts_wasm_store_load_precompiled_language to libtree-sitter. It consumes both the precompiled module and its source Wasm module, the latter of which is used to read dynamic linking memory information necessary to load the precompiled module.
• cwasm-parser-format teaches Neovim's parser discovery to rank parser/{lang}.cwasm between native and plain Wasm parsers within a runtimepath entry, and to load it together with the sibling parser/{lang}.wasm through the new entry point.

Because the cwasm format is not self-contained, it is always present in conjunction with the source Wasm module.

To use this format in Neovim, compile Neovim with both patches applied, its libtree-sitter being of version 0.26.10 or newer, and compile the syntaxes with exactly the wasmtime version used in libtree-sitter. The Neovim patch is maintained against the version pinned in neovim-pin.nix. The Nix configuration looks like the following:

syntaxes.lib.mkSyntaxes {
  inherit pkgs;
  formats = ["cwasm"];
  inherit wasmtime;
}

pkgs.tree-sitter.overrideAttrs (old: {
  patches =
    (old.patches or [])
    ++ [syntaxes.packages.${pkgs.system}.patches.tree-sitter.load-precompiled-language];
})

pkgs.neovim-unwrapped.overrideAttrs (old: {
  patches =
    (old.patches or [])
    ++ [syntaxes.packages.${pkgs.system}.patches.neovim.cwasm-parser-format];
})

The machine code is compiled with x86-64-v2 optimisations on x86_64. If you want to enable more specialised optimisations, pass cwasmTarget as { triple, flags }, flags being Cranelift flag names.

Configuring Wasmtime

syntaxes.lib.mkSyntaxes {
  inherit pkgs wasmtime;
  formats = ["cwasm"];
  cwasmConfig.epoch_interruption = true;
}

Every attribute set in cwasmConfig interpolates into a wasmtime_config_<name>_set definition.

Updating a grammar

just update ocaml <new-rev>
just check

If the scanner's native code changed, just update prints the diff and stops without writing anything. After reading it, re-run with --bless to record the change.

just update ocaml <new-rev> --bless
just check

Patches which have been merged into the new head are automatically removed after the update succeeds. If a patch no longer applies properly, the update is cancelled and must be continued by hand.

Patching a grammar

An entry may carry upstream pull requests as patches:

# Parses `where` clauses on unit structs.
[[rust.patches]]
pr = "https://github.com/tree-sitter/tree-sitter-rust/pull/271"
rev = "<pr-head-commit>"
hash = "sha256-..."
paths = ["grammar.js"]

A fix that has no upstream pull request may be carried as an inline diff instead:

# Rewrites the `\verb` rule into a single token to restore error recovery.
[[latex.patches]]
diff = '''
diff --git a/grammar.js b/grammar.js
...
'''

Every patch requires a comment stating its concrete improvement or defect it fixes. Patches which touch external scanners must have their delta on the external scanners documented according to the same model applied to full external scanner files.

Adding a language

1. Add the language entry to grammars.toml. You can use nix flake prefetch --json <reference> for convenience.
   • If the parser is not in the expected top-level location, use the location option to specify where it lies.
2. Read and review any required native code according to the process described in the trust model section and add it to the files list.
3. Add Neovim queries into the queries directory.
4. Add language tests in the tests directory.
5. Run just check.