extensible `#lang` support

[jyn514]

this is a stretch goal, i know it's a bunch of work and i don't expect you to do a bunch of free labour. just jotting my thoughts down so i don't lose them.

racket is a meta-language for building programming languages. each language in the racket ecosystem identifies itself through a #lang my-lang at the top of the file (docs¹). furthermore, each language may possibly have a different file extension. in an ideal world, a racket plugin would support all languages built on top of racket, not just racket itself and some hand-coded extensions.

there are three main ways to detect whether a given file is built on top of racket.

1. sniff the shebang.
2. sniff the #lang attribute.
3. info.rkt files record the file extension used by a given language. @jackfirth tells me the recommended interface to this is get-module-suffixes, which we could call with e.g. racket -e '(require compiler/module-suffix) (print (get-module-suffixes))'

this plugin already partly does 1 and 2. but it only does so for a hard-coded list of file extensions. part of this work would be to do it for any racket file; probably by caching the racket -e output. we could avoid unnecessary work by only running if we see a #lang header.

once we discover the filetype, there are 4 main things we want to support:

• commentstring
• autoindent through indentexpr
• indent/detent
• LSP (which gets us syntax highlighting for free)

we can get commentstring by calling racket like so (see Comments):

$ racket -e "((read-language (open-input-string \"#lang rhombus\")) 'drracket:comment-delimiters #f)"

Indentation is harder.
This returns a racket function. Technically we could use a racket library that drives Neovim via the RPC API,
but this seems complicated and also slow. Not sure what to do here.

for LSP, things are actually not too tricky. the hard part is just integrating with the editor's LSP extension.
the main ones are nvim + lspconfig, coc, and ALE. for neovim we can also just call the LSP APIs directly.
to start, i only plan to work with lspconfig and tell people using other plugins to configure them manually; but it shouldn't be too hard to extend this.
note that we do need to inform lspconfig where the workspace root is. i think in practice this is "the closest info.rkt upward from this directory".

Footnotes

1. actually even this syntax is extensible; see https://docs.racket-lang.org/choose-lang/index.html ↩

[jyn514]

ok i did a hyperfocus https://github.com/jyn514/nvim-autoracket

[benknoble]

FWIW, while I appreciate the effort, the overlap of Vim and Racket enthusiasts is small enough that I believe fragmenting efforts will hamper improvement rather than help it. (I've similar feelings about some parts of the Neovim ecosystem.)

That said, I hope we can use this opportunity to collaborate and trade ideas and implementations. Thanks for putting something working together (which is better than my typing about it with nothing to show!). For example, I'd like to know how you circumvented performance problems [with dynamic lookups] to keep Vim snappy. I can read the code, but the commits didn't have much info, so a short summary would help when I get to that point.

[jyn514]

@benknoble the algorithm is something like this:

1. on plugin load, register a BufReadPost autocommand that checks if the buffer has #lang in the first three lines.
2. if so, load all file extensions with racket -e get-module-suffixes and unregister the autocommand. otherwise keep running it on each new buffer.
3. call vim.filetype.add for all those extensions, which registers a callback to be executed when matching files are loaded. additionally run that callback immediately for the current file.
4. in that callback, use a global lang -> settings cache, populated by racket -e (read-language 'drracket:comment-delimiters'), to set filetype and commentstring. additionally configure lspconfig if it's enabled.

some things to notice about this:

• the startup time is completely dominated by the time to run racket. racket -e '(require compiler/module-suffix)' takes a quarter second, and i run racket twice, so this delays startup by half a second. if there's any way at all to speed up racket's startup times, please do let me know.
  • i have an idea for how to only run racket once per #lang (combine steps 2 and 4), which would cut it to a quarter second, but that's still a lot more than all the other plugins combined.
  • maybe i could use a filesystem cache to avoid the overhead; but i couldn't invalidate the cache without calling racket, which defeats the point.
  • i could make the loading asynchronous. this is not ideal but better than completely blocking startup for that long.
• that half second is only for #lang files. for other files the startup is basically negligible (e.g. nvim -u bench.lua README.md +qall takes 24 ms without this plugin installed and 25 ms with), where bench.lua is just require('nvim-autoracket').setup {}.
• loading new buffers where the #lang is already cached has no noticeable overhead

[jyn514]

i found that racket -l racket/base -e get-modules-suffixes takes about as half as long as the default (racket/init). so that combined with only running racket once at startup should get it down to 130 ms. that's still a lot longer than most plugins, but it's at least tolerable.

[jyn514]

i had a long discussion on the racket discord where i found that raco demodularize suffixes.rkt gives about a 2x speedup. and with that i think the startup times are reasonable. i'll implement that soon.