Brýden - an (accidental) MoinMoin fork

[dvdkon]

Hi all,
we've been maintaining an internal MoinMoin fork for a few years now, and I think it's time I finally shared it with you, along with our reasons for doing so.

Around 2021, we decided to make a new wiki for our faculty. There was a previous, abandoned one based on MediaWiki, but we had bigger ambitions than just reviving it. The only software that matched our requirements was XWiki and MoinMoin v2; we chose MoinMoin, mostly because XWiki seemed too large for us to use and modify.

When testing, we made a few changes (mostly a new theme), and almost everything worked fine. Sadly when we imported the old wiki's content, page loads slowed down to a crawl - multiple seconds per page. Our efforts to speed up the Whoosh index by at least an order of magnitude failed, so we decided to replace it by Postgres.

The first attempt, which kept most of the code intact and changed just the indexing middleware, never quite worked right. The second attempt, a complete rewrite of storage (and much more), finally finished last year. The result is a codebase that'll probably be familiar to you in a few places (converters and HTML templates), but is very different elsewhere (storage, views). This improved performance drastically (50ms average page load time with 40k+ revisions on a busy server) and the code now works better with IDEs due to static typing. We also removed a lot of features to make the switch manageable. (In our defence, though, most of those were half- or fully broken.)

The original plan was to look through our changes and contribute the site-agnostic ones upstream in small PRs, but we've now changed the code so much, it's probably no longer possible. I'd love it if our work could be useful to people outside our single faculty, possibly reused by upstream MoinMoin; I also see you've made some nice changes to MoinMoin proper. If you have any ideas on this, let me know.

You can see the source code on our faculty's GitLab instance, and look at the resulting wiki.

[ThomasWaldmann]

Interesting, thanks for sharing!

There was some recent work here to make the whoosh search (and general response times) faster.

But whoosh seems to be a bit in a zombie state, which also led me to thinking about going postgresql (#715, but was never realized). Is your postgresql storage able to deal with large binary items efficiently and how good is the search?

[dvdkon]

Thanks. I saw some of that work on making MoinMoin faster, and while I haven't tried it myself, the rough numbers I saw sadly didn't seem enough to overcome Whoosh's failings. By my crude measurements, grep is faster than Whoosh, so something's clearly very wrong.

Currently we only store revision data smaller than 4 KiB in the DB (about 1/3rd of our items), using the filesystem for larger content.

We use Postgres' "full text search" with a custom Czech lemmatisation dictionary. It's far from state-of-the-art, without any typo tolerance, but I think it works well enough. Recently I was talking with one of the other contributors about the possibility of using a better, external, search index, but it's hard to do so without creating side-channel vulnerabilities.

[ThomasWaldmann]

4KiB is a quite a low threshold (DB vs fs). You just use the size and not the content-type for that decision? Guess searching in the fs files is not implemented (or slow)?

[dvdkon]

Yes, I'll admit that the value was chosen without any testing, mostly to not make our DB admins mad :)

Even larger files are indexed, since the fulltext-searchable content extracted via the text_out converter is stored in a separate column, so it's just an optimisation issue. There is a separate, larger, limit on what prefix of a document should be searchable.