rust rewrite

[qknight]

development continues here:

nixcloud/pankat-rs#2

motivation

i consider a rust rewrite because:

• the rust workflow is better: go language feels too limited, the compiling process like a patchwork, standard project structure freaks me out
• rust makes WASM possible and has leptosUIs

technical specification

The source code:

https://github.com/nixcloud/pankat-replit

requirements

this is mainly a rewrite of

• https://lastlog.de/blog

i want to use these components for

• tour_of_nix (specialized teaching module)
• labsystem (generalized teaching module)
• nixcloud remote editor
• nixcloud.io webpage

for now this rust rewrite mainly focuses on the lastlog.de/blog module

ux

pankat main mission is a static page generator but with highly interactive elements:

• using websocket with live editing updates/rebuilds
• permission schema with login

backend

• execute pandoc from shell
• replace diffDOM.js by rust implementation
  • client side prototype working, it is fast! sauron: https://docs.rs/sauron/latest/sauron/dom/index.html
  • reintegrate diffDOM.js as sauron replacement because of <pre> issue with rphtml and html5ever
• pankat in rust rewrite

  • create index.html (most recent page)

  • timeline support

  • command line toggle for

    • basic command line interface concept
    • site brand
    • static blog generator command line option: --build-only

  • article logic

    • integrate websocket life-updates using sauron dom/vdom updates
    • a global flat switch to output input/libnix/libnix_roadmap.mdwn to either output/libnix/libnix_roadmap.html or output/libnix_roadmap.html
    • rewrite ArticleWithTags so id is not optional (same as Article) and use NewArticle/NewArticleWithTags for creating new entries
    • GC check after traverse_and_collect_articles to remove old articles from db
    • cache table
      • write cache into db
      • render optimization: implement checks (hash on the html string after plugin run has finished) so we can use the previous render and don't have to redo it on each article change
    • article finder
    • fs-change finder
    • plugins decode support
    • database support (https://diesel.rs/guides/relations.html)
      • Article / NewArticle sqlite/diesel concept
      • sqlite abstraction (convert GORM to DIESEL logic)
      • reimplement the tests from go for db interface
      • extend set()/del() to return affected neighbours in next/prev article, next/prev series and most_recent_article
      • relational support for tags
      • Articles' src/dst filenames (in db) should be relative to $input (ATM they are $input/dir/file.mdwn)
      • support for series queries
      • next/previous in article / series
        • fix bug with series
      • garbage collection in $output of unreferences files with scan+fs-notify

  • https://github.com/notify-rs/notify/tree/main/examples prototype integration

  • rendering

    • tempaltes support using https://docs.rs/handlebars/latest/handlebars/
    • markdown to html pandoc conversion
    • article rendering using the two templates
    • article navigation i.e. next/previous link
    • article series i.e. next/prevous link

  • README.md documentation

bugs

• refactor handlers::serve_assets / handlers::serve_wasm / handlers::serve_posts / ... into one function
• create output folder if no existing (or at least fail, that it is not there)
• fs-change updates are sent via ws but the file it actually not written (probably should also update timeline & misc)
• remove dst_file_name.html on mdwn delete in articles/mod.rs:149: EventKind::Remove(_)
• add /blog subdirectory support
• articles with 'foo bar.mdwn' are referenced wrongly (URL encoding bad?) in the 'article source' link. seems to be the router which gets a correct path but then can't decode it and goes into serve_output instead of serve_input
• check fs delete, fs new as well by adding new.mdwn and deleting it
• removing [[!draft]] does not:
  • add the article to the timeline (needs manual reload)
  • create the dst_file_name
• nav claims next article where there is none in 'lastlog.de' example for libnix.html article
• removing [[!series libnix]] won't update the nav to have no series
• GC for directories is not properly implemented

new stuff

• config file support
• cargo zigbuild support
• update javascript libraries, consider npm or pnpm
• REST intgerface concept
  • REST interface with login prototype (https://github.com/diesel-rs/diesel)
  • JWT_SECRET from ENV or config file
  • default admin user in config file
  • auth via rest/token (https://codeberg.org/fairsync/fairsync/src/branch/main/service/src/main.rs)
  • https://codeberg.org/fairsync/fairsync/src/commit/afab4f9ae4265dbc6a28faa42bda1cb35668ccf5/service/src/main.rs#L246
  • /logfile route to have some logs what is going on

content hashing

• separate article parser, plugins & html creator into standalone program using IPC
• create hash of templates
• add hashes into static documents output and registry call

frontend

textarea editor

• integrate rustpad (coop editing)
  • integrate it into the page for fast article editing with preview
  • use /api/login for auth
  • generate error messages with position annotations on parse/plugin/rendering issues

rustpad

https://rustpad.io (uses https://github.com/microsoft/monaco-editor/)

it builds on top of https://github.com/spebern/operational-transform-rs which means FS writes from normal editors don't work anymore (maybe i can write a FS-watcher which creates a bunch of transformations so that it wraps an override of a file into a 'huge' update of the document state)

see also:

https://github.com/siku2/rust-monaco

leptos rewrite

• timeline (from javascript to rust)
• show menu entries to draft/special_page/roadmap after login
• /api/ws-draft with websocket (leptos)
• /api/ws-specialpage with websocket (leptos)
• https://xtermjs.org terminal for git using leptos_xterm

related issues

• Add example of diffing/patching over a network chinedufn/percy#208
• diff algorithm ivanceras/sauron#105

future

• consider https://tweag.github.io/pandoc-wasm/
• integrate https://github.com/excalidraw/excalidraw
• integrate web-shell with git/tig
• fork button
• copy'n'paste for images to [[!imag ...]]
• tidy html
  • https://crates.io/crates/tidy-sys sucks on nixos
  • https://gist.github.com/qknight/968e11fe2b1c7531493a3665ddcd7aad (html5server) parser+serializer seems interesting
  • tidy html output (https://chatgpt.com/share/679f7102-fc84-800c-bddc-2db88661f337)
• add a new [[!link foo/bar/example.mdwn]] plugin which checks existence and draft status
  • deployment flake (with nginx)
  • replace anchor.js with lua / pandoc https://chatgpt.com/share/67bdd355-28d0-800c-9ed7-2a412b54e7c1

[qknight]

sever side events, http3 (quic)

[qknight]

leptos SSR, hydration

[qknight]

CRDT als OT alternative (https://github.com/automerge/automerge)

[qknight]

See flosse/pankat-replit@d591b30#diff-ada3de7a47efab14718dc6bd5006cf5b10f913f8d7d0d7ce6d3f778c9ee0481d

[qknight]

https://docs.rs/tokio-util/latest/tokio_util/sync/struct.CancellationToken.html

[qknight]

Decided to abandon the idea of using sauron DOM/VDOM because of:

• parse_html ignoring white-spaces and newlines for <pre><code> ... </pre></code> html ivanceras/sauron#107

But also abandoned the idea of using sauron + html5ever parser because of:

• parse_html adds unwanted tags like <html><head>...<body></html> servo/html5ever#583

I hope that either gets fixed beucase this would then enable:

• vdom patching (server and client side)
• nice html formatting
• modifying html so: tocify.js and anchorjs can be done server-side