nix packaging

[teto]

To use wordbase as a dependency of my program and as a nix fan, I am trying to pakcage this for nix (possibily upstreamed as hinted at #1, otherwise just in a fork).

On the suggestion of my lux friends, I went with https://github.com/ipetkov/crane which is more granular than other rust nix packagers.

I create this ticket mostly to track down issues I meet:

First one is absence of a workspace.version in Cargo.toml which triggers this:

commonCargoSourcesevaluation warning: crane will use a placeholder value since `name` cannot be found in /nix/store/vvmg0szsqa0nyp2x51x7qb97vfjr47in-Cargo.toml
                    to silence this warning consider one of the following:
                    - setting `pname = "...";` in the derivation arguments explicitly
                    - setting `package.name = "..."` or `package.metadata.crane.name` = "..." or `workspace.metadata.crane.name` = "..." in the root Cargo.toml
                    - explicitly looking up the values from a different Cargo.toml via
                      `craneLib.crateNameFromCargoToml { cargoToml = ./path/to/Cargo.toml; }`

                    To find the source of this warning, rerun nix with:
                    `NIX_ABORT_ON_WARN=1 nix --option pure-eval false --show-trace ...`

Lindera needs some external data. To start with I apply this patch:

diff --git i/crates/wordbase/Cargo.toml w/crates/wordbase/Cargo.toml
index b964a20..d555475 100644
--- i/crates/wordbase/Cargo.toml
+++ w/crates/wordbase/Cargo.toml
@@ -39,7 +39,7 @@ foldhash             = { workspace = true }
 futures              = { workspace = true }
 indexmap             = { workspace = true, features = ["serde"] }
 itertools            = { workspace = true }
-lindera              = { workspace = true, features = ["unidic", "compress"] }
+lindera              = { workspace = true, features = ["compress"] }
 maud                 = { workspace = true }
 paste                = { workspace = true }
 pin-project          = { workspace = true }

to avoid

       >   cargo:rerun-if-changed=build.rs
       >   cargo:rerun-if-changed=Cargo.toml
       >   cargo::rustc-env=LINDERA_WORKDIR=/build/source/target/release/build/lindera-unidic-fbad8ed6b976bd63/out
       >
       >   --- stderr
       >   Error: "Failed to download a valid file from all sources"

https://github.com/NixOS/nixpkgs/blob/e32cfd2e9abfe73cb3ed49c22bd1e29677ce4f9a/pkgs/applications/misc/pagefind/default.nix#L102 may show how to patch lindera build.

My current error is this:

     >     = note: this error originates in the macro `$crate::sqlx_macros::expand_query` which comes from the expansion of the macro `sqlx::query` (in Nightly builds, run with -Z macro-backtrace for more info)
       >
       > error: set `DATABASE_URL` to use query macros online, or run `cargo sqlx prepare` to update the query cache
       >    --> crates/wordbase/src/profile.rs:147:9
       >     |
       > 147 | /         sqlx::query!(
       > 148 | |             "UPDATE profile SET font_family = $1 WHERE id = $2",
       > 149 | |             font_family,
       > 150 | |             profile_id.0
       > 151 | |         )
       >     | |_________^
       >     |
       >     = note: this error originates in the macro `$crate::sqlx_macros::expand_query` which comes from the expansion of the macro `sqlx::query` (in Nightly builds, run with -Z macro-backtrace for more info)
       >
       > error: set `DATABASE_URL` to use query macros online, or run `cargo sqlx prepare` to update the query cache
       >    --> crates/wordbase/src/profile.rs:166:23
       >     |
       > 166 |       let mut records = sqlx::query!(
       >     |  _______________________^
       > 167 | |         "SELECT
       > 168 | |             profile.id,
       > 169 | |             profile.name,
       > ...   |
       > 177 | |         ORDER BY profile.id"
       > 178 | |     )
       >     | |_____^
       >     |
       >     = note: this error originates in the macro `$crate::sqlx_macros::expand_query` which comes from the expansion of the macro `sqlx::query` (in Nightly builds, run with -Z macro-backtrace for more info)
       >
       > error: set `DATABASE_URL` to use query macros online, or run `cargo sqlx prepare` to update the query cache
       >   --> crates/wordbase/src/texthook.rs:38:19
       >    |
       > 38 |         let url = query!("SELECT texthooker_url FROM config")
       >    |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
       >    |
       >    = note: this error originates in the macro `$crate::sqlx_macros::expand_query` which comes from the expansion of the macro `query` (in Nightly builds, run with -Z macro-backtrace for more info)
       >
       > error: set `DATABASE_URL` to use query macros online, or run `cargo sqlx prepare` to update the query cache
       >   --> crates/wordbase/src/texthook.rs:75:9
       >    |
       > 75 |         query!("UPDATE config SET texthooker_url = $1", url)
       >    |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
       >    |
       >    = note: this error originates in the macro `$crate::sqlx_macros::expand_query` which comes from the expansion of the macro `query` (in Nightly builds, run with -Z macro-backtrace for more info)
       >
       > warning: unused import: `RecordKind`
       >   --> crates/wordbase/src/import/insert.rs:32:65
       >    |
       > 32 |     DictionaryId, FrequencyValue, NormString, Record, RecordId, RecordKind, RecordType, Term,
       >    |                                                                 ^^^^^^^^^^
       >    |
       >    = note: `#[warn(unused_imports)]` on by default
       >
       > warning: `wordbase` (lib) generated 1 warning
       > error: could not compile `wordbase` (lib) due to 31 previous errors; 1 warning emitted
       > warning: build failed, waiting for other jobs to finish...
       For full logs, run:
         nix log /nix/store/8dvzx0yg6kq1s554sbpxsymv3jkhwh9m-wordbase-cli-0.0.1.drv

Not sure what the build is trying to do here, are these macros/SQL commands executed at buildtime ? any idea how to make it work in a sandbox (pass on an already generated data etc) ?

[aecsocket]

wordbase-cli doesn't have a package.version since it's not something I intend on publishing, and is more of a debug/testing tool rather than a proper piece of released software. Cargo will default to giving it the version 0.0.0 if it has no package.version, but I can specify this explicitly if you need it for packaging.

Lindera downloads a dictionary from the internet at build-time using its build.rs script. This is admittedly really annoying, and I've had issues with it before, but I'm not really sure what the best approach here would be. However, Lindera follows this logic: if LINDERA_CACHE env var is set, and $LINDERA_CACHE/${current lindera version} exists, it will use the built dictionary from that path instead of downloading. So, maybe you can set LINDERA_CACHE to a path in the Nix store? I'm not sure how Nix packaging works exactly. Here's an example of what LINDERA_CACHE should look like:

sqlx uses some (admittedly cursed) magic at macro-runtime to type-check sqlx::query! statements. It connects to a live database (or a live SQLite file) to do this. However, if you run cargo sqlx prepare --workspace, it will dump all the data it needs into a .sqlx folder, and no longer require the live database. So, if you include .sqlx in the Nix build environment as well, and place it in the workspace root, this should fix the build errors.

[teto]

wordbase-cli doesn't have a package.version since it's not something I intend on publishing, and is more of a debug/testing tool rather than a proper piece of released software.

ok. It's a bit sad since I think this would be popular :) but the crane message explains how I can work around that so no pb.

I had (unknowingly) removed the .sqlx folder from the build sandbox, adding it back made the wordbase-cli build pass.

cli fails at runtime:

Caused by:
    0: failed to create deinflectors
    1: failed to create Lindera deinflector
    2: failed to load dictionary
    3: LinderaError(kind=Args, source=Invalid dictionary type: UniDic)

because of my change

-lindera              = { workspace = true, features = ["unidic", "compress"] }
+lindera              = { workspace = true, features = ["compress"] }

I will try to restore the unidic feature with your LINDERA_CACHE suggestion

EDIT: for my future self: https://github.com/qdrant/qdrant/blob/master/shell.nix

[aecsocket]

I added the version = "0.0.0" in main to address your first issue, so you shouldn't need to do anything special with crane to get it to build now.

For Lindera, this might be trickier. Lindera does this at build time:

• lindera-unidic's build.rs downloads UniDic from Lindera's own servers (S3 bucket? not sure)
• build.rs parses UniDic and builds up the dict.vals, dict.words, etc. Basically builds that entire folder
• Afterwards, it uses $LINDERA_CACHE/${current lindera version} as a cache for that build process

So Nix will need to not only provide the UniDic dictionary archive, but also allow lindera-unidic's build.rs to build the files from that archive. I would look to existing derivations which include Lindera to figure out how to package this properly.