feat: export a compact version of the database

[marcoconti83]

What's new in this PR

Added a function to flush and export the database. This allows for a compact export without the WAL file.
This only works with SQLite DBs. It won't work on WASM.

[coriolinus]

Looks plausible, pending green CI. I had to double-check the docs, but I agree that using VACUUM INTO on an encrypted database produces encrypted output.

Getting CI green might take a few more iterations, but just doing cargo +nightly fmt and avoiding re-adding the close method should handle the currently-most-obvious failures.

[istankovic]

@marcoconti83 what is the motivation for this?

[github-actions]

Bencher Report

Branch	marco/flush-main
Testbed	ubuntu-latest

⚠️ WARNING: No Threshold found!

Without a Threshold, no Alerts will ever be generated.

• Latency (nanoseconds (ns))

Click here to create a new Threshold
For more information, see the Threshold documentation.
To only post results if a Threshold exists, set the --ci-only-thresholds flag.

Click to view all benchmark results

Benchmark	Latency	microseconds (µs)
Commit add f(group size)/cs1/mem/1002	📈 view plot ⚠️ NO THRESHOLD	18,529.00 µs
Commit add f(group size)/cs1/mem/2	📈 view plot ⚠️ NO THRESHOLD	747.46 µs
Commit add f(group size)/cs1/mem/202	📈 view plot ⚠️ NO THRESHOLD	4,522.50 µs
Commit add f(group size)/cs1/mem/402	📈 view plot ⚠️ NO THRESHOLD	8,326.30 µs
Commit add f(group size)/cs1/mem/602	📈 view plot ⚠️ NO THRESHOLD	11,934.00 µs
Commit add f(group size)/cs1/mem/802	📈 view plot ⚠️ NO THRESHOLD	14,778.00 µs
Commit add f(number clients)/cs1/mem/1002	📈 view plot ⚠️ NO THRESHOLD	988,880.00 µs
Commit add f(number clients)/cs1/mem/2	📈 view plot ⚠️ NO THRESHOLD	703.32 µs
Commit add f(number clients)/cs1/mem/202	📈 view plot ⚠️ NO THRESHOLD	79,187.00 µs
Commit add f(number clients)/cs1/mem/402	📈 view plot ⚠️ NO THRESHOLD	215,460.00 µs
Commit add f(number clients)/cs1/mem/602	📈 view plot ⚠️ NO THRESHOLD	423,830.00 µs
Commit add f(number clients)/cs1/mem/802	📈 view plot ⚠️ NO THRESHOLD	674,040.00 µs
Commit pending proposals f(group size)/cs1/mem/1002	📈 view plot ⚠️ NO THRESHOLD	115,940.00 µs
Commit pending proposals f(group size)/cs1/mem/2	📈 view plot ⚠️ NO THRESHOLD	23,100.00 µs
Commit pending proposals f(group size)/cs1/mem/202	📈 view plot ⚠️ NO THRESHOLD	41,959.00 µs
Commit pending proposals f(group size)/cs1/mem/402	📈 view plot ⚠️ NO THRESHOLD	57,395.00 µs
Commit pending proposals f(group size)/cs1/mem/602	📈 view plot ⚠️ NO THRESHOLD	77,408.00 µs
Commit pending proposals f(group size)/cs1/mem/802	📈 view plot ⚠️ NO THRESHOLD	93,389.00 µs
Commit pending proposals f(pending size)/cs1/mem/1	📈 view plot ⚠️ NO THRESHOLD	17,760.00 µs
Commit pending proposals f(pending size)/cs1/mem/101	📈 view plot ⚠️ NO THRESHOLD	115,830.00 µs
Commit pending proposals f(pending size)/cs1/mem/21	📈 view plot ⚠️ NO THRESHOLD	35,417.00 µs
Commit pending proposals f(pending size)/cs1/mem/41	📈 view plot ⚠️ NO THRESHOLD	56,347.00 µs
Commit pending proposals f(pending size)/cs1/mem/61	📈 view plot ⚠️ NO THRESHOLD	75,914.00 µs
Commit pending proposals f(pending size)/cs1/mem/81	📈 view plot ⚠️ NO THRESHOLD	95,289.00 µs
Commit remove f(group size)/cs1/mem/1002	📈 view plot ⚠️ NO THRESHOLD	11,071.00 µs
Commit remove f(group size)/cs1/mem/2	📈 view plot ⚠️ NO THRESHOLD	583.14 µs
Commit remove f(group size)/cs1/mem/202	📈 view plot ⚠️ NO THRESHOLD	2,456.40 µs
Commit remove f(group size)/cs1/mem/402	📈 view plot ⚠️ NO THRESHOLD	4,374.90 µs
Commit remove f(group size)/cs1/mem/602	📈 view plot ⚠️ NO THRESHOLD	7,123.30 µs
Commit remove f(group size)/cs1/mem/802	📈 view plot ⚠️ NO THRESHOLD	8,805.70 µs
Commit remove f(number clients)/cs1/mem/1002	📈 view plot ⚠️ NO THRESHOLD	14,072.00 µs
Commit remove f(number clients)/cs1/mem/2	📈 view plot ⚠️ NO THRESHOLD	136,160.00 µs
Commit remove f(number clients)/cs1/mem/202	📈 view plot ⚠️ NO THRESHOLD	111,310.00 µs
Commit remove f(number clients)/cs1/mem/402	📈 view plot ⚠️ NO THRESHOLD	86,525.00 µs
Commit remove f(number clients)/cs1/mem/602	📈 view plot ⚠️ NO THRESHOLD	62,206.00 µs
Commit remove f(number clients)/cs1/mem/802	📈 view plot ⚠️ NO THRESHOLD	38,217.00 µs
Commit update f(group size)/cs1/mem/1002	📈 view plot ⚠️ NO THRESHOLD	137,170.00 µs
Commit update f(group size)/cs1/mem/2	📈 view plot ⚠️ NO THRESHOLD	754.18 µs
Commit update f(group size)/cs1/mem/202	📈 view plot ⚠️ NO THRESHOLD	28,041.00 µs
Commit update f(group size)/cs1/mem/402	📈 view plot ⚠️ NO THRESHOLD	55,130.00 µs
Commit update f(group size)/cs1/mem/602	📈 view plot ⚠️ NO THRESHOLD	82,866.00 µs
Commit update f(group size)/cs1/mem/802	📈 view plot ⚠️ NO THRESHOLD	109,680.00 µs

🐰 View full continuous benchmarking report in Bencher

[SimonThormeyer]

If we feature-gated this entire function instead:

#[cfg(not(feature = "wasm"))]

we could prevent this function to be generated for typescript entirely. Or was there a particular reason why you wanted to expose this function for TypeScript?

[marcoconti83]

I tried to feature-gate the entire function. I then spent hours trying to get the WASM tests to run - they would fail, because even if the tests were not using this function, there was a step in the test setup that checks that all functions part of the FFI are defined, and it would fail. I didn't dig deeper into why this happens, but removing this check seems like a big architectural change that is beyond the scope of this PR.

Auto-generating stubs for WASM did not work because the function just does not exists when using the WASM target. So I tried manually generating stubs, but either we hardcode the stubs (and we need to keep them in sync with the code as the code changes), or we would have a script that looks for functions that don't exists and generates stubs for them (which feels very hacky).
I decided to just throw an error when running this in WASM, and this makes the tests happy.

[marcoconti83]

Note that this function will only ever be called by Android anyway

[SimonThormeyer]

Are you sure that you annotated the function with exactly the attribute I'm mentioning above?

It's unintuitive at first that if we want to hide a part of the API for typescript, we cannot compile conditionnally by target family, but have to use that feature-gate. It's due to how uniffi-bindgen-for-react-native (ubrn) generates the bindings. They are generated from a binary that is compiled for the host platform. So if you'd compile conditionally by target family, the function would still appear in the binary the bindings are generated from, which would then try to bind to a function that doesn't exist on the target.

I did an experiment in 96307cc where I feature-gate the function. Typescript tests are passing there for me.

[coriolinus]

IMO building on Simon's work is the way to go here. Possibly that will also solve the interop test issue?

[SimonThormeyer]

The interop test issue will probably be fixed with a rebase on main: on main, we're creating android emulators freshly for each android test (including interop), and this branch assumes emulators are reused across test runs.