Skip to content

Feat: add async VCF streaming scanner#179

Open
wjixiang wants to merge 2 commits into
abdenlab:mainfrom
wjixiang:feat/async-vcf-scanner
Open

Feat: add async VCF streaming scanner#179
wjixiang wants to merge 2 commits into
abdenlab:mainfrom
wjixiang:feat/async-vcf-scanner

Conversation

@wjixiang

@wjixiang wjixiang commented Jul 2, 2026

Copy link
Copy Markdown

Add async streaming support for VCF files via a new AsyncScanner trait.

Changes

  • Add AsyncScanner trait (async_scanner.rs) with a stream-based
    scan() method
  • Implement async scan on the existing Scanner struct using
    futures::try_unfold
  • Move VcfModelBuilder to variant::model as ModelBuilder
  • Add async test coverage (tokiotest)
  • Remove unused generic Select<R>

Example usage

let scanner = Scanner::with_model(header, model);
let stream = scanner.scan(async_reader, Some(1024), None);
while let Some(batch) = stream.next().await {
    // process RecordBatch
}

Future work

  • Async scan_query (requires AsyncSeek)
  • Async scan_byte_ranges / scan_virtual_ranges

wjixiang added 2 commits July 2, 2026 11:52
- Remove `enum Select<R>` with `Select`, using String to select field.
- Verify String is the only type passed through this generic
- Pass all unit tests
Implement AsyncScanner trait with a stream-based scan method using
futures::try_unfold. The trait is implemented on the existing Scanner
struct, allowing both sync and async scanning from the same instance.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant