Skip to content

Support loading/streaming data from s3 locations #1

@nihaljn

Description

@nihaljn

This is what's supported at the moment:

  • Local - JSON, JSONL, HuggingFace parquet
  • Remote
    • HuggingFace Hub - HuggingFace parquet

Supporting S3 loading/streaming would entail supporting the following:

  • Local - JSON, JSONL, HuggingFace parquet
  • Remote
    • HuggingFace Hub - HuggingFace parquet
    • S3 - JSON, JSONL, HuggingFace parquet

Metadata

Metadata

Assignees

Labels

No labels
No labels

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions