how to use wids on local dataset

[WudiJoey]

webdataset==0.2.86

shards = [dict(url="lines-000000.tar", nsamples=250)]
ds = wids.ShardListDataset(shards, keep=True)
print(ds[0])

then happen

File "/usr/local/lib/python3.6/dist-packages/wids/wids.py", line 515, in __getitem__
    shard, inner_idx, desc = self.get_shard(index)
  File "/usr/local/lib/python3.6/dist-packages/wids/wids.py", line 510, in get_shard
    shard = self.cache.get_shard(url)
  File "/usr/local/lib/python3.6/dist-packages/wids/wids.py", line 348, in get_shard
    itf = IndexedTarSamples(downloaded, source=url)
  File "/usr/local/lib/python3.6/dist-packages/wids/wids.py", line 205, in __init__
    self.reader = MMIndexedTar(tar_file)
  File "/usr/local/lib/python3.6/dist-packages/wids/wids_mmtar.py", line 50, in __init__
    self._build_index()
  File "/usr/local/lib/python3.6/dist-packages/wids/wids_mmtar.py", line 58, in _build_index
    name = header.name.decode("utf-8").strip("\x00")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x92 in position 2: invalid start byte

[WudiJoey]

网络数据集==0.2.86

shards = [dict(url="lines-000000.tar", nsamples=250)]
ds = wids.ShardListDataset(shards, keep=True)
print(ds[0])

然后发生

File "/usr/local/lib/python3.6/dist-packages/wids/wids.py", line 515, in __getitem__
    shard, inner_idx, desc = self.get_shard(index)
  File "/usr/local/lib/python3.6/dist-packages/wids/wids.py", line 510, in get_shard
    shard = self.cache.get_shard(url)
  File "/usr/local/lib/python3.6/dist-packages/wids/wids.py", line 348, in get_shard
    itf = IndexedTarSamples(downloaded, source=url)
  File "/usr/local/lib/python3.6/dist-packages/wids/wids.py", line 205, in __init__
    self.reader = MMIndexedTar(tar_file)
  File "/usr/local/lib/python3.6/dist-packages/wids/wids_mmtar.py", line 50, in __init__
    self._build_index()
  File "/usr/local/lib/python3.6/dist-packages/wids/wids_mmtar.py", line 58, in _build_index
    name = header.name.decode("utf-8").strip("\x00")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x92 in position 2: invalid start byte

I generate my .tar file using ShardWriter.

[tmbdev]

What you describe should work in principle.

Is it possible that you are using non-ASCII file names/keys inside the tar archive?

As you can tell, mmtar tries to decode file names in tar archives as UTF-8, but I don't know how TarWriter writes them (since it just uses the Python tarfile library).

[WudiJoey]

您好，您的来信我已经收到，会尽快回复。