Skip to content

ZSTD decoder: cleanups, new sequence executor and fixes for subcomponents #3636

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
magancarz wants to merge 85 commits into
base: main
Choose a base branch
from
Open

ZSTD decoder: cleanups, new sequence executor and fixes for subcomponents #3636

magancarz wants to merge 85 commits into from
+15,112 −5,743

Conversation

@magancarz
Copy link

@magancarz magancarz commented Dec 31, 2025

Added functionality:

  • Cleaned up the ZSTD decoder and cocotb tests environment
  • Fixed the ZSTD decoder subcomponents
  • Added a detailed testing environment with intermediate values checks
  • Added a CLI tool for running decoder test cases
  • Refactored the sequence executor architecture
  • Added fixes for parameterizing AXI buses width
  • Added a verilog model for 2RW ram
  • Added Verilator as a new dependency to improve cocotb tests runtime
  • Updated README
    • Updated and added new information about the components of the ZSTD Decoder
    • Added diagrams to present the updated structure
  • Added a workaround for a bug causing the xls_benchmark_ir targets to fail

Supersedes #3035

rw1nkler and others added 30 commits December 30, 2025 11:03
@rw1nkler @magancarz
Add a separate test case for each test file
5d6cc6c 
Signed-off-by: Robert Winkler <rwinkler@antmicro.com>
@sgizler @magancarz
Add "pytype binary, library" hint
b7e3741
@magancarz
Add cocotb tests
ce75b89 
Add cocotb testing utilities:
- XLSStruct for easier handling and serializing/deserializing XLS structs
- XLSChannel that serves as a dummy receiving channel
- XLSMonitor that monitors transactions on an XLS channel
- XLSDriver that can send data on an XLS channel
- LatencyScoreboard that can measure latency between corresponding transactions on input and output buses
- File-backed AXI memory python model

Add cocotb tests for:
- modules/zstd/memory/MemReader
- modules/zstd/memory/AxiWriter
- modules/zstd/memory/MemWriter

Co-authred-by: Pawel Czarnecki <pczarnecki@antmicro.com>
Co-authred-by: Robert Winkler <rwinkler@antmicro.com>
Co-authred-by: Michal Czyz <mczyz@antmicro.com>
Signed-off-by: Michal Czyz <mczyz@antmicro.com>
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>
Signed-off-by: Robert Winkler <rwinkler@antmicro.com>
Signed-off-by: Krzysztof Obłonczek <koblonczek@antmicro.com>
@lpawelcz @magancarz
xls/modules/zstd: expose fifo verilog module
5b3f1d0 
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>
@lpawelcz @magancarz
modules/zstd/cocotb: Add ZSTD frame generator library
82b442f 
This reverts commit 04ad379225b706ddf492d440c673e77348d7a409.
@lpawelcz @magancarz
dependency_support: Add zstandard python library
3eec9be 
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>
@lpawelcz @magancarz
modules/zstd: Add verilog simulation of the ZstdDecoder
595df0f 
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>
@magancarz @rw1nkler
Use frame generator in tests
6a7294f 
Co-authored-by: Robert Winkler <rwinkler@antmicro.com>
Signed-off-by: Robert Winkler <rwinkler@antmicro.com>
Signed-off-by: Krzysztof Obłonczek <koblonczek@antmicro.com>
@lpawelcz @magancarz
modules/zstd/zstd_dec: Write decoded data to the memory
8db2cc7 
ZstdDecoder:
  * Cocotb tests:
    * Move third-party verilog modules (AXI Interconnect) to external directory
    * Replace AXI Interconnect with AXI Crossbar that handles simultaneous AXI Read
      and Write transactions
    * Add reference memory and fill it with expected data for comparison
      against testbench memory at the end of the decoding

Internal-tag: [#67272]
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>
@lpawelcz @magancarz
modules/zstd/zstd_dec: Remove stream-based output interface
a0fafec 
Adjust cocotb test to:
* Removal of Repacketizer proc
* Removal of stream-based output channels from
  * SequenceExecutor
  * ZstdDecoder

Internal-tag: [#67272]
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>
@lpawelcz @magancarz
modules/zstd/zstd_dec_cocotb_test: Improve Verilog simulation
a585d16 
* Decode multiple ZSTD frames in a single cocotb testbench
* Add one cocotb testbench per type of the ZSTD frames:
  * Frames with RAW blocks only
  * Frames with RLE blocks only
  * Frames with Compressed blocks only (disabled)
  * Frames with mixed blocks (disabled)

Internal-tag: [#67272]
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>
@lpawelcz @magancarz
modules/zstd/memory/mem_writer: Reduce the amount of random test cases
3796c90 
Caused by timeouts after rebase - looks like regression in the
performance of codegen

Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>
@lpawelcz @magancarz
modules/zstd/memory: Move verilog sources to rtl subdirectory
3de5586 
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>
@lpawelcz @magancarz
xls/modules/zstd/external: Add docstring and license for third party …
fe86865 
…sources

Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>
@lpawelcz @magancarz
modules/zstd: Adjust cocotb test to removal of RESET CSR
806c4b4 
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>
@wsipak @lpawelcz @magancarz
Style changes
25d9d9a 
Improve formatting, wording and fix lint issues

Co-authored-by: Pawel Czarnecki <pczarnecki@antmicro.com>
Signed-off-by: Wojciech Sipak <wsipak@antmicro.com>
@rw1nkler @magancarz
Extend cocotb tests of ZSTD decoder
1cfd144 
Co-authored-by: Pawel Czarnecki <pczarnecki@antmicro.com>
Co-authored-by: Krzysztof Oblonczek <koblonczek@antmicro.com>
Co-authored-by: Wojciech Sipak <wsipak@antmicro.com>
Co-authored-by: Dominik Lau <dlau@antmicro.com>
Co-authored-by: Szymon Gizler <sgizler@antmicro.com>

Signed-off-by: Robert Winkler <rwinkler@antmicro.com>
@rw1nkler @magancarz
Collect P&R benchmarks
1179e34 
Signed-off-by: Robert Winkler <rwinkler@antmicro.com>
@rw1nkler @magancarz
Add fixed version of RamDemux together with a test
6810b5d 
Signed-off-by: Robert Winkler <rwinkler@antmicro.com>
@sgizler @magancarz
Document decoder testing
5e0701a
@rw1nkler @magancarz
dependency_support: Add verilog_axi package
4687a49 
Signed-off-by: Robert Winkler <rwinkler@antmicro.com>
@rw1nkler @magancarz
Use bazel package with verilog_axi
112d089 
Signed-off-by: Robert Winkler <rwinkler@antmicro.com>
@rw1nkler @magancarz
Downgrade dependencies
a05cfba 
Signed-off-by: Robert Winkler <rwinkler@antmicro.com>
@rw1nkler @magancarz
Run tests selectively in GitHub CI
d17570b 
Signed-off-by: Robert Winkler <rwinkler@antmicro.com>
@rw1nkler @magancarz
Remove zstandard python library
2ce8a34 
Signed-off-by: Robert Winkler <rwinkler@antmicro.com>
@rw1nkler @magancarz
modules: zstd: update readme
06faeb1 
Co-autored-by: Dominik Lau <dlau@antmicro.com>
Signed-off-by: Robert Winkler <rwinkler@antmicro.com>
@rw1nkler @magancarz
Add cocotb check for verifying huffman codes
4296f1d 
Signed-off-by: Robert Winkler <rwinkler@antmicro.com>
@rw1nkler @magancarz
modules: zstd: cocotb: drop unused testcase
904a651 
Co-autored-by: Dominik Lau <dlau@antmicro.com>
Signed-off-by: Robert Winkler <rwinkler@antmicro.com>
@rw1nkler @magancarz
modules: zstd: disable huffman code builder PnR
7073d51 
Co-autored-by: Dominik Lau <dlau@antmicro.com>
Signed-off-by: Robert Winkler <rwinkler@antmicro.com>
@rw1nkler @magancarz
modules: zstd: fix formatting
047ce03 
Co-autored-by: Dominik Lau <dlau@antmicro.com>
Signed-off-by: Robert Winkler <rwinkler@antmicro.com>
sgizler and others added 26 commits December 31, 2025 14:22
@sgizler @magancarz
modules: zstd: bump verilator dependency
f28568c
@sgizler @magancarz
modules: zstd: cocotb: run verilator on a single thread
d56a9c7
@sgizler @magancarz
modules: zstd: cocotb: don't make unused signals public
e985861 
cocotb marks all signals as public, siginficantly restricting verilator
from doing optimizations - public signal can change anytime and verilator
must take it into account

Instead of using global public flag, we list individual public
signals/modules in verilator config
@sgizler @magancarz
modules: zstd: huffman literals dec: fix broken raw huffman weights o…
603a7a1 
…rdering

First symbol is stored in the most significant nibble of given input
byte requiring us to swap each adjecent weights (rfc8878#section-4.2.1.1)
@sgizler @magancarz
modules: zstd: huffman weights dec: fix broken tree_description_size …
558b85a 
…calculation

As defined in https://datatracker.ietf.org/doc/html/rfc8878#section-4.2.1.1
raw weights consume ceil(n_symb/2) bytes.

The correct equivalent of that ceiled div is:
(n_symb+1)>>1
rather than
(n_symb>>1)+1
we previously used
@sgizler @magancarz
modules: zstd: huffman weights dec: fix bad insertion position of las…
1d38b0c 
…t weight

Previous version did not take into account weight swapping.
@sgizler @magancarz
modules: zstd: huffman weights dec: test weight decoding bugfixes
cd8d923 
Old test reimplemented parts of decoder and compared output of proc
against reimplementation. It had very little value (as we are likely to
slip same bug into both implementations) and was hard to reason about.

New test compares decoder against readable explicit golden prepared to
test all recently fixed bugs (swapped weights, bad description size
and wrong last weight position)
@magancarz
modules: zstd: fse table decoder: fix handling last read bit if it is…
297ecfb 
… a remainder
@wsipak @magancarz
modules: zstd: rtl: ram_1r1w: fix handling of simultaneous read and w…
4b7594e 
…rite
@magancarz
modules: zstd: huffman ctrl: omit building new codes when decoding tr…
c95e9fe 
…eeless literals
@magancarz
modules: zstd: cocotb: print correct packet values less often
0770058
@magancarz
modules: zstd: huffman weights dec: remove duplicate call
9bbcc7b
@magancarz
modules: zstd: huffman weights dec: add more test cases for raw weights
a464972
@magancarz
modules: zstd: huffman weights dec: send last raw weight in the next …
ccd52c5 
…iteration if required
@magancarz
modules: zstd: performance: add generating a throughput benchmark
b42afce
@magancarz
modules: zstd: sequence executor: fix clearing current history buffer…
e69c28b 
… length
@rw1nkler @magancarz
modules: zstd: remove outdated/commented code
9d67da9 
Signed-off-by: Robert Winkler <rwinkler@antmicro.com>
@rw1nkler @wsipak @magancarz
modules: zstd: update README
355fc0a 
Co-authored-by: Wojciech Sipak <wsipak@antmicro.com>
Signed-off-by: Robert Winkler <rwinkler@antmicro.com>
@magancarz
modules: zstd: fix errors after rebase
6fbfeb4
@magancarz
modules: zstd: cocotb: update deps to python 3.12
a2eded8
@magancarz
dependencies: update python lock file
1d3a2d7
@magancarz
examples: ram: make 2RW ram proc public
12fe66f
@magancarz
modules: zstd: sequence executor: rewrite seq exec architecture
71c2dcd
@magancarz
modules: zstd: fix parametrizing AXI bus width
b345c56
@magancarz
modules: zstd: set worst case throughput to 5
5a55abf
@magancarz
passes: check if a node is tracked before getting info
412ca7f
@magancarz magancarz mentioned this pull request Dec 31, 2025
Open
@rw1nkler rw1nkler mentioned this pull request Jan 2, 2026
Closed
@grebe grebe requested a review from proppy January 2, 2026 22:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@proppy proppy Awaiting requested review from proppy

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@magancarz @rw1nkler @sgizler @lpawelcz @wsipak