FPGA for Computer Vision Tasks over PCIe

Hardware

RHS Research NiteFury II

• Artix-7 200T (XC7A200T-2FBG484E)
• 1GB DDR3-800 RAM
• 256 MB Flash

Computer with M.2 M-Key Port

• USB-A to M.2 DOES NOT WORK
• Need direct M.2 on motherboard or M.2 to Thunderbolt (need verification)
• Current PoC will be on an Orange Pi 5 Plus, which includes an M.2 M-Key slot

Software/Drivers

AMD Vitis 2024.2

• Includes Vivado and Vitis HLS

RapidStream TAPA

• HLS C++ framework for high-frequency FPGA dataflow accelerators
• Requires Vitis

Xilinx XDMA IP Drivers

• For memory-mapping over PCIe

Resources

RHS Research NiteFury/LiteFury example project

• Includes basic Vivado project (outdated, needs IP upgrades for 2024.2) and sample host code for DDR transfer

Project Timeline

✅ Initial sample testing, data transfer

• Successfully installed software/drivers on Arch Linux
• Successfully built RHS Research sample project kernel (with IP upgrades)
• Successfully ran RHS Research sample host code script for both device info reading and DDR transfer (1 GB in ~1.3s)

🔳 Test arbitrary operation on data, performance log

• Design basic vector operation (vadd/vmul/dot product) as TAPA HLS kernel
• Test correctness with C-sim
• Integrate with Vivado Block Diagram and XDMA IP for communication
• Create testing script for correctness and measuring latency

🔳 Godot initial project structure

• Set up basic Godot project structure
• Set up assets

✅ Godot basic city scene

• Add roads, traffic signs/lights, sidewalks, buildings, trees, vehicles

❌ Godot vehicle controls

• Add user controls for vehicle
• Add movement for NPC vehicles

❌ Godot simulator sensor I/O

• Save stereo camera buffers from Godot and write to file or stream
• Read processed depth buffer from file or stream

❌ C++ implementation and simulation SGM kernel, performance log cycles

• Implement SGM kernel in C++, test with C-sim, and verify results in Python with OpenCV
• Log cycle count

❌ Hardware emulation SGM kernel, performance log cycles

• Test SGM kernel with v++ hardware emulation
• Log cycle count

❌ Hardware implementation SGM kernel, performance log cycles/latency

• Integrate SGM kernel with Vivado Block Diagram with XDMA IP
• Create testing script for communication
• Log cycle count/end-to-end latency

❌ Godot-XDMA Integration

• Transfer image data between Godot simulator and XDMA memory-mapped buffer