[OPTIMIZATION] make source-ref line extraction faster

[nachumg]

tl;dr: DirectSourceRef should store the line-number in addition to start_idx and end_idx.

• Then the line property does not need to look at the source file.

DirectSourceRef._extract_line_info scans the entire file in order to discover a component's line number in its source file.

Profiling shows that peakrdl_html export spends 30% of the time in this method.

For very large models, the peakrdl_html runtime explodes to many hours.

• Disabling source-links reduces the runtime to a small fraction of that.

[nachumg]

Turns out that passing-to (or calculating) the line-number of each DirectSourceRef when it is created, is quite difficult.

• All we have is the source-text, and the start-index in that text where the source-ref starts.
• Discovering the line number requires counting the number of newlines until the start-index.
• Doing that for every source-ref will require multiple passes over the source-text, which will slow down the compiler.

Seems to me that the easiest way to do this is to add a fast_line property alongside DirectSourceRef.line(https://github.com/SystemRDL/systemrdl-compiler/blob/main/src/systemrdl/source_ref.py#L143) that will:

1. Cache already-visited source file content.
2. Scan that content efficiently to count #-of-lines until the DirectSourceRef._start_idx

Then HTML exporter (and other exporters) could use fast_line instead of line and hopefully run faster.

Does someone have a better idea?

[amykyta3]

Thanks for the analysis! I'll have to take a closer look at this to see what is feasible.

The current implementation for SourceRef objects essentially made the assumption that they would only ever get "resolved" upon firing off an error message. In this situation performance was never particularly important. Of course as you point out, the HTML exporter resolves all source refs which I agree would certainly become expensive!

I suspect the reason I originally implemented it this way (to discard the source line number) is realated to how preprocessors can mangle the input text. Very likely that when I implemented this, it was easier to reverse-transform the start/stop index and re-compute the line number later. I wonder if somehow carrying forward the line number through the transformations would be a better idea.

[seimei-d]

+1 from the LSP angle. In systemrdl-pro (an LSP plus viewer on top of systemrdl-compiler), source-ref resolution is on the hot path for diagnostics, hover, and goto-definition - typically dozens of resolutions per viewport refresh during editing. Pre-computing line/col offsets at parse time would help LSPs as much as it helps peakrdl_html. Can share LSP-side traces if useful.