Temporal Memory extreamly slow when benchmarking with random inputs

[marty1885]

Hi,

Since @breznak mentioned about TM performance in #680 - I was benchmarking HTM.core's TM implementation vs mine and got really weird results.

Methodology

I adopted the tmbench.cpp code from here. The idea is to:

1. Set both TM's parameter to run in the same condition
2. Generate an array of 1000 random SDR with 10% density (using a scalar encoder)
3. Send the SDRs to the TM one-by-one, until we passed all the SDRs
4. Measure how long 3 takes
5. Repeat for different SDR sizes

Source code (modified so I can generate plot from them):
For HTM.core
For Etaler

Result

The result is weird. I think I'm doing something horribly wrong. But I have no idea without deep knowledge about HTM.core's internal.

---

Edit: I believe the problem might be related to some old forum posts
TemporalMemory running very slow after long training

[breznak]

Thanks a lot @marty1885 for benchmarking and reporting back!

• The methodology looks sound.
  • (I haven't reviewed the TM params if there's some hiccup there, but still fair as all implementations use the same setting)
• is your cmake correct?

My current best results (on branch speedup_comparison (I'll update it with progress soon) which we used for the comparison on the forums before, it is inspired by your bench and has similar (not same as here) params):

Built target benchmark_hotgym
starting test. DIM_INPUT=9000, DIM=9000, CELLS=16
EPOCHS = 1000

Init:   0.328663
Random: 0.259727
Encode: 0.112999
SP (g): 1.42247
TM:     23.006
AN:     0.00689343
Total elapsed time = 24 seconds

Which is roughly 23ms/iter for TM, so the reported 95ms/iter looks off somewhere.

Edit: I believe the problem might be related to some old forum posts
TemporalMemory running very slow after long training

I remember this issue (think it was valid for TP, not TM, but not sure). I'll revisit it 👍
Seems the benchmark is not that "long training", 1000 iters, right? That's what I'm using in the results above.

[marty1885]

is your cmake correct

What you you mean? I'm using a Release build of HTM.core/linked to the shared lib. The benchmarking code is compiled with c++ tm_benchmark_htmcore.cpp -o tm_benchmark_htmcore -O3 -flto -march=native -lhtm_core -std=c++17

so the reported 95ms/iter looks off somewhere.

Something have to be horribly wrong. The 95ms/iter is for 1000 bits/SDR. It grows to 15000ms/iter at 4000 bits/SDR. The line literally goes off the chart.

May you check my bencharking code when you have time? It would be great to have performance compares in my project report.

[breznak]

Just an update: I've added TM test that runs this benchmark and "checks" for reasonable performance.

Some notes:

• py bindings TM speed: ~30ms/it (per 1000 iters, 9000 input/output size, default TM params)

Which is roughly 23ms/iter for TM [for C++ TM]

• so python bindings are roughly 1.5x slower (in this case)

For large number of steps (iters) the speed decreases. So there is still that problem? Needs to be said that this scenario is somewhat "unnatural" (completely random inputs, 10% random each step).