Reducing Tracing overhead with LTTng

[yanyh15]

So far, based on Lulesh example, we have 12-45% (2,4,8,16,32,48,56 threads on carina) overhead of tracing when using gcc-5.4 without energy measurement. With energy measurement, the overhead is 250% - 750%. Our goal is to reduce the overhead to 5% or even less, overhead >=10% is hardly acceptable.

Approaches:

1. Reduce unnecessary trace events, sync_region and sync_regin_wait pairs are all together, so we only need to trace one pair, say sync_region_begin and sync_region_end. We need to check other events.
2. Turn on and off tracing at runtime, needs to check OMPT to see whether this can be done from OMPT callback. Or using the approach of assigning log level to a tracepoint so we can turn on/off tracepoint.
3. Eliminate thread_id field in each events, this needs to be checked with LTTng channel streaming, etc. E.g. if an OpenMP thread migrates from one core to another, how the traces stream to LTTng
4. Choose smaller data types for certain event files to reduce event record size, e.g. thread_id could be a short type if it needed.
5. Consult with LTTng about performance and the mechanism of tracing records, e.g. can we not creating LTTng threads for each channel and let the tracepoints just write to memory buffer

[yanyh15]

In HPC, performance is a bug :-)

[philipaconrad]

Record size reduction

I think of the items mentioned in the original post, item 4 (reduce size of types used in trace records) seems like the best option to me for reducing overheads everywhere.

In my experience with LULESH, the application's traces are almost entirely events-of-interest (parallel_begin, implicit_task_begin, etc.), and thus there's not a lot of low-value events to remove.

Reducing the size of the trace records we generate should reduce the write pressure on LTTng's in-memory buffers.

Live client

One crazier idea is that if we wanted to do all of our trace analysis "live", we could hook into the undocumented LTTng live protocol that Babeltrace 1 knows how to use. We would essentially be writing a live client (like Babeltrace, or TraceCompass's Live Mode) that would aggregate trace data.

It would be semi-painful to write up (due to overall lack of docs), and would take time, but it might be an efficient approach worth checking out later on.

[philipaconrad]

Whoops, sorry. Clicked the wrong button in the Github user interface.

[philipaconrad]

In response to Approach 3, I'm going to run an experiment or two with our existing tracing setup to see if thread migration happens in real workloads (like LULESH or kdtree).

• If no migration occurs, we only need to track the thread_id in the thread_begin and thread_end fields.
• If migration does occur, then we are in trouble, and will need to keep the thread_id field around for all events.

[yanyh15]

Check this thread of discussion with LTTng developer: https://lists.lttng.org/pipermail/lttng-dev/2018-December/028450.html

Basically, we need to keep thread_id for all events and LTTng collects tracepoint events from each CPU. If a thread migrates from one cpu to another, events feeds will go from one CPU to another.

In reality, thread migration is rare, so it likely won't happen at all in most cases. But even with that, since LTTng keeps all tracepoints events from all CPUs linearly in chronological order, we likely need that thread_id for all events. This also indicates that parallel processing of trace records will be very limited.