GDB only reports partial call stack after indirect cross-compartment call

[randomPoison]

While working on the Firefox demo, I ran into a situation where GDB wasn't able to display the full call stack after a compartment transition. When following an indirect function call from libjpeg (the compartmentalized library) back to firefox, GDB will only display the callstack starting at the compartment transition:

(gdb) bt
#0  0x00007fffe7abc4cc in mozilla::image::my_error_exit (cinfo=0x21fc00244238) at /home/legare/ff-rewritten/image/decoders/nsJPEGDecoder.cpp:761
#1  0x00007fffe7ab8f7c in __ia2_my_error_exit () at /home/legare/ff-rewritten/obj-x86_64-pc-linux-gnu/dist/bin/libxul.so

Note how the call stack starts at __ia2_my_error_exit, which is the indirect callgate, instead of showing the rest of the calls that came before the compartment transition.

For comparison, when Firefox calls directly into libjpeg, GDB is able to report the full call stack across the compartment transition:

(gdb) bt
#0  jpeg_read_scanlines (cinfo=0x3d880023c238, scanlines=0x3d8800204080, max_lines=1) at /home/legare/ff-rewritten/media/libjpeg/jdapistd.c:315
#1  0x00007ffff7e18ff3 in __wrap_jpeg_read_scanlines () at /home/legare/ff-rewritten/libcallgates.so
#2  0x00007fffe7abe566 in mozilla::image::nsJPEGDecoder::OutputScanlines()::$_0::operator()(unsigned int*, int) const (this=0x7ffe7bfff340, aPixelBlock=0x3d8800244000, aBlockSize=160)
    at /home/legare/ff-rewritten/image/decoders/nsJPEGDecoder.cpp:697
#3  0x00007fffe7abe2d3 in mozilla::image::SurfaceFilter::DoWritePixelBlockToRow<unsigned int, mozilla::image::nsJPEGDecoder::OutputScanlines()::$_0>(mozilla::image::nsJPEGDecoder::OutputScanlines()::$_0) (this=0xfe01045aca0, aFunc=...) at /home/legare/ff-rewritten/image/SurfacePipe.h:549
#4  0x00007fffe7abe112 in mozilla::image::SurfaceFilter::WritePixelBlocks<unsigned int, mozilla::image::nsJPEGDecoder::OutputScanlines()::$_0>(mozilla::image::nsJPEGDecoder::OutputScanlines()::$_0) (this=0xfe01045aca0, aFunc=...) at /home/legare/ff-rewritten/image/SurfacePipe.h:219
#5  0x00007fffe7abc408 in mozilla::image::SurfacePipe::WritePixelBlocks<unsigned int, mozilla::image::nsJPEGDecoder::OutputScanlines()::$_0>(mozilla::image::nsJPEGDecoder::OutputScanlines()::$_0) (this=0x3d880023c6c8, aFunc=...) at /home/legare/ff-rewritten/image/SurfacePipe.h:677
#6  0x00007fffe7abc1c8 in mozilla::image::nsJPEGDecoder::OutputScanlines (this=0x3d880023c000) at /home/legare/ff-rewritten/image/decoders/nsJPEGDecoder.cpp:688
#7  0x00007fffe7abb5af in mozilla::image::nsJPEGDecoder::ReadJPEGData (this=0x3d880023c000, aData=0x3d8800238000 "\377\330\377", <incomplete sequence \340>, aLength=3643)
    at /home/legare/ff-rewritten/image/decoders/nsJPEGDecoder.cpp:476
#8  0x00007fffe7abdd4c in mozilla::image::nsJPEGDecoder::DoDecode(mozilla::image::SourceBufferIterator&, mozilla::image::IResumable*)::$_0::operator()(mozilla::image::nsJPEGDecoder::State, char const*, unsigned long) const (this=0x7ffe7bfff798, aState=mozilla::image::nsJPEGDecoder::State::JPEG_DATA, aData=0x3d8800238000 "\377\330\377", <incomplete sequence \340>, aLength=3643)
    at /home/legare/ff-rewritten/image/decoders/nsJPEGDecoder.cpp:212
#9  0x00007fffe7abde2a in mozilla::image::StreamingLexer<mozilla::image::nsJPEGDecoder::State, 16ul>::ContinueUnbufferedRead<mozilla::image::nsJPEGDecoder::DoDecode(mozilla::image::SourceBufferIterator&, mozilla::image::IResumable*)::$_0>(char const*, unsigned long, unsigned long, mozilla::image::nsJPEGDecoder::DoDecode(mozilla::image::SourceBufferIterator&, mozilla::image::IResumable*)::$_0) (this=0x3d880023c190, aData=0x3d8800238000 "\377\330\377", <incomplete sequence \340>, aLength=3643, aChunkLength=3643, aFunc=...)
    at /home/legare/ff-rewritten/image/StreamingLexer.h:555
#10 0x00007fffe7abd806 in mozilla::image::StreamingLexer<mozilla::image::nsJPEGDecoder::State, 16ul>::UnbufferedRead<mozilla::image::nsJPEGDecoder::DoDecode(mozilla::image::SourceBufferIterator&, mozilla::image::IResumable*)::$_0>(mozilla::image::SourceBufferIterator&, mozilla::image::nsJPEGDecoder::DoDecode(mozilla::image::SourceBufferIterator&, mozilla::image::IResumable*)::$_0)
    (this=0x3d880023c190, aIterator=..., aFunc=...) at /home/legare/ff-rewritten/image/StreamingLexer.h:501
#11 0x00007fffe7aba84a in mozilla::image::StreamingLexer<mozilla::image::nsJPEGDecoder::State, 16ul>::Lex<mozilla::image::nsJPEGDecoder::DoDecode(mozilla::image::SourceBufferIterator&, mozilla::image::IResumable*)::$_0>(mozilla::image::SourceBufferIterator&, mozilla::image::IResumable*, mozilla::image::nsJPEGDecoder::DoDecode(mozilla::image::SourceBufferIterator&, mozilla::image::IResumable*)::$_0) (this=0x3d880023c190, aIterator=..., aOnResume=0xfe01052c7f8, aFunc=...) at /home/legare/ff-rewritten/image/StreamingLexer.h:469
#12 0x00007fffe7aba37a in mozilla::image::nsJPEGDecoder::DoDecode (this=0x3d880023c000, aIterator=..., aOnResume=0xfe01052c7f8)
    at /home/legare/ff-rewritten/image/decoders/nsJPEGDecoder.cpp:208
#13 0x00007fffe7988b69 in mozilla::image::Decoder::Decode (this=0x3d880023c000, aOnResume=0xfe01052c7f8) at /home/legare/ff-rewritten/image/Decoder.cpp:190
#14 0x00007fffe7986502 in mozilla::image::DecodedSurfaceProvider::Run (this=0xfe01052c780) at /home/legare/ff-rewritten/image/DecodedSurfaceProvider.cpp:125
#15 0x00007fffe79859f2 in mozilla::image::DecodingTask::Run (this=0xfe0104cd360) at /home/legare/ff-rewritten/image/DecodePool.cpp:153
#16 0x00007fffe48b10c5 in mozilla::TaskController::RunTask (aTask=0xfe0104cd360) at /home/legare/ff-rewritten/xpcom/threads/TaskController.cpp:196
#17 0x00007fffe48b18e5 in mozilla::TaskController::RunPoolThread (this=0xfe00006d180, aThread=0xfe00127bf80) at /home/legare/ff-rewritten/xpcom/threads/TaskController.cpp:433
#18 0x00007fffe48b1765 in mozilla::ThreadFuncPoolThread (aData=0xfe00127bf80) at /home/legare/ff-rewritten/xpcom/threads/TaskController.cpp:319
#19 0x00007ffff7ccfbbc in _pt_root (arg=0x555555a0c600) at /home/legare/ff-rewritten/nsprpub/pr/src/pthreads/ptthread.c:190
#20 0x000055555560b541 in set_alt_signal_stack_and_start (params=0xfe00161cde0) at /home/legare/ff-rewritten/mozglue/interposers/pthread_create_interposer.cpp:83
#21 0x00007fffef583408 in ia2_thread_begin (arg=0xaaaaaaaa00000000) at /home/legare/IA2-Phase2/runtime/libia2/threads.c:52

I haven't tried to put together a minimal repro of this, but here are some additional details that may be relevant:

• libjpeg is being compartmentalized with library-only mode.
• We don't have the usual IA2_CALL macro in use when doing the indirect call from libjpeg (since in library-only mode we don't modify the sources of the compartmentalized library).