[Offload] Change Offload build to dlopen libhsa-runtime

[jhuber6]

Summary:
There were some previous discussions about directly linking HSA in the
offload build. Currently, we have special logic for The Rock build to
pull out the in-progres ROCr build so we can directly link against it. I
am asserting that this just introduces build insanity for very little
benefit.

The only true benefit of direct linking over the dlopen shim is that
we get compile-time verification of symbols. using dlopen will already
respect the runpath that we set for The Rock build. I would not expect
there to be a significant amount of overhead for this either, as either
ld.so walks the symbols or we do.

The reamaining build issue is the Emissary APIs, these are enabled by
default and do not cover any runtime cases that the upstream support
does not. This prevents it from building offload at all if you do
not have flang_rt configured, unlike what upstream does. This PR works
around that without fully disabling it.

My desperate attempt to make the downstream build of offload/ more
sane.

[z1-cciauto]

PSDB Link: https://compiler-ci.amd.com/job/compiler-psdb-amd-staging/5825

[estewart08]

There needs to be more of a discussion moving to dlopen. That being said I would prefer if you split the emissary fortran changes to a separate patch.

[lamb-j]

There needs to be more of a discussion moving to dlopen. That being said I would prefer if you split the emissary fortran changes to a separate patch.

Is there ever a context where we'd have static HSA libs and not be able to dlopen?

[jhuber6]

There needs to be more of a discussion moving to dlopen. That being said I would prefer if you split the emissary fortran changes to a separate patch.

Yes, I will likely bring this up on Monday again. Can split them once PSDB runs together.

Is there ever a context where we'd have static HSA libs and not be able to dlopen?

You can theoretically build it, but I don't think we use it in practice. I'd assume that would work if you put it in a place CMake could find and disabled the DLOPEN_PLUGINS

[z1-cciauto]

PSDB Link: https://compiler-ci.amd.com/job/compiler-psdb-amd-staging/5829