GenServers block per message. IO frames between Elixir/JACK need work without interruption which requires having two GenServers. Ideally the interface wouldn't change, they would just communicate with two different processes.
This issue may require forking rust-jack. While it's not possible to pass the JACK client as a resource to ExJack, it may be possible to run operations on it in a adhoc manner.