issue with Cassandra in pekko-sample-distributed-workers-scala

[pierreboudes]

On a stable Debian (OpenJDK 17), the example pekko-sample-distributed-workers-scala will fail on a exception while using sbt run or sbt "runMain worker.Main cassandra".

The logs

Starting Cassandra on port client port: 9042 storage port 45209 host 127.0.0.1 java version 17.0.10+7-Debian-1deb12u1
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.apache.cassandra.security.ThreadAwareSecurityManager (file …/pekko-samples/pekko-sample-distributed-workers-scala/target/cassandra-db/cassandra-bundle.jar)
WARNING: Please consider reporting this to the maintainers of org.apache.cassandra.security.ThreadAwareSecurityManager
WARNING: System::setSecurityManager will be removed in a future release
Exception (java.lang.AssertionError) encountered during startup: java.lang.reflect.InaccessibleObjectException: Unable to make field private int java.io.FileDescriptor.fd accessible: module java.base does not "opens java.io" to unnamed module @11b03c1f
java.lang.AssertionError: java.lang.reflect.InaccessibleObjectException: Unable to make field private int java.io.FileDescriptor.fd accessible: module java.base does not "opens java.io" to unnamed module @11b03c1f
…
[error] 	at worker.Main$.startCassandraDatabase(Main.scala:101)

[pjfanning]

you could add something like this to the build.sbt

run / javaOptions ++= Seq("--add-opens", "java.base/java.lang=ALL-UNNAMED")

unfortunately, this will cause issues if you use Java 8 or 11

when we get around to doing a PR to fix this, we'll probably need to add a condition that checks the java runtime version

[pierreboudes]

Thank you for your quick answer ! I also had to add run / fork := true as warned by sbt ([warn] Compile / run / javaOptions will be ignored, Compile / run / fork is set to false). Unfortunately I still have the error. I also tried adding the same option for java.util (random pick on Internet).

[peterDijk]

Hi, I have the same issue. I would really like to run this sample.
Tried to run it using JDK11, that gives a different error.
I have tried to debug this issue but I'm not able to find a solution.

Would it be possible for a maintainer to look into this issue so the sample could be run on JDK17, 21 ?

[pjfanning]

A fuller version of the exception appears to be:

10/24 10:40:59 ERROR[main] o.a.c.s.CassandraDaemon - Exception encountered during startup
java.lang.AssertionError: java.lang.reflect.InaccessibleObjectException: Unable to make field private int java.io.FileDescriptor.fd accessible: module java.base does not "opens java.io" to unnamed module @46d59067
	at org.apache.cassandra.utils.FBUtilities.getProtectedField(FBUtilities.java:608) ~[cassandra-bundle.jar:0.0.0-1068-9a5c7580-SNAPSHOT]
	at org.apache.cassandra.utils.NativeLibrary.<clinit>(NativeLibrary.java:80) ~[cassandra-bundle.jar:0.0.0-1068-9a5c7580-SNAPSHOT]
	at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:196) ~[cassandra-bundle.jar:0.0.0-1068-9a5c7580-SNAPSHOT]
	at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:602) ~[cassandra-bundle.jar:0.0.0-1068-9a5c7580-SNAPSHOT]
	at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:691) ~[cassandra-bundle.jar:0.0.0-1068-9a5c7580-SNAPSHOT]
Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make field private int java.io.FileDescriptor.fd accessible: module java.base does not "opens java.io" to unnamed module @46d59067
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:354) ~[na:na]
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297) ~[na:na]
	at java.base/java.lang.reflect.Field.checkCanSetAccessible(Field.java:178) ~[na:na]
	at java.base/java.lang.reflect.Field.setAccessible(Field.java:172) ~[na:na]
	at org.apache.cassandra.utils.FBUtilities.getProtectedField(FBUtilities.java:603) ~[cassandra-bundle.jar:0.0.0-1068-9a5c7580-SNAPSHOT]
	... 4 common frames omitted

This issue is happening in Cassandra code and maybe it would be possible to ask Cassandra experts. I don't use Cassandra much.
The sample runs with Java 8 and 11 so you try the sample with an older Java version.

[peterDijk]

Thank you for your fast reply. I understand if the project works in Java 8 or 11 I can just try it with that version.

So it's runs on JDK11 for you ?
I've switched back to Java 11 now, when running the first cassandra step I get this error.

[info] welcome to sbt 1.11.7 (Homebrew Java 11.0.29)
[info] loading global plugins from /Users/peter.van.dijk/.sbt/1.0/plugins
[info] loading project definition from /Users/peter.van.dijk/sideprojects/pekko-samples/pekko-sample-distributed-workers-scala/project
[info] loading settings for project pekko-sample-distributed-workers-scala from build.sbt...
[info] set current project to pekko-distributed-workers-scala (in build file:/Users/peter.van.dijk/sideprojects/pekko-samples/pekko-sample-distributed-workers-scala/)
[info] compiling 10 Scala sources to /Users/peter.van.dijk/sideprojects/pekko-samples/pekko-sample-distributed-workers-scala/target/scala-3.3.7/classes ...
[warn] -- [E029] Pattern Match Exhaustivity Warning: /Users/peter.van.dijk/sideprojects/pekko-samples/pekko-sample-distributed-workers-scala/src/main/scala/worker/Main.scala:21:9
[warn] 21 |    args.headOption match {
[warn]    |    ^^^^^^^^^^^^^^^
[warn]    |    match may not be exhaustive.
[warn]    |
[warn]    |    It would fail on pattern case: Some(_)
[warn]    |
[warn]    | longer explanation available when compiling with `-explain`
[warn] one warning found
[info] running (fork) worker.Main cassandra
[info] Starting Cassandra on port client port: 9042 storage port 54341 host 127.0.0.1 java version 11.0.29+0
[error] WARNING: An illegal reflective access operation has occurred
[error] WARNING: Illegal reflective access by org.apache.cassandra.utils.FBUtilities (file:/Users/peter.van.dijk/sideprojects/pekko-samples/pekko-sample-distributed-workers-scala/target/cassandra-db/cassandra-bundle.jar) to field java.io.FileDescriptor.fd
[error] WARNING: Please consider reporting this to the maintainers of org.apache.cassandra.utils.FBUtilities
[error] WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
[error] WARNING: All illegal access operations will be denied in a future release
[info] 10/24 11:49:42 ERROR[main] o.a.c.u.NativeLibraryDarwin - Failed to link the C library against JNA. Native methods will be unavailable.
[info] java.lang.UnsatisfiedLinkError: Can't load library: /var/folders/vn/tl9yvd2n42v_348c7f67fm7h0000gq/T/jna-1038551571/jna3654608662811854637.tmp
[info] 	at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2638) ~[na:na]
[info] 	at java.base/java.lang.Runtime.load0(Runtime.java:768) ~[na:na]
[info] 	at java.base/java.lang.System.load(System.java:1854) ~[na:na]
[info] 	at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:851) ~[cassandra-bundle.jar:1.1.0]
[info] 	at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:826) ~[cassandra-bundle.jar:1.1.0]
[info] 	at com.sun.jna.Native.<clinit>(Native.java:140) ~[cassandra-bundle.jar:1.1.0]
[info] 	at org.apache.cassandra.utils.NativeLibraryDarwin.<clinit>(NativeLibraryDarwin.java:53) ~[cassandra-bundle.jar:1.1.0]
[info] 	at org.apache.cassandra.utils.NativeLibrary.<clinit>(NativeLibrary.java:88) ~[cassandra-bundle.jar:1.1.0]
[info] 	at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:196) ~[cassandra-bundle.jar:1.1.0]
[info] 	at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:602) ~[cassandra-bundle.jar:1.1.0]
[info] 	at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:691) ~[cassandra-bundle.jar:1.1.0]
[info] 10/24 11:49:42 ERROR[main] o.a.c.s.StartupChecks - cassandra.jmx.local.port missing from cassandra-env.sh, unable to start local JMX service.
[info] 10/24 11:49:42 ERROR[main] o.a.c.s.CassandraDaemon - The native library could not be initialized properly.```

[pjfanning]

I don't see that when running in Java 11. But it is a Cassandra failure not a Pekko one and I am not a Cassandra expert.

[pjfanning]

Maybe Cassandra doesn't run on all Macs.

[peterDijk]

I can run the project using Cassandra running in docker 👍 It also runs well in Java 21.
Comment out/ remove startCassandraDatabase in Main.scala, have cassandra running in docker before running the Scala project

docker run \
    --name cass_cluster \
    --rm \
    -d \
    -p 9042:9042 \
    cassandra:latest

[mdedetrich]

Thank you for your fast reply. I understand if the project works in Java 8 or 11 I can just try it with that version.

So it's runs on JDK11 for you ? I've switched back to Java 11 now, when running the first cassandra step I get this error.

[info] welcome to sbt 1.11.7 (Homebrew Java 11.0.29)
[info] loading global plugins from /Users/peter.van.dijk/.sbt/1.0/plugins
[info] loading project definition from /Users/peter.van.dijk/sideprojects/pekko-samples/pekko-sample-distributed-workers-scala/project
[info] loading settings for project pekko-sample-distributed-workers-scala from build.sbt...
[info] set current project to pekko-distributed-workers-scala (in build file:/Users/peter.van.dijk/sideprojects/pekko-samples/pekko-sample-distributed-workers-scala/)
[info] compiling 10 Scala sources to /Users/peter.van.dijk/sideprojects/pekko-samples/pekko-sample-distributed-workers-scala/target/scala-3.3.7/classes ...
[warn] -- [E029] Pattern Match Exhaustivity Warning: /Users/peter.van.dijk/sideprojects/pekko-samples/pekko-sample-distributed-workers-scala/src/main/scala/worker/Main.scala:21:9
[warn] 21 |    args.headOption match {
[warn]    |    ^^^^^^^^^^^^^^^
[warn]    |    match may not be exhaustive.
[warn]    |
[warn]    |    It would fail on pattern case: Some(_)
[warn]    |
[warn]    | longer explanation available when compiling with `-explain`
[warn] one warning found
[info] running (fork) worker.Main cassandra
[info] Starting Cassandra on port client port: 9042 storage port 54341 host 127.0.0.1 java version 11.0.29+0
[error] WARNING: An illegal reflective access operation has occurred
[error] WARNING: Illegal reflective access by org.apache.cassandra.utils.FBUtilities (file:/Users/peter.van.dijk/sideprojects/pekko-samples/pekko-sample-distributed-workers-scala/target/cassandra-db/cassandra-bundle.jar) to field java.io.FileDescriptor.fd
[error] WARNING: Please consider reporting this to the maintainers of org.apache.cassandra.utils.FBUtilities
[error] WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
[error] WARNING: All illegal access operations will be denied in a future release
[info] 10/24 11:49:42 ERROR[main] o.a.c.u.NativeLibraryDarwin - Failed to link the C library against JNA. Native methods will be unavailable.
[info] java.lang.UnsatisfiedLinkError: Can't load library: /var/folders/vn/tl9yvd2n42v_348c7f67fm7h0000gq/T/jna-1038551571/jna3654608662811854637.tmp
[info] 	at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2638) ~[na:na]
[info] 	at java.base/java.lang.Runtime.load0(Runtime.java:768) ~[na:na]
[info] 	at java.base/java.lang.System.load(System.java:1854) ~[na:na]
[info] 	at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:851) ~[cassandra-bundle.jar:1.1.0]
[info] 	at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:826) ~[cassandra-bundle.jar:1.1.0]
[info] 	at com.sun.jna.Native.<clinit>(Native.java:140) ~[cassandra-bundle.jar:1.1.0]
[info] 	at org.apache.cassandra.utils.NativeLibraryDarwin.<clinit>(NativeLibraryDarwin.java:53) ~[cassandra-bundle.jar:1.1.0]
[info] 	at org.apache.cassandra.utils.NativeLibrary.<clinit>(NativeLibrary.java:88) ~[cassandra-bundle.jar:1.1.0]
[info] 	at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:196) ~[cassandra-bundle.jar:1.1.0]
[info] 	at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:602) ~[cassandra-bundle.jar:1.1.0]
[info] 	at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:691) ~[cassandra-bundle.jar:1.1.0]
[info] 10/24 11:49:42 ERROR[main] o.a.c.s.StartupChecks - cassandra.jmx.local.port missing from cassandra-env.sh, unable to start local JMX service.
[info] 10/24 11:49:42 ERROR[main] o.a.c.s.CassandraDaemon - The native library could not be initialized properly.```

Considering we are getting a JNA error against a C library, this very well seems to be an issue regarding running Cassandra on architectures it wasn't compiled on (and Apple M series silicon can be one of them)

I am with @pjfanning here though, this is entirely a Cassandra issue (Pekko just wraps the underlying core library). The best we can do is add some error detection depending on the architecture and version of Cassandra, but even that is a stretch (and honestly this functionality should be in Cassandra core)

[pjfanning]

This a related issue in akka-persistence-cassandra.
akka/akka-persistence-cassandra#986

The pekko-persistence-cassandra-launcher should be removed as far as I am concerned and the sample rewritten to use testcontainers project to start Cassandra using Docker - or we just assume that the user has Cassandra running for themselves and the sample just tries to connect to the running instance.

I think we should remove pekko-persistence-cassandra-launcher from the pekko-persistence-cassandra repo too. It is currently built using Cassandra 3.11.3, a very old release.

[mdedetrich]

This a related issue in akka-persistence-cassandra. akka/akka-persistence-cassandra#986

The pekko-persistence-cassandra-launcher should be removed as far as I am concerned and the sample rewritten to use testcontainers project to start Cassandra using Docker - or we just assume that the user has Cassandra running for themselves and the sample just tries to connect to the running instance.

I think we should remove pekko-persistence-cassandra-launcher from the pekko-persistence-cassandra repo too. It is currently built using Cassandra 3.11.3, a very old release.

Agreed on all counts, especially on using testcontainers to startup a cassandra instance rather than having to run wire up things with docker manually.