JNI + Timer, weird crash

Question

Hello people.
First of all, I'd like to apologize beforehand for the long post. Also, I hope I'm posting on the right forum section.
Here's the story:
I'm working on an academic project, which involves a Java application calling C++ code, via JNI. I'm currently facing an issue, the exact cause of which I cannot track down, at least given my current grasp of Java, JNI, JVM etc.
Before I get straight to the point, here's some background on the project, which I think may help in clarifying any possible subtleties of the problem. You can probably skip this paragraph safely though, and jump to the picture that is provided on the link below it, without missing too much. Basically, I'm working on implementing an AI (specifically Reinforcement Learning) experiment domain which is based on an open-source game, written in C++. A Java application (which can be thought of, as a platform for deploying AI experiments) is responsible for the control of the experiment execution, which is conducted on a step-by-step basis. The game is compiled as a dynamically loadable shared library, so that it can be loaded by a specific JNI component of the Java AI platform. Let me state at this point, that the Java application, as well as the JNI dynamic library loader that it features, were not implemented by myself. As a consequence, I'm not aware of the details concerning the JNI part of the implementation (I've peeked at the pure Java part though, and it seems pretty OK to me). My work lies on modifying the game code (C++), so that it can be integrated with the Java AI platform. The application features 2 modes of executing experiment steps. In the "manual" mode, a step is executed when a specific JButton is hit (I will henceforth refer to this execution mode, as JButton mode). On the other hand, there is a "batch" mode, where step execution is scheduled to happen on a fixed time interval, using the scheduleAtFixedRate method of the java.util.Timer class (Timer mode, henceforth). To make matters even more clear, here's a a sketch of the architecture of the system:
[Sketch of the problem|http://users.auth.gr/~idaroglo/pub/fsitu.png]
The problem is, that while the JButton mode works OK, the Timer mode doesn't. In that case, things crash on the native side (specifically on the OpenGL library, which is used to render the graphics of the game). This is quite weird I think, as the only difference between these 2 execution modes, is the way they call the native step function (I tried to illustrate in the picture linked above). So, a wild guess could be that something gets messed up, when the Timer gets involved in orchestrating the execution. I cannot put blame on the native shared library, as it functions properly in the case of the JButton mode. Having no other hint or indication of what causes this situation, I can only blame it on the Timer involvement (hence the thread title). I will post a sample JVM Hotspot crash log, as a reply to this one.
Here are some things I've tried so far, to resolve this issue:
1) Debugging with gdb. I tracked down which function of the OpenGL library causes the crash (btw, segmentation fault - SISEGV). I've been told by a fellow though, that this specific function cannot produce a segmentation fault and that the crash probably happens due to memory corruption. In case this speculation holds, a wild guess could be that the use of a timer, could be introducing memory corruption.
2) I've skimmed through the JVM Troubleshooting guide, in the hope that I'll find some advice on how I should troubleshoot this situation. I'm bound to revisit this, hopefully after some feedback from the community.
3) I've considered using valgrind to verify that memory corruption, but I'm not sure if that would make sense in the case of Java/JNI.
4) I've tried setting the Timer's fixed time interval to a 20 seconds (20000ms), to make absolutely sure that the native method returns, before it is invoked by the timer for a consecutive time. Still things crash.
5) I've tested the Java AI platform with a considerably simpler dynamic library (in terms of implementation), and both modes functioned OK. This probably indicates that I cannot blame it all, on the Timer alone.
So, I would really appreciate some feedback on the following matters:
1) Does any of the speculations above, concerning the cause of the problem, make sense to you?
2) How should I proceed troubleshooting? Are there any recommended tools that might prove helpful in tracking down the cause of the problem?
3) Should one employ extra care when using Timers in combination with JNI & native methods?
4) How can one track down and troubleshoot memory corruption issues in the mixed mode scenario (JAVA+JNI)?
That's all. Thanks for the time you spent reading this. Once more, I sincerely apologize for the long post.
I really hope that collective experience and wisdom, will help me squash this issue.
Thanks again.

Answer 1

Hello again!
Sorry to revive this thread after a long time, but this post might be helpful to anyone that might encounter a similar problem in the future.
I am obliged to Paul Pluzhnikov, for identifying the cause of the problem. Wouldn't it be for his invaluable help, I would still have been stuck with this one.
I quote part of his response, a while ago:
Paul Pluzhnikov wrote:
....JVM log has register dump as well, and
RAX=0x0000000000000000 in it. So your program tried to jump through a
NULL pointer, and crashed with SIGSEGV since zero page of memory is
not mapped into the process space on Linux.
The more interesting question is "why is RAX NULL?"
RAX was loaded in the previous instruction from the FS (thread-local
storage on x86_64) segment register.
Given all of the above, I will guess that the difference between the
"JButton" and "Timer" execution is that in the "JButton" case you
execute the call to JNI/game_step in the event dispatch thread, while
in the "Timer" case you are not.
The following (buggy) program demonstrates what I believe is
essentially the same crash:
// compile with "gcc -g main.c /usr/lib/libGL.so.1 -pthread"
#include <pthread.h>
#include <stdio.h>
void *fn(void *p)
printf("calling glHint on other thread\n");
return glHint();
int main(int argc, char *argv[])
pthread_t tid;
if (argc > 1) {
printf("calling glHint on main thread\n");
return glHint();
pthread_create(&tid, 0, fn, 0);
pthread_join(tid, 0);
return 0;
}Here is what I see on my system:
gdb -q ./a.out
(gdb) run
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/libthread_db.so.1".
[New Thread 0x7ffff7ed4720 (LWP 19872)]
[New Thread 0x40802950 (LWP 19875)]
calling glHint on other thread
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x40802950 (LWP 19875)]
0x00007ffff7db62c9 in glHint () from /usr/lib/libGL.so.1
(gdb) x/i $pc
0x7ffff7db62c9 <glHint+9>:     jmpq *0x378(%rax)
(gdb) p $rax
$1 = 0 Notice the same instruction crash, and the same RAX == NULL (gdb) run 1
[Thread debugging using libthread_db enabled]
[New Thread 0x7ffff7ed4720 (LWP 19877)]
calling glHint on main thread
Program exited normally.
(gdb) quitSo how do you fix this? The following may be helpful:
[http://java.sun.com/docs/books/tutorial/uiswing/concurrency/index.html] A few days later, I got some further input from Paul. I quote part of it as well, as I think it might also prove to be helpful for someone:
Paul Pluzhnikov wrote:
You probably don't actually have to jump through hoops to get the
"game" executing on the event dispatch thread.
What I think must happen is that the dlopen() of the "game" must
be done in the same thread that the "step" will be done in.
Alternatively, you may be able to force libGL to initialize its
TLS storage for arbitrary thread, by using glXMakeCurrent.
Here is some additional info which may be helpful:
[http://www.equalizergraphics.com/documentation/parallelOpenGLFAQ.html]
[http://www.opengl.org/documentation/specs/man_pages/hardcopy/GL/html/glx/xmakecurrent.html]
I'd like to thank all of you, who provided any sort of input on this. Most of all, I'd like to thank Paul Pluzhnikov once more, for figuring the problem out.
Hope this post will be helpful to someone in the future!
Cheers!