Application and jdi/jvmti agent in single vm?

Question

We're building a suite of monitoring tools that will help us inspect and manage our enterprise-class app. We currently have working an instrumenting interface (via mbeans) that uses javassist to instrument bytecode; this tool runs as an agent within the app's vm using the java.lang.instrument package -- no native code involved. The next aspect of the tool suite is a more sophisticated "inspector" that will offer debugging-like facilities: method tracing, local variable inspection, thread inspection, etc.
We prototyped a version using jdi and that works fine if the agent does not run in the same vm as the app. However, setting things up so that the JDI agent does run in the app's vm does not work -- the entire vm (app, agent) hang after a call to VirtualMachine.suspend().
Intuitively, I think I knew this would happen but wanted to confirm it here. I have two questions:
1. Is it possible for a JDI-based agent to run in the same vm as a monitored app?
2. If not (1), and we write our agent in C, will the RunAgentThread() jvmti api allow us to run the inspector in the app's vm?
Thanks for your help.

Answer 1

JDI wasn't intended to be used to debug the same process in which the JDI client is running. We haven't tested that.
However, it is an interesting question and it seems that theoretically, if you are very careful, you might be able to do
a limited amount of debugging.
I gave it a try. My test program is launched with
java -agentlib:jdwp=transport=dt_socket,address=60000,server=y,suspend=n
The program starts a thread (named by default Thread-0) and then uses the SocketAttachingConnector
to attach to 60000. After it gets back a VirtualMachine object, it does
VirtualMachine.allThreads()
and prints the result:
instance of java.lang.Thread(name='Thread-1', id=66),
instance of java.lang.Thread(name='JDI Target VM Interface', id=4),
instance of java.lang.Thread(name='JDI Internal Event Handler', id=7),
instance of java.lang.Thread(name='Thread-0', id=69),
instance of java.lang.Thread(name='Signal Dispatcher', id=70),
instance of java.lang.ref.Finalizer$FinalizerThread(name='Finalizer', id=71),
instance of java.lang.ref.Reference$ReferenceHandler(name='Reference Handler', id=72),
instance of java.lang.Thread(name='main', id=1)
So, this method worked. It sounds like you have basically gotten this far since you said your entire
app hangs after a call to VirutalMachine.suspend(). This hang is to be expected since you are suspending
'yourself'. You have to be very careful to not do this. EG, don't set a SUSPEND_ALL breakpoint at a location
which your 'debugger' will execute. There might be other pitfalls too that make this idea impractical.
But, if you are careful, eg, restrict actions to threads in the part of your app that you want to 'monitor', you
might be able to do something useful.
Let us know how it goes.