Embed VLC video output into a Scene Graph

Question

Hello,
I'm trying to embed a VLC player in a JavaFX 2.2 application.
For that I'm using the libvlc.dll library's API.
VLC provided two ways to render a video:
1. Call API function that takes HWND: +libvlc_media_player_set_hwnd (libvlc_media_player_t p_mi, void drawable)+ Set a Win32/Win64 API window handle (HWND) where the media player should render its video output.
2. Register a callback that gets called every time a frame should be rendered: +libvlc_video_display_cb (void opaque, void picture) Callback prototype to display a picture.+
With the HWND approach, I managed to extract an HWND pointer of the Scene using Glass framework, causing the video to render on the whole scene, no other nodes are visible.
With the Callback approach, and by using JavaFX 2.2 new javafx.scene.canvas.Canvas node, I was able to throw pixels at the canvas, but it turns to be very inefficient in terms of memory and thread usage.
I would love to have a solution as with AWT + JNA: the HWND can be extracted from java.awt.Canvas component (using JNA), and be given to libvlc API function. This canvas can be placed anywhere in the frame, not occupying all the entire window.
Hard Guidelines:
1. Integrating AWT/Swing with FX application is not an option, I want it to be a pure JavaFX application.
2. Using JavaFX Media API for playback is not an option, since I have a requirement to modify the media player's source code.
3. The video must not occupy the entire window, allowing other components to be visible.
Possible solution is to create my own node or control that will wrap Glass component View that has a HWND but I don't know how to bind and make it work with other Glass components.
Any help would be appreciated.
Thank you.
Eli.

Answer 1

I would be interested in a solution for this as well.
I've already done a lot of experiments in this area trying many frameworks/players (Xuggler, GStreamer, VLC (VLCJ), MediaPlayer Home Cinema, DS4J, MPlayer) and it seems to be very hard to do what you ask. Best chances at the moment are to use GStreamer, which I've succesfully been able to overlay with JavaFX components in the same window (with a stiff CPU penalty though).
The other frameworks all have their drawbacks (Gstreamer as well). Most are low level which makes things like switching streams or doing subtitles a lot harder, some have poor control interfaces that donot expose enough functionality, etc). GStreamers main drawback is efficiency when using the Canvas (about 50% slower than native) but solvable by throwing more hardware at it. Also until recently its builds were not very up-to-date. I intend to soon test it again and see if I can get a fully working Canvas style solution with it.
Currently I use VLC (with VLCJ) and I play back videos in a seperate Frame, and have a second window (a Stage) that is transparent on top of it. This way I can mix JavaFX controls and a video. It is hackish and also annoying that you have to juggle two windows but it works fairly ok. This approach might also work for you if you resize the Canvas in the seperate frame to where it needs to be displayed and keep that area transparent in your JFX frame... but it is hardly optimal (but very fast since it can do direct rendering).
I haven't tried recently to mix it directly with JavaFX, but from what I read it seems possible now... but I'm pretty sure it will incur the same kind of CPU penalty as with GStreamer's Canvas :(