Media Foundation transforms for BackgroundMediaPlayer

Question

So I want to apply some media foundation effect to the backgoundmediaplayer samples before they get rendered. While this seems an obvious mission with the media stream source class, I do not really have the time to implement media stream sources for all
formats supported by windows phone just to get to the raw PCM inside them, and then do the things I want to do with them.
The media foundation transforms does something similar to what I want to do. Is there any way I can use a transform for BackgroundMediaPlayer, intercept PCM samples delivered by built-in system codecs and modifiy them accordingly? Or do I need to relay on
media stream sources?

Answer 1

Hello,
I'm sorry if I caused any confusion. Let me try to offer some clarification. The MediaStreamSource was designed specifically to address
the need to ingest and parse 3rd party streaming protocols such as HLS. In this scenario you connect to, parse and pass the encoded video samples downstream and allow the hardware to decode the samples.
While it is certainly possible to implement a stream source and codec in the same MediaStreamSource context you may not
get the performance needed to present a good user experience. Again we just didn't intend the MediaStreamSource to be used in this way.
While using C++ to implement the decoder may improve performance there is still a need to marshal the data between managed and unmanaged
code. Once in managed code the GC may run and cause sample delivery to stop unexpectedly. As I'm sure you are aware actively managing the buffer size can help to reduce this effect at the expense of latency. This is an extremely advanced topic.
That said, your business requirements may allow for dropouts to occur. This is up to you to decide. I just want to make sure that you are
aware of the repercussions of choosing this architecture and that it is not a scenario that we intended or tested.
So... Because of this I would never recommend this approach. I guess if it works great but be aware of the risk.
Okay, let’s talk about the BackgroundMediaPlayer architecture a bit. Windows Phone 8 (WP8) was a three process architecture. Windows Phone 8.1 (WP8.1) is a two process architecture.
I won’t go into how WP8 works. In WP8.1 you have a UI process and a background player process. The background player process is registered as a singleton instance OOPServer.
This means that the background player will only be created if one does not already exist in the current app container.
The background player process contains your background audio code. Just like Windows 8 this process is given a PLM exemption that allows it to continue to run in the background
while your UI process can be suspended. Once the exemption is registered, management of the background player process is handed off to the media manger. The media manager coordinates closely with the PLM to ensure there is only one active background player
process at a given time. This means only one app can be playing audio in the background at any time.
Okay so if you have gotten this far you get some potentially good news and an apology. I humbly admit that I was incorrect. I spent most of the afternoon going over
specs and the phone source code. I found that it should theoretically be possible to add additional components to the underlying background player MF topology.
From what I have been able to decipher, you can register a custom MFT in the background player code and have it automatically join the topology when it is created. You can
do this via the
MediaExtensionManger. Sounds good right?  Keep in mind this is theoretical. I haven’t tried this and there may be some caveats, explained below.
Here are what I can see as caveats: The MediaExtensionManger only allows you to register certain types of MFTs. In particular encoders, decoders and stream handlers. Also you
can’t override any of the MF components that are handled by a known format. In other words you can’t override our MP4 decoder. So it’s not as easy as just saying “stick my MFT here”.
After much thought it still might be possible to get a MFT to join the topology after the decoder. In the case of audio I’m thinking that we need to create an MFT that takes
uncompressed audio in and outputs uncompressed audio. The question here is: “How does the topology manager evaluate components available to participate in the topology?”
The topology we want looks like this: S->D->C->R where C is our custom MFT. The decoder (D) outputs PCM. The renderer (R) takes PCM as the input. Our C takes PCM as
an input and output. When the topology loader builds the topology it connects the nodes from left to right. The question is: “Will our C be evaluated before the R?” If our C is evaluated before the R then we are money. However if the R is evaluated first or
C will be left out of the topology. I honestly don’t know enough about the intricacies of the topology loader to say with any certainty if this will work (and what about format conversions?)
Again we are trying to do things that the original developers never intended and didn’t test. If it works great! Please let me know. If it doesn’t I will certainly request
this feature for the next version of the Phone.
I hope this helps,
James
Windows SDK Technologies - Microsoft Developer Services - http://blogs.msdn.com/mediasdkstuff/