Hello
I am testing the new Media Extension API in macOS 15 Beta 4.
Firstly, THANK YOU FOR THIS API!!!!!! This is going to be huge for the video ecosystem on the platform. Seriously!
My understanding is that to support custom container formats you make a MEFormatReader extension, and to support a specific custom codec, you create a MEVideoDecoder for that codec.
Ok - I have followed the docs - esp the inline header info and have gotten quite far
A Host App which hosts my Media Extenion (MKV files)
A Extension Bundle which exposes the UTTYpes it supports to the system and plugin class ID as per the docs
Entitlements as per docs
I'm building debug - but I have a valid Developer ID / Account associated in Teams in Xcode
My Plugin is visible to the Media Extension System preference
My Plugin is properly initialized, I get the MEByteReader and can read container level metadata in callbacks
I can instantiate my tracks readers, and validate the tracks level information and provide the callbacks
I can instantiate my sample cursors, and respond to seek requests for samples for the track in question
Now, here is where I get hit some issues.
My format reader is leveraging FFMPEGs libavformat library, and I am testing with MKV files which host AVC1 h264 samples, which should be decodable as I understand it out of the box from VideoToolbox (ie, I do not need a separate MEVideoDecoder plugin to handle this format).
Here is my CMFormatDescription which I vend from my MKV parser to AVFoundation via the track reader
Made Format Description: <CMVideoFormatDescription 0x11f005680 [0x1f7d62220]> {
mediaType:'vide'
mediaSubType:'avc1'
mediaSpecific: {
codecType: 'avc1' dimensions: 1920 x 1080
}
extensions: {(null)}
}
My MESampleCursor implementation implements all of the callbacks - and some of the 'optional' sample cursor location methods: (im only sharing the optional ones here)
- (MESampleLocation * _Nullable) sampleLocationReturningError:(NSError *__autoreleasing _Nullable * _Nullable) error
- (MESampleCursorChunk * _Nullable) chunkDetailsReturningError:(NSError *__autoreleasing _Nullable * _Nullable) error
I also populate the AVSampleCursorSyncInfo and AVSampleCursorDependencyInfo structs per each AVPacket* I decode from libavformat
Now my issue:
I get these log files in my host app:
<<<< VRP >>>> figVideoRenderPipelineSetProperty signalled err=-12852 (kFigRenderPipelineError_InvalidParameter) (sample attachment collector not enabled) at FigStandardVideoRenderPipeline.c:2231
<<<< VideoMentor >>>> videoMentorDependencyStateCopyCursorForDecodeWalk signalled err=-12836 (kVideoMentorUnexpectedSituationErr) (Node not found for target cursor -- it should have been created during videoMentorDependencyStateAddSamplesToGraph) at VideoMentor.c:4982
<<<< VideoMentor >>>> videoMentorThreadCreateSampleBuffer signalled err=-12841 (err) (FigSampleGeneratorCreateSampleBufferAtCursor failed) at VideoMentor.c:3960
<<<< VideoMentor >>>> videoMentorThreadCreateSampleBuffer signalled err=-12841 (err) (FigSampleGeneratorCreateSampleBufferAtCursor failed) at VideoMentor.c:3960
Which I presume is telling me I am not providing the GOP or dependency metadata correctly to the plugin.
I've included console logs from my extension and host app:
LibAVExtension system logs
And my SampleCursor implementation is here
https://github.com/vade/FFMPEGMediaExtension/blob/main/LibAVExtension/LibAVSampleCursor.m
Any guidance is very helpful.
Thank you!