Explore the integration of media technologies within your app. Discuss working with audio, video, camera, and other media functionalities.

All subtopics
Posts under Media Technologies topic

Post

Replies

Boosts

Views

Activity

Should the MV-HEVC Encoder Support Multiple Passes?
As a straightforward example, I've taken Apple's MV-HEVC sample project and added two lines. First, after the AVAssetWriterInput is created: frameInput.performsMultiPassEncodingIfSupported = true Second, after the call to multiviewWriter.startWriting(): print("canPerformMultiplePasses: \(frameInput.canPerformMultiplePasses)") Which prints true. This leads me to believe that the first encoding pass should proceed as-normal (even though I haven't handled the logic for the completion of the first pass, etc.). However, I receive this error when the code attempts to appendTaggedBuffers to the AVAssetWriterInputTaggedPixelBufferGroupAdaptor: Fatal error: Failed to append tagged buffers to multiview output Am I missing a step? Or is the multi-pass encoding only supported for standard sample/pixel buffers (and not tagged buffers)?
1
3
743
Mar ’24
Understanding AVAudioTime in AVAudioNodeTapBlock? Is there a way to get time relative to a scheduled Buffer?
I'm using AVAudioEngine to play AVAudioPCMBuffers. I'd like to synchronize some events with the playback. For example if the audio's frame position is >= some point && less than some point trigger some code. So I'm looking at - (void)installTapOnBus:(AVAudioNodeBus)bus bufferSize:(AVAudioFrameCount)bufferSize format:(AVAudioFormat * __nullable)format block:(AVAudioNodeTapBlock)tapBlock; Now I have frame positions calculated (predetermined before audio is scheduled I already made all necessary computations) . So I just need to fire code at certain points during playback: [playerNode installTapOnBus:bus bufferSize:bufferSize format:format block:^(AVAudioPCMBuffer * _Nonnull buffer, AVAudioTime * _Nonnull when) { //Inspect current audio here and fire... }]; [playerNode scheduleBuffer:fullbuffer atTime:startTime options:0 completionCallbackType:AVAudioPlayerNodeCompletionDataPlayedBack completionHandler:^(AVAudioPlayerNodeCompletionCallbackType callbackType) { // some code is here, not important to this question. }]; The problem I'm having is figuring out at what point in full buffer I'm at within the tap block. The tap block passes chunks (not the full audio buffer). I tried using the when parameter of the block to calculate the frame position relative to the entire audio but have be unsuccessful so far. I'm assuming the when parameter is relative to the buffer passed in the tap block (not my entire audio buffer I scheduled). Not installing a tap and just using a timer before scheduling my fullBuffer has given me good results but I'd rather avoid using a timer if possible and use sample time.
3
0
1.4k
Apr ’24
How to display artwork images from MusicKit with UIKit?
In SwiftUI there is a built-in component for displaying album artworks called Artwork but there is no equivalent for UIKit. My current approach is to use the .url() method to read image's URL and download the image or read it from the disk but the performance is much worse than it was previously with MPMediaItem's artworkImage method. let artworkQueue = DispatchQueue( label: "MusicKit-ArtworkQueue", qos: .default, attributes: .concurrent ) let artworkSemaphore = DispatchSemaphore(value: 5) extension Song { func artworkImage(for size: CGSize, completion: @escaping (UIImage?) -> Void) { artworkQueue.async { artworkSemaphore.wait() defer { artworkSemaphore.signal() } let imageURL = artwork?.url( width: Int(size.width), height: Int(size.height) ) // I hate doing this as it might very well break in the future guard let imageURL, imageURL.scheme == "musicKit" else { return completion(nil) } guard let imageData = try? Data(contentsOf: imageURL), let image = UIImage(data: imageData) else { return completion(nil) } completion(image) } } } I really dislike this approach because it feels hacky but somewhat works. You might ask what's the semaphore for? Well, without it I could notice that MusicKit was choking and after reading too many artworks at once. Can someone from Apple please provide us with an example on how to use MusicKit with UIKit properly? Ideally (IMO) we would have a method defined on Song and other MusicKit structures that returns the image for us, just like MPMediaItem had the .artwork() method. It would make our lives so much easier.
2
0
960
Apr ’24
AVAudioEngine Dolby Atmos
Hi! I have a music app using AVAudioEngine. Right now, I have set it up to play multi channel tracks and show "Multichannel" in the volume controls. However, I am unable to figure out how to get it to use Dolby Atmos. Is there something that needs to be enabled? Is it even possible for AVAudioEngine? I saw some apps that are able of playing with Dolby Atmos, but they do not have EQ feature, so I'm guessing that they are not using AVAudioEngine.
2
0
929
Apr ’24
"Remote call timed out" error when trying to play large collection of music items with MusicKit's ApplicationMusicPlayer
I am using MusicKit ApplicationMusicPlayer to play music in my app. Everything works fine as long as I'm not playing large playlists that contain hundreds of songs. When I to play collection of songs that is larger than around 300 I'm always getting the error message saying: "Prepare to play failed" UserInfo={NSDebugDescription=Prepare to play failed, NSUnderlyingError=0x121d42dc0 {Error Domain=MPMusicPlayerControllerErrorDomain Code=9 "Remote call timed out" UserInfo={NSDebugDescription=Remote call timed out}}})) It doesn't matter if songs are downloaded to the device or not. I am aware that there is another initializer for player's queue that accepts Playlist instances but in my app users can choose to sort playlist tracks in different order than the default and that makes using that initializer not feasible for me. I tried everything I could think of, I tried to fall back on MPMusicPlayerController and pass array of MPMusicPlayerPlayParameters to it but the result was the same. typealias QueueEntry = ApplicationMusicPlayer.Queue.Entry let player = ApplicationMusicPlayer.shared let entries: [QueueEntry] = tracks .compactMap { guard let song = $0 as? Song else { return nil } return QueueEntry(song) } Task(priority: .high) { [player] in do { player.queue = .init(entries, startingAt: nil) try await player.play() // prepareToPlay failed } catch { print(error) } }
1
0
622
Apr ’24
HDR10 MVHECV can not play on Safari
Hi, just generated a HDR10 MVHEVC file, mediainfo is below: Color range : Limited Color primaries : BT.2020 Transfer characteristics : PQ Matrix coefficients : BT.2020 non-constant Codec configuration box : hvcC+lhvC then generate the segment files with below command: mediafilesegmenter --iso-fragmented -t 4 -f av_1 av_new_1.mov then upload the segment files and prog_index.m3u8 to web server. just find that can not play the HLS stream on Safari... the url is http://ip/vod/prog_index.m3u8 just checked that if i remove the tag Transfer characteristics : PQ when generating the MVHEVC file. above same mediafilesegmenter command and upload the files to web server. the new version of HLS stream is can play on Safari... Is there any way to play HLS PQ video on Safari. thanks.
2
1
738
May ’24
iPad app on macOS not asking for microphone permission
Hello, I have an iOS app that is recording audio that is working fine on iPads/iPhones. It asks for microphone permission and after that recording works. I installed the same app on my M3 MacBook via TestFlight since iPad apps are supposed to work without a change that way. The app starts fine and everything, but it never asks for Microphone permission, so I can't record. Do I need to do something to make this happen (this is not macCatalyst, its running the arm64 iPhone binary on macOS) thanks
2
1
721
May ’24
FCPXML Creation issue...
I have generated FCPXML, but i can't figure out issue: <?xml version="1.0"?> <fcpxml version="1.11"> <resources> <format id="r1" name="FFVideoFormat3840x2160p2997" frameDuration="1001/30000s" width="3840" height="2160" colorSpace="1-1-1 (Rec. 709)"/> <asset id="video0" name="11a(1-5).mp4" start="0s" hasVideo="1" videoSources="1" duration="6.81s"> <media-rep kind="original-media" src="file:///Volumes/Dropbox/RealMedia Dropbox/Real Media/Media/Test/Test AE videos, City, testOLOLO/video/11a(1-5).mp4"/> </asset> <asset id="video1" name="12(4)r8 mute.mp4" start="0s" hasVideo="1" videoSources="1" duration="9.94s"> <media-rep kind="original-media" src="file:///Volumes/Dropbox/RealMedia Dropbox/Real Media/Media/Test/Test AE videos, City, testOLOLO/video/12(4)r8 mute.mp4"/> </asset> <asset id="video2" name="13 mute.mp4" start="0s" hasVideo="1" videoSources="1" duration="6.51s"> <media-rep kind="original-media" src="file:///Volumes/Dropbox/RealMedia Dropbox/Real Media/Media/Test/Test AE videos, City, testOLOLO/video/13 mute.mp4"/> </asset> <asset id="video3" name="13x (8,14,24,29,38).mp4" start="0s" hasVideo="1" videoSources="1" duration="45.55s"> <media-rep kind="original-media" src="file:///Volumes/Dropbox/RealMedia Dropbox/Real Media/Media/Test/Test AE videos, City, testOLOLO/video/13x (8,14,24,29,38).mp4"/> </asset> </resources> <library> <event name="Untitled"> <project name="Untitled Project" uid="28B2D4F3-05C4-44E7-8D0B-70A326135EDD" modDate="2024-04-17 15:44:26 -0400"> <sequence format="r1" duration="4802798/30000s" tcStart="0s" tcFormat="NDF" audioLayout="stereo" audioRate="48k"> <spine> <asset-clip ref="video0" offset="0/10000s" name="11a(1-5).mp4" duration="0/10000s" format="r1" tcFormat="NDF"/> <asset-clip ref="video1" offset="12119/10000s" name="12(4)r8 mute.mp4" duration="0/10000s" format="r1" tcFormat="NDF"/> <asset-clip ref="video2" offset="22784/10000s" name="13 mute.mp4" duration="0/10000s" format="r1" tcFormat="NDF"/> <asset-clip ref="video3" offset="34544/10000s" name="13x (8,14,24,29,38).mp4" duration="0/10000s" format="r1" tcFormat="NDF"/> </spine> </sequence> </project> </event> </library> </fcpxml> Any ideas?
2
0
798
May ’24
How to detect the end of playback with the system music player?
Since iOS 12 it has become difficult to detect the end of playback using the system music player. In earlier iOS versions, the now playing item would be set nil and you would receive a notification that the player stopped. In iOS 12 and later, nowPlayingItem still contains the current song and the only notification you get is MPMusicPlayerControllerPlaybackStateDidChangeNotification with the playbackState set to MPMusicPlaybackStatePaused. Pressing pause in my car (or any remote access) generates the same conditions making it difficult to correctly detect the difference. It would be nice if they added a notification that playback was done (similar to the other players). Any suggestions?
1
1
710
May ’24
AudioComponentInstanceNew takes up to five seconds to complete
We are using a VoiceProcessingIO audio unit in our VoIP application on Mac. In certain scenarios, the AudioComponentInstanceNew call blocks for up to five seconds (at least two). We are using the following code to initialize the audio unit: OSStatus status; AudioComponentDescription desc; AudioComponent inputComponent; desc.componentType = kAudioUnitType_Output; desc.componentSubType = kAudioUnitSubType_VoiceProcessingIO; desc.componentFlags = 0; desc.componentFlagsMask = 0; desc.componentManufacturer = kAudioUnitManufacturer_Apple; inputComponent = AudioComponentFindNext(NULL, &desc); status = AudioComponentInstanceNew(inputComponent, &unit); We are having the issue with current MacOS versions on a host of different Macs (x86 and x64 alike). It takes two to three seconds until AudioComponentInstanceNew returns. We also see the following errors in the log multiple times: AUVPAggregate.cpp:2560 AggInpStreamsChanged wait failed and those right after (which I don't know if they matter to this issue): KeystrokeSuppressorCore.cpp:44 ERROR: KeystrokeSuppressor initialization was unsuccessful. Invalid or no plist was provided. AU will be bypassed. vpStrategyManager.mm:486 Error code 2003332927 reported at GetPropertyInfo
3
1
1.1k
Jun ’24
Get downloaded tracks when querying MPMediaItem and can't detect them
Hello, We've a music app reading MPMediaItem. We got items using MPMediaQuery. But we realized that some downloaded tracks from Apple Music were fetched too. Not all downloaded track but only those who were played recently. Of course, since these tracks are protected with DRM we can't play them in our player. It's weird to get them in our query because we added predicate in order to dont fetch protected asset and iCloud item MPMediaPropertyPredicate(value: false, forProperty: MPMediaItemPropertyHasProtectedAsset) MPMediaPropertyPredicate(value: false, forProperty: MPMediaItemPropertyIsCloudItem) To be sure, we made a second check on each item we've fetched extension MPMediaItem { public func isValid() -> Bool { return self.assetURL != nil && !self.isCloudItem && !self.hasProtectedAsset } } But we still get these items. Their hasProtectedAsset attribute always return false. I dont know if it's a bug, but since we can't detect this items as Apple Music downloaded track, we can't either: filter them to not add them in our application library OR switch on a MPMusicPlayerController.applicationMusicPlayer to allow the user to play them
1
1
775
Jun ’24
Issue with iPhone QR Code Contact Preview Displaying Email Instead of Organization
Post Content: Hi everyone, I’m encountering an issue with how iPhone displays contact information from a vCard QR code in the contact preview. When I scan the QR code with my iPhone camera, the contact preview shows the email address between the name and the contact image, instead of displaying the organization name. Here’s the structure of the vCard I’m using: BEGIN:VCARD VERSION:3.0 FN:Ahmad Rana N:Rana;Ahmad;;; ORG:Company 3 TEL;TYPE=voice,msg:+1234567890 EMAIL:a(at the rate)gmail.com URL:https://example.com IMPP:facebook:fb END:VCARD What I Expect: When I scan it with camera and in the contact preview before creating the camera I want organization name between name and image of the preview but I get email instead of ogrganization name. If only organisation is passed then it displays correctly but when I pass email it displayed email in between. Steps I’ve Taken: Verified the vCard structure to ensure it follows the standard format. Reordered the fields in the vCard to prioritize the organization name and job title. Tested with a simplified vCard containing only the name, organization, and email. Despite these efforts, the email address continues to be displayed in the contact preview between the name and the contact image, while the organization name is not shown as expected. Question: How can I ensure that the organization name is displayed correctly in the contact preview on iPhone when scanning a QR code? Are there specific rules or best practices for field prioritization in vCards that I might be missing? I would appreciate any insights or suggestions on how to resolve this issue. Thank you!
2
0
819
Jun ’24
PiP not launching from a WKWebview Sandboxed app
Hi, I am developing an app that has a WKWebView and it can open sites like Youtube. The app is sandboxed as it is meant to be uploaded to the mac App Store. It has a feature PiP where we start the native PiP by calling a browser Javascript where we tell the WKWEBView to fire the PiP. It works well when we are running the code from XCODE in Debug scheme. When we run the code from release mode by archiving it or directly from the build folder, the WKWebView is not able to fire the PiP Agent and thus the Native PiP window is not visible, while the site shows that PiP is opened and we can here the sound being played. But PiP window is not visible. I cannot see PiPAgent in activity monitor. Why does it not work from within the release build outside xcode. But when I try to run the build directly from the Finder in builds folder, this PiP feature does not work. Request technical help for this. Thanks!
1
1
583
Jul ’24
Generating Live Photo from JPG and MOV fails
I am working on an iOS application using SwiftUI where I want to convert a JPG and a MOV file to a live photo. I am utilizing the LivePhoto Class from Github for this. The JPG and MOV files are displayed correctly in my WallpaperDetailView, but I am facing issues when trying to download the live photo to the gallery and generate the Live Photo. Here is the relevant code and the errors I am encountering: Console prints: Play button should be visible Image URL fetched and set: Optional("https://firebasestorage.googleapis.com/...") Video is ready to play Video downloaded to: file:///var/mobile/Containers/Data/Application/.../tmp/CFNetworkDownload_7rW5ny.tmp Failed to generate Live Photo I have verified that the app has the necessary permissions to access the Photo Library. The JPEG and MOV files are successfully downloaded and can be displayed in the app. The issue seems to occur when generating the Live Photo from the downloaded files. struct WallpaperDetailView: View { var wallpaper: Wallpaper @State private var isLoading = false @State private var isImageSaved = false @State private var imageURL: URL? @State private var livePhotoVideoURL: URL? @State private var player: AVPlayer? @State private var playerViewController: AVPlayerViewController? @State private var isVideoReady = false @State private var showBuffering = false var body: some View { ZStack { if let imageURL = imageURL { GeometryReader { geometry in KFImage(imageURL) .resizable() ... } } if let playerViewController = playerViewController { VideoPlayerViewController(playerViewController: playerViewController) .frame(maxWidth: .infinity, maxHeight: .infinity) .clipped() .edgesIgnoringSafeArea(.all) } } .onAppear { PHPhotoLibrary.requestAuthorization { status in if status == .authorized { loadImage() } else { print("User denied access to photo library") } } } private func loadImage() { isLoading = true if let imageURLString = wallpaper.imageURL, let imageURL = URL(string: imageURLString) { self.imageURL = imageURL if imageURL.scheme == "file" { self.isLoading = false print("Local image URL set: \(imageURL)") } else { fetchDownloadURL(from: imageURLString) { url in self.imageURL = url self.isLoading = false print("Image URL fetched and set: \(String(describing: url))") } } } if let livePhotoVideoURLString = wallpaper.livePhotoVideoURL, let livePhotoVideoURL = URL(string: livePhotoVideoURLString) { self.livePhotoVideoURL = livePhotoVideoURL preloadAndPlayVideo(from: livePhotoVideoURL) } else { self.isLoading = false print("No valid image or video URL") } } private func preloadAndPlayVideo(from url: URL) { self.player = AVPlayer(url: url) let playerViewController = AVPlayerViewController() playerViewController.player = self.player self.playerViewController = playerViewController let playerItem = AVPlayerItem(url: url) playerItem.preferredForwardBufferDuration = 1.0 self.player?.replaceCurrentItem(with: playerItem) ... print("Live Photo Video URL set: \(url)") } private func saveWallpaperToPhotos() { if let imageURL = imageURL, let livePhotoVideoURL = livePhotoVideoURL { saveLivePhotoToPhotos(imageURL: imageURL, videoURL: livePhotoVideoURL) } else if let imageURL = imageURL { saveImageToPhotos(url: imageURL) } } private func saveImageToPhotos(url: URL) { ... } private func saveLivePhotoToPhotos(imageURL: URL, videoURL: URL) { isLoading = true downloadVideo(from: videoURL) { localVideoURL in guard let localVideoURL = localVideoURL else { print("Failed to download video for Live Photo") DispatchQueue.main.async { self.isLoading = false } return } print("Video downloaded to: \(localVideoURL)") self.generateAndSaveLivePhoto(imageURL: imageURL, videoURL: localVideoURL) } } private func generateAndSaveLivePhoto(imageURL: URL, videoURL: URL) { LivePhoto.generate(from: imageURL, videoURL: videoURL, progress: { percent in print("Progress: \(percent)") }, completion: { livePhoto, resources in guard let resources = resources else { print("Failed to generate Live Photo") DispatchQueue.main.async { self.isLoading = false } return } print("Live Photo generated with resources: \(resources)") self.saveLivePhotoToLibrary(resources: resources) }) } private func saveLivePhotoToLibrary(resources: LivePhoto.LivePhotoResources) { LivePhoto.saveToLibrary(resources) { success in DispatchQueue.main.async { if success { self.isImageSaved = true print("Live Photo saved successfully") } else { print("Failed to save Live Photo") } self.isLoading = false } } } private func fetchDownloadURL(from gsURL: String, completion: @escaping (URL?) -> Void) { let storageRef = Storage.storage().reference(forURL: gsURL) storageRef.downloadURL { url, error in if let error = error { print("Failed to fetch image URL: \(error)") completion(nil) } else { completion(url) } } } private func downloadVideo(from url: URL, completion: @escaping (URL?) -> Void) { let task = URLSession.shared.downloadTask(with: url) { localURL, response, error in guard let localURL = localURL, error == nil else { print("Failed to download video: \(String(describing: error))") completion(nil) return } completion(localURL) } task.resume() } }```
1
1
712
Jul ’24