ios – Is it doable to make use of ARKit with `AVCaptureMultiCamSession`?

What am I Doing?

I’m constructing an software that takes feed from an exterior webcam and maps that feed to a UIView overlaying all the display screen.

This software additionally wants to trace facial options, particularly how open the consumer’s mouth is, and cross that information throughout a USB connection to a receiving gadget. This worth for a way open the consumer’s mouth is is used to replace the peak of a mouth asset on the receiving gadget. If the consumer of the app opens their mouth, the face displayed on the receiving gadget additionally opens its mouth, and many others.

I’m presently in a position to do all of this by utilizing the Imaginative and prescient API along with AVCaptureMultiCamSession. Right here you possibly can see an instance, with a preview of the mouth monitoring information that will likely be despatched to the receiving gadget within the backside proper nook.

Imaginative and prescient’s VNFaceObservation.landmarks?.innerLips information principally offers you a hexagon of the consumer’s mouth.

We then calculate the “top” of the mouth by way of:

let factors: [CGPoint] = [ /* add some points here */ ]
let pointYs = factors.map { level in
    level.y
}
if let maxY = pointYs.max(), let minY = pointYs.min() {
    let mouthHeight: Double = (maxY - minY)

    // multiply mouthHeight by some issue, serialize, and cross throughout USB connection
}

The Subject

The problem is that this strategy makes the mouthHeight very jumpy, i.e. the mouth asset, whose top the info is mapping to, sparkles lots as a result of there’s not a clean transferring common of mouth heights. Notice that I used to be not transferring my mouth on this beneath instance, but the UI is doing this flickering.

Utilizing ARKit could be rather more relevant for this use case. After I take a look at how an Animoji mimics the consumer’s mouth “openness”, I do not observe this flickering.

What I’ve Tried

1. I’ve explored utilizing ARSCNView whose .session is operating an ARWorldTrackingConfiguration with userFaceTrackingEnabled = true as seen in this instance. The issue with this strategy is that, whereas this can present me the exterior world whereas additionally giving me ARFaceAnchor information, I’m pretty sure that there is no such thing as a solution to change the ARWorldTrackingConfiguration‘s digital camera to be an exterior digital camera.

2. I’ve tried eradicating the Imaginative and prescient framework fully and use an ARSCNView whose .session is operating an ARFaceTrackingConfiguration whereas persevering with to get the exterior webcam feed by way of

let multiCamSession = AVCaptureMultiCamSession()
let deviceDiscoverySession = AVCaptureDevice.DiscoverySession(deviceTypes: [.external], mediaType: .video, place: .unspecified)
if let gadget = deviceDiscoverySession.units.first {
    if let deviceInput = attempt? AVCaptureDeviceInput(gadget: gadget) {
        captureSession.addInput(deviceInput)
    }
}

however, sadly, I can’t get each the exterior webcam feed and the selfie digital camera feed to show on the similar time of their respective UIView and ARSCNView. It appears that evidently the ARSCNView simply makes the feed from the exterior digital camera and AVCaptureMultiCamSession cease as soon as it hundreds its digital camera.

What I’m questioning

What paths ahead do I’ve to make use of ARKit’s face monitoring whereas additionally having an exterior webcam feed? Take into account:

A. Is my assumption that “there is no such thing as a solution to change the ARWorldTrackingConfiguration‘s digital camera to be an exterior digital camera” true?

B. Is there a means to make use of ARSCNView or another technique of grabbing face monitoring information by way of ARKit from the gadget’s selfie cam whereas additionally seeing the exterior digital camera feed, by way of an AVCaptureMultiCamSession or in any other case?