So, your fancy webcam or video capture card won’t work as a video input for FaceTime. Let’s take a look at why that happens.
sandboxd: () FaceTime(766) deny mach-lookup com.blackmagic-design.desktopvideo.DeckLinkHardwareXPCService
The driver for the video capture device appears to be implemented as an XPC Service and FaceTime is denied access by the sandbox. But don’t Apple’s own apps get special entitlements for stuff like this? Let’s take a look:
$ codesign -dvvv --entitlements - /Applications/FaceTime.app/ ... <key>com.apple.security.temporary-exception.sbpl</key> <string>(allow mach-lookup (global-name-regex #"^[0-9]+$"))</string>
So there is an exemption for FaceTime that allows mach-lookups for services that match this regex, but it will only match numeric-only strings. Obviously, this won’t match “com.blackmagic-design.desktopvideo.DeckLinkHardwareXPCService” or any other service named in reverse domain notation.
Let’s try ripping out the code signature, and fiddling with the entitlement to say:
<string>(allow mach-lookup (global-name-regex #"^******$"))</string>
Restart FaceTime, and indeed it will detect the video device with the correct resolution and frame rate (still no picture, but the mach-lookup went through!). Also, after messing with the signature, FaceTime pulls a Skype and will not establish any connection. So, no real success in getting the device to work, but a little win in at least getting around the bad regex for the mach-lookup.