Archive for the ‘MacOS Programming’ Category


Preface

This Article is designated for developers with C++ & Objective-C experience, A basic knowledge of iOS Operating System is assumed.

Introduction

In this article I will describe an approach enabling code injection to an existing AppStore Application, In a nut-shell, this involves breaking FairPlay encryption, Injecting code, re-signing and re-packaging the App.

Once the process described in this article is complete, the App ( having the injected code on board ) can be freely installed on any type of device, whether Jail-broken OR NOT.

High-level Flow diagram

Pre-requisites

  • A developer account with Apple
    This is required so after Injecting the DynamicLib the application could get re-packaged and installed on the iOS device, use this link to create one.
  • X-code 6 ( supporting iOS Framework Compilation )
    With X-code 6 it is possible to create custom frameworks, which are, in essence, DynamicLibraries.
  • Two iOS device
    Two devices are required for the process, one, that will run the patched app, and another, JailBroken device with cydia installed where FairPlay is Removed.
  • Mach-O View
    An application used to parse Apple Executable files, use this link to download.

The Mach-O File format

Executable files on OS-X & iOS are stored using the Mach-O format, this executable format is specific for apple and is used to store multiple instances of the binary code, each using a different instruction set ( x86, ARM, … ) along with metadata telling the OS how the code is to be executed.

The Mach-O file format consists of a header followed by a set of commands and one or more segments, Of specific interest is the set of load commands, I will cover this later in details

Application Packaging

Applications are packaged in bundles, a bundle is a ZIP file with an ‘.IPA’ extension, these, contain the Program/Framework executable, and, all of the related resources such as Strings, Images, NIBs, and most importantly the developer certificates and application Entitlements.

A simple IPA is presented bellow, this IPA consists of the Mach-O Executable ( named Tester ), the program resources, located under the Base.lproj folder, the provisioning profile ( named mbedded.mobileprovision ) where the authorized devices are defined,

the Entitlements file ( named archived-expanded-entitlements.xcent ) where specific application capabilities and or security permissions are defined, and ‘Info.plist‘ where the bundle id and other application specific properties are set, later on, these are used to re-sign an re-pack the IPA with the Injected code.

FairPlay Workaround

FairPlay is apples DRM used to protect applications downloaded from the AppStore, it prevents execution of the application on unauthorized devices.

FairPlay removal requires a JailBroken device, Having Cydia and the Clutch tool installed, I have found this site to contain the most up-to-date iOS JailBreaks, high level instructions of the process are provided by this Tutorial, next, I will provide explanation regarding FairPlay Mach-O protection, How Clutch is removing encryption, and, why a Jailbroken device is needed.

Mach-O Executables are signed with the developer certificate, this certificate is used in conjunction with other OS information during FairPlay decryption, The code injection process mandate Mach-O modifications, and these, mandate re-signing with a different certificate ( eg. a custom developer certificate ), this is why it is needed to remove FairPlay before doing any change to the Mach-O Executable.

FairPlay encrypts part of the code segment of the Mach-O executable, the encrypted region is indicated using the LC_ENCRYPTION_INFO load command ( consisting of the ‘encryption_info_command’ structure ) as can be seen in the following Mach-O View snap:

Having ‘encryption_info_command::cryptid’ set to a value different than zero indicate that the Mach-O executable is FairPlay protected.
During the process of Mach-O loading, the operating system decrypt the protected region, and thus, the binary code reside un-encrypted in memory during execution, Clutch is taking advantage of that fact by starting the application, and, once loaded in memory, copies the decrypted section back to the physical Mach-O file replacing the encrypted segment, obviously, accessing another process memory mandate root privileges, and this, is why a Jailbroken device is needed.

Code Injection

Once we have FairPlay removed and the Mach-O is un-encrypted we can inject custom code to be executed on behalf of the application ( at the application sandbox ), this is done by having the code compiled as a framework ( consisting of a DynamicLib ), and, having a LC_LOAD_DYLIB command ( referring that DynamicLib ) added to the Mach-O executable.

As can be seen on the image to the right, With X-code 6, Creating an iOS DynamicLib was made much simpler, with older X-code versions, one would have to tinker through with the solution files to have X-code produce iOS compatible DynamicLib, X-code 6 has introduced a new type of project, the “Cocoa Touch Framework”, which in essence, is a DynamicLib packed together with it’s associated resources.

The compiler attribute “__attribute__((constructor))” is used to ensure the Framework code execute upon ‘module loading’/’application start-up’, this is illustrated bellow:

#import <Foundation/Foundation.h>
#import <UIKit/UIKit.h>

__attribute__((constructor))
void EntryPoint() {
    NSLog(@“Injected routine…\n”);
    dispatch_async(dispatch_get_main_queue(), ^{
        UIAlertView *alert = [[UIAlertView alloc] initWithTitle:@“Hello”
                                                        message:@“Code Injected”
                                                     delegate:nil
                                             cancelButtonTitle:@“OK”
                                             otherButtonTitles:nil];
        [alert show];
    });
}

The framework code injected to the application executable might have references to common frameworks, for example, both, the Injected code and the Application code might use/refer UIKit.framework, this is a tricky situation since only a single framework version is loaded during execution, using different versions of the same type of framework might cause un-predictable behaviour, having that said, it is essential, either to have the injected code dependent on a minimal set of external frameworks, OR, to make sure to compile the Injected code with the same frameworks used by the application being injected too.

Once the framework to be injected was generated, the Application IPA ( Zip file ) must be extracted and the Injected framework must be copied to the Application executable folder so later on it could get loaded in to the application memory address space, this is demonstrated on the image to the right.
The Mach-O executable is then to be added a load command referring the Injected Framework, this, will make the OS load the framework when the Mach-O executable is lunched.

After the injection has taken place, the Application Executable looks as follows ( in red is the path to the Injected Framework ), Note the ‘@executable_path’ notation, this notation is used resemble a path relative to the current executing Mach-O.

The Injected framework is added as the last LC_LOAD_DYLIB Command to ensure that all the application dependent modules were already loaded.

Sample Code ( download )

This Article is accompanied a simple single file sample-code implementing the Injection logic, the following include an explanation of the major parts of the code.

As stated before, In-order to be able to inject the framework the Mach-O must be un-encrypted ( otherwise signature verification will fail, and, it’ll not get loaded by the OS ), the following code-snap iterate through all of the commands until it finds the ‘LC_ENCRYPTION_INFO‘ command, if found, it verify no encryption is applied by evaluating ‘cryptid‘, the file is concluded un-encrypted if either ‘cryptid‘ is evaluated to zero, OR, when the ‘LC_ENCRYPTION_INFO‘ command is not found.

m_pCmdFirst‘ in the code bellow is found immediately after the mach_header ( whether 64 bit or not )

template< typename T >// Supports both the 32bit and 64 bit versions of ‘mach_header’
bool MachOParser<T>::IsMachOEncrypted() {
    uint8_t*        pPtr = (uint8_t*)m_pCmdFirst;
    load_command*    pCmd = m_pCmdFirst;
    for (uint32_t i = 0;
        i < m_pMachO->ncmds;
        i++, pPtr += pCmd->cmdsize, pCmd = (load_command*)pPtr)
    {
        if (LC_ENCRYPTION_INFO != pCmd->cmd)
            continue;
        if (0 != ((encryption_info_command*)pCmd)->cryptid)
            return true;
        break;// We have found the encryption info section, no need to keep on searching
    }
    return false;
}

The next code-snap is responsible for injecting the ‘dylib_command‘ as the last ‘dylib_command‘ of the Mach-O, this is done using ‘m_pCmdLastLoadLib ‘ which is pre-initialized to the last ‘dylib_command‘ upon Mach-O loading ( see the ‘ReloadCommands()‘ method of the accompanied code ).

cmdInjected‘ define the injected command, important variables are ‘dylib_command::cmd‘ that indicate the command type, and, ‘dylib_command::dylib::name::offset‘ that indicate an offset to where the name of the framework is located within the Mach-O, relative to the beginning of the ‘dylib_command‘ command, in our case, the name of the framework is directly located after the command, and thus, the relative offset is ‘sizeof(dylib_command)

One more thing to note, all commands must be aligned to 4 bytes length, and thus, the name of the framework is padded with zeros making sure it is appropriately aligned.

m_vecCommands‘ is pre-initialized with all commands upon Mach-O loading ( see the ‘ReloadCommands()‘ method of the accompanied code ).

template< typename T >// Supports both the 32bit and 64 bit versions of ‘mach_header’
int MachOParser<T>::InjectDyLib(const char* pDynLibPath) {
    union {
        dylib_command    cmdInjected;
        char            __pRaw__[512];
    };
    cmdInjected.cmd = LC_LOAD_DYLIB;
    cmdInjected.dylib.compatibility_version = 0x00010000;
    cmdInjected.dylib.current_version = 0x00020000;
    cmdInjected.dylib.timestamp = 2;
    cmdInjected.dylib.name.offset = (uint32_t)sizeof(dylib_command);

    char* pLibNameStart = (char*)(&cmdInjected + 1);
    strncpy(pLibNameStart, pDynLibPath, sizeof(__pRaw__)-cmdInjected.dylib.name.offset);
    cmdInjected.cmdsize = cmdInjected.dylib.name.offset + (uint32_t)strlen(pLibNameStart);
    const div_t d = div(cmdInjected.cmdsize, 4);
    if (0 != d.rem) {// Commands size must be aligned to 4
        memset((char*)&cmdInjected + cmdInjected.cmdsize, 0, 4 – d.rem);
        cmdInjected.cmdsize += (4 – d.rem);
    }

    if (FALSE == IsThereEnoughSpaceForCmd((load_command*)&cmdInjected)) {
        // TBD: In case no space is available in the existing Mach-O, enlarge
        // the size of the file and update section offsets/RVAs
        return ENOBUFS;
    }

    char* pInjectionOffset = (char*)m_pCmdLastLoadLib + m_pCmdLastLoadLib->cmdsize;
    const char* pLoadCmdsEnd = (char*)m_vecCommands[m_vecCommands.size() – 1] +
                                m_vecCommands[m_vecCommands.size() – 1]->cmdsize;
    // Make space for the new command
    memmove(pInjectionOffset + cmdInjected.cmdsize,
            pInjectionOffset,
            (size_t)(pLoadCmdsEnd – pInjectionOffset));
    // Inject the dynlib command
    memcpy(pInjectionOffset, &cmdInjected, cmdInjected.cmdsize);
    m_pMachO->ncmds++;
    m_pMachO->sizeofcmds += cmdInjected.cmdsize;
    return 0;
}

Sign & Re-package

This is the last step before the patched application can be used on a non Jail-broken device, In this section a method for signing/packaging using a provisioning profile is demonstrated although any other approach will work ( potentially, even AppStore distribution )

  1. Extract application IPA
    iOS applications are packed in IPA files, which, in essence, are zip files containing all of the application resources, the first thing to do is to extract the Application IPA to a known folder, this can be done using the unzip command-line utility in the following manner:

    unzip %Filename%.ipa -d %dest folder%
  2. Remove existing signature
    When applications are signed, a per file signature is generated, these are stored in a special file named ‘CodeResources‘ which is located under the ‘_CodeSignature‘ folder, to remove these signatures the ‘_CodeSignature‘ is to be removed:

    rm -fR %dest folder%/Payload/%App name%.app/_CodeSignature/

  3. Update Provisioning Profile
    The provisioning profile defines the devices the application is allowed to execute on, An iOS developer account is required, click here to generate one, once the provisioning profile is ready it should be copied to the application @executable_path and named ‘embedded.mobileprovision‘:

    cp %profile file%.mobileprovision %dest folder%/Payload/%App name%.app/embedded.mobileprovision

  4. Update Entitlements
    The Entitlements file is a simple ‘.plist‘ file that “confer specific capabilities or security permissions to your iOS” App, it should be copied to the application @executable_path and named ‘Entitlements.plist‘, it refers the developer via the ‘application-identifier‘ variable which must prefix with the developer id, this is illustrated bellow:

    <?xml version=“1.0” encoding=“UTF-8”?>
    <!DOCTYPE plist PUBLIC “-//Apple//DTD PLIST 1.0//EN” http://www.apple.com/DTDs/PropertyList-1.0.dtd&#8221;>
    <plist version=“1.0”>
    <dict>
        <key>application-identifier</key>
        <string>%Developer Id%.*</string>
        <key>get-task-allow</key>
        <true/>
    </dict>
    </plist>

    The %Developer Id% can be retrieved directly form the developer portal, OR, under the ‘User Id’/’Org Unit’ when inspecting the developer certificate using using the KeyChain tool.

    Copy the ‘Entitlements‘ file to the ‘@executable_folder‘:

    cp %entitlements file%.pinfo %dest folder%/Payload/%app name%.app/entitlements.plist

  5. Update Framework
    Once the framework is compiled ( into a folder named %Proj Name%.framework ) it should be copied into the the applications @executable_path, make sure to have any descendent _CodeSignature folder removed.

    cp -R %Name%.framework %dest folder%/Payload/%app name%.app/%Name%.framework/

  6. Inject Framework to the App Executable
    This is where we use the above mentioned code, compile the accompanied code and run the following to Inject the framework into the application executable:

    Injector “%app path%/Payload/%Name%.app/%Name%” “@executable_path/%Proj name%.framework/%Proj name%

  7. Sign
    It is needed to replace the signature embedded in the Mach-O executables, and, to re-generate the _CodeSignature folder using the developer identity, during the process of creating a developer account the development signing certificates were installed on the local machine ( this might have been done automatically by xcode ), these can be seen using the Key-Chain tool, OR, via the xcode project ‘Code Signing Identity’ of the project target ‘Build Settings’ ( located at ‘project properties->Target->Build settings->Code signing->Code signing Identity‘ ), the codesign tool is to be used to sign the Mach-O binaries with this identity as demonstrated bellow:

    codesign -s “%KeyChain cert name%” –force –deep “%dest folder%/Payload/%app name%.app/%Name%”

    And

    codesign -s “%KeyChain cert name%” –force –deep “%dest folder%/Payload/%app name%.app/%Proj Name%.framework/%Proj Name%”

    Once the application executables signature is updated/replaced, the _CodeSignature folder is to be reconstructed using the development signing certificates and the Entitlements file:

    codesign -s “%KeyChain cert name%” –entitlements “%dest folder%/Payload/%app name%.app/Entitlements.plist” –force –deep %dest folder%/Payload/%app name%.app/

  8. Re-package
    The last step is to zip together all of the resulting files into an IPA, this is done using the zip command in the following manner:

    zip -r %dest folder%/%Name%.ipa “%dest folder%/Payload/

  9. Deploy
    That’s it, all is set, simply drag the resulting IPA on to iTunes and install the patched application on a provisioned device as described on the ‘Installing Your App on Test Devices’ of this link.
  10. IMPORTANT NOTE:
    If the app being packaged include 3rd party Frameworks and/or Extensions these are also to be signed, of specific importance is the order in which they are signed, for example, if the app is composed of the following dependent modules

    %app name%.app/Plugins/%Extension%.appex/%tool app%.app

    the order in which they are to be signed must be bottom to top, thus, %tool app% -> %Extension% -> %app name%, otherwise package verification will fail and the app will not get installed on the device.

    When signing a module ( App/Framework/Extension/… ) the generated _CodeSignature refer all of it’s decedents, if any of the decedents is changed AFTER the _CodeSignature was generated verification will fail due to signature inconsistency and the app will not be installed, To avoid that, module signing must be bottom to top.

    A handy way to verify validity is to use the following for each of the embedded modules

    codesign –verify –verbose %App/Extension/Framework Name%


End result


Background is intentionally blurred

Risks & Limitations

  • Usage of a provisioning profile ( ad-hoc distribution ) limits usage on up to 100 devices
  • Injected framework must consist of a minimal set of dependencies to avoid different version of the same framework being loaded by the Mach-O
  • The current injection code takes as granted that there is enough space available between the last command and the first section for the Injected ‘dylib_command‘, while I have seen no cases where there was not enough space, in theory such a case can exist, to deal with it, a page aligned block should be inserted, and, all sections RVA should get correspondingly modified.

Disclaimer

The write of this article takes no liability for any direct of indirect damage that usage of the accompanied source code and the demonstrated approach might cause.

References

Register as a developer with Apple, Mach-O file format, Mach-O View, Appels FairPlay, Creating provisioning profiles,
Application Entitlements, Bundle Structure, Cydia, Clutch Open-Source project, Creating Your Team Provisioning Profile, Working with the System Key-Chain, codesign tool, Beta Testing iOS Apps

Perspective

The intended audience of this article are MacOS C++/Obj-C developers and architects, It is assumed that the reader of this article is familiar with object oriented programming and design.

For the purpose of brevity and clarity, thread synchronization aspect is omitted and not discussed in details in this the article.

Introduction

The Objective-C AVFoundation framework is encapsulating media processing ( capture, editing, … ), it is robust, well document and covers most of the A/V use-cases, however, some edge case use-cases are not supported by this framework, for example, being able to directly access the buffers sent out from the device, this, is specifically important when the payload sent out from the device is already muxed and/or compressed, in such cases, AVFoundation ( AVCaptureSession in-specific ) will de-mux and/or decode the payload before making it accessible to the user, to get direct access to the buffers sent out from the device w/o any intermediate intervention we will have to use a lower-level API, namely, the CoreMediaIO.

Apples CoreMediaIO is a low-level C++ framework for accessing and interacting with audio/video devices such as cameras, capture cards and even Mirroring sessions of iOS devices

The problem with CoreMediaIO is lack of documentation, and, the fact that the existing sample code is old and require quite some tinkering to have it compiling with latest SDKs

In this short article I will provide a simple sample code demonstrating capture and format resolution using CoreMediaIO and some AVFoundation

Implementation

CoreMediaIO API are provided through the “CoreMediaIO.framework“, make sure to have it included by the project, and to have “CoreMediaIO/CMIOHardware.h” included/imported.

The first thing we have to do in-order to be able to start capture is to find the device of interest, if we are interested in screen capture ( for example capturing the screen of an attached iOS device ) we need to enable CoreMediaIO ‘DAL’ plug-ins, This, is demonstrated in the following code snap:

void EnableDALDevices()
{
    CMIOObjectPropertyAddress prop = {
        kCMIOHardwarePropertyAllowScreenCaptureDevices,
        kCMIOObjectPropertyScopeGlobal,
        kCMIOObjectPropertyElementMaster
    };

    UInt32 allow = 1;
    CMIOObjectSetPropertyData(kCMIOObjectSystemObject,
                            &prop, 0, NULL,
                            sizeof(allow), &allow );
}

Some devices are added or removed on runtime, to get runtime indications for device addition or removal, an A/V Capture device notification is set using the NSNotificationCenter class, the AVCaptureDevice added/removed is indicated by the ‘object‘ variable of the ‘note‘ ^block argument, This is demonstrated by the following code snap, Be aware that no notifications will be received unless a Run Loop is executed.

NSNotificationCenter *notiCenter = [NSNotificationCenter defaultCenter];
id connObs =[notiCenter addObserverForName:AVCaptureDeviceWasConnectedNotification
                                    object:nil
                                     queue:[NSOperationQueue mainQueue]
                                usingBlock:^(NSNotification *note)
                                            {
                                                // Device addition logic
                                            }];

id disconnObs =[notiCenter addObserverForName:AVCaptureDeviceWasDisconnectedNotification
                                     object:nil
                                        queue:[NSOperationQueue mainQueue]
                                 usingBlock:^(NSNotification *note)
                                            {
                                                // Device removal logic
                                            }];

[[NSRunLoop mainRunLoop] run];
[notiCenter removeObserver:connObs];
[notiCenter removeObserver:disconnObs];

The next step is to enumerate the attached capture devices, this is either done using AVCaptureDevice class of AVFoundation or, directly using CoreMediaIO C++ APIs, each capture device provide an uniquely identifier, in the next code snap, that id will be used to find the device of interest

The Code Snap bellow demonstrate device enumeration using AVFoundation APIs, To filter a specific type of device use the ‘devicesWithMediaType’ method of the AVCaptureDevice class.

// Use the ‘devicesWithMediaType’ to filter devs by media type
// NSArray* devs = [AVCaptureDevice devicesWithMediaType:AVMediaTypeMuxed];
NSArray* devs = [AVCaptureDevice devices];
NSLog(@“devices: %d\n”, (int)[devs count]);

for(AVCaptureDevice* d in devs) {
    NSLog(@“uniqueID: %@\n”, [d uniqueID]);
    NSLog(@“modelID: %@\n”, [d modelID]);
    NSLog(@“description: %@\n”, [d localizedName]);
}

The next step is to find the device we want to use for capture, Capture devices in CoreMediaIO are identified by CMIODeviceID, the following code-snap demonstrate how to resolve the devices CMIODeviceID according to their unique ID which is a-priori known and externally provided.

OSStatus GetPropertyData(CMIOObjectID objID, int32_t sel, CMIOObjectPropertyScope scope,
                         UInt32 qualifierDataSize, const void* qualifierData, UInt32 dataSize,
                         UInt32& dataUsed, void* data) {
    CMIOObjectPropertyAddress addr={ (CMIOObjectPropertySelector)sel, scope,
                                     kCMIOObjectPropertyElementMaster };
    return CMIOObjectGetPropertyData(objID, &addr, qualifierDataSize, qualifierData,
                                     dataSize, &dataUsed, data);
}

OSStatus GetPropertyData(CMIOObjectID objID, int32_t selector, UInt32 qualifierDataSize,
                         const void* qualifierData, UInt32 dataSize, UInt32& dataUsed,
                         void* data) {
    return GetPropertyData(objID, selector, 0, qualifierDataSize,
                         qualifierData, dataSize, dataUsed, data);
}

OSStatus GetPropertyDataSize(CMIOObjectID objID, int32_t sel,
                             CMIOObjectPropertyScope scope, uint32_t& size) {
    CMIOObjectPropertyAddress addr={ (CMIOObjectPropertySelector)sel, scope,
                                     kCMIOObjectPropertyElementMaster };
    return CMIOObjectGetPropertyDataSize(objID, &addr, 0, 0, &size);
}

OSStatus GetPropertyDataSize(CMIOObjectID objID, int32_t selector, uint32_t& size) {
    return GetPropertyDataSize(objID, selector, 0, size);
}

OSStatus GetNumberDevices(uint32_t& cnt) {
    if(0 != GetPropertyDataSize(kCMIOObjectSystemObject, kCMIOHardwarePropertyDevices, cnt))
        return -1;
    cnt /= sizeof(CMIODeviceID);
    return 0;
}

OSStatus GetDevices(uint32_t& cnt, CMIODeviceID* pDevs) {
    OSStatus status;
    uint32_t numberDevices = 0, used = 0;
    if((status = GetNumberDevices(numberDevices)) < 0)
        return status;
    if(numberDevices > (cnt = numberDevices))
        return -1;
    uint32_t size = numberDevices * sizeof(CMIODeviceID);
    return GetPropertyData(kCMIOObjectSystemObject, kCMIOHardwarePropertyDevices,
                         0, NULL, size, used, pDevs);
}

template< const int C_Size >
OSStatus GetDeviceStrProp(CMIOObjectID objID, CMIOObjectPropertySelector sel,
                         char (&pValue)[C_Size]) {
    CFStringRef answer = NULL;
    UInt32     dataUsed= 0;
    OSStatus    status = GetPropertyData(objID, sel, 0, NULL, sizeof(answer),
                                         dataUsed, &answer);
    if(0 == status)// SUCCESS
        CFStringCopyUTF8String(answer, pValue);
    return status;
}

template< const int C_Size >
Boolean CFStringCopyUTF8String(CFStringRef aString, char (&pText)[C_Size]) {
    CFIndex length = CFStringGetLength(aString);
    if(sizeof(pText) < (length + 1))
        return false;
    CFIndex maxSize = CFStringGetMaximumSizeForEncoding(length, kCFStringEncodingUTF8);
    return CFStringGetCString(aString, pText, maxSize, kCFStringEncodingUTF8);
}

Utility methods

OSStatus FindDeviceByUniqueId(const char* pUID, CMIODeviceID& devId) {
    OSStatus status = 0;
    uint32_t numDev = 0;
    if(((status = GetNumberDevices(numDev)) < 0) || (0 == numDev))
        return status;
    // Allocate memory on the stack
    CMIODeviceID* pDevs = (CMIODeviceID*)alloca(numDev * sizeof(*pDevs));
    if((status = GetDevices(numDev, pDevs)) < 0)
        return status;
    for(uint32_t i = 0; i < numDev; i++) {
        char pUniqueID[64];
        if((status = GetDeviceStrProp(pDevs[i], kCMIODevicePropertyDeviceUID, pUniqueID)) < 0)
            break;
        status = afpObjectNotFound;// Not Found…
        if(0 != strcmp(pUID, pUniqueID))
            continue;
        devId = pDevs[i];
        return 0;
    }
    return status;
}

Device resolution by UID

CoreMediaIO Capture devices expose streams, each such stream is a data source and is indicated using a CMIOStreamID type, one stream might provide Video payload, another can provide Audio payload and others might provide multiplexed payload, while capturing we have to select a stream and start pumping out data, the following code-snap demonstrate how to enumerate the available streams for a given device ( indicated by it’s CMIODeviceID ) and how to resolve the payload format.

uint32_t GetNumberInputStreams(CMIODeviceID devID)
{
    uint32 size = 0;
    GetPropertyDataSize(devID, kCMIODevicePropertyStreams,
                        kCMIODevicePropertyScopeInput, size);
    return size / sizeof(CMIOStreamID);
}

OSStatus GetInputStreams(CMIODeviceID devID, uint32_t&
                        ioNumberStreams, CMIOStreamID* streamList)
{
    ioNumberStreams = std::min(GetNumberInputStreams(devID), ioNumberStreams);
    uint32_t size     = ioNumberStreams * sizeof(CMIOStreamID);
    uint32_t dataUsed = 0;
    OSStatus err = GetPropertyData(devID, kCMIODevicePropertyStreams,
                                    kCMIODevicePropertyScopeInput, 0,
                                    NULL, size, dataUsed, streamList);
    if(0 != err)
        return err;
    ioNumberStreams = size / sizeof(CMIOStreamID);
    CMIOStreamID* firstItem = &(streamList[0]);
    CMIOStreamID* lastItem = firstItem + ioNumberStreams;
    std::sort(firstItem, lastItem);
    return 0;
}

Utility methods

CMIODeviceID devId;
FindDeviceByUniqueId(“4e58df701eb87”, devId);

uint32_t numStreams = GetNumberInputStreams(devId);
CMIOStreamID* pStreams = (CMIOStreamID*)alloca(numStreams * sizeof(CMIOStreamID));
GetInputStreams(devId, numStreams, pStreams);
for(uint32_t i = 0; i < numStreams; i++) {
    CMFormatDescriptionRef fmt = 0;
    uint32_t                used;
    GetPropertyData(pStreams[i], kCMIOStreamPropertyFormatDescription,
                    0, NULL, sizeof(fmt), used, &fmt);
    CMMediaType mt     = CMFormatDescriptionGetMediaType(fmt);
    uint8_t     null1 = 0;// ‘mt’ is a 4 char string, we use ‘null1’ so
                         // it could be printed.
    FourCharCode fourcc= CMFormatDescriptionGetMediaSubType(fmt);
    uint8_t     null2 = 0;// ‘fourcc’ is a 4 char string, we use ‘null1’
                         // so it could be printed.
    printf(“media type: %s\nmedia sub type: %s\n”, (char*)&mt, (char*)&fourcc);
}

Stream format resolution

The next and final stage is to start pumping data out of the stream, this is done by registering a callback to be called upon by CoreMediaIO with the sampled payload, the following code-snap demonstrate how this is done and how to get access to the raw payload bytes.

CMSimpleQueueRef    queueRef = 0;// The queue that will be used to
                                 // process the incoming data
CMIOStreamCopyBufferQueue(strmID, [](CMIOStreamID streamID, void*, void* refCon) {
    // The callback ( lambda in out case ) being called by CoreMediaIO
    CMSimpleQueueRef queueRef = *(CMSimpleQueueRef*)refCon;
    CMSampleBufferRef sb = 0;
    while(0 != (sb = (CMSampleBufferRef)CMSimpleQueueDequeue(queueRef))) {
        size_t            len     = 0;// The ‘len’ of our payload
        size_t            lenTotal = 0;
        char*             pPayload = 0;// This is where the RAW media
                                     // data will be stored
        const CMTime     ts         = CMSampleBufferGetOutputPresentationTimeStamp(sb);
        const double     dSecTime = (double)ts.value / (double)ts.timescale;
        CMBlockBufferRef bufRef     = CMSampleBufferGetDataBuffer(sb);
        CMBlockBufferGetDataPointer(bufRef, 0, &len, &lenTotal, &pPayload);
        assert(len == lenTotal);
        // TBD: Process ‘len’ bytes of ‘pPayload’
    }
}, &queueRef, &queueRef);

One last thing to note, on more tan few cases the actual capture format is not available until the first sample is sent, in such cases it should be resolved upon first sample reception, the following code-snap demonstrate how to resolve Audio sample format using CMSampleBufferRef, the same can be done for video and other media types with a little more effort.

bool PrintAudioFormat(CMSampleBufferRef sb)
{
    CMFormatDescriptionRef    fmt    = CMSampleBufferGetFormatDescription(sb);
    CMMediaType                mt    = CMFormatDescriptionGetMediaType(fmt);

    if(kCMMediaType_Audio != mt) {
        printf(“Not an audio sample\n”);
        return false;
    }
    
    CMAudioFormatDescriptionRef afmt = (CMAudioFormatDescriptionRef)fmt;
    const auto pAud = CMAudioFormatDescriptionGetStreamBasicDescription(afmt);
    if(0 == pAud)
        return false;
    // We are expecting PCM Audio
    if(‘lpcm’ != pAud->mFormatID)// ‘pAud->mFormatID’ == fourCC
        return false;// Not a supported format
    printf(“mChannelsPerFrame: %d\nmSampleRate: %.1f\n”\
            “mBytesPerFrame: %d\nmBitsPerChannel: %d\n”,
         pAud->mChannelsPerFrame, pAud->mSampleRate,
         pAud->mBytesPerFrame, pAud->mBitsPerChannel);
    return true;
}

Final words

What provided in this article is just a glimpse of what is doable with CoreMediaIO, further information of can be found in the reference links bellow.

References

CoreMediaIO, AVFoundation, AVCaptureSession, NSNotificationCenter, Run Loop, AVCaptureDevice