Life in The Clouds

IOS multitasking and external accessories

When Apple first announced the External Accessory Framework with IOS 3 there was a great deal of excitement, it was thought that the new framework would open the iphone up to a host of new third party devices which could be attached via the dock connector, camera adaptor , USB and even bluetooth.

But along with the new Framework came new rules and limitations that tempered those early expectations.

For one the new framework required enrollment in the Made for Ipod program or MFI (which can be interpreted nowadays as Made for Iphone or whatever "I" device you care to postfix the "I" with. MFI membership is pretty much limited to OEMs that have a clear vision of a real product and can prove that to Apple.

USB and Bluetooth framworks were restriced to only MFI members.

Also the new framework emphasized one of the biggest complaints with IOS , the lack of mult-tasking.

prior to IOS version 4 the lifecycle of an application was not very complicated.

.The app is started by calling application:didFinishLaunchingWithOptions which starts the event loop, When the user quits the application, the whole app is closed and the method applicationWillTerminate is called. The next time the user opens the application, the method application:didFinishLaunchingWithOptions: is called again. 

This Presented a Problem for Apple  which they have been incrementally addressing since IOS 4.0.

Apple was not about to allow the wholesale multitasking of the kind available on Android. Android multitasking can be a headache for some applications. i.e.  streaming IPTV which may have need to maintain large buffers for audio and video, On iphone the applications are free to grab all the memory they can but on Android, the OS provides only for limited amount of memory and the app "lives" in fear of the OS killing it at any time for higher priority tasks.

But No Multi-tasking is equally a problem.

Lets take as a scenario a External Heart Monitoring accessory such as provided by Wahoo Fitness and a Iphone with 2 applications .

HealthMonitor which is attached to the accessory via  ANT protocol.

Pandora Radio, for those tunes you just must listen to while you exercise.

Prior to OS 4 whenever the user switched between Pandora Radio and HealthMonitor the former application had to completely quit and the application in the foreground had to completely reinitialize itself. Developers were required to implement there own protocols for fast restarts by caching data locally. The connection to the monitor would be servered and have to be reopended when the monitor was forground again. Not a happy lifecycle for a application that needed realtime sensor data.

Once more Bluetooth allows for multiple socket connections and multiple devices but the lack of multitaking meant that there could be a "single connection" to a "single accessory" only.

With IOS 4 Apple attempted to address these limitiations in a number of ways.

First "fast task switching" gave the os the ability to save enough of the state of the application to quickly restore it upon resumption of the application.

iOS4 introduces a more complicated lifecyle, when the user quits the app, it enters the background and method application:didEnterBackground is called. The application remains in the background for some determinied time period depending on what the application was doing before it entered background usually not exceeding 10 minutes, this allows time for the application to complete whatever task it was doing before entering a suspended state . If the application is still in the middle of some long task it may "request" additional time from the os in small increments to complete whatever task it was doing. Then the application enters the suspended state.

An IOS application can be in one of 5 states

• Not running - the app has been terminated or has not been launched.
• Inactive - the app is in the foreground but not receiving events (for example, the user has locked the device with the app active)
• Active - the normal state of "in use" for an app
• Background - the app is no longer on-screen but is still executing code
• Suspended - the app is still resident in memory but is not executing code

Apple Also defined a set of elite tasks type which are allowed to run in the background.

- Background audio - allows apllications to play audio continuously. - Voice over IP - Users can now receive VoIP calls and have conversations while using another app.  - Background location - Navigation apps continue to guide users who are using other apps - Push notifications - Receive alerts from your remote servers even when your app isn't running. - Local notifications - Applications can alert users of scheduled events and alarms in the background. - Task finishing - If your app is in mid-task, the app can now keep running to finish the task. - Fast app switching - Quick restore between swapping applications

Recall we said Apple was making incremental tweaks to multitasking. IOS 5 added 2 additons to this list.

• Newsstand apps that are downloading new content.
• Apps that receive continuous updates from an external accessory in the Background.

A New notification was added to the acessory protocol that allowed an accessory to launch an application when it had updated data.

Considering now are heart Monitor example.

The user launches HealthMonitor which opens an NSStream to the  heart monitoring accessory and begins a brisk Juant. While running she switches to the Pandora Application which causes the HealthMonitor app to enter into a background state. Because HealthMonitor has indicated in its plist that it is one of the Apple's blessed background types with the key "external-accessory" the application continues to collect the data stream from the accessory. The user is free to switch back and forth between the apps , forcing each in turn to swap states.

To Make it interesting lets add a third application, HealthShare, which logs the users heart rate and makes it available to facebook. HealthShare also accesses the accessory. The heart monitor accessory can only handle a single serial connection at a time. So when the HealthShare accessory is started it forces HealthMonitor into the background, HealthMonitor is still connected to the accessory but get an NSStreamEcounteredEnd event because HealthShare now has the connection to the stream. This handing off of streams can be repeated as needed.

The new lifecycle is an improvement over the previous but still has flaws, for instance the user must physically bring each application back to the foreground state. Apple did provide Accessories with the abililty to send a launch notification, but the programming for most accessories is embedded into firmware, and in most cases needs to physically flashed with new programming. For some OEMs this can be a simple process of providing the user with an over the air update but for others it can involve a tedius trip by the customer to a dealer or actually having to wait for a new model, which can be several months or even years down the road.

Having to manually switch applications can be especially troublesome for In-Car infotainment or management systems because of the implications of having to physically handle the device causing a driver distraction. This would be a problem with many a state legislature.

In the next segment we will discuss a way of switching between applications without the user having to intervene.

 

IOS multitasking and external accessories part II

In this segment continuing our discussion on IOS external accessories and Multitasking we are going to look at ways an application can be launched without any action on the part of the user.

Case Study:

In-car navigation and management systems , so called infotainment systems. These are Radio head units that may offer depending on the OEM Handsfree calling, messaging , navigation and control over the vehicle's environmental systems. Examples would be SYNC from Microsoft and Ford, Mylink from GM and Entune from Toyota.

All of the above named systems and many others also provide some level of integration between the Radio Headunit and Applications on the Driver's Mobile device Using simple steering wheel or voice controls.  Pandora Radio is a widely included application, But the list is growing.

These infotainment systems have special challenges associated with them in that many State Legislature and NTSB are developing strict regulations for Driver Distraction, Mandating and in some States enforcing use of handsfree devices for making calls and limiting certain control of the equipment to when the vehicle is at rest or a safe speed. No one whats to see an accident report that states the driver hit the pedestrial while their attention was focused on Searching for "lost control" by Unwritten Law.

These systems work with a few smart phones e.g. Blackberry, Android and Iphone. Android is by far the easiest to interface with Android devices can connect to these systems via bluetooth socket connections. Since Android allows for true multitasking, multiple socket connections can be opened it is fairly simple for the infotainment system to provide a mechanism to swap from application to application.

Apple presents a more difficult challenge. Apple didn't add Bluetooth Support to the MFI certification until after many of the current crop of devices were on the market which means that for these devices the only connection available is using the I(Pod,Phone,Pad) accessory protocol or IAP over the dock Connector which is a serial connection. 

IAP is a very simple control protocol which uses standard serial RS232 . With the Approved apple api the only way you can do serial communication between an iphone and another device is using an MFI approved appliance. There are a few backdoor approaches to serial communication involving copying portions of the IOKit api from mac OS or jailbreaking the device and installing a serial port handler but for an application that the developer hopes to get through the Apple Review process the only approved way is to use the Apple Dock Connector and belong to the MFI program.

All MFI accessories include a decoder chip, the dock connector also has a decoder chip, when the IDevice is connected to the accessory the devices go through something resembling an ssl handshake, After successful authentication The TTY port is available for standard communication. The IAP framework handles all the low level conversions , the user's application uses the standard IOS stream protocol for all traffic. 

The whole process is fairly fluid but there is a major flaw, only one accessory can be connected at a time and because of the lack of multitasking only one app can be connected to the accessory.

With the latest ios apple has introduced Airplay and Bluetooth 4.0 , Bluetooth 4.0 is a recent standard for new low powered bluetooth devices now coming on the market. It is clear that apple plans Airplay to be the new direction for all future accessories.

Apple announces new MFI components

This is an exciting new direction but software is a lot easier to change than hardware, and new chips mean new engineering and with regards to Automotive product cycles can mean months or years before the public sees it. Meantime what about current hardware.

With IOS 5 Apple has tried to address this issue by providing a new notification to the Accessory. A current generation accessory can now send an IAP command to the device to alert the user to Launch a particular app. "Health Monitor needs your attention" Allow or Dismiss. For the heart monitor example the user simply taps "allow" which starts the application bringing it to the foreground. This is a great improvement it means the User doesn't have to search for the app icon on the phone or remember which app was used with the accessory, But still presents problems for a driver who must take their eyes off the road to tap the allow button. Better but not useful just yet.

I have observed a  regulatory test of an infotainment system . The test consisted of the smart phone being paired with the system and then the device placed in the cars glove compartment. From  that point on the only way to interact with the apps on the phone was via the handsfree microphone or steering column buttons. 

Apple also included new commands to send meta data to existing MFI accessories. eg. Artist, title etc. Nice touches but still for existing accessories it means that the user has to select a favorite app ie Pandora Radio and stick with it until they can safely (or unsafely) switch to another application. 

But.. A method has existed since IOS 2 for launching applications without using the application springboard. That method is the custom URL Scheme. Apple uses custom url schemes for many standard applications. If you open Safari and type mail://johnsmith@apple.com, Safari will close (or nowadays go into background) and the mail application will open. You can designate custom url schemes for any native applications including the phone application and facetime or define schemes for your own applications.

To communicate with an app using a custom URL, create an NSURL object with some properly formatted content and pass that object to the openURL: method of the shared UIApplication object. The openURL: method launches the app that registered to receive URLs of that type and passes it the URL. At that point, control passes to the new app.

NSURL *myURL = [NSURL URLWithString:@"Pandora://play?Sweet+Dreams&artist=sucker+punch"] [[UIApplication sharedApplication] openURL:myURL];

The above code snippet assumes the Pandora app developer has coded a url handler which would process the search query specified on the url.

To create an application with a custom URL scheme

1.	Right-click the Information Property List key, and select Add Row. Select “URL types” from the list .  Adding a URL type.
2.	Expand Item 1, right-click URL identifier, and again select Add Row. Select URL Schemes from the list
3.	Select Item 1, and set the value to yourapp .

An app that has its own custom URL scheme must be able to handle URLs passed to it. All URLs are passed to your app delegate, either at launch time or while your app is running or in the background. To handle incoming URLs, your delegate should implement the following methods:

• Use the application:didFinishLaunchingWithOptions: method to retrieve information about the URL and decide whether you want to open it. This method is called only when your app is launched.

• In iOS 4.2 and later, use the application:openURL:sourceApplication:annotation: method to open the file.

• In iOS 4.1 and earlier, use the application:handleOpenURL: method to open the file.

If your app is not running when a URL request arrives, it is launched and moved to the foreground so that it can open the URL.

if your app is running but is in the background or suspended when a URL request arrives, it is moved to the foreground to open the URL.

The system calls the delegate’s application:openURL:sourceApplication:annotation: to check the URL and open it. If your delegate does not implement this method (or the current system version is iOS 4.1 or earlier), the system calls your delegate’s application:handleOpenURL:

 

Using the information above it is a fairly trivial task to construct a simple launcher with a button representing each application providing a function and the IBOutlet of the button having the code to openURL to the application.

-(IBAction)openSms {

[[UIApplication sharedApplication] openURL:[NSURL URLWithString:@"sms://466453"]];

 }

 

Its also not very difficult to see how you can code a simple voice command application using Siri or Nuance.

- (void)recognizer:(SKRecognizer *)recognizer didFinishWithResults:(SKRecognition *)results

{    

if ([results.results count] > 0) {

NSString *wellFormedRequest =[self  getUrlRequest:[results firstResult]];

if (wellFormedRequest != nil) {

  NSURL *url = [NSURL URLWithString:wellFormedRequest];

  [[UIApplication sharedApplication] openURL:url];

}else{

[self speakText:MESSAGE_I_DID_NOT_UNDERSTAND_Y0U];

}else {

        [self speakText:MESSAGE_NO_RESULTS_RETURNED];

}

}

In the code snippet above for a typical speech recognizer the method didFinishwithResults passes the results of the speech transcription to getUrlRequest which returns either a well formed url request in the form of appName?parameters or nil if the request couldn't be mapped to an application. The Url request calls openUrl which opens the application and puts the launcher into the background. 

If getUrlRequest returned nil or [results count] = 0 then an appropriate message is passed to the text to speech engine.

 So now we have a fairly decent little voice controlled launcher that can be used by any handsfree device not just MFI external accessories. Its easy to imagine one of those small bluetooth headsets e.g. Jambra used as a voice interface to your applications. This is quite possible using the new WB 16KHZ SCO support in IOS 5. 

So now with a voice command we can start an SMS message, listen to Pandora Radio or get the latest Podcast from Stitcher. Our launcher has all the compatible phone apps hardcoded into a list of choices.

But.. What is the application isn't available on the device. It would be nice to find all the Applications that are present on the phone and compatible with the accessory.

At First thought you might consider as each application is started and backgrounded it writes its Id to ICloud after all isn't that what the IOS 5 "key value storage" feature is for , storing small bits of data usually for restoring the state of the app. Such as the page the reader is on in an ebook reader application.

But Icloud requires an internet connection, and it would be ideal if the launcher had as few dependencies as needed.

The next consideration might be, when an application starts it could register with the accessory , and the accessory could keep a list of active applications but this may require special coding on the accessory . Accessory code is usually embedded into firmware that must be flashed to add new features, For some Accessories this may only require an Over the Air update but for others as previously mentioned this could be a complex and time expensive task. And we wanted the launcher to function with non-MFI devices like the Jambra Headsets mentioned .

A somewhat simpler approach is to use a little bit of standard c code to provide an enumeration of all the running processes on the Device. Much like that pictured below in The AirTracer Processes View. A shameless plug for Airtracer which can List all the processes on an IDevice and there related PID and other information. AirTracer Pro can also terminate processes as well.

Once we have the list of Processes we can build a list of voice commands by comparing the process list against the hardcoded list of compatible processes. 

A fragment from a typical siri voice tree using the above technique might go as follows.

User: Sir please play "Sweet Dreams" by Eurythmics

Siri: Ok , I found 2 applications on your device that can play the original mix of Sweet Dreams", Spotify or Tunes , which app would you like to you.

If the user replies Tunes  Siri will launch etunes  and play the the requested single, if the user fails to respond either Spotify or Tunes may be launched depending on which application the user has selected as a default.

 

IOS multitasking and external accessories part III - HealthMonitor First Cut

So now using All the information presented so far we can turn to our hypothetical Marathon Runner trainee. She has the Pulse monitor band wrapped around her wrist and the Jambra headset mic over her right earlobe. Using the headset she can switch between Music applications on her phone as well as call up spoken statistics from the HeartMonitor application. She can  also Post her progress to her Facebook Wall using the HealthShare application.

The Heart Monitor accessory allows for a single serial radio connection.

When she requests the Heart Monitor application using a voice command, HealthMonitor starts if it's not running already and connects to the Accessory. The launcher enters background and then is suspended because it's not one of the apps "blessed" to run in the background. On receiving AccessoryDidConnect HealthMonitor issues an OpenUrl request for the Launcher. HealthMonitor moves to the background but because it is a "background allowed" application it stays connected to the accessory and can still alert the runner of changes in Pulse Rate via notification Alerts.

The User now issues the "Pandora" voice command to start Pandora Radio, once again the Launcher moves to the background and is suspended. HealthMonitor continues to receive updates from the Accessory.

Pandora now is in complete control, until the runner exits it. after which Pandora enters applicationDidEnterBackground and issues an openUrl back to the launcher. Now the Launcher is the foreground application, HealthMonitor continues to run in the background.

She can continue this round Robin cycle  between opening applications, bringing to foreground or sending to backgrounds indefinitely. HealthMonitor will continue to receive updates until it becomes disconnected from the heart monitor or another forground app opens a EASession.

HealthMonitor talks to the heart monitor accessory using the ubiquitous NSStream that apple uses for virtually all stream connection. The same protocol can be used for tcp, bluetooth and IAP.

The runner now issues  a voice command to start HealthShare, as before the launcher enters the background and shortly after a suspended state. HealthShare does open a session to the accessory. When HealthShare opens a stream to the heart Monitor  HealthMonitor receives a NSStreamEndEncountered message , Since HealthMonitor no longer has an active session to the accessory it will eventually be suspended until such time it is brought again to the foreground. HealthShare issues an openURL request to the Launcher and enters the background. It is still receiving updates from the accessory.

Reality check : There doesn't seem to be any means to start an application in the background it must be "pushed" there by a foreground application, so apps must be specifically coded to return to the launcher with the openUrl method. Unfortunately there is no voice controled version of Pandora, the app mentioned here is simply a hypothetical version. 

IOS multitasking and external accessories part IV - Interapp communication

So now we have a fairly useful multi-application usage scheme. The runner can use her voice to start and swap between applications. She can also connect or disconnect applications to the heart monitor that perform different functions.

But there are definite refinements that can be made surrounding HealthMonitor and HealthShare. Since the monitor accessory only allows a single connection only one of the applications can be getting updates from the monitor. HealthShare in particular would benefit from a persistant connection because as implemented it can only deliver a log of resperatory activity over a particular time slice.

Also any applications which like HealthShare collect statistics from the monitor need to be certified by the MFI program, leaving potential but small independent developers out in the code. It would be far more efficient and benificial if the HealthShare application could just get the pulse updates from HealthMonitor then directly from the accessory, but in a way that did not require changes to the firmware to add newer Apple protocols. It turns out such a mechanism exits and we will examine that now. 

This mechanism is the Pasteboard which was introduced by Apple to provide cut and paste between and within applications. The Pasteboard is often thought to be like simple clipboard but this is not an entirely true anology.

There are two types of pasteboards that an application can interact with. The first is the general, systemwide pasteboard. This pasteboard can be accessed by all applications on the system and is the default pasteboard that is used by UITextView, UITextField, and so on.

The second type is a named pasteboard. These pasteboards are created by your application and given a name. Other applications that know the name of your pasteboard can also access that pasteboard and retrieve your data. In this way, you can share data amongst a select set of applications if you need to. Local notifications can be set up between applications. System pasteboards are persistent by default named pasteboards can be made persistent by setting the persistence property.

The UIPasteboard class provides the mechanism by which applications interact at a low level with both the general system pasteboard and specific named pasteboards that they themselves create.

To access the general system pasteboard, you call the method generalPasteboard on the UIPasteboard class. This returns the general pasteboard singleton.

To access a named pasteboard, you call the method pasteboardWithName:create: on the UIPasteboard class. If you would like the UIPasteboard class to create a unique name for you, then you can pass nil as the first parameter. The second parameter indicates whether or not you want the UIPasteboard class to create the pasteboard for you if it does not exist.

  You can get a reference to the pasteboard as follows:

UIPasteboard *pasteboard = [UIPasteboard generalPasteboard];

 You can then place a string on the pasteboard by using this:

pasteboard.string = @"here is some data";

 After launching your app via a call to openURL:, you can then retrieve the string from the pasteboard by writing this:

UIPasteboard *pasteboard = [UIPasteboard generalPasteboard];

NSString *myString = pasteboard.string;

 The UIPasteboard class contains convenience properties like this for NSString, UIImage, NSURL, andUIColor.

 

Pasteboards are implemented as an array of items. Each item may have multiple representations, and is represented by a dictionary. For instance, a URL can have a number of representations:

• A text representation (kUTTypeUTF8PlainText) for text editors
• A URL representation (kUTTypeURL) for tools that accept URLs
• An image representation (kUTTypePNG or kUTTypeJPEG) for tools that accept images

To retrieve a pasteboard’s items, invoke items. To add an item, create the appropriate dictionary and add it to the pasteboard with addItems:.

NSDictionary* item

  = [NSDictionary dictionaryWithObjectsAndKeys: string, kUTTypeUTF8PlainText,

                                                   url, kUTTypeURL,

                                                   png, kUTTypePNG, nil];

[pasteboard addItems:[NSArray

arrayWithObject:item]];

Standard UTI types are used to access each item but you can also define custom keys if needed.

Due to a limitation of the iPhoneOS, the size of each item on the pasteboard cannot exceed 5~8 MB. If an item is greater than 5120 kb it should be broken into smaller items of less than 5120 kb.

Using Pasteboards it is not hard to imagine that you could come up with a protocol for sharing information between applications. A company called INTUA markets an application named BeatMaker that uses such a custom sharing protocol to share audio data between BeatMaker compatible applications. They have opened source the protocol under MIT license.

Overview INTUA audioCopy

BeatMaker uses this UTI to achieve copy and paste operations of uncompressed audio data with other applications. 

Items with type kUTTypeAudio contains audio data in a specific sample format.

BeatMaker assembles all contiguous items of type kUTTypeAudio, into one audio file.    

Each item is added to the pasteboard and made available to other applications.    

INTUA audioCopy only works for pcm audio but its not much of a stretch to devise a general protocol for interapplication sharing. Mooncatventures is in the process of developing such a General Interapp Protocol (GIP) and we will provide more details soon.

IOS multitasking and external accessories part V - Endgame

So now we come to the end of this series, the final cut if you will ,we hoped you found it both educational and enjoyable.

So looking again at our Marathon runner traniee. She has the pulse rate monitor band wrapped around her left wrist, and the jambra headset behind her right ear. The heart monitor "talks" to the applications on her phone using the same ANT+ protocol used by the resident Nike application. The headset allows her to control a springboard application on her iphone using Siri (or perhaps Nuance) over the new WB band 16 khz SCO channel.

As Before she has several audio applications including her favorite Radio application.  The device also has the HealthMonitor application and a hypothetical version of HealthShare using a GIP protocol.

As Before she starts HealthMonitor with a voice command, HealthMonitor connects to the accessory then does an openURL transfer back to the launcher.

She now starts Pandora which becomes the forground application. When she is done with Pandora she exits and returns to the launcher.

She now issues the voice command to start HealthShare which sends the launcher to the background.

HealthMonitor continues to send updates from the background, whenever it has an update for HealthShare it retrieves HealthShares "SharedMessageQueue" (a uipasteboard) and encodes the update as a GIP message. Messages greater then 2048 bytes are broken into smaller messages, each message contains a protocol frame header. Multiple messages are in the form of a circular linked-list and have a message id to indicate the message the frame belongs too. Again each frame is no more than 2048 bytes.

After adding the message to the sharedMessageQueue HealthMonitor sends a notification to HealthShare that it has an update. Remember this protocol is in early development, perhaps it may be better from a message traffic prospective if the HealthShare just had a background task that watched the Uipasteboard named sharedMessageQueue every 100 ms or so. In any case, HealthShare gets the update and sends it to facebook. It may also update the runners location to facebook checkin.

The one drawback here is that when the runner swaps  HealthShare for another application, HealthShare will become suspended, A workaround may be to have healthShare receive periodic location changes which allows it to join the list of Blessed background applications.

So ends our saga, this seem to be a fairly nice mechanism for interapp communication. Undoubtedly there are holes that may be need to be filled, but all in all theoretically it should work.

The technique is also quite cross-platform compatible and may be duplicated on android using Intents.