Building an Apple Media Service Client with the Moddable SDK

iOS devices from Apple provide BLE services for accessories to enhance user experience and interactions. The Apple Media Service (AMS) allows accessories to interact with the iOS media player. We thought it would be fun to build an ESP32 color touch screen interactive media player accessory to control the iPhone media player. We built it all in JavaScript using the Moddable SDK. This blog post describes the application architecture and challenges we faced building the BLE accessory on a constrained microcontroller device. You will learn how we used BLE, HTTPS, JSON, JPEG, and our Commodetto graphics library to implement the app.

Apple Media Service 101

The Apple Media Service provides BLE accessories access to the iOS media player. A device can issue commands to control playback, e.g. play, pause, next track and previous track. The service provides notifications to accessories of changes to the current track name, album, artist, and duration. Accessories may change the media volume and playback queue shuffle mode.

As with other BLE services, clients interact with the service by reading and writing the service characteristics. The Apple Media Service provides three characteristics:

Remote Command -- to control playback state, e.g. play, pause, next track, etc. Our BLE client writes commands to this characteristic.
Entity Update -- to notify the BLE client when media attributes change. Our application's BLE client registers for Entity Update notifications corresponding to track title, artist, album, and duration. When the iOS media player changes tracks, our application receives notifications and updates the display accordingly.
Entity Attribute -- allows a BLE client to retrieve extended information corresponding to a media attribute. This characteristic is typically only used when an attribute value is delivered truncated due to MTU size limitations. For simplicity, our BLE client does not support this characteristic.

Our implementation is comprised of two JavaScript modules. The ams-client module implements a generic Apple Media Service BLE client class AMSClient. The client connects to the iOS service, performs service discovery, and registers for media attribute change notifications. The module exports constants corresponding to the various AMS ID and flag values.

To streamline AMS application development, the AMSClient class provides two functions for application clients to override:

The onTrackChanged function is called when the currently playing media track changes.

onTrackChanged(artist, album, title, duration)

The artist, album, and title parameters are strings and the duration is a floating point number corresponding to the track duration in seconds.

The onPlaybackInfoChanged function is called when the track playback state changes, e.g. when the playback is paused or the time is changed.

onPlaybackInfoChanged(state, rate, elapsed)

The state corresponds to the current playback state defined by the PlaybackState object:

const PlaybackState = {
    Paused: 0,
    Playing: 1,
    Rewinding: 2,
    FastForwarding: 3,
}

The rate parameter is the playback rate as a floating point number. The elapsed parameter is the elapsed time of the current track in seconds as a floating point number.

The ios-media-sync module extends the AMSClient class to display and interact with the currently playing iOS media. It implements onTrackChanged and onPlaybackInfoChanged to render the media player user interface using the Commodetto graphics library on a display connected to the ESP32.

Connecting to the iOS Device

For an accessory to access any BLE service on an iOS device, it must first pair with the iOS device over an encrypted connection. Pairing is initiated by the iOS device. Our accessory advertises as a BLE server/peripheral interested in connecting with iOS devices that support the Apple Media Service. Our accessory does this using the Service Solicitation advertising type, as described in this Bluetooth blog article.

The AMSAuthenticator class in the ams-client module implements the required advertising, connection, and pairing flow. While the requirements seem complicated, the amount of code required is small:

const AMS_UUID = uuid`89D3502B-0F36-433A-8EF4-C502AD55F8DC`;

class AMSAuthenticator extends BLEServer {
    constructor(client) {
        super();
        this.client = client;
    }
    onConnected(device) {
        this.device = device;
        this.stopAdvertising();
    }
    onAuthenticated() {
        this.client.onAuthenticated(this.device);
    }
    onDisconnected() {
        this.startAdvertising({
            advertisingData: {flags: 6, completeName: this.deviceName, solicitationUUID128List: [AMS_UUID]}
        });
    }
}

When the BLE stack is ready, the AMSAuthenticator uses the securityParameters property accessor function to request an encrypted connection with pairing and Man-in-the-Middle (MITM) protection. Encryption is enabled by default when using this accessor function. The AMSAuthenticator then advertises as a connectable device named Moddable, soliciting the AMS 128-bit service UUID. After our device starts advertising, the iOS device can connect to and pair with our accessory.

Once the iOS device successfully pairs with our accessory, our onAuthenticated callback is invoked. It switches the ESP32 from server to client and connects to the iOS device as an AMS client.

Because the AMSAuthenticator handles the details of establishing an encrypted connection, the Apple Media Service client application class only needs to instantiate the Poco renderer, kick off the authentication process, and instantiate the media player client:

class AppleMediaServiceClient {
    constructor(render) {
        this.render = render;
    }
    start() {
        this.server = new AMSAuthenticator(this);
    }
    onAuthenticated(device) {
        this.client = new AMSPlayerClient(this.render, device);
    }
}

let render = new Poco(screen);
let client = new AppleMediaServiceClient(render);
client.start();

ESP32 Media Player

The AMSPlayerClient class implements the media player client. The media player displays the media attributes (title, artist, album) and elapsed time with a progress bar. The touch media control buttons support play/pause, next track and previous track. The ESP32 player client is a remote control for the iOS media player.

All the coding techniques used to render the various on-screen media player UI elements can be found in our Commodetto docs, text-ticker, progress and mini-drag example apps. All the bitmap assets used are RLE-encoded 4-bit gray bitmap masks to save space. The bitmaps are tinted to the desired color at runtime.

Very cool... but what about the album art?

Great question! The Apple Media Service does not provide album cover art as it is impractical to transfer images over BLE. Fortunately, the iTunes Search API can be used to fetch the album cover art over Wi-Fi.

iTunes Search API

The iTunes Search API is built for web developers to include search fields on their web sites for content on the iTunes and Apple Books (iBooks) stores. Our media player uses the API to search for album cover art whenever the track changes. The Search API returns a JSON result set that includes album cover art URIs. We use the Request object with a SecureSocket to retrieve the JSON (limited to 4 results) over HTTPS:

onTrackChanged(artist, album, title, duration) {
    ...
    let term = encodeURIComponent(album.replace(/ /g, "+"));
    let path = `/search?media=music&amp;entity=album&amp;attribute=albumTerm&amp;term=${term}&amp;limit=4`;
    this.fetchAlbumURI(path);
}

fetchAlbumURI(path) {
    this.request = new Request({
        host: "itunes.apple.com",
        path, response: String,
        port: 443, Socket: SecureSocket, secure: {protocolVersion: 0x303}
    });
    this.request.callback = this.fetchAlbumURICallback.bind(this);
}

Unfortunately the JSON returned includes dozens of properties that we don't need:

// Request:
https://itunes.apple.com/search?media=music&amp;entity=album&amp;attribute=albumTerm&amp;term=Computer%2BWorld&amp;limit=4

// Abbreviated JSON single result response:
{
    "resultCount": 4,
    "results": [
        {
            "artistViewUrl": "https://itunes.apple.com/us/artist/kraftwerk/553899?uo=4",
            "releaseDate": "1981-05-10T07:00:00Z",
            "collectionType": "Album",
            "collectionName": "Computer World (Remastered)",
            "amgArtistId": 4706,
            "copyright": "\u2117 1987 Elektra Records. Marketed by Rhino Entertainment Company, a Warner Music Group Company",
            "collectionId": 830083942,
            "artworkUrl60": "https://is2-ssl.mzstatic.com/image/thumb/Music7/v4/14/1c/69/141c691f-e0bf-0a5f-6f49-092d59837279/source/60x60bb.jpg",
            "wrapperType": "collection",
            "collectionViewUrl": "https://itunes.apple.com/us/album/computer-world-remastered/830083942?uo=4",
            "artistId": 553899,
            "collectionCensoredName": "Computer World (Remastered)",
            "artworkUrl100": "https://is2-ssl.mzstatic.com/image/thumb/Music7/v4/14/1c/69/141c691f-e0bf-0a5f-6f49-092d59837279/source/100x100bb.jpg",
            "trackCount": 7,
            "currency": "USD",
            "artistName": "Kraftwerk",
            "country": "USA",
            "primaryGenreName": "Electronic",
            "collectionExplicitness": "notExplicit",
            "collectionPrice": 8.99
        },
        ...
    ]
}

Parsing all those unnecessary properties into a JavaScript object requires additional memory that we don't have available. On a ESP-WROOM-32 with 520 KB RAM, only roughly 80 KB RAM is available to our application after the FreeRTOS, Espressif BLE server/client stacks and Moddable SDK runtime are loaded. That limited RAM needs to accommodate BLE and secure HTTP requests, graphics rendering, UI timers, and touch screen handling. Because applications frequently only need to parse a handful of JSON keys, the XS JavaScript engine extends the JSON.parse() function to allow callers to specify which keys to parse. Our media application only needs four of the JSON key/value pairs. We pass the required keys as a second Array parameter to JSON.parse:

fetchAlbumURICallback(message, value, etc) {
    ...
    let entries = JSON.parse(value, ["resultCount","results","collectionName","artworkUrl100"]);
    let result = entries.results.find(entry => {
        return (entry.collectionName.startsWith(this.album) && ("artworkUrl100" in entry))
    });
}

Note that the Array parameter is different from the optional JSON.parse() reviver function parameter.

By limiting the number of key/value pairs parsed, we significantly reduce the amount of RAM required to contain the properties required to identify the album art URIs.

Album Art Fetch & Display

The iTunes Search API JSON results include JPEG album art URIs corresponding to 60 x 60 and 100 x 100 square pixel images. These image sizes are too small for our 240 x 320 display. We found Ben Dodson's itunes-artwork-finder project fetches album art images at a variety of sizes not included in the search results JSON. Following the same idea, we are able to request 110 x 110 album art images over HTTPS by modifying the 100 x 100 album art artworkUrl100 property provided in the JSON search results:

const ART_WIDTH = 110;
const ART_HEIGHT = 110;

let url = result.artworkUrl100.replace(/100x100/, `${ART_WIDTH}x${ART_HEIGHT}`);
this.fetchAlbumArt(url);

fetchAlbumArt(url) {
    let end = url.indexOf("/", 8);
    let host = url.slice(8, end);
    let path = url.slice(end);
    this.request = new Request({
        host, path,
        response: ArrayBuffer,
        port: 443, Socket: SecureSocket, secure: {protocolVersion: 0x303}
    });
    this.request.callback = this.fetchAlbumArtCallback.bind(this);
}

The resulting JPEG encoded image is delivered to the fetchAlbumArtCallback in an ArrayBuffer and decoded/rendered block-by-block to the display to conserve memory:

fetchAlbumArtCallback(message, value, etc) {
    if (5 == message) {
        this.drawAlbumArt(value);
    }
}

drawAlbumArt(imageData) {
    let render = this.render;
    let decoder = new JPEG(imageData);
    let x = LAYOUT.art.x;
    let y = LAYOUT.art.y;
    let block;
    while (block = decoder.read()) {
        render.begin(block.x+x, block.y+y, block.width, block.height);
        render.drawBitmap(block, block.x+x, block.y+y);
        render.end();
    }
}

Note: Though it isn't used here, Commodetto supports streaming JPEG decode to reduce the RAM required to display a JPEG image, since the entire image isn't required to be completely downloaded into RAM before decoding. Our jpegstream example app shows how this works.

Conclusion

Our Apple Media Service client app demonstrates how to cleanly integrate many different technologies, all available in the Moddable SDK with efficient and modular JavaScript APIs. The ams-client module subclasses the BLEClient and BLEServer classes to establish a secure BLE connection to the iOS device and read/write service characteristics. The ios-media-sync application uses the Moddable SDK networking modules to fetch JPEG album artwork, using the iTunes Search API, over a secure HTTP connection and decode the image using our JPEG module. The interactive color touch screen UI is implemented with the Commodetto graphics library. All of these technologies are running simultaneously on a resource-constrained microcontroller.

Additional Resources

The Moddable SDK includes many BLE client and server example apps for exploring BLE development in JavaScript on the ESP32. If you are interested in working with other BLE services on iOS, our ios-time-sync example app shows how to set the ESP32 clock by connecting to an iPhone and reading the current time over BLE.