Web Services Series Part 3: Building Custom Modules to Replace JavaScript Libraries for the Web

There are countless JavaScript libraries available for the web. Since the XS JavaScript engine in the Moddable SDK implements the same JavaScript language used in web browsers, it seems reasonable to assume these libraries may also be used on embedded devices. We often get the question, "can I use [insert name of web JavaScript library here] with the Moddable SDK?"

The short answer is: probably not, at least not without modifications.

This article explains why it is not always practical or possible to use JavaScript libraries designed for the web in embedded applications built with the Moddable SDK. It then walks through how we built the PubNub module for the Moddable SDK based on PubNub's REST API spec using PubNub's web JavaScript SDK as reference.

Why don't JavaScript web libraries work?

The two most common issues that arise when trying to use libraries designed for the web on embedded devices are:

memory constraints and
host object differences.

Memory constraints

Embedded JavaScript applications have to be designed with memory constraints in mind. We often run applications on the ESP8266, which has just 45 KB of free RAM and 1 MB of available flash storage for code, and the Thunderboard Sense, which has just 32 KB of RAM and 256 KB of flash. The difference between the amount of memory available on these devices and the amount available to browsers is measured in orders of magnitude.

There are many JavaScript libraries that are too large to get even close to fitting on these microcontrollers. Smaller libraries may fit, but be impractical to use because of their RAM requirements. An example of this is the excellent QR Code Generator by Kazuhiko Arase mentioned in our QR Code blog post.

Host object differences

The XS JavaScript engine follows the ECMAScript 2018 standard, the specification that defines the JavaScript language. It defines the built-in JavaScript objects (String, Number, Date, etc.). The built-in objects are not enough to build most applications, so host environments provide additional objects -- host objects -- that are not part of the ECMAScript standard.

For web applications, the host is the browser. Many objects widely used by web libraries, including the DOM interfaces for accessing and changing elements of HTML documents and Fetch API for making HTTP requests, are host objects provided by browsers.

When JavaScript runs outside of the browser, the available host objects are different. The Moddable SDK is designed to run on microcontrollers, where it is impractical to provide the same host objects as a web browser. Consequently, libraries built for the web that rely on host objects provided by browsers need modification to work with the Moddable SDK.

Building custom modules

Often the problem is a mixture of both issues described above. It is difficult to sift through each line of code to identify and resolve compatibility issues. It can be much easier to create your own module from scratch based on the framework's API documentation, as we did with the PubNub module for the Moddable SDK. An important benefit of this approach is that the module built for use on embedded will typically be smaller because it is focused on providing the needed the functionality and no more.

PubNub

PubNub is a global data streaming network. Their APIs allow for cross-platform, cross-device communication over this network with a simple publish/subscribe messaging system.

The PubNub web JavaScript SDK consists of several thousand lines of code. This is no problem for use in web applications, but it is not practical for embedded applications running on constrained devices. It also uses XMLHTTPRequest objects to make HTTP requests. Again, no problem for use in browsers, but XMLHttpRequest is not part of the Moddable SDK.

The PubNub module for the Moddable SDK implements a subset of the features of the web JavaScript SDK -- the publish, subscribe, and addListener methods. It is only 235 lines of code so there is certainly room for additional functionality should your application need it. Web libraries are generally meant to be all-purpose libraries that make every feature of a service available, even ones that are rarely used. Minimizing code size is a high priority when writing embedded applications, so it is generally a good idea to remove unnecessary functions.

Module Design

The module consists of two classes:

The PubNub class, which is based on the PubNub web JavaScript SDK
The REST_API class, which follows the PubNub REST API spec

Applications create an instance of the PubNub class and call the created instance's functions. Functions of the PubNub class call functions of the REST_API class.

We designed this module to emulate the PubNub JavaScript API so the experience is the same for a developer writing applications with the Moddable SDK as it is for a developer writing applications for the web. For example, here's part of the Publish section of the web JavaScript API reference:

publish(
    {Object message, String channel, Boolean storeInHistory, Boolean sendByPost, Object meta, Number ttl},
    Function callback
)

The publish function of the PubNub class in the Moddable SDK module takes the same arguments, although only a subset of the first argument's properties supported by the PubNub web JavaScript library are supported by our version. While the developer API offered by the PubNub web library and our embedded version are about the same, the two modules have completely different implementations.

Here is the source code for the publish function from our PubNub module:

export class PubNub {
    ...
    publish(params, callback, scope) {
        let channel = params.channel;
        let message = params.message;
        REST_API.publish(this.config, { channel, message, store:0 }, (error, data) => {
            if (error) trace("Publish Error: " + error + "\n");
            if (callback) {
                if (scope) callback.bind(scope);
                callback(error, data);
            }
        }, this);
    }
    ...
}

The corresponding publish function of the REST_API class does some additional processing and calls makeRequest. The makeRequest function is called by multiple functions in the REST_API class to avoid duplicating code to create an HTTP request object.

export class REST_API {
    ...
    static publish(config, params, callback, scope) {
        params = Object.assign({}, config, params);
        let query = params.query = {};
        if ("store" in params) {
            query.store = params.store;
        }
        if ("auth" in params) {
            query.auth = params.auth;
            debugger; // auth not supported
        }
        let channel = encodeURIComponent(params.channel);
        params.location = `/publish/${params.pub_key}/${params.sub_key}/0/${channel}/0`;
        return makeRequest(params, callback, scope);
    }
    ...
}

For more information about working with REST APIs, see the first article of this series.

PubNub example

The pubnub example shows how to use the PubNub module in an application to subscribe to a PubNub channel and publish a single message to it.. This example is small enough that it runs comfortably on the ESP8266.

Conclusion

JavaScript libraries for the web are designed with different goals than JavaScript modules for embedded devices. By writing your own modules, you can achieve the same interactions with the same web services while keeping code size and RAM usage small. Using these modules in applications built with the Moddable SDK allows the familiar experience of working with web services to become practical within the constraints of embedded devices.thingicon.png