How We Test XS and the Moddable SDK

Making sure that the Moddable SDK and the XS JavaScript engine run correctly and reliably is a big project. There are hundreds of features and APIs that must be verified on a wide variety of devices. Testing on devices is particularly challenging, as many don't even have enough storage space to hold the test suite.

Moddable has developed a technique to test both our JavaScript engine and our APIs that works on many embedded devices and our desktop simulator platforms. Just this month we are making our tests and test tools available so everyone can run these tests. This blog post is an overview of Moddable's testing effort, explaining our goals and introducing how we test on embedded device targets. A separate document provides step-by-step instructions on how to run the tests.

Testing Goals

The Moddable SDK tests are an important tool to ensure that the Moddable SDK is ready for developers creating commercial products or personal projects. Here are some of the goals of the testing effort.

Works as intended. Ensuring proper behavior of each module and API is the most fundamental purpose of testing.
Keeps working. Changes to modules and XS have the potential to break code that was working. Thorough, regular testing helps to ensure that improvements in one area don't unintentionally break another.
Consistent results. A key goal of the Moddable SDK is ensuring that scripts written for one device can run unchanged on other similar devices. Running the same tests on each device helps ensure consistent results, even when the underlying implementation of the module or API is completely different on each device.
Port baseline. When porting the Moddable SDK to a new device target, the test suites provide thorough tests so that issues are identified as early as possible.
Fails gracefully. As a rule, JavaScript code should not be able to cause a hardware or operating system crash. By carefully testing parameter validation, the APIs are more resilient when presented with invalid input, whether that's caused by a developer's mistake or a malicious script.
No resource leaks. Running a long sequence of tests tends to expose leaks of memory and other host resources. This is especially critical on embedded devices with limited memory that need to run for months or years at a time without rebooting.

Security is another goal of testing. For example, testing is needed to verify that XS and the Moddable SDK prevent scripts from accessing memory and other resources that they weren't given access to. To perform this kind of vulnerability testing, Moddable uses fuzz testing with several different fuzzing engines. See the article Hardening the XS JavaScript engine for more information.

The Tests

There are two separate test suites used to validate the Moddable SDK.

The first is test262, created by TC39, the JavaScript language committee, to guarantee interoperability of scripts across JavaScript engines through rigorous conformance testing. With tens of thousands of test scripts, test262 makes sure that the JavaScript language implemented by XS for the Moddable SDK is the same JavaScript language found on the web.

The second is an extensive suite of test scripts in the Moddable SDK, created by Moddable to achieve the goals described above. These tests cover the most widely used modules in the Moddable SDK and coverage continues to grow.

The Moddable SDK test scripts follow the same conventions as the test262 test scripts, including frontmatter, implementation of asynchronous tests, and the runtime environment for test scripts. Adopting the test262 test script conventions allows Moddable to use the same tools for running test262 and Moddable SDK tests. It also allows developers with experience reading and writing test262 tests to apply their skills to Moddable SDK tests.

The Moddable SDK tests include initial tests for parts of Ecma-419, the ECMAScript Embedded Systems API Specification. As these Ecma-419 tests evolve, it may be appropriate to migrate them to a shared test419 project for use across independent implementations.

The Moddable SDK tests also include tests that exercise areas of the XS JavaScript engine that are not covered by test262. These include issues reported via GitHub and built-in objects and methods either unique to XS or not yet part of test262.

Running Tests

Running the test262 and Moddable SDK tests on the target devices is the biggest challenge. These devices have limited RAM and storage. To lighten the load on the device being tested, Moddable uses the xsbug JavaScript debugger to select the tests to run, download one test at a time to the target device, and collect the test results. The device being tested runs a test app which receives the test scripts from xsbug and runs them in an environment compatible with test262. This allows tests to be run on many devices, but not all. For example, an ESP32 can run the tests but an ESP8266 cannot at this time.

Using xsbug to manage the running of tests is convenient. When porting the Moddable SDK to a new device, JavaScript debugging support is one of the first features ported because it is valuable for debugging the port. Extending xsbug to run tests allows the early bring-up effort to easily run tests against the port.

Conclusion

The Moddable SDK plays a key role in mission critical aspects of consumer and industrial IoT products. It is essential that the Moddable SDK be as robust and reliable as possible to provide a stable foundation for products.

Moddable has invested considerable time and effort to both develop tests and to create a technique to efficiently run them on embedded device targets. These tools are now available for everyone to use to perform their own tests and the tests are open for review and contributions to expand the breadth and depth of test coverage.

Testing large software systems, like the Moddable SDK and its XS JavaScript engine, allows these systems to continue to evolve with new features, optimizations, and ports by minimizing the risk of instabilities and incompatibilities being introduced. By being as open with our test tools and test scripts as with the Moddable SDK itself, Moddable looks forward to engaging with our growing community to make further improvements to how we test XS and the Moddable SDK.