Share

Yosuke Furukawa is a Node.js Core Collaborator and one of the passionate champions of the Japanese Node.js community.

Yosuke recently published a blog post in Japanese regarding the challenges Node.js was facing with considering ES Modules support. As there is a lack of concise information laying out the complex factors involved in making decisions around ES Modules in Node.js, we asked him if we could publish his post in English. We have worked with him to translate and update the content to reflect the current state of events and hope you find this article instructive.

---

ECMAScript 2015 (ES2015, formerly ES6) was published almost a year ago. Node.js v6 supports 93% of the ES2015 syntax and features and most modern browsers exceed 90%. However, no JavaScript runtime currently supports ES Modules. (Note that kangax's compatibility table does not yet have an ES Modules column.)

ECMAScript 2015 defines the ES Modules syntax but ECMAScript does not define a "Loader" specification which determines how Modules are inserted into the runtime. The Loader spec is being defined by WHATWG, but is not yet finalized.

The WHATWG Loader spec needs to define the following items for Milestone 0 on its roadmap:

• Name resolution (relative and absolute URLs and paths)
• Fetch integration
• How to describe script tag: <script type="module">
• Memoization / caching

The Module script tag has been defined, but the other items are still under discussion. You can check the status of this discussion on GitHub. Some browsers have started implementation, but most are waiting for finalization of the Loader spec.

Why does Node.js need ES Modules?

When Node.js came into existence, an ES Modules proposal didn't exist. Node.js decided to use CommonJS Modules. While CommonJS as an organization is no longer an active concern, Node.js and npm have evolved the specification to create a very large JavaScript ecosystem. Browserify and more recently webpack bring Node's version of CommonJS to the browser and solve module problems gracefully. As a result, the Node/npm JavaScript module ecosystem spans both server and client and is growing rapidly.

But how do we deal with interoperability between standard ES Modules and CommonJS-style modules in such a big ecosystem? This question has been debated heavily since the beginning of the ES Modules spec process.

Browserify and webpack currently bridge the gap between browser and server to make JavaScript development easy and somewhat unified. If we lose interoperability, we increase the friction between the existing ecosystem and new standard. If front-end developers choose ES Modules as their preferred default and server-side engineers continue to use Node's CommonJS, the gap will only widen.

An interoperability proposal for Node.js

Bradley Farias (a.k.a Bradley Meck) has written a proposal for interoperability between CommonJS and ES Modules. The proposal is presented in the form of a Node.js EP (Enhancement Proposal) and the pull request generated record amounts of discussion but also helped shape and tune the proposal. The EP was merged but still retains DRAFT status, indicating a preference rather than a clear intention to even implement ES Modules in Node.js. You can read the proposal here: https://github.com/nodejs/node-eps/blob/master/002-es6-modules.md.

Discussion and options explored during the development of this proposal are mostly found throughout the initial pull request comments thread but a partial summary can be found on the Node.js wiki.

The biggest challenge for Node.js is that it doesn't have the luxury of a <script type="module"> tag to tell it whether any given file is in CommonJS format or an ES Module. Unfortunately you can't even be sure in all cases what type of file you have simply by parsing it as the Modules spec presents us with some ambiguities in the distinction. It's clear that we need some signal that Node.js can use to determine whether to load a file as CommonJS (a "Script") or as an ES Module.

Some constraints that were applied in the decision making process include:

• Avoiding a "boilerplate tax" (e.g. "use module")
• Avoiding double-parsing if possible as Modules and Scripts parse differently
• Don't make it too difficult for non-JavaScript tools to make the determination (e.g. build toolchains such as Sprockets or Bash scripts)
• Don't impose a noticeable performance cost on users (e.g. by double-parsing large files)
• No ambiguity
• Preferably self-contained
• Preferably without vestiges in a future where ES Modules may be the most prominent type

Clearly compromise has to be made somewhere to find a path forward as some of these constraints are in conflict when considering the options available.

The route chosen for the Node.js EP, and currently accepted by the Node.js CTC for ES Modules is detection via filename extension, .mjs (alternatives such as .es, .jsm were ruled out for various reasons).

Detection via filename extension provides a simple route to determining the intended contents of a JavaScript file: if a file's extension is .mjs then the file will load as an ES Module, but .js files will be loaded as a Script via CommonJS.

Basic interoperability algorithm

The following algorithm describes how interoperability between ES Modules and CommonJS can be achieved:

1. Determine if file is an ES Module (ES) or CommonJS (CJS)
2. If CJS:
  2.1. Wrap CJS code to bootstrap code
  2.1. Evaluate as Script
  2.2. Produce a DynamicModuleRecord from `module.exports`
3. If ES:
  3.1. Parse for `import`/`export`s and keep record, in order to create bindings
  3.2. Gather all submodules by performing recursive dependency loading
  3.3. Connect `import` bindings for all relevant submodules
  3.4. Evaluate as Module

For example, if a developer wanted to create a module that exports both module types (CommonJS and ES Modules) for backward compatibility, their package.json may be defined as:

{
  "name": "test",
  "version": "0.0.1",
  "description": "",
  "main": "./index", // no file extension
}

The package will then have both an index.mjs and an index.js. The index.mjs is an ES Module, using the new export / import syntax:

// index.mjs
export default class Foo {
  //..
}

And the index.js is a CommonJS style module, using the module.exports object:

// index.js
class Foo {
  // ...
}
module.exports = Foo;

If the version of Node.js being used supports ES Modules via the .mjs file extension, it will first try to find an index.mjs. On the other hand, if the version of Node.js does not support ES Modules (such as Node.js v4 or v6), or it can not find an index.mjs, it will look for an index.js.

According to the EP, you would be able to use both require and import to find packages in your node_modules:

import mkdirp from 'mkdirp';
require('mkdirp');

For resolving modules local to your own project or package, you do not need to add a file extensions in your require() or import statements unless you want to be precise. The standard Node.js file resolution algorithm applies when you don't supply an extension, but an .mjs version is looked for before a .js:

require('./foo');
import './foo';
// these both look at
//   ./foo.mjs
//   ./foo.js
//   ./foo/index.mjs
//   ./foo/index.js

// to explicitly load a CJS module, add '.js':
import './foo.js';
// to explicitly load an ES module add '.mjs'
import './bar.mjs';

Examples: Consuming CommonJS with ES Modules

Example 1: Load CommonJS from ES Modules

// cjs.js
module.exports = {
  default:'my-default',
  thing:'stuff'
};

// es.mjs

import * as baz from './cjs.js';
// baz = {
//   get default() {return module.exports;},
//   get thing() {return this.default.thing}.bind(baz)
// }
// console.log(baz.default.default); // my-default

import foo from './cjs.js';
// foo = {default:'my-default', thing:'stuff'};

import {default as bar} from './cjs.js';
// bar = {default:'my-default', thing:'stuff'};

Example 2: Export value and assigning "default"

// cjs.js
module.exports = null;

// es.mjs
import foo from './cjs.js';
// foo = null;

import * as bar from './cjs.js';
// bar = {default:null};

Example 3: Single-function export

// cjs.js
module.exports = function two() {
  return 2;
};

// es.mjs
import foo from './cjs.js';
foo(); // 2

import * as bar from './cjs.js';
bar.name; // 'two' ( get function name)
bar.default(); // 2 ( assigned default function )
bar(); // throws, bar is not a function

Examples: Consuming ES Modules with CommonJS

Example 1: Using export default

// es.mjs
let foo = {bar:'my-default'};
export default foo;
foo = null; // this null value does not effect import value.

// cjs.js
const es_namespace = require('./es');
// es_namespace ~= {
//   get default() {
//     return result_from_evaluating_foo;
//   }
// }
console.log(es_namespace.default);
// {bar:'my-default'}

Example 2: Using export

// es.mjs
export let foo = {bar:'my-default'};
export {foo as bar};
export function f() {};
export class c {};

// cjs.js
const es_namespace = require('./es');
// es_namespace ~= {
//   get foo() {return foo;}
//   get bar() {return foo;}
//   get f() {return f;}
//   get c() {return c;}
// }

Current state of discussion

Although built in a collaborative process, taking into account proposals for alternatives, Bradley's landed EP received a prominent counter-proposal from outside of the EP process. Going by the name "In Defense of .js", this counter-proposal relies on the use of package.json rather than a new file extension. Even though this option had been previously discussed, this new proposal contains some interesting additions.

In Defense of .js presents the following rules for determining what format to load a file, with the same rules for both require and import:

• If package.json has "main" field but not a "module" field, all files in that package are loaded as CommonJS.
• If a package.json has a "module" field but not "main" field, all files in that package are loaded as ES Modules.
• If a package.json has neither "main" nor "module" fields, it will depend on on whether an index.js or a module.js exists in the package as to whether to load files in the package as CommonJS or ES Modules respectively.
• If a package.json has both "main" and "module" fields, files in the package will be loaded as CommonJS unless they are enumerated in the "module" field in which case they will be loaded as ES Modules, this may also include directories.
• If there is no package.json in place (e.g. require('c:/foo')), it will default to being loaded as CommonJS.
• A special "modules.root" field in package.json, files under the directory specified will be loaded as ES Modules. Additionally, files loaded relative to the package itself (e.g. require('lodash/array')) will load from within this directory.

In Defense of .js Examples

// package.json
// all files loaded as CommonJS
{
  "main": "index.js" // default module for package
}

// package.json
// default to CommonJS, conditional loading of ES Modules
{
  "main": "index.js", // used by older versions of Node.js as default module, CommonJS
  "module": "module.js" // used by newer versions of Node.js as default module, ES Module
}

// package.json
// CommonJS with directory exceptions
{
  "main": "index.js",
  "module": "module.js",
  "modules.root": "lib" // all files loaded within this directory will be ES Modules
}

The above example is used to show how to maintain backward compatibility for packages. For older versions of Node.js, require('foo/bar') will look for a CommonJS bar.js in the root of the package. However, for newer versions of Node.js, the "modules.root": "lib" directory will dictate that loading 'foo/bar' will look for an ES Module at lib/bar.js.

Supporting both CommonJS and ES Modules

Under most proposals, including the Node.js EP and In Defense of .js, it is assumed that packages wishing to provide support for old and newer versions of Node.js will use a transpilation mechanism. Under the .mjs solution, the ES Modules would be transpiled to .js files next to their originals and the different versions of Node.js would resolve to the right file. Under In Defense of .js, the ES Modules would exist under a subdirectory specified by "modules.root" and be transpiled to CommonJS forms in the parent directory; additionally, package.json would have both "main" and "module" entry-points.

Hard choices

In Defense of .js presents a view that we need to switch to ES Modules from CommonJS and prioritizes such a future. On the other hand, the Node.js EP prioritizes compatibility and interoperability.

Bradley recently wrote a post attempting to further explain the difficult choice and why a file extension was an appropriate way forward. In it, he goes into further details about why it is not possible to parse a file to determine whether it is an ES Module or not. He also further explores the difficulties of having an out-of-band descriptor (e.g. package.json) determine what type of content is in a .js file.

Although it may be sad to consider the loss of a universal .js file extension, it's worth noting that other languages have already paved this path. Perl for instance uses .pl for Perl Script, and .pm for Perl Module.

Getting involved

Even though the Node.js CTC has accepted the EP in its current form and stated its preference on how ES Modules would be implemented in Node.js (if they are implemented in Node.js at all), discussion continues and there is still room for change. You can engage with the Node.js community on this topic in the Node.js EP repository issues list. Be sure to first review existing comments to see if your concerns have already been addressed.

Bradley and the Node.js CTC are very concerned about getting this decision right, in the interests of Node.js users everywhere. The choices that Node.js is having to make to accommodate ES Modules are difficult and are not being approached lightly.