Node.js / io.js State of the Build August 2015: ARM
Reposted from Medium
My last State of the Build was in April and I wanted to get another one out but instead of doing the whole thing, which would be more effort and time than I can afford at the moment, I’m going to just bite off the ARM bits.
See my article Node and ARM for my thoughts on why ARM hardware is important to Node’s future.
ARM Test Coverage
We have 4 main areas of test coverage at the moment for ARM, with a 5th that’s temporarily offline but will be back soon.
- pi1-raspbian-wheezy: Compile and test on ARMv6 Raspberry Pi 1 B+ boards running Raspbian
- pi2-raspbian-wheezy: Compile and test on ARMv7 Raspberry Pi 2 B boards running Raspbian
- armv7-wheezy: Compile and test ARMv7 Marvell ARMADA 370 based boards running Debian Wheezy
- armv7-ubuntu1404: Compile and test on ODROID-XU3 Samsung Exynos5422 Cortex-A15 based boards running an Ubuntu 14.04 based OS
Additionally we have access to an ARMv8 / AARCH64 server via the Linaro Server Cluster, but we’re having some technical problems with that and have deferred dealing with that because we have new replacement hardware coming online soon.
How Tests are Used
Test failures on ARM platforms are now treated in the same way as on other platforms for the current Node.js master. An example of this is the active PR for an upgrade to V8 4.5 by @ofrobots which was found to be broken for ARMv6 (original Raspberry Pi) with a fix being investigated (and hopefully submitted upstream to V8).
Some tests get special treatment by the test suite when running on ARM; mostly to account for varying speed. This logic is embedded in test/common.js with ARMv6 getting minimum timeouts in some tests multiplied by 7 and ARMv7+ having them multiplied by 2. Search for platformTimeout to see which tests are currently receiving this special attention.
ARM tests have been used as a reference for the entire lifespan of io.js, meaning that Node.js v4 will inherit first-class ARMv6, ARMv7 and AARCH64 (pending some current hardware work to bring that back online) support. Node.js v0.10 and v0.12 on the other hand predate a lot of this work and the test slaves for ARM do not apply to these branches, even under converged Node / CI. ARM bugs filed against Node.js v0.12 may be addressed if there is a clear path to fixing or where a working pull request is provided. Node.js v0.10 is unlikely to see attention for its ARM support as it is moving into maintenance mode for the rest of its supported life.
Some new work has been put in by @joaocgreis and @orangemocha to the test runner for core, adding the --run=n,m option that will allow us to divide our ARM clusters into groups that will run in parallel, executing independent sets of tests and thereby parallelising our test runs on ARM! This will be particularly helpful on ARMv6 / Raspberry Pi v1 which are quite slow because just starting a node process for each of the ~800 tests takes in the order of 3s each time! More on this below (we are looking for donations of a couple more Raspberry Pi’s).
ARM Hardware Resources
pi1-raspbian-wheezy / ARMv6 Raspberry Pi 1 B+
We have 5 Raspberry Pi 1 B+ boards active in CI currently. They are running Raspbian Wheezy. You can see them in Jenkins by looking for the name “iojs-nodesource-raspbian-wheezy-pi1p-X” where “X” is a number between 1 and 5. For example: https://jenkins-iojs.nodesource.com/computer/iojs-nodesource-raspbian-wheezy-pi1p-1/ (“nodesource” is in the name because they are hosted in my office).
The boards were all donated to the project:
- One board was donated by NodeSource
- Two boards were donated by Andi Neck / @andineck
- Two boards were donated by Bryan English / @bengl
They are configured with NFS mounted SSD access to help speed up builds. This doesn’t make a huge difference for general compiling but it makes a large difference when coupled with ccache which has a large cache distributed across all of the machines. This means that an object file compiled on one of the Pi’s will be cached for reuse by any of the others, shaving many minutes (usually hours) off compiles. Because most of the time compiling is spent in V8 and V8 doesn’t change that much between pull requests, we get to take advantage of identical object file generation. Even when we get a new V8 version it just needs to be compiled once (usually) to prime the cache.
This cluster is configured to produce the armv6l binaries that land with io.js releases and will land with Node.js v4+ releases.
We are about to begin work on parallelising tests across the Pi 1 B+ cluster using the new test runner -run flag, see above for details on that.
pi2-raspbian-wheezy / ARMv7 Raspberry Pi 2 B
We have 5 Raspberry Pi 2 B boards active in CI currently. They are running Raspbian Wheezy. You can see them in Jenkins by looking for the name “iojs-nodesource-raspbian-wheezy-pi2-X” where “X” is a number between 1 and 5. For example: https://jenkins-iojs.nodesource.com/computer/iojs-nodesorce-raspbian-wheezy-pi2-1/
The boards were all donated to the project:
- One board was donated by NodeSource
- Two boards were donated by Joey van Dijk / @joeyvandijk & http://www.techtribe.nl/
- One board was donated by Simeon Vincent / @svincent
- One board was donated by Matteo Collina / @mcollina
Our Raspberry Pi 2 B boards are set up almost identically to the Pi 1 B+ boards with access to shared SSD and ccache. Because they are newer CPUs they compile and run significantly faster than the Pi 1 B+ boards but they could still do with some parallelisation.
armv7-wheezy / ARMv7 Marvell ARMADA 370
These ARMv7 Marvell ARMADA 370 based boards come courtesy of Scaleway from their ARM IaaS. They couple custom hardware with SSDs to provide a great alternative to heavy-weight IaaS providers and are doing us a service by providing a free account for up to 5 machines to use for testing. We are currently using two machines for testing on pure Debian Wheezy (armhf) and also using a dedicated release machine running Debian Wheezy to produce the armv7l binaries that land with io.js releases and will land with Node.js v4+ releases. We switched to these machines for producing binaries from an ODROID-XU3 machine (below) because Wheezy gives us a much older libc to compile against, making the binaries compatible with older armv7 OSs including the custom one used by many BeagleBone Black users. You can see these machines in Jenkins by looking for the name “iojs-online_net-armv7-wheezy-*” (Scaleway used to be online.net).
I’d encourage you to check out Scaleway if you’re interested in alternative hosting options for your Node.js applications.
FWIW I’m happy to spruik for them because I think this is a fantastic platform and really wish for their success! My spruiking is my own, nothing to do with my employer or the Node.js Foundation or Build Working Group of course.
armv7-ubuntu1404 / Samsung Exynos5422 Cortex-A15
These Samsung Exynos5422 Cortex-A15 based boards are made by Hardkernel / ODROID, they are the ODROID-XU3 which run a mildly customised version of Ubuntu 14.04. You can see this machine in Jenkins by looking for the name “iojs-nodesource-armv7-ubuntu1401”. They are small and brilliantly fast and even come with a tiny fan that spins up occasionally under load. If you’re looking for some some ARM hardware with some serious grunt then I’d recommend these, the XU4 is now out with very similar specs but nearly 1/2 the size and less than 1/2 the price.
Currently we only have one in active use by the project CI, originally donated by NodeSource but we also have two more identical boards that still need to be set up, donated by Dav Glass / @davglass of Yahoo!. I have had these boards for some time but haven’t been able to allocate the time for the final yak shave needed to get them online but will do so soon!
ARMv8 / AARCH64 hardware
ARM themselves were kind enough to put in a word to get us access to the Linaro Server Cluster for an ARMv8 machine running Ubuntu 14.04 which enabled us to add ARMv8 support to io.js. This server is currently inactive in CI due to access problems which I have not prioritised sorting out, I will be informing Linaro soon that they can deallocate the machine for us.
As I mentioned in a previous article, ARM have taken a keen interest in Node.js/io.js now that we are catching up to modern V8 as they are driving hard into the server market and Node has a great story to tell there obviously. So, ARM have raised their stakes in the project by donating 3 AppliedMicro X-Gene ARMv8 / AARCH64 server-class machines. I have them in my office and am just waiting on a serial adapter (!) to get them bootstrapped onto my network and then straight in to CI. They are running Ubuntu 14.04 and are super-fast compared to any of our other ARM hardware so we may want to consider virtualising (containerising more likely) multiple CI slaves on each machine so we can test different Linux variants (there have been lots of announcements in the past couple of months about Linux distro support for ARMv8).
We’ll also be using these to produce AARCH64 binaries for Node.js v4 and onward.
Unused ARM hardware
We have some devices that are currently not in use, including the two ODROID-XU3 boards donated by Dav Glass and the three ARMv8 machines donated by ARM. While these machines will end up in our cluster very soon there are some additional hardware that remain unused:
1 x BeagleBone Black: Donated by Julian Duque / @julianduque. This board is interesting because it’s popular among NodeBots enthusiasts and BeagleBone insist on shipping their own odd distribution(s) with them. Unfortunately, until we have enough to make a cluster, hooking a single board up to CI will only slow down the testing phase. I have been considering hooking it up as a special that doesn’t sit within the standard testing set but can be run occasionally to check status. We remain open to donations of BeagleBone Black boards of course!
1 x ODROID-C1: Donated by Joey van Dijk / @joeyvandijk. Similar to the BeagleBone Black in that having only one of these means that it’ll only slow down CI and introduce a single point of failure that we can’t afford. If you have a C1 laying around that you’re looking to offload then please send it my way!
A Call For Donations
We have a Node.js Foundation now, so technically we could afford to spend some money on hardware. However, so far, our ARM cluster has been made up entirely of community donated resources and there’s something special about that. ARM hardware is currently mostly used by hobbyists and it’s among those users of Node that we’ve received many of our donations. What’s more, being able to receive donations from the community provides yet another avenue for contribution to the project, particularly for those who are too busy to contribute their time.
Our cluster is growing, so much so that I’ve recently had to purchase (thanks NodeSource!) some new networking and power equipment to run it. But it can grow further and we’d love to receive devices that could add to our test cluster so we can ensure we are shipping code that is solid on all of the ARM devices that are in use by the community.
The real needs we have right now are for Raspberry Pi boards. As noted above, we now have the ability to distribute test runs across multiple machines, so we need an even number so make this practical. Currently we have 5 of each Pi 1 B+ and Pi 2 B boards which we could divide up in to two clusters of two with one spare of each. What we really want is to be able to have two clusters of three!
If you would like to contribute financially, we’re seeking to add at least one of each of these to our cluster:
Pi 1 B+ with SD card @ $45 + Enclosure @ $6.50 = $51.50 AUD (~$38 USD)
Pi 1 B+ with MicroSD card @ $54.67 + Enclosure @ $6.50 = $61.17 AUD (~$45 USD)
In the past I’ve had folks purchase through my preferred local supplier but as I can now get free shipping with my personal account I’d suggest that transferring the donation to me via PayPal so that I can place the order would be best (also, I now receive multiple copies of junk mail from them addressed to each of the people who have ordered hardware to be delivered to me!).
Drop me a line @ rod@vagg.org if you’d like to help out. I can either give you an address for shipping actual hardware or give you details on how you can transfer funds.