The Ultimate Guide to Performance Monitoring in Node.js

Contents

Application Performance in Node.js
Node.js Performance Metrics
Garbage Collection and Memory Leaks in Node.js
Garbage Collection
Memory Leaks
CPU Usage
CPU-Intensive Tasks
The Event Loop: Managing Async Behavior
The Event Loop Phases
Understanding Node.js Application Health
Analyzing Performance with Heap Snapshots and CPU Profiles
Using N|Solid to Monitor and Optimize Performance
Thresholds and Alerts
Monitoring the Event Loop
Lag Time in the Event Loop
Idle Time in the Event Loop
Comparing Lag and Idle Times
Conclusion
Contact us

Application Performance in Node.js

If your applications are customer-facing or internally relied on for critical business functions, performance is not a bonus but a requirement. For any of these mission-critical applications, performant behavior should include:

Minimal or no downtime due to preventable outages
Predictable resource usage based off of informed benchmarks
Effective scaling to meet peak usage demands

When dealing with performance in Node.js, there are several metrics that can be vitally important to understanding and managing your application health. Identifying these key metrics and monitoring them on an ongoing basis informs application performance by allowing engineers to:

Identify the impact of design and architecture choices
Use infrastructure resources more effectively
Be prepared with relevant data should there be a critical performance issue

Node.js Performance Metrics

The diverse Node.js ecosystem has provided many ways of accessing application metrics, and parsing this can make it difficult to determine which core metrics to focus on optimizing. This paper will highlight the three metric classes that, if leveraged correctly, help provide a comprehensive view of your Node.js applications:

Heap usage
CPU usage
Event loop behavior

As each of these three metric classes are explored, this paper will look at how their data is generated and analyzed with N|Solid, the Node.js platform built for mission-critical applications in the enterprise.

Garbage Collection and Memory Leaks in Node.js

For dynamic memory allocation of objects of unknown size, quantity, or duration, there is the heap. New objects are typically stored on the heap and, unlike allocation on the stack, the memory storage of an object persists after the function that instantiated the object returns.

By default, the V8 Heap allocates 1.5 GB of memory to a Node.js process, regardless of how much memory is available on the machine running the process. As this finite amount of memory is used, there is less and less space for new objects to be stored. If not addressed, this will lead to significant performance hits or outright application failure.

If the default heap size allotted to your Node.js process needs to be adjusted, it can be set with the --max-old-space-size option.

The heap does not have a predictable allocation/deallocation paradigm for blocks of memory. Because memory on the heap can be allocated and freed at any time, manual management of heap memory can become unwieldy and complex. In some languages, the memory allocated needs to be explicitly freed by a programmer. Node.js, however, is memory managed—Google’s V8 allocates memory on the heap and uses garbage collection to free it.

Garbage Collection

Garbage collection in Node.js relies on a combination of several algorithms that track references to objects. If an object on the heap is no longer referenced by any other objects it is considered dead. This makes it an ‘unreachable object’, the foundation of what garbage collection algorithms are looking for to free up to the heap.

Google V8’s garbage collector thinks of the heap as having spaces and of objects as belonging to generations. The heap is divided into Old Space and New Space. The New Space is where newly created objects, referred to as the Young Generation, are allocated. Garbage collection happens quickly in the New Space. If an object survives a number of garbage collection runs in the New Space it is promoted to the Old Space and becomes a member of the Old Generation. Allocation happens very quickly in the Old Space. Garbage collection in the Old Space is costly and thus less frequently performed.

Memory Leaks

Memory leaks are a common issue in Node.js and occur when memory is not freed up for garbage collection. If memory usage is growing steadily in your application, and garbage collection runs do not reduce it, your code is most likely suffering from a memory leak.

There are three main classes of memory leaks in Node.js:

Leaks from native modules.
Leaks from JS objects that retain external memory (from outside the heap).
Leaks from JS objects that are in the heap.

Depending on what is being allocated and when, memory consumption from these leaks may be very slow, or quite rapid. Of particular concern are functions or objects that are being added to other long-lived objects on a recurring basis. JavaScript uses closures, a function object that has access to the parent scope even after the parent function has ended. If not implemented correctly, this inherited scope can create unwanted references to strings, arrays, objects, buffers, and other data. The severity of the memory leak is proportional to the size of the unwanted reference multiplied by the number of times it is executed.

LEAKY RSS MEMORY

Native modules (typically used to provide an interface between JavaScript running in Node.js and C/C++ libraries) and JS objects retaining external memory (such as buffers, etc.) can not be directly referenced from JavaScript because they use RSS memory and Google’s V8 only enables developers to interact with heap memory. This paper will focus on the interactions between Node.js and V8 heap memory.

LEAKY HEAP MEMORY

Leaks from JavaScript objects in the heap are the most common for a Node.js developer. Because V8 handles memory allocation and deallocation in your Node.js applications, it is typically syntactical or logical error on the developer’s part that leads to memory leaks. Commons sources of unwanted references in JavaScript include accidental global variables, forgotten observers, and improper use of closures.

CPU Usage

In Node.js applications, high CPU usage can be an indicator that your application is doing a lot of synchronous work. Not all synchronous work is created equally: synchronous I/O work, and synchronous processing of large data structures and parsing tasks can prevent the event loop from moving on to the next task in the queue. If not handled well, these types of tasks will block the event loop.

CPU-Intensive Tasks

Processes or tasks which run on a computer require various resources like CPU cycles, memory, and disk space. The operating system and event loop work together to manage tasks and their use of resources to help ensure that each task executes efficiently and, ideally, without an end user ever having to 'wait'.

However, some tasks, especially when combined with the scale of real world data sets, can prove to be limiting to application performance if special attention is not paid to CPU usage. Writing code that uses synchronous iterative functions like .forEach and parsing functions like JSON.parse to handle data structures of unknown size can lead to bottlenecks. Node.js champions asynchronous I/O, and packages like async.js have been developed in the ecosystem to help extend asynchronous flow control to CPU usage.

CPU-INTENSIVE PROCESS	NODE.JS SOLUTION
Synchronous I/O, such as the fs.Sync methods	Node.js’s asynchronous I/O APIs
Serial execution of asynchronous workloads	Parallelize asynchronous execution wherever possible
Parsing large, blocking JSON objects	Streaming and selective-parsing solutions such as `JSONstream` and `json-depth-stream` may improve throughput
XML processing	Avoid; introduce abstracted translation and filtering layers where possible

The Event Loop: Managing Async Behavior

The event loop and associated asynchronous logic is available to Node.js through the C library libuv. As Node.js is single-threaded, the event loop lives on the same thread executing all JavaScript code. To orchestrate its asynchronous flow, the event loop has a schedule of six phases that correspond with core programmatic operations. The six phases are:

Timers
Pending Callbacks
Idle/Prepare
I/O Polling
Check
Close Events

Within each phase, a first-in-first-out queue forms and will execute waiting callbacks until the queue has been depleted or a set maximum number of callbacks have been executed, at which point the event loop moves to the next phase. Let’s look at each phase in more detail.

The Event Loop Phases

TIMERS

The Timers phase is the starting point of the the event loop, and executes callbacks scheduled by setTimeout() and setInterval(). The times provided by setTimeout() and setInterval() describe the threshold after which a provided callback may be executed. Callbacks in these timers can still be delayed by other callbacks or by scheduling that the kernel has taken over.

PENDING CALLBACKS

This phase handles callbacks pushed onto the pending queue by previous operations. These callbacks are typically concerned with callbacks for systems operations like TCP/IP connections and error callbacks, more generally.

IDLE/PREPARE

This phase is internal-use-only. Node.js does not extend interactive functionality to this phase.

I/O POLLING

This phase handles new connections and incoming data. The queue attached to the poll phase is the watch queue. The callbacks in the watch queue will be handled first (until empty or max callbacks limit is reached), after which the phase will listen for new connections for a set amount of time.

CHECK

The check phase handles setImmediate() callbacks.

CLOSE EVENTS

Callbacks that close connections or otherwise ‘clean up’ event listening are handled in this phase. This phase primarily helps prepare the environment for a fresh turn of the event loop.

nextTickQueue and microTaskQueueThese two queues don’t live in a phase per se. nextTickQueue accepts callbacks invoked by process.nextTick() and microTaskQueue holds callbacks invoked by resolved Promises. These two queues are Node.js augmentations to the libuv implementation of the event loop, and are intended to provide more direct control for critical callback execution in the event loop.

This phase-based movement within the event loop dictates the programmatic idioms that developers need to keep in mind when building performant Node.js applications. Based off of these mechanics and workflow, there are three aspects of event loop behavior that we will focus on for event loop monitoring: lag time, idle time, and the amount of active handlers. More details on event loop metrics in the section "Monitoring the Event Loop".

Understanding Node.js Application Health

Thus far, we have discussed heap usage, CPU usage, and event loop behavior as three critical classes of performance metrics when looking to debug or optimize a Node.js application.

“Normal" and “abnormal” ranges for each of these metrics will reflect the specific needs of a given application, and will thus vary between applications. As needs vary, each application should have its own clear performance benchmarks to be established as a point of reference.

Driven by the nature of your application and how users are engaged with it, specific patterns of performance demands will emerge.

Performance Load and Benchmarking

The point of benchmarking is to have a sane understanding of the maximum threshold an application can handle. It does not indicate efficiency in code. Establishing a baseline through benchmarking allows engineers to make adjustments to the architecture and code with measurable results.

Establish clear metrics to measure the predictability of your application.
Ideally capture real world data to replay against your app (tools: jmeter / gor / k6, etc).
As applications grow (new features added) adjust and measure tests to identify baseline performance.

The point of benchmarking is to have a sane understanding of the maximum threshold an application can handle. It does not indicate efficiency in code. Establishing a baseline through benchmarking allows engineers to make adjustments to the architecture and code with measurable results.

Analyzing Performance with Heap Snapshots and CPU Profiles

Thus far, most development of diagnostic tooling in Node.js has focused on monitoring heap usage and CPU usage. "Heap Snapshots" have risen as a powerful way to interact with an application’s memory profile. Taking "CPU Profiles" provides analogous insight into how your application is using CPU resources.

HEAP SNAPSHOTS

Heap snapshots help us understand where memory is being used, and provide a starting point for debugging memory leaks in Node.js applications.

The below image shows a heap snapshot in the N|Solid Console. The image that follows shows a deeper dive into the downloaded snapshot viewed with Chrome DevTools.

To be beneficial, at least two heap snapshots need to be generated. If heap snapshot file sizes for a given process are increasing in size, you have a clear sign that a memory leak is most likely the issue at hand.

Assume we have two application states. The first is the ‘control’ state; benchmarks for memory usage have been established and are being realized effectively and consistently. The control state is the ‘healthy’ state. The second is the ‘experimental’ state; the application is crashing, user experience is suffering, and you suspect memory usage to be at the heart of the issue. For constructive comparison, take a heap snapshot of both application states and look for deltas.

The N|Solid Console now provides cross-process snapshot diffing, which can directly compare two processes and their memory metrics. Ideally, the two processes show 0 deltas when compared. The appearance of deltas helps narrow the search for memory leaks.

CPU PROFILES

CPU Profiles, as discussed earlier, are a key tool for digging down and investigating issues around your Node.js applications’ use of CPU resources . With N|Solid, CPU profiles can be triggered automatically and remotely across any of your Node.js applications and processes. These profiles can then be examined within the N|Solid Console to gain insight into the issues causing abnormal CPU usage.

The above screenshot shows a CPU profile displayed as a flame graph. The flame graph displays the call stack, showing functions nested in the order they are executed. Parent functions are on the bottom. The y-axis, or height, of the call stack is not particularly illuminating, as it is only a reference to function complexity. The x-axis, or width, of the call stack shows how much time it takes for each function to execute.

Three regions of this example CPU profile beg further attention. The first column, labeled 1, seems significant because it features a spike, but the width of the block shows that despite a complex call depth, it is consuming a minor percentage of the overall CPU time. From this we can infer a relatively high complexity function, but it processes quickly, which is desired behavior. Investigating this area further is unlikely to reveal the source of CPU usage issue.

The other two regions to pay attention to are labeled 2 and 3. Region 2 is not very tall, suggesting relatively low function complexity, and is not particularly wide either, meaning a relatively low CPU cost. Region 3 is taller (comparatively high function complexity) and maintains a fairly wide profile down the call stack.

Because it has deeper nested functions that are relatively CPU expensive, region 3 would be a good starting point for debugging or optimizing your Node.js application’s use of CPU resources.

Third-party APMs and Your Node.js Applications

Certain tooling, like some third-party application performance monitors (APMs), take an approach to metric gathering that can have quite a negative impact on the performance of Node.js applications.

While these tools are usually great in polyglot environments, they require additional instrumentation and code augmentation to be able to inspect the event loop from ‘inside’ a Node.js application. These factors can have a pretty severe impact on performance at scale. In some cases we’ve witnessed, they can have up to a 40% impact on performance.

Besides the direct performance hit an application is likely to take, it is worth noting that including third-party functionality into your application can also complicate debugging, especially if the APM has black-box qualities.

Thankfully, N|Solid can help you decrease this APM overhead in certain circumstances. If using AppDynamics, it will automatically utilize the N|Solid API to avoid some of the more costly monitoring of its own. Likewise, if your APM provider can consume data from StatsD, you can set up a StatsD integration from the N|Solid agent on each Node.js process.

Using N|Solid to Monitor and Optimize Performance

Through the N|Solid Runtime and Console, NodeSource offers a unique approach to performance monitoring in Node.js. N|Solid collects real-time performance data at the runtime level via the N|Solid Agent. A benefit of this direct connection to the core components of Node.js is that N|Solid is relatively resource cheap in a running application—around 1%—which is within an acceptable range for just about any Node.js application that needs this level of monitoring.

Thresholds and Alerts

The N|Solid console enables setting performance thresholds on over 50+ performance metrics related to the heap, CPU, and event loop. If these thresholds are crossed, actions like heap snapshots and CPU profiles can be triggered, as well as a range of alert options, such as emails and Slack webhooks.

The following are some common use cases for setting N|Solid thresholds and triggering actions accordingly:

EXCEEDED HEAP USAGE THRESHOLD

Heap usage varies greatly from app to app, so take this number with a grain of salt—if your applications typically perform near or above 65% memory, you’ll want to adjust to a higher memory threshold.

Common Triggers:

Memory Leaks

THRESHOLD	ACTION	ALERT
65% Heap Memory Usage	Trigger a Heap Snapshot	An incoming webhook that alerts the Ops team on Slack

EXCEEDED CPU USAGE THRESHOLD

Common Triggers

Blocking Synchronous Tasks
CPU-intensive computation

THRESHOLD	ACTION	ALERT
85% CPU Usage	Trigger a CPU Profile	An incoming webhook that alerts the Ops team on Slack

Monitoring the Event Loop

Instrumenting Node.js scripts to facilitate insight into the event loop is either incredibly inaccurate or impossible without a native C++ extension to extend the core functionality of Node.js. This is one of the principal reasons we built N|Solid; being able to directly monitor the metrics from each process’ event loop.

With the N|Solid StatsD interface, you can easily view real-time event loop performance data in your preferred APM tool. Of the 50+ application metrics that N|Solid provides, the following are specifically focused on providing insight into the event loop:

loopIdlePercent - Percent of time that Node.js is waiting for I/O ("idle").
loopEstimatedLag - The amount of time an I/O response may have to wait before being processed.
loopsPerSecond - The amount of event loop "turns" elapsed in the last second.
loopAvgTasks - The average amount of async JavaScript entries per loop.
activeHandlers - The number of longer-lived event-loop operations currently in flight or attached to resources.

Lag Time in the Event Loop

Lag occurs when the event loop is unable to keep pace with incoming events. Understanding when and why lag occurs in the event loop is important when optimizing your Node.js applications. A steady increase in lag can slow Node.js down to the point of unresponsiveness. Following are some common scenarios that encourage event loop lag:

LONG - RUNNING, SYNCHRONOUS TASKS

Spending too much time during a single tick of the event loop can be a source of lag. CPU-bound work can’t be avoided entirely, but it is important to be mindful of getting ‘trapped’ in a phase of the event loop. If a synchronous task is taking too long to complete and causing lag, it might make sense to perform that work in a different process, use flow controls like setImmediate() and process.nextTick(), or finding an asynchronous alternative to handling the data.

CONSTANT INCREASE IN AVERAGE TASKS PER LOOP

Node.js keeps track of all the functions and callbacks that need to be handled at the various phases of the event loop. When the rate of inbound events outpaces the event loop’s ability to process those events, your average amount of tasks per loop (loopAvgTasks) will increase and users will start to experience lag as these tasks start to ‘pile up’. Thankfully, scaling up the number of processes running your application often helps alleviate this source of lag.

An event loop without lag (a "healthy" event loop) will show that no task has been queued to wait for the completion of another task. Instead they are all processed as they are received. Because Node.js relies on the operating system’s kernel for control of the task queue, it is impossible to determine the moment a task is received; you can only know when it is ready to be processed. Thus, if two tasks are ready to be processed at the ‘same time’, Node.js can only know that the ‘second’ task is available after the ‘first’ has been completed.

Idle Time in the Event Loop

Idle time in the event loop is any time that there is no computation going on (no JavaScript to run) because the event loop is waiting for I/O. Other things ideal, low idle time is preferred because it represents efficient cycling of the event loop.

Comparing Lag and Idle Times

Avoiding bottlenecks is the name of the game in managing event loop performance. To identify bottlenecks in the event loop, analyze the lag (if any) and time spent idle. The relationship between lag time and idle time is useful in diagnosing the source of your event loop woes. How these two metrics relate to the amount of active handlers in the event loop can provide an even broader and use-specific picture of application performance.

HIGH EVENT LOOP LAG, LOW EVENT LOOP IDLE

When an application is simultaneously experiencing high lag and low idle time in the event loop, it is a clear sign that your application is attempting to do too much work and is CPU-bound, meaning it is experiencing a bottleneck due to its interaction with the CPU.

For applications that typically handle a high volume of networking requests, this high lag/low idle relationship is especially problematic if the amount of active handlers on the event loop is low. This caveat is highly dependent on uniquely established performance and use benchmarks.

The standard Sync() methods that you find in the Node.js core API, particularly in the fs module (e.g. fs.readFileSync()), have the unfortunate property of being synchronous. Choosing to use these methods means that with each cycle of the event loop, there is a pause until execution of these synchronous methods completes; nothing else can process in the meantime. For this reason, you should avoid using the synchronous core methods in your application whenever possible.

A typical example of problematic JavaScript is complex string concatenation or parsing. Often, rendering a very complex template or parsing a very large JSON object will take up enough CPU time to cause a noticeable impact on your application’s responsiveness due to the high lag time, low idle time scenario. This is why, generally speaking, Node.js is not ideal for long-running, CPU-intensive tasks.

The more CPU time your application needs in a single cycle of the event loop, the more time it will take for a single event loop cycle to complete, potentially preventing other tasks from being processed in a performant manner. Combing through your code and guessing what might be causing event loop delays is a tough chore without proper insight. In the following examples, N|Solid’s CPU profiler helps explore and identify where problematic synchronous execution is taking place.

LOW EVENT LOOP LAG, HIGH EVENT LOOP IDLE

If you are experiencing timeouts, or general performance degradation in an application that is demonstrating low lag and high idle time in the event loop, an I/O or downstream resource is likely acting as a bottleneck. In this case, your application is failing to fully use available CPU resources.

In this case, if the amount of active handlers being handled by the event loop is high, it indicates a performance problem elsewhere in the stack: slow, unresponsive, or otherwise blocked backend services.

A problematic I/O resource is likely to have a performance constraint of its own. This may either be an expected responsiveness issue, i.e. the resource simply isn’t designed to respond as quickly as you need, or it could be that the service is overloaded and your Node.js application is able to throw much more traffic at it than it can reasonably handle.

This is a very common scenario when Node.js is inserted into an existing environment and has to make external calls to existing services. An experience that NodeSource sees repeatedly when assisting companies migrate to Node.js is that their new application can handle much more throughput than their existing stack can, thereby exposing bottlenecks that have always existed but were never noticed.

In a services-based architecture, the performance of your application is dependent on every service involved in the handling of each transaction. Diagnosing problems of this kind requires insight into each service that is part of the transaction change and comparing the event loop and resource usage of each. A side-by-side comparison of heap snapshots like those provided by the N|Solid Console can easily spot the points in your services that need to be tweaked to optimize application performance.

Conclusion

The asynchronous operation flow of Node.js is powered by libuv and exists as the event loop, a construct with 6 phases optimized for handling callbacks (events) efficiently. Because of its foundational role in managing requests, the behavior of the event loop has a critical influence on CPU resource usage and vice versa. Because of this, it is important that Node.js developers write code that optimizes asynchronous functionality and integrates fortified best practices to avoid costly memory problems like unwanted or forgotten object references.

Application expectations in the enterprise are always shifting, and your Node.js application will have to adapt. Understanding your performance demands and priorities is integral to establishing performance criteria and benchmarks needed to effectively test and monitor your application.

When problems do arise with event loop management, CPU usage, and memory performance in mission-critical Node.js applications, the premium tooling and unique insights of the N|Solid Runtime and N|Solid Console offer enterprise users of Node.js unparalleled diagnostic capabilities. Easily generate CPU profiles based off of metric performance triggers that provide real-time visibility when it matters most. N|Solid offers memory tooling with cross-process snapshot diffing, providing comparison data that points engineers to problematic objects quickly and intuitively.

Contact us

Trusted By