Streams and Buffers in Node.js

Streams and buffers are fundamental concepts in Node.js for handling data efficiently. Understanding how they work is essential for building performant applications that deal with files, network requests, and large datasets.

What Are Buffers?

A buffer is a temporary storage area for raw binary data. It is a part of the stream’s flow, like drops of water in a waterfall.

When data is read in chunks through a stream, each chunk is stored in a buffer. Similarly, when writing data to a stream, buffers are used to hold the data before sending it.

// Creating a buffer
const buffer = Buffer.from('Hello');
console.log(buffer); // <Buffer 48 65 6c 6c 6f>

// Converting back to string
console.log(buffer.toString()); // Hello

Buffers are particularly useful because JavaScript originally didn’t have a way to handle binary data. Node.js introduced buffers to work with TCP streams, file system operations, and other binary data sources.

What Are Streams?

Streams are a flow of data piece by piece, like a waterfall’s stream. This is especially useful when you don’t want to load everything into memory at once.

A stream can be represented as buffer1 + buffer2 + ... where the full data comes together. Data received or watched can be thought of as buffer1 and buffer2.

Stream --> Part1 --> Part2 --> Part3 --> Full data

In Node.js, you cannot work directly with parts; instead, buffers or chunks are received and worked on once they are complete.

Four Types of Streams

1. Readable Streams

For reading data. Examples: fs.createReadStream, http.IncomingMessage.

const fs = require('fs');

// Create a readable stream
const ourReadStream = fs.createReadStream(`${__dirname}/bigdata.txt`);

// Listen for data events (when buffer is filled)
ourReadStream.on('data', (chunk) => {
  console.log('Received chunk:');
  console.log(chunk.toString());
});

console.log('Reading started');

The createReadStream efficiently reads the file in chunks and emits the data event whenever new data is available for processing (when the buffer is filled).

2. Writable Streams

For writing data. Examples: fs.createWriteStream, http.ServerResponse.

const http = require('http');

const server = http.createServer((req, res) => {
  if (req.url == '/') {
    res.write('<html><head><title>Form</title></head>');
    res.write('<body><form method="post" action="/process">');
    res.write('<input name="message" /></form></body>'); 
    res.end();
  } else if (req.url == '/process' && req.method == 'POST') {
    let body = '';
    
    // Reading data from the request stream
    req.on('data', (chunk) => {
      body += chunk;
    });
    
    req.on('end', () => {
      console.log('Received:', body);
      res.write('Data processed');
      res.end();
    });
  } else {
    res.write('Not Found');
    res.end();
  }
});

server.listen(3000);

In this example, req is a readable stream, and res is a writable stream. We listen to the data and end events on req to capture the incoming data.

3. Duplex Streams

These can be both readable and writable. Example: TCP sockets (net.Socket).

const net = require('net');

const server = net.createServer((socket) => {
  // Socket is duplex - can read and write
  socket.on('data', (data) => {
    console.log('Received:', data.toString());
    socket.write('Echo: ' + data);
  });
});

server.listen(8000);

4. Transform Streams

A type of duplex stream that can modify the data as it’s read or written. Example: zlib.createGzip for compression.

const fs = require('fs');
const zlib = require('zlib');

// Create a transform stream for compression
const gzip = zlib.createGzip();

// Read, compress, and write
fs.createReadStream('input.txt')
  .pipe(gzip)
  .pipe(fs.createWriteStream('input.txt.gz'));

Common Stream Methods

.pipe()

Connects a readable stream to a writable stream, allowing data to flow from one to the other.

Example without .pipe():

const fs = require('fs');

const ourReadStream = fs.createReadStream(`${__dirname}/bigdata.txt`);
const ourWriteStream = fs.createWriteStream(`${__dirname}/output.txt`);

ourReadStream.on('data', (chunk) => {
  ourWriteStream.write(chunk);
});

Example with .pipe() (cleaner):

const fs = require('fs');

const ourReadStream = fs.createReadStream(`${__dirname}/bigdata.txt`);
const ourWriteStream = fs.createWriteStream(`${__dirname}/output.txt`);

ourReadStream.pipe(ourWriteStream);

The pipe() method automatically handles the data flow between the readable and writable streams, making the code cleaner and more efficient.

.on()

Event listener for stream events like data, end, and error.

readStream.on('data', (chunk) => {
  // Handle chunk
});

readStream.on('end', () => {
  console.log('Reading complete');
});

readStream.on('error', (err) => {
  console.error('Error:', err);
});

Why Use Streams?

Memory Efficiency

Instead of loading an entire file into memory, streams process data in chunks:

// Without streams - loads entire file into memory
const fs = require('fs');
const data = fs.readFileSync('huge-file.txt'); // Dangerous with large files

// With streams - processes in chunks
const readStream = fs.createReadStream('huge-file.txt');
readStream.on('data', (chunk) => {
  // Process chunk (much less memory)
});

Time Efficiency

Streams allow you to start processing data before it’s all available:

const http = require('http');
const fs = require('fs');

http.createServer((req, res) => {
  // Start sending response immediately
  fs.createReadStream('large-video.mp4').pipe(res);
  // User starts receiving data right away
}).listen(3000);

Composability

Streams can be chained together for complex operations:

const fs = require('fs');
const zlib = require('zlib');
const crypto = require('crypto');

// Read → Compress → Encrypt → Write
fs.createReadStream('input.txt')
  .pipe(zlib.createGzip())
  .pipe(crypto.createCipher('aes192', 'password'))
  .pipe(fs.createWriteStream('output.txt.gz.enc'));

Stream Events

All streams emit events that you can listen to:

Event	Description	Stream Type
data	Fired when chunk is available	Readable
end	Fired when no more data	Readable
error	Fired when error occurs	All
finish	Fired when all data written	Writable
pipe	Fired when .pipe() is called	Readable
unpipe	Fired when .unpipe() is called	Readable

Practical Example: File Upload

const http = require('http');
const fs = require('fs');

http.createServer((req, res) => {
  if (req.method === 'POST' && req.url === '/upload') {
    const writeStream = fs.createWriteStream('uploaded-file.txt');
    
    let uploadedBytes = 0;
    
    req.on('data', (chunk) => {
      uploadedBytes += chunk.length;
      console.log(`Uploaded: ${uploadedBytes} bytes`);
    });
    
    req.pipe(writeStream);
    
    writeStream.on('finish', () => {
      res.end('Upload complete!');
    });
    
    writeStream.on('error', (err) => {
      res.end('Upload failed: ' + err.message);
    });
  }
}).listen(3000);

Buffer Operations

Buffers provide various methods for manipulating binary data:

// Create buffers
const buf1 = Buffer.from('Hello');
const buf2 = Buffer.alloc(10); // 10 bytes of zeros
const buf3 = Buffer.allocUnsafe(10); // Faster but not zeroed

// Write to buffer
buf2.write('World');

// Concatenate buffers
const buf4 = Buffer.concat([buf1, buf2]);

// Compare buffers
console.log(buf1.equals(buf2)); // false

// Slice buffer
const slice = buf4.slice(0, 5);
console.log(slice.toString()); // 'Hello'

Summary

Streams and buffers are powerful tools in Node.js for handling data efficiently:

Concept	Description	Use Case
Buffer	Temporary storage for binary data	Handling raw data chunks
Readable	Stream for reading data	Reading files, HTTP requests
Writable	Stream for writing data	Writing files, HTTP responses
Duplex	Stream for both reading and writing	Network sockets
Transform	Stream that modifies data in transit	Compression, encryption

Understanding streams and buffers is essential for building efficient Node.js applications. They allow you to handle large amounts of data without overwhelming memory, start processing data before it’s fully available, and compose complex data transformations in a clean, readable way.

By leveraging streams, you can build applications that are both memory-efficient and performant, handling data of any size gracefully.