Basic Buffer Operations in Node.js

Basic Buffer Operations in Node.js

·

3 min read

Buffer stands for binary data in Node.js, its different from arraybuffer and typed array in JavaScript. In this article, let's explore its basic usage.

Create

Normally, there are 4 methods can be used to create a Buffer.

First one is Buffer.alloc(size[, fill[, encoding]]).

const buf1 = Buffer.alloc(5);
console.log(buf1); // <Buffer 00 00 00 00 00>

const buf2 = Buffer.alloc(5, "abc")
console.log(buf2)  // <Buffer 61 62 63 61 62>

const buf3 = Buffer.alloc(11, 'aGVsbG8gd29ybGQ=', 'base64');
console.log(buf3)  // <Buffer 68 65 6c 6c 6f 20 77 6f 72 6c 64>

Second one is Buffer.allocUnsafe(size).

Buffer created by allocUnsafe is not initialized like alloc, so it may contains arbitrary data.

const buf = Buffer.allocUnsafe(5);
console.log(buf); // <Buffer 08 68 05 05 01>

Third one is Buffer.allocUnsafeSlow(size).

This method is similiar to allocUnsafe. The difference is that small buffer(<4KB) create by allocUnsafe is sliced from a single pre-allocated Buffer, and allocUnsafeSlow ensures that all created buffers are new created from operate system.

Fourth one is Buffer.from.

This method is useful for creating buffer from other objects.

const buf1 = Buffer.from([1, 2, 3]);
console.log(buf1); // <Buffer 01 02 03>

const buf2 = Buffer.from("haha");
console.log(buf2);  // <Buffer 68 61 68 61>

Read/Write String

We can use toString method to read String from Buffer.

const buf1 = Buffer.from("abcdef");
console.log(buf1.toString());           // abcdef
console.log(buf1.toString("hex"));      // 616263646566
console.log(buf1.toString("hex", 0, 3)); // 616263

We can use write(string[, offset[, length]][, encoding]) method to write String into Buffer.

const buf = Buffer.alloc(256);
const len = buf.write('\u00bd + \u00bc = \u00be', 0);
console.log(`${len} bytes: ${buf.toString('utf8', 0, len)}`);
// 12 bytes: ½ + ¼ = ¾

Read/Write Number

Node.js provides a list of methods for reading/writing number from/into Buffer.

const buf = Buffer.alloc(10);
buf.writeUInt8(1, 0);
console.log(buf); // <Buffer 01 00 00 00 00 00 00 00 00 00>
buf.writeUInt8(2, 1);
console.log(buf); // <Buffer 01 02 00 00 00 00 00 00 00 00>

console.log(buf.readUInt16BE(0)); // 258
console.log(buf.readUInt16LE(0)); // 513

Operations Between Buffers

There are also a list of methods for operations between buffers.

// subarray
const buf1 = Buffer.from([1, 2, 3, 4]);
console.log(buf1.subarray(1, 3));

// copy
const buf2 = Buffer.from([6, 7, 8, 9]);
// copy buf1[1:3) to buf2[0:]
buf1.copy(buf2, 0, 1, 3);
console.log(buf2); // <Buffer 02 03 08 09>

// concat
const buf3 = Buffer.concat([buf1, buf2]);
console.log(buf3); // <Buffer 01 02 03 04 02 03 08 09>

// equals
console.log(Buffer.from([1,2,3]).equals(Buffer.from([1,2,3]))) // true

Conversion with TypedArray

TypedArray is the binary standard in JavaScript.

If we have a Buffer, we can pass it to the TypedArray's constructor to create a TypedArray.

const buf = Buffer.from([1, 2, 3, 4]);
const uint32array = new Uint32Array(buf);

console.log(uint32array); // [ 1, 2, 3, 4 ]

In this way, same size memory is copied, every element in the Buffer should be interpreted as an element in the TypedArray.

We can also pass Buffer's original buffer into TypedArray directly. In this way, TypedArray and Buffer should share the same memory, and TypedArray will interpret elements according to the buffer's content directly.

const Uint32Array2 = new Uint32Array(
    buf.buffer, 
    buf.byteOffset, 
    buf.length / Uint32Array.BYTES_PER_ELEMENT);

console.log(Uint32Array2) // [ 67305985 ]

The same logic goes to conversion from TypedArray to Buffer. If we pass TypedArray to Buffer directly, then memory is copied. If we pass its original buffer to Buffer, then they should share the same memory.

const arr = new Uint32Array(2);
console.log(arr) // Uint32Array(2) [ 0, 0 ]

// memory copied
const buf1 = Buffer.from(arr)
console.log(buf1) // <Buffer 00 00>

// memory shared
const buf2 = Buffer.from(arr.buffer)
console.log(buf2) // <Buffer 00 00 00 00 00 00 00 00>