Node.js 18 🟢 Binary Data Manipulation with Buffers and TypedArrays
JavaScript was designed for text. Its strings are sequences of UTF-16 code units, its numbers are 64-bit floats, and its arrays are flexible collections that can hold anything. None of these types is suitable for binary data. When Node.js needs to read a file, receive a network packet, or generate a cryptographic hash, it works with raw bytes. The Buffer class and the TypedArray family exist to represent those bytes. A Buffer is Node.js’s original binary type, a subclass of Uint8Array that adds Node-specific methods. A TypedArray is the ECMAScript standard for typed binary views: Uint8Array, Int32Array, Float64Array, and others, each representing a fixed-width numeric type.
The two are closely related. Buffer is a Uint8Array with extra methods, and the same underlying memory can be viewed through multiple typed arrays with different element sizes. This aliasing is what makes binary manipulation efficient: a Uint8Array of 4 bytes can be read as an Int32Array of 1 element, or as a Float32Array of 1 element, without copying. The ArrayBuffer is the raw memory block, and the typed arrays are views into it. The DataView is a third view that allows reading and writing multi-byte values at arbitrary offsets with explicit endianness.
This chapter covers the Buffer class, the TypedArray family, the ArrayBuffer and DataView, the encoding and decoding functions, the conversion between these types, the endianness rules, and the patterns that make binary manipulation safe and readable.
Key point: A Buffer is a Uint8Array with Node-specific methods. All typed arrays are views into an underlying ArrayBuffer, and multiple views can share the same memory. Buffer handles text encodings (utf8, hex, base64, and others). DataView handles multi-byte numeric values with explicit endianness. Use Buffer.from to create a buffer from a string, array, or another buffer, and use the typed array constructors to create numeric views.
Why Buffer and TypedArray exist
The text-only problem. JavaScript strings cannot represent arbitrary byte sequences. A byte like 0xFF is not a valid UTF-8 sequence on its own, and the String type would convert it to a replacement character. Binary data — file contents, network packets, cryptographic digests — must be represented as bytes, not as text. The Buffer and Uint8Array types are the representation.
The performance problem. A regular JavaScript array of numbers is a flexible object with per-element overhead. Storing a megabyte of binary data as [0, 1, 2, ...] uses many times the memory and is far slower than storing it in a contiguous typed array. Typed arrays are backed by a single ArrayBuffer, and their elements are stored contiguously with no per-element overhead.
The interoperability problem. Node.js APIs return Buffer objects. Web APIs return ArrayBuffer objects. A code path that goes from a Node.js file read to a Web Crypto operation must convert between them. Buffer is a Uint8Array, so it can be passed to any API that expects a Uint8Array. The ArrayBuffer can be extracted from a Buffer with .buffer, though the view may not cover the whole buffer.
The endianness problem. A 32-bit integer can be stored in memory with the most significant byte first (big-endian) or last (little-endian). Network protocols use big-endian; most CPUs use little-endian. The Buffer methods have explicit endianness (readUInt32BE versus readUInt32LE), and the DataView requires the endianness as a parameter. The explicit choice prevents the silent bugs that occur when the platform’s byte order is assumed.
The view problem. The same bytes can be interpreted as different numeric types depending on the view. A four-byte sequence can be a Uint8Array of four elements, a Uint16Array of two elements, an Int32Array of one element, or a Float32Array of one element. The ArrayBuffer is the shared memory, and each typed array is a view. This makes binary parsing efficient: the same buffer can be read with different views without copying.
a. Creating a Buffer
The Buffer class has several factory methods. The Buffer constructor itself is deprecated; use the static methods instead.
From a string:
const buf = Buffer.from('hello', 'utf8');
console.log(buf); // <Buffer 68 65 6c 6c 6f>
console.log(buf.length); // 5The second argument is the encoding. The default is 'utf8'. Other encodings include 'hex', 'base64', 'latin1', 'ascii', 'utf16le', and 'base64url' .
From an array of bytes:
const buf = Buffer.from([0x48, 0x65, 0x6c, 0x6c, 0x6f]);Each element must be an integer between 0 and 255. Values outside the range are truncated modulo 256.
From an ArrayBuffer:
const ab = new ArrayBuffer(5);
const buf = Buffer.from(ab);From an existing Buffer:
const buf2 = Buffer.from(buf); // copies the dataBuffer.from(buffer) copies the bytes. To share the memory, use Buffer.from(buffer.buffer, buffer.byteOffset, buffer.byteLength).
Allocating uninitialized memory:
const buf = Buffer.alloc(10); // filled with zeros
const buf2 = Buffer.allocUnsafe(10); // not initializedBuffer.alloc zeros the memory. Buffer.allocUnsafe does not, which is faster but can expose old data if the buffer is read before it is written. Use allocUnsafe only when the buffer will be fully overwritten before use .
From a size:
const buf = Buffer.alloc(1024);b. Reading and writing Buffer data
The Buffer class provides read and write methods for integers, floats, and strings. The integer methods have explicit endianness.
Integers:
const buf = Buffer.alloc(8);
buf.writeUInt8(0x48, 0);
buf.writeUInt16BE(0x1234, 1);
buf.writeUInt32LE(0xdeadbeef, 3);
console.log(buf.readUInt8(0)); // 0x48
console.log(buf.readUInt16BE(1)); // 0x1234
console.log(buf.readUInt32LE(3)); // 0xdeadbeefThe BE and LE suffixes mean big-endian and little-endian. The numeric argument is the offset in bytes.
Floats:
const buf = Buffer.alloc(8);
buf.writeFloatBE(3.14, 0);
buf.writeDoubleLE(2.71828, 4);
console.log(buf.readFloatBE(0)); // 3.14
console.log(buf.readDoubleLE(4)); // 2.71828Strings:
const buf = Buffer.alloc(20);
buf.write('hello', 0, 'utf8');
buf.write('world', 6, 'utf8');
console.log(buf.toString('utf8', 0, 5)); // 'hello'
console.log(buf.toString('utf8', 6, 11)); // 'world'The toString method takes the encoding and optional start and end offsets.
BigInt:
const buf = Buffer.alloc(8);
buf.writeBigUInt64BE(12345678901234567890n, 0);
console.log(buf.readBigUInt64BE(0));The BigInt methods handle 64-bit integers that exceed the safe range for JavaScript numbers.
c. The TypedArray family
Buffer is a Uint8Array. The TypedArray family includes:
| Type | Element size | Range |
|---|---|---|
Uint8Array | 1 byte | 0 to 255 |
Int8Array | 1 byte | -128 to 127 |
Uint16Array | 2 bytes | 0 to 65535 |
Int16Array | 2 bytes | -32768 to 32767 |
Uint32Array | 4 bytes | 0 to 4294967295 |
Int32Array | 4 bytes | -2147483648 to 2147483647 |
Float32Array | 4 bytes | 32-bit float |
Float64Array | 8 bytes | 64-bit float |
BigInt64Array | 8 bytes | 64-bit signed BigInt |
BigUint64Array | 8 bytes | 64-bit unsigned BigInt |
Each type has a constructor that creates a new array or a view into an existing ArrayBuffer.
const u8 = new Uint8Array(4); // 4 bytes, zeroed
const i32 = new Int32Array(2); // 8 bytes, zeroed
const f64 = new Float64Array(1); // 8 bytes, zeroedThe length property is the number of elements, and the byteLength property is the number of bytes.
const u32 = new Uint32Array(4);
console.log(u32.length); // 4
console.log(u32.byteLength); // 16Typed arrays have methods for iteration, mapping, and transformation, similar to regular arrays but optimized for numeric data.
d. ArrayBuffer and views
An ArrayBuffer is a fixed-length block of raw memory. It has no type and no methods for reading or writing; it is just a buffer.
const ab = new ArrayBuffer(16);
console.log(ab.byteLength); // 16A typed array is a view into an ArrayBuffer. Multiple views can share the same buffer, which is how the same bytes can be read as different types.
const ab = new ArrayBuffer(4);
const u8 = new Uint8Array(ab);
const u16 = new Uint16Array(ab);
const u32 = new Uint32Array(ab);
u8[0] = 0x01;
u8[1] = 0x02;
u8[2] = 0x03;
u8[3] = 0x04;
console.log(u16[0]); // 0x0201 or 0x0102, depending on endianness
console.log(u32[0]); // 0x04030201 or 0x01020304The u16 view reads two bytes as one 16-bit value, and the u32 view reads four bytes as one 32-bit value. The values depend on the platform’s byte order.
A view can be offset and length-limited:
const ab = new ArrayBuffer(16);
const view = new Uint8Array(ab, 4, 8); // offset 4, length 8
console.log(view.byteOffset); // 4
console.log(view.length); // 8The byteOffset is the index in the buffer where the view starts, and the length is the number of elements.
A Buffer can be created over an ArrayBuffer without copying:
const ab = new ArrayBuffer(8);
const buf = Buffer.from(ab);The Buffer.from(ab) creates a view that shares the memory. Modifications through the buffer are visible through any typed array over the same buffer .
e. DataView for explicit endianness
The DataView is a view that provides methods for reading and writing multi-byte values with explicit endianness. It does not have the aliasing behavior of typed arrays; it reads and writes at a specified byte offset.
const ab = new ArrayBuffer(8);
const view = new DataView(ab);
view.setUint16(0, 0x1234, false); // big-endian
view.setUint32(2, 0xdeadbeef, true); // little-endian
console.log(view.getUint16(0, false)); // 0x1234
console.log(view.getUint32(2, true)); // 0xdeadbeefThe third argument is the littleEndian boolean. false means big-endian, and true means little-endian. This explicit choice is the advantage of DataView over typed arrays: the byte order is part of the call, not the platform.
The DataView methods include getInt8, getUint8, getInt16, getUint16, getInt32, getUint32, getFloat32, getFloat64, getBigInt64, and getBigUint64, each with a corresponding set method.
DataView is the right tool for parsing binary formats that specify the byte order, such as network protocols, file headers, and serialization formats.
f. Encoding and decoding
The Buffer class handles text encodings. The toString method decodes a buffer to a string, and the Buffer.from method encodes a string to a buffer.
UTF-8:
const buf = Buffer.from('hello', 'utf8');
console.log(buf.toString('utf8')); // 'hello'Hex:
const buf = Buffer.from('deadbeef', 'hex');
console.log(buf); // <Buffer de ad be ef>
console.log(buf.toString('hex')); // 'deadbeef'The hex encoding is commonly used for cryptographic hashes, identifiers, and human-readable binary data. Each byte is two hex characters.
Base64:
const buf = Buffer.from('hello', 'utf8');
console.log(buf.toString('base64')); // 'aGVsbG8='
const decoded = Buffer.from('aGVsbG8=', 'base64');
console.log(decoded.toString('utf8')); // 'hello'Base64 is used for encoding binary data in text contexts, such as JSON payloads and data URLs. The base64url variant uses URL-safe characters.
Latin1:
const buf = Buffer.from([0xe9, 0xe8]);
console.log(buf.toString('latin1')); // 'éè'Latin1 (also called ISO-8859-1) maps each byte to a character in the range U+0000 to U+00FF.
The TextEncoder and TextDecoder classes are the Web standard alternatives:
const encoder = new TextEncoder();
const buf = encoder.encode('hello'); // Uint8Array
const decoder = new TextDecoder('utf8');
console.log(decoder.decode(buf)); // 'hello'The TextEncoder produces a Uint8Array, and the TextDecoder accepts an ArrayBufferView. These are portable across Node.js and browsers.
Complete Example Session
// ============================================
// PART 1: CREATE A BUFFER FROM A STRING
// ============================================
const buf = Buffer.from('hello', 'utf8');
console.log(buf); // <Buffer 68 65 6c 6c 6f>
console.log(buf.length); // 5// ============================================
// PART 2: CREATE A BUFFER FROM BYTES
// ============================================
const bytes = Buffer.from([0x48, 0x65, 0x6c, 0x6c, 0x6f]);
console.log(bytes.toString('utf8')); // 'Hello'// ============================================
// PART 3: ALLOCATE A BUFFER
// ============================================
const buf = Buffer.alloc(10);
console.log(buf); // <Buffer 00 00 00 00 00 00 00 00 00 00>
const unsafe = Buffer.allocUnsafe(10);
console.log(unsafe.length); // 10// ============================================
// PART 4: WRITE AND READ INTEGERS
// ============================================
const buf = Buffer.alloc(8);
buf.writeUInt8(0x48, 0);
buf.writeUInt16BE(0x1234, 1);
buf.writeUInt32LE(0xdeadbeef, 3);
console.log(buf.readUInt8(0)); // 0x48
console.log(buf.readUInt16BE(1)); // 0x1234
console.log(buf.readUInt32LE(3)); // 0xdeadbeef// ============================================
// PART 5: WRITE AND READ FLOATS
// ============================================
const buf = Buffer.alloc(8);
buf.writeFloatBE(3.14, 0);
buf.writeDoubleLE(2.71828, 4);
console.log(buf.readFloatBE(0)); // 3.14
console.log(buf.readDoubleLE(4)); // 2.71828// ============================================
// PART 6: TYPED ARRAYS
// ============================================
const u8 = new Uint8Array(4);
const i32 = new Int32Array(2);
const f64 = new Float64Array(1);
console.log(u8.length); // 4
console.log(i32.byteLength); // 8
console.log(f64.byteLength); // 8// ============================================
// PART 7: SHARED ARRAYBUFFER
// ============================================
const ab = new ArrayBuffer(4);
const u8 = new Uint8Array(ab);
const u32 = new Uint32Array(ab);
u8[0] = 0x01;
u8[1] = 0x02;
u8[2] = 0x03;
u8[3] = 0x04;
console.log(u32[0]); // platform-dependent// ============================================
// PART 8: DATAVIEW WITH EXPLICIT ENDIANNESS
// ============================================
const ab = new ArrayBuffer(8);
const view = new DataView(ab);
view.setUint16(0, 0x1234, false); // big-endian
view.setUint32(2, 0xdeadbeef, true); // little-endian
console.log(view.getUint16(0, false)); // 0x1234
console.log(view.getUint32(2, true)); // 0xdeadbeef// ============================================
// PART 9: HEX AND BASE64
// ============================================
const buf = Buffer.from('deadbeef', 'hex');
console.log(buf); // <Buffer de ad be ef>
console.log(buf.toString('hex')); // 'deadbeef'
const b64 = Buffer.from('hello', 'utf8').toString('base64');
console.log(b64); // 'aGVsbG8='
console.log(Buffer.from(b64, 'base64').toString('utf8')); // 'hello'// ============================================
// PART 10: TEXTENCODER AND TEXTDECODER
// ============================================
const encoder = new TextEncoder();
const decoder = new TextDecoder('utf8');
const encoded = encoder.encode('hello');
console.log(encoded); // Uint8Array(5) [104, 101, 108, 108, 111]
console.log(decoder.decode(encoded)); // 'hello'These ten parts cover creating a buffer from a string, creating a buffer from bytes, allocating a buffer, reading and writing integers, reading and writing floats, typed arrays, shared ArrayBuffer, DataView with explicit endianness, hex and base64 encoding, and TextEncoder and TextDecoder.
Quick Reference
Buffer Creation
| Method | Purpose |
|---|---|
Buffer.from(string, encoding) | From a string |
Buffer.from(array) | From byte values |
Buffer.from(arrayBuffer) | View over an ArrayBuffer |
Buffer.from(buffer) | Copy of a buffer |
Buffer.alloc(n) | Zeroed buffer of size n |
Buffer.allocUnsafe(n) | Uninitialized buffer |
Buffer Read/Write
| Method | Type |
|---|---|
readUInt8 / writeUInt8 | 8-bit unsigned |
readInt16BE / writeInt16BE | 16-bit signed big-endian |
readUInt32LE / writeUInt32LE | 32-bit unsigned little-endian |
readFloatBE / writeFloatBE | 32-bit float |
readDoubleLE / writeDoubleLE | 64-bit float |
readBigUInt64BE | 64-bit unsigned BigInt |
toString(encoding) | Decode to string |
Typed Arrays
| Type | Bytes per Element |
|---|---|
Uint8Array | 1 |
Int8Array | 1 |
Uint16Array | 2 |
Int16Array | 2 |
Uint32Array | 4 |
Int32Array | 4 |
Float32Array | 4 |
Float64Array | 8 |
BigInt64Array | 8 |
Encodings
| Encoding | Purpose |
|---|---|
utf8 | Text (default) |
hex | Two chars per byte |
base64 | Text-safe binary |
base64url | URL-safe base64 |
latin1 | Byte-to-char mapping |
utf16le | UTF-16 little-endian |
ascii | 7-bit ASCII |
DataView Methods
| Method | Type |
|---|---|
getUint8 / setUint8 | 8-bit unsigned |
getInt16 / setInt16 | 16-bit signed |
getUint32 / setUint32 | 32-bit unsigned |
getFloat32 / setFloat32 | 32-bit float |
getFloat64 / setFloat64 | 64-bit float |
getBigUint64 / setBigUint64 | 64-bit BigInt |
Best Practices
✅ Do This:
// Use Buffer.from for creation
const buf = Buffer.from('hello', 'utf8');
// Use Buffer.alloc for zeroed memory
const safe = Buffer.alloc(10);
// Use explicit endianness
buf.writeUInt32BE(value, 0);
buf.writeUInt32LE(value, 0);
// Use DataView for multi-byte parsing
const view = new DataView(ab);
view.getUint16(0, false);
// Use hex for hashes and identifiers
const hex = buf.toString('hex');
// Use TextEncoder for portable code
const encoded = new TextEncoder().encode('hello');❌ Don’t Do This:
// Use the deprecated Buffer constructor
const buf = new Buffer(10); // ❌ use Buffer.alloc
// Use allocUnsafe without writing first
const buf = Buffer.allocUnsafe(10); // ❌ may expose old data
// Assume endianness
buf.writeUInt32(value, 0); // ❌ does not exist
// Mix up byte and element offsets
const u32 = new Uint32Array(ab, 2); // ❌ offset must be a multiple of 4
// Concatenate strings for binary data
const data = '\xff\xfe'; // ❌ use a Buffer
// Use latin1 for arbitrary binary
const text = buf.toString('latin1'); // ⚠️ may lose dataCommon Pitfalls
| Pitfall | Why It Happens | Fix |
|---|---|---|
allocUnsafe exposes data | Buffer not written before read | Use alloc or write first |
| Endianness mismatch | Assumed the wrong byte order | Use BE/LE explicitly |
| Alignment error | Typed array offset not a multiple | Align the offset |
RangeError on read | Offset beyond the buffer | Check byteLength |
| Unicode string length | UTF-16 length versus byte length | Use Buffer.byteLength |
| Incorrect base64 | Used the wrong variant | Use base64url for URLs |
| Shared memory surprise | Views share the same buffer | Copy if independence is needed |
Real-World Examples
1. Read a File
const data = await readFile('image.png'); // Buffer2. Hex Hash
const hash = createHash('sha256').update('data').digest('hex');3. Base64 Image
const b64 = data.toString('base64');
const url = `data:image/png;base64,${b64}`;4. Parse a Header
const view = new DataView(buffer);
const version = view.getUint8(0);
const length = view.getUint16(1, false);5. Write a Protocol Message
const buf = Buffer.alloc(8);
buf.writeUInt32BE(0xdeadbeef, 0);
buf.writeUInt32LE(0x12345678, 4);6. Convert to Uint8Array
const u8 = new Uint8Array(buffer.buffer, buffer.byteOffset, buffer.byteLength);7. Concatenate Buffers
const combined = Buffer.concat([buf1, buf2]);8. Compare Buffers
if (buf1.equals(buf2)) { }9. Slice a Buffer
const slice = buf.subarray(0, 5);10. Generate Random Bytes
const random = randomBytes(16);Visual
Buffer, ArrayBuffer, and Views
┌──────────────────────────────────────────────────────────────┐
│ ArrayBuffer (raw memory) │
│ ┌────┬────┬────┬────┐ │
│ │ 01 │ 02 │ 03 │ 04 │ │
│ └────┴────┴────┴────┘ │
│ ▲ ▲ │
│ │ │ │
│ Uint8Array Uint32Array │
│ └── 4 elements, 4 bytes │
│ └── 1 element, 4 bytes │
│ │
│ Multiple views share the same memory. │
└──────────────────────────────────────────────────────────────┘Buffer Encoding Flow
┌──────────────────────────────────────────────────────────────┐
│ String → Buffer: │
│ Buffer.from('hello', 'utf8') │
│ └── <Buffer 68 65 6c 6c 6f> │
│ │
│ Buffer → String: │
│ buf.toString('hex') │
│ └── '68656c6c6f' │
│ │
│ Buffer → Base64: │
│ buf.toString('base64') │
│ └── 'aGVsbG8=' │
└──────────────────────────────────────────────────────────────┘Endianness
┌──────────────────────────────────────────────────────────────┐
│ VALUE: 0x12345678 │
│ │
│ BIG-ENDIAN (network byte order): │
│ ┌────┬────┬────┬────┐ │
│ │ 12 │ 34 │ 56 │ 78 │ │
│ └────┴────┴────┴────┘ │
│ │
│ LITTLE-ENDIAN (x86, ARM): │
│ ┌────┬────┬────┬────┐ │
│ │ 78 │ 56 │ 34 │ 12 │ │
│ └────┴────┴────┴────┘ │
│ │
│ Use the BE/LE suffix explicitly. │
└──────────────────────────────────────────────────────────────┘DataView vs TypedArray
┌──────────────────────────────────────────────────────────────┐
│ TYPED ARRAY: │
│ u32[0] = 0x12345678; │
│ └── Uses platform endianness │
│ └── Offset must be aligned to element size │
│ │
│ DATAVIEW: │
│ view.setUint32(0, 0x12345678, false); // big-endian │
│ view.setUint32(0, 0x12345678, true); // little-endian │
│ └── Explicit endianness │
│ └── Any offset │
└──────────────────────────────────────────────────────────────┘Summary
| Item | Value |
|---|---|
| Buffer | Node.js binary type, subclass of Uint8Array |
| ArrayBuffer | Raw fixed-length memory block |
| TypedArray | Typed view into an ArrayBuffer |
| DataView | View with explicit endianness |
| Creation | Buffer.from, Buffer.alloc, Buffer.allocUnsafe |
| Read/write | readUInt8, writeUInt32BE, readDoubleLE |
| Endianness | BE (big) and LE (little) |
| Encodings | utf8, hex, base64, base64url, latin1 |
| Alignment | Typed array offset must be a multiple of element size |
| TextEncoder | Portable encode to Uint8Array |
| TextDecoder | Portable decode from a typed array |
Key takeaways:
Bufferis aUint8Arraywith Node-specific methods. It is the original binary type in Node.js, and it is what every I/O API returns and accepts.- All typed arrays are views into an
ArrayBuffer. Multiple views can share the same memory, which allows the same bytes to be read as different numeric types. - The
DataViewprovides explicit endianness. Use it for parsing binary formats that specify the byte order, such as network protocols and file headers. - The
BEandLEsuffixes on the Buffer methods specify endianness.readUInt32BEreads big-endian, andreadUInt32LEreads little-endian. Buffer.fromcreates a buffer from a string, array, or another buffer.Buffer.alloczeros the memory, andBuffer.allocUnsafedoes not.- The
Bufferclass handles text encodings.utf8,hex,base64,base64url, andlatin1are the common ones. TextEncoderandTextDecoderare the portable alternatives. They produce and accept typed arrays, so the same code works in Node.js and the browser.
Remember: JavaScript’s text-oriented types are not suitable for binary data. The Buffer and TypedArray family exists to represent raw bytes, and the ArrayBuffer is the shared memory that all views are built on. The Buffer class adds Node-specific read and write methods with explicit endianness. The DataView provides the same control at any offset. The encoding functions convert between bytes and text. Use the typed arrays when the data is numeric and the layout is uniform, use the DataView when the byte order must be explicit, and use the Buffer when the data is text or when the Node.js API requires it.
Stop using slow, ad-bloated tool sites! 🤮
🔎 Search “KandZ Tools” on Google to use many professional utilities for free.
KandZ.me is the ultimate minimalist hub for:
✅ Finance (Mortgage, Interest, Inflation)
✅ Tech (Base64, JSON, Dev Suite, IP)
✅ Health (BMI, BMR, TDEE)
✅ Productivity (Timer, Workspace, QR)
⚡️ Fast & Private
🔒 No data leaves your device
💎 100% Free
🔗 Use it now: https://tools.kandz.me
🔖 Bookmark it—you’ll need it later!