The fs module (File System module) is a built-in Node.js module that lets you work with files and directories on your computer.

Think of the fs module as a set of tools for reading, writing, and managing files. It's like having a digital file cabinet where you can store, retrieve, and organize your data.

Basic usage:

const fs = require('fs');

What you can do with fs module:

• Read files
• Write files
• Create and delete directories
• Rename files and directories
• Check file information
• Watch files for changes
• Set file permissions

Example - reading a file:

const fs = require('fs');

fs.readFile('example.txt', 'utf8', (error, data) => {
    if (error) {
        console.log('Error reading file:', error);
        return;
    }
    console.log('File content:', data);
});

The fs module has both synchronous (blocking) and asynchronous (non-blocking) methods. Asynchronous methods are preferred because they don't stop your program while working with files.

This module is essential for any Node.js application that needs to work with files, like web servers, data processing tools, or configuration managers.

Reading files asynchronously means your program doesn't wait for the file to be read. It continues doing other work and gets notified when the file is ready.

Basic async file reading:

const fs = require('fs');

fs.readFile('data.txt', 'utf8', (error, data) => {
    if (error) {
        console.log('Error reading file:', error);
        return;
    }
    console.log('File content:', data);
});

console.log('This runs immediately, before file reading completes');

Output would be:

This runs immediately, before file reading completes
File content: [file contents]

With error handling:

const fs = require('fs');

fs.readFile('missing-file.txt', 'utf8', (error, data) => {
    if (error) {
        if (error.code === 'ENOENT') {
            console.log('File does not exist');
        } else {
            console.log('Other error:', error);
        }
        return;
    }
    console.log('File content:', data);
});

Reading binary files (images, etc.):

const fs = require('fs');

fs.readFile('image.jpg', (error, data) => {
    if (error) throw error;
    // data is a Buffer object
    console.log('Image size:', data.length, 'bytes');
});

Key points about async reading:

• Uses callbacks - functions that run when the operation completes
• Doesn't block your program - other code can run while waiting
• Essential for web servers - can handle multiple requests
• More complex error handling with callbacks

Async file reading is the preferred method in Node.js because it keeps your application responsive.

Reading files synchronously means your program waits until the file is completely read before continuing. It's simpler but blocks other operations.

Basic sync file reading:

const fs = require('fs');

try {
    const data = fs.readFileSync('data.txt', 'utf8');
    console.log('File content:', data);
} catch (error) {
    console.log('Error reading file:', error);
}

console.log('This runs only after file reading completes');

Output would be:

File content: [file contents]
This runs only after file reading completes

Without try-catch (not recommended):

const fs = require('fs');

const data = fs.readFileSync('data.txt', 'utf8');
console.log('File content:', data);

Reading binary files synchronously:

const fs = require('fs');

try {
    const buffer = fs.readFileSync('image.jpg');
    console.log('Image size:', buffer.length, 'bytes');
} catch (error) {
    console.log('Error:', error);
}

When to use synchronous reading:

• Command-line tools (not web servers)
• Configuration files at startup
• Simple scripts where blocking doesn't matter
• When you want simpler, linear code

When NOT to use synchronous reading:

• Web servers (would block other requests)
• Applications that need to stay responsive
• When reading large files
• In event handlers or callbacks

Synchronous methods are easier to understand but can make your application unresponsive.

Writing to files lets you save data from your program to disk. The fs module provides several ways to write files.

Basic async file writing:

const fs = require('fs');

const content = 'Hello, this is my file content!';

fs.writeFile('output.txt', content, 'utf8', (error) => {
    if (error) {
        console.log('Error writing file:', error);
        return;
    }
    console.log('File written successfully!');
});

Synchronous file writing:

const fs = require('fs');

const content = 'Hello, this is my file content!';

try {
    fs.writeFileSync('output.txt', content, 'utf8');
    console.log('File written successfully!');
} catch (error) {
    console.log('Error writing file:', error);
}

Writing JSON data:

const fs = require('fs');

const user = {
    name: 'John Doe',
    age: 30,
    email: 'john@example.com'
};

fs.writeFile('user.json', JSON.stringify(user, null, 2), 'utf8', (error) => {
    if (error) throw error;
    console.log('JSON file written!');
});

Important notes about writeFile:

• Creates the file if it doesn't exist
• Overwrites the file if it already exists
• Creates directories in the path if needed
• Always handle errors - writing can fail for many reasons

Common error scenarios:

• No permission to write to the location
• Disk is full
• Path contains invalid characters
• File is currently open by another program

File writing is essential for saving application data, logs, user uploads, and configuration files.

Appending to a file means adding content to the end of an existing file without deleting what's already there. It's perfect for logs and data collection.

Basic async file appending:

const fs = require('fs');

fs.appendFile('log.txt', 'New log entry\n', 'utf8', (error) => {
    if (error) {
        console.log('Error appending to file:', error);
        return;
    }
    console.log('Content appended!');
});

Synchronous file appending:

const fs = require('fs');

try {
    fs.appendFileSync('log.txt', 'New log entry\n', 'utf8');
    console.log('Content appended!');
} catch (error) {
    console.log('Error:', error);
}

Appending multiple times:

const fs = require('fs');

function logMessage(message) {
    const timestamp = new Date().toISOString();
    const logEntry = `[${timestamp}] ${message}\n`;
    
    fs.appendFile('application.log', logEntry, 'utf8', (error) => {
        if (error) {
            console.log('Logging failed:', error);
        }
    });
}

// Usage
logMessage('Application started');
logMessage('User logged in');
logMessage('Data processed');

Appending vs Writing:

• writeFile: Creates new file or overwrites existing
• appendFile: Adds to existing file or creates new one

Use cases for appendFile:

• Log files (application logs, error logs)
• Data collection (sensor readings, user actions)
• Chat applications (message history)
• Progress tracking (download logs, processing status)

Performance consideration:

For high-frequency appends (like logging), consider using streams or buffering to reduce disk I/O operations.

Appending is much safer than writing when you want to preserve existing content.

Deleting files (also called "unlinking") removes them from the filesystem. Be careful - deleted files are usually gone forever!

Basic async file deletion:

const fs = require('fs');

fs.unlink('file-to-delete.txt', (error) => {
    if (error) {
        console.log('Error deleting file:', error);
        return;
    }
    console.log('File deleted successfully');
});

Synchronous file deletion:

const fs = require('fs');

try {
    fs.unlinkSync('file-to-delete.txt');
    console.log('File deleted');
} catch (error) {
    console.log('Error:', error);
}

Safe deletion with existence check:

const fs = require('fs');
const path = require('path');

function safeDelete(filename) {
    fs.access(filename, fs.constants.F_OK, (error) => {
        if (error) {
            console.log('File does not exist');
            return;
        }
        
        fs.unlink(filename, (error) => {
            if (error) {
                console.log('Error deleting:', error);
                return;
            }
            console.log('File deleted safely');
        });
    });
}

safeDelete('temp-file.txt');

Common deletion errors:

• ENOENT: File doesn't exist
• EACCES: No permission to delete
• EBUSY: File is in use by another program
• EISDIR: Path is a directory (use rmdir instead)

Why it's called "unlink":

In Unix systems, files are connected to the filesystem through links. Deleting a file actually means removing its link. The name stuck in Node.js.

Safety tips:

• Double-check filenames before deleting
• Consider moving to trash instead of deleting
• Implement confirmation in user applications
• Log all delete operations

File deletion is permanent in most cases, so always be cautious.

Renaming files changes their name or moves them to a different location. In Node.js, renaming and moving are the same operation.

Basic async file renaming:

const fs = require('fs');

fs.rename('old-name.txt', 'new-name.txt', (error) => {
    if (error) {
        console.log('Error renaming file:', error);
        return;
    }
    console.log('File renamed successfully');
});

Synchronous file renaming:

const fs = require('fs');

try {
    fs.renameSync('old-name.txt', 'new-name.txt');
    console.log('File renamed');
} catch (error) {
    console.log('Error:', error);
}

Moving files to different directories:

const fs = require('fs');

// Move file to different folder
fs.rename('current-file.txt', '../archive/old-file.txt', (error) => {
    if (error) {
        console.log('Error moving file:', error);
        return;
    }
    console.log('File moved successfully');
});

Safe rename with checks:

const fs = require('fs');

function safeRename(oldPath, newPath) {
    // Check if destination already exists
    fs.access(newPath, (error) => {
        if (!error) {
            console.log('Destination already exists');
            return;
        }
        
        // If we get here, destination doesn't exist
        fs.rename(oldPath, newPath, (error) => {
            if (error) {
                console.log('Error renaming:', error);
                return;
            }
            console.log('File renamed safely');
        });
    });
}

safeRename('old.txt', 'new.txt');

Common rename errors:

• ENOENT: Source file doesn't exist
• EACCES: No permission to rename
• EBUSY: File is in use
• EXDEV: Cross-device link (can't rename across drives)

Important notes:

• Renaming is atomic - it either completes fully or fails completely
• You can't rename across different drives/filesystems
• Renaming directories works the same way
• The operation is very fast - it just changes directory entries

File renaming is useful for organizing files, versioning, and moving data between locations.

Creating directories (folders) helps organize your files. The fs module provides methods to create single directories or nested directory structures.

Basic async directory creation:

const fs = require('fs');

fs.mkdir('new-folder', (error) => {
    if (error) {
        console.log('Error creating directory:', error);
        return;
    }
    console.log('Directory created successfully');
});

Synchronous directory creation:

const fs = require('fs');

try {
    fs.mkdirSync('new-folder');
    console.log('Directory created');
} catch (error) {
    console.log('Error:', error);
}

Creating nested directories:

const fs = require('fs');

// Create parent/child/grandchild structure
fs.mkdir('parent/child/grandchild', { recursive: true }, (error) => {
    if (error) {
        console.log('Error creating directories:', error);
        return;
    }
    console.log('All directories created');
});

Creating directories with specific permissions:

const fs = require('fs');

// Create directory with read/write/execute permissions for owner
fs.mkdir('secure-folder', { mode: 0o755 }, (error) => {
    if (error) throw error;
    console.log('Directory created with specific permissions');
});

Complete example with error handling:

const fs = require('fs');

function createProjectStructure() {
    const folders = [
        'project/src',
        'project/dist',
        'project/docs',
        'project/tests'
    ];
    
    folders.forEach(folder => {
        fs.mkdir(folder, { recursive: true }, (error) => {
            if (error) {
                console.log(`Error creating ${folder}:`, error);
            } else {
                console.log(`Created: ${folder}`);
            }
        });
    });
}

createProjectStructure();

Common errors:

• EEXIST: Directory already exists
• ENOENT: Parent directory doesn't exist (without recursive)
• EACCES: No permission to create directory

Key points:

• Use recursive: true for nested directories
• Directories are created with default permissions (usually 755)
• The operation is fast - just creates directory entries
• Always handle errors - creation can fail for many reasons

Directory creation is essential for organizing files in applications, temporary data, and project structures.

Removing directories deletes them from the filesystem. You need different methods for empty directories vs directories with content.

Removing empty directory (async):

const fs = require('fs');

fs.rmdir('empty-folder', (error) => {
    if (error) {
        console.log('Error removing directory:', error);
        return;
    }
    console.log('Directory removed');
});

Removing empty directory (sync):

const fs = require('fs');

try {
    fs.rmdirSync('empty-folder');
    console.log('Directory removed');
} catch (error) {
    console.log('Error:', error);
}

Removing directory with contents (Node.js 14+):

const fs = require('fs');

// Remove directory and all its contents
fs.rm('folder-with-files', { recursive: true }, (error) => {
    if (error) {
        console.log('Error:', error);
        return;
    }
    console.log('Directory and all contents removed');
});

Manual removal of directory with contents (older approach):

const fs = require('fs');
const path = require('path');

function removeDirectory(dirPath) {
    if (fs.existsSync(dirPath)) {
        fs.readdirSync(dirPath).forEach(file => {
            const curPath = path.join(dirPath, file);
            
            if (fs.lstatSync(curPath).isDirectory()) {
                // Recursively remove subdirectories
                removeDirectory(curPath);
            } else {
                // Delete file
                fs.unlinkSync(curPath);
            }
        });
        
        // Remove empty directory
        fs.rmdirSync(dirPath);
    }
}

removeDirectory('old-project');

Safe directory removal:

const fs = require('fs');

function safeRemoveDirectory(dirPath) {
    fs.stat(dirPath, (error, stats) => {
        if (error) {
            console.log('Directory does not exist');
            return;
        }
        
        if (!stats.isDirectory()) {
            console.log('Path is not a directory');
            return;
        }
        
        fs.rm(dirPath, { recursive: true }, (error) => {
            if (error) {
                console.log('Error removing:', error);
                return;
            }
            console.log('Directory removed safely');
        });
    });
}

safeRemoveDirectory('temp-data');

Important warnings:

• Recursive removal is permanent and can't be undone
• Double-check the path before deleting
• Consider moving to trash instead of deleting
• Test removal code carefully

Directory removal is useful for cleaning up temporary files, cache directories, and old project versions.

fs.promises is a modern version of the fs module that uses Promises instead of callbacks. It's cleaner and works well with async/await.

Using fs.promises:

const fs = require('fs').promises;

Reading files with promises:

const fs = require('fs').promises;

async function readFile() {
    try {
        const data = await fs.readFile('data.txt', 'utf8');
        console.log('File content:', data);
    } catch (error) {
        console.log('Error reading file:', error);
    }
}

readFile();

Writing files with promises:

const fs = require('fs').promises;

async function writeFile() {
    try {
        await fs.writeFile('output.txt', 'Hello World', 'utf8');
        console.log('File written!');
    } catch (error) {
        console.log('Error:', error);
    }
}

writeFile();

Multiple file operations with promises:

const fs = require('fs').promises;

async function processFiles() {
    try {
        // Read multiple files
        const [file1, file2] = await Promise.all([
            fs.readFile('file1.txt', 'utf8'),
            fs.readFile('file2.txt', 'utf8')
        ]);
        
        // Process the data
        const result = file1 + file2;
        
        // Write result
        await fs.writeFile('combined.txt', result, 'utf8');
        console.log('Files processed successfully');
        
    } catch (error) {
        console.log('Error processing files:', error);
    }
}

processFiles();

Comparing callback vs promise style:

// Callback style (old way)
fs.readFile('file.txt', 'utf8', (error, data) => {
    if (error) throw error;
    console.log(data);
});

// Promise style (modern way)
fs.promises.readFile('file.txt', 'utf8')
    .then(data => console.log(data))
    .catch(error => console.log(error));

// Async/await style (cleanest)
async function readFile() {
    try {
        const data = await fs.promises.readFile('file.txt', 'utf8');
        console.log(data);
    } catch (error) {
        console.log(error);
    }
}

Benefits of fs.promises:

• Cleaner code with async/await
• Better error handling with try/catch
• Easier to chain multiple operations
• Modern JavaScript standard

Available methods in fs.promises:

• readFile(), writeFile(), appendFile()
• mkdir(), rmdir(), rm()
• rename(), unlink()
• stat(), readdir()
• And many more - most fs methods have promise versions

For new projects, fs.promises is the recommended way to work with files in Node.js.

Checking file existence is a common operation before reading, writing, or deleting files. There are several ways to do this in Node.js.

Using fs.access() (recommended):

const fs = require('fs');

fs.access('file.txt', fs.constants.F_OK, (error) => {
    if (error) {
        console.log('File does not exist');
    } else {
        console.log('File exists');
    }
});

Using fs.promises.access():

const fs = require('fs').promises;

async function checkFile() {
    try {
        await fs.access('file.txt');
        console.log('File exists');
    } catch (error) {
        console.log('File does not exist');
    }
}

checkFile();

Checking different permissions:

const fs = require('fs');

fs.access('file.txt', fs.constants.F_OK | fs.constants.R_OK, (error) => {
    if (error) {
        console.log('File does not exist or is not readable');
    } else {
        console.log('File exists and is readable');
    }
});

Common permission constants:

• fs.constants.F_OK - File exists
• fs.constants.R_OK - File is readable
• fs.constants.W_OK - File is writable
• fs.constants.X_OK - File is executable

Why NOT to use fs.exists() (deprecated):

// DON'T DO THIS - it's deprecated
if (fs.existsSync('file.txt')) {
    // Race condition: file might be deleted between check and use
    fs.readFile('file.txt', (error, data) => {
        // Could get error here if file was deleted
    });
}

Better pattern - try to use the file directly:

const fs = require('fs');

// Instead of checking first, just try the operation
fs.readFile('file.txt', 'utf8', (error, data) => {
    if (error) {
        if (error.code === 'ENOENT') {
            console.log('File does not exist');
        } else {
            console.log('Other error:', error);
        }
        return;
    }
    console.log('File content:', data);
});

Key points:

• Use fs.access() for checking existence/permissions
• Don't use deprecated fs.exists()
• Often better to try the operation and handle errors
• Race conditions can occur between check and use

File existence checking is essential for validation, security, and error prevention.

Copying files creates a duplicate of a file in a new location. Node.js provides several methods for file copying, each with different use cases.

Using fs.copyFile() (simple copying):

const fs = require('fs');

fs.copyFile('source.txt', 'destination.txt', (error) => {
    if (error) {
        console.log('Error copying file:', error);
        return;
    }
    console.log('File copied successfully');
});

Using fs.promises.copyFile():

const fs = require('fs').promises;

async function copyFile() {
    try {
        await fs.copyFile('source.jpg', 'backup.jpg');
        console.log('File copied');
    } catch (error) {
        console.log('Error:', error);
    }
}

copyFile();

Copying with streams (for large files):

const fs = require('fs');

function copyLargeFile(source, destination) {
    return new Promise((resolve, reject) => {
        const readStream = fs.createReadStream(source);
        const writeStream = fs.createWriteStream(destination);
        
        readStream.on('error', reject);
        writeStream.on('error', reject);
        writeStream.on('finish', resolve);
        
        readStream.pipe(writeStream);
    });
}

// Usage
copyLargeFile('large-video.mp4', 'video-backup.mp4')
    .then(() => console.log('Large file copied'))
    .catch(error => console.log('Error:', error));

Copying with progress tracking:

const fs = require('fs');

function copyWithProgress(source, destination) {
    return new Promise((resolve, reject) => {
        const readStream = fs.createReadStream(source);
        const writeStream = fs.createWriteStream(destination);
        
        let bytesCopied = 0;
        const fileSize = fs.statSync(source).size;
        
        readStream.on('data', (chunk) => {
            bytesCopied += chunk.length;
            const percent = ((bytesCopied / fileSize) * 100).toFixed(1);
            console.log(`Copy progress: ${percent}%`);
        });
        
        readStream.on('error', reject);
        writeStream.on('error', reject);
        writeStream.on('finish', resolve);
        
        readStream.pipe(writeStream);
    });
}

Copy modes (preserving timestamps):

const fs = require('fs');

// COPYFILE_EXCL - fails if destination exists
// COPYFILE_FICLONE - try to use copy-on-write (if supported)
// COPYFILE_FICLONE_FORCE - use copy-on-write (fail if not supported)
fs.copyFile('source.txt', 'destination.txt', fs.constants.COPYFILE_EXCL, (error) => {
    if (error) {
        console.log('Error:', error);
        return;
    }
    console.log('File copied (exclusive)');
});

When to use each method:

• fs.copyFile(): Small to medium files, simple copying
• Streams: Large files, progress tracking, memory efficiency
• Manual read/write: When you need to process data during copy

File copying is essential for backups, data processing, and file management operations.

Streams are one of the most powerful concepts in Node.js. They allow you to handle data in chunks as it becomes available, rather than loading everything into memory at once.

Think of streams like a water pipe:

• Water flows continuously from source to destination
• You don't need to store all the water in a giant tank
• You can process water as it flows through the pipe

Why streams are important:

// Without streams - loads entire file into memory
fs.readFile('huge-file.txt', (error, data) => {
    // All 2GB of data is now in memory!
    console.log(data.length);
});

// With streams - processes data in chunks
const stream = fs.createReadStream('huge-file.txt');
stream.on('data', (chunk) => {
    // Only small chunks in memory at a time
    console.log('Received chunk:', chunk.length, 'bytes');
});

Types of streams:

• Readable: Data source (files, HTTP requests)
• Writable: Data destination (files, HTTP responses)
• Duplex: Both readable and writable (sockets)
• Transform: Modify data as it flows (compression, encryption)

Real-world stream examples:

// Video streaming
const videoStream = fs.createReadStream('movie.mp4');
response.writeHead(200, { 'Content-Type': 'video/mp4' });
videoStream.pipe(response);

// Data processing pipeline
fs.createReadStream('data.csv')
    .pipe(csvParser())        // Parse CSV
    .pipe(dataFilter())       // Filter rows
    .pipe(jsonConverter())    // Convert to JSON
    .pipe(fs.createWriteStream('output.json'));

Stream events:

const stream = fs.createReadStream('file.txt');

stream.on('data', (chunk) => {
    console.log('Data chunk:', chunk.length);
});

stream.on('end', () => {
    console.log('No more data');
});

stream.on('error', (error) => {
    console.log('Error:', error);
});

stream.on('close', () => {
    console.log('Stream closed');
});

Benefits of streams:

• Memory efficiency: Process data in chunks
• Time efficiency: Start processing immediately
• Composability: Chain operations with pipe()
• Backpressure handling: Automatic flow control

Streams are fundamental to Node.js performance and are used everywhere - from file operations to HTTP servers to data processing.

Readable streams are used to read data from a source. They emit data events as chunks become available and are perfect for handling large files or continuous data.

Creating a readable stream from a file:

const fs = require('fs');

const readableStream = fs.createReadStream('large-file.txt', 'utf8');

readableStream.on('data', (chunk) => {
    console.log('Received chunk:', chunk.length, 'characters');
    console.log('Chunk content:', chunk.substring(0, 50) + '...');
});

readableStream.on('end', () => {
    console.log('Finished reading file');
});

readableStream.on('error', (error) => {
    console.log('Error reading file:', error);
});

Stream options for better control:

const fs = require('fs');

const stream = fs.createReadStream('large-file.txt', {
    encoding: 'utf8',
    highWaterMark: 64 * 1024, // 64KB chunks
    start: 0,                 // Start reading from byte 0
    end: 1024 * 1024          // Read only first 1MB
});

stream.on('data', (chunk) => {
    console.log('Chunk size:', chunk.length);
});

Creating a custom readable stream:

const { Readable } = require('stream');

class NumberStream extends Readable {
    constructor(max) {
        super();
        this.current = 1;
        this.max = max;
    }
    
    _read() {
        if (this.current > this.max) {
            this.push(null); // No more data
        } else {
            const number = this.current.toString();
            this.push(number + '\n');
            this.current++;
        }
    }
}

// Usage
const numberStream = new NumberStream(10);
numberStream.pipe(process.stdout);

Using readable streams with async iteration (Node.js 10+):

const fs = require('fs');

async function processFile() {
    const stream = fs.createReadStream('data.txt', 'utf8');
    
    for await (const chunk of stream) {
        console.log('Processing chunk:', chunk.length);
        // Process each chunk
    }
    
    console.log('File processing complete');
}

processFile().catch(console.error);

Pausing and resuming streams:

const fs = require('fs');

const stream = fs.createReadStream('data.txt', 'utf8');

stream.on('data', (chunk) => {
    console.log('Received data');
    
    // Pause stream to process data slowly
    stream.pause();
    
    setTimeout(() => {
        console.log('Processing complete, resuming...');
        stream.resume();
    }, 1000);
});

Common readable stream events:

• data - emitted when data is available
• end - emitted when no more data
• error - emitted on error
• close - emitted when stream is closed
• readable - emitted when data can be read

Readable streams are essential for efficient data processing, especially with large files or continuous data sources.

Writable streams are used to write data to a destination. They receive data and handle the writing process, managing backpressure and errors automatically.

Creating a writable stream to a file:

const fs = require('fs');

const writableStream = fs.createWriteStream('output.txt', 'utf8');

writableStream.write('First line\n');
writableStream.write('Second line\n');
writableStream.write('Third line\n');
writableStream.end('Final line\n');

Using events with writable streams:

const fs = require('fs');

const stream = fs.createWriteStream('log.txt', 'utf8');

stream.on('finish', () => {
    console.log('All data has been written');
});

stream.on('error', (error) => {
    console.log('Error writing:', error);
});

stream.on('drain', () => {
    console.log('Buffer is empty, ready for more data');
});

// Write some data
stream.write('Log entry 1\n');
stream.write('Log entry 2\n');
stream.end('Final entry\n');

Creating a custom writable stream:

const { Writable } = require('stream');

class LoggerStream extends Writable {
    constructor(options) {
        super(options);
    }
    
    _write(chunk, encoding, callback) {
        const message = chunk.toString().trim();
        console.log(`[LOG] ${new Date().toISOString()}: ${message}`);
        callback(); // Signal that writing is complete
    }
}

// Usage
const logger = new LoggerStream();
logger.write('Application started');
logger.write('User logged in');
logger.end('Application stopped');

Handling backpressure with drain event:

const fs = require('fs');

function writeLargeData(data, writableStream) {
    let i = 0;
    
    function write() {
        let ok = true;
        
        while (i < data.length && ok) {
            // Try to write next chunk
            ok = writableStream.write(data[i]);
            i++;
        }
        
        if (i < data.length) {
            // Had to stop early - wait for drain
            writableStream.once('drain', write);
        } else {
            // All data written
            writableStream.end();
        }
    }
    
    write();
}

const stream = fs.createWriteStream('large-output.txt');
const data = Array(1000).fill().map((_, i) => `Line ${i}\n`);
writeLargeData(data, stream);

Writable stream options:

const fs = require('fs');

const stream = fs.createWriteStream('file.txt', {
    encoding: 'utf8',
    flags: 'a', // Append mode
    highWaterMark: 16 * 1024 // 16KB buffer
});

Common flags for file streams:

• 'w' - Write (default, creates or overwrites)
• 'a' - Append (creates or appends)
• 'wx' - Write exclusive (fails if file exists)
• 'ax' - Append exclusive (fails if file exists)

Writable streams are essential for efficient data writing, especially when dealing with large amounts of data or continuous output.

Duplex streams are streams that are both readable and writable. They represent a two-way communication channel where data flows in both directions independently.

Think of duplex streams like a telephone:

• You can talk (write) and listen (read) at the same time
• Both directions work independently
• Common in network communication

Common examples of duplex streams:

• TCP sockets
• WebSocket connections
• STDIN/STDOUT (terminal input/output)
• Child process stdin/stdout

Creating a custom duplex stream:

const { Duplex } = require('stream');

class EchoStream extends Duplex {
    constructor(options) {
        super(options);
        this.buffer = [];
    }
    
    // Readable side
    _read(size) {
        if (this.buffer.length > 0) {
            // Push data from buffer to readable side
            this.push(this.buffer.shift());
        } else {
            // No data available
            this.push(null);
        }
    }
    
    // Writable side
    _write(chunk, encoding, callback) {
        // Echo the data back (convert to uppercase)
        const echoed = chunk.toString().toUpperCase();
        this.buffer.push(Buffer.from(echoed));
        
        // Trigger read to make data available
        this._read();
        
        callback(); // Writing complete
    }
}

// Usage
const echo = new EchoStream();
echo.on('data', (data) => {
    console.log('Received:', data.toString());
});

echo.write('hello');
echo.write('world');
echo.end();

Using TCP socket (built-in duplex stream):

const net = require('net');

const server = net.createServer((socket) => {
    // socket is a duplex stream
    console.log('Client connected');
    
    // Read data from client
    socket.on('data', (data) => {
        console.log('Received:', data.toString());
        
        // Write data back to client
        socket.write('Echo: ' + data);
    });
    
    socket.on('end', () => {
        console.log('Client disconnected');
    });
});

server.listen(3000, () => {
    console.log('Server listening on port 3000');
});

Duplex stream with transform capability:

const { Duplex } = require('stream');

class DelayStream extends Duplex {
    constructor(delayMs, options) {
        super(options);
        this.delay = delayMs;
        this.buffer = [];
    }
    
    _read(size) {
        if (this.buffer.length > 0) {
            const chunk = this.buffer.shift();
            setTimeout(() => {
                this.push(chunk);
            }, this.delay);
        } else {
            this.push(null);
        }
    }
    
    _write(chunk, encoding, callback) {
        this.buffer.push(chunk);
        callback();
    }
}

// Usage - adds delay to data flow
const delay = new DelayStream(1000); // 1 second delay
delay.on('data', (data) => {
    console.log('Delayed data:', data.toString());
});

delay.write('Hello');
delay.write('World');
delay.end();

Key characteristics of duplex streams:

• Have both readable and writable interfaces
• Both sides operate independently
• Common in network protocols
• Can be piped to and from other streams
• Manage backpressure in both directions

Duplex streams are fundamental for any bidirectional communication in Node.js, from network protocols to custom data processing pipelines.

Transform streams are a special type of duplex stream that modify or transform data as it passes through. They're perfect for data processing pipelines.

Think of transform streams like a food processor:

• Raw ingredients go in (input)
• Processed food comes out (output)
• The transformation happens inside

Common examples of transform streams:

• Compression (zlib.createGzip())
• Encryption (crypto.createCipher())
• Data parsers (JSON, CSV)
• Data validators
• Line splitters

Creating a custom transform stream:

const { Transform } = require('stream');

class UpperCaseTransform extends Transform {
    _transform(chunk, encoding, callback) {
        // Transform the data
        const upperChunk = chunk.toString().toUpperCase();
        
        // Push the transformed data
        this.push(upperChunk);
        
        // Signal that transformation is complete
        callback();
    }
}

// Usage
const upperCase = new UpperCaseTransform();

// Pipe data through the transform
process.stdin
    .pipe(upperCase)
    .pipe(process.stdout);

// Type in terminal, see uppercase output

More complex transform - CSV to JSON:

const { Transform } = require('stream');

class CSVToJSONTransform extends Transform {
    constructor(options) {
        super({ ...options, objectMode: true });
        this.headers = null;
        this.isFirstChunk = true;
    }
    
    _transform(chunk, encoding, callback) {
        const lines = chunk.toString().split('\n');
        
        for (const line of lines) {
            if (line.trim() === '') continue;
            
            const values = line.split(',');
            
            if (this.isFirstChunk) {
                // First line contains headers
                this.headers = values;
                this.isFirstChunk = false;
            } else {
                // Convert to object
                const obj = {};
                this.headers.forEach((header, index) => {
                    obj[header.trim()] = values[index] ? values[index].trim() : '';
                });
                
                // Push the JSON object
                this.push(JSON.stringify(obj) + '\n');
            }
        }
        
        callback();
    }
}

// Usage
const csvToJson = new CSVToJSONTransform();
fs.createReadStream('data.csv')
    .pipe(csvToJson)
    .pipe(fs.createWriteStream('output.json'));

Using built-in transform streams:

const fs = require('fs');
const zlib = require('zlib');

// Compression pipeline
fs.createReadStream('large-file.txt')
    .pipe(zlib.createGzip())           // Transform: compress
    .pipe(fs.createWriteStream('large-file.txt.gz'));

// Decompression pipeline  
fs.createReadStream('large-file.txt.gz')
    .pipe(zlib.createGunzip())         // Transform: decompress
    .pipe(fs.createWriteStream('decompressed.txt'));

Transform stream with error handling:

const { Transform } = require('stream');

class NumberMultiplier extends Transform {
    constructor(factor) {
        super({ objectMode: true });
        this.factor = factor;
    }
    
    _transform(chunk, encoding, callback) {
        try {
            const number = parseInt(chunk.toString());
            
            if (isNaN(number)) {
                throw new Error('Invalid number');
            }
            
            const result = number * this.factor;
            this.push(result.toString());
            callback();
        } catch (error) {
            callback(error); // Pass error to stream
        }
    }
}

const multiplier = new NumberMultiplier(10);
multiplier.on('error', (error) => {
    console.log('Transform error:', error.message);
});

multiplier.pipe(process.stdout);
multiplier.write('5');  // Output: 50
multiplier.write('abc'); // Will emit error

Key benefits of transform streams:

• Reusable data transformation logic
• Composable with pipe()
• Automatic backpressure handling
• Memory efficient for large data
• Easy error propagation

Transform streams are the building blocks of data processing pipelines in Node.js, enabling efficient and modular data transformation.

Proper error handling in streams is crucial because unhandled stream errors can crash your application. Streams use EventEmitter for error handling.

Basic error handling with event listeners:

const fs = require('fs');

const readStream = fs.createReadStream('nonexistent-file.txt');

readStream.on('error', (error) => {
    console.log('Read stream error:', error.message);
});

const writeStream = fs.createWriteStream('/readonly/location.txt');

writeStream.on('error', (error) => {
    console.log('Write stream error:', error.message);
});

Error handling in pipelines (Node.js 10+):

const fs = require('fs');
const { pipeline } = require('stream');

// pipeline automatically handles errors and cleanup
pipeline(
    fs.createReadStream('input.txt'),
    fs.createWriteStream('output.txt'),
    (error) => {
        if (error) {
            console.log('Pipeline failed:', error);
        } else {
            console.log('Pipeline succeeded');
        }
    }
);

Error handling with pipe() (manual approach):

const fs = require('fs');

function handleStreamErrors(source, destination) {
    source.on('error', (error) => {
        console.log('Source error:', error);
        destination.destroy(error); // Destroy destination too
    });
    
    destination.on('error', (error) => {
        console.log('Destination error:', error);
        source.destroy(error); // Destroy source too
    });
}

const source = fs.createReadStream('input.txt');
const destination = fs.createWriteStream('output.txt');

handleStreamErrors(source, destination);
source.pipe(destination);

Custom error handling in transform streams:

const { Transform } = require('stream');

class SafeTransform extends Transform {
    _transform(chunk, encoding, callback) {
        try {
            // Simulate potential error
            if (chunk.includes('error')) {
                throw new Error('Invalid data detected');
            }
            
            const processed = chunk.toString().toUpperCase();
            this.push(processed);
            callback();
        } catch (error) {
            // Pass error to stream
            callback(error);
        }
    }
}

const transform = new SafeTransform();

transform.on('error', (error) => {
    console.log('Transform error handled:', error.message);
});

transform.write('hello');
transform.write('this will cause error'); // Triggers error
transform.write('world'); // This won't be processed

Graceful error recovery:

const fs = require('fs');
const { Transform } = require('stream');

class ResilientTransform extends Transform {
    constructor() {
        super();
        this.errorCount = 0;
        this.maxErrors = 3;
    }
    
    _transform(chunk, encoding, callback) {
        // Simulate occasional errors
        if (Math.random() < 0.3 && this.errorCount < this.maxErrors) {
            this.errorCount++;
            console.log(`Error ${this.errorCount}, skipping chunk`);
            callback(); // Continue without pushing data
        } else {
            this.push(chunk);
            callback();
        }
    }
}

const resilient = new ResilientTransform();
resilient.on('error', (error) => {
    console.log('Fatal error:', error);
});

// This stream will tolerate some errors before failing

Best practices for stream error handling:

• Always attach error listeners to all streams
• Use pipeline() for complex chains
• Destroy related streams when one fails
• Handle errors gracefully - don't let them crash the app
• Log errors for debugging
• Consider retry mechanisms for transient errors

Proper error handling makes your stream-based applications robust and production-ready.

File descriptors are low-level identifiers that operating systems use to track open files. In Node.js, you can work with file descriptors for more control over file operations.

Think of file descriptors like library cards:

• Each open file gets a unique number (the descriptor)
• The OS uses this number to track the file
• You need the card to check out books (perform operations)

Opening a file and getting its descriptor:

const fs = require('fs');

// Open file and get file descriptor
fs.open('example.txt', 'r', (error, fd) => {
    if (error) {
        console.log('Error opening file:', error);
        return;
    }
    
    console.log('File descriptor:', fd);
    
    // Always close the file descriptor when done
    fs.close(fd, (error) => {
        if (error) console.log('Error closing:', error);
    });
});

Reading from a file using file descriptor:

const fs = require('fs');

fs.open('data.txt', 'r', (error, fd) => {
    if (error) throw error;
    
    const buffer = Buffer.alloc(1024); // Buffer to hold data
    
    fs.read(fd, buffer, 0, buffer.length, 0, (error, bytesRead, buffer) => {
        if (error) throw error;
        
        console.log('Bytes read:', bytesRead);
        console.log('Content:', buffer.toString('utf8', 0, bytesRead));
        
        fs.close(fd, (error) => {
            if (error) console.log('Error closing:', error);
        });
    });
});

Writing to a file using file descriptor:

const fs = require('fs');

fs.open('output.txt', 'w', (error, fd) => {
    if (error) throw error;
    
    const content = 'Hello, World!';
    const buffer = Buffer.from(content, 'utf8');
    
    fs.write(fd, buffer, 0, buffer.length, 0, (error, written, buffer) => {
        if (error) throw error;
        
        console.log('Bytes written:', written);
        
        fs.close(fd, (error) => {
            if (error) console.log('Error closing:', error);
        });
    });
});

Common file open flags:

• 'r' - Read (file must exist)
• 'r+' - Read and write (file must exist)
• 'w' - Write (creates or truncates)
• 'w+' - Read and write (creates or truncates)
• 'a' - Append (creates or appends)
• 'a+' - Read and append (creates or appends)

Getting file information with file descriptor:

const fs = require('fs');

fs.open('example.txt', 'r', (error, fd) => {
    if (error) throw error;
    
    fs.fstat(fd, (error, stats) => {
        if (error) throw error;
        
        console.log('File size:', stats.size);
        console.log('Created:', stats.birthtime);
        console.log('Modified:', stats.mtime);
        
        fs.close(fd, (error) => {
            if (error) console.log('Error closing:', error);
        });
    });
});

Standard file descriptors:

• 0: stdin (standard input)
• 1: stdout (standard output)
• 2: stderr (standard error)

When to use file descriptors:

• Need low-level file control
• Working with file metadata
• Random access reading/writing
• File locking operations
• When high performance is critical

File descriptors give you fine-grained control over file operations but require more careful resource management.

File watching allows your application to react when files or directories change. This is useful for live reloading, automatic builds, and monitoring.

Watching a single file for changes:

const fs = require('fs');

const watcher = fs.watch('config.json', (eventType, filename) => {
    if (filename) {
        console.log(`File ${filename} changed: ${eventType}`);
        
        if (eventType === 'change') {
            console.log('Content was modified');
            // Reload configuration, etc.
        }
    }
});

// Stop watching after 30 seconds
setTimeout(() => {
    watcher.close();
    console.log('Stopped watching file');
}, 30000);

Watching a directory for changes:

const fs = require('fs');

const watcher = fs.watch('src/', (eventType, filename) => {
    console.log(`Event: ${eventType} on file: ${filename}`);
    
    if (eventType === 'rename') {
        // File was created or deleted
        console.log('File was created or deleted');
    } else if (eventType === 'change') {
        // File content changed
        console.log('File content was modified');
    }
});

Using fs.watchFile() for polling (less efficient):

const fs = require('fs');

// Check file every 2 seconds
fs.watchFile('data.txt', { interval: 2000 }, (curr, prev) => {
    console.log(`File modified at: ${curr.mtime}`);
    
    if (curr.mtime !== prev.mtime) {
        console.log('File content changed');
        // Handle the change
    }
});

// Stop watching
// fs.unwatchFile('data.txt');

More robust watching with fs.watch (recursive):

const fs = require('fs');

const watcher = fs.watch('project/', { 
    recursive: true  // Watch subdirectories too
}, (eventType, filename) => {
    console.log(`Change detected in: ${filename}`);
    
    // Debounce rapid changes
    clearTimeout(watcher.debounceTimer);
    watcher.debounceTimer = setTimeout(() => {
        console.log('Processing changes...');
        // Your change handling logic here
    }, 100);
});

File watching with error handling:

const fs = require('fs');

function startWatching(path) {
    try {
        const watcher = fs.watch(path, { recursive: true }, (event, filename) => {
            console.log(`Change: ${event} on ${filename}`);
        });
        
        watcher.on('error', (error) => {
            console.log('Watcher error:', error);
            // Attempt to restart watching
            setTimeout(() => startWatching(path), 1000);
        });
        
        return watcher;
    } catch (error) {
        console.log('Failed to start watching:', error);
    }
}

const watcher = startWatching('./src');

Using chokidar (popular third-party library):

// First: npm install chokidar
const chokidar = require('chokidar');

const watcher = chokidar.watch('src/', {
    ignored: /node_modules/, // Ignore node_modules
    persistent: true,
    ignoreInitial: true
});

watcher
    .on('add', path => console.log(`File ${path} added`))
    .on('change', path => console.log(`File ${path} changed`))
    .on('unlink', path => console.log(`File ${path} removed`))
    .on('error', error => console.log(`Watcher error: ${error}`));

Common use cases for file watching:

• Development servers with live reload
• Build systems that trigger on file changes
• Configuration hot-reloading
• Log file monitoring
• Backup systems

File watching is essential for creating responsive applications that react to filesystem changes in real-time.

File permissions control who can read, write, or execute files. Node.js provides methods to set and modify these permissions using Unix-style permission codes.

Understanding permission codes:

// Permission codes use octal notation
// 0o644 = owner: read/write, group: read, others: read
// 0o755 = owner: read/write/execute, group: read/execute, others: read/execute

// Breakdown:
// 4 = read, 2 = write, 1 = execute
// First digit: owner, second: group, third: others
// 6 = 4+2 (read+write), 5 = 4+1 (read+execute), 7 = 4+2+1 (all)

Setting permissions with chmod:

const fs = require('fs');

// Make file readable/writable by owner, readable by others
fs.chmod('config.json', 0o644, (error) => {
    if (error) {
        console.log('Error setting permissions:', error);
        return;
    }
    console.log('Permissions updated');
});

Using symbolic notation:

const fs = require('fs');

// Make script executable
fs.chmod('script.sh', '755', (error) => {
    if (error) throw error;
    console.log('Script is now executable');
});

// Add write permission for group
fs.chmod('data.txt', 'g+w', (error) => {
    if (error) throw error;
    console.log('Group write permission added');
});

Setting permissions when creating files:

const fs = require('fs');

// Create file with specific permissions
fs.writeFile('secret.txt', 'confidential data', { 
    mode: 0o600  // Only owner can read/write
}, (error) => {
    if (error) throw error;
    console.log('Secure file created');
});

// Create directory with specific permissions
fs.mkdir('private-folder', {
    mode: 0o700  // Only owner has access
}, (error) => {
    if (error) throw error;
    console.log('Private directory created');
});

Common permission patterns:

const fs = require('fs');

const permissions = {
    PUBLIC_FILE: 0o644,    // -rw-r--r--
    PRIVATE_FILE: 0o600,   // -rw-------
    EXECUTABLE: 0o755,     // -rwxr-xr-x
    PRIVATE_DIR: 0o700,    // drwx------
    PUBLIC_DIR: 0o755      // drwxr-xr-x
};

fs.chmod('app.js', permissions.EXECUTABLE, (error) => {
    if (error) throw error;
    console.log('App is now executable');
});

Checking current permissions:

const fs = require('fs');

fs.stat('file.txt', (error, stats) => {
    if (error) throw error;
    
    // stats.mode contains permission information
    console.log('File mode:', stats.mode.toString(8)); // Octal representation
    console.log('Is directory:', stats.isDirectory());
    console.log('Is file:', stats.isFile());
});

A symbolic link (symlink) is a special file that points to another file or directory. It's like a shortcut or reference to the actual file.

Creating symbolic links:

const fs = require('fs');

// Create symlink to a file
fs.symlink('./original.txt', './link-to-original.txt', 'file', (error) => {
    if (error) {
        console.log('Error creating symlink:', error);
        return;
    }
    console.log('Symbolic link created');
});

// Create symlink to a directory
fs.symlink('../parent-folder', './parent-link', 'dir', (error) => {
    if (error) throw error;
    console.log('Directory symlink created');
});

Reading symbolic links:

const fs = require('fs');

// Get the target of a symlink
fs.readlink('./link-to-original.txt', (error, linkString) => {
    if (error) throw error;
    console.log('Symlink points to:', linkString);
});

Checking if a path is a symbolic link:

const fs = require('fs');

fs.lstat('./possible-link', (error, stats) => {
    if (error) throw error;
    
    if (stats.isSymbolicLink()) {
        console.log('This is a symbolic link');
        fs.readlink('./possible-link', (error, target) => {
            console.log('It points to:', target);
        });
    } else {
        console.log('This is a regular file/directory');
    }
});

Real-world use cases:

const fs = require('fs');

// Use case 1: Version management
fs.symlink('/apps/myapp-v1.2.3', '/apps/myapp-current', 'dir', (error) => {
    if (error) throw error;
    console.log('Current version symlink created');
});

// Use case 2: Configuration switching
fs.symlink('./configs/production.json', './config/current.json', 'file', (error) => {
    if (error) throw error;
    console.log('Production config activated');
});

Difference between hard links and symbolic links:

const fs = require('fs');

// Hard link - direct reference to file data
fs.link('original.txt', 'hard-link.txt', (error) => {
    if (error) console.log('Hard link creation failed');
});

// Symbolic link - points to file path
fs.symlink('original.txt', 'soft-link.txt', 'file', (error) => {
    if (error) console.log('Soft link creation failed');
});

Key differences:

• Symbolic links: Can cross filesystems, point to directories, break if original is deleted
• Hard links: Same filesystem only, can't link directories, persist if original is deleted

The path module provides utilities for working with file and directory paths. It helps you write cross-platform code that works on Windows, macOS, and Linux.

Importing the path module:

const path = require('path');

Basic path operations:

const path = require('path');

const filePath = '/users/john/documents/file.txt';

console.log('Directory:', path.dirname(filePath));
console.log('Filename:', path.basename(filePath));
console.log('Extension:', path.extname(filePath));
console.log('Filename without extension:', path.basename(filePath, '.txt'));

Output:

Directory: /users/john/documents
Filename: file.txt
Extension: .txt
Filename without extension: file

Path parsing:

const path = require('path');

const parsed = path.parse('/home/user/project/src/index.js');
console.log(parsed);

Output:

{
  root: '/',
  dir: '/home/user/project/src',
  base: 'index.js',
  ext: '.js',
  name: 'index'
}

Platform-specific path handling:

const path = require('path');

console.log('Platform path separator:', path.sep);
console.log('Path delimiter:', path.delimiter);
console.log('Current working directory:', process.cwd());

// Cross-platform path joining
const fullPath = path.join('users', 'john', 'documents', 'file.txt');
console.log('Joined path:', fullPath);

Resolving relative paths:

const path = require('path');

// Resolve to absolute path
console.log('Absolute path:', path.resolve('src', 'app.js'));
console.log('With CWD:', path.resolve('./config.json'));

// Get relative path between two paths
const from = '/users/john/documents';
const to = '/users/john/downloads/file.zip';
console.log('Relative path:', path.relative(from, to));

Normalizing paths:

const path = require('path');

// Remove redundant segments
console.log('Normalized:', path.normalize('/users/../john/./documents//file.txt'));
// Output: /john/documents/file.txt

// Handle .. and . correctly
console.log('Resolved with ..:', path.resolve('/users/john', '../mary/file.txt'));
// Output: /users/mary/file.txt

Why use path module instead of string manipulation:

// ❌ Don't do this - platform specific
const badPath = 'folder' + '\\' + 'subfolder' + '\\' + 'file.txt';

// ✅ Do this - cross-platform
const goodPath = path.join('folder', 'subfolder', 'file.txt');

Path joining and resolution are essential for building correct file paths in a cross-platform way.

path.join() - Combine path segments:

const path = require('path');

// Basic joining
const fullPath = path.join('users', 'john', 'documents', 'file.txt');
console.log(fullPath);
// On Unix: users/john/documents/file.txt
// On Windows: users\john\documents\file.txt

// With absolute paths
const absolutePath = path.join('/users', 'john', 'documents', 'file.txt');
console.log(absolutePath);
// Output: /users/john/documents/file.txt

// Handles trailing slashes
const cleanPath = path.join('users/', '/john/', '/documents/');
console.log(cleanPath);
// Output: users/john/documents

path.resolve() - Create absolute paths:

const path = require('path');

// Resolve relative to current working directory
console.log(path.resolve('src', 'app.js'));
// If CWD is /home/user, output: /home/user/src/app.js

// With absolute path segment
console.log(path.resolve('/etc', 'config.json'));
// Output: /etc/config.json

// Using .. to go up directories
console.log(path.resolve('/users/john', '../mary', 'file.txt'));
// Output: /users/mary/file.txt

path.relative() - Find relative path between two paths:

const path = require('path');

const from = '/users/john/documents';
const to = '/users/john/downloads/file.zip';

console.log(path.relative(from, to));
// Output: ../downloads/file.zip

const from2 = '/project/src/components';
const to2 = '/project/assets/images/logo.png';

console.log(path.relative(from2, to2));
// Output: ../../assets/images/logo.png

Practical examples:

const path = require('path');
const fs = require('fs');

// Building project paths safely
function getProjectPaths(projectRoot) {
    return {
        src: path.join(projectRoot, 'src'),
        dist: path.join(projectRoot, 'dist'),
        config: path.join(projectRoot, 'config.json'),
        nodeModules: path.join(projectRoot, 'node_modules')
    };
}

const paths = getProjectPaths('/my-project');
console.log(paths.src); // /my-project/src

// Resolving module paths
function requireLocal(modulePath) {
    const absolutePath = path.resolve(process.cwd(), modulePath);
    return require(absolutePath);
}

Working with file extensions:

const path = require('path');

function changeExtension(filePath, newExtension) {
    const dir = path.dirname(filePath);
    const name = path.basename(filePath, path.extname(filePath));
    return path.join(dir, name + newExtension);
}

console.log(changeExtension('/src/app.js', '.ts'));
// Output: /src/app.ts

console.log(changeExtension('config.json', '.backup'));
// Output: config.backup

Common path patterns:

const path = require('path');

// Get directory of current module
const currentDir = __dirname;

// Get path to parent directory
const parentDir = path.join(__dirname, '..');

// Create path relative to current file
const configPath = path.join(__dirname, 'config', 'app.json');

// Resolve from project root
const projectRoot = path.resolve(__dirname, '..', '..');
const assetPath = path.join(projectRoot, 'assets', 'images');

Reading JSON files is a common task in Node.js applications. The fs module makes it straightforward to read and parse JSON data.

Basic JSON file reading:

const fs = require('fs');

// Read and parse JSON file
fs.readFile('config.json', 'utf8', (error, data) => {
    if (error) {
        console.log('Error reading file:', error);
        return;
    }
    
    try {
        const config = JSON.parse(data);
        console.log('Database host:', config.database.host);
        console.log('App port:', config.app.port);
    } catch (parseError) {
        console.log('Error parsing JSON:', parseError);
    }
});

Using require() for JSON files:

// Simple way for static configuration
// Note: This caches the result and is synchronous
const config = require('./config.json');
console.log('App name:', config.appName);

// For dynamic reloading, use fs.readFile
function loadConfig() {
    delete require.cache[require.resolve('./config.json')];
    return require('./config.json');
}

Reading JSON with fs.promises:

const fs = require('fs').promises;

async function readJSONFile(filePath) {
    try {
        const data = await fs.readFile(filePath, 'utf8');
        return JSON.parse(data);
    } catch (error) {
        console.log('Error reading JSON file:', error);
        throw error;
    }
}

// Usage
async function main() {
    try {
        const users = await readJSONFile('users.json');
        const settings = await readJSONFile('settings.json');
        
        console.log('Loaded users:', users.length);
        console.log('App theme:', settings.theme);
    } catch (error) {
        console.log('Failed to load configuration');
    }
}

main();

Handling large JSON files with streams:

const fs = require('fs');
const { Transform } = require('stream');

class JSONParser extends Transform {
    constructor(options) {
        super({ ...options, objectMode: true });
        this.buffer = '';
    }
    
    _transform(chunk, encoding, callback) {
        this.buffer += chunk.toString();
        
        // Try to parse complete JSON objects
        try {
            const data = JSON.parse(this.buffer);
            this.push(data);
            this.buffer = '';
        } catch (error) {
            // Incomplete JSON, wait for more data
        }
        
        callback();
    }
}

// For newline-delimited JSON files
fs.createReadStream('large-data.jsonl')
    .pipe(split()) // Split by newlines
    .pipe(new JSONParser())
    .on('data', (obj) => {
        console.log('Parsed object:', obj);
    });

Validating JSON schema:

const fs = require('fs').promises;

async function loadValidatedConfig() {
    const schema = {
        type: 'object',
        required: ['appName', 'port', 'database'],
        properties: {
            appName: { type: 'string' },
            port: { type: 'number', minimum: 1, maximum: 65535 },
            database: {
                type: 'object',
                required: ['host', 'name'],
                properties: {
                    host: { type: 'string' },
                    name: { type: 'string' }
                }
            }
        }
    };
    
    const data = await fs.readFile('config.json', 'utf8');
    const config = JSON.parse(data);
    
    // Simple validation
    if (!config.appName || !config.port) {
        throw new Error('Invalid configuration: missing required fields');
    }
    
    return config;
}

Writing JSON files with formatting:

const fs = require('fs');

const data = {
    appName: 'My Application',
    version: '1.0.0',
    settings: {
        theme: 'dark',
        notifications: true
    },
    users: ['john', 'jane', 'bob']
};

// Write with 2-space indentation
fs.writeFile('app-config.json', JSON.stringify(data, null, 2), 'utf8', (error) => {
    if (error) throw error;
    console.log('Configuration saved');
});

Parsing large files requires streaming and chunk processing to avoid memory issues. Here are efficient techniques for handling large files.

Streaming large CSV files:

const fs = require('fs');
const readline = require('readline');

async function processLargeCSV(filePath) {
    const fileStream = fs.createReadStream(filePath);
    const rl = readline.createInterface({
        input: fileStream,
        crlfDelay: Infinity // Handle Windows line endings
    });
    
    let lineCount = 0;
    let headers = [];
    
    for await (const line of rl) {
        if (lineCount === 0) {
            // First line contains headers
            headers = line.split(',');
        } else {
            // Process data line
            const values = line.split(',');
            const row = {};
            
            headers.forEach((header, index) => {
                row[header] = values[index] || '';
            });
            
            // Process the row
            console.log('Processing row:', row);
            
            // Simulate processing time
            await new Promise(resolve => setTimeout(resolve, 10));
        }
        
        lineCount++;
        if (lineCount % 1000 === 0) {
            console.log(`Processed ${lineCount} lines`);
        }
    }
    
    console.log(`Finished processing ${lineCount} lines`);
}

processLargeCSV('large-data.csv').catch(console.error);

Processing large JSON files with streams:

const fs = require('fs');
const { Transform } = require('stream');

class JSONStreamProcessor extends Transform {
    constructor(options) {
        super({ ...options, objectMode: true });
        this.buffer = '';
        this.objectCount = 0;
    }
    
    _transform(chunk, encoding, callback) {
        this.buffer += chunk.toString();
        
        // Process complete JSON objects (assuming one per line)
        const lines = this.buffer.split('\n');
        this.buffer = lines.pop(); // Keep incomplete line
        
        for (const line of lines) {
            if (line.trim()) {
                try {
                    const obj = JSON.parse(line);
                    this.objectCount++;
                    this.processObject(obj);
                } catch (error) {
                    console.log('Error parsing JSON line:', error);
                }
            }
        }
        
        callback();
    }
    
    processObject(obj) {
        // Your processing logic here
        this.push(JSON.stringify(obj) + '\n');
        
        if (this.objectCount % 1000 === 0) {
            console.log(`Processed ${this.objectCount} objects`);
        }
    }
}

// Usage
fs.createReadStream('large-data.jsonl')
    .pipe(new JSONStreamProcessor())
    .pipe(fs.createWriteStream('processed-data.jsonl'));

Memory-efficient file processing with buffers:

const fs = require('fs');

function processFileInChunks(filePath, chunkSize = 64 * 1024) {
    return new Promise((resolve, reject) => {
        const stats = fs.statSync(filePath);
        const fileSize = stats.size;
        let bytesProcessed = 0;
        let position = 0;
        
        const processNextChunk = () => {
            if (position >= fileSize) {
                resolve();
                return;
            }
            
            const readSize = Math.min(chunkSize, fileSize - position);
            const buffer = Buffer.alloc(readSize);
            
            fs.open(filePath, 'r', (error, fd) => {
                if (error) {
                    reject(error);
                    return;
                }
                
                fs.read(fd, buffer, 0, readSize, position, (error, bytesRead) => {
                    if (error) {
                        fs.close(fd, () => reject(error));
                        return;
                    }
                    
                    // Process the chunk
                    processChunk(buffer, bytesRead, position);
                    
                    bytesProcessed += bytesRead;
                    position += bytesRead;
                    
                    // Progress reporting
                    const percent = ((bytesProcessed / fileSize) * 100).toFixed(1);
                    console.log(`Progress: ${percent}%`);
                    
                    fs.close(fd, (error) => {
                        if (error) console.log('Close error:', error);
                        setImmediate(processNextChunk); // Prevent stack overflow
                    });
                });
            });
        };
        
        processNextChunk();
    });
}

function processChunk(buffer, bytesRead, position) {
    // Your chunk processing logic here
    const chunkText = buffer.toString('utf8', 0, bytesRead);
    // Process chunkText...
}

Using readline for log file analysis:

const fs = require('fs');
const readline = require('readline');

async function analyzeLogFile(filePath) {
    const fileStream = fs.createReadStream(filePath);
    const rl = readline.createInterface({
        input: fileStream,
        crlfDelay: Infinity
    });
    
    const analysis = {
        totalLines: 0,
        errorCount: 0,
        warningCount: 0,
        infoCount: 0,
        ipAddresses: new Set()
    };
    
    for await (const line of rl) {
        analysis.totalLines++;
        
        if (line.includes('ERROR')) analysis.errorCount++;
        if (line.includes('WARNING')) analysis.warningCount++;
        if (line.includes('INFO')) analysis.infoCount++;
        
        // Extract IP addresses (simple regex)
        const ipMatch = line.match(/\b(?:\d{1,3}\.){3}\d{1,3}\b/);
        if (ipMatch) {
            analysis.ipAddresses.add(ipMatch[0]);
        }
        
        // Process every 10,000 lines to show progress
        if (analysis.totalLines % 10000 === 0) {
            console.log(`Processed ${analysis.totalLines} lines...`);
        }
    }
    
    return {
        ...analysis,
        uniqueIPs: analysis.ipAddresses.size
    };
}

// Usage
analyzeLogFile('server.log')
    .then(result => {
        console.log('Analysis result:', result);
    })
    .catch(console.error);

Key strategies for large files:

• Use streams instead of loading entire file
• Process data in chunks
• Use readline for line-based files
• Implement progress reporting
• Handle backpressure properly
• Use async processing for CPU-intensive tasks

zlib is a Node.js module that provides compression and decompression functionality using the Gzip, Deflate, and Brotli algorithms.

Basic file compression:

const fs = require('fs');
const zlib = require('zlib');

// Compress a file
fs.createReadStream('large-file.txt')
    .pipe(zlib.createGzip())
    .pipe(fs.createWriteStream('large-file.txt.gz'))
    .on('finish', () => {
        console.log('File compressed successfully');
    });

File decompression:

const fs = require('fs');
const zlib = require('zlib');

// Decompress a file
fs.createReadStream('large-file.txt.gz')
    .pipe(zlib.createGunzip())
    .pipe(fs.createWriteStream('decompressed-file.txt'))
    .on('finish', () => {
        console.log('File decompressed successfully');
    });

Node.js provides multiple compression methods through the zlib module for different use cases.

Different compression methods:

const fs = require('fs');
const zlib = require('zlib');

// Gzip compression (most common)
fs.createReadStream('data.txt')
    .pipe(zlib.createGzip())
    .pipe(fs.createWriteStream('data.txt.gz'));

// Deflate compression
fs.createReadStream('data.txt')
    .pipe(zlib.createDeflate())
    .pipe(fs.createWriteStream('data.txt.deflate'));

// Brotli compression (modern, efficient)
fs.createReadStream('data.txt')
    .pipe(zlib.createBrotliCompress())
    .pipe(fs.createWriteStream('data.txt.br'));

Synchronous compression:

const fs = require('fs');
const zlib = require('zlib');

const data = fs.readFileSync('large-file.txt');
const compressed = zlib.gzipSync(data);
fs.writeFileSync('large-file.txt.gz', compressed);

fs-extra is a popular third-party module that enhances the built-in fs module with additional methods and better promise support.

Installing fs-extra:

npm install fs-extra

Key differences and advantages:

const fs = require('fs');
const fse = require('fs-extra');

1. Promise-based by default:

// fs-extra uses promises (no callbacks needed)
async function withFsExtra() {
    try {
        const data = await fse.readFile('file.txt', 'utf8');
        await fse.writeFile('output.txt', data);
        await fse.copy('src/', 'dist/');
        console.log('All operations completed');
    } catch (error) {
        console.log('Error:', error);
    }
}

// vs native fs with promises
async function withNativeFs() {
    const fs = require('fs').promises;
    try {
        const data = await fs.readFile('file.txt', 'utf8');
        await fs.writeFile('output.txt', data);
        // No built-in copy method
        console.log('Operations completed');
    } catch (error) {
        console.log('Error:', error);
    }
}

2. Additional utility methods:

const fse = require('fs-extra');

// Copy directories recursively
await fse.copy('/source/dir', '/dest/dir');

// Ensure directory exists (create if not)
await fse.ensureDir('/path/to/directory');

// Remove directory and all contents
await fse.remove('/path/to/directory');

// Read JSON file and parse automatically
const config = await fse.readJson('config.json');

// Write JSON with automatic stringification
await fse.writeJson('data.json', { name: 'John', age: 30 });

// Check if path exists (returns boolean)
const exists = await fse.pathExists('/some/path');

// Empty directory (remove all contents)
await fse.emptyDir('/path/to/directory');

3. Better error messages and handling:

const fse = require('fs-extra');

// More descriptive errors
try {
    await fse.move('/nonexistent/src', '/dest');
} catch (error) {
    console.log(error.message);
    // "ENOENT: no such file or directory, stat '/nonexistent/src'"
}

4. Additional useful methods:

const fse = require('fs-extra');

// File operations
await fse.move('/old/path', '/new/path'); // Move/rename
await fse.ensureFile('/path/to/file.txt'); // Ensure file exists
await fse.createFile('/path/to/newfile.txt'); // Create empty file

// Directory operations
const files = await fse.readdir('/path'); // Returns array
const stats = await fse.stat('/path/to/file');

// Path operations
await fse.ensureSymlink('/target', '/link'); // Create symlink
await fse.ensureLink('/target', '/link'); // Create hard link

When to use fs-extra vs native fs:

• Use fs-extra: When you need convenience methods, better promise support, or frequently work with directories
• Use native fs: When you want zero dependencies, minimal footprint, or only need basic file operations

The encoding option specifies how to interpret file data - as text with a specific character encoding or as raw binary data.

Common encoding options:

const fs = require('fs');

// Read as UTF-8 text (most common)
fs.readFile('file.txt', 'utf8', (error, data) => {
    console.log('UTF-8 text:', data);
});

// Read as ASCII
fs.readFile('file.txt', 'ascii', (error, data) => {
    console.log('ASCII text:', data);
});

// Read as base64 (for binary data)
fs.readFile('image.jpg', 'base64', (error, data) => {
    console.log('Base64 encoded:', data.substring(0, 100) + '...');
});

// Read as hex
fs.readFile('file.bin', 'hex', (error, data) => {
    console.log('Hex representation:', data.substring(0, 50));
});

// No encoding - get Buffer object
fs.readFile('file.txt', (error, data) => {
    console.log('Buffer:', data);
    console.log('As string:', data.toString('utf8'));
});

Encoding in write operations:

const fs = require('fs');

// Write with UTF-8 encoding
fs.writeFile('text.txt', 'Hello World', 'utf8', (error) => {
    if (error) throw error;
});

// Write binary data from base64
const base64Data = 'SGVsbG8gV29ybGQ='; // "Hello World" in base64
fs.writeFile('from-base64.txt', base64Data, 'base64', (error) => {
    if (error) throw error;
});

// Write without encoding (Buffer)
const buffer = Buffer.from('Hello World', 'utf8');
fs.writeFile('from-buffer.txt', buffer, (error) => {
    if (error) throw error;
});

Detecting file encoding:

const fs = require('fs');
const jschardet = require('jschardet'); // npm install jschardet

function detectEncoding(filePath) {
    const buffer = fs.readFileSync(filePath);
    const detection = jschardet.detect(buffer);
    
    console.log('Detected encoding:', detection.encoding);
    console.log('Confidence:', detection.confidence);
    
    return detection.encoding;
}

// Usage
const encoding = detectEncoding('unknown-file.txt');
const content = fs.readFileSync('unknown-file.txt', encoding);

Handling different encodings automatically:

const fs = require('fs');
const iconv = require('iconv-lite'); // npm install iconv-lite

function readFileWithEncoding(filePath, possibleEncodings = ['utf8', 'win1252', 'iso-8859-1']) {
    const buffer = fs.readFileSync(filePath);
    
    for (const encoding of possibleEncodings) {
        try {
            const content = iconv.decode(buffer, encoding);
            // Check if decoding produced valid text
            if (!content.includes('�')) { // No replacement characters
                return { content, encoding };
            }
        } catch (error) {
            continue;
        }
    }
    
    throw new Error('Could not decode file with any of the provided encodings');
}

// Usage
try {
    const result = readFileWithEncoding('european-file.txt');
    console.log('File encoding:', result.encoding);
    console.log('Content:', result.content);
} catch (error) {
    console.log('Error:', error.message);
}

Common encoding scenarios:

const fs = require('fs');

// Scenario 1: Reading configuration files (usually UTF-8)
const config = JSON.parse(fs.readFileSync('config.json', 'utf8'));

// Scenario 2: Reading legacy files (might be different encoding)
const legacyContent = fs.readFileSync('old-file.txt', 'latin1');

// Scenario 3: Working with binary files (images, etc.)
const imageBuffer = fs.readFileSync('photo.jpg');
// No encoding specified - returns Buffer

// Scenario 4: Creating data URLs for web
const imageBase64 = fs.readFileSync('icon.png', 'base64');
const dataUrl = `data:image/png;base64,${imageBase64}`;

// Scenario 5: Reading files for cryptographic operations
const fileHash = fs.readFileSync('document.pdf', 'binary');

Supported encodings in Node.js:

• utf8 - Standard Unicode (default)
• utf16le - Little-endian UTF-16
• latin1 - ISO-8859-1
• ascii - 7-bit ASCII
• base64 - Base64 encoding
• hex - Hexadecimal
• ucs2 - Alias of utf16le
• binary - Binary data (deprecated)

Temporary files are useful for intermediate processing, caching, or storing data that doesn't need to persist. Node.js provides several ways to create them.

Using the os.tmpdir() method:

const fs = require('fs');
const os = require('os');
const path = require('path');

function createTempFile(content, extension = '.tmp') {
    const tempDir = os.tmpdir();
    const tempFile = path.join(tempDir, `temp-${Date.now()}${extension}`);
    
    fs.writeFileSync(tempFile, content);
    return tempFile;
}

// Usage
const tempPath = createTempFile('Temporary data', '.txt');
console.log('Temp file created:', tempPath);

// Clean up when done
fs.unlinkSync(tempPath);

Using the tmp package (recommended):

const tmp = require('tmp'); // npm install tmp

// Create temporary file
tmp.file((error, path, fd, cleanupCallback) => {
    if (error) throw error;
    
    console.log('Temp file path:', path);
    console.log('File descriptor:', fd);
    
    // Write to temp file
    fs.writeSync(fd, 'Temporary content');
    
    // Automatically clean up on process exit
    cleanupCallback();
});

// Create temporary directory
tmp.dir((error, path, cleanupCallback) => {
    if (error) throw error;
    
    console.log('Temp directory:', path);
    
    // Create files in temp directory
    const filePath = path + '/data.txt';
    fs.writeFileSync(filePath, 'Some data');
    
    // Clean up when done
    cleanupCallback();
});

With promises using tmp-promise:

const tmp = require('tmp-promise'); // npm install tmp-promise

async function useTempFiles() {
    try {
        // Create temp file with automatic cleanup
        const { path, fd, cleanup } = await tmp.file();
        
        await fs.promises.writeFile(path, 'Temporary data');
        const data = await fs.promises.readFile(path, 'utf8');
        
        console.log('Read from temp file:', data);
        
        // Manual cleanup (or it happens automatically)
        await cleanup();
        
    } catch (error) {
        console.log('Error:', error);
    }
}

useTempFiles();

Secure temporary files with specific permissions:

const tmp = require('tmp');

// Create secure temp file (readable only by current user)
tmp.file({ 
    mode: 0o600,  // Only owner can read/write
    prefix: 'secret-',
    postfix: '.tmp'
}, (error, path, fd) => {
    if (error) throw error;
    
    fs.writeFileSync(path, 'Sensitive data');
    console.log('Secure temp file created:', path);
});

Temporary files with automatic cleanup:

const fs = require('fs');
const os = require('os');
const path = require('path');

class TempFileManager {
    constructor() {
        this.files = new Set();
        this.setupCleanup();
    }
    
    createTempFile(content, extension = '.tmp') {
        const tempPath = path.join(os.tmpdir(), `app-${process.pid}-${Date.now()}${extension}`);
        fs.writeFileSync(tempPath, content);
        this.files.add(tempPath);
        return tempPath;
    }
    
    cleanup() {
        for (const filePath of this.files) {
            try {
                if (fs.existsSync(filePath)) {
                    fs.unlinkSync(filePath);
                    console.log('Cleaned up:', filePath);
                }
            } catch (error) {
                console.log('Error cleaning up', filePath, error.message);
            }
        }
        this.files.clear();
    }
    
    setupCleanup() {
        // Clean up on process exit
        process.on('exit', () => this.cleanup());
        process.on('SIGINT', () => {
            this.cleanup();
            process.exit();
        });
    }
}

// Usage
const tempManager = new TempFileManager();
const temp1 = tempManager.createTempFile('Data 1');
const temp2 = tempManager.createTempFile('Data 2', '.json');

console.log('Temp files created:', temp1, temp2);

Use cases for temporary files:

const fs = require('fs');
const tmp = require('tmp');

// Use case 1: File processing pipeline
async function processLargeFile(inputPath) {
    const { path: tempPath, cleanup } = await tmp.file();
    
    try {
        // Step 1: Filter data to temp file
        await filterData(inputPath, tempPath);
        
        // Step 2: Transform data
        await transformData(tempPath);
        
        // Step 3: Generate output
        const result = await generateOutput(tempPath);
        
        return result;
    } finally {
        // Always clean up
        await cleanup();
    }
}

// Use case 2: Caching expensive operations
const cache = new Map();

async function getCachedData(key, dataGenerator) {
    if (cache.has(key)) {
        return fs.promises.readFile(cache.get(key), 'utf8');
    }
    
    const { path: tempPath, cleanup } = await tmp.file();
    const data = await dataGenerator();
    
    await fs.promises.writeFile(tempPath, data);
    cache.set(key, tempPath);
    
    // Set up cleanup after 1 hour
    setTimeout(() => {
        if (cache.has(key)) {
            fs.unlinkSync(cache.get(key));
            cache.delete(key);
        }
        cleanup();
    }, 60 * 60 * 1000);
    
    return data;
}

File uploads in Node.js typically involve handling multipart/form-data from HTTP requests and saving files securely.

Using multer (most popular solution):

const express = require('express');
const multer = require('multer');
const path = require('path');

const app = express();

// Configure storage
const storage = multer.diskStorage({
    destination: (req, file, cb) => {
        cb(null, 'uploads/');
    },
    filename: (req, file, cb) => {
        // Create safe filename
        const uniqueName = Date.now() + '-' + Math.round(Math.random() * 1E9);
        const extension = path.extname(file.originalname);
        cb(null, file.fieldname + '-' + uniqueName + extension);
    }
});

// File filter for security
const fileFilter = (req, file, cb) => {
    const allowedTypes = ['image/jpeg', 'image/png', 'application/pdf'];
    if (allowedTypes.includes(file.mimetype)) {
        cb(null, true);
    } else {
        cb(new Error('Invalid file type'), false);
    }
};

const upload = multer({
    storage: storage,
    limits: {
        fileSize: 5 * 1024 * 1024 // 5MB limit
    },
    fileFilter: fileFilter
});

// Handle single file upload
app.post('/upload', upload.single('document'), (req, res) => {
    if (!req.file) {
        return res.status(400).json({ error: 'No file uploaded' });
    }
    
    res.json({
        message: 'File uploaded successfully',
        filename: req.file.filename,
        size: req.file.size,
        path: req.file.path
    });
});

// Handle multiple files
app.post('/upload-multiple', upload.array('photos', 5), (req, res) => {
    const files = req.files.map(file => ({
        filename: file.filename,
        size: file.size
    }));
    
    res.json({ message: 'Files uploaded', files });
});

app.listen(3000);

Manual file upload handling with streams:

const http = require('http');
const fs = require('fs');
const path = require('path');

const server = http.createServer((req, res) => {
    if (req.method === 'POST' && req.url === '/upload') {
        const uploadDir = './uploads';
        if (!fs.existsSync(uploadDir)) {
            fs.mkdirSync(uploadDir);
        }
        
        const filename = `file-${Date.now()}.bin`;
        const filePath = path.join(uploadDir, filename);
        const writeStream = fs.createWriteStream(filePath);
        
        let uploadedBytes = 0;
        
        req.on('data', (chunk) => {
            uploadedBytes += chunk.length;
            writeStream.write(chunk);
            
            // Basic progress tracking
            console.log(`Uploaded: ${uploadedBytes} bytes`);
        });
        
        req.on('end', () => {
            writeStream.end();
            res.writeHead(200, { 'Content-Type': 'application/json' });
            res.end(JSON.stringify({
                message: 'Upload complete',
                filename: filename,
                size: uploadedBytes
            }));
        });
        
        req.on('error', (error) => {
            writeStream.destroy();
            fs.unlinkSync(filePath); // Clean up partial file
            res.writeHead(500, { 'Content-Type': 'application/json' });
            res.end(JSON.stringify({ error: 'Upload failed' }));
        });
        
    } else {
        res.writeHead(404);
        res.end('Not found');
    }
});

server.listen(3000);

Secure file upload validation:

const multer = require('multer');
const path = require('path');

function createSecureUpload(config = {}) {
    const { allowedTypes = [], maxSize = 5 * 1024 * 1024 } = config;
    
    const storage = multer.diskStorage({
        destination: 'uploads/',
        filename: (req, file, cb) => {
            // Sanitize filename
            const originalName = file.originalname;
            const sanitized = originalName.replace(/[^a-zA-Z0-9.\-]/g, '_');
            const uniqueName = Date.now() + '-' + sanitized;
            cb(null, uniqueName);
        }
    });
    
    const fileFilter = (req, file, cb) => {
        // Check file type
        if (allowedTypes.length > 0 && !allowedTypes.includes(file.mimetype)) {
            return cb(new Error(`File type ${file.mimetype} not allowed`), false);
        }
        
        // Check file extension
        const ext = path.extname(file.originalname).toLowerCase();
        const allowedExts = ['.jpg', '.jpeg', '.png', '.pdf', '.txt'];
        if (!allowedExts.includes(ext)) {
            return cb(new Error(`File extension ${ext} not allowed`), false);
        }
        
        cb(null, true);
    };
    
    return multer({
        storage,
        limits: { fileSize: maxSize },
        fileFilter
    });
}

// Usage
const secureUpload = createSecureUpload({
    allowedTypes: ['image/jpeg', 'image/png', 'application/pdf'],
    maxSize: 10 * 1024 * 1024 // 10MB
});

app.post('/secure-upload', secureUpload.single('file'), (req, res) => {
    // File is validated and safe to use
    res.json({ message: 'Secure upload successful', file: req.file });
});

Handling large file uploads with progress:

const express = require('express');
const multer = require('multer');
const app = express();

// In-memory storage for progress tracking (use Redis in production)
const uploadProgress = new Map();

const upload = multer({ dest: 'uploads/' });

app.post('/upload-with-progress', upload.single('largeFile'), (req, res) => {
    const uploadId = req.headers['x-upload-id'];
    
    if (uploadId && uploadProgress.has(uploadId)) {
        uploadProgress.delete(uploadId); // Clean up
    }
    
    res.json({ message: 'Upload complete', file: req.file });
});

// Progress tracking endpoint
app.get('/upload-progress/:id', (req, res) => {
    const progress = uploadProgress.get(req.params.id) || 0;
    res.json({ progress });
});

// Client would send chunks with progress tracking
/* 
// Client-side example:
const uploadFile = async (file, uploadId) => {
    const chunkSize = 1024 * 1024; // 1MB chunks
    const totalChunks = Math.ceil(file.size / chunkSize);
    
    for (let chunkIndex = 0; chunkIndex < totalChunks; chunkIndex++) {
        const chunk = file.slice(chunkIndex * chunkSize, (chunkIndex + 1) * chunkSize);
        const formData = new FormData();
        formData.append('chunk', chunk);
        formData.append('chunkIndex', chunkIndex);
        formData.append('totalChunks', totalChunks);
        formData.append('uploadId', uploadId);
        formData.append('filename', file.name);
        
        await fetch('/upload-chunk', {
            method: 'POST',
            body: formData
        });
        
        // Update progress
        const progress = ((chunkIndex + 1) / totalChunks) * 100;
        await fetch(`/upload-progress/${uploadId}`, {
            method: 'POST',
            body: JSON.stringify({ progress })
        });
    }
};
*/

Video files require special stream handling due to their large size and the need for efficient processing and streaming.

Basic video streaming server:

const http = require('http');
const fs = require('fs');
const path = require('path');

const server = http.createServer((req, res) => {
    const videoPath = path.join(__dirname, 'videos', 'sample.mp4');
    const videoSize = fs.statSync(videoPath).size;
    
    // Parse Range header for partial content
    const range = req.headers.range;
    
    if (range) {
        // Handle partial content request (video seeking)
        const parts = range.replace(/bytes=/, '').split('-');
        const start = parseInt(parts[0], 10);
        const end = parts[1] ? parseInt(parts[1], 10) : videoSize - 1;
        const chunksize = (end - start) + 1;
        
        // Create read stream for specific range
        const file = fs.createReadStream(videoPath, { start, end });
        
        res.writeHead(206, {
            'Content-Range': `bytes ${start}-${end}/${videoSize}`,
            'Accept-Ranges': 'bytes',
            'Content-Length': chunksize,
            'Content-Type': 'video/mp4'
        });
        
        file.pipe(res);
        
    } else {
        // Full video request
        res.writeHead(200, {
            'Content-Length': videoSize,
            'Content-Type': 'video/mp4'
        });
        
        fs.createReadStream(videoPath).pipe(res);
    }
});

server.listen(3000, () => {
    console.log('Video server running on port 3000');
});

Video processing with ffmpeg and streams:

const { spawn } = require('child_process');
const fs = require('fs');
const path = require('path');

function convertVideo(inputPath, outputPath, format = 'mp4') {
    return new Promise((resolve, reject) => {
        const ffmpeg = spawn('ffmpeg', [
            '-i', inputPath,
            '-c:v', 'libx264',
            '-c:a', 'aac',
            '-f', format,
            'pipe:1' // Output to stdout
        ]);
        
        const outputStream = fs.createWriteStream(outputPath);
        let errorData = '';
        
        ffmpeg.stdout.pipe(outputStream);
        
        ffmpeg.stderr.on('data', (data) => {
            errorData += data.toString();
        });
        
        ffmpeg.on('close', (code) => {
            if (code === 0) {
                resolve(outputPath);
            } else {
                reject(new Error(`FFmpeg failed: ${errorData}`));
            }
        });
        
        ffmpeg.on('error', reject);
    });
}

// Usage
convertVideo('input.avi', 'output.mp4')
    .then(outputPath => console.log('Video converted:', outputPath))
    .catch(error => console.log('Conversion failed:', error));

Video thumbnail generation:

const { spawn } = require('child_process');
const fs = require('fs');

function generateThumbnail(videoPath, thumbnailPath, time = '00:00:01') {
    return new Promise((resolve, reject) => {
        const ffmpeg = spawn('ffmpeg', [
            '-i', videoPath,
            '-ss', time,           // Seek to specific time
            '-vframes', '1',       // Capture 1 frame
            '-vf', 'scale=320:-1', // Resize to 320px width
            '-f', 'image2',
            'pipe:1'
        ]);
        
        const thumbnailStream = fs.createWriteStream(thumbnailPath);
        let errorData = '';
        
        ffmpeg.stdout.pipe(thumbnailStream);
        ffmpeg.stderr.on('data', (data) => errorData += data.toString());
        
        ffmpeg.on('close', (code) => {
            if (code === 0) {
                resolve(thumbnailPath);
            } else {
                reject(new Error(`Thumbnail generation failed: ${errorData}`));
            }
        });
        
        ffmpeg.on('error', reject);
    });
}

Video streaming with adaptive bitrate:

const express = require('express');
const fs = require('fs');
const path = require('path');

const app = express();

// Serve different quality versions
app.get('/video/:quality/:filename', (req, res) => {
    const { quality, filename } = req.params;
    const videoDir = path.join(__dirname, 'videos', quality);
    const videoPath = path.join(videoDir, filename);
    
    if (!fs.existsSync(videoPath)) {
        return res.status(404).send('Video not found');
    }
    
    const videoSize = fs.statSync(videoPath).size;
    const range = req.headers.range;
    
    if (range) {
        const parts = range.replace(/bytes=/, '').split('-');
        const start = parseInt(parts[0], 10);
        const end = parts[1] ? parseInt(parts[1], 10) : videoSize - 1;
        const chunksize = (end - start) + 1;
        
        const file = fs.createReadStream(videoPath, { start, end });
        
        res.writeHead(206, {
            'Content-Range': `bytes ${start}-${end}/${videoSize}`,
            'Accept-Ranges': 'bytes',
            'Content-Length': chunksize,
            'Content-Type': 'video/mp4'
        });
        
        file.pipe(res);
    } else {
        res.writeHead(200, {
            'Content-Length': videoSize,
            'Content-Type': 'video/mp4'
        });
        fs.createReadStream(videoPath).pipe(res);
    }
});

// Serve master playlist for HLS
app.get('/master.m3u8', (req, res) => {
    const masterPlaylist = `#EXTM3U
#EXT-X-VERSION:3
#EXT-X-STREAM-INF:BANDWIDTH=800000,RESOLUTION=640x360
/video/360p/index.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=1400000,RESOLUTION=854x480
/video/480p/index.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=2800000,RESOLUTION=1280x720
/video/720p/index.m3u8`;
    
    res.set('Content-Type', 'application/vnd.apple.mpegurl');
    res.send(masterPlaylist);
});

app.listen(3000);

Video upload with progress and validation:

const multer = require('multer');
const path = require('path');

const videoStorage = multer.diskStorage({
    destination: 'uploads/videos/',
    filename: (req, file, cb) => {
        const uniqueName = `video-${Date.now()}${path.extname(file.originalname)}`;
        cb(null, uniqueName);
    }
});

const videoUpload = multer({
    storage: videoStorage,
    limits: {
        fileSize: 100 * 1024 * 1024 // 100MB limit
    },
    fileFilter: (req, file, cb) => {
        // Check if file is a video
        if (file.mimetype.startsWith('video/')) {
            cb(null, true);
        } else {
            cb(new Error('Only video files are allowed'), false);
        }
    }
});

app.post('/upload-video', videoUpload.single('video'), (req, res) => {
    // Additional video validation
    const videoInfo = {
        filename: req.file.filename,
        originalName: req.file.originalname,
        size: req.file.size,
        mimetype: req.file.mimetype,
        path: req.file.path
    };
    
    // You could now process the video further
    // - Generate thumbnails
    // - Create different quality versions
    // - Extract metadata
    
    res.json({
        message: 'Video uploaded successfully',
        video: videoInfo
    });
});

Buffers and streams are both used for handling binary data, but they serve different purposes and have different characteristics.

Buffers - Fixed-size memory chunks:

// Buffers store fixed amounts of data in memory
const buffer = Buffer.from('Hello World', 'utf8');
console.log('Buffer size:', buffer.length);
console.log('Buffer content:', buffer.toString());

// Buffers are good for small, complete data
fs.readFile('small-file.txt', (error, buffer) => {
    // Entire file content is loaded into memory
    console.log('File content as buffer:', buffer);
});

Streams - Continuous data flow:

// Streams process data in chunks as it becomes available
const stream = fs.createReadStream('large-file.txt');
stream.on('data', (chunk) => {
    // Only small chunks in memory at a time
    console.log('Received chunk:', chunk.length, 'bytes');
});

highWaterMark controls the internal buffer size of streams, affecting memory usage and backpressure behavior.

Configuring highWaterMark:

const fs = require('fs');

// Custom highWaterMark for better memory control
const readStream = fs.createReadStream('large-file.txt', {
    highWaterMark: 64 * 1024 // 64KB chunks instead of default 16KB
});

const writeStream = fs.createWriteStream('output.txt', {
    highWaterMark: 128 * 1024 // 128KB buffer
});

Backpressure occurs when data is being produced faster than it can be consumed, causing memory issues and potential crashes. It's like a traffic jam in your data pipeline.

Understanding backpressure:

const fs = require('fs');

// Problem: Fast producer, slow consumer
const fastReadStream = fs.createReadStream('huge-file.txt'); // Fast
const slowWriteStream = fs.createWriteStream('output.txt');  // Slow (maybe disk is busy)

// Without backpressure handling, this could cause memory issues
fastReadStream.pipe(slowWriteStream);

How backpressure works automatically with pipe():

const fs = require('fs');

// pipe() automatically handles backpressure
fs.createReadStream('input.txt')
  .pipe(fs.createWriteStream('output.txt'));
  
// What happens internally:
// 1. Read stream reads data until write stream's buffer is full
// 2. Write stream emits 'drain' event when buffer is empty
// 3. Read stream resumes reading
// 4. Process repeats until complete

Manual backpressure handling with drain event:

const fs = require('fs');

function copyWithBackpressure(source, destination) {
    return new Promise((resolve, reject) => {
        const readStream = fs.createReadStream(source);
        const writeStream = fs.createWriteStream(destination);
        
        readStream.on('data', (chunk) => {
            // Try to write the chunk
            const canContinue = writeStream.write(chunk);
            
            if (!canContinue) {
                // Pause reading until drain event
                readStream.pause();
                console.log('Backpressure: Pausing read stream');
                
                writeStream.once('drain', () => {
                    // Resume reading when buffer is empty
                    readStream.resume();
                    console.log('Backpressure: Resuming read stream');
                });
            }
        });
        
        readStream.on('end', () => {
            writeStream.end();
        });
        
        writeStream.on('finish', resolve);
        readStream.on('error', reject);
        writeStream.on('error', reject);
    });
}

// Usage
copyWithBackpressure('large-file.txt', 'copy.txt')
    .then(() => console.log('Copy completed with backpressure handling'))
    .catch(console.error);

Backpressure in transform streams:

const { Transform } = require('stream');

class SlowTransform extends Transform {
    constructor(options) {
        super(options);
    }
    
    _transform(chunk, encoding, callback) {
        console.log('Transforming chunk of', chunk.length, 'bytes');
        
        // Simulate slow processing
        setTimeout(() => {
            const transformed = chunk.toString().toUpperCase();
            this.push(transformed);
            callback();
        }, 100); // 100ms delay per chunk
    }
}

// This will naturally create backpressure
const slowTransform = new SlowTransform();

fs.createReadStream('input.txt', { highWaterMark: 1024 })
  .pipe(slowTransform)
  .pipe(fs.createWriteStream('output.txt'));

Monitoring backpressure:

const fs = require('fs');

function monitorStreams(source, destination) {
    const readStream = fs.createReadStream(source);
    const writeStream = fs.createWriteStream(destination);
    
    let readPaused = false;
    let drainEvents = 0;
    
    readStream.on('data', (chunk) => {
        const canContinue = writeStream.write(chunk);
        
        if (!canContinue && !readPaused) {
            readStream.pause();
            readPaused = true;
            console.log('🚦 Read stream PAUSED due to backpressure');
        }
    });
    
    writeStream.on('drain', () => {
        drainEvents++;
        if (readPaused) {
            readStream.resume();
            readPaused = false;
            console.log('✅ Read stream RESUMED after drain event #' + drainEvents);
        }
    });
    
    readStream.on('end', () => {
        writeStream.end();
        console.log('📊 Statistics:');
        console.log('  - Drain events:', drainEvents);
        console.log('  - Final read paused:', readPaused);
    });
    
    writeStream.on('finish', () => {
        console.log('✨ Write completed successfully');
    });
}

monitorStreams('large-file.txt', 'output.txt');

Backpressure solutions and best practices:

const fs = require('fs');

// Solution 1: Use pipeline() for automatic backpressure
const { pipeline } = require('stream');

pipeline(
    fs.createReadStream('input.txt'),
    fs.createWriteStream('output.txt'),
    (error) => {
        if (error) {
            console.log('Pipeline failed:', error);
        } else {
            console.log('Pipeline completed');
        }
    }
);

// Solution 2: Adjust highWaterMark for your use case
const optimizedReadStream = fs.createReadStream('file.txt', {
    highWaterMark: 64 * 1024 // 64KB chunks
});

const optimizedWriteStream = fs.createWriteStream('output.txt', {
    highWaterMark: 128 * 1024 // 128KB buffer
});

// Solution 3: Use async processing for CPU-intensive operations
class AsyncTransform extends Transform {
    async _transform(chunk, encoding, callback) {
        try {
            // Simulate async operation (database call, API request, etc.)
            const result = await processChunkAsync(chunk);
            this.push(result);
            callback();
        } catch (error) {
            callback(error);
        }
    }
}

Signs of backpressure problems:

• High memory usage
• Streams frequently pausing and resuming
• Slow performance despite fast producers
• Application crashes with memory errors

File size detection is essential for validation, progress tracking, and memory management. Node.js provides several methods to get file size information.

Using fs.stat() for file size:

const fs = require('fs');

// Async file size detection
fs.stat('document.pdf', (error, stats) => {
    if (error) {
        console.log('Error getting file stats:', error);
        return;
    }
    
    console.log('File size:', stats.size, 'bytes');
    console.log('Human readable:', formatFileSize(stats.size));
    console.log('Is file:', stats.isFile());
    console.log('Is directory:', stats.isDirectory());
    console.log('Created:', stats.birthtime);
    console.log('Modified:', stats.mtime);
});

// Helper function for human-readable sizes
function formatFileSize(bytes) {
    const units = ['B', 'KB', 'MB', 'GB', 'TB'];
    let size = bytes;
    let unitIndex = 0;
    
    while (size >= 1024 && unitIndex < units.length - 1) {
        size /= 1024;
        unitIndex++;
    }
    
    return `${size.toFixed(2)} ${units[unitIndex]}`;
}

Using fs.promises.stat():

const fs = require('fs').promises;

async function getFileInfo(filePath) {
    try {
        const stats = await fs.stat(filePath);
        
        return {
            size: stats.size,
            sizeHuman: formatFileSize(stats.size),
            isFile: stats.isFile(),
            isDirectory: stats.isDirectory(),
            created: stats.birthtime,
            modified: stats.mtime,
            accessed: stats.atime
        };
    } catch (error) {
        console.log('Error getting file info:', error);
        throw error;
    }
}

// Usage
getFileInfo('large-video.mp4')
    .then(info => {
        console.log('File information:');
        console.log('  Size:', info.sizeHuman);
        console.log('  Type:', info.isFile ? 'File' : 'Directory');
        console.log('  Modified:', info.modified.toLocaleString());
    })
    .catch(console.error);

Getting size of multiple files:

const fs = require('fs').promises;
const path = require('path');

async function getDirectorySize(directoryPath) {
    try {
        const files = await fs.readdir(directoryPath);
        let totalSize = 0;
        const fileDetails = [];
        
        for (const file of files) {
            const filePath = path.join(directoryPath, file);
            const stats = await fs.stat(filePath);
            
            if (stats.isFile()) {
                totalSize += stats.size;
                fileDetails.push({
                    name: file,
                    size: stats.size,
                    sizeHuman: formatFileSize(stats.size)
                });
            }
        }
        
        return {
            totalSize,
            totalSizeHuman: formatFileSize(totalSize),
            fileCount: fileDetails.length,
            files: fileDetails.sort((a, b) => b.size - a.size) // Sort by size descending
        };
    } catch (error) {
        console.log('Error reading directory:', error);
        throw error;
    }
}

// Usage
getDirectorySize('./documents')
    .then(result => {
        console.log(`Directory contains ${result.fileCount} files`);
        console.log(`Total size: ${result.totalSizeHuman}`);
        console.log('Largest files:');
        result.files.slice(0, 5).forEach(file => {
            console.log(`  ${file.name}: ${file.sizeHuman}`);
        });
    });

File size validation for uploads:

const fs = require('fs');

class FileSizeValidator {
    constructor(maxSizeBytes) {
        this.maxSizeBytes = maxSizeBytes;
    }
    
    async validate(filePath) {
        try {
            const stats = await fs.promises.stat(filePath);
            
            if (stats.size > this.maxSizeBytes) {
                const maxSizeHuman = formatFileSize(this.maxSizeBytes);
                const actualSizeHuman = formatFileSize(stats.size);
                throw new Error(
                    `File too large: ${actualSizeHuman} exceeds maximum ${maxSizeHuman}`
                );
            }
            
            return {
                isValid: true,
                size: stats.size,
                sizeHuman: formatFileSize(stats.size)
            };
        } catch (error) {
            return {
                isValid: false,
                error: error.message
            };
        }
    }
}

// Usage
const validator = new FileSizeValidator(10 * 1024 * 1024); // 10MB limit

validator.validate('uploaded-file.jpg')
    .then(result => {
        if (result.isValid) {
            console.log('✅ File size OK:', result.sizeHuman);
        } else {
            console.log('❌ File validation failed:', result.error);
        }
    });

Monitoring file size changes:

const fs = require('fs');

class FileSizeMonitor {
    constructor(filePath, intervalMs = 5000) {
        this.filePath = filePath;
        this.intervalMs = intervalMs;
        this.previousSize = 0;
        this.intervalId = null;
    }
    
    start() {
        console.log(`Starting file size monitor for: ${this.filePath}`);
        
        this.intervalId = setInterval(async () => {
            try {
                const stats = await fs.promises.stat(this.filePath);
                const currentSize = stats.size;
                
                if (currentSize !== this.previousSize) {
                    const change = currentSize - this.previousSize;
                    const changeSymbol = change > 0 ? '📈' : '📉';
                    
                    console.log(
                        `${changeSymbol} File size: ${formatFileSize(currentSize)} ` +
                        `(${change > 0 ? '+' : ''}${formatFileSize(change)})`
                    );
                    
                    this.previousSize = currentSize;
                }
            } catch (error) {
                console.log('Error monitoring file:', error.message);
            }
        }, this.intervalMs);
    }
    
    stop() {
        if (this.intervalId) {
            clearInterval(this.intervalId);
            console.log('File size monitor stopped');
        }
    }
}

// Usage - great for monitoring log files or downloads
const monitor = new FileSizeMonitor('application.log', 2000);
monitor.start();

// Stop after 30 seconds
setTimeout(() => monitor.stop(), 30000);

Getting file size without loading content:

const fs = require('fs');

// Efficient way to get size without reading file content
function getFileSizeSync(filePath) {
    try {
        const stats = fs.statSync(filePath);
        return stats.size;
    } catch (error) {
        return -1; // File doesn't exist or error
    }
}

// For very large files, use streams to estimate size
function estimateFileSize(filePath) {
    return new Promise((resolve, reject) => {
        const stream = fs.createReadStream(filePath);
        let size = 0;
        
        stream.on('data', (chunk) => {
            size += chunk.length;
        });
        
        stream.on('end', () => {
            resolve(size);
        });
        
        stream.on('error', reject);
        
        // Read just enough to get size, then destroy
        stream.on('readable', () => {
            stream.destroy(); // Stop reading
            resolve(size);
        });
    });
}

Reading specific byte ranges is essential for partial file access, seeking in large files, and efficient data extraction.

Using fs.read() with file descriptors:

const fs = require('fs');

function readBytes(filePath, start, length) {
    return new Promise((resolve, reject) => {
        fs.open(filePath, 'r', (error, fd) => {
            if (error) {
                reject(error);
                return;
            }
            
            const buffer = Buffer.alloc(length);
            
            fs.read(fd, buffer, 0, length, start, (error, bytesRead, buffer) => {
                fs.close(fd, (closeError) => {
                    if (error) {
                        reject(error);
                    } else if (closeError) {
                        reject(closeError);
                    } else {
                        resolve({
                            bytesRead,
                            data: buffer.slice(0, bytesRead)
                        });
                    }
                });
            });
        });
    });
}

// Usage: Read first 100 bytes
readBytes('large-file.bin', 0, 100)
    .then(({ bytesRead, data }) => {
        console.log(`Read ${bytesRead} bytes`);
        console.log('Hex dump:', data.toString('hex'));
    })
    .catch(console.error);

Using createReadStream with start/end options:

const fs = require('fs');

function readFileRange(filePath, start, end) {
    return new Promise((resolve, reject) => {
        const chunks = [];
        const stream = fs.createReadStream(filePath, {
            start: start,
            end: end - 1, // end is inclusive, so subtract 1
            highWaterMark: 64 * 1024 // 64KB chunks
        });
        
        stream.on('data', (chunk) => {
            chunks.push(chunk);
        });
        
        stream.on('end', () => {
            const data = Buffer.concat(chunks);
            resolve({
                start,
                end,
                length: data.length,
                data: data
            });
        });
        
        stream.on('error', reject);
    });
}

// Usage: Read bytes 1000-2000
readFileRange('large-file.txt', 1000, 2000)
    .then(result => {
        console.log(`Read bytes ${result.start}-${result.end}: ${result.length} bytes`);
        console.log('Content:', result.data.toString('utf8'));
    });

Reading file headers and metadata:

const fs = require('fs');

class FileHeaderReader {
    static async readJPEGHeader(filePath) {
        // JPEG files start with FF D8 FF
        const header = await readBytes(filePath, 0, 20);
        const signature = header.data.toString('hex', 0, 4);
        
        if (signature !== 'ffd8ff') {
            throw new Error('Not a valid JPEG file');
        }
        
        return {
            type: 'JPEG',
            signature: signature,
            data: header.data
        };
    }
    
    static async readPNGHeader(filePath) {
        // PNG files start with 89 50 4E 47
        const header = await readBytes(filePath, 0, 8);
        const signature = header.data.toString('hex', 0, 8);
        
        if (signature !== '89504e470d0a1a0a') {
            throw new Error('Not a valid PNG file');
        }
        
        return {
            type: 'PNG',
            signature: signature,
            data: header.data
        };
    }
    
    static async readFileType(filePath) {
        const header = await readBytes(filePath, 0, 8);
        const hexSignature = header.data.toString('hex');
        
        const signatures = {
            'ffd8ff': 'JPEG',
            '89504e470d0a1a0a': 'PNG',
            '47494638': 'GIF',
            '25504446': 'PDF',
            '504b0304': 'ZIP',
            '7f454c46': 'ELF' // Executable
        };
        
        for (const [signature, type] of Object.entries(signatures)) {
            if (hexSignature.startsWith(signature)) {
                return type;
            }
        }
        
        return 'Unknown';
    }
}

// Usage
FileHeaderReader.readFileType('image.jpg')
    .then(type => console.log('File type:', type))
    .catch(console.error);

Efficient large file processing with chunks:

const fs = require('fs');

class FileChunkReader {
    constructor(filePath, chunkSize = 64 * 1024) {
        this.filePath = filePath;
        this.chunkSize = chunkSize;
        this.position = 0;
    }
    
    async next() {
        try {
            const result = await readBytes(this.filePath, this.position, this.chunkSize);
            
            if (result.bytesRead === 0) {
                return { done: true }; // End of file
            }
            
            const chunk = {
                data: result.data,
                start: this.position,
                end: this.position + result.bytesRead - 1,
                size: result.bytesRead
            };
            
            this.position += result.bytesRead;
            
            return { done: false, value: chunk };
        } catch (error) {
            throw error;
        }
    }
    
    async *[Symbol.asyncIterator]() {
        let chunk;
        do {
            chunk = await this.next();
            if (!chunk.done) {
                yield chunk.value;
            }
        } while (!chunk.done);
    }
}

// Usage with async iteration
async function processLargeFile(filePath) {
    const reader = new FileChunkReader(filePath, 1024 * 1024); // 1MB chunks
    
    let totalBytes = 0;
    
    for await (const chunk of reader) {
        console.log(`Processing chunk: ${chunk.start}-${chunk.end} (${chunk.size} bytes)`);
        
        // Process the chunk here
        totalBytes += chunk.size;
        
        // Simulate processing time
        await new Promise(resolve => setTimeout(resolve, 10));
    }
    
    console.log(`Total bytes processed: ${totalBytes}`);
}

processLargeFile('huge-file.bin').catch(console.error);

Reading specific file structures:

const fs = require('fs');

async function readCSVRow(filePath, rowNumber, delimiter = ',') {
    // This is a simplified example - real CSV parsing is more complex
    const stats = await fs.promises.stat(filePath);
    const approximateRowSize = 100; // Estimate
    const startByte = Math.max(0, (rowNumber - 1) * approximateRowSize - 1000);
    const readSize = 2000; // Read extra to ensure we get the full row
    
    const chunk = await readFileRange(filePath, startByte, startByte + readSize);
    const text = chunk.data.toString('utf8');
    const lines = text.split('\n');
    
    // Find our target row (adjusting for the partial read)
    const targetLine = lines[lines.length - 2]; // Simplified
    
    if (targetLine) {
        return targetLine.split(delimiter);
    }
    
    return null;
}

// Reading specific binary structures
async function readBinaryStructure(filePath, structure) {
    const reads = structure.map(async ({ offset, size, name, type }) => {
        const data = await readBytes(filePath, offset, size);
        
        let value;
        switch (type) {
            case 'uint32':
                value = data.data.readUInt32LE(0);
                break;
            case 'string':
                value = data.data.toString('utf8').replace(/\0/g, '');
                break;
            case 'hex':
                value = data.data.toString('hex');
                break;
            default:
                value = data.data;
        }
        
        return { name, value, offset, size };
    });
    
    return Promise.all(reads);
}

// Usage: Read specific fields from a binary file
const fileStructure = [
    { offset: 0, size: 4, name: 'magicNumber', type: 'hex' },
    { offset: 4, size: 4, name: 'fileSize', type: 'uint32' },
    { offset: 8, size: 16, name: 'creator', type: 'string' }
];

readBinaryStructure('data.bin', fileStructure)
    .then(fields => {
        console.log('File structure:');
        fields.forEach(field => {
            console.log(`  ${field.name}: ${field.value} (offset: ${field.offset})`);
        });
    });

fs.access() checks a user's permissions for a file or directory without opening it. It's the modern replacement for the deprecated fs.exists().

Basic file existence checking:

const fs = require('fs');

// Check if file exists
fs.access('config.json', fs.constants.F_OK, (error) => {
    if (error) {
        console.log('File does not exist');
    } else {
        console.log('File exists');
    }
});

Checking multiple permissions:

const fs = require('fs');

// Check read and write permissions
fs.access('data.txt', fs.constants.R_OK | fs.constants.W_OK, (error) => {
    if (error) {
        console.log('Cannot read/write file:', error.message);
    } else {
        console.log('File is readable and writable');
    }
});

Available permission constants:

const fs = require('fs');

// Common permission constants
const permissions = {
    F_OK: fs.constants.F_OK, // File exists
    R_OK: fs.constants.R_OK, // Readable
    W_OK: fs.constants.W_OK, // Writable
    X_OK: fs.constants.X_OK  // Executable
};

// Check all permissions
fs.access('script.sh', 
    fs.constants.F_OK | fs.constants.R_OK | fs.constants.W_OK | fs.constants.X_OK,
    (error) => {
        if (error) {
            console.log('Missing permissions:', error.message);
        } else {
            console.log('File exists and is readable, writable, and executable');
        }
    }
);

Using fs.access() with promises:

const fs = require('fs').promises;

async function checkFileAccess(filePath) {
    try {
        // Check if file exists and is readable
        await fs.access(filePath, fs.constants.F_OK | fs.constants.R_OK);
        
        console.log('File is accessible and readable');
        return true;
        
    } catch (error) {
        if (error.code === 'ENOENT') {
            console.log('File does not exist');
        } else if (error.code === 'EACCES') {
            console.log('No permission to access file');
        } else {
            console.log('Other error:', error.message);
        }
        return false;
    }
}

// Usage
checkFileAccess('important-file.txt')
    .then(accessible => {
        if (accessible) {
            console.log('Proceeding with file operations...');
        }
    });

Safe file operations pattern:

const fs = require('fs').promises;

async function safeFileOperation(filePath, operation) {
    try {
        // First check if we can access the file
        await fs.access(filePath, fs.constants.F_OK | fs.constants.R_OK);
        
        // Then perform the actual operation
        return await operation(filePath);
        
    } catch (error) {
        if (error.code === 'ENOENT') {
            throw new Error(`File not found: ${filePath}`);
        } else if (error.code === 'EACCES') {
            throw new Error(`Permission denied: ${filePath}`);
        } else {
            throw error;
        }
    }
}

// Usage
async function readConfig() {
    return safeFileOperation('config.json', async (filePath) => {
        const data = await fs.readFile(filePath, 'utf8');
        return JSON.parse(data);
    });
}

readConfig()
    .then(config => console.log('Config loaded:', config))
    .catch(error => console.log('Error:', error.message));

Checking directory permissions:

const fs = require('fs').promises;

async function checkDirectoryAccess(dirPath) {
    try {
        await fs.access(dirPath, fs.constants.F_OK | fs.constants.R_OK | fs.constants.W_OK);
        
        console.log('Directory is accessible, readable, and writable');
        return true;
        
    } catch (error) {
        switch (error.code) {
            case 'ENOENT':
                console.log('Directory does not exist');
                break;
            case 'EACCES':
                console.log('No permission to access directory');
                break;
            case 'ENOTDIR':
                console.log('Path is not a directory');
                break;
            default:
                console.log('Unknown error:', error.message);
        }
        return false;
    }
}

// Usage
checkDirectoryAccess('/tmp')
    .then(accessible => {
        if (accessible) {
            console.log('Can use this directory for temporary files');
        }
    });

Why fs.access() is better than fs.exists():

// ❌ OLD WAY (deprecated and has race conditions)
if (fs.existsSync('file.txt')) {
    // File might be deleted between check and use
    fs.readFile('file.txt', (error, data) => {
        // Could get ENOENT error here!
    });
}

// ✅ NEW WAY (race condition safe)
fs.access('file.txt', fs.constants.F_OK, (error) => {
    if (error) {
        console.log('File does not exist or cannot be accessed');
    } else {
        // Safe to use the file
        fs.readFile('file.txt', (error, data) => {
            // Handle read operation
        });
    }
});

// ✅ EVEN BETTER: Just try the operation and handle errors
fs.readFile('file.txt', 'utf8', (error, data) => {
    if (error) {
        if (error.code === 'ENOENT') {
            console.log('File does not exist');
        } else if (error.code === 'EACCES') {
            console.log('No permission to read file');
        } else {
            console.log('Other error:', error);
        }
    } else {
        console.log('File content:', data);
    }
});

Common error codes from fs.access():

• ENOENT - File or directory does not exist
• EACCES - Permission denied
• ENOTDIR - Path is not a directory
• EISDIR - Path is a directory (when expecting file)

fs.readdir() reads the contents of a directory, returning an array of filenames. It's essential for file management, directory scanning, and building file explorers.

Basic directory reading:

const fs = require('fs');
const path = require('path');

// Read directory contents (async)
fs.readdir('./documents', (error, files) => {
    if (error) {
        console.log('Error reading directory:', error);
        return;
    }
    
    console.log('Files in directory:');
    files.forEach(file => {
        console.log('  -', file);
    });
});

Using fs.promises.readdir():

const fs = require('fs').promises;

async function listDirectory(dirPath) {
    try {
        const files = await fs.readdir(dirPath);
        
        console.log(`Contents of ${dirPath}:`);
        files.forEach(file => {
            console.log(`  📄 ${file}`);
        });
        
        return files;
    } catch (error) {
        console.log('Error reading directory:', error.message);
        throw error;
    }
}

// Usage
listDirectory('./src')
    .then(files => {
        console.log(`Found ${files.length} files`);
    })
    .catch(console.error);

Reading with file types using withFileTypes:

const fs = require('fs').promises;

async function listDirectoryDetailed(dirPath) {
    try {
        // Use withFileTypes to get Dirent objects
        const entries = await fs.readdir(dirPath, { withFileTypes: true });
        
        console.log(`Detailed contents of ${dirPath}:`);
        
        const result = {
            files: [],
            directories: [],
            symlinks: [],
            others: []
        };
        
        for (const entry of entries) {
            const info = {
                name: entry.name,
                path: path.join(dirPath, entry.name),
                isFile: entry.isFile(),
                isDirectory: entry.isDirectory(),
                isSymbolicLink: entry.isSymbolicLink()
            };
            
            if (entry.isFile()) {
                result.files.push(info);
                console.log(`  📄 ${entry.name} (file)`);
            } else if (entry.isDirectory()) {
                result.directories.push(info);
                console.log(`  📁 ${entry.name}/ (directory)`);
            } else if (entry.isSymbolicLink()) {
                result.symlinks.push(info);
                console.log(`  🔗 ${entry.name} (symlink)`);
            } else {
                result.others.push(info);
                console.log(`  ❓ ${entry.name} (other)`);
            }
        }
        
        return result;
    } catch (error) {
        console.log('Error reading directory:', error);
        throw error;
    }
}

// Usage
listDirectoryDetailed('./project')
    .then(({ files, directories }) => {
        console.log(`Found ${files.length} files and ${directories.length} directories`);
    });

Recursive directory scanning:

const fs = require('fs').promises;
const path = require('path');

async function scanDirectory(dirPath, maxDepth = 3, currentDepth = 0) {
    if (currentDepth > maxDepth) {
        return [];
    }
    
    try {
        const entries = await fs.readdir(dirPath, { withFileTypes: true });
        const results = [];
        
        for (const entry of entries) {
            const fullPath = path.join(dirPath, entry.name);
            const item = {
                name: entry.name,
                path: fullPath,
                depth: currentDepth,
                isFile: entry.isFile(),
                isDirectory: entry.isDirectory(),
                size: 0
            };
            
            if (entry.isFile()) {
                try {
                    const stats = await fs.stat(fullPath);
                    item.size = stats.size;
                    item.modified = stats.mtime;
                } catch (error) {
                    item.error = error.message;
                }
            }
            
            results.push(item);
            
            // Recursively scan subdirectories
            if (entry.isDirectory() && currentDepth < maxDepth) {
                const subItems = await scanDirectory(fullPath, maxDepth, currentDepth + 1);
                results.push(...subItems);
            }
        }
        
        return results;
    } catch (error) {
        console.log(`Error scanning ${dirPath}:`, error.message);
        return [];
    }
}

// Usage
scanDirectory('.', 2) // Scan up to 2 levels deep
    .then(files => {
        console.log(`Scanned ${files.length} items total`);
        
        const totalSize = files
            .filter(f => f.isFile)
            .reduce((sum, file) => sum + file.size, 0);
            
        console.log(`Total file size: ${formatFileSize(totalSize)}`);
        
        // Show largest files
        const largestFiles = files
            .filter(f => f.isFile)
            .sort((a, b) => b.size - a.size)
            .slice(0, 5);
            
        console.log('Largest files:');
        largestFiles.forEach(file => {
            console.log(`  ${file.path}: ${formatFileSize(file.size)}`);
        });
    });

Filtering and searching files:

const fs = require('fs').promises;
const path = require('path');

class FileSearcher {
    static async findByExtension(dirPath, extensions) {
        const files = await fs.readdir(dirPath, { withFileTypes: true });
        const matched = [];
        
        for (const entry of files) {
            if (entry.isFile()) {
                const ext = path.extname(entry.name).toLowerCase();
                if (extensions.includes(ext)) {
                    matched.push({
                        name: entry.name,
                        path: path.join(dirPath, entry.name),
                        extension: ext
                    });
                }
            }
        }
        
        return matched;
    }
    
    static async findByName(dirPath, pattern, caseSensitive = false) {
        const files = await fs.readdir(dirPath, { withFileTypes: true });
        const matched = [];
        const regex = new RegExp(
            caseSensitive ? pattern : pattern, 
            caseSensitive ? '' : 'i'
        );
        
        for (const entry of files) {
            if (regex.test(entry.name)) {
                matched.push({
                    name: entry.name,
                    path: path.join(dirPath, entry.name),
                    isFile: entry.isFile(),
                    isDirectory: entry.isDirectory()
                });
            }
        }
        
        return matched;
    }
    
    static async findLargeFiles(dirPath, minSizeBytes) {
        const files = await fs.readdir(dirPath, { withFileTypes: true });
        const largeFiles = [];
        
        for (const entry of files) {
            if (entry.isFile()) {
                const fullPath = path.join(dirPath, entry.name);
                try {
                    const stats = await fs.stat(fullPath);
                    if (stats.size >= minSizeBytes) {
                        largeFiles.push({
                            name: entry.name,
                            path: fullPath,
                            size: stats.size,
                            sizeHuman: formatFileSize(stats.size)
                        });
                    }
                } catch (error) {
                    // Skip files we can't stat
                }
            }
        }
        
        return largeFiles.sort((a, b) => b.size - a.size);
    }
}

// Usage examples
async function searchFiles() {
    // Find all JavaScript and TypeScript files
    const codeFiles = await FileSearcher.findByExtension('./src', ['.js', '.ts', '.jsx', '.tsx']);
    console.log('Code files:', codeFiles.map(f => f.name));
    
    // Find files containing "test" in name
    const testFiles = await FileSearcher.findByName('./src', 'test');
    console.log('Test files:', testFiles.map(f => f.name));
    
    // Find files larger than 1MB
    const largeFiles = await FileSearcher.findLargeFiles('.', 1024 * 1024);
    console.log('Large files:');
    largeFiles.forEach(file => {
        console.log(`  ${file.name}: ${file.sizeHuman}`);
    });
}

searchFiles().catch(console.error);

Directory monitoring with readdir:

const fs = require('fs').promises;

class DirectoryWatcher {
    constructor(dirPath, intervalMs = 5000) {
        this.dirPath = dirPath;
        this.intervalMs = intervalMs;
        this.previousFiles = new Set();
        this.intervalId = null;
    }
    
    async getCurrentFiles() {
        try {
            const files = await fs.readdir(this.dirPath);
            return new Set(files);
        } catch (error) {
            console.log('Error reading directory:', error.message);
            return new Set();
        }
    }
    
    start() {
        console.log(`Starting directory watcher for: ${this.dirPath}`);
        
        this.intervalId = setInterval(async () => {
            const currentFiles = await this.getCurrentFiles();
            
            // Find new files
            const added = [...currentFiles].filter(file => !this.previousFiles.has(file));
            // Find deleted files
            const removed = [...this.previousFiles].filter(file => !currentFiles.has(file));
            
            if (added.length > 0 || removed.length > 0) {
                console.log('📁 Directory changes detected:');
                added.forEach(file => console.log(`  ➕ Added: ${file}`));
                removed.forEach(file => console.log(`  ➖ Removed: ${file}`));
                
                // Update previous files
                this.previousFiles = currentFiles;
            }
        }, this.intervalMs);
    }
    
    stop() {
        if (this.intervalId) {
            clearInterval(this.intervalId);
            console.log('Directory watcher stopped');
        }
    }
}

// Usage
const watcher = new DirectoryWatcher('./uploads', 2000);
watcher.start();

// Stop after 1 minute
setTimeout(() => watcher.stop(), 60000);

The http module is a core Node.js module that provides functionality to create HTTP servers and make HTTP requests. It's the foundation for web servers and API clients in Node.js.

Importing the http module:

const http = require('http');

Key capabilities of the http module:

• Create HTTP servers to handle requests
• Make HTTP requests to other servers
• Handle HTTP methods (GET, POST, PUT, DELETE, etc.)
• Manage request/response headers
• Handle streaming data
• Work with cookies and sessions

Basic HTTP server example:

const http = require('http');

const server = http.createServer((req, res) => {
    res.writeHead(200, { 'Content-Type': 'text/plain' });
    res.end('Hello World!');
});

server.listen(3000, () => {
    console.log('Server running at http://localhost:3000/');
});

Making HTTP requests:

const http = require('http');

const options = {
    hostname: 'jsonplaceholder.typicode.com',
    port: 80,
    path: '/posts/1',
    method: 'GET'
};

const req = http.request(options, (res) => {
    let data = '';
    
    res.on('data', (chunk) => {
        data += chunk;
    });
    
    res.on('end', () => {
        console.log('Response:', JSON.parse(data));
    });
});

req.on('error', (error) => {
    console.log('Error:', error);
});

req.end();

Key classes in the http module:

• http.Server - HTTP server class
• http.ClientRequest - HTTP client requests
• http.ServerResponse - Server response object
• http.IncomingMessage - Incoming request object
• http.Agent - Manages connection persistence

The http module is low-level but powerful, giving you complete control over HTTP communication. Most web frameworks (like Express) are built on top of it.

Creating an HTTP server in Node.js is straightforward using the http module's createServer method. Here are different ways to create and configure servers.

Basic HTTP server:

const http = require('http');

// Create server with request handler
const server = http.createServer((request, response) => {
    response.writeHead(200, { 'Content-Type': 'text/plain' });
    response.end('Hello from Node.js server!');
});

// Start listening on port 3000
server.listen(3000, () => {
    console.log('Server is running on http://localhost:3000');
});

Server with different response types:

const http = require('http');

const server = http.createServer((req, res) => {
    // Set content type based on URL
    if (req.url === '/json') {
        res.writeHead(200, { 'Content-Type': 'application/json' });
        res.end(JSON.stringify({ message: 'Hello JSON', timestamp: new Date() }));
    } else if (req.url === '/html') {
        res.writeHead(200, { 'Content-Type': 'text/html' });
        res.end('
Hello HTML
Welcome to our server
');
    } else {
        res.writeHead(200, { 'Content-Type': 'text/plain' });
        res.end('Hello World!');
    }
});

server.listen(3000);

Using separate request event handlers:

const http = require('http');

// Create server without immediate request handler
const server = http.createServer();

// Handle requests using event listeners
server.on('request', (req, res) => {
    console.log(`${new Date().toISOString()} - ${req.method} ${req.url}`);
    
    res.writeHead(200, { 'Content-Type': 'text/plain' });
    res.end('Request received!');
});

// Handle server errors
server.on('error', (error) => {
    console.error('Server error:', error);
});

// Handle client errors
server.on('clientError', (error, socket) => {
    console.error('Client error:', error);
    socket.end('HTTP/1.1 400 Bad Request\r\n\r\n');
});

server.listen(3000, () => {
    console.log('Server ready on port 3000');
});

Advanced server with routing:

const http = require('http');
const url = require('url');

const server = http.createServer((req, res) => {
    const parsedUrl = url.parse(req.url, true);
    const pathname = parsedUrl.pathname;
    
    // Simple routing
    switch (pathname) {
        case '/':
            res.writeHead(200, { 'Content-Type': 'text/html' });
            res.end(`
                <html>
                    <body>
                        <h1>Home Page</h1>
                        <a href="/about">About</a>
                        <a href="/api/data">API</a>
                    </body>
                </html>
            `);
            break;
            
        case '/about':
            res.writeHead(200, { 'Content-Type': 'text/html' });
            res.end('<h1>About Us</h1><p>This is our about page</p>');
            break;
            
        case '/api/data':
            res.writeHead(200, { 'Content-Type': 'application/json' });
            res.end(JSON.stringify({
                users: ['John', 'Jane', 'Bob'],
                timestamp: new Date().toISOString()
            }));
            break;
            
        default:
            res.writeHead(404, { 'Content-Type': 'text/html' });
            res.end('<h1>404 - Page Not Found</h1>');
    }
});

server.listen(3000);

Server with request logging middleware:

const http = require('http');

function logger(req, res, next) {
    const timestamp = new Date().toISOString();
    console.log(`[${timestamp}] ${req.method} ${req.url} - ${req.headers['user-agent']}`);
    next();
}

function requestHandler(req, res) {
    res.writeHead(200, { 'Content-Type': 'text/plain' });
    res.end('Hello World!');
}

const server = http.createServer((req, res) => {
    logger(req, res, () => {
        requestHandler(req, res);
    });
});

server.listen(3000);

Key server methods:

• server.listen() - Start listening for connections
• server.close() - Stop the server from accepting new connections
• server.setTimeout() - Set socket timeout in milliseconds
• server.maxHeadersCount - Limit maximum header count

Graceful shutdown:

const server = http.createServer((req, res) => {
    res.end('Server is running');
}).listen(3000);

// Handle graceful shutdown
process.on('SIGTERM', () => {
    console.log('Received SIGTERM, shutting down gracefully');
    server.close(() => {
        console.log('Server closed');
        process.exit(0);
    });
});

GET requests are used to retrieve data from the server. They should be idempotent and safe, meaning they don't change server state.

Basic GET request handling:

const http = require('http');
const url = require('url');

const server = http.createServer((req, res) => {
    // Only handle GET requests
    if (req.method !== 'GET') {
        res.writeHead(405, { 'Content-Type': 'text/plain' });
        res.end('Method Not Allowed');
        return;
    }
    
    const parsedUrl = url.parse(req.url, true);
    const pathname = parsedUrl.pathname;
    
    // Handle different routes
    if (pathname === '/') {
        res.writeHead(200, { 'Content-Type': 'text/html' });
        res.end(`
            <html>
                <head><title>Home</title></head>
                <body>
                    <h1>Welcome</h1>
                    <form action="/search" method="GET">
                        <input type="text" name="q" placeholder="Search...">
                        <button type="submit">Search</button>
                    </form>
                </body>
            </html>
        `);
    } else if (pathname === '/search') {
        // Handle search with query parameters
        const query = parsedUrl.query.q || '';
        res.writeHead(200, { 'Content-Type': 'application/json' });
        res.end(JSON.stringify({
            searchTerm: query,
            results: [`Result 1 for ${query}`, `Result 2 for ${query}`],
            timestamp: new Date().toISOString()
        }));
    } else {
        res.writeHead(404, { 'Content-Type': 'text/plain' });
        res.end('404 - Not Found');
    }
});

server.listen(3000);

GET request with query parameters:

const http = require('http');
const url = require('url');

const products = [
    { id: 1, name: 'Laptop', price: 999 },
    { id: 2, name: 'Phone', price: 599 },
    { id: 3, name: 'Tablet', price: 399 }
];

const server = http.createServer((req, res) => {
    if (req.method === 'GET' && req.url.startsWith('/api/products')) {
        const parsedUrl = url.parse(req.url, true);
        const query = parsedUrl.query;
        
        let responseData = products;
        
        // Filter by name if provided
        if (query.name) {
            responseData = responseData.filter(product => 
                product.name.toLowerCase().includes(query.name.toLowerCase())
            );
        }
        
        // Filter by max price if provided
        if (query.maxPrice) {
            responseData = responseData.filter(product => 
                product.price <= parseFloat(query.maxPrice)
            );
        }
        
        // Sort if specified
        if (query.sort) {
            responseData.sort((a, b) => {
                if (query.sort === 'price') return a.price - b.price;
                if (query.sort === 'name') return a.name.localeCompare(b.name);
                return 0;
            });
        }
        
        res.writeHead(200, { 
            'Content-Type': 'application/json',
            'Cache-Control': 'public, max-age=300' // Cache for 5 minutes
        });
        res.end(JSON.stringify({
            products: responseData,
            count: responseData.length,
            filters: query
        }));
    }
});

server.listen(3000);

GET request with path parameters simulation:

const http = require('http');
const url = require('url');

const users = {
    1: { id: 1, name: 'John Doe', email: 'john@example.com' },
    2: { id: 2, name: 'Jane Smith', email: 'jane@example.com' },
    3: { id: 3, name: 'Bob Johnson', email: 'bob@example.com' }
};

const server = http.createServer((req, res) => {
    if (req.method === 'GET') {
        // Match /api/users/123 pattern
        const userMatch = req.url.match(/^\/api\/users\/(\d+)$/);
        
        if (userMatch) {
            const userId = parseInt(userMatch[1]);
            const user = users[userId];
            
            if (user) {
                res.writeHead(200, { 'Content-Type': 'application/json' });
                res.end(JSON.stringify(user));
            } else {
                res.writeHead(404, { 'Content-Type': 'application/json' });
                res.end(JSON.stringify({ error: 'User not found' }));
            }
            return;
        }
        
        // Get all users
        if (req.url === '/api/users') {
            res.writeHead(200, { 'Content-Type': 'application/json' });
            res.end(JSON.stringify(Object.values(users)));
            return;
        }
    }
    
    res.writeHead(404, { 'Content-Type': 'text/plain' });
    res.end('Not Found');
});

GET with conditional requests (ETag):

const http = require('http');
const crypto = require('crypto');

const server = http.createServer((req, res) => {
    if (req.method === 'GET' && req.url === '/api/config') {
        const config = {
            appName: 'My Application',
            version: '1.0.0',
            features: ['auth', 'payments', 'reports'],
            lastUpdated: new Date().toISOString()
        };
        
        // Generate ETag from config content
        const configString = JSON.stringify(config);
        const etag = crypto.createHash('md5').update(configString).digest('hex');
        
        // Check if client has cached version
        const clientETag = req.headers['if-none-match'];
        
        if (clientETag === etag) {
            // Content hasn't changed
            res.writeHead(304, { // Not Modified
                'ETag': etag,
                'Cache-Control': 'public, max-age=300'
            });
            res.end();
        } else {
            // Send fresh content
            res.writeHead(200, {
                'Content-Type': 'application/json',
                'ETag': etag,
                'Cache-Control': 'public, max-age=300'
            });
            res.end(configString);
        }
    }
});

GET request with pagination:

const http = require('http');
const url = require('url');

// Sample data
const allItems = Array.from({ length: 100 }, (_, i) => ({
    id: i + 1,
    name: `Item ${i + 1}`,
    value: Math.random() * 1000
}));

const server = http.createServer((req, res) => {
    if (req.method === 'GET' && req.url.startsWith('/api/items')) {
        const parsedUrl = url.parse(req.url, true);
        const query = parsedUrl.query;
        
        const page = parseInt(query.page) || 1;
        const limit = parseInt(query.limit) || 10;
        const startIndex = (page - 1) * limit;
        const endIndex = startIndex + limit;
        
        const items = allItems.slice(startIndex, endIndex);
        const totalPages = Math.ceil(allItems.length / limit);
        
        res.writeHead(200, { 'Content-Type': 'application/json' });
        res.end(JSON.stringify({
            items,
            pagination: {
                page,
                limit,
                totalItems: allItems.length,
                totalPages,
                hasNext: page < totalPages,
                hasPrev: page > 1
            }
        }));
    }
});

Best practices for GET requests:

• Use appropriate HTTP status codes (200, 404, 304)
• Implement caching with ETags and Cache-Control
• Validate and sanitize query parameters
• Use pagination for large datasets
• Support filtering and sorting via query parameters
• Set proper Content-Type headers

POST requests are used to submit data to the server, typically for creating new resources. Unlike GET requests, POST requests can have a request body.

Basic POST request handling:

const http = require('http');

const server = http.createServer((req, res) => {
    if (req.method === 'POST') {
        let body = '';
        
        // Collect data from request
        req.on('data', (chunk) => {
            body += chunk.toString();
        });
        
        // Process when data is complete
        req.on('end', () => {
            try {
                const data = JSON.parse(body);
                
                // Process the data (e.g., save to database)
                console.log('Received data:', data);
                
                res.writeHead(201, { 
                    'Content-Type': 'application/json',
                    'Location': '/api/users/123' // URL of created resource
                });
                res.end(JSON.stringify({
                    message: 'Resource created successfully',
                    id: 123,
                    data: data
                }));
                
            } catch (error) {
                res.writeHead(400, { 'Content-Type': 'application/json' });
                res.end(JSON.stringify({
                    error: 'Invalid JSON data'
                }));
            }
        });
        
    } else {
        res.writeHead(405, { 'Content-Type': 'text/plain' });
        res.end('Method Not Allowed');
    }
});

server.listen(3000);

Handling different content types:

const http = require('http');
const querystring = require('querystring');

const server = http.createServer((req, res) => {
    if (req.method === 'POST') {
        const contentType = req.headers['content-type'];
        let body = '';
        
        req.on('data', (chunk) => {
            body += chunk.toString();
        });
        
        req.on('end', () => {
            let parsedData;
            
            try {
                if (contentType === 'application/json') {
                    parsedData = JSON.parse(body);
                } else if (contentType === 'application/x-www-form-urlencoded') {
                    parsedData = querystring.parse(body);
                } else if (contentType === 'text/plain') {
                    parsedData = { text: body };
                } else {
                    throw new Error('Unsupported content type');
                }
                
                // Process the data
                console.log('Parsed data:', parsedData);
                
                res.writeHead(200, { 'Content-Type': 'application/json' });
                res.end(JSON.stringify({
                    status: 'success',
                    received: parsedData
                }));
                
            } catch (error) {
                res.writeHead(400, { 'Content-Type': 'application/json' });
                res.end(JSON.stringify({
                    error: error.message
                }));
            }
        });
        
    } else {
        res.writeHead(405, { 'Content-Type': 'text/plain' });
        res.end('Method Not Allowed');
    }
});

POST with file upload handling:

const http = require('http');
const fs = require('fs');
const path = require('path');

const server = http.createServer((req, res) => {
    if (req.method === 'POST' && req.url === '/upload') {
        const contentLength = parseInt(req.headers['content-length']);
        const contentType = req.headers['content-type'];
        
        // Basic validation
        if (contentLength > 10 * 1024 * 1024) { // 10MB limit
            res.writeHead(413, { 'Content-Type': 'application/json' });
            return res.end(JSON.stringify({ error: 'File too large' }));
        }
        
        if (!contentType.includes('multipart/form-data')) {
            res.writeHead(415, { 'Content-Type': 'application/json' });
            return res.end(JSON.stringify({ error: 'Unsupported media type' }));
        }
        
        const uploadDir = './uploads';
        if (!fs.existsSync(uploadDir)) {
            fs.mkdirSync(uploadDir);
        }
        
        const filename = `upload-${Date.now()}.dat`;
        const filepath = path.join(uploadDir, filename);
        const writeStream = fs.createWriteStream(filepath);
        
        let uploadedBytes = 0;
        
        req.on('data', (chunk) => {
            uploadedBytes += chunk.length;
            writeStream.write(chunk);
            
            // Simple progress tracking
            const progress = ((uploadedBytes / contentLength) * 100).toFixed(1);
            console.log(`Upload progress: ${progress}%`);
        });
        
        req.on('end', () => {
            writeStream.end();
            res.writeHead(200, { 'Content-Type': 'application/json' });
            res.end(JSON.stringify({
                message: 'Upload successful',
                filename: filename,
                size: uploadedBytes,
                path: filepath
            }));
        });
        
        req.on('error', (error) => {
            writeStream.destroy();
            fs.unlinkSync(filepath); // Clean up
            res.writeHead(500, { 'Content-Type': 'application/json' });
            res.end(JSON.stringify({ error: 'Upload failed' }));
        });
        
    } else {
        res.writeHead(404, { 'Content-Type': 'text/plain' });
        res.end('Not Found');
    }
});

POST with authentication and validation:

const http = require('http');
const crypto = require('crypto');

// In-memory store (use database in production)
const users = [];
const sessions = {};

const server = http.createServer((req, res) => {
    if (req.method === 'POST') {
        if (req.url === '/api/register') {
            handleRegistration(req, res);
        } else if (req.url === '/api/login') {
            handleLogin(req, res);
        } else {
            res.writeHead(404, { 'Content-Type': 'application/json' });
            res.end(JSON.stringify({ error: 'Endpoint not found' }));
        }
    }
});

function handleRegistration(req, res) {
    let body = '';
    
    req.on('data', (chunk) => {
        body += chunk.toString();
    });
    
    req.on('end', () => {
        try {
            const { email, password, name } = JSON.parse(body);
            
            // Validation
            if (!email || !password || !name) {
                throw new Error('Missing required fields');
            }
            
            if (users.find(user => user.email === email)) {
                throw new Error('User already exists');
            }
            
            if (password.length < 6) {
                throw new Error('Password too short');
            }
            
            // Create user (in real app, hash the password!)
            const user = {
                id: users.length + 1,
                email,
                password, // In production, use bcrypt!
                name,
                createdAt: new Date()
            };
            
            users.push(user);
            
            res.writeHead(201, { 'Content-Type': 'application/json' });
            res.end(JSON.stringify({
                message: 'User registered successfully',
                user: { id: user.id, email: user.email, name: user.name }
            }));
            
        } catch (error) {
            res.writeHead(400, { 'Content-Type': 'application/json' });
            res.end(JSON.stringify({ error: error.message }));
        }
    });
}

function handleLogin(req, res) {
    let body = '';
    
    req.on('data', (chunk) => {
        body += chunk.toString();
    });
    
    req.on('end', () => {
        try {
            const { email, password } = JSON.parse(body);
            const user = users.find(u => u.email === email && u.password === password);
            
            if (!user) {
                throw new Error('Invalid credentials');
            }
            
            // Create session
            const sessionId = crypto.randomBytes(16).toString('hex');
            sessions[sessionId] = {
                userId: user.id,
                createdAt: new Date()
            };
            
            res.writeHead(200, {
                'Content-Type': 'application/json',
                'Set-Cookie': `sessionId=${sessionId}; HttpOnly; Max-Age=3600`
            });
            res.end(JSON.stringify({
                message: 'Login successful',
                user: { id: user.id, email: user.email, name: user.name }
            }));
            
        } catch (error) {
            res.writeHead(401, { 'Content-Type': 'application/json' });
            res.end(JSON.stringify({ error: error.message }));
        }
    });
}

POST with rate limiting:

const http = require('http');

const requestCounts = new Map();

function rateLimit(ip, limit = 100, windowMs = 60000) {
    const now = Date.now();
    const windowStart = now - windowMs;
    
    if (!requestCounts.has(ip)) {
        requestCounts.set(ip, []);
    }
    
    const requests = requestCounts.get(ip);
    
    // Remove old requests outside the window
    while (requests.length > 0 && requests[0] < windowStart) {
        requests.shift();
    }
    
    // Check if over limit
    if (requests.length >= limit) {
        return false;
    }
    
    // Add current request
    requests.push(now);
    return true;
}

const server = http.createServer((req, res) => {
    const clientIP = req.socket.remoteAddress;
    
    if (req.method === 'POST') {
        // Apply rate limiting
        if (!rateLimit(clientIP, 10, 60000)) { // 10 requests per minute
            res.writeHead(429, { 
                'Content-Type': 'application/json',
                'Retry-After': '60'
            });
            return res.end(JSON.stringify({
                error: 'Too many requests',
                message: 'Please try again later'
            }));
        }
        
        // Process the request
        let body = '';
        req.on('data', (chunk) => body += chunk.toString());
        req.on('end', () => {
            res.writeHead(200, { 'Content-Type': 'application/json' });
            res.end(JSON.stringify({ message: 'Request processed' }));
        });
    }
});

Best practices for POST requests:

• Always validate and sanitize input data
• Use appropriate status codes (201 for created, 400 for bad request)
• Implement rate limiting to prevent abuse
• Handle different content types properly
• Use HTTPS in production for security
• Consider using CSRF protection
• Set proper CORS headers if needed

In Node.js HTTP servers, the request (http.IncomingMessage) and response (http.ServerResponse) objects represent the incoming HTTP request and outgoing response.

Request Object (http.IncomingMessage):

const http = require('http');

const server = http.createServer((req, res) => {
    // Request properties
    console.log('HTTP Method:', req.method);
    console.log('URL:', req.url);
    console.log('HTTP Version:', req.httpVersion);
    console.log('Headers:', req.headers);
    console.log('Remote Address:', req.socket.remoteAddress);
    console.log('Remote Port:', req.socket.remotePort);
    
    // Read request body
    let body = '';
    req.on('data', chunk => body += chunk.toString());
    req.on('end', () => {
        console.log('Request Body:', body);
    });
});

Common request properties:

const http = require('http');

const server = http.createServer((req, res) => {
    // Method and URL
    console.log('Method:', req.method); // GET, POST, etc.
    console.log('URL:', req.url);       // /path?query=string
    
    // Headers (all lowercase)
    console.log('User Agent:', req.headers['user-agent']);
    console.log('Content-Type:', req.headers['content-type']);
    console.log('Authorization:', req.headers['authorization']);
    console.log('Accept:', req.headers['accept']);
    
    // Connection info
    console.log('Remote IP:', req.socket.remoteAddress);
    console.log('Remote Family:', req.socket.remoteFamily);
    console.log('Secure:', req.socket.encrypted ? 'HTTPS' : 'HTTP');
    
    // URL components (with url module)
    const url = require('url');
    const parsedUrl = url.parse(req.url, true);
    console.log('Pathname:', parsedUrl.pathname);
    console.log('Query:', parsedUrl.query);
    
    // Request timing
    console.log('Request received at:', new Date().toISOString());
});

Response Object (http.ServerResponse):

const http = require('http');

const server = http.createServer((req, res) => {
    // Set status code and headers
    res.statusCode = 200; // or res.writeHead(200, headers)
    res.setHeader('Content-Type', 'application/json');
    res.setHeader('X-Powered-By', 'Node.js');
    
    // Write response body
    res.write('Hello ');
    res.write('World');
    res.end('!'); // End the response
    
    // Alternative: write everything at once
    // res.end('Hello World!');
});

Common response methods and properties:

const http = require('http');

const server = http.createServer((req, res) => {
    // Method 1: Set status and headers separately
    res.statusCode = 201;
    res.setHeader('Content-Type', 'application/json');
    res.setHeader('Cache-Control', 'no-cache');
    
    // Method 2: Set everything at once with writeHead
    res.writeHead(200, {
        'Content-Type': 'text/html',
        'Set-Cookie': ['session=abc123', 'theme=dark']
    });
    
    // Check if headers are sent
    console.log('Headers sent:', res.headersSent);
    
    // Write response in chunks
    res.write('');
    res.write('
Hello
');
    res.write('');
    
    // End the response
    res.end();
    
    // Events
    res.on('finish', () => {
        console.log('Response sent successfully');
    });
    
    res.on('close', () => {
        console.log('Connection closed before response could be sent');
    });
});

Advanced request handling with streams:

const http = require('http');
const fs = require('fs');

const server = http.createServer((req, res) => {
    // Log request details
    console.log(`${req.method} ${req.url} from ${req.socket.remoteAddress}`);
    
    // Handle different content types in request
    if (req.headers['content-type'] === 'application/json') {
        let data = '';
        req.on('data', chunk => data += chunk);
        req.on('end', () => {
            try {
                const jsonData = JSON.parse(data);
                console.log('Received JSON:', jsonData);
                
                res.writeHead(200, { 'Content-Type': 'application/json' });
                res.end(JSON.stringify({ received: true, data: jsonData }));
            } catch (error) {
                res.writeHead(400, { 'Content-Type': 'application/json' });
                res.end(JSON.stringify({ error: 'Invalid JSON' }));
            }
        });
    } else {
        // Stream the request to a file (for large uploads)
        const writeStream = fs.createWriteStream(`upload-${Date.now()}.tmp`);
        req.pipe(writeStream);
        
        writeStream.on('finish', () => {
            res.writeHead(200, { 'Content-Type': 'text/plain' });
            res.end('Upload received');
        });
    }
});

Advanced response streaming:

const http = require('http');
const fs = require('fs');

const server = http.createServer((req, res) => {
    if (req.url === '/stream') {
        // Stream large file efficiently
        const fileStream = fs.createReadStream('large-file.txt');
        
        res.writeHead(200, {
            'Content-Type': 'text/plain',
            'Transfer-Encoding': 'chunked'
        });
        
        fileStream.pipe(res);
        
        fileStream.on('error', (error) => {
            console.log('Stream error:', error);
            if (!res.headersSent) {
                res.writeHead(500, { 'Content-Type': 'text/plain' });
            }
            res.end('Error streaming file');
        });
        
    } else if (req.url === '/events') {
        // Server-sent events
        res.writeHead(200, {
            'Content-Type': 'text/event-stream',
            'Cache-Control': 'no-cache',
            'Connection': 'keep-alive'
        });
        
        let count = 0;
        const interval = setInterval(() => {
            count++;
            res.write(`data: Message ${count} at ${new Date().toISOString()}\n\n`);
            
            if (count >= 10) {
                clearInterval(interval);
                res.write('data: END\n\n');
                res.end();
            }
        }, 1000);
        
        req.on('close', () => {
            clearInterval(interval);
        });
    }
});

Key request object properties:

• req.method - HTTP method (GET, POST, etc.)
• req.url - Request URL string
• req.headers - Request headers object
• req.httpVersion - HTTP version
• req.socket - Underlying socket
• req.body - Not available by default (need to parse)

Key response object methods:

• res.writeHead() - Set status and headers
• res.setHeader() - Set a single header
• res.write() - Write response body
• res.end() - End the response
• res.statusCode - Set status code
• res.getHeader() - Get a header value

URL parameters can be parsed using Node.js's built-in url module or by manually parsing the URL string. There are two types: query parameters and path parameters.

Using the url module (recommended):

const http = require('http');
const url = require('url');

const server = http.createServer((req, res) => {
    // Parse URL with query string
    const parsedUrl = url.parse(req.url, true);
    
    console.log('Pathname:', parsedUrl.pathname);
    console.log('Query parameters:', parsedUrl.query);
    
    // Access specific parameters
    const name = parsedUrl.query.name || 'World';
    const age = parsedUrl.query.age;
    
    res.writeHead(200, { 'Content-Type': 'application/json' });
    res.end(JSON.stringify({
        message: `Hello ${name}`,
        age: age ? parseInt(age) : null,
        query: parsedUrl.query
    }));
});

server.listen(3000);

Manual URL parsing for path parameters:

const http = require('http');

const server = http.createServer((req, res) => {
    // Parse path parameters like /users/123/posts/456
    const pathParams = req.url.match(/^\/users\/(\d+)\/posts\/(\d+)$/);
    
    if (pathParams) {
        const userId = pathParams[1];
        const postId = pathParams[2];
        
        res.writeHead(200, { 'Content-Type': 'application/json' });
        res.end(JSON.stringify({
            userId: parseInt(userId),
            postId: parseInt(postId),
            message: `User ${userId}'s post ${postId}`
        }));
    } else {
        res.writeHead(404, { 'Content-Type': 'application/json' });
        res.end(JSON.stringify({ error: 'Not found' }));
    }
});

Advanced parameter parsing with validation:

const http = require('http');
const url = require('url');

class ParameterParser {
    static parseSearchParams(req) {
        const parsedUrl = url.parse(req.url, true);
        const query = parsedUrl.query;
        
        // Validate and transform parameters
        const params = {
            page: this.validateNumber(query.page, 1, 1, 1000),
            limit: this.validateNumber(query.limit, 10, 1, 100),
            sort: this.validateString(query.sort, 'createdAt', ['createdAt', 'updatedAt', 'title']),
            order: this.validateString(query.order, 'desc', ['asc', 'desc']),
            search: this.sanitizeString(query.search, '')
        };
        
        return params;
    }
    
    static validateNumber(value, defaultValue, min, max) {
        const num = parseInt(value);
        if (isNaN(num)) return defaultValue;
        return Math.max(min, Math.min(max, num));
    }
    
    static validateString(value, defaultValue, allowedValues) {
        if (!value) return defaultValue;
        return allowedValues.includes(value) ? value : defaultValue;
    }
    
    static sanitizeString(value, defaultValue) {
        if (!value) return defaultValue;
        // Basic sanitization
        return value.toString().trim().substring(0, 100);
    }
}

const server = http.createServer((req, res) => {
    if (req.url.startsWith('/api/products')) {
        const params = ParameterParser.parseSearchParams(req);
        
        res.writeHead(200, { 'Content-Type': 'application/json' });
        res.end(JSON.stringify({
            params,
            message: 'Search parameters processed'
        }));
    }
});

Query strings are the part of the URL after the ? character. They contain key-value pairs that can be easily parsed in Node.js.

Using the url module:

const http = require('http');
const url = require('url');

const server = http.createServer((req, res) => {
    const parsedUrl = url.parse(req.url, true); // true to parse query string
    
    // Access query parameters
    const query = parsedUrl.query;
    
    console.log('Full query object:', query);
    console.log('Name:', query.name);
    console.log('Age:', query.age);
    console.log('Filters:', query.filters);
    
    res.writeHead(200, { 'Content-Type': 'application/json' });
    res.end(JSON.stringify({
        query: query,
        path: parsedUrl.pathname
    }));
});

Manual query string parsing:

const http = require('http');
const querystring = require('querystring');

const server = http.createServer((req, res) => {
    const urlParts = req.url.split('?');
    const path = urlParts[0];
    const queryString = urlParts[1] || '';
    
    // Parse query string manually
    const query = querystring.parse(queryString);
    
    // Handle array parameters (filters=js&filters=node)
    if (query.filters) {
        if (!Array.isArray(query.filters)) {
            query.filters = [query.filters];
        }
    }
    
    res.end(JSON.stringify({ query, path }));
});

The url module provides utilities for URL resolution and parsing. It's essential for working with URLs in Node.js applications.

URL parsing and manipulation:

const url = require('url');

// Parse a URL string
const urlString = 'https://example.com:8080/path/name?query=string#hash';
const parsedUrl = url.parse(urlString, true);

console.log('Protocol:', parsedUrl.protocol); // https:
console.log('Host:', parsedUrl.host);         // example.com:8080
console.log('Hostname:', parsedUrl.hostname); // example.com
console.log('Port:', parsedUrl.port);         // 8080
console.log('Pathname:', parsedUrl.pathname); // /path/name
console.log('Query:', parsedUrl.query);       // { query: 'string' }
console.log('Hash:', parsedUrl.hash);         // #hash

URL formatting:

const url = require('url');

// Create URL from object
const urlObject = {
    protocol: 'https',
    host: 'api.example.com',
    pathname: '/v1/users',
    query: {
        page: '1',
        limit: '10'
    }
};

const formattedUrl = url.format(urlObject);
console.log(formattedUrl); // https://api.example.com/v1/users?page=1&limit=10

Sending JSON responses involves setting the correct Content-Type header and stringifying JavaScript objects.

Basic JSON response:

const http = require('http');

const server = http.createServer((req, res) => {
    const data = {
        message: 'Hello World',
        timestamp: new Date().toISOString(),
        status: 'success'
    };
    
    res.writeHead(200, {
        'Content-Type': 'application/json',
        'Cache-Control': 'no-cache'
    });
    
    res.end(JSON.stringify(data));
});

JSON response with error handling:

const http = require('http');

const server = http.createServer((req, res) => {
    try {
        // Your business logic
        if (req.url === '/api/users') {
            const users = [
                { id: 1, name: 'John' },
                { id: 2, name: 'Jane' }
            ];
            
            sendJSON(res, 200, { users });
        } else {
            sendJSON(res, 404, { error: 'Not found' });
        }
    } catch (error) {
        sendJSON(res, 500, { error: 'Internal server error' });
    }
});

function sendJSON(res, statusCode, data) {
    res.writeHead(statusCode, {
        'Content-Type': 'application/json',
        'Access-Control-Allow-Origin': '*'
    });
    res.end(JSON.stringify(data));
}

HTTP headers provide metadata about the request or response. They can be set using res.setHeader() or res.writeHead().

Setting individual headers:

const http = require('http');

const server = http.createServer((req, res) => {
    // Set headers individually
    res.setHeader('Content-Type', 'application/json');
    res.setHeader('X-Powered-By', 'Node.js');
    res.setHeader('Cache-Control', 'no-cache, no-store, must-revalidate');
    res.setHeader('Access-Control-Allow-Origin', '*');
    
    res.end(JSON.stringify({ message: 'Hello' }));
});

Setting headers with writeHead:

const http = require('http');

const server = http.createServer((req, res) => {
    // Set all headers at once
    res.writeHead(200, {
        'Content-Type': 'application/json',
        'X-Powered-By': 'Node.js',
        'Cache-Control': 'public, max-age=3600',
        'Set-Cookie': ['session=abc123; HttpOnly', 'theme=dark']
    });
    
    res.end(JSON.stringify({ message: 'Hello' }));
});

HTTP methods (verbs) indicate the desired action to be performed on a resource. The most common methods are GET, POST, PUT, DELETE, PATCH.

Common HTTP methods:

const http = require('http');

const server = http.createServer((req, res) => {
    switch (req.method) {
        case 'GET':
            // Retrieve resource
            res.end('GET: Reading resource');
            break;
        case 'POST':
            // Create new resource
            res.end('POST: Creating resource');
            break;
        case 'PUT':
            // Update entire resource
            res.end('PUT: Updating resource');
            break;
        case 'PATCH':
            // Partial update
            res.end('PATCH: Partial update');
            break;
        case 'DELETE':
            // Delete resource
            res.end('DELETE: Deleting resource');
            break;
        case 'OPTIONS':
            // CORS preflight
            res.writeHead(204, {
                'Access-Control-Allow-Origin': '*',
                'Access-Control-Allow-Methods': 'GET, POST, PUT, DELETE'
            });
            res.end();
            break;
        default:
            res.writeHead(405, { 'Content-Type': 'application/json' });
            res.end(JSON.stringify({ error: 'Method not allowed' }));
    }
});

CORS (Cross-Origin Resource Sharing) is a security mechanism that allows restricted resources on a web page to be requested from another domain outside the domain from which the first resource was served.

Simple CORS implementation:

const http = require('http');

const server = http.createServer((req, res) => {
    // Set CORS headers
    res.setHeader('Access-Control-Allow-Origin', '*');
    res.setHeader('Access-Control-Allow-Methods', 'GET, POST, PUT, DELETE, OPTIONS');
    res.setHeader('Access-Control-Allow-Headers', 'Content-Type, Authorization');
    
    // Handle preflight requests
    if (req.method === 'OPTIONS') {
        res.writeHead(204);
        res.end();
        return;
    }
    
    // Your normal request handling
    res.end(JSON.stringify({ message: 'CORS enabled' }));
});

CORS can be enabled manually by setting appropriate headers and handling preflight OPTIONS requests.

Complete CORS implementation:

const http = require('http');

function enableCORS(res) {
    res.setHeader('Access-Control-Allow-Origin', process.env.ALLOWED_ORIGIN || '*');
    res.setHeader('Access-Control-Allow-Methods', 'GET, POST, PUT, DELETE, OPTIONS, PATCH');
    res.setHeader('Access-Control-Allow-Headers', 'Content-Type, Authorization, X-Requested-With');
    res.setHeader('Access-Control-Allow-Credentials', 'true');
    res.setHeader('Access-Control-Max-Age', '86400'); // 24 hours
}

const server = http.createServer((req, res) => {
    // Enable CORS for all responses
    enableCORS(res);
    
    // Handle preflight requests
    if (req.method === 'OPTIONS') {
        res.writeHead(204);
        res.end();
        return;
    }
    
    // Your application logic
    if (req.method === 'GET' && req.url === '/api/data') {
        res.writeHead(200, { 'Content-Type': 'application/json' });
        res.end(JSON.stringify({ data: 'CORS enabled response' }));
    } else {
        res.writeHead(404, { 'Content-Type': 'application/json' });
        res.end(JSON.stringify({ error: 'Not found' }));
    }
});

HTTP redirects are handled using 3xx status codes and the Location header to tell the client where to go next.

Different types of redirects:

const http = require('http');

const server = http.createServer((req, res) => {
    if (req.url === '/old-page') {
        // 301 Moved Permanently
        res.writeHead(301, {
            'Location': '/new-page'
        });
        res.end();
    } else if (req.url === '/temp-redirect') {
        // 302 Found (Temporary)
        res.writeHead(302, {
            'Location': '/temporary-page'
        });
        res.end();
    } else if (req.url === '/see-other') {
        // 303 See Other (for POST redirect)
        res.writeHead(303, {
            'Location': '/result-page'
        });
        res.end();
    } else {
        res.end('Current page');
    }
});

HTTP status codes are 3-digit numbers returned by servers to indicate the result of a request. They are grouped into five classes.

Common status codes:

const http = require('http');

const server = http.createServer((req, res) => {
    // 2xx Success
    if (req.url === '/success') {
        res.writeHead(200, { 'Content-Type': 'application/json' });
        res.end(JSON.stringify({ message: 'OK' }));
    }
    
    // 3xx Redirection
    else if (req.url === '/moved') {
        res.writeHead(301, { 'Location': '/new-location' });
        res.end();
    }
    
    // 4xx Client Errors
    else if (req.url === '/not-found') {
        res.writeHead(404, { 'Content-Type': 'application/json' });
        res.end(JSON.stringify({ error: 'Not found' }));
    }
    else if (req.url === '/unauthorized') {
        res.writeHead(401, { 'Content-Type': 'application/json' });
        res.end(JSON.stringify({ error: 'Unauthorized' }));
    }
    
    // 5xx Server Errors
    else if (req.url === '/error') {
        res.writeHead(500, { 'Content-Type': 'application/json' });
        res.end(JSON.stringify({ error: 'Internal server error' }));
    }
});

HTTPS servers provide encrypted communication using SSL/TLS. They are essential for secure web applications.

Creating an HTTPS server:

const https = require('https');
const fs = require('fs');

// Read SSL certificate and key
const options = {
    key: fs.readFileSync('server-key.pem'),
    cert: fs.readFileSync('server-cert.pem')
};

const server = https.createServer(options, (req, res) => {
    res.writeHead(200, { 'Content-Type': 'text/plain' });
    res.end('Secure HTTPS server!');
});

server.listen(443, () => {
    console.log('HTTPS server running on port 443');
});

SSL certificates are used to enable HTTPS and secure communication. You can use self-signed certificates for development or obtain certificates from a CA for production.

Using SSL certificates:

const https = require('https');
const fs = require('fs');

// For development (self-signed)
const options = {
    key: fs.readFileSync('./ssl/private-key.pem'),
    cert: fs.readFileSync('./ssl/certificate.pem'),
    
    // Optional: Add CA bundle for production
    // ca: fs.readFileSync('./ssl/ca-bundle.crt'),
    
    // Security settings
    secureProtocol: 'TLSv1_2_method',
    ciphers: 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384'
};

const server = https.createServer(options, (req, res) => {
    res.end('Secure connection established');
});

server.listen(443);

The net module provides asynchronous network API for creating stream-based TCP servers and clients.

Basic TCP server:

const net = require('net');

const server = net.createServer((socket) => {
    console.log('Client connected from:', socket.remoteAddress);
    
    socket.write('Hello from TCP server!\r\n');
    
    socket.on('data', (data) => {
        console.log('Received:', data.toString());
        socket.write('Echo: ' + data);
    });
    
    socket.on('end', () => {
        console.log('Client disconnected');
    });
});

server.listen(3000, () => {
    console.log('TCP server listening on port 3000');
});

TCP servers handle raw TCP connections, useful for custom protocols, real-time applications, or when you need low-level control.

Advanced TCP server:

const net = require('net');

class TCPServer {
    constructor(port) {
        this.port = port;
        this.clients = new Map();
        this.server = net.createServer(this.handleConnection.bind(this));
    }
    
    handleConnection(socket) {
        const clientId = `${socket.remoteAddress}:${socket.remotePort}`;
        console.log(`Client ${clientId} connected`);
        
        this.clients.set(clientId, socket);
        
        socket.on('data', (data) => {
            this.handleMessage(clientId, data.toString());
        });
        
        socket.on('end', () => {
            console.log(`Client ${clientId} disconnected`);
            this.clients.delete(clientId);
        });
        
        socket.on('error', (error) => {
            console.log(`Client ${clientId} error:`, error.message);
            this.clients.delete(clientId);
        });
        
        // Send welcome message
        socket.write(`Welcome! You are client ${clientId}\r\n`);
    }
    
    handleMessage(clientId, message) {
        console.log(`Message from ${clientId}: ${message.trim()}`);
        
        // Broadcast to all clients
        this.broadcast(`${clientId}: ${message}`, clientId);
    }
    
    broadcast(message, excludeClientId = null) {
        this.clients.forEach((socket, clientId) => {
            if (clientId !== excludeClientId) {
                socket.write(message);
            }
        });
    }
    
    start() {
        this.server.listen(this.port, () => {
            console.log(`TCP server running on port ${this.port}`);
        });
    }
}

// Usage
const tcpServer = new TCPServer(3000);
tcpServer.start();

TCP clients connect to TCP servers to send and receive data. The net module provides the Socket class for this purpose.

TCP client connection:

const net = require('net');

// Create TCP client
const client = new net.Socket();

client.connect(3000, 'localhost', () => {
    console.log('Connected to TCP server');
    client.write('Hello from client!');
});

client.on('data', (data) => {
    console.log('Received from server:', data.toString());
});

client.on('close', () => {
    console.log('Connection closed');
});

client.on('error', (error) => {
    console.log('Connection error:', error.message);
});

A socket is an endpoint for sending or receiving data across a computer network. In Node.js, sockets are used for both TCP and UDP communication.

Socket characteristics:

const net = require('net');

const server = net.createServer((socket) => {
    // Socket properties
    console.log('Remote address:', socket.remoteAddress);
    console.log('Remote port:', socket.remotePort);
    console.log('Local address:', socket.localAddress);
    console.log('Local port:', socket.localPort);
    console.log('Bytes read:', socket.bytesRead);
    console.log('Bytes written:', socket.bytesWritten);
    
    // Socket events
    socket.on('data', (data) => {
        console.log('Data received:', data.toString());
    });
    
    socket.on('end', () => {
        console.log('Client ended connection');
    });
    
    socket.on('close', () => {
        console.log('Socket closed');
    });
    
    socket.on('error', (error) => {
        console.log('Socket error:', error.message);
    });
    
    // Socket methods
    socket.write('Hello client!');
    socket.setEncoding('utf8');
    socket.setTimeout(30000); // 30 seconds
});

WebSockets provide full-duplex communication channels over a single TCP connection, ideal for real-time applications.

Using the ws library:

const WebSocket = require('ws');
const wss = new WebSocket.Server({ port: 8080 });

wss.on('connection', (ws) => {
    console.log('New WebSocket connection');
    
    ws.on('message', (message) => {
        console.log('Received:', message.toString());
        
        // Echo back to client
        ws.send(`Echo: ${message}`);
        
        // Broadcast to all clients
        wss.clients.forEach((client) => {
            if (client !== ws && client.readyState === WebSocket.OPEN) {
                client.send(`Broadcast: ${message}`);
            }
        });
    });
    
    ws.on('close', () => {
        console.log('WebSocket connection closed');
    });
    
    // Send welcome message
    ws.send('Welcome to WebSocket server!');
});

Socket.io is a library that enables real-time, bidirectional and event-based communication between web clients and servers.

Socket.io server setup:

const http = require('http');
const socketIo = require('socket.io');

const server = http.createServer();
const io = socketIo(server, {
    cors: {
        origin: "*",
        methods: ["GET", "POST"]
    }
});

io.on('connection', (socket) => {
    console.log('User connected:', socket.id);
    
    // Join room
    socket.on('join-room', (room) => {
        socket.join(room);
        socket.to(room).emit('user-joined', socket.id);
    });
    
    // Handle chat messages
    socket.on('send-message', (data) => {
        io.to(data.room).emit('new-message', {
            user: socket.id,
            message: data.message,
            timestamp: new Date()
        });
    });
    
    // Handle disconnection
    socket.on('disconnect', () => {
        console.log('User disconnected:', socket.id);
    });
});

server.listen(3000);

A WebSocket server handles WebSocket connections and enables real-time communication between server and clients.

Complete WebSocket server:

const WebSocket = require('ws');

class WebSocketServer {
    constructor(port) {
        this.port = port;
        this.wss = new WebSocket.Server({ port });
        this.clients = new Map();
        
        this.setupEventHandlers();
    }
    
    setupEventHandlers() {
        this.wss.on('connection', (ws, req) => {
            const clientId = this.generateClientId();
            console.log(`Client ${clientId} connected from ${req.socket.remoteAddress}`);
            
            this.clients.set(clientId, ws);
            this.setupClientHandlers(clientId, ws);
            
            // Send welcome message
            this.sendToClient(clientId, {
                type: 'welcome',
                clientId: clientId,
                message: 'Connected to WebSocket server'
            });
            
            // Notify other clients
            this.broadcast({
                type: 'user-joined',
                clientId: clientId,
                timestamp: new Date()
            }, clientId);
        });
    }
    
    setupClientHandlers(clientId, ws) {
        ws.on('message', (data) => {
            try {
                const message = JSON.parse(data);
                this.handleMessage(clientId, message);
            } catch (error) {
                this.sendToClient(clientId, {
                    type: 'error',
                    message: 'Invalid JSON format'
                });
            }
        });
        
        ws.on('close', () => {
            console.log(`Client ${clientId} disconnected`);
            this.clients.delete(clientId);
            this.broadcast({
                type: 'user-left',
                clientId: clientId,
                timestamp: new Date()
            });
        });
        
        ws.on('error', (error) => {
            console.log(`Client ${clientId} error:`, error.message);
            this.clients.delete(clientId);
        });
    }
    
    handleMessage(clientId, message) {
        switch (message.type) {
            case 'chat':
                this.broadcast({
                    type: 'chat',
                    from: clientId,
                    text: message.text,
                    timestamp: new Date()
                });
                break;
                
            case 'ping':
                this.sendToClient(clientId, { type: 'pong' });
                break;
                
            default:
                this.sendToClient(clientId, {
                    type: 'error',
                    message: 'Unknown message type'
                });
        }
    }
    
    sendToClient(clientId, data) {
        const ws = this.clients.get(clientId);
        if (ws && ws.readyState === WebSocket.OPEN) {
            ws.send(JSON.stringify(data));
        }
    }
    
    broadcast(data, excludeClientId = null) {
        this.clients.forEach((ws, clientId) => {
            if (clientId !== excludeClientId && ws.readyState === WebSocket.OPEN) {
                ws.send(JSON.stringify(data));
            }
        });
    }
    
    generateClientId() {
        return Math.random().toString(36).substring(2, 15);
    }
    
    start() {
        console.log(`WebSocket server running on port ${this.port}`);
    }
}

// Usage
const wsServer = new WebSocketServer(8080);
wsServer.start();

HTTP streaming allows sending data in chunks as it becomes available, rather than waiting for the complete response.

HTTP response streaming:

const http = require('http');

const server = http.createServer((req, res) => {
    res.writeHead(200, {
        'Content-Type': 'text/plain',
        'Transfer-Encoding': 'chunked'
    });
    
    // Send data in chunks
    let count = 0;
    const interval = setInterval(() => {
        count++;
        res.write(`Chunk ${count}\n`);
        
        if (count >= 10) {
            clearInterval(interval);
            res.end('Stream complete');
        }
    }, 1000);
});

Streaming responses are handled using Node.js streams to efficiently send large amounts of data or real-time updates.

File streaming response:

const http = require('http');
const fs = require('fs');

const server = http.createServer((req, res) => {
    const fileStream = fs.createReadStream('large-file.txt');
    
    res.writeHead(200, {
        'Content-Type': 'text/plain',
        'Content-Disposition': 'attachment; filename="large-file.txt"'
    });
    
    fileStream.pipe(res);
    
    fileStream.on('error', (error) => {
        res.writeHead(500);
        res.end('Error streaming file');
    });
});

HTTP pipelining is a technique where multiple HTTP requests are sent on a single connection without waiting for the corresponding responses, improving performance by reducing latency.

Understanding HTTP pipelining:

const http = require('http');

// Server that handles pipelined requests
const server = http.createServer((req, res) => {
    console.log(`Processing ${req.method} ${req.url}`);
    
    // Simulate processing time
    setTimeout(() => {
        res.writeHead(200, { 'Content-Type': 'text/plain' });
        res.end(`Response for ${req.url}\n`);
    }, 1000);
});

server.listen(3000);

// Client making pipelined requests
function makePipelinedRequests() {
    const requests = [
        { method: 'GET', path: '/api/users' },
        { method: 'GET', path: '/api/products' },
        { method: 'POST', path: '/api/orders' }
    ];
    
    const options = {
        hostname: 'localhost',
        port: 3000,
        method: 'GET'
    };
    
    // Note: Modern HTTP/1.1 clients handle pipelining automatically
    // This is a simplified demonstration
    requests.forEach((reqConfig, index) => {
        const req = http.request({ ...options, path: reqConfig.path }, (res) => {
            let data = '';
            res.on('data', chunk => data += chunk);
            res.on('end', () => {
                console.log(`Response ${index + 1}:`, data.trim());
            });
        });
        req.end();
    });
}

// Note: HTTP/2 multiplexing has largely replaced pipelining
// Pipelining can be problematic with head-of-line blocking

Key points about HTTP pipelining:

• Requests are sent without waiting for previous responses
• Responses must be returned in the same order as requests
• Can cause "head-of-line blocking" if one request is slow
• Largely superseded by HTTP/2 multiplexing
• Not all servers and clients support it well

Timeouts are crucial for preventing hanging requests and managing resources efficiently in HTTP servers and clients.

Server-side timeout management:

const http = require('http');

const server = http.createServer((req, res) => {
    // Set timeout for this request (10 seconds)
    req.setTimeout(10000, () => {
        console.log('Request timeout for:', req.url);
        if (!res.headersSent) {
            res.writeHead(408, { 'Content-Type': 'application/json' });
            res.end(JSON.stringify({ error: 'Request timeout' }));
        }
    });
    
    // Simulate long processing
    if (req.url === '/slow') {
        setTimeout(() => {
            res.writeHead(200, { 'Content-Type': 'application/json' });
            res.end(JSON.stringify({ message: 'Slow response completed' }));
        }, 15000); // This will timeout
    } else {
        res.end('OK');
    }
});

// Set server-level timeout
server.setTimeout(30000, (socket) => {
    console.log('Socket timeout, closing connection');
    socket.destroy();
});

server.listen(3000);

Client-side timeout management:

const http = require('http');

function makeRequestWithTimeout() {
    const options = {
        hostname: 'example.com',
        port: 80,
        path: '/api/data',
        method: 'GET',
        timeout: 5000 // 5 second timeout
    };
    
    const req = http.request(options, (res) => {
        let data = '';
        res.on('data', chunk => data += chunk);
        res.on('end', () => {
            console.log('Response:', data);
        });
    });
    
    // Handle timeout
    req.setTimeout(5000, () => {
        console.log('Request timeout');
        req.destroy();
    });
    
    // Handle errors
    req.on('error', (error) => {
        console.log('Request error:', error.message);
    });
    
    req.end();
}

Advanced timeout strategy:

class TimeoutManager {
    constructor(defaultTimeout = 10000) {
        this.defaultTimeout = defaultTimeout;
        this.timeouts = new Map();
    }
    
    setRequestTimeout(req, res, timeout = this.defaultTimeout) {
        const timeoutId = setTimeout(() => {
            this.handleTimeout(req, res);
        }, timeout);
        
        this.timeouts.set(req, timeoutId);
        
        // Clear timeout when request completes
        res.on('finish', () => {
            this.clearTimeout(req);
        });
    }
    
    handleTimeout(req, res) {
        if (!res.headersSent) {
            res.writeHead(408, { 'Content-Type': 'application/json' });
            res.end(JSON.stringify({ 
                error: 'Request timeout',
                message: 'The request took too long to process'
            }));
        }
        req.destroy();
        this.clearTimeout(req);
    }
    
    clearTimeout(req) {
        const timeoutId = this.timeouts.get(req);
        if (timeoutId) {
            clearTimeout(timeoutId);
            this.timeouts.delete(req);
        }
    }
}

// Usage
const timeoutManager = new TimeoutManager(8000);
const server = http.createServer((req, res) => {
    timeoutManager.setRequestTimeout(req, res);
    
    // Your request handling logic
    setTimeout(() => {
        res.end('Response after delay');
    }, 5000); // Within timeout
});

http.Agent is responsible for managing connection persistence and reuse for HTTP clients. It maintains a pool of sockets for keep-alive connections.

Using http.Agent:

const http = require('http');

// Custom agent with specific settings
const agent = new http.Agent({
    keepAlive: true,
    keepAliveMsecs: 10000,
    maxSockets: 10,
    maxFreeSockets: 5,
    timeout: 60000
});

// Make requests using the custom agent
function makeRequests() {
    const options = {
        hostname: 'jsonplaceholder.typicode.com',
        port: 80,
        path: '/posts/1',
        method: 'GET',
        agent: agent // Use our custom agent
    };
    
    const req = http.request(options, (res) => {
        console.log('Status:', res.statusCode);
        console.log('Connection reused:', res.reusedSocket);
    });
    
    req.end();
}

// Monitor agent statistics
console.log('Agent stats:', {
    totalSockets: agent.totalSocketsCount,
    freeSockets: Object.keys(agent.freeSockets).length,
    requests: Object.keys(agent.requests).length
});

Global agent configuration:

const http = require('http');

// Configure global agent
http.globalAgent.keepAlive = true;
http.globalAgent.maxSockets = 50;
http.globalAgent.maxFreeSockets = 10;

// All requests will now use keep-alive by default
function makeMultipleRequests() {
    for (let i = 1; i <= 5; i++) {
        const req = http.request({
            hostname: 'api.example.com',
            path: `/data/${i}`
        }, (res) => {
            console.log(`Request ${i} - Reused: ${res.reusedSocket}`);
        });
        req.end();
    }
}

Custom agent for specific domains:

const http = require('http');

// Create specialized agents for different services
const apiAgent = new http.Agent({
    keepAlive: true,
    maxSockets: 20,
    timeout: 30000
});

const imageAgent = new http.Agent({
    keepAlive: false, // Don't keep alive for image requests
    maxSockets: 50
});

class HttpClient {
    constructor() {
        this.agents = {
            'api.example.com': apiAgent,
            'images.example.com': imageAgent,
            'default': new http.Agent({ keepAlive: true })
        };
    }
    
    request(url, options = {}) {
        const urlObj = new URL(url);
        const agent = this.agents[urlObj.hostname] || this.agents.default;
        
        return http.request({
            hostname: urlObj.hostname,
            path: urlObj.pathname + urlObj.search,
            agent: agent,
            ...options
        });
    }
}

// Usage
const client = new HttpClient();
const req = client.request('https://api.example.com/data');
req.end();

Concurrent request handling is essential for building scalable servers. Node.js's event-driven architecture makes it excellent for handling many simultaneous connections.

Basic concurrent request handling:

const http = require('http');

const server = http.createServer(async (req, res) => {
    const requestId = Math.random().toString(36).substring(7);
    console.log(`[${requestId}] Started processing ${req.url}`);
    
    // Simulate async work (database query, API call, etc.)
    await simulateAsyncWork();
    
    res.writeHead(200, { 'Content-Type': 'application/json' });
    res.end(JSON.stringify({ 
        requestId,
        message: 'Request completed',
        url: req.url 
    }));
    
    console.log(`[${requestId}] Completed processing`);
});

async function simulateAsyncWork() {
    // Simulate variable processing time (100-2000ms)
    const delay = Math.random() * 1900 + 100;
    return new Promise(resolve => setTimeout(resolve, delay));
}

server.listen(3000, () => {
    console.log('Server ready for concurrent requests on port 3000');
});

Concurrent requests with connection limiting:

const http = require('http');

class ConcurrentRequestHandler {
    constructor(maxConcurrent = 100) {
        this.maxConcurrent = maxConcurrent;
        this.activeRequests = 0;
        this.queue = [];
    }
    
    handleRequest(req, res) {
        return new Promise((resolve) => {
            const requestHandler = () => {
                this.activeRequests++;
                
                // Simulate request processing
                setTimeout(() => {
                    res.end('Request processed');
                    this.activeRequests--;
                    this.processQueue();
                    resolve();
                }, 1000);
            };
            
            if (this.activeRequests < this.maxConcurrent) {
                requestHandler();
            } else {
                this.queue.push(requestHandler);
                console.log(`Request queued. Queue length: ${this.queue.length}`);
            }
        });
    }
    
    processQueue() {
        while (this.queue.length > 0 && this.activeRequests < this.maxConcurrent) {
            const nextHandler = this.queue.shift();
            nextHandler();
        }
    }
}

const requestHandler = new ConcurrentRequestHandler(5); // Max 5 concurrent
const server = http.createServer((req, res) => {
    requestHandler.handleRequest(req, res);
});

Making concurrent client requests:

const http = require('http');

async function makeConcurrentRequests(urls, concurrencyLimit = 5) {
    const results = [];
    let currentIndex = 0;
    
    async function processNext() {
        if (currentIndex >= urls.length) return;
        
        const url = urls[currentIndex++];
        try {
            const result = await fetchUrl(url);
            results.push({ url, status: 'success', data: result });
        } catch (error) {
            results.push({ url, status: 'error', error: error.message });
        }
        
        // Process next URL
        await processNext();
    }
    
    // Start concurrent processing
    const workers = Array(concurrencyLimit).fill().map(() => processNext());
    await Promise.all(workers);
    
    return results;
}

function fetchUrl(url) {
    return new Promise((resolve, reject) => {
        const req = http.request(url, (res) => {
            let data = '';
            res.on('data', chunk => data += chunk);
            res.on('end', () => resolve(data));
        });
        
        req.on('error', reject);
        req.setTimeout(5000, () => {
            req.destroy();
            reject(new Error('Timeout'));
        });
        
        req.end();
    });
}

// Usage
const urls = [
    'http://jsonplaceholder.typicode.com/posts/1',
    'http://jsonplaceholder.typicode.com/posts/2',
    'http://jsonplaceholder.typicode.com/posts/3'
];

makeConcurrentRequests(urls, 2)
    .then(results => console.log('Concurrent results:', results));

A proxy server acts as an intermediary between clients and other servers, useful for caching, filtering, load balancing, or bypassing restrictions.

Basic HTTP proxy server:

const http = require('http');
const https = require('https');

const proxy = http.createServer((clientReq, clientRes) => {
    console.log(`Proxying: ${clientReq.method} ${clientReq.url}`);
    
    // Parse the target URL from request
    const targetUrl = new URL(clientReq.url, `http://${clientReq.headers.host}`);
    
    const options = {
        hostname: targetUrl.hostname,
        port: targetUrl.port || (targetUrl.protocol === 'https:' ? 443 : 80),
        path: targetUrl.pathname + targetUrl.search,
        method: clientReq.method,
        headers: { ...clientReq.headers }
    };
    
    // Remove hop-by-hop headers
    delete options.headers['proxy-connection'];
    delete options.headers['connection'];
    delete options.headers['keep-alive'];
    
    // Choose protocol
    const protocol = targetUrl.protocol === 'https:' ? https : http;
    
    const proxyReq = protocol.request(options, (proxyRes) => {
        // Forward status and headers
        clientRes.writeHead(proxyRes.statusCode, proxyRes.headers);
        
        // Pipe the response
        proxyRes.pipe(clientRes);
    });
    
    // Handle errors
    proxyReq.on('error', (error) => {
        console.log('Proxy error:', error.message);
        clientRes.writeHead(500, { 'Content-Type': 'text/plain' });
        clientRes.end('Proxy error: ' + error.message);
    });
    
    // Pipe the request body
    clientReq.pipe(proxyReq);
});

proxy.listen(8080, () => {
    console.log('Proxy server running on port 8080');
});

Advanced proxy with caching:

const http = require('http');
const https = require('https');
const fs = require('fs');
const path = require('path');

class CachingProxy {
    constructor(cacheDir = './cache') {
        this.cacheDir = cacheDir;
        this.ensureCacheDir();
    }
    
    ensureCacheDir() {
        if (!fs.existsSync(this.cacheDir)) {
            fs.mkdirSync(this.cacheDir, { recursive: true });
        }
    }
    
    getCacheKey(url) {
        return Buffer.from(url).toString('base64').replace(/[^a-zA-Z0-9]/g, '_');
    }
    
    getCachedResponse(url) {
        const cacheFile = path.join(this.cacheDir, this.getCacheKey(url));
        try {
            if (fs.existsSync(cacheFile)) {
                const cached = JSON.parse(fs.readFileSync(cacheFile, 'utf8'));
                // Check if cache is still valid (5 minutes)
                if (Date.now() - cached.timestamp < 5 * 60 * 1000) {
                    return cached;
                }
            }
        } catch (error) {
            // Cache corrupted, ignore
        }
        return null;
    }
    
    cacheResponse(url, response) {
        const cacheFile = path.join(this.cacheDir, this.getCacheKey(url));
        const cached = {
            url,
            timestamp: Date.now(),
            statusCode: response.statusCode,
            headers: response.headers,
            body: response.body
        };
        
        fs.writeFileSync(cacheFile, JSON.stringify(cached));
    }
    
    handleRequest(clientReq, clientRes) {
        const targetUrl = clientReq.url.substring(1); // Remove leading /
        
        // Check cache first
        const cached = this.getCachedResponse(targetUrl);
        if (cached) {
            console.log('Serving from cache:', targetUrl);
            clientRes.writeHead(cached.statusCode, cached.headers);
            clientRes.end(cached.body);
            return;
        }
        
        // Forward request
        const url = new URL(targetUrl);
        const options = {
            hostname: url.hostname,
            path: url.pathname + url.search,
            method: clientReq.method,
            headers: { ...clientReq.headers }
        };
        
        const protocol = url.protocol === 'https:' ? https : http;
        const proxyReq = protocol.request(options, (proxyRes) => {
            let body = '';
            
            proxyRes.on('data', chunk => body += chunk);
            proxyRes.on('end', () => {
                // Cache successful responses
                if (proxyRes.statusCode === 200) {
                    this.cacheResponse(targetUrl, {
                        statusCode: proxyRes.statusCode,
                        headers: proxyRes.headers,
                        body: body
                    });
                }
                
                clientRes.writeHead(proxyRes.statusCode, proxyRes.headers);
                clientRes.end(body);
            });
        });
        
        clientReq.pipe(proxyReq);
    }
}

// Usage
const cachingProxy = new CachingProxy();
const server = http.createServer((req, res) => {
    cachingProxy.handleRequest(req, res);
});

server.listen(8080);

An HTTP client is a program that sends HTTP requests to servers and processes the responses. Node.js provides built-in HTTP client capabilities through the http and https modules.

Built-in HTTP client examples:

const http = require('http');
const https = require('https');

// Simple GET request
function simpleGet(url) {
    return new Promise((resolve, reject) => {
        const protocol = url.startsWith('https') ? https : http;
        
        const req = protocol.get(url, (res) => {
            let data = '';
            
            res.on('data', chunk => data += chunk);
            res.on('end', () => {
                resolve({
                    statusCode: res.statusCode,
                    headers: res.headers,
                    body: data
                });
            });
        });
        
        req.on('error', reject);
        req.setTimeout(5000, () => {
            req.destroy();
            reject(new Error('Request timeout'));
        });
    });
}

// POST request with JSON body
function postJSON(url, data) {
    return new Promise((resolve, reject) => {
        const urlObj = new URL(url);
        const protocol = urlObj.protocol === 'https:' ? https : http;
        const postData = JSON.stringify(data);
        
        const options = {
            hostname: urlObj.hostname,
            port: urlObj.port,
            path: urlObj.pathname,
            method: 'POST',
            headers: {
                'Content-Type': 'application/json',
                'Content-Length': Buffer.byteLength(postData)
            }
        };
        
        const req = protocol.request(options, (res) => {
            let responseData = '';
            res.on('data', chunk => responseData += chunk);
            res.on('end', () => {
                resolve({
                    statusCode: res.statusCode,
                    body: responseData
                });
            });
        });
        
        req.on('error', reject);
        req.write(postData);
        req.end();
    });
}

// Usage examples
simpleGet('http://jsonplaceholder.typicode.com/posts/1')
    .then(result => console.log('GET result:', result));

postJSON('http://jsonplaceholder.typicode.com/posts', {
    title: 'Test Post',
    body: 'This is a test',
    userId: 1
}).then(result => console.log('POST result:', result));

HTTP requests can be made using Node.js's built-in modules or third-party libraries. Here are various ways to make HTTP requests.

Using built-in http module:

const http = require('http');
const https = require('https');

class HTTPClient {
    static request(options) {
        return new Promise((resolve, reject) => {
            const protocol = options.protocol === 'https:' ? https : http;
            
            const req = protocol.request(options, (res) => {
                const response = {
                    statusCode: res.statusCode,
                    headers: res.headers,
                    body: ''
                };
                
                res.on('data', chunk => response.body += chunk);
                res.on('end', () => resolve(response));
            });
            
            req.on('error', reject);
            req.setTimeout(options.timeout || 5000, () => {
                req.destroy();
                reject(new Error('Request timeout'));
            });
            
            if (options.body) {
                req.write(options.body);
            }
            req.end();
        });
    }
    
    static get(url, options = {}) {
        const urlObj = new URL(url);
        return this.request({
            protocol: urlObj.protocol,
            hostname: urlObj.hostname,
            port: urlObj.port,
            path: urlObj.pathname + urlObj.search,
            method: 'GET',
            ...options
        });
    }
    
    static post(url, data, options = {}) {
        const urlObj = new URL(url);
        const body = JSON.stringify(data);
        
        return this.request({
            protocol: urlObj.protocol,
            hostname: urlObj.hostname,
            port: urlObj.port,
            path: urlObj.pathname,
            method: 'POST',
            headers: {
                'Content-Type': 'application/json',
                'Content-Length': Buffer.byteLength(body),
                ...options.headers
            },
            body: body,
            ...options
        });
    }
}

// Usage
HTTPClient.get('https://jsonplaceholder.typicode.com/posts/1')
    .then(response => console.log('GET:', response.body));

HTTPClient.post('https://jsonplaceholder.typicode.com/posts', {
    title: 'Test',
    body: 'Content',
    userId: 1
}).then(response => console.log('POST:', response.body));

node-fetch is a light-weight module that brings window.fetch to Node.js, providing a familiar API for making HTTP requests similar to browsers.

Using node-fetch:

const fetch = require('node-fetch');

// Basic GET request
async function fetchData() {
    try {
        const response = await fetch('https://jsonplaceholder.typicode.com/posts/1');
        
        if (!response.ok) {
            throw new Error(`HTTP error! status: ${response.status}`);
        }
        
        const data = await response.json();
        console.log('Fetched data:', data);
        
    } catch (error) {
        console.log('Fetch error:', error.message);
    }
}

// POST request with JSON
async function postData() {
    const response = await fetch('https://jsonplaceholder.typicode.com/posts', {
        method: 'POST',
        body: JSON.stringify({
            title: 'Test Post',
            body: 'This is a test',
            userId: 1
        }),
        headers: {
            'Content-Type': 'application/json'
        }
    });
    
    return await response.json();
}

// Advanced usage with timeouts and retries
class FetchClient {
    constructor(defaultOptions = {}) {
        this.defaultOptions = {
            timeout: 5000,
            retries: 3,
            ...defaultOptions
        };
    }
    
    async fetchWithRetry(url, options = {}) {
        let lastError;
        
        for (let attempt = 1; attempt <= this.defaultOptions.retries; attempt++) {
            try {
                const controller = new AbortController();
                const timeoutId = setTimeout(() => controller.abort(), this.defaultOptions.timeout);
                
                const response = await fetch(url, {
                    ...this.defaultOptions,
                    ...options,
                    signal: controller.signal
                });
                
                clearTimeout(timeoutId);
                
                if (!response.ok) {
                    throw new Error(`HTTP ${response.status}: ${response.statusText}`);
                }
                
                return response;
                
            } catch (error) {
                lastError = error;
                console.log(`Attempt ${attempt} failed:`, error.message);
                
                if (attempt < this.defaultOptions.retries) {
                    await this.delay(1000 * attempt); // Exponential backoff
                }
            }
        }
        
        throw lastError;
    }
    
    delay(ms) {
        return new Promise(resolve => setTimeout(resolve, ms));
    }
}

// Usage
const client = new FetchClient({ timeout: 3000, retries: 2 });
client.fetchWithRetry('https://api.example.com/data')
    .then(response => response.json())
    .then(data => console.log('Data:', data))
    .catch(error => console.log('All attempts failed:', error.message));

Axios is a popular Promise-based HTTP client for both browser and Node.js that provides a simple and clean API.

Basic axios usage:

const axios = require('axios');

// GET request
async function getData() {
    try {
        const response = await axios.get('https://jsonplaceholder.typicode.com/posts/1');
        console.log('Response:', response.data);
    } catch (error) {
        console.log('Error:', error.message);
    }
}

// POST request
async function createPost() {
    const response = await axios.post('https://jsonplaceholder.typicode.com/posts', {
        title: 'Test Post',
        body: 'This is a test',
        userId: 1
    });
    return response.data;
}

// Concurrent requests
async function makeConcurrentRequests() {
    try {
        const [userResponse, postResponse] = await Promise.all([
            axios.get('https://jsonplaceholder.typicode.com/users/1'),
            axios.get('https://jsonplaceholder.typicode.com/posts/1')
        ]);
        
        console.log('User:', userResponse.data);
        console.log('Post:', postResponse.data);
    } catch (error) {
        console.log('Error in concurrent requests:', error.message);
    }
}

Advanced axios configuration:

const axios = require('axios');

// Create axios instance with default config
const apiClient = axios.create({
    baseURL: 'https://api.example.com/v1',
    timeout: 5000,
    headers: {
        'Content-Type': 'application/json',
        'Authorization': 'Bearer your-token-here'
    }
});

// Request interceptor
apiClient.interceptors.request.use(
    (config) => {
        console.log(`Making ${config.method.toUpperCase()} request to ${config.url}`);
        return config;
    },
    (error) => {
        return Promise.reject(error);
    }
);

// Response interceptor
apiClient.interceptors.response.use(
    (response) => {
        console.log(`Received ${response.status} from ${response.config.url}`);
        return response;
    },
    (error) => {
        if (error.response) {
            console.log(`Server responded with ${error.response.status}`);
        } else if (error.request) {
            console.log('No response received:', error.message);
        } else {
            console.log('Request setup error:', error.message);
        }
        return Promise.reject(error);
    }
);

// Usage with the configured client
async function useApiClient() {
    try {
        const response = await apiClient.get('/users');
        return response.data;
    } catch (error) {
        if (error.response && error.response.status === 401) {
            // Handle unauthorized
            console.log('Authentication required');
        }
        throw error;
    }
}

Keep-alive connections allow multiple HTTP requests to be sent over the same TCP connection, reducing latency and improving performance.

Enable keep-alive in Node.js:

const http = require('http');

// Server with keep-alive
const server = http.createServer((req, res) => {
    res.writeHead(200, { 
        'Content-Type': 'text/plain',
        'Connection': 'keep-alive'
    });
    res.end('Hello with keep-alive');
});

server.keepAliveTimeout = 30000; // 30 seconds
server.headersTimeout = 35000; // 35 seconds

// Client with keep-alive
function makeKeepAliveRequests() {
    const agent = new http.Agent({
        keepAlive: true,
        keepAliveMsecs: 1000,
        maxSockets: 10,
        maxFreeSockets: 5
    });
    
    // Make multiple requests using the same agent
    for (let i = 0; i < 5; i++) {
        const req = http.request({
            hostname: 'localhost',
            port: 3000,
            path: `/request-${i}`,
            agent: agent,
            headers: { 'Connection': 'keep-alive' }
        }, (res) => {
            console.log(`Request ${i} - Reused socket: ${res.reusedSocket}`);
            res.on('data', () => {}); // Consume data
        });
        
        req.end();
    }
}

Response compression reduces bandwidth usage by compressing response bodies using algorithms like gzip, deflate, or brotli.

Compressing responses with zlib:

const http = require('http');
const zlib = require('zlib');

const server = http.createServer((req, res) => {
    const responseData = {
        message: 'This is a compressed response',
        timestamp: new Date().toISOString(),
        data: Array(1000).fill('Some repeated data to make compression worthwhile')
    };
    
    const jsonData = JSON.stringify(responseData);
    
    // Check client's accepted encodings
    const acceptEncoding = req.headers['accept-encoding'] || '';
    
    if (acceptEncoding.includes('gzip')) {
        zlib.gzip(jsonData, (err, compressed) => {
            if (err) {
                res.writeHead(500);
                res.end('Compression error');
                return;
            }
            
            res.writeHead(200, {
                'Content-Type': 'application/json',
                'Content-Encoding': 'gzip',
                'Vary': 'Accept-Encoding'
            });
            res.end(compressed);
        });
    } else if (acceptEncoding.includes('deflate')) {
        zlib.deflate(jsonData, (err, compressed) => {
            res.writeHead(200, {
                'Content-Type': 'application/json',
                'Content-Encoding': 'deflate'
            });
            res.end(compressed);
        });
    } else {
        // No compression
        res.writeHead(200, { 'Content-Type': 'application/json' });
        res.end(jsonData);
    }
});

Content negotiation allows clients and servers to agree on the best representation of a resource based on factors like content type, language, and encoding.

Content negotiation implementation:

const http = require('http');

const server = http.createServer((req, res) => {
    const resource = {
        id: 1,
        title: 'Sample Resource',
        content: 'This is the content',
        languages: {
            en: 'Hello World',
            es: 'Hola Mundo',
            fr: 'Bonjour le monde'
        }
    };
    
    // Content-Type negotiation
    const acceptHeader = req.headers['accept'] || 'text/plain';
    
    if (acceptHeader.includes('application/json')) {
        res.writeHead(200, { 'Content-Type': 'application/json' });
        res.end(JSON.stringify(resource));
    } else if (acceptHeader.includes('text/html')) {
        res.writeHead(200, { 'Content-Type': 'text/html' });
        res.end(`
            <html>
                <body>
                    <h1>${resource.title}</h1>
                    <p>${resource.content}</p>
                </body>
            </html>
        `);
    } else if (acceptHeader.includes('text/xml')) {
        res.writeHead(200, { 'Content-Type': 'text/xml' });
        res.end(`
            <resource>
                <id>${resource.id}</id>
                <title>${resource.title}</title>
                <content>${resource.content}</content>
            </resource>
        `);
    } else {
        res.writeHead(200, { 'Content-Type': 'text/plain' });
        res.end(`${resource.title}: ${resource.content}`);
    }
});

Cookies are small pieces of data stored by the browser and sent with HTTP requests. They're commonly used for session management, personalization, and tracking.

Cookie management in Node.js:

const http = require('http');

function parseCookies(cookieHeader) {
    const cookies = {};
    if (cookieHeader) {
        cookieHeader.split(';').forEach(cookie => {
            const [name, value] = cookie.trim().split('=');
            cookies[name] = value;
        });
    }
    return cookies;
}

function setCookie(res, name, value, options = {}) {
    const cookieOptions = [
        `${name}=${value}`,
        options.path ? `Path=${options.path}` : 'Path=/',
        options.domain ? `Domain=${options.domain}` : '',
        options.expires ? `Expires=${options.expires.toUTCString()}` : '',
        options.maxAge ? `Max-Age=${options.maxAge}` : '',
        options.secure ? 'Secure' : '',
        options.httpOnly ? 'HttpOnly' : '',
        options.sameSite ? `SameSite=${options.sameSite}` : ''
    ].filter(opt => opt).join('; ');
    
    res.setHeader('Set-Cookie', cookieOptions);
}

const server = http.createServer((req, res) => {
    // Parse incoming cookies
    const cookies = parseCookies(req.headers.cookie);
    console.log('Received cookies:', cookies);
    
    // Session management example
    if (req.url === '/login') {
        // Set session cookie
        setCookie(res, 'sessionId', 'abc123', {
            httpOnly: true,
            secure: true,
            sameSite: 'Strict',
            maxAge: 24 * 60 * 60 // 1 day
        });
        
        setCookie(res, 'user', 'john_doe', {
            maxAge: 7 * 24 * 60 * 60 // 1 week
        });
        
        res.end('Logged in successfully');
    } else if (req.url === '/logout') {
        // Clear cookies by setting expired dates
        setCookie(res, 'sessionId', '', { maxAge: 0 });
        setCookie(res, 'user', '', { maxAge: 0 });
        res.end('Logged out successfully');
    } else {
        // Check for session
        if (cookies.sessionId) {
            res.end(`Welcome back! User: ${cookies.user || 'Unknown'}`);
        } else {
            res.end('Please log in');
        }
    }
});

Security headers help protect web applications from common attacks like XSS, clickjacking, and MIME sniffing.

Implementing security headers:

const http = require('http');

function setSecurityHeaders(res) {
    // Content Security Policy
    res.setHeader('Content-Security-Policy', 
        "default-src 'self'; script-src 'self' 'unsafe-inline'; style-src 'self' 'unsafe-inline'"
    );
    
    // Prevent MIME sniffing
    res.setHeader('X-Content-Type-Options', 'nosniff');
    
    // Prevent clickjacking
    res.setHeader('X-Frame-Options', 'DENY');
    
    // XSS protection
    res.setHeader('X-XSS-Protection', '1; mode=block');
    
    // Referrer policy
    res.setHeader('Referrer-Policy', 'strict-origin-when-cross-origin');
    
    // Feature policy
    res.setHeader('Feature-Policy', 
        "geolocation 'none'; microphone 'none'; camera 'none'"
    );
    
    // Remove server identification
    res.removeHeader('X-Powered-By');
}

const server = http.createServer((req, res) => {
    // Set security headers for all responses
    setSecurityHeaders(res);
    
    // Your application logic
    if (req.url === '/secure') {
        res.writeHead(200, { 'Content-Type': 'application/json' });
        res.end(JSON.stringify({ message: 'Secure endpoint' }));
    } else {
        res.writeHead(200, { 'Content-Type': 'text/html' });
        res.end(`
            <html>
                <head>
                    <title>Secure App</title>
                </head>
                <body>
                    <h1>Secure Application</h1>
                    <p>This app uses security headers</p>
                </body>
            </html>
        `);
    }
});

// HTTPS redirect middleware
function requireHTTPS(req, res, next) {
    if (req.headers['x-forwarded-proto'] !== 'https' && process.env.NODE_ENV === 'production') {
        return res.writeHead(301, {
            'Location': `https://${req.headers.host}${req.url}`
        }).end();
    }
    next();
}

Express provides flexible ways to handle multiple routes using different HTTP methods, route parameters, and organized routing structures.

Basic multiple route handling:

const express = require('express');
const app = express();

// Multiple HTTP methods for same path
app.get('/products', (req, res) => {
    res.send('Get all products');
});

app.post('/products', (req, res) => {
    res.send('Create new product');
});

app.get('/products/:id', (req, res) => {
    res.send(`Get product ${req.params.id}`);
});

app.put('/products/:id', (req, res) => {
    res.send(`Update product ${req.params.id}`);
});

app.delete('/products/:id', (req, res) => {
    res.send(`Delete product ${req.params.id}`);
});

// Different paths with similar logic
app.get('/users', (req, res) => {
    res.send('Get all users');
});

app.get('/admins', (req, res) => {
    res.send('Get all admins');
});

app.get('/customers', (req, res) => {
    res.send('Get all customers');
});

Using app.route() for cleaner code:

app.route('/articles')
    .get((req, res) => {
        res.send('Get all articles');
    })
    .post((req, res) => {
        res.send('Create new article');
    });

app.route('/articles/:id')
    .get((req, res) => {
        res.send(`Get article ${req.params.id}`);
    })
    .put((req, res) => {
        res.send(`Update article ${req.params.id}`);
    })
    .delete((req, res) => {
        res.send(`Delete article ${req.params.id}`);
    });

Route parameters are named URL segments used to capture values at specific positions in the URL. They're defined with colon syntax and accessible via req.params.

Basic route parameters:

const express = require('express');
const app = express();

// Single parameter
app.get('/users/:userId', (req, res) => {
    res.send(`User ID: ${req.params.userId}`);
});

// Multiple parameters
app.get('/posts/:postId/comments/:commentId', (req, res) => {
    res.json({
        postId: req.params.postId,
        commentId: req.params.commentId
    });
});

// Optional parameters (with ?)
app.get('/books/:category?', (req, res) => {
    if (req.params.category) {
        res.send(`Books in category: ${req.params.category}`);
    } else {
        res.send('All books');
    }
});

// Parameter with constraints using regex
app.get('/products/:id(\\d+)', (req, res) => {
    // Only matches if :id is numeric
    res.send(`Product ID: ${req.params.id}`);
});

Query parameters are key-value pairs in the URL after the ? symbol. Express automatically parses them and makes them available in req.query.

Accessing query parameters:

const express = require('express');
const app = express();

// Basic query parameters
app.get('/search', (req, res) => {
    const { q, category, sort } = req.query;
    res.json({
        query: q,
        category: category,
        sort: sort,
        allParams: req.query
    });
});

// URL: /search?q=javascript&category=books&sort=price

// Multiple values for same parameter
app.get('/filters', (req, res) => {
    // URL: /filters?tags=js&tags=node&tags=express
    const tags = Array.isArray(req.query.tags) ? req.query.tags : [req.query.tags];
    res.json({ tags });
});

// With default values
app.get('/products', (req, res) => {
    const page = parseInt(req.query.page) || 1;
    const limit = parseInt(req.query.limit) || 10;
    const sort = req.query.sort || 'name';
    
    res.json({
        page,
        limit,
        sort,
        message: `Showing page ${page} with ${limit} items, sorted by ${sort}`
    });
});

app.use() is used to mount middleware functions at a specified path. It can be used for application-level middleware, router-level middleware, and error handling.

Using app.use() for middleware:

const express = require('express');
const app = express();

// Application-level middleware (runs for every request)
app.use((req, res, next) => {
    console.log(`${req.method} ${req.url} at ${new Date()}`);
    next();
});

// Mount middleware at specific path
app.use('/api', (req, res, next) => {
    console.log('API route accessed');
    next();
});

// Multiple middleware functions
app.use(express.json()); // Parse JSON bodies
app.use(express.urlencoded({ extended: true })); // Parse form data

// Third-party middleware
app.use(require('cors')()); // Enable CORS
app.use(require('helmet')()); // Security headers

// Serving static files
app.use(express.static('public'));

// Router-level middleware
app.use('/users', require('./routes/users'));

Middleware are functions that have access to the request object (req), response object (res), and the next middleware function in the application's request-response cycle.

Middleware execution flow:

Request → Middleware 1 → Middleware 2 → ... → Route Handler → Response

Types of middleware:

const express = require('express');
const app = express();

// 1. Application-level middleware
app.use((req, res, next) => {
    console.log('Application middleware - runs for every request');
    next();
});

// 2. Router-level middleware
const router = express.Router();
router.use((req, res, next) => {
    console.log('Router middleware - runs for router routes');
    next();
});

// 3. Error-handling middleware
app.use((err, req, res, next) => {
    console.error(err.stack);
    res.status(500).send('Something broke!');
});

// 4. Built-in middleware
app.use(express.json());
app.use(express.urlencoded({ extended: true }));

// 5. Third-party middleware
app.use(require('cors')());
app.use(require('helmet')());

Custom middleware functions are created to handle specific tasks like authentication, logging, data validation, etc.

Creating custom middleware:

const express = require('express');
const app = express();

// Simple logging middleware
app.use((req, res, next) => {
    console.log(`${req.method} ${req.originalUrl} - ${new Date().toISOString()}`);
    next();
});

// Authentication middleware
const authenticate = (req, res, next) => {
    const token = req.headers.authorization;
    
    if (!token) {
        return res.status(401).json({ error: 'No token provided' });
    }
    
    // Verify token (simplified)
    if (token !== 'secret-token') {
        return res.status(401).json({ error: 'Invalid token' });
    }
    
    req.user = { id: 1, name: 'John Doe' }; // Attach user to request
    next();
};

// Validation middleware
const validateUser = (req, res, next) => {
    const { name, email, age } = req.body;
    
    if (!name || !email) {
        return res.status(400).json({ error: 'Name and email are required' });
    }
    
    if (age && (age < 0 || age > 150)) {
        return res.status(400).json({ error: 'Invalid age' });
    }
    
    next();
};

// Usage
app.post('/users', authenticate, validateUser, (req, res) => {
    res.json({ message: 'User created', user: req.user });
});

next() is a function passed to middleware that, when called, passes control to the next middleware function in the stack.

Using next() in middleware:

const express = require('express');
const app = express();

// Calling next() to continue
app.use((req, res, next) => {
    console.log('Middleware 1');
    next(); // Pass control to next middleware
});

app.use((req, res, next) => {
    console.log('Middleware 2');
    next(); // Continue to route handler
});

app.get('/', (req, res) => {
    res.send('Hello World');
});

// Using next('route') to skip remaining middleware
app.get('/user/:id', (req, res, next) => {
    if (req.params.id === '0') {
        next('route'); // Skip to next route with same path
    } else {
        next(); // Continue to next middleware
    }
}, (req, res, next) => {
    res.send('Regular user');
});

app.get('/user/:id', (req, res) => {
    res.send('Special user');
});

// Error handling with next(err)
app.get('/error', (req, res, next) => {
    const err = new Error('Something went wrong');
    err.status = 500;
    next(err); // Pass error to error handler
});

404 errors are handled by creating a catch-all route at the end of your middleware stack that responds when no other routes match.

404 error handling:

const express = require('express');
const app = express();

// Your regular routes
app.get('/', (req, res) => {
    res.send('Home Page');
});

app.get('/about', (req, res) => {
    res.send('About Page');
});

// 404 handler - catches all unmatched routes
app.use('*', (req, res) => {
    res.status(404).json({
        error: 'Route not found',
        path: req.originalUrl,
        method: req.method,
        timestamp: new Date().toISOString()
    });
});

// Alternative 404 handler with HTML response
app.use('*', (req, res) => {
    res.status(404).send(`
        
            
            
                
404 - Page Not Found
                
The requested URL ${req.originalUrl} was not found on this server.
                Go to Home Page
            
        
    `);
});

Global error handling is done using error-handling middleware with four parameters (err, req, res, next) placed after all other middleware.

Global error handler:

const express = require('express');
const app = express();

// Regular routes
app.get('/', (req, res) => {
    res.send('Home Page');
});

app.get('/error', (req, res) => {
    throw new Error('This is a test error');
});

// Async error example
app.get('/async-error', async (req, res, next) => {
    try {
        await someAsyncOperation();
        res.send('Success');
    } catch (error) {
        next(error); // Pass to error handler
    }
});

// Global error handling middleware
app.use((err, req, res, next) => {
    console.error('Error stack:', err.stack);
    
    // Set default status code
    const statusCode = err.status || err.statusCode || 500;
    
    // Development vs production error responses
    const isDevelopment = process.env.NODE_ENV === 'development';
    
    res.status(statusCode).json({
        error: isDevelopment ? err.message : 'Something went wrong!',
        ...(isDevelopment && { stack: err.stack }),
        ...(err.details && { details: err.details }),
        timestamp: new Date().toISOString()
    });
});

JSON responses are sent using res.json() which automatically sets the Content-Type header to application/json.

Sending JSON responses:

const express = require('express');
const app = express();

// Basic JSON response
app.get('/api/user', (req, res) => {
    res.json({
        id: 1,
        name: 'John Doe',
        email: 'john@example.com',
        active: true
    });
});

// JSON with status code
app.post('/api/users', (req, res) => {
    // Create user logic...
    res.status(201).json({
        success: true,
        message: 'User created successfully',
        data: {
            id: 123,
            name: 'New User'
        }
    });
});

// Error JSON response
app.get('/api/error', (req, res) => {
    res.status(400).json({
        success: false,
        error: {
            code: 'VALIDATION_ERROR',
            message: 'Invalid input data',
            details: ['Name is required', 'Email must be valid']
        }
    });
});

// JSON with custom headers
app.get('/api/data', (req, res) => {
    res.set({
        'Content-Type': 'application/json',
        'Cache-Control': 'no-cache',
        'X-Custom-Header': 'custom-value'
    }).json({
        data: [1, 2, 3, 4, 5],
        count: 5,
        timestamp: new Date().toISOString()
    });
});

res.sendFile() sends a file as an HTTP response. It automatically sets the Content-Type based on the file extension.

Using res.sendFile():

const express = require('express');
const path = require('path');
const app = express();

// Send file with absolute path
app.get('/download', (req, res) => {
    const filePath = path.join(__dirname, 'files', 'document.pdf');
    res.sendFile(filePath);
});

// Send file with custom filename
app.get('/download-custom', (req, res) => {
    const filePath = path.join(__dirname, 'files', 'report.pdf');
    res.sendFile(filePath, {
        headers: {
            'Content-Disposition': 'attachment; filename="monthly-report.pdf"'
        }
    });
});

// Send HTML file
app.get('/page', (req, res) => {
    res.sendFile(path.join(__dirname, 'views', 'index.html'));
});

// Send file with error handling
app.get('/safe-download', (req, res) => {
    const fileName = req.query.file;
    
    // Security: Validate filename to prevent directory traversal
    if (!fileName || fileName.includes('..')) {
        return res.status(400).send('Invalid filename');
    }
    
    const filePath = path.join(__dirname, 'uploads', fileName);
    
    res.sendFile(filePath, (err) => {
        if (err) {
            console.error('File send error:', err);
            if (!res.headersSent) {
                res.status(404).send('File not found');
            }
        }
    });
});

Static files (CSS, JavaScript, images) are served using express.static() middleware.

Serving static files:

const express = require('express');
const path = require('path');
const app = express();

// Serve files from 'public' directory
app.use(express.static('public'));

// Now you can access:
// http://localhost:3000/css/style.css
// http://localhost:3000/js/app.js
// http://localhost:3000/images/logo.png

// Multiple static directories
app.use(express.static('public'));
app.use(express.static('uploads'));

// Virtual path prefix
app.use('/static', express.static('public'));
// Now files are available at:
// http://localhost:3000/static/css/style.css

// With cache control
app.use('/assets', express.static('public', {
    maxAge: '1d', // Cache for 1 day
    etag: false   // Disable ETag generation
}));

// Security: Don't serve sensitive files
app.use(express.static('public', {
    dotfiles: 'deny', // Deny access to dotfiles (.env, .gitignore)
    index: false       // Don't serve directory index
}));

express.static() is built-in Express middleware that serves static files and is the simplest way to serve images, CSS, JavaScript, etc.

express.static() options:

const express = require('express');
const app = express();

app.use(express.static('public', {
    // Cache control
    maxAge: '1d', // Cache for 1 day
    etag: true,   // Enable ETag (default)
    
    // Security options
    dotfiles: 'ignore', // 'allow', 'deny', 'ignore'
    index: 'index.html', // Default file to serve
    
    // Redirect trailing slash
    redirect: true,
    
    // Custom headers
    setHeaders: (res, path, stat) => {
        res.set('X-Custom-Header', 'static-file');
        
        // Set specific headers for certain file types
        if (path.endsWith('.css')) {
            res.set('Content-Type', 'text/css');
        }
    }
}));

// Multiple static configurations
app.use('/css', express.static('styles', { 
    maxAge: '7d', 
    index: false 
}));

app.use('/js', express.static('scripts', { 
    maxAge: '1h' 
}));

JSON request bodies are parsed using express.json() middleware, which makes the parsed data available in req.body.

Parsing JSON bodies:

const express = require('express');
const app = express();

// Parse JSON bodies
app.use(express.json());

// Now req.body contains parsed JSON
app.post('/api/users', (req, res) => {
    const { name, email, age } = req.body;
    
    // Validate and process the data
    if (!name || !email) {
        return res.status(400).json({
            error: 'Name and email are required'
        });
    }
    
    res.status(201).json({
        message: 'User created',
        user: { name, email, age }
    });
});

// With custom options
app.use(express.json({
    limit: '10mb',        // Maximum request body size
    strict: true,         // Only accept arrays and objects
    type: 'application/json' // Only parse when Content-Type matches
}));

// Example request that would be parsed:
// POST /api/users
// Content-Type: application/json
// {
//   "name": "John Doe",
//   "email": "john@example.com",
//   "age": 30
// }

body-parser was previously a separate middleware for parsing request bodies, but is now built into Express as express.json() and express.urlencoded().

Using body-parser (legacy approach):

// Old way (before Express 4.16)
const express = require('express');
const bodyParser = require('body-parser');
const app = express();

app.use(bodyParser.json());
app.use(bodyParser.urlencoded({ extended: true }));

// Modern way (Express 4.16+)
app.use(express.json());        // Replaces bodyParser.json()
app.use(express.urlencoded({ extended: true })); // Replaces bodyParser.urlencoded()

// Comparison of old vs new
const oldWay = {
    json: "app.use(bodyParser.json())",
    urlencoded: "app.use(bodyParser.urlencoded({ extended: true }))"
};

const newWay = {
    json: "app.use(express.json())",
    urlencoded: "app.use(express.urlencoded({ extended: true }))"
};

// Both parse:
// - JSON bodies (Content-Type: application/json)
// - URL-encoded bodies (Content-Type: application/x-www-form-urlencoded)

express.Router() creates modular, mountable route handlers. A Router instance is a complete middleware and routing system.

Using express.Router():

// routes/users.js
const express = require('express');
const router = express.Router();

// Middleware specific to this router
router.use((req, res, next) => {
    console.log('User routes accessed');
    next();
});

// Define routes
router.get('/', (req, res) => {
    res.json({ message: 'Get all users' });
});

router.get('/:id', (req, res) => {
    res.json({ message: `Get user ${req.params.id}` });
});

router.post('/', (req, res) => {
    res.json({ message: 'Create user', data: req.body });
});

router.put('/:id', (req, res) => {
    res.json({ message: `Update user ${req.params.id}` });
});

router.delete('/:id', (req, res) => {
    res.json({ message: `Delete user ${req.params.id}` });
});

module.exports = router;

// In main app file
const express = require('express');
const app = express();
const userRoutes = require('./routes/users');

app.use('/api/users', userRoutes);
// Now all user routes are available under /api/users

Route modularization involves organizing routes into separate files using express.Router() for better maintainability.

Route modularization structure:

// Project structure:
// routes/
//   users.js
//   products.js
//   orders.js
// app.js

// routes/users.js
const express = require('express');
const router = express.Router();

router.get('/', (req, res) => {
    res.json({ users: [] });
});

router.post('/', (req, res) => {
    // Create user logic
    res.status(201).json({ message: 'User created' });
});

module.exports = router;

// routes/products.js
const express = require('express');
const router = express.Router();

router.get('/', (req, res) => {
    res.json({ products: [] });
});

router.get('/:id', (req, res) => {
    res.json({ product: { id: req.params.id } });
});

module.exports = router;

// app.js
const express = require('express');
const app = express();

const userRoutes = require('./routes/users');
const productRoutes = require('./routes/products');

app.use('/api/users', userRoutes);
app.use('/api/products', productRoutes);

URL parameters are captured from the route path and accessed via req.params object.

URL parameter examples:

const express = require('express');
const app = express();

app.get('/users/:userId/posts/:postId', (req, res) => {
    res.json({
        userId: req.params.userId,
        postId: req.params.postId
    });
});

Multiple middlewares can be applied by passing an array or multiple arguments to app.use() or route methods.

Multiple middleware examples:

const express = require('express');
const app = express();

const auth = (req, res, next) => { next(); };
const log = (req, res, next) => { next(); };
const validate = (req, res, next) => { next(); };

app.use([auth, log]); // Array of middlewares
app.get('/protected', auth, validate, (req, res) => {
    res.send('Protected route');
});

Request logging can be implemented using custom middleware or third-party packages like morgan.

Custom request logging:

const express = require('express');
const app = express();

app.use((req, res, next) => {
    console.log(`${new Date().toISOString()} - ${req.method} ${req.url}`);
    next();
});