JS Rendering Proxy

Project Overview

JS Rendering Proxy is an open-source Docker-based proxy service that solves a critical problem in web scraping: enabling legacy crawlers to process modern JavaScript-heavy websites. Traditional PHP crawlers cannot execute JavaScript, making them ineffective for React, Angular, Vue, and other client-side rendered applications. This proxy bridges that gap by rendering pages server-side using headless Chrome.

The Problem

Modern web applications increasingly rely on client-side JavaScript for rendering content. This creates a significant challenge:

Legacy crawlers cannot process JavaScript
Search engines need fully rendered HTML for indexing
Data extraction tools fail on single-page applications (SPAs)
Testing tools require complete DOM structures

The Solution

JS Rendering Proxy acts as an intermediary layer that:

Receives requests from traditional crawlers
Uses Puppeteer (headless Chrome) to fully render the page
Executes all JavaScript and waits for content to load
Returns complete HTML to the requester
Remains transparent to the original crawler

Key Features

JavaScript Execution

Full page rendering with headless Chrome (Puppeteer)
Waits for dynamic content to load before returning
Configurable wait conditions for specific elements or network idle
JavaScript blocking option for performance optimization

Resource Optimization

Blocks CSS to speed up rendering
Blocks media files (images, fonts) to reduce bandwidth
Configurable JavaScript blocking for fine-tuned control
Concurrent request management for high throughput

Crawler Compatibility

Preserves HTTP error codes for accurate status handling
No automatic redirects - maintains crawler behavior
Timeout handling with configurable limits
Custom headers support for advanced control
Transparent proxy mode - works with existing crawlers

API Interface

Two usage modes:

Query parameter mode: /?render_url=[URL] with API key header
HTTP proxy mode: Configure as standard HTTP proxy

Technical Implementation

Architecture

Built with high-performance Fastify framework:

Fastify - Ultra-fast Node.js web server
Puppeteer - Headless Chrome automation
Docker - Containerized deployment
Node.js 18+ - Modern JavaScript runtime

Performance Features

Concurrent requests handling
Resource blocking for faster rendering
Request queuing for load management
Detailed logging for monitoring and debugging
Timeout management prevents hanging requests

Configuration

Customizable through headers:

X-API-Key - Authentication
X-Render-Timeout - Custom timeout values
X-Wait-Until - Network idle conditions
X-Block-Resources - Resource blocking options

Technologies Used

Runtime: Node.js 18+
Framework: Fastify (high-performance web server)
Browser Automation: Puppeteer
Containerization: Docker
Language: JavaScript (94.2%)
Package Manager: npm

Use Cases

Legacy Crawler Modernization

Enable existing PHP or Python crawlers to process modern JavaScript frameworks without code changes.

Search Engine Indexing

Provide fully-rendered HTML for search engines to properly index SPAs and dynamic content.

Data Extraction

Extract data from JavaScript-heavy websites that traditional tools cannot process.

Testing & Monitoring

Test how websites render with JavaScript enabled, monitor dynamic content changes.

Content Archival

Archive complete rendered state of web applications, not just static HTML.

Challenges & Solutions

Challenge 1: Performance Overhead

Solution: Implemented resource blocking (CSS, media) and concurrent request handling to minimize rendering time while maintaining functionality.

Challenge 2: Crawler Compatibility

Solution: Designed transparent proxy mode that preserves HTTP codes and redirect behavior, ensuring existing crawlers work without modification.

Challenge 3: Resource Management

Solution: Added timeout handling, request queuing, and detailed logging to prevent resource exhaustion and enable monitoring.

Results & Impact

Successfully renders React, Angular, Vue, and other modern frameworks
Enables legacy crawlers to process JavaScript without code changes
High-performance architecture handles concurrent requests efficiently
Open source with 1 star on GitHub
Docker deployment for easy installation and scaling
Active maintenance (updated April 2025)

Installation & Deployment

Docker Deployment

docker pull ssv445/js-rendering-proxy
docker run -p 3000:3000 ssv445/js-rendering-proxy

API Usage

# Query parameter mode
curl -H "X-API-Key: your-key" \
  "http://localhost:3000/?render_url=https://example.com"

# HTTP proxy mode
export http_proxy=http://localhost:3000
curl https://spa-website.com

Open Source Contribution

Repository: github.com/ssv445/JS-Rendering-Proxy-Docker
License: Open Source
Community: Open for contributions
Status: Active Development

Project Duration

Released: April 2025
Status: Active, Maintained

Technical Highlights

Bridges gap between legacy crawlers and modern web
Transparent proxy design for zero-code integration
High-performance Fastify framework
Docker containerization for easy deployment
Production-ready with timeout and error handling

Project Gallery