Node Unblocker : Node-unblocker for Web Scraping Github
Unblocker was originally a web proxy for evading internet censorship, similar to CGIproxy / PHProxy / Glype but written in node.js. It’s since morphed into a general-purpose library for proxying and rewriting remote webpages.
All data is processed and relayed to the client on the fly without unnecessary buffering, making unblocker one of the fastest web proxies available.
The magic part
The script uses “pretty” urls which, besides looking pretty, allow links with relative paths to just work without modification. (E.g. <a href="path/to/file2.html"></a>)
In addition to this, links that are relative to the root (E.g. <a href="/path/to/file2.html"></a>) can be handled without modification by checking the referrer and 307 redirecting them to the proper location in the referring site. (Although the proxy does attempt to rewrite these links to avoid the redirect.)
Cookies are proxied by adjusting their path to include the proxy’s URL, and a bit of extra work is done to ensure they remain intact when switching protocols or subdomains.
Limitations
Although the proxy works well for standard login forms and even most AJAX content, OAuth login forms and anything that uses postMessage (Google, Facebook, etc.) are not likely to work out of the box. This is not an insurmountable issue, but it’s not one that I expect to have fixed in the near term.
More advanced websites, such as Roblox, Discord, YouTube*, Instagram, etc. do not currently work. At the moment, there is no timeframe for when these might be supported.
- There is an example that detects YouTube video pages and replaces them with a custom page that just streams the video.
Patches are welcome, including both general-purpose improvements to go into the main library, and site-specific fixes to go in the examples folder.
Running the website on your computer
See https://github.com/nfriedly/nodeunblocker.com
Using unblocker as a library in your software
npm install --save unblocker
Unblocker exports an express-compatible API, so using in an express application is trivial:
var express = require('express')
var Unblocker = require('unblocker');
var app = express();
var unblocker = new Unblocker({prefix: '/proxy/'});
// this must be one of the first app.use() calls and must not be on a subdirectory to work properly
app.use(unblocker);
app.get('/', function(req, res) {
//...
});
// the upgrade handler allows unblocker to proxy websockets
app.listen(process.env.PORT || 8080).on('upgrade', unblocker.onUpgrade);
Configuration
Unblocker supports the following configuration options, defaults are shown:
{
prefix: '/proxy/', // Path that the proxied URLs begin with. '/' is not recommended due to a few edge cases.
host: null, // Host used in redirects (e.g `example.com` or `localhost:8080`). Default behavior is to determine this from the request headers.
requestMiddleware: [], // Array of functions that perform extra processing on client requests before they are sent to the remote server. API is detailed below.
responseMiddleware: [], // Array of functions that perform extra processing on remote responses before they are sent back to the client. API is detailed below.
standardMiddleware: true, // Allows you to disable all built-in middleware if you need to perform advanced customization of requests or responses.
clientScripts: true, // Injects JavaScript to force things like WebSockets and XMLHttpRequest to go through the proxy.
processContentTypes: [ // All built-in middleware that modifies the content of responses limits itself to these content-types.
'text/html',
'application/xml+xhtml',
'application/xhtml+xml',
'text/css'
],
httpAgent: null, //override agent used to request http response from server. see https://nodejs.org/api/http.html#http_class_http_agent
httpsAgent: null //override agent used to request https response from server. see https://nodejs.org/api/https.html#https_class_https_agent
}
Setting process.env.NODE_ENV='production' will enable more aggressive caching on the client scripts and potentially other optimizations in the future.
Custom Middleware
Unblocker “middleware” are small functions that allow you to inspect and modify requests and responses. The majority of Unblocker’s internal logic is implimented as middleware, and it’s possible to write custom middleware to augment or replace the built-in middleware.
Custom middleware should be a function that accepts a single data argument and runs synchronously.
To process request and response data, create a Transform Stream to perform the processing in chunks and pipe through this stream. (Example below.)
To respond directly to a request, add a function to config.requestMiddleware that handles the clientResponse (a standard http.ServerResponse when used directly, or a Express Response when used with Express. Once a response is sent, no further middleware will be executed for that request.
