Reads data from an upstream-backed nginx location by running several concurrent range requests.

Add --add-module when configuring nginx:

./configure --add-module=$PATH_TO_PARALLEL_MODULE

• syntax: parallel uri_prefix
• default: none
• context: location

Enables the parallel module on the enclosing location, the uri_prefix is added to the request URI to form the URI of the subrequests

• syntax: parallel_fiber_count count
• default: 8
• context: http, server, location

Sets the maximum number of concurrent subrequests

• syntax: parallel_min_chunk_size size
• default: 128k
• context: http, server, location

Sets the minimum size that is requested in a single subrequest, only the request for the last chunk is allowed to return less than this value. This parameter also controls the size that is requested in the initial subrequests, that are issued before the response size is known.

• syntax: parallel_max_chunk_size size
• default: 512k
• context: http, server, location

Sets the maximum size that can be fetched in a single subrequest

• syntax: parallel_max_headers_size size
• default: 4k
• context: http, server, location

Sets the size that should be allocated for holding the subrequests headers

• syntax: parallel_max_buffer_count count
• default: 16
• context: http, server, location

Sets the maximum number of buffers that the module is allowed to allocate. Once the limit is reached, the module will until data is flushed to the client before issuing more subrequests.

• syntax: parallel_consistency_check_etag on/off
• default: on
• context: http, server, location

Enables/disables ETag consistency validation, when enabled, the server will make sure all ETag headers returned for subrequests are consistent and fail the request otherwise.

• syntax: parallel_consistency_check_last_modified on/off
• default: on
• context: http, server, location

Enables/disables Last-Modified consistency validation, when enabled, the server will make sure all Last-Modified headers returned for subrequests are consistent and fail the request otherwise.

• syntax: parallel_content_length_cache zone_name zone_size
• default: off
• context: http, server, location

Configures the size and shared memory object name of the content length cache. This cache holds the content lengths of requested URLs in order to optimize their retrieval in the future. Cached lengths serve as a hint to the module when making the initial batch of subrequests. If the content length is large, a larger chunk size can be used, if the content length is small, the module can avoid making subrequests that will fail with 416 error. An incorrect length that is saved in the cache can not affect the response generated by this module, but it can lead to a less efficient pull from the origin. Each saved content length record consumes about 80 bytes, so a cache of few hundred megabytes can probably last for a long time.

http {

upstream backend {
	server backendhost:80;
	keepalive 32;
}

server {

	location /parallel {
		parallel /proxy;
	}
	
	location /proxy/parallel/ {
		proxy_pass http://backend/;
		proxy_http_version 1.1;
		proxy_set_header Connection "";
		proxy_set_header Host $http_host;
		
		internal;
	}
}

}

All code in this project is released under the AGPLv3 license unless a different license for a particular library is specified in the applicable library path.

Copyright © Kaltura Inc. All rights reserved.