A pure stream http push technology for your Nginx setup.
Comet made easy and really scalable.
Always give a look to changes in Changelog.textile.
# clone
git clone http://github.com/wandenberg/nginx-push-stream-module.git
cd nginx-push-stream-module
# build with 0.8.54 or any other nginx version later than 0.7.67
./build.sh master 0.8.54
cd build/nginx-0.8.54
# finish
sudo make install
# check
sudo /usr/local/nginx/sbin/nginx -v
nginx version: nginx/0.8.54
sudo /usr/local/nginx/sbin/nginx -c nginx-push-stream-module/misc/nginx.conf -t
the configuration file nginx-push-stream-module/misc/nginx.conf syntax is ok
configuration file nginx-push-stream-module/misc/nginx.conf test is successful
# run
sudo /usr/local/nginx/sbin/nginx -c nginx-push-stream-module/misc/nginx.conf
location /channels_stats {
# activate channels statistics mode for this location
push_stream_channels_statistics;
# query string based channel id
set $push_stream_channel_id $arg_id;
}
location /pub {
# activate publisher mode for this location
push_stream_publisher;
# query string based channel id
set $push_stream_channel_id $arg_id;
# message template
push_stream_message_template "<script>p(~id~,'~channel~','~text~');</script>";
# store messages in memory
push_stream_store_messages on;
# max messages to store in memory
push_stream_max_message_buffer_length 20;
# message ttl
push_stream_min_message_buffer_timeout 5m;
# client_max_body_size MUST be equal to client_body_buffer_size or
# you will be sorry.
client_max_body_size 32k;
client_body_buffer_size 32k;
}
location ~ /sub/(.*) {
# activate subscriber mode for this location
push_stream_subscriber;
# positional channel path
set $push_stream_channels_path $1;
# header to be sent when receiving new subscriber connection
push_stream_header_template "<html><head><meta http-equiv=\"Content-Type\" content=\"text/html; charset=utf-8\">\r\n<meta http-equiv=\"Cache-Control\" content=\"no-store\">\r\n<meta http-equiv=\"Cache-Control\" content=\"no-cache\">\r\n<meta http-equiv=\"Pragma\" content=\"no-cache\">\r\n<meta http-equiv=\"Expires\" content=\"Thu, 1 Jan 1970 00:00:00 GMT\">\r\n<script type=\"text/javascript\">\r\nwindow.onError = null;\r\ndocument.domain = 'localhost';\r\nparent.PushStream.register(this);\r\n</script>\r\n</head>\r\n<body onload=\"try { parent.PushStream.reset(this) } catch (e) {}\">";
# message template
push_stream_message_template "<script>p(~id~,'~channel~','~text~');</script>";
# content-type
push_stream_content_type "text/html; charset=utf-8";
# subscriber may create channels on demand or only authorized
# (publisher) may do it?
push_stream_authorized_channels_only off;
# ping frequency
push_stream_ping_message_interval 10s;
# connection ttl to enable recycle
push_stream_subscriber_connection_timeout 15m;
# solving some leakage problem with persistent connections in
# Nginx's chunked filter (ngx_http_chunked_filter_module.c)
chunked_transfer_encoding off;
}
You can feel the flavor right now at the command line. Try using more than
one terminal and start playing http pubsub:
# Pub
curl -s -v -X POST "http://localhost/pub?id=my_channel_1" -d "Hello World!"
# Sub
curl -s -v "http://localhost/sub/my_channel_1.b20"
# Channel Stats for publisher (json format)
curl -s -v "http://localhost/pub?id=my_channel_1"
# All Channels Stats summarized (json format)
curl -s -v "http://localhost/channels_stats"
# All Channels Stats detailed (json format)
curl -s -v "http://localhost/channels_stats?id=ALL"
# Channels Stats (json format)
curl -s -v "http://localhost/channels_stats?id=my_channel_1"
| variable | values | location |
| push_stream_channel_id | channel id | push_stream_publisher, push_stream_channels_statistics |
| push_stream_channels_path | set of channels id and backtrack desired messages | push_stream_subscriber |
A string to uniquely identify a communication channel. Must be present on location of the push_stream_publisher and push_stream_channels_statistics.
set $push_channel_id $arg_id;
#channel id is now the url query string parameter "id"
#(/pub?id=channel_id_string or /channels_stats?id=channel_id_string)
A string representing a set of channels id and backtrack desired messages separated by slash, example /channel1.b3/channel2.b5/channel3.b2.
The backtrack means the amount of old messages from each of the channels that will be delivered to the subscriber. On the example will be 3 messages from channel1, 5 from channel2 and 2 from channel3.
Backtrack isn’t needed, you can only sign channels without get old messages, or you can mix things.
More accepted examples: /channel1, /channel1/channel2, /channel1.b5/channel2, /channel1/channel2.b6_, …
Must be present on location of the push_streamsubscriber.
location /sub/(.*) {
set $push_stream_channels_path $1;
}
#channels path is now part of url
#(/sub/channel_id_string or /sub/channel_id_string.b2/other_channel)
| directive | default value | values | context | location |
| push_stream_channels_statistics | - | - | location | - |
| push_stream_publisher | - | - | location | - |
| push_stream_subscriber | - | - | location | - |
| push_stream_max_reserved_memory | 32M | size greater than 8 * ngx_pagesize | http | main nginx configuration |
| push_stream_store_messages | off | on, off | location | push_stream_publisher |
| push_stream_min_message_buffer_timeout | unset | time constant | location | push_stream_publisher |
| push_stream_max_message_buffer_length | unset | number | location | push_stream_publisher |
| push_stream_authorized_channels_only | off | on, off | location | push_stream_subscriber |
| push_stream_header_template | unset | any string | location | push_stream_subscriber |
| push_stream_content_type | text/plain | any valid content type | location | push_stream_subscriber |
| push_stream_ping_message_interval | unset | time constant | location | push_stream_subscriber |
| push_stream_subscriber_connection_timeout | unset | time constant | location | push_stream_subscriber |
| push_stream_broadcast_channel_max_qtd | unset | number | location | push_stream_subscriber |
| push_stream_max_channel_id_length | unset | number | http, location | (push_stream_subscriber and push_stream_publisher) or main nginx configuration |
| push_stream_broadcast_channel_prefix | unset | any string | http, location | (push_stream_subscriber and push_stream_publisher) or main nginx configuration |
| push_stream_message_template | unset | any string | http, location | (push_stream_subscriber and push_stream_publisher) or main nginx configuration |
| push_stream_max_number_of_channels | unset | number | http, location | (push_stream_subscriber and push_stream_publisher) or main nginx configuration |
| push_stream_max_number_of_broadcast_channels | unset | number | http, location | (push_stream_subscriber and push_stream_publisher) or main nginx configuration |
| push_stream_memory_cleanup_timeout | 30 seconds | time constant | http, location | (push_stream_subscriber and push_stream_publisher) or main nginx configuration |
New in version 0.2.0
context: location
Defines a location as a source of statistics. You can use this location to get statistics about a specific channel, or about all channels, in a resumed ou summarized way.
To get statistics about all channels in a summarized way you have to make a GET in this location without specify a name in the push_stream_channel_id variable.
To get statistics about all channels in a detailed way you have to specify “ALL” in the push_stream_channel_id.
To get statistics about a channel you have to specify the name in the push_stream_channel_id.
You can get statistics in the formats plain, xml, yaml and json. The default is json, to change this behavior you can use Accept header parameter passing values like “text/plain”, “application/xml”, “application/yaml” and “application/json” respectivelly.
location /channels_stats {
push_stream_channels_statistics;
set $push_stream_channel_id $arg_id;
}
# /channels_stats -> get statistics about all channels in a summarized way
# /channels_stats?id=ALL -> get statistics about all channels in a detailed way
# /channels_stats?id=channel_id -> get statistics about a channel
context: location
Defines a location as a message publisher. Requests to a publisher location are treated as messages to be sent to subscribers.
This location supports two http methods:
GET, make possible to get statistics about the channel
POST, publish a message to the channel
location /pub {
push_stream_publisher;
set $push_stream_channel_id $arg_id;
}
# GET /pub?id=channel_id -> get statistics about a channel
# POST /pub?id=channel_id -> publish a message to the channel
context: location
Defines a location as a subscriber. This location represents a subscriber’s interface to a channel’s message queue.
This location only supports GET http method to receive published messages in a stream.
default: -
context: location
location: push_stream_subscriber
The text that will be sended to subscribers when they arrive.
default: -
context: location
location: (push_stream_subscriber and push_stream_publisher) or main nginx configuration
The text template that will be used to format the message before be sended to subscribers. The template can contain any number of the reserved words: ~id~, ~text~ and ~channel~, example: ""
default: text/plain
context: location
location: push_stream_subscriber
The content type used on responses to subscribers. Must be complient with push_stream_header_template and push_stream_message_template.
default: -
context: location
location: push_stream_subscriber
The time interval in which a keepalive message is sent to subscribers.
default: 32M
context: http
The size of the memory chunk this module will use to store published messages, channels and other shared structures.
When this memory is full any new request for publish a message or subscribe a channel will receive an 500 Internal Server Error response.
default: off
context: location
location: push_stream_publisher
Whether or not message queuing is enabled.
If store messages is “On” is needed to set at least one of these two directives push_stream_min_message_buffer_timeout or push_stream_max_message_buffer_length.
default: -
context: location
location: push_stream_publisher
The length of time a message may be queued before it is considered expired. If you do not want messages to expire, just not set this directive.
default: -
context: location
location: push_stream_publisher
The maximum number of messages to store per channel. A channel’s message buffer will retain at most this many most recent messages. If you do not want messages to be discarded by length, just not set this directive.
default: off
context: location
location: push_stream_subscriber
Whether or not a subscriber may create a channel by making a request to a push_stream_subscriber location. If set to on, a publisher must send a POST request before a subscriber can request messages on the channel. Otherwise, all subscriber requests to nonexistent channels will get a 403 Forbidden response.
This restriction is not applied to broadcast channels, but to subscribe to a broadcast channel is necessary to subscribe at least to one normal channel, and if this directive is set to on this channel has to be created before.
default: -
context: location
location: push_stream_subscriber
The length of time a subscriber will stay connected before it is considered expired and disconnected. If you do not want subscribers to be automatically disconnected, just not set this directive.
But, this operation is very important to help Nginx recycle memory consumed to send messages to susbscriber, allocated at pool request.
default: -
context: location
location: (push_stream_subscriber and push_stream_publisher) or main nginx configuration
Maximum permissible channel id length (number of characters). Longer ids will receive an 400 Bad Request response.
default: -
context: location
location: (push_stream_subscriber and push_stream_publisher) or main nginx configuration
The string prefix used to identify when a channel is as normal or broadcast channel, example: when you set this directive as “bd_”, “bd_ch1” will be a broadcast channel
default: -
context: location
location: (push_stream_subscriber and push_stream_publisher) or main nginx configuration
The maximum number of broadcast channels that a subscriber may sign on the request.
This directive works in conjunction with push_stream_authorized_channels_only to preserve the server from a kind of attack where a subscriber sign one normal channel and many nonexistent broadcast channels.
default: -
context: location
location: (push_stream_subscriber and push_stream_publisher) or main nginx configuration
The maximum number of concurrent channels on the server. If you do not want to limit the number of channels, just not set this directive.
default: -
context: location
location: (push_stream_subscriber and push_stream_publisher) or main nginx configuration
The maximum number of concurrent broadcats channels on the server. If you do not want to limit the number of broadcast channels, just not set this directive.
default: 30 seconds
context: location
location: (push_stream_subscriber and push_stream_publisher) or main nginx configuration
The length of time a message or a channel will stay on garbage collection area before it is completly discarded, freeing the shared memory. The minimum length is 30 seconds to ensure that no one is using these elements.
This operation is very important to help Nginx recycle memory consumed to create messages and channels, so do not use a large time.
This module controls everything needed to send the messages to subscribers.
So is strongly recommended to you disable Nginx’s chuncked filter to reduce memory consumption.
You can do this using the chunked_transfer_encoding directive on subscriber location
# solving some leakage problem with persistent connections in
# Nginx's chunked filter (ngx_http_chunked_filter_module.c)
chunked_transfer_encoding off;