PSPACER RELEASE NOTES
=====================
Version 2.2.1

								2008-02-10

CREDIT
------

This software was developed by the National Institute of Advanced 
Industrial Science and Technology (AIST), Japan. A part of the development 
was supported by a grant from the Ministry of Education, Sports, Culture, 
Science and Technology (MEXT) of Japan through the NAREGI (National 
Research Grid Initiative) Project.
PSPACER IS LICENSED UNDER THE GNU GPL.


WHAT IS PSPACER?
----------------
PSPacer (Precise Software Pacer) is a qdisc module which realizes 
precise transmission bandwidth control. It makes bursty traffic which is 
often generated by TCP smooth without any special hardware.

Bursty traffic can degrade the communication performance, because it 
causes buffer overflow at intermediate network nodes and results in 
packet losses. In a bursty traffic, packets are sent back to back. By 
adding a short pause in between the packets, traffic bursts can be 
avoided.

PSPacer controls the interval between outgoing packets very precisely. 
The key idea of PSPacer is to determine transmission timing of packets 
by the number of bytes transferred. If packets are transferred back to 
back, the timing a packet is sent can be determined by the number of 
bytes sent before the packet. PSPacer fills the gaps between time 
aligned "real packets" (the packets which are sent by user program) by 
"gap packets". The real packets and gap packets are sent back to back, 
and thus the timing of transmission of each real packet can be precisely 
controlled by adjusting the gap packet size. As the gap packets, the IEEE 
802.3x PAUSE frames are used. PAUSE frames are discarded at a switch 
input port, and only real packets go through the switch keeping the 
original intervals. 

In the past, some software-based pacing schemes have been proposed. 
These schemes use timer interrupt based packet transmission timing control. 
Therefore, to achieve precise pacing, they require the operating system 
to maintain a high resolution timer, which could incur a large overhead.

For detailed description and the usage of PSPacer, please refer to 
our project page (http://www.gridmpi.org/pspacer.jsp), and the paper
"Design and Evaluation of Precise Software Pacing Mechanisms for Fast 
Long-Distance Networks," in PFLDnet2005.

NOTE: PSPacer version 2.x does not support the Linux kernel 2.4!
If you want to run PSPacer on the Linux kernel 2.4, you should use
PSPacer version 1.x.

......................................................................
LICENSE		- The GPL license.
Makefile	- Makefile.
README		- This file.
README.ja	- This file (in Japanese).
ChangeLog	- The ChangeLog.
doc/		- Miscellaneous documents (user guide, paper).
kernel/		- The kernel module.
man/		- The man pages.
scripts/	- Some sample setup/test scripts.
tc/		- The shared library for tc (8).
pspd/		- The PSPacer control daemon.
......................................................................


LIMITATIONS
-----------
(1) PSPacer controls the bandwidth according to the ratio of the target
bandwidth in the maximum transmission bandwidth of the system.
Therefore, the system (computer, network interface, operating system, 
buffer settings, etc.) should have a capability to transmit packets at 
the maximum transmission rate (i.e. 1 Gbps for 1000BASE, 100 
Mbps for 100BASE) to realize a precise pacing.

For example, if a 33MHz/32bit PCI bus is used to connect a Gigabit 
Ethernet network interface, the bottleneck is the PCI bus, and the 
system can not transmit packets at the rate of gigabit/sec. In this case, 
you can explicitly specify the maximum transmission bandwidth using
qdisc's "rate" parameter as follows:

	# /sbin/tc qdisc add ... psp ... rate 700mbit
	                                 ^^^^^^^^^^^^

However, in this case, the output traffic may be imprecise since the behavior
of PCI bus is not stable enough.

Therefore, if you want to control Gigabit Ethernet traffic, we recommend 
to use PCI-X, 66MHz/64bit PCI or CSA connected network interface. If 
the total of target bandwidth of the output streams is less than 100Mbps,
you can set the network interface to use 100BASE mode so as to obtain
precise pacing. For the same reason, avoid using a shared switch (dumb hub)
for the edge switch to which the PC with PSPacer is connected.

(2) PSPacer uses the IEEE 802.3x PAUSE frame as the gap between packets. 
Therefore, you can not use the PAUSE frame to stop transmission from the 
switch/router to the PC. Since PSPacer generates PAUSE frames with zero 
pause time, there should not be any side effects other than you can not 
stop transmission from the switch. However, it is recommended to disable 
IEEE 802.3x flow control function of the switch (to which a PC with 
PSPacer is connected) in order to avoid unexpected behavior.

(3) PSPacer does not support TCP Segmentation Offloading (TSO). You have 
to disable TSO by using the ethtool command (ethtool -K eth0 tso off).

(4) PSPacer can only work as a root qdisc since it produces (gap) packets
itself and thereby violates the rule that a qdisc can only hand out packets
that were enqueued to it. Using it as a leaf qdisc causes that qlen counters
become inconsistent between itself and the upper qdiscs.


COMPILATION AND INSTALLATION
----------------------------
Installation from the source tallball
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The following source codes are required for the compilation:

	- Linux kernel 2.6.x
	- iproute2

NOTE: Since the iproute2 package is tightly coupled with the kernel, 
you must use a proper combination. see TESTED ENVIRONMENT for more 
information.

1. Unpack the iproute2 tarball:

	$ scripts/dl_iproute2.py
	Success: iproute2-<version>.tar.gz
	$ tar zxvf iproute2-<version>.tar.gz -C /opt
	$ ln -s /opt/iproute2-<version> /opt/iproute2
or
	$ tc -V  # check the iproute2 version.
	tc utility, iproute2-<version>
	$ cd /opt
	$ wget http://devresources.linux-foundation.org/dev/iproute2/download/
	  iproute2-<version>.tar.gz
	$ tar zxvf iproute2-<version>.tar.gz
	$ ln -s iproute2-<version> iproute2
	
2. Unpack the PSPacer tarball:

	$ tar zxvf pspacer-<version>.tar.gz

3. Do make and install:

	$ cd pspacer-<version>
	$ ./configure
	$ make
	$ su
	# make install

Installation from the binary package
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You can also build a RedHat RPM or Debian deb package from the tarball.
See the PSPacer User Guide (doc/usage.en.txt).


SIMPLE USAGE
------------
You check whether PSPacer works properly by using iperf as follows:

1. Setup

	(add the PSPacer qdisc as the root qdisc)
	# /sbin/tc qdisc add dev eth0 root handle 1: psp default 1
	(add the PSPacer class whose target rate is 500Mbps)
	# /sbin/tc class add dev eth0 parent 1: classid 1:1 psp rate 500mbit
	(add the PFIFO qdisc as the sub qdisc)
	# /sbin/tc qdisc add dev eth0 parent 1:1 handle 10: pfifo

2. Run iperf

	$ iperf -c <server host> -i 10 -t 60
	------------------------------------------------------------
	Client connecting to XXXX, TCP port 5122
	TCP window size: 16.0 KByte (default)
	------------------------------------------------------------
	[  3] local 192.168.1.1 port 46457 connected with 192.168.1.2 port 5122
	[  3]  0.0-10.0 sec    567 MBytes    476 Mbits/sec
	[  3] 10.0-20.0 sec    567 MBytes    476 Mbits/sec
		:

3. Cleanup

	(remove the PFIFO sub qdisc)
	# /sbin/tc qdisc del dev eth0 parent 1:1 handle 10:
	(remove the PSPacer class)
	# /sbin/tc class del dev eth0 parent 1: classid 1:1
	(remove the PSPacer qdisc)
	# /sbin/tc qdisc del dev eth0 root handle 1:
	(remove the PSPacer module)
	# /sbin/rmmod sch_psp

For more information, see the PSPacer Usage Guide (doc/user.en.txt).


TESTED ENVIRONMENT
------------------
The software is tested on the following environments:

- Linux distribution:
	* FedoraCore 3 (kernel 2.6.9 + iproute ss040831)
	* FedoraCore 3 (kernel 2.6.11 + iproute ss050330)
	* FedoraCore 3 (kernel 2.6.13 + iproute ss040831)
	* FedoraCore 4 (kernel 2.6.11 + iproute ss050314)
	* FedoraCore 5 (kernel 2.6.15-1.2054_FC5 + iproute ss060110)
	* FedoraCore 5 (kernel 2.6.20 + iproute ss060110)
	* SUSE SLES 9 AMD64 (kernel 2.6.5 + iproute ss020116)
	* Debian GNU/Linux 3.1 (kernel 2.6.8 + iproute ss041019)
	* Debian GNU/Linux sid (kernel 2.6.12.2 + iproute ss041019)

- Network interface card:
	* Intel PRO/1000 (PCI-X, CSA)
	* Intel PRO/100
	* Broadcom BCM5704 (PCI-X)