#R Benchmark Suite

R Benchmark Suite is a collection of benchmarks for R programming language as well as a benchmarking environment for measuring the performance of different R VM implementations.

The benchmark type is defined as

Example

#R-benchmark-25: creation of Toeplitz matrix
for (j in 1:500) {
    for (k in 1:500) {
        jk<-j - k;
        b[k,j] <- abs(jk) + 1
    }
}

Example

#Riposte benchmark: age and gender are large vectors
males_over_40 <- function(age, gender) {
    age >= 40 & gender == 1
}

Example

#R-benchmark-25:  FFT over 2.4Mill random values
a <- rnorm(2400000);
b <- fft(a)

The driver is rbench.py under utility directory. You can use "-h" to get the help.

$ rbench.py -h
usage: rbench.py [-h] [--meter {time,perf,system.time}]
                 [--rvm {R,R-bytecode,rbase2.4,...}]
                 [--warmup_rep WARMUP_REP] [--bench_rep BENCH_REP]
                 source [args [args ...]]
...

Note: on Windows platform, you may use "python rbench.py -h"

Do a simple benchmark

$ cd examples
$ ../utility/rbench.py hello_rbenchmark.R

It will use the default R VM (R-bytecode) and the default meter to benchmark the application. The output of benchmark application will be thrown away (redirect to "/dev/null"), and the timing result will be recorded in "rbench.csv" file.

The default benchmark method has two phases

• pure warmup: run run() 2 times
• warmup + benchmark: run run() 2 + 5 times

Then the post processing will diff the two phases, and reports the average value for the 5 benchmark iterations.

You can use command line to do more controls

$ cd examples
$ ../utility/rbench.py --meter perf --rvm R --bench_log stdout hello_rbenchmark.R 1000

Then it will use Linux perf (only on Linux Platform) for the data measuring, choose the R (without byte-code compiler) as the VM for benchmarking, and dump the benchmark's output to the standard output.

You can run benchmark for all .R files in a directory, or run benchmarks defined in a .list file.

Please refer Running Benchmark for additional controls of running a benchmark. And the driver supports many RVMs for benchmarking. Here is the list.

A benchmark R program should have a mandatory run() function. The driver will call run() function in the benchmarking.

#hello_rbenchmark.R
run <- function () {
    print("Executing hello_rbenchmark run()")
}

The benchmark R program could have an optional setup() function. The driver will call setup() first, then use the return value of the setup() to call the run().

#hello_rbenchmark.R

setup <- function(cmdline_args=character(0)) {
   return(cmdline_args)
}

run <- function (input) { # input = setup(cmdline_args)
    print("Executing hello_rbenchmark run() with input")
    print(input)
}

Please refer Writing Benchmark for additional controls of the benchmark program

FAQ

The R-benchmark-25 benchmark is ported from http://r.research.att.com/benchmarks/.

The R version shootout benchmark is ported from UIUC ORBIT project and Purdue FastR project (https://github.com/allr/fastr).

The Riposte benchmark is ported from Riposte project (https://github.com/jtalbot/riposte/)

Please contact Haichuan Wang (hwang154@illinois.edu) and Arun Chauhan(arunchauhan@google.com) for any questions and suggestions.