Region-based Dynamic Frequency Scaling Toolset

• gcc 5.1.1 or above
• libcpupower or libcpufreq

Default options should be enough. If you want to hack and understand what you are doing, start customization in files src/libredfst/CMakeLists.txt and src/redfst/CMakeLists.txt. This should get improved in the future, with user options available in the command-line or through ccmake.

mkdir build; cd build; cmake ..; make;
sudo make install

ReDFST uses Intel RAPL to measure the energy consumption of the package, power plane 0 and DRAM of the processors. Besides the hardware support, which is not available on all Intel processor, it requires the msr kernel module, which can be loaded with:

sudo modprobe msr

You will also need read/write permissions for /dev/cpu/*/msr. These permissions can be obtained in two ways:

• by running your application as root

sudo ./application

• by giving your application the necessary access

sudo chmod 666 /dev/cpu/*/msr
sudo setcap cap_sys_rawio=eip ./application
./application

The CPUs which will have their energy measured are given in the REDFST_CPUS environment variable. This is a comma-separated list of:

• number
• number:number

Example: 0,3:5,7 will measure the energy of cpus 0, 3, 4, 5 and 7.

Alternatively, the list of CPUs can be set by calling redfst_cpus:

int cpus[] = {0,3,4,5,7,-1}; // terminate the list with -1
if(!redfst_cpus(cpus))
  printf("failed to read energy of cpus\n");

The functions used to measure energy consumption are:

int redfst_support()

Returns the type of energy measuring support available. Possible values: REDFST_NONE, REDFST_POWERCAP, REDFST_MSR, REDFST_LIKWID.

int redfst_cpus(const int *cpus)

Sets which cpus will have their energy measured. The array should end with -1.

void redfst_reset()

Zeroes the energy counters.

void redfst_print()

Output the current value of the counters.

void redfst_printf(...)

Calls printf with the given arguments then outputs the current value of the counters.

redfst_dev_t * redfst_dev_init(redfst_dev_t *d)

Initializes the passed device structure, which is used to store energy data. If the passed value is NULL, a new device will be allocated and it’s the caller’s responsibility to free it. After use, redfst_dev_destroy should be called.

void redfst_dev_destroy(redfst_dev_t *dev)

Frees internal data allocated by redfst_dev_init.

redfst_dev_t * redfst_get(redfst_dev_t *dev)

Write the current value of the energy counters into dev. If dev is NULL, a pointer to an internal structure will be returned and this pointer should not be freed. If dev is non-NULL, it must first be initialized by calling redfst_dev_init, and before being freed it must be passed to redfst_dev_destroy. redfst_dev_t is a structure with the following members:

char **name

Array with the names of the energy counters. Example: {“cpu.0.pkg”, “cpu.0.pp0”, “cpu.0.dram”, “cpu.1.pkg”, “cpu.1.pp0”, “cpu.1.dram”}.

float *energy

Array of energy counters, in Joules.

double time

Seconds passed since last call to redfst_reset.

int count

Number of energy counters.

Sample code:

#include <stdlib.h>
#include <unistd.h>
#include <redfst.h>

int main(int argc, char **argv){
  int n;
  int x;
  int i;
  int j;

  n = argc > 1 ? atoi(argv[1]) : 8*1024;

  redfst_reset(); // zero energy counters to skip initialization code
#pragma omp parallel for reduction(+:x)
  for(i = 0; i < n; ++i){
    for(j = 0; j < n; ++j){
      x ^= i^j;
    }
  }
  redfst_print(); // print energy counters
  
  redfst_reset(); // reset energy counters so we can measure something else
  usleep(3000000);
  redfst_print(); // print energy counters

  return x;
}

Compile with

gcc -o example1 example1.c -fopenmp -lredfst -lcpupower -lm

or

gcc -o example1 example1.c -fopenmp -lredfst -lcpufreq -lm

Attention: If the library was not installed to /lib or /usr/lib, you have to pass -Wl,-rpath=/INSTALLPATH/lib to the compiler or run the example as root.

Run it:

• as root

sudo ./example1

or

• as your user:

First set the permissions

sudo setcap cap_sys_rawio=pei example1

Run it:

./example1

The output should be similar to:

51.217300, 38.908722, 2.748489, 44.485428, 32.220169, 2.158203, 95.702728, 71.128891, 4.906693, 0.868234
35.392227, 5.524429, 5.437225, 36.257050, 6.249054, 5.364868, 71.649277, 11.773483, 10.802094, 3.000128

To print a header explaining what each column means, set the variable REDFST_HEADER:

REDFST_HEADER=1 ./example1

pkg.0, pp0.0, dram.0, pkg.1, pp0.1, dram.1, pkg, pp0, dram, time
51.217300, 38.908722, 2.748489, 44.485428, 32.220169, 2.158203, 95.702728, 71.128891, 4.906693, 0.868234
35.392227, 5.524429, 5.437225, 36.257050, 6.249054, 5.364868, 71.649277, 11.773483, 10.802094, 3.000128

In the output above, the first row of numbers comes from the first redfst_print and represents the energy spent on the first loop. The second row of numbers, printed by the second call to redfst_print, displays the energy spent on the sleep function.

The columns labeled pkg.0, pp0.0 and dram.0 show, respectively, the energy consumed by the package, power plane 0 and DRAM of CPU 0. Likewise, pkg.1, pp0.1 and dram.1 are the energy of CPU 1. The columns labeled pkg, pp0 and dram are, respectively, the sum of the package, power plane 0 and DRAM of all CPUs. The last column, time, contains the number of seconds passed since the last call to redfst_reset.

#include <stdlib.h>
#include <unistd.h>
#include <redfst.h>

int main(int argc, char **argv){
  int n;
  int x;
  int i;
  int j;

  n = argc > 1 ? atoi(argv[1]) : 8*1024;

  redfst_reset(); // clear the energy counters to skip initialization
#pragma omp parallel for reduction(+:x)
  for(i = 0; i < n; ++i){
    redfst_region(1); // set the code region of the current thread to 1
    for(j = 0; j < n; ++j){
      x ^= i^j;
    }
  }
  redfst_print(); // print the energy counters
  
  redfst_reset(); // reset the energy counters
  redfst_region_all(2); // set the code region of all threads to 2
  usleep(3000000);
  redfst_print(); // print the energy counters
  
  return x;
}

Compile with:

gcc -o example2 example2.c -fopenmp -lredfst -lcpupower -lm

or

gcc -o example2 example2.c -fopenmp -lredfst -lcpufreq -lm

Let’s run the code region 1 at 2.0GHz frequency and the code region 2 at the 1.5GHz:

export REDFST_HIGH=2000000
export REDFST_LOW=1500000
export REDFST_FASTREGIONS=1
export REDFST_SLOWREGIONS=2
export OMP_PROC_BIND=TRUE
./example2

The frequency for the fast regions is given by the variable REDFST_HIGH. Likewise, the frequency to be used in the slow regions is given by the variable REDFST_LOW. The set of fast (slow) regions is given by REDFST_FASTREGIONS (REDFST_SLOWREGIONS) as a list in the format region[,region…] (example: 1,3,4). The use of OMP_PROC_BIND to bind the threads to cores is obligatory.

For ReDFST to change the frequency it requires permissions and the use of the userspace governor. You can see which governor you’re using with:

cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

or

cpufreq-info

or

cpupower frequency-info

The governor can be changed with:

echo userspace | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

or, for every CPU,

sudo cpufreq-set -c CPU -g userspace

or

sudo cpupower frequency-set -c all -g userspace

Besides using the userspace governor you also need read/write permissions to /sys/devices/system/cpu/cpu*/cpufreq/scaling_setspeed.

int redfst_support()

Returns the type of energy measuring support available. Possible values: REDFST_NONE, REDFST_POWERCAP, REDFST_MSR, REDFST_LIKWID.

int redfst_cpus(const int *cpus)

Sets which cpus will have their energy measured. The array cpus should end with -1.

void redfst_reset()

Reset the energy counters.

void redfst_print()

Print the energy counters.

void redfst_printf(...)

Print the given arguments like printf, then print the energy counters. Example: redfst_print(“%s,”,”here”) will output “here,cpuX.pkg,cpuX.pp0,cpuX.dram,…,total.pkg,total.pp0,total.dram,time”

redfst_dev_t * redfst_dev_init(redfst_dev_t *d)

Initializes the passed device structure, which is used to store energy data. If the passed value is NULL, a new device will be allocated and it’s the caller’s responsibility to free it. After use, redfst_dev_destroy should be called.

void redfst_dev_destroy(redfst_dev_t *dev)

Frees internal data allocated by redfst_dev_init.

redfst_dev_t * redfst_get(redfst_dev_t *dev)

Write the current value of the energy counters into dev. If dev is NULL, a pointer to an internal structure will be returned and this pointer should not be freed. If dev is non-NULL, it must first be initialized by calling redfst_dev_init, and before being freed it must be passed to redfst_dev_destroy. redfst_dev_t is a structure with the following members:

char \*\*name

Array with the names of the energy counters. Example: {“cpu.0.pkg”, “cpu.0.pp0”, “cpu.0.dram”, “cpu.1.pkg”, “cpu.1.pp0”, “cpu.1.dram”}.

float \*energy

Array of energy counters, in Joules.

double time

Seconds passed since last call to redfst_reset.

int count

Number of energy counters.

void redfst_region(int id)

Set the code region of the calling core to “id”. May trigger a frequency change.

void redfst_region_all(int id)

Set the code region of all cores to “id”. May trigger a frequency change on all cores.

void redfst_monitor_set_status(int n)

Sets the value of a variable used by the execution monitor, which is explained in the Environment Variables section

The following variables are accepted by libredfst. Besides them, OMP_PROC_BIND must be set to true.

REDFST_CPUS

List of CPUs that will have their energy consumption measured

REDFST_OUT

Energy measurements are output to this file

REDFST_HEADER

If defined, write a CSV header to REDFST_OUT

REDFST_LOW

Frequency to use on slow regions. Example: 1500000 (for 1.5GHz)

REDFST_HIGH

Frequency for the fast regions. Example: 2000000 (for 2.0GHz)

REDFST_SLOWREGIONS

Set of regions which will trigger the low frequency. The regions are given as a comma-separated list. Example: to use the regions 1, 3 and 4, this variable should be set to 1,3,4. The 63th region is entered when the program finishes. The other regions are user-defined.

REDFST_FASTREGIONS

Regions which will trigger the high frequency. See REDFST_SLOWREGIONS.

REDFST_MONITOR

If defined to anything other than “0”, “F” or “f”, dumps some execution information to the file “monitor.csv”. This file contains a table with the following columns:

time

When the data was measured. This is sort of a guess, and depends on usleep not screwing up too much. If you want something less imprecise you’ll have to change the function redfst_monitor_loop on the file monitor.c.

status

Integer you can manually set by calling the function redfst_monitor_set_status(int n). You can, for instance, set it to 0 on the portions of the code you’re not interested in.

freqN

Frequency of the N-th core.

regionN

Current region of the N-th thread, which corresponds to the n-th core.

events

Hardware counters. The defaults are last level cache (LLC) references, LLC misses, cycles, instructions.

REDFST_TRACEFILE

If the library was compiled with tracing enabled, the trace will be written to this file. If undefined, the trace is output to the file descriptor 4, or otherwise to stderr.