pdbg is a simple application to allow debugging of the host POWER processors from the BMC. It works in a similar way to JTAG programmers for embedded system development in that it allows you to access GPRs, SPRs and system memory.
A remote gdb sever is under development to allow integration with standard debugging tools.
The output of autoconf is not included in the git tree so it needs to be
generated using autoreconf. This can be done by running ./bootstrap.sh in the
top level directory. Static linking is supported and can be performed by adding
CFLAGS=-static to the command line passed to configure.
First, work out if your BMC is using the hard or soft float ABI. If the file /lib/ld-linux.so.3 exists, soft-float. If /lib/ld-linux-armhf.so.3 exists, hard-float.
To build for soft-float:
apt-get install gcc-arm-linux-gnueabi
./bootstrap.sh
./configure --host=arm-linux-gnueabi CFLAGS="-static"
make
rsync pdbg root@bmc:/usr/local/bin
For hard-float:
apt-get install gcc-arm-linux-gnueabihf
./bootstrap.sh
./configure --host=arm-linux-gnueabihf CFLAGS="-static"
make
rsync pdbg root@bmc:/usr/local/bin
Some generated files are shipped with the source to reduce the toolchain
requirement for builds. Currently src/gdb_parser.rl creates
src/gdb_parser_precompile.c so that needs to be updated if the .rl file is
changed. make generated to update such generated files.
There is a test suite to perform some basic testing. The tests on the host are mostly unit tests that exercise the device tree based targetting code.
make check
To test on the bmc, make a build in a separate directory:
cd pdbg
mkdir obj-arm
cd obj-arm
../configure --host=arm-openbmc-linux-gnueabi
make -j
And create a configuration file .test.pdbg in the pdbg source directory:
BMC_HOST=rain71bmc.aus.stglabs.ibm.com
BMC_USER=root
BMC_PASS=passw0rd
PDBG_ARM_BUILD=obj-arm
The BMC must have the host powered on and be accessable via ssh.
Then run make check. It will test both the host unit tests, and copy the arm binary to the BMC and run it there:
-- BMC HW tests
sending incremental file list
created directory /tmp/pdbg
...
Checking if the host is up... yes
SKIP: /tmp/pdbg/pdbg -p0 getcfam 0xc09
SKIP: /tmp/pdbg/pdbg -p0 getscom 0xf000f
SKIP: /tmp/pdbg/pdbg -p0 putmem 0x31000000
SKIP: /tmp/pdbg/pdbg -p0 getmem --raw 0x31000000 0x8
Several backends are supported depending on which system you are using and are
selected using the -b option:
POWER8 Backends:
- i2c (default): Uses an i2c connection between BMC and host processor
- fsi: Uses a bit-banging GPIO backend which accesses BMC registers directly via
/dev/mem/. Requires
-d p8to specify you are running on a POWER8 system.
POWER9 Backends:
- kernel (default): Uses the in kernel OpenFSI driver provided by OpenBMC
- fsi: Uses a bit-banging GPIO backend which accesses BMC registers directly via
/dev/mem. Requiers
-d p9w/p9r/p9zas appropriate for the system. - sbefifo: Uses the in kernel OpenFSI & SBEFIFO drivers provided by OpenBMC
When using the fsi backend POWER8 AMI based BMC's must first be put into debug mode to allow access to the relevant GPIOs:
ipmitool -H <host> -U <username> -P <password> raw 0x3a 0x01
On POWER9 when using the fsi backend it is also a good idea to put the BMC into debug mode to prevent conflicts with the OpenFSI driver. On the BMC run:
systemctl start fsi-disable.service && systemctl stop host-failure-reboots@0.service
Usage is straight forward. Note that if the binary is not statically linked all commands need to be prefixed with LD_LIBRARY_PATH= in addition to the arguments for selecting a backend.
pdbg has commands that operate on specific hardware unit(s) inside the POWER processor. To select appropriate hardware unit (commonly referred as target), pdbg provides two different mechanisms.
Many commands typically operate on hardware thread(s) or CPU(s) as identified by Linux.
- all threads (
-a) - core 0 of processor 0 (
-p0 -c0) - all threads on processor 0 (
-p0 -a) - all threads on core 1 of processor 0 (
-p0 -c1 -a) - thread 2 on core 1 of processor 0 (
-p0 -c1 -t2) - thread 0 on all cores of processor 0 (
-p0 -t0 -a) - threads 1,2,3,4 on cores 1,3,5 of processor 1 (
-p1 -c1,3,5 -t1-4) - CPUs 15 and 17 as identified by Linux (
-l15,17)
Note: -l option is only available when running pdbg on the host.
To select any target in a device tree, it can be specified using -P.
The -P option takes path specification as an argument. This path specification
is constructed using the class names of targets present in a device tree.
Some of the targets currently available for selection are:
pibcorethreadadufsichiplet
Path specification can be either an individual target or a path constructed using more than one targets.
- all threads (
-P thread) - core 0 of processor 0 (
-P pib0/core0) - all threads on processor 0 (
-P pib0/thread) - all threads on core 1 of processor 0 (
-P pib0/core1/thread) - thread 2 on core 1 of processor 0 (
-P pib0/core1/thread2) - thread 0 on all cores of processor 0 (
-P pib0/thread0) - threads 1,2,3,4 on cores 1,3,5 of processor 1 (
-P pib1/core[1,3,5]/thread[1-4]) - chiplet at address 21000000 (-P
chiplet@21000000) - all adus (
-P adu) - First FSI (
-P fsi0)
$ ./pdbg --help
Usage: ./pdbg [options] command ...
Options:
-p, --processor=processor-id
-c, --chip=chiplet-id
-t, --thread=thread
-a, --all
Run command on all possible processors/chips/threads (default)
-b, --backend=backend
fsi: An experimental backend that uses
bit-banging to access the host processor
via the FSI bus.
i2c: The P8 only backend which goes via I2C.
kernel: The default backend which goes the kernel FSI driver.
-d, --device=backend device
For I2C the device node used by the backend to access the bus.
For FSI the system board type, one of p8 or p9w
Defaults to /dev/i2c4 for I2C
-s, --slave-address=backend device address
Device slave address to use for the backend. Not used by FSI
and defaults to 0x50 for I2C
-V, --version
-h, --help
Commands:
getcfam <address>
putcfam <address> <value> [<mask>]
getscom <address>
putscom <address> <value> [<mask>]
getmem <address> <count>
putmem <address>
getvmem <virtual address>
getgpr <gpr>
putgpr <gpr> <value>
getnia
putnia <value>
getspr <spr>
putspr <spr> <value>
start
step <count>
stop
threadstatus
probe
$ ./pdbg -a probe
proc0: Processor Module
fsi0: Kernel based FSI master (*)
pib0: Kernel based FSI SCOM (*)
chiplet16: POWER9 Chiplet
eq0: POWER9 eq
ex0: POWER9 ex
chiplet32: POWER9 Chiplet
core0: POWER9 Core (*)
thread0: POWER9 Thread (*)
thread1: POWER9 Thread (*)
thread2: POWER9 Thread (*)
thread3: POWER9 Thread (*)
chiplet33: POWER9 Chiplet
core1: POWER9 Core (*)
thread0: POWER9 Thread (*)
thread1: POWER9 Thread (*)
thread2: POWER9 Thread (*)
thread3: POWER9 Thread (*)
ex1: POWER9 ex
chiplet34: POWER9 Chiplet
core2: POWER9 Core (*)
thread0: POWER9 Thread (*)
thread1: POWER9 Thread (*)
thread2: POWER9 Thread (*)
thread3: POWER9 Thread (*)
chiplet35: POWER9 Chiplet
core3: POWER9 Core (*)
thread0: POWER9 Thread (*)
thread1: POWER9 Thread (*)
thread2: POWER9 Thread (*)
thread3: POWER9 Thread (*)
chiplet17: POWER9 Chiplet
eq1: POWER9 eq
ex0: POWER9 ex
chiplet36: POWER9 Chiplet
core4: POWER9 Core (*)
thread0: POWER9 Thread (*)
thread1: POWER9 Thread (*)
thread2: POWER9 Thread (*)
thread3: POWER9 Thread (*)
chiplet37: POWER9 Chiplet
core5: POWER9 Core (*)
thread0: POWER9 Thread (*)
thread1: POWER9 Thread (*)
thread2: POWER9 Thread (*)
thread3: POWER9 Thread (*)
ex1: POWER9 ex
chiplet18: POWER9 Chiplet
eq2: POWER9 eq
ex0: POWER9 ex
chiplet40: POWER9 Chiplet
core8: POWER9 Core (*)
thread0: POWER9 Thread (*)
thread1: POWER9 Thread (*)
thread2: POWER9 Thread (*)
thread3: POWER9 Thread (*)
chiplet41: POWER9 Chiplet
core9: POWER9 Core (*)
thread0: POWER9 Thread (*)
thread1: POWER9 Thread (*)
thread2: POWER9 Thread (*)
thread3: POWER9 Thread (*)
ex1: POWER9 ex
chiplet19: POWER9 Chiplet
eq3: POWER9 eq
ex0: POWER9 ex
chiplet44: POWER9 Chiplet
core12: POWER9 Core (*)
thread0: POWER9 Thread (*)
thread1: POWER9 Thread (*)
thread2: POWER9 Thread (*)
thread3: POWER9 Thread (*)
chiplet45: POWER9 Chiplet
core13: POWER9 Core (*)
thread0: POWER9 Thread (*)
thread1: POWER9 Thread (*)
thread2: POWER9 Thread (*)
thread3: POWER9 Thread (*)
ex1: POWER9 ex
chiplet46: POWER9 Chiplet
core14: POWER9 Core (*)
thread0: POWER9 Thread (*)
thread1: POWER9 Thread (*)
thread2: POWER9 Thread (*)
thread3: POWER9 Thread (*)
chiplet47: POWER9 Chiplet
core15: POWER9 Core (*)
thread0: POWER9 Thread (*)
thread1: POWER9 Thread (*)
thread2: POWER9 Thread (*)
thread3: POWER9 Thread (*)
chiplet20: POWER9 Chiplet
eq4: POWER9 eq
ex0: POWER9 ex
chiplet48: POWER9 Chiplet
core16: POWER9 Core (*)
thread0: POWER9 Thread (*)
thread1: POWER9 Thread (*)
thread2: POWER9 Thread (*)
thread3: POWER9 Thread (*)
chiplet49: POWER9 Chiplet
core17: POWER9 Core (*)
thread0: POWER9 Thread (*)
thread1: POWER9 Thread (*)
thread2: POWER9 Thread (*)
thread3: POWER9 Thread (*)
ex1: POWER9 ex
chiplet21: POWER9 Chiplet
eq5: POWER9 eq
ex0: POWER9 ex
ex1: POWER9 ex
chiplet54: POWER9 Chiplet
core22: POWER9 Core (*)
thread0: POWER9 Thread (*)
thread1: POWER9 Thread (*)
thread2: POWER9 Thread (*)
thread3: POWER9 Thread (*)
chiplet55: POWER9 Chiplet
core23: POWER9 Core (*)
thread0: POWER9 Thread (*)
thread1: POWER9 Thread (*)
thread2: POWER9 Thread (*)
thread3: POWER9 Thread (*)
proc1: Processor Module
proc2: Processor Module
proc3: Processor Module
proc4: Processor Module
proc5: Processor Module
proc6: Processor Module
proc7: Processor Module
Note that only selected targets (marked with *) and targets in the
hierarchy of the selected targets will be shown above. If none are shown
try adding '-a' to select all targets.
Core-IDs are core/chip numbers which should be passed as arguments to -c
when performing operations such as getgpr that operate on particular cores.
Processor-IDs should be passed as arguments to -p to operate on different
processor chips. Specifying no targets is an error and will result in the
following error message:
Note that only selected targets will be shown above. If none are shown
try adding '-a' to select all targets
If the above error occurs even though targets were specified it means the specified targets were not found when probing the system.
$ ./pdbg -P pib getscom 0xf000f
p0: 0x00000000000f000f = 0x222d104900008040 (/proc0/pib)
p1: 0x00000000000f000f = 0x222d104900008040 (/proc1/pib)
$ ./pdbg -P pib1 putscom 0x8013c02 0x0
$ ./pdbg -a threadstatus
p0t: 0 1 2 3
c22: A A A A
c21: A A A A
c20: A A A A
c19: A A A A
c15: A A A A
c14: A A A A
c07: A A A A
c05: A A A A
p1t: 0 1 2 3
c23: A A A A
c22: A A A A
c21: A A A A
c20: A A A A
c19: A A A A
c18: A A A A
c17: A A A A
c16: A A A A
Reading thread register values requires all threads on a given core to be in the quiesced state.
$ ./pdbg -p0 -c22 -t0 -t1 -t2 -t3 stop
$ ./pdbg -p0 -c22 -t0 -t1 -t2 -t3 threadstatus
p0t: 0 1 2 3
c22: Q Q Q Q
$ ./pdbg -p0 -c22 -t0 getgpr 2
p0:c22:t0:gpr02: 0xc000000000f09900
$ ./pdbg -p0 -c22 -t0 getspr 8
p0:c22:t0:spr008: 0xc0000000008a97f0
./pdbg -p0 -c22 -t0 -t1 -t2 -t3 start
./pdbg -p0 -c22 -t0 -t1 -t2 -t3 threadstatus
p0t: 0 1 2 3
c22: A A A A
$ echo hello | sudo ./pdbg -p 1 putmem 0x250000001
Wrote 6 bytes starting at 0x0000000250000001
$ sudo ./pdbg -p 1 getmem 0x250000001 6 | hexdump -C
0x0000000250000000: 68 65 6c 6c 6f 0a
$ sudo ./pdbg -p 1 getmem 0x250000001 6 --raw | hexdump -C
00000000 68 65 6c 6c 6f 0a |hello.|
00000006
$ echo hello | sudo ./pdbg -p 1 putmem --ci 0x3fe88202
Wrote 6 bytes starting at 0x000000003fe88202
$ sudo ./pdbg -p 1 getmem --ci 0x3fe88202 6 --raw | hexdump -C
00000000 68 65 6c 6c 6f 0a |hello.|
00000006
$ lsprop /proc/device-tree/hwrng@3ffff40000000/
ibm,chip-id 00000000
compatible "ibm,power-rng"
reg 0003ffff 40000000 00000000 00001000
phandle 100003bd (268436413)
name "hwrng"
$ sudo ./pdbg -p 0 getmem --ci 0x0003ffff40000000 4 --raw |hexdump -C
00000000 01 c0 d1 79 |...y|
00000004
$ sudo ./pdbg -p 0 getmem --ci 0x0003ffff40000000 4 --raw |hexdump -C
00000000 77 9b ab ce |w...|
00000004
$ sudo ./pdbg -p 0 getmem --ci 0x0003ffff40000000 4 --raw |hexdump -C
00000000 66 8d fb 42 |f..B|
00000004
$ sudo ./pdbg -p 0 getmem --ci 0x0003ffff40000000 4 --raw |hexdump -C
00000000 fa 9b e3 44 |...D|
00000004
Exploitation of HTM is limited to POWER8 Core from the powerpc host.
Core HTM on POWER8 needs to run SMT1 and no power save, so you need to run this first:
ppc64_cpu --smt=1
for i in /sys/devices/system/cpu/cpu*/cpuidle/state*/disable;do echo 1 > $i;done
Also, using HTM requires a kernel built with both CONFIG_PPC_MEMTRACE=y
(v4.14) and CONFIG_SCOM_DEBUGFS=y. debugfs should be mounted at
/sys/kernel/debug. Ubuntu 18.04 has this by default.
pdbg provides a htm command with a variety of sub-commands. The most
useful command is record which will start the trace, wait for buffer
to fill (~1 sec), stop and then dump the trace to a file (~5 sec). eg.
pdbg -l 0 htm core record
pdbg -l allows users to specify CPUs using the same addressing as scheme as taskset -c. This can be useful for tracing workloads. eg.
taskset -c 0 myworkload
sleep 1
pdbg -l 0 htm core record
There are also low level htm commands which can also be used:
startwill configure the hardware and start tracing in wrapping mode.stopwill still stop the trace and de-configure the hardware.dumpwill dump the trace to a file.
At the moment gdbserver is only supported on P8 and P9 and P10.
Memory access can only be performed on kernel memory.
To run a gdbserver on a machine from a BMC running OpenBMC:
-
Read NOTES and set up the BMC and host as recommended.
-
(Optional) Stop the threads of the core(s) you want to look at. Ideally all threads in the machine should be debugged:
$ ./pdbg -a stop
- Run gdbserver on the target threads, accessible through port 44
$ ./pdbg -a gdbserver 44
The thread-id tid is set to the PIR of the corresponding thread, the hard_smp_processor_id. Be warned, "info threads" or other gdb operations that iterate over all threads may be very slow when debugging a lot of threads, especially over slow remote links.
On your local machine: $ gdb (gdb) target remote palm5-bmc:44
Debugging info: (gdb) set debug remote 10
Long-running operations or high latency links: The gdb client timeout defaults to 2 seconds after which it re-transmits commands. The gdb server does not deal with this robustly today and this can cause hangs and other unexpected results. If gdb client stops responding, behaves strangely or complains about bad or unexpected remote packets, try increasing the timeout. E.g., (gdb) set remotetimeout 60
Notes:
-
DON'T RUN PDBG OVER FSI WHILE HOSTBOOT IS RUNNING. Weird things seem to happen.
-
If you want to view the kernel call trace then run gdb on the vmlinux that the host is running (the kernel needs to be compiled with debug symbols).
-
The kernel HARDLOCKUP watchdog can interact badly with GDBSERVER (and all pdbg direct controls for that matter). Disabling it before debugging is a good idea.
-
Idle states have often had problems with pdbg direct controls. If things are misbehaving, booting Linux with powersave=off is the first thing to try.
-
attn instructions seem to cause host hangs on POWER9. gdb breakpoints should not be used.
-
attn instructions can cause the service processor to begin error handling. If breakpoints are to be used, the attn handler service on the BMC should be stopped first (don't forget to start it again when done).
systemctl stop attn_handler.service
Development and patch review happens on the mailing list at:
Patches are tracked through patchwork:
https://patchwork.ozlabs.org/project/pdbg/list
Pull requests via Github are also acceptable if you are not familiar with email based patch submission.