Using a Network Debug Session for Posit Workbench with Launcher and Kubernetes

When wanting to trace networking with Posit Workbench using the Job Launcher and Kubernetes to run sessions in dedicated pods, there are a wide array of tools at your disposal.

One such tool, albeit limited, is to launch a session with network diagnostics enabled. This is the approach discussed here.

Configure Posit Workbench

Unfortunately, to make use of this functionality, you will need to change the configuration for Posit Workbench.

We will be following this guide to configure the entrypoint script for new Posit Workbench sessions.

IMPORTANT NOTE: this will render most images unusable for Posit Workbench sessions, and should thus be considered as taking downtime, even though the service will remain up.

Create a file named /etc/rstudio/entrypoint.json with the following contents

/etc/rstudio/entrypoint.json

"/usr/local/bin/startup.sh"

NOTE: This script is embedded in the rstudio/r-session-debug image that we will be using for our session further down. If you have your own image, this is the link to the startup script that you need.

Add or append the following section to your job-json-overrides definitions for /etc/rstudio/launcher.kubernetes.profiles.conf 

NOTE: You can restrict to just your username here by using [user] or a group with [@group], so that other users are not affected

/etc/rstudio/launcher.kubernetes.profiles.conf

[*]
job-json-overrides="/spec/template/spec/containers/0/command/0":"/etc/rstudio/entrypoint.json"
# setting default container images
# using pattern of rstudio/r-session-debug:<os>-<version>
container-images="rstudio/r-session-debug:bionic-1.4.1717-3"

Other Options

If you want to configure the entrypoint script, you can set the following env vars using the /etc/rstudio/launcher-env file

• TCPDUMP_OUTPUT = the file to put the tcpdump output into. This can be helpful if setting to a persistent directory
• TCPDUMP_MAX_PACKETS = the max number of packets to output (restricts file size)

/etc/rstudio/launcher-env

JobType: session
Environment: RSTUDIO_TYPE=session
  TCPDUMP_OUTPUT=/tmp/tcpdump.pcap
  TCPDUMP_MAX_PACKETS=10000

JobType: adhoc
Environment: RSTUDIO_TYPE=adhoc

Launch a Session

Now restart Posit Workbench and the Job Launcher:

rstudio-server restart
rstudio-launcher restart

And launch a session in the IDE, choosing the following as your image name:

# rstudio/r-session-debug:<os>-<version>
# for example
rstudio/r-session-debug:bionic-1.4.1717-3

NOTE: we restricted user selections to this image in the example above

Get Diagnostics

There are several useful network and process diagnostics tools installed into the r-session-debug container.

However, the principal debugging information included by default is a tcpdump packet capture that starts at container startup and captures the first 10000 packets (configurable) - this is written to /tmp/tcpdump.pcap by default (configurable)

The best way to utilize this packet capture is to exec into the container, kill the tcpdump process (if it has not finished yet), and then copy the .pcap file to a local / persistent storage.

# kubectl -n rstudio exec -it session-7afefcfc78a310ced79a5-cole---rstudio-session-5-lxft8d95 -- bash

# ps -ef
UID PID PPID C STIME TTY TIME CMD
cole 1 0 0 01:21 ? 00:00:00 /bin/bash /usr/lib/rstudio-server/bin/rsession-run -u cole -p 7afefcfc78a31 -s 0ced79a5 --session-use-secure-cookies 1 --session-root-path / --session-
root 7 1 0 01:21 ? 00:00:00 tcpdump -i any -w /tmp/tcpdump.pcap -c 10000
cole 145 1 0 01:21 ? 00:00:04 /usr/lib/rstudio-server/bin/rsession -u cole -p 7afefcfc78a31 -s 0ced79a5 --session-use-secure-cookies 1 --session-root-path / --session-same-site 0 --
cole 185 145 0 01:27 pts/0 00:00:00 bash -l
root 211 0 0 01:40 pts/1 00:00:00 bash
root 226 211 0 01:40 pts/1 00:00:00 ps -ef

# kill 7

# exit

# kubectl -n rstudio cp session-7afefcfc78a310ced79a5-cole---rstudio-session-5-lxft8d95:/tmp/tcpdump.pcap tcpdump.pcap

Oftentimes, we would then ask that you send this file to Posit Support, so that we can get more information about what is going wrong!

Building your own

If you are building your own images, you can reference our r-session-debug image creation at this GitHub repository.