When wanting to trace networking with RStudio Workbench using the Job Launcher and Kubernetes to run sessions in dedicated pods, there are a wide array of tools at your disposal.
One such tool, albeit limited, is to launch a session with network diagnostics enabled. This is the approach discussed here.
Configure RStudio Workbench
Unfortunately, to make use of this functionality, you will need to change the configuration for RStudio Workbench.
We will be following this guide to configure the entrypoint script for new RStudio Workbench sessions.
IMPORTANT NOTE: this will render most images unusable for RStudio Workbench sessions, and should thus be considered as taking downtime, even though the service will remain up.
Create a file named /etc/rstudio/entrypoint.json with the following contents
/etc/rstudio/entrypoint.json
"/usr/local/bin/startup.sh"
NOTE: This script is embedded in the rstudio/r-session-debug image that we will be using for our session further down. If you have your own image, this is the link to the startup script that you need.
Add or append the following section to your job-json-overrides definitions for /etc/rstudio/launcher.kubernetes.profiles.conf
NOTE: You can restrict to just your username here by using [user] or a group with [@group], so that other users are not affected
/etc/rstudio/launcher.kubernetes.profiles.conf
[*]
job-json-overrides="/spec/template/spec/containers/0/command/0":"/etc/rstudio/entrypoint.json"
# setting default container images
# using pattern of rstudio/r-session-debug:<os>-<version>
container-images="rstudio/r-session-debug:bionic-1.4.1717-3"
Other Options
If you want to configure the entrypoint script, you can set the following env vars using the /etc/rstudio/launcher-env file
- TCPDUMP_OUTPUT = the file to put the tcpdump output into. This can be helpful if setting to a persistent directory
- TCPDUMP_MAX_PACKETS = the max number of packets to output (restricts file size)
/etc/rstudio/launcher-env
JobType: session
Environment: RSTUDIO_TYPE=session
TCPDUMP_OUTPUT=/tmp/tcpdump.pcap
TCPDUMP_MAX_PACKETS=10000
JobType: adhoc
Environment: RSTUDIO_TYPE=adhoc
Launch a Session
Now restart RStudio Workbench and the Job Launcher:
rstudio-server restart
rstudio-launcher restart
And launch a session in the IDE, choosing the following as your image name:
# rstudio/r-session-debug:<os>-<version>
# for example
rstudio/r-session-debug:bionic-1.4.1717-3
NOTE: we restricted user selections to this image in the example above
Get Diagnostics
There are several useful network and process diagnostics tools installed into the r-session-debug container.
However, the principal debugging information included by default is a tcpdump packet capture that starts at container startup and captures the first 10000 packets (configurable) - this is written to /tmp/tcpdump.pcap by default (configurable)
The best way to utilize this packet capture is to exec into the container, kill the tcpdump process (if it has not finished yet), and then copy the .pcap file to a local / persistent storage.
# kubectl -n rstudio exec -it session-7afefcfc78a310ced79a5-cole---rstudio-session-5-lxft8d95 -- bash
# ps -ef
UID PID PPID C STIME TTY TIME CMD
cole 1 0 0 01:21 ? 00:00:00 /bin/bash /usr/lib/rstudio-server/bin/rsession-run -u cole -p 7afefcfc78a31 -s 0ced79a5 --session-use-secure-cookies 1 --session-root-path / --session-
root 7 1 0 01:21 ? 00:00:00 tcpdump -i any -w /tmp/tcpdump.pcap -c 10000
cole 145 1 0 01:21 ? 00:00:04 /usr/lib/rstudio-server/bin/rsession -u cole -p 7afefcfc78a31 -s 0ced79a5 --session-use-secure-cookies 1 --session-root-path / --session-same-site 0 --
cole 185 145 0 01:27 pts/0 00:00:00 bash -l
root 211 0 0 01:40 pts/1 00:00:00 bash
root 226 211 0 01:40 pts/1 00:00:00 ps -ef
# kill 7
# exit
# kubectl -n rstudio cp session-7afefcfc78a310ced79a5-cole---rstudio-session-5-lxft8d95:/tmp/tcpdump.pcap tcpdump.pcap
Oftentimes, we would then ask that you send this file to RStudio Support, so that we can get more information about what is going wrong!
Building your own
If you are building your own images, you can reference our r-session-debug image creation at this GitHub repository.
Comments