Control Groups are a feature of the Linux kernel that allows resource utilisation limits to be set on processes.
The two primary resources we see questions about in Support are CPU and Memory utilisation - here I will walk through how to create a Control Group limit on memory for a Workbench session.
The resources available to limit are all listed in the kernel documentation, we are looking for the section on memory.
memory.high A read-write single value file which exists on non-root cgroups. The
default is “max”. Memory usage throttle limit. This is the main mechanism to control memory
usage of a cgroup. If a cgroup’s usage goes over the high boundary, the
processes of the cgroup are throttled and put under heavy reclaim pressure. Going over the high limit never invokes the OOM killer and under extreme
conditions the limit may be breached. memory.max A read-write single value file which exists on non-root cgroups. The
default is “max”. Memory usage hard limit. This is the final protection mechanism. If a
cgroup’s memory usage reaches this limit and can’t be reduced, the OOM
killer is invoked in the cgroup. Under certain circumstances, the usage
may go over the limit temporarily. In default configuration regular
0-order allocations always succeed unless OOM killer chooses current task
as a victim. Some kinds of allocations don’t invoke the OOM killer. Caller could retry
them differently, return into userspace as -ENOMEM or silently ignore in
cases like disk readahead. This is the ultimate protection mechanism. As long as the high limit is
used and monitored properly, this limit’s utility is limited to providing
the final safety net.
To configure this for Workbench, we want to adjust the rstudio-launcher.service
, as that is the service that creates and controls each session. Here we will set the memory.high
to 2GB:
systemctl set-property rstudio-launcher.service MemoryHigh=2G
We can see that this has created a systemd
control file:
[root@rocky]# cat /etc/systemd/system.control/rstudio-launcher.service.d/50-MemoryHigh.conf
# This is a drop-in unit file extension, created via "systemctl set-property"
# or an equivalent operation. Do not edit.
[Service]
MemoryHigh=2147483648
In this way, you can enforce CPU and memory limits at an OS level, rather than just for R sessions within Workbench.
On a launcher based setup with AWS EC2 RHEL 8, MemoryHigh cgroup condition did not work as we would expect - i.e. 256 GB with 85% cap around 217 GB still caused the session to occupy complete memory and launcher service to go down.
MemoryMax on the other hand was able to invoke OOM killer and terminate the specific launcher session that was racing towards the memory threshold set.
Hope this helps anyone looking at the article.