Monitoring Posit Team Using Prometheus and Graphite

Avatar
Cole Arendt
Follow

Posit Connect and Posit Workbench (previously RStudio Workbench) both support exporting server metrics for monitoring. The only current option for metric exposure is Graphite.

If you are unfamiliar with Graphite, there is general documentation on Graphite here: https://graphite.readthedocs.io/en/latest/overview.html

If you are interested in using Graphite, each product has a guide to the configuration:

Posit Package Manager, by contrast, exposes a Prometheus endpoint directly. This can be configured by using the "Metrics" section of the configuration. By default, the Prometheus endpoint is accessible at /metrics on port :2112.


Today, there are many competing alternatives for exporting metrics and monitoring server health. Prometheus is one such alternative that has fostered wide adoption. This article is a guide to using Prometheus to monitor Posit Team by converting Graphite into a Prometheus endpoint.

Getting Started

There are many good "Getting Started" guides for Prometheus. If you are unfamiliar, this may be a good place to start. 

The general steps for using Prometheus to monitor Posit Team are to:

  1. Configure a mapping from Graphite to Prometheus metrics via a mapping.yaml file.
  2. Install Prometheus' Graphite Exporter.
  3. Run the Graphite Exporter, manually or via a systemd unit file.
  4. Test that the system is working correctly.
  5. Configure Posit Team products to output to Graphite.

1. Configure the mapping.yaml

You will need to define the mapping that translates graphite metrics to Prometheus metrics. An example mapping.yaml that covers both products is the following. This file must be present on each server where you'll want to configure Prometheus. 

mappings:
 - match: "rstudio\\.(\\w+)\\.(.*)"
   match_type: regex
   name: "rstudio_$2"
 - match: "rsconnect\\.(\\w+)\\.(.*)"
   match_type: regex
   name: "rsconnect_$2"

Since each server likely only has one product, remove the entry for the other Posit product on that server. You can save this file wherever you like, but it might be convenient to place both the mapping.yaml and the binary you'll download in the next step in /opt/prometheus-graphite. Note in this case that we dropped the second component for each graphite metric name (the hostname) to reduce metric cardinality.

Note that it is possible to configure a "central" Graphite Exporter so that you do not need to run it on each server. In this case, you will want to alter this mapping file to preserve the second component (hostname) and configure each Posit Connect / Posit Workbench server to utilize this central exporting server.

mappings:
 - match: "rstudio\\.(\\w+)\\.(.*)"
 match_type: regex
 name: "rstudio_$1_$2"
 - match: "rsconnect\\.(\\w+)\\.(.*)"
 match_type: regex
 name: "rsconnect_$1_$2"

2. Install the Graphite Exporter

In order to begin scraping metrics, you need a Prometheus endpoint to pull information from. We will do this with the "Graphite Exporter," a supported component of the Prometheus project that converts Graphite data into a Prometheus endpoint. It is also featured on the Prometheus "Downloads" page.

You will need to install the Graphite Exporter onto each server that you want to monitor with Prometheus. The downloads page is here, and the releases page is here.

3. Run the Graphite Exporter

Once you have the Graphite Exporter downloaded, you need to run it. This can be done manually from the command line or via a systemd unit file.

To run the Graphite Exporter manually, run the following on the command line. If you put the binary and mapping.yaml somewhere other than /opt/prometheus-graphite, change the paths below as appropriate.

/opt/prometheus-graphite/graphite_exporter\
 --web.listen-address=":9108"\
 --web.telemetry-path="/metrics"\
 --graphite.listen-address=":9109"\
 --graphite.mapping-config="/opt/prometheus-graphite/mapping.yaml"\
 --graphite.sample-expiry="5m"

If you'd like to use a systemd unit file, the chunk below should suffice. If you put the binary and mapping.yaml somewhere other than /opt/prometheus-graphite, change the paths below as appropriate.

[Unit]
Description=Run prometheus graphite exporter

[Service]
Type=simple
User=prom-graphite
Group=prom-graphite
ExecStart=/opt/prometheus-graphite/graphite_exporter\
 --web.listen-address=":9108"\
 --web.telemetry-path="/metrics"\
 --graphite.listen-address=":9109"\
 --graphite.mapping-config="/opt/prometheus-graphite/mapping.yaml"\
 --graphite.sample-expiry="5m"
PIDFile=/var/run/prometheus-graphite.pid
Restart=on-failure
KillMode=process

[Install]
WantedBy=multi-user.target

4. Test that the system is working properly

To test that conversion is happening properly, you can run the following example from the Graphite Exporter documentation on the command line:

echo "rsconnect.test1.test2 1234 $(date +%s)" | nc localhost 9109        # TCP
echo "rsconnect.test1.test2 1234 $(date +%s)" | nc -u -w1 localhost 9109 # UDP

Then look for the converted output on the correct endpoint:

curl -s localhost:9108/metrics | grep rsconnect_test2

Which should result in something like:

# HELP rsconnect_test2 Graphite metric rsconnect_test2
# TYPE rsconnect_test2 gauge
rsconnect_test2 1234

Now we are ready to convert real metrics!

5. Configure Posit Professional Products

Configure Posit Connect

To set up Posit Connect, we will add the following lines to the configuration file:

[Metrics]
GraphiteEnabled = true
GraphiteClientId = rsconnect
GraphiteHost = 127.0.0.1
GraphitePort = 9109

Configure Posit Workbench

To set up Posit Workbench, you will:

  • Configure the Posit Workbench Health Check to export a Prometheus endpoint
  • Configure Posit Workbench to export graphite metrics

The Posit Workbench Health Check has a configurable structure, and an example Prometheus template file is available in the admin guide: https://docs.posit.co/ide/server-pro/auditing_and_monitoring/server_health_checks.html#customizing-responses

Place this template into a file: /etc/rstudio/health-check

Then add the following configuration to the server configuration:

/etc/rstudio/rserver.conf

server-health-check-enabled=1

monitor-graphite-enabled=1
monitor-graphite-host=127.0.0.1
monitor-graphite-port=9109
monitor-graphite-client-id=rstudio

Finishing Touches

Once we restart all services, we should be able to test that the metrics are getting picked up properly with:

curl -s localhost:9108/metrics

# check the Posit Workbench health-check
curl -s localhost:8787/health-check

NOTE: we set our scraping interval to 5 minutes up above (with --graphite.sample-expiry="5m"). This means that your initial metrics and any changes may take a few minutes to show up.

Congratulations! You are exporting Prometheus metrics!

Next Steps

Once you have a Prometheus endpoint, you can collect and store metrics by using any metrics aggregator that supports the ability to scrape from a Prometheus endpoint. Prometheus provides such a server (even packaged as a docker container), and Grafana is an open-source option for exploring and visualizing metrics.

Alternatives (both paid and open source) abound. We would recommend checking with your IT department to see if any existing standard is already in use that you can leverage.

You can also look at other types of exporters. For instance, the Node Exporter is often used for standard server metrics.

Please share how you have set up monitoring for Posit Team, or let us know if you have any questions! The "R Admins" section on RStudio Community is a great place to discuss best practices and architectures for Posit product administration.

Comments