Writing Native Plugins

While the Outlyer agent can run standard Nagios plugins and scrape Promtheus endpoints, it also supports a plugin format called Native which is specific to the Outlyer agent. All our standard integrations are written in this format which outputs a JSON payload with collected metrics that the agent can forward to Outlyer.

Although Open-Source standards is at the core of our design philosophy, there is currently no Open-Source plugin standard which allows different metric types (counters, gauges) to be collected with labels alongside events, which resulted in us creating this new plugin format for Outlyer. Over time we will evolve this plugin type to support more metric types and responses that enable our standard integrations to collect more information about your services when they run.

The plugin type supports Python and inherits from the outlyer_plugin module which will be made available as a seperate module outside the agent runtime soon for developing Outlyer plugins outside Outlyer. If you develop plugins using our built in plugin IDE in the Outlyer web application, they will run from the agent successfully.

The Basic Plugin Structure

At the most basic level, a plugin inherits the Plugin class which provides utility methods to collect metrics, status, events and logs from a plugin when it runs. It also outputs the JSON format required by the agent to parse the plugin output to forward this information to Outlyer.

As plugins run as a subprocess scheduled by the Outlyer agent, they must contain an ___main__ function at the end so they can run as a stand alone process when scheduled. A Status class is also provided to output different exit status codes so the agent can tell if the plugin ran successfully, was warning, critical or unknown, after it runs.

import sys

from outlyer_plugin import Plugin, Status

class MyPlugin(Plugin):

    def collect(self, _):
        
        return Status.OK

if __name__ == '__main__':
    sys.exit(MyPlugin().run())

The code above forms the basis of the Outlyer Native plugin and at this point will simply run and return an OK status.

Available Plugin Methods

To make the plugin more useful, the following methods are available from inside your Plugin class:

Get a Check Variable

If you run the plugin as part of a check with variable settings set (i.e. port number or username) you can get these check variables as Strings using the following method:

port = self.get('port', 'default')

This will get the plugin variable ‘port’ if set on the check running the plugin, and the variable hasn’t been applied will return the default value ‘default’ instead.

Collect Metrics

Outlyer plugins can collect Counters and Gauges, with more metric types (i.e. Histograms) planned soon. A metric has a name, optional labels, a value and an optional timestamp.

A counter should be used for collected values that increase over time and don’t go up and down. You can collect a counter using the following method:

self.counter(name: str, labels: Optional[dict]) -> Counter

A gauge should be used to collect values that can go up and down over time. You can collect a gauge using the following method:

self.gauge(name: str, labels: Optional[dict]) -> Gauge

All metric classes have a set method that allows you to set a value and an optional timestamp for the metric. If no timestamp is given it will use the current time the plugin was run as the timestamp for the metric value.

For example, the following code will set a counter ‘sys.cpu_time’ and a gauge ‘sys.cpu_pct’ with the timestamp overriden:

self.counter('sys.cpu_time', {'process': 'mysql'}).set(1243343.00)
# Get current time in unix UTC time seconds
ts = int(datetime.utcnow() * 1000)
self.gauge('sys.cpu_pct', {'process': 'mysql'}).set(34.34, ts=ts)

Status Codes

The Status class is provided to return standard status codes which indicate the status of a plugin:

When you exit the collect() method of the plugin class, simply return one of the enumerations above to send the plugin status code.

Further Reading

For more inspiration on how to run Outlyer Native plugins you can look through our integrations repo.