Docs

Deployment Modes

Outlyer is unique in the fact it can push configuration down to agents. This allows for incredibly fast and easy setup of Outlyer without the need to learn and write a lot of configuration and configuration management to deploy Outlyer across your entire environment. It also allows you to delegate the complete monitoring setup to your teams who no longer need to rely on your central operations/DevOps team to setup monitoring checks for their services.

However we understand for some enviroments and use cases, a more traditional deployment model is preferred, with configuration pushed to out via configuration management and local configuration on the server. We call this Solo mode, and the rest of this article talks about the differences between both deployment modes and how to use them.

All agents can be deployed independently in your environment, which enables you to use a combination of modes across your environment(s). For example you may have Collaborative Mode enabled for your test and development environments, and all agents in your production environment running in Solo mode. Or you could have only certain servers such as your database servers deployed in Solo mode for extra security. It really is up to you.

The great thing about the Outlyer agent however, is with its built in auto-discovery, you don’t need to apply different configuration to each of your agents to monitor the different services on each server, you can simply put every one of your checks and plugins on every agent, and the agent will automatically run them based on the service/server labels it discovers. Whichever mode you decide to use, setting up your monitoring will be incredibly simple!

Collaborative Mode (Default)

Outlyer has a unique technology that allows teams outside of operations to quickly write plugins and deploy them to groups of services. We recommend that this feature is enabled and used in development and test so that you get the full value out of our self service capability.

In this mode, the agent will pull all its plugins and check configuration from Outlyer centrally, allowing users to quickly setup and configure their monitoring using Outlyer’s web UI, API or Command Line Client (CLI).

You can also take a hybrid approach when this mode is enabled, where configuration can be pulled from Outlyer centrally, but local checks and plugins can also be deployed and run by the agent as well. However this can get quite messy as locally deployed checks and plugins will not appear in Outlyer at this time, so we recommend picking one mode and sticking to that so all your configuration is stored in one place.

When Collaborative mode is enabled, you can still manage all your configuration as code by using the APIs or Command Line Client (CLI). Typically users will use Collaborative mode so they can get the benefits of delegated setup across their teams and fast and easy setup, and use our CLI to export their account configuration (including dashboards and alerts) so it can be version controlled in Github or another version control repository.

Solo Mode

In some cases, usually on production or in more tightly controlled environments, you will need to turn off the remote configuration capabilities. For this scenario we provide Solo mode which completely disables remote script execution and deployment. This is as simple as updating the agent.yaml configuration with solo: true and restarting the agent.

In Solo Mode the Outlyer Agent no longer polls the Outlyer API for plugin and check changes and instead polls the local disk. Simply drop all your check configuration YAML files into the agent’s conf.d directory and your plugins into the agent’s plugins directory. These locations can be overriden in your agent.yaml configuration but by default on a Linux install you will find these directories in the following locations:

  • conf.d: /etc/outlyer/conf.d
  • plugins: /etc/outlyer/plugins

Please refer to your agent.yaml configuration to check these haven’t been changed or on Windows environments.

Check Configuration

Solo mode configuration requires that you manually place the configuration files for checks on the host. When the Agent starts it will look in its conf.d directory for YAML files containing Check configuration.

Check configuration is required to be in the following format:

checks: #checks block
	<check_name>:
		command: 'my_plugin.py --format=nagios'
		interval: 30
		handler: 'nagios'
		plugins:
			- 'my_plugin.py'
		selectors:
			- 'host=my-host-name'
		env:
			plugin_var: '1234'
		disabled: false

checks block (required)

All check configuration must be under a checks block in the containing YAML file. This informs the agent that the config is intended as check configurations

check_name (required)

Specify the name you would like to use for a check these should be unique within your account. We do not impose uniqueness for check names, but it can lead to confusing side effects if there are duplicates

command (required)

The command attribute of a check is where you specify the executable command the agent should run to collect stats about a service. Typically this would execute a plugin with certain arguments to control how the plugin should operate for this check. Outlyer intends plugins be abstract components for reuse across multiple service checks. Command-line arguments and environment variables a key to supporting plugin reuse.

interval (optional: default 30)

Check can specify how often it should be run using the interval attribute, specifying an integer value for the number of seconds. You should consider how much load your command is likely to generate when setting the interval. Running an intensive operation too frequently can have negative side effects.

handler (optional: default nagios)

Handler tells the agent what it should do with the result from your command. Currently, we support the following handlers:

  • Nagios - Parses standard Nagios plugin output format and returns the performance metrics and status.
  • Prometheus - Parses Prometheus format and forwards the dimensional metrics.
  • Native - Parses the native Outlyer plugin response and forwards the dimensional metrics.

For example, you can use an off-the-shelf Nagios plugin, just set the handler to Nagios and the agent takes care of the rest.

plugins (optional: default none)

List the plugins required by this check command. These plugins should will be loaded from the plugin directory (see plugin_dir)

selectors (optional: default none)

Selectors specify the host or container instances you would like to target with this check, and this should be an array of selectors. A selector is a comma-separated list of labels. For example, you would use service=mysql,env=prod to target all instances with the labels service: mysql and env: prod.

Note: if no selectors are specified then the check should not be scheduled to run

env (optional: default none)

The env attribute is for specifying any environment variables to set when the check runs; this is useful for plugins configured using environment variables.

Note: these must be strings

disabled

The disabled attibute is for completely disabling the check.