Docs

OpenShift Container Platform Installation

One agent needs to be deployed on each OpenShift node to monitor all the containers and services in the cluster.

The agent will automatically collect all labels applied to containers running on OpenShift. You can use these labels as selectors to run plugins and schedule checks against services running in your cluster.

To deploy the Outlyer agent on OpenShift Container Platform, you will need to install oc and log in to your OpenShift cluster as admin or any user who has the cluster-admin role.

A Red Hat Customer Portal login is required to access our RHEL-Atomic certified Docker image. If you do not already have one, you can obtain one here.

First, create a secret in your cluster, in which to store your Agent key. The outlyer-agent container will be deployed into the kube-system namespace:

oc create secret generic outlyer-agent-key --namespace=kube-system --from-literal=accountKey='<PUT YOUR ACCOUNT AGENT KEY HERE>'

Because Kubernetes doesn’t provide a cluster name by default, and you could potentially add multiple OpenShift clusters to your Outlyer account, you will need to set a cluster name in a configmap that the agent daemonset can read to ensure all your instances and metrics can be distinguished in your account using the k8s.cluster label:

oc create configmap outlyer-config --namespace=kube-system --from-literal=k8s.cluster='<PUT A UNIQUE CLUSTER NAME HERE, i.e. openshift.prod.acme.com>'

Next, apply the RBAC (Role Based Access Control) permissions required for the Outlyer Agent to run:

oc apply -f https://www.outlyer.com/assets/files/rbac.yaml

Create a secret containing the Red Hat Customer Portal credentials:

oc create secret docker-registry rh-container-catalog \
    --docker-server=registry.connect.redhat.com \
    --docker-username=<USERNAME HERE> \
    --docker-password=<PASSWORD HERE> \
    --docker-email=<EMAIL HERE> \
    --namespace=kube-system

Associate this secret with the outlyer-agent service account:

oc secrets link outlyer-agent rh-container-catalog --for=pull --namespace=kube-system

Apply the DaemonSet to launch Outlyer across all nodes:

oc apply -f https://www.outlyer.com/assets/files/daemonset-openshift.yaml

To confirm the DaemonSet is deployed properly:

oc get pods --namespace=kube-system | grep outlyer-agent

You will see one running for each schedulable node you have in your cluster. Make sure your OpenShift master and infra nodes configuration allow the running of Outlyer pods.

Deploying Kube-State-Metrics

kube-state-metrics is a simple service that listens to the Kubernetes API server and generates metrics about the state of the objects. The default Kubernetes dashboards for Outlyer use the metrics from kube-state-metrics to give an overview of the cluster. You will need to deploy kube-state-metrics on your Kubernetes cluster following the instructions here:

git clone https://github.com/kubernetes/kube-state-metrics.git
cd kube-state-metrics
oc apply -f kubernetes

Once successfully deployed, you will have access to a lot of useful metrics that can be used to see the state of your cluster.

Editing Outlyer DaemonSet and RBAC Configuration

The configuration for the Outlyer agent DaemonSet and RBAC can be downloaded from here for local changes:

curl -O https://www.outlyer.com/assets/files/rbac.yaml
curl -O https://www.outlyer.com/assets/files/daemonset-openshift.yaml

The DaemonSet manifest daemonset-openshift.yaml can be edited to add additional deployment configuration options:

  • If you want to monitor Java services via JMX you will need to deploy a different Docker image that has Java installed alongside the Outlyer agent. This is included seperately as Java increases the image size so should only be downloaded to all your hosts if needed. To pull the Java version of the Docker image edit the daemonset-openshift.yaml file container image to add ‘-java’ at the end of the image tag. i.e. ‘registry.connect.redhat.com/outlyer/agent2:1.4.6-java’
  • You can add additional environment variables to pass to the agent configuration when it starts up. For full details of what environment settings are available, please refer to our docker instructions.
  • You can add nodeSelectors to only run the Daemonset on specific nodes.

Role-Based Access Control (“RBAC”)

Role-Based Access Control (“RBAC”) uses the “rbac.authorization.k8s.io” API group to drive authorization decisions, allowing admins to dynamically configure policies through the Kubernetes API. You can read more about using RBAC here.

To simplify deployment of the Outlyer DaemonSet, a service account with cluster role policies is included alongside the Daemonset manifest and deployed automatically when running the instructions above. If your cluster doesn’t have RBAC enabled, these additional manifests will not affect the running of the Outlyer DaemonSet.

Updating and Upgrading the DaemonSet

If you want to deploy a newer version of the agent, you can update the container image version, or apply new settings to the DaemonSet. Once your manifest is updated with the changes you can use the following command to update the DaemonSet on Kubernetes:

oc apply -f <path-to>/daemonset.yaml

Or to simply deploy a newer image version to the DaemonSet:

oc --namespace=kube-system set image ds/outlyer-agent outlyer-agent=registry.connect.redhat.com/outlyer/agent2:1.4.6

Troubleshooting

If you do not see your OpenShift cluster Nodes appear in your account’s Host list after running the above instructions successfully you can check if the DaemonSet is running using the following command:

oc get daemonsets --namespace=kube-system

This will return a message like below if the DaemonSet is successfully deployed and running on a 5 Node cluster:

NAME            DESIRED   CURRENT   READY     UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
outlyer-agent   5         5         5         5            5           <none>          4s

If you see any of the numbers are not the same as expected this should indicate there was an issue deploying the DaemonSet across the cluster. You will then need to list the Kubernetes Pods and get the logs for the outlyer-agent pods via oc to see exactly what the issue was:

$ oc get pods --namespace=kube-system
NAME                  READY     STATUS    RESTARTS   AGE
outlyer-agent-c2lqg   1/1       Running   0          10m

$ oc logs outlyer-agent-c2lqg --namespace=kube-system
2018-03-02 23:22:47 INFO apscheduler.executors.default - Running job "local_sync (trigger: interval[0:00:10], next run at: 2018-03-02 23:22:57 UTC)" (scheduled at 2018-03-02 23:22:47.591177+00:00)
......

Removing the Outlyer Agent

You can remove the Outlyer agent with the following commands:

oc delete --namespace=kube-system daemonset outlyer-agent
oc delete --namespace=kube-system serviceaccount outlyer-agent
oc delete --namespace=kube-system clusterrole outlyer-agent
oc delete --namespace=kube-system secret outlyer-agent-key rh-container-catalog
oc delete --namespace=kube-system configmap outlyer-config