Kubernetes Installation (1.6+)
The Outlyer agent can be deployed as a container. One agent needs to be deployed on each Kubernetes node to monitor all the containers and services in the cluster.
The agent will automatically collect all labels applied to containers running on Kubernetes. You can use these labels as selectors to run plugins and schedule checks against services running in your cluster.
These instructions also apply for managed Kubernetes services such as EKS and AKS from AWS and Azure.
To deploy the Outlyer agent on Kubernetes, first create a secret in your cluster, in which to store your Agent key.
The outlyer-agent container will be deployed into the kube-system namespace so it has access to monitor all the components of the Kubernetes cluster.
kubectl create secret generic outlyer-agent-key --namespace=kube-system --from-literal=accountKey='<PUT YOUR ACCOUNT AGENT KEY HERE>'
Because Kubernetes doesn’t provide a cluster name by default, and you could potentially add multiple Kubernetes clusters to your
Outlyer account, you will need to set a cluster name in a configmap that the agent daemonset can read to ensure all your instances
and metrics can be distinguished in your account using the k8s.cluster label:
kubectl create configmap outlyer-config --namespace=kube-system --from-literal=k8s.cluster='<PUT A UNIQUE CLUSTER NAME HERE, i.e. kube.prod.acme.com>'
Next, apply the RBAC (Role Based Access Control) permissions required for the Outlyer Agent to run:
kubectl apply -f https://www.outlyer.com/assets/files/rbac.yaml
And apply the DaemonSet to launch Outlyer across all nodes, including any masters.
kubectl apply -f https://www.outlyer.com/assets/files/daemonset.yaml
To confirm the DaemonSet is deployed properly:
kubectl get pods --namespace=kube-system | grep outlyer-agent
You will see one running for each node you have in your cluster
Deploying Kube-State-Metrics
kube-state-metrics is a simple service that listens to the Kubernetes API server and generates metrics about the state of the objects. The default Kubernetes dashboards for Outlyer use the metrics from kube-state-metrics to give an overview of the cluster. You will need to deploy kube-state-metrics on your Kubernetes cluster following the instructions here:
git clone https://github.com/kubernetes/kube-state-metrics.git
cd kube-state-metrics
kubectl apply -f kubernetes
Once successfully deployed, you will have access to a lot of useful metrics that can be used to see the state of your cluster.
Editing Outlyer daemonset.yaml configuration
The configuration for the Outlyer agent DaemonSet can be downloaded from here for local changes:
curl -O https://www.outlyer.com/assets/files/rbac.yaml
curl -O https://www.outlyer.com/assets/files/daemonset.yaml
The DaemonSet manifest daemonset.yaml can be edited to add additional deployment configuration options:
- If you want to monitor Java services via JMX you will need to deploy a different Docker image that has Java installed alongside the Outlyer agent. This is included seperately as Java increases the image size by around 60MB so should only be downloaded to all your hosts if needed. To pull the Java version of the Docker image edit the daemonset.yaml file container image to add ‘-java’ at the end of the image tag. i.e. ‘outlyer/agent2:0.1.12-java’
- You can add additional environment variables to pass to the agent configuration when it starts up. For full details of what environment settings are available, please refer to our docker instructions.
- You can add nodeSelectors to only run the Daemonset on specific nodes.
Role-Based Access Control (“RBAC”)
Role-Based Access Control (“RBAC”) uses the “rbac.authorization.k8s.io” API group to drive authorization decisions, allowing admins to dynamically configure policies through the Kubernetes API. You can read more about using RBAC here.
To simplify deployment of the Outlyer DaemonSet, a service account with cluster role policies is included alongside the Daemonset manifest and deployed automatically when running the instructions above. If your cluster doesn’t have RBAC enabled, these additional manifests will not affect the running of the Outlyer DaemonSet.
Updating and Upgrading the DaemonSet
If you want to deploy a newer version of the agent, you can update the container image version, or apply new settings to the DaemonSet. Once your manifest is updated with the changes you can use the following command to update the DaemonSet on Kubernetes:
kubectl apply -f <path-to>/daemonset.yaml
Or to simply deploy a newer image version to the DaemonSet
kubectl --namespace=kube-system set image ds/outlyer-agent outlyer-agent=outlyer/agent2:1.5.11
Troubleshooting
If you do not see your Kubernetes cluster Nodes appear in your account’s Host list after running the above instructions successfully you can check if the DaemonSet is running using the following command:
kubectl get daemonsets
This will return a message like below if the DaemonSet is successfully deployed and running on a 5 Node cluster:
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
outlyer-agent 5 5 5 5 5 <none> 4s
If you see any of the numbers are not the same as expected this should indicate there was an issue deploying the DaemonSet across the cluster. You will then need to list the Kubernetes Pods and get the logs for the outlyer-agent pods via kubectl to see exactly what the issue was:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
outlyer-agent-c2lqg 1/1 Running 0 10m
$ kubectl logs outlyer-agent-c2lqg
2018-03-02 23:22:47 INFO apscheduler.executors.default - Running job "local_sync (trigger: interval[0:00:10], next run at: 2018-03-02 23:22:57 UTC)" (scheduled at 2018-03-02 23:22:47.591177+00:00)
......
Removing the Outlyer Agent
You can remove the Outlyer agent with the following commands:
kubectl delete --namespace=kube-system daemonset outlyer-agent
kubectl delete --namespace=kube-system serviceaccount outlyer-agent
kubectl delete --namespace=kube-system clusterrole outlyer-agent
kubectl delete --namespace=kube-system clusterrolebinding outlyer-agent