This integration will collect metrics on every Kinesis Stream in a specified AWS region via Cloudwatch so you can easily monitor your Kinesis Streams in Outlyer.
AWS Kinesis provides high throughput message queues (streams) as a managed service so producers can put messages on the streams for consumers to process.
The following diagram illustrates the high-level architecture of Kinesis Data Streams. The producers continually push data to Kinesis Data Streams and the consumers process the data in real time. Consumers (such as a custom application running on Amazon EC2, or an Amazon Kinesis Data Firehose delivery stream) can store their results using an AWS service such as Amazon DynamoDB, Amazon Redshift, or Amazon S3.
A full list of available Cloudwatch metrics for Amazon Kinesis is available here.
| Metric Name | Type | Labels | Unit | Description |
|---|---|---|---|---|
| aws.kinesis_getrecords_bytes_sum | gauge | stream, region | Bytes | The number of bytes retrieved from the Kinesis stream. |
| aws.kinesis_getrecords_iteratoragemilliseconds_max | gauge | stream, region | Milliseconds | The age of the last record in all GetRecords calls made against an Kinesis stream. |
| aws.kinesis_getrecords_latency_max | gauge | stream, region | Count | The maximum time taken per GetRecords operation. |
| aws.kinesis_getrecords_records_sum | gauge | stream, region | Count | The number of records retrieved from the shard. |
| aws.kinesis_getrecords_success_sum | gauge | stream, region | Count | The number of successful GetRecords operations per stream. |
| aws.kinesis_incomingbytes_sum | gauge | stream, region | Bytes | The number of bytes successfully put to the Kinesis stream. |
| aws.kinesis_incomingrecords_sum | gauge | stream, region | Count | The number of records successfully put to the Kinesis stream. |
| aws.kinesis_putrecord_bytes_sum | gauge | stream, region | Bytes | The number of bytes put to the Kinesis stream using the PutRecord operation. |
| aws.kinesis_putrecord_latency_max | gauge | stream, region | Milliseconds | The maximum time taken per PutRecord operation. |
| aws.kinesis_putrecord_success_sum | gauge | stream, region | Count | The number of successful PutRecord operations per Kinesis stream. |
| aws.kinesis_putrecords_bytes_sum | gauge | stream, region | Bytes | The number of bytes put to the Kinesis stream using the PutRecords operation. |
| aws.kinesis_putrecords_latency_max | gauge | stream, region | Milliseconds | The maximum time taken per PutRecords operation. |
| aws.kinesis_putrecords_records_sum | gauge | stream, region | Count | The number of successful records in a PutRecords operation per Kinesis stream. |
| aws.kinesis_putrecords_success_sum | gauge | stream, region | Count | The number of PutRecords operations where at least one record succeeded, per Kinesis stream. |
| aws.kinesis_readprovisionedthroughputexceeded_avg | gauge | stream, region | Count | The number of GetRecords calls throttled for the stream over the specified time period. |
| aws.kinesis_writeprovisionedthroughputexceeded_avg | gauge | stream, region | Count | The number of records rejected due to throttling for the stream over the specified time period. |
In order for this integration to run, you must create an IAM role and access keys for the plugin to connect to your AWS Kinesis APIs and AWS Cloudwatch metrics:
- To get started, open the AWS Management Console
- Click the IAM tab.
- Click the Create a New Group of Users button.
- Enter a Group Name called Outlyer.
- Select the Read Only Access Policy Template then click Continue.
- Click the Create New Users tab.
- Enter a new User Name called Outlyer and click Continue and then Finish.
- Click Show User Security Credentials.
- Copy and paste your Access Key Id and the Secret Access Key somewhere safe.
The plugin must set the following variables to run:
AWS_ACCESS_KEY_ID: The AWS Access Key copied aboveAWS_SECRET_ACCESS_KEY: The AWS Secret Key copied above
In addition the following variables can be set to override the default values:
time_range: The time range in minutes to query your metrics over. By default this is set to the last 10 minutes.
It is recommended this plugin is run at most every 5 minutes to avoid high additional charges on your AWS acount
| Version | Release Date | Description |
|---|---|---|
| 1.0 | 24-Jun-2018 | Initial version of our AWS Kinesis integration. |