James Wing By James Wing on 2016-04-21

This article describes how to configure the CloudWatch Logs Agent for NiFi log files.

Using CloudWatch Logs with BatchIQ Instances

CloudWatch Logs can provide useful and cost-effective visibility into the Apache NiFi logs on your BatchIQ NiFi instance. The CloudWatch Logs Agent is installed and configured on BatchIQ instances starting with Apache NiFi 0.6.1 (April 2016), although it is turned off by default. Activating the CloudWatch Logs Agent requires AWS permissions for writting logs and some simple admin commands to start and run the agent.

EC2 Role Permissions

Using CloudWatch logs requires your EC2 instance role have permission to write to CloudWatch Logs.

EC2 Roles may only be assigned to instances at launch. Because the AWS Marketplace 1-Click launch experience does allow role assignment, you should use the Manual Launch option to go through the normal EC2 launch screen flow and assign an IAM Role to your instance. Permissions added to an existing and assigned role will be dynamically applied to existing instances.
A typical policy that you can attach to your IAM role looks like this:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents",
        "logs:DescribeLogStreams"
    ],
      "Resource": [
        "arn:aws:logs:*:*:*"
    ]
  }
 ]
}

Please see Configure your IAM role or user for CloudWatch Logs for more information.

Activating CloudWatch Logs

The CloudWatch Logs agent is installed on your BatchIQ instance, but is deactivated by default. To activate the agent, do the following:

  1. Connect to your instance using SSH
  2. Start the agent using the service controller utility
    sudo service awslogs start
    The results should look like this:
    [ec2-user@ip-172-31-46-242 ~]$ sudo service awslogs start
    Starting awslogs:                                          [  OK  ]
  3. Last, run the chkconfig utility to activate the agent at boot time in the future:
    sudo chkconfig awslogs on

The default BatchIQ AMI for Apache NiFi includes the CloudWatch agent, but it has not been turned on.

sudo chkconfig awslogs on
sudo service awslogs start

Or, you can include the following Cloud-Init template in your instance user data at launch:

#cloud-config
runcmd:
- chkconfig awslogs on
- service awslogs start

Need Some Help?

Please use the BatchIQ Support Portal to open a ticket and get customized assistance.