James Wing By James Wing on 2016-06-07

CloudWatch Logs provides a hosted interface for viewing, searching, and alerting for your log data. Of course, you can always SSH into your instance and tail the logs. But you may find CloudWatch Logs a good visual tool for monitoring your NiFi instance logs easily and inexpensively. Use cases include:

  • Storing logs off-instance for reliability
  • Troubleshooting processor issues
  • Monitoring activity levels in NiFi
  • Alerting on errors
  • Using CloudWatch Logs as a gateway to ElasticSearch for more advanced analysis and monitoring

Get Started

BatchIQ instances come with the CloudWatch logs agent installed and configured, but not turned on by default. Please see Configuring CloudWatch Logs for details on turning on the logs agent and the required permissions.

Find NiFi Logs

The default log configuration on BatchIQ instances sends the NiFi log files to CloudWatch Logs in the same region as your instance. The logs appear under the log file name prefixed by "nifi", like nifi/nifi-app.log. NiFi log files are grouped by their name/type, with log streams for individual instances appearing within each group. This grouping allows you to specify sensible custom metrics and alarms that apply to content of nifi-app, nifi-bootstrap, or nifi-user logs respectively.

NiFi log groups in CloudWatch Logs

View NiFi Logs

You can view the latest log entries within the log stream for your instance. NiFi App Log in CloudWatch Logs

Search for Errors

CloudWatch Logs provides text search over the logs in the stream, allowing you to easily find system and processor errors. NiFi App Log in CloudWatch Logs

Found some NiFi errors in CloudWatch Logs

Define Custom Merics

You can also define custom metrics derived from your log data. These metrics will be accessible in CloudWatch and can be used as the basis for alarms. Metrics might be defined for particular error conditions, or simply to track the activity level of NiFi on an ongoing basis. Defining a custom metric in CloudWatch Logs (1 of 2)

Defining a custom metric in CloudWatch Logs (2 of 2)