CloudWatch Logs provides a hosted interface for viewing, searching, and alerting for your log data. Of course, you can always SSH into your instance and tail the logs. But you may find CloudWatch Logs a good visual tool for monitoring your NiFi instance logs easily and inexpensively. Use cases include:
- Storing logs off-instance for reliability
- Troubleshooting processor issues
- Monitoring activity levels in NiFi
- Alerting on errors
- Using CloudWatch Logs as a gateway to ElasticSearch for more advanced analysis and monitoring
BatchIQ instances come with the CloudWatch logs agent installed and configured, but not turned on by default. Please see Configuring CloudWatch Logs for details on turning on the logs agent and the required permissions.
Find NiFi Logs
The default log configuration on BatchIQ instances sends the NiFi log files to CloudWatch Logs in the same region as your instance.
The logs appear under the log file name prefixed by "nifi", like
NiFi log files are grouped by their name/type, with log streams for individual instances appearing within each group.
This grouping allows you to specify sensible custom metrics and alarms that apply to content of nifi-app, nifi-bootstrap, or nifi-user logs respectively.
View NiFi Logs
Search for Errors
Define Custom Merics
You can also define custom metrics derived from your log data. These metrics will be accessible in CloudWatch and can be used as the basis for alarms. Metrics might be defined for particular error conditions, or simply to track the activity level of NiFi on an ongoing basis.