James Wing By James Wing on 2016-06-15
NiFi-Init Context Diagram

BatchIQ instances now include NiFi-Init, a boot-time configuration tool for easily creating secure and repeatable Apache NiFi installations from YAML templates. NiFi-Init was inspired by the popular Cloud-Init tool for general-purpose instance configuration. Obviously, NiFi-Init is specific to Apache NiFi, with a template grammar customized to NiFi configuration options and requirements. Templates may be provided through instance User Data or URLs to HTTP, HTTPS, and S3 resources.

James Wing By James Wing on 2016-06-07

CloudWatch Logs provides a hosted interface for viewing, searching, and alerting for your log data. Of course, you can always SSH into your instance and tail the logs. But you may find CloudWatch Logs a good visual tool for monitoring your NiFi instance logs easily and inexpensively. Use cases include:

  • Storing logs off-instance for reliability
  • Troubleshooting processor issues
  • Monitoring activity levels in NiFi
  • Alerting on errors
  • Using CloudWatch Logs as a gateway to ElasticSearch for more advanced analysis and monitoring

James Wing By James Wing on 2016-04-21

The latest NiFi release, 0.6.1, is now available in the AWS Marketplace as Apache NiFi provided by BatchIQ. You can read about the release in the official Apache NiFi Release Notes for 0.6.0 and 0.6.1, but I would like to call out a couple of key AWS-related enhancements:

  • Lambda - New PutLambda processor for routing flow files to Lambda functions and capturing the Lambda response
  • Kinesis - New PutKinesisFirehose processor for routing flow files to Kinesis
  • S3 Multipart Upload - The PutS3Object processor has been upgrade with multi-part upload support for large objects

AMI Updates

The BatchIQ AMI has also been improved:

  • Updated to Amazon Linux 2016.03
  • Java 8 now included by default
  • CloudWatch Logs agent configured for NiFi, but not turned on by default (more).

Anyone migrating from an earlier version of NiFi should take a look at our upgrade guidance.

James Wing By James Wing on 2016-04-19
Using NiFi for Database Ingest

Apache NiFi is a great tool for building flexible and performant data ingestion pipelines. Since relational databases are a staple for many data cleaning, storage, and reporting applications, it makes sense to use NiFi as an ingestion tool for MySQL, SQL Server, Postgres, Oracle, etc. In this article we will look at Apache NiFi's built-in features for getting FlowFile data into your database, and some strategies you should consider in implementing a database flow.

James Wing By James Wing on 2016-03-10

I'm very happy to announce that NiFi 0.5.1 is now available in the AWS Marketplace as Apache NiFi provided by BatchIQ. This is BatchIQ's first AWS Marketplace offering, currently available for free ($0 above EC2 charges).

Please try it. Bring your friends. And let me know how it goes, I'm quite eager to make improvements to the deployment, configuration, and monitoring experience.

James Wing By James Wing on 2016-01-23

This is a hands-on walkthrough configuring SSL/TLS authentication in Apache NiFi. The tasks we will accomplish include:

  1. Creating and installing a user certificate
  2. Setting up the server's KeyStore
  3. Setting up the server's TrustStore
  4. Installing the user certificate into the TrustStore
  5. Configuring authorization for our user

At the end, our user will be able to securely log in to our NiFi server. Our walkthrough will use self-signed certificates. The good news is that this simplifies many tasks and gets to a running secure installation very quickly. The bad news is that this may not match your experience working with real, signed certificates in some respects. Getting a firm grip on the process and the end result is critical, more so than the details of working with a particular certificate authority.

James Wing By James Wing on 2016-01-17
Overview of X.509 Certificate Security in NiFi

Configuring security in NiFi can be a bit tricky. NiFi's base security mechanism, mutual authentication with X.509 (a.k.a. TLS/SSL) certificates provides great security. But great security comes with a price, and that price is the complexity of configuring X.509 mutual authentication. There are a log of settings, and it may not be clear up front what all needs to be done, in what order. The error messages can be cryptic and frustrating. In this post, we'll start with an overview of the security situation in NiFi using X.509 certificates, focusing on human users rather than system-to-system access.