James Wing By James Wing on 2016-06-15

BatchIQ instances now include NiFi-Init, a boot-time configuration tool for easily creating secure and repeatable Apache NiFi installations from YAML templates. NiFi-Init was inspired by the popular Cloud-Init tool for general-purpose instance configuration. Obviously, NiFi-Init is specific to Apache NiFi, with a template grammar customized to NiFi configuration options and requirements. Templates may be provided through instance User Data or URLs to HTTP, HTTPS, and S3 resources.

NiFi-Init Context Diagram

The initial feature set targets security configuration, making it easy to deploy a secured NiFi installation using SSL/X.509 certificates, or just plain username/password authentication. Username/password authentication might be as simple as this:

#nifi-init
name: NiFi with Simple Username/Password Security
security:
  scheme: simple-username-password
  users:
    james:
      roles: ROLE_DFM, ROLE_PROVENANCE
      password: $2a$10$m/BtaawA/1yI2zacRZtQ2OUdC/ANnb08lYMyU1dKRMxfDIjGNplUp

Where NiFi is provisioned to let me log in with a password, specified in Bcrypt hash format in the template. For SSL/X.509 security, we might want to start up NiFi with our company's Certificate Authority configured in the Truststore:

#nifi-init
name: NiFi with SSL Security
security:
  scheme: ssl
  keystore: auto
  truststore:
    certificates:
      my_company_ca: s3://mybucket/certs/my_company_ca.pem
  authorization:
    'CN=james, OU=test, O=TestCo, L=Seattle, ST=WA, C=US': ROLE_ADMIN, ROLE_DFM, ROLE_PROVENANCE
aws:
  cloudwatch-logs: true
notify:
  sns:
    topic-arn: arn:aws:sns:us-east-1:123456789012:NiFi_Operators

The template above configures NiFi security with a self-signed Keystore, and build a Truststore using our company CA certificate. It sets up authorization for a user to log in and start using NiFi. It turns on the CloudWatch Logs agent to capture NiFi logs. And it notifies us when NiFi is ready.

That's pretty cool, and even cooler that we can check these templates into source control and recreate a predicatable NiFi configuration. Or, if you store templates on S3, you can reference them directly:

#nifi-init
s3://sample-bucket/nifi-init/template.yaml

Ready to Try It?

You can get started with NiFi-Init by launching a BatchIQ instance from the AWS Marketplace. NiFi-Init is supported in the most recent version, 0.6.1-1. Templates must be included in the instance User Data at launch.

In the AWS Marketplace, select the "Manual Launch" option to use the familiar EC2 Launch Instance flow. AWS Marketplace Manual Launch Option

While configuring the launch settings, you will find User Data under Advanced Details on the Configure Instance page. User Data in the EC2 Launch Flow

Please see the NiFi-Init User Guide and Reference for more on specific features. I am looking forward to hearing how NiFi-Init works for you, and what features can be added to make it more useful.