Overview

NiFi-Init is a boot-time configuration tool for easily creating secure and repeatable Apache NiFi installations from YAML templates. NiFi-Init was inspired by the popular Cloud-Init tool for general-purpose instance configuration. BatchIQ AMIs for Apache NiFi have been pre-configured to include NiFi-Init. Using NiFi-Init customization is optional, NiFi will be installed in a sensible default configuration with no extra effort on your part. You can use NiFi-Init to do many things:

  • Configure security
    • SSL/X.509 authentication
    • importing your certificates
    • Simple username/password authentication
  • Authorize users
  • Install custom libraries and NARs
  • Configure logging with CloudWatch Logs
  • Notify you when it is finished
NiFi-Init Context Diagram

Templates

NiFi-Init uses YAML templates to specify a NiFi configuration. Templates must start with #nifi-init to be recognized, otherwise they will be ignored.

Sections within the template define particular areas of functionality, detailed separately below.

name
Identifier for this template, used in notifications and logs.
#nifi-init
name: My Sample NiFi
security:
    scheme: ssl
    truststore:
        password: password1
        source: s3://mybucket/path/to/truststore.jks
    keystore:
        password: Super$ecret
        source: s3://mybucket/path/to/keystore.pfx
    authorization:
        nifi_admin: ROLE_ADMIN, ROLE_DFM, ROLE_PROVENANCE
notify:
    sns:
        topic-arn: arn:aws:sns:us-east-1:123456789012:NiFi_Operators

Templates from User Data

In cloud or virtualization environments that support User Data, you can specify your NiFi-Init template as text in the User Data.

#nifi-init
name: My User Data Template
...

Templates from URLs

You may also specify a list of one or more URLs to templates to load. Templates will be loaded and merged in sequence. This may be convenient to reuse common templates or template segments.

HTTP, HTTPS, and S3 URLs are supported.

#nifi-init
s3://s3bucket/path/to/template1.yaml
https://myserver/path/to/template2.yaml

Security

NiFi-Init supports a number of security-related settings for authentication, authorization, and the underlying security configuration of NiFi. High-level authentication schemes support configuration based on intent. Granular configuration of keystores, truststores and authorizations is provided for fine tuning

  • High-level authentication schemes
    • Simple - Authenticate clients with username/password
    • SSL - Authenticate clients with SSL/X.509 certificates
    • Anonymous - Do not authenticate clients
  • Reference
    • Authorization - permissions for users
    • Keystore configuration
    • Truststore configuration
scheme
ssl simple anonymous

Simple Username/Password Security

Simple username/password authentication provides basic security for NiFi, with less complexity than X.509 SSL certificates.

scheme simple
users
<user id>
Object map of users and their properties (see below)
user
roles
List of NiFi roles to assign, ROLE_ADMIN ROLE_DFM ROLE_MONITOR ROLE_PROVENANCE
password
Bcrypt 2a password hash (recommended!) or plain-text password. Plain-text passwords will be hashed by NiFi-Init and stored in Bcrypt 2a format.
#nifi-init
name: NiFi with Simple Username/Password Security
security:
  scheme: simple-username-password
  users:
    james:
      roles: ROLE_DFM, ROLE_PROVENANCE
      password: $2a$10$m/BtaawA/1yI2zacRZtQ2OUdC/ANnb08lYMyU1dKRMxfDIjGNplUp
    guest:
      roles: ROLE_MONITOR
      password: Super$ecure

SSL/X.509 Security

NiFi provides strong security through X.509 or SSL certificates. Configuring SSL security requires the following:

  • Keystore - The keystore specifies how the NiFi server identifies itself to clients.
  • Truststore - The truststore defines which client certificates and certificate chains to trust.
  • Client Certificates - Clients connecting to NiFi must have individual certificates that are trusted by NiFi or are in the trust chain.
  • Authorizations - Authorizations for clients allowing them to work in NiFi.

Please see the Security Configuration section in the NiFi System Administrator's Guide for more about NiFi security.

scheme ssl
keystore

auto to autogenerate a keystore with a self-signed key. To download and use a certificate store file:

source
HTTP, HTTPS, or S3 url to a certificate store file
password
Password or passphrase for the certificate store
type
JKS or PKCS12
truststore

You may download an entire certificate store file to use as the Truststore, and/or specify individual certificates to import.

source
HTTP, HTTPS, or S3 url to a certificate store file
password
Password or passphrase for the certificate store
type
JKS or PKCS12
certificates
Object map of aliases to certificates to import (see below)
certificates
<alias>
Url of a certificate to import and trust

See the sections below for more detail on the keystore and truststore configuration.

#nifi-init
name: SSL Security Example 1
security:
  scheme: ssl
  truststore:
    source: s3://mybucket/cert_stores/truststore.jks
    password: password1
  keystore:
    source: s3://mybucket/cert_stores/keystore.pfx
    password: 8V0vc1Igk50Qx0RkW5PZ
    type: PKCS12
  authorization:
    'CN=james, OU=test, O=TestCo, L=Seattle, ST=WA, C=US': ROLE_ADMIN, ROLE_DFM, ROLE_PROVENANCE
#nifi-init
name: SSL Security Example 2
security:
  scheme: ssl
  keystore: auto
  truststore:
    certificates:
      my_company_ca: s3://mybucket/certs/my_company_ca.pem
  authorization:
    'CN=james, OU=test, O=TestCo, L=Seattle, ST=WA, C=US': ROLE_ADMIN, ROLE_DFM, ROLE_PROVENANCE

Anonymous Security

NiFi can be configured to use anonymous security, such that all users have maximum permissions. Strict network access rules are recommended for this configuration.

scheme anonymous
#nifi-init
name: NiFi with Anonymous Security
security:
  scheme: anonymous

Authorization

Authorizations are specified as a list of objects, with user IDs as the key and the list of authorized roles as the value.

Please see the Controlling Access section in the NiFi System Administrator's Guide for more information about the roles.

<user id>
ROLE_ADMIN ROLE_DFM ROLE_MONITOR ROLE_PROVENANCE

If your users are identified by the distinguished names of their X.509 certificates, quote the distinguished name.

#nifi-init
security:
  authorization:
    user1: ROLE_ADMIN, ROLE_DFM, ROLE_PROVENANCE
    user2: ROLE_DFM
    user3: ROLE_MONITOR

Keystore

The Keystore is used to identify your instance to clients in an SSL/TLS configuration. You may specify a Keystore file, or allow NiFi-Init to generate a self-signed key for your instance.

auto
true to generate self-signed certificate
source
HTTP, HTTPS, or S3 path to a Keystore file
type
JKS or PKCS12 type of the Keystore file
password
Password or passphrase of the Keystore and private key
#nifi-init
security: sample
  keystore:
    source: s3://mys3bucket/path/to/keystore.jks
    type: JKS
    password: Super$ecret

Truststore

The Truststore is used to establish trust for authenticating clients. You may specify a Truststore file, one or more public key certificates to install.

auto
true to generate a Truststore with no client certificates
source
HTTP, HTTPS, or S3 path to a Truststore file
type
JKS or PKCS12 type of the Truststore file
password
Password or passphrase of the Truststore file provided in source
certificates
An object map of aliases and URLs to certificates to install
#nifi-init
security: sample
  truststore:
    source: https://myserver/path/to/truststore.pfx
    type: PKCS12
    password: Super$ecret
#nifi-init
security: sample
  truststore:
    certificates:
       my_company_ca_cert: s3://mybucket/company_ca_cert.pem

Notification

NiFi-Init can be configured to send a notification when it is complete. AWS Simple Notification Services (SNS) is currently supported.

SNS

Notification to an AWS Simple Notification Services topic.

topic
SNS Topic ARN. Your instance's IAM role must have Publish permission on this topic.
#nifi-init
notify:
    sns:
        topic-arn: arn:aws:sns:us-east-1:123456789012:NiFi_Operators
            

Amazon Web Services (AWS)

This section contains features particular to Amazon Web Services.

  • CloudWatch Logs

CloudWatch Logs

Specifies the configuration of the CloudWatch Logs agent for handling NiFi logs.

cloudwatch-logs
true to activate the CloudWatch Logs agent with a default configuration for NiFi logs.
#nifi-init
aws:
  cloudwatch-logs: true