James Wing By James Wing on 2016-01-23

This is a hands-on walkthrough configuring SSL/TLS authentication in Apache NiFi. The tasks we will accomplish include:

  1. Creating and installing a user certificate
  2. Setting up the server's KeyStore
  3. Setting up the server's TrustStore
  4. Installing the user certificate into the TrustStore
  5. Configuring authorization for our user

At the end, our user will be able to securely log in to our NiFi server. Our walkthrough will use self-signed certificates. The good news is that this simplifies many tasks and gets to a running secure installation very quickly. The bad news is that this may not match your experience working with real, signed certificates in some respects. Getting a firm grip on the process and the end result is critical, more so than the details of working with a particular certificate authority.

For an overview of NiFi security, please take a look at Overview of X.509 SSL Security on NiFi. For a streamlined approach to configuring security in NiFi quickly and easily, see Introducing NiFi-Init.

NiFi Security with X.509 certificates

User Certificate

We're going to start with the admin user certificate. The first step here is to create a private key and public key certificate pair. For example purposes, we will do this for an imaginary "Admin Q. User".

openssl req -x509 -newkey rsa:2048 -keyout admin-private-key.pem -out admin-cert.pem -days 365 -subj "/CN=Admin Q. User/C=US/L=Seattle" -nodes

You should see output like the following from OpenSSL:

Loading 'screen' into random state - done
Generating a 2048 bit RSA private key
............................................................................................................................................................................+++
..........+++
writing new private key to 'admin-private-key.pem'
-----

If you are doing this on Windows and get an error "Subject does not start with '/'", please see the Troubleshooting notes below.

In the above openssl command, we create a new 2048-bit RSA key pair consisting of both a private key and certificate with public key. The certificate is valid for 365 days. The -subj argument specifies the certificate identity or "Distinguished Name" (DN). For now, I've just used the CN (common name), C (country), and L (locality). We can put anything we want into this test certificate, but there are standards and Certificate Authority rules that specify standard organizational, geographical, and personal data fields. Just remember that we will need this Distinguished Name later to configure NiFi authorization for this user. Last, we used the -nodes parameter to not specify a password for the key. OpenSSL writes out two files for both the private key and the public key certificate, both in PEM encoding.

Packaging the User Certificate

Next, we need to package the key pair into an archive acceptable to your web browser. This can be a point of confusion because browsers prompt you for certificate files by file extensions like .pem, .cer, or .pfx, without clarifying what the actual storage format should be. I recommend he PKCS-12 format because it is widely acceptable to browsers, typically saved as a .pfx file. Our PKCS-12 file is protected with a password to prevent casual misuse.

openssl pkcs12 -inkey admin-private-key.pem -in admin-cert.pem -export -out admin-q-user.pfx -passout pass:"SuperSecret"

Attaching the Certificate to a Browser

Last, we will import the PFX file into our browser or OS store for a user's personal keys. Pretty much every web browser has a utility for managing certificates found via "Settings", "Advanced Settings", "Security". In Google Chrome, for example, follow these steps:

  1. Select Settings from the menu
  2. Scroll down to the bottom and click Show advanced settings...
  3. Find the HTTPS/SSL heading and click the button for Manage certificates...
  4. On the Personal tab, click the Import... button Managing Certificates in Google Chrome
  5. Go through the wizard to import your .PFX file. You will be prompted to enter the passphrase we created the .PFX file with earlier. Import the certificate to the "Personal" store.

Chrome Certificate Imported

Our test admin user is now ready to log into a NiFi server, which we will now configure.

Configuring the Server

Our NiFi server needs three things

  1. KeyStore
  2. TrustStore
  3. Authorization settings

Creating the KeyStore

The server's KeyStore is an archive containing the private key and public key certificate identifying the server. We can create a KeyStore file with a key pair in one go using the JDK's Keytool utility.

keytool -genkeypair -alias nifiserver -keyalg RSA -keypass SuperSecret -storepass SuperSecret -keystore server_keystore.jks -dname "CN=Test NiFi Server" -noprompt

This creates the KeyStore file in Java KeyStore format (.jks). Both our KeyStore and our key are protected by our super-secret password. As we did above with the admin user, we have taken gross liberties with the Distinguished Name (-dname argument). Typically, the Common Name portion of the Distinguished Name should match the public DNS name of the server. Since we are just testing NiFi configuration, this is not important.

Creating the TrustStore

The TrustStore contains public key certificates for users trusted to log in to this NiFi server in any way. Detailed role authorization will be configured separately below.

First, upload or copy the admin users's public key certificate file, admin-cert.pem to your server.

Next, we'll use the keytool utility again to create a TrustStore archive with this certificate.

keytool -importcert -v -trustcacerts -alias admin -file admin-cert.pem -keystore server_truststore.jks  -storepass SuperSecret -noprompt

And we get a bit of output:

Certificate was added to keystore
[Storing server_truststore.jks]

Authorization

NiFi's authorization data matches standard roles with specific users. This information is stored in conf/authorized-users.xml. The file comes pre-populated with helpful comments, we'll fill in the template with edits for our admin user:

<users>
    <user dn="L=Seattle, C=US, CN=Admin Q. User">
        <role name="ROLE_ADMIN"/>
        <role name="ROLE_DFM"/>
    </user>
</users>

Our admin user is identified by Distinguished Name, the same data we gave to OpenSSL's -subj parameter above, but reformated as comma-delimited pairs. The ROLE_ADMIN permissions are specific to editing permissions for users, rather than an all-powerful-super-user flag. So we'll also give our admin ROLE_DFM to allow manipulation of the data flow.

Configuration for the KeyStore and TrustStore

Now we need to fill in the appropriate sections of our nifi.properties configuration file for this KeyStore:

# security properties #
...
nifi.security.keystore=server_keystore.jks
nifi.security.keystoreType=JKS
nifi.security.keystorePasswd=SuperSecret
nifi.security.keyPasswd=SuperSecret
nifi.security.truststore=server_truststore.jks
nifi.security.truststoreType=JKS
nifi.security.truststorePasswd=SuperSecret
...

Also, we need to edit the web configuration to reflect HTTPS, by commenting out HTTP port 8080 and configuring HTTPS on port 8443 (typically):

...
# web properties #
nifi.web.war.directory=./lib
nifi.web.http.host=
#nifi.web.http.port=8080
nifi.web.https.host=
nifi.web.https.port=8443
nifi.web.jetty.working.directory=./work/jetty
nifi.web.jetty.threads=200
...

Testing it Out

That should do it, let's test it out. After editing the security settings, we should watch our logs/nifi-app.log carefully to make sure NiFi is happy with our settings.

./nifi.sh start; tail -f ../logs/nifi-app.log

When the dust settles, we can see that the server has started successfully and is waiting for clients:

...
2016-02-12 20:40:43,021 INFO [main] org.eclipse.jetty.server.Server Started @27313ms
2016-02-12 20:40:44,577 INFO [main] org.apache.nifi.web.server.JettyServer NiFi has started. The UI is available at the following URLs:
2016-02-12 20:40:44,577 INFO [main] org.apache.nifi.web.server.JettyServer https://172.31.44.214:8443/nifi
2016-02-12 20:40:44,578 INFO [main] org.apache.nifi.web.server.JettyServer https://127.0.0.1:8443/nifi
2016-02-12 20:40:44,579 INFO [main] org.apache.nifi.BootstrapListener Successfully initiated communication with Bootstrap
2016-02-12 20:40:44,579 INFO [main] org.apache.nifi.NiFi Controller initialization took 13969558566 nanoseconds.

One thing to note in the output above is that the URLs should reflect the HTTPS protocol and the port we configured above. If there is a firewall between you and your server, you need to make sure port 8443 is open. I had to update my EC2 Security Group to allow 8443 instead of 8080.

When you open NiFi in your browser at https://yournifiserver:8443/nifi, you will be prompted with the following scary-looking message:

Chrome: Your connection is not private

In this case, this nasty-gram does not mean your credit card has been stolen, you are just using a self-signed certificate that your browser does not trust. We can accept this shortcoming and proceed.

And... we're in! We are now securely logged into NiFi as our admin user. You can see the Admin Q. User common name in the top-right toolbar.

Logged in to NiFi

Wrapping Up

I hope you found it helpful to take a hands-on walkthrough of NiFi security configuration. Of course, our test setup leaves many things to be desired

  • Trust - proper TLS/SSL security involves a trust chain, where clients and servers have some mutual basis for trusting each other, and users are not trained to robotically click through those dire security warning dialogs.
  • Key and Certificate Formats - There are a baffling array of certificate file formats, extensions, and encodings. It's pretty easy to generate a self-signed cert, things get much more interesting when trust chains are involved.
  • Cloud Deployment - This type of manual configuration is not a great fit for a short-lived cloud server.

We'll come back to these issues in future posts.

Further Reading

Some other resources to check out:


Troubleshooting Notes

OpenSSL errors in Git Bash on Windows

Using MinGW64 or Git Bash on Windows, the -subj parameter to OpenSSL should be modified according to this StackOverflow answer. The modification works around the built-in Windows->Unix path conversion logic. OpenSSL receives a converted subject and that results in the "Subject does not start with '/'" error.

openssl req -x509 -newkey rsa:2048 -keyout admin-private-key.pem -out admin-cert.pem -days 365 -subj "//CN=Admin Q. User\C=US\L=Seattle" -nodes

Distinguished Name Ordering

While testing out my own instructions I ran into an unexpected issue on my first test of the completed setup. I was prompted by the "Submit Justification" dialog below:

NiFi Submit Justification

I was just about to write a 500-character justification for my existence, but I eventually figured out this was not a philosophical question, but a request for access form. I believe this was caused by a mismatch between the client Distinguished Name read by the server from the certificate vs. the way I had transcribed it into the authorized_users.xml file. I changed the order of the fields in authorized_users.xml to match the error message that followed this dialog, and it worked. This needs more experimentation and perhaps a separate post, I'm not confident in my understanding here.