Install and Configure MiNiFi agents

MiNiFi logo from Google Photos

In this article, we are going to learn how to install and configure Cloudera MiNiFi agents to communicate with a secured NiFi in a Hadoop cluster with auto-TLS.

Auto-TLS greatly simplifies the process of enabling and managing TLS encryption on your cluster.
Auto-TLS automates the creation of an internal certificate authority (CA) and deployment of certificates across all cluster hosts. It can also automate the distribution of existing certificates, such as those signed by a public CA. Adding new cluster hosts or services to a cluster with auto-TLS enabled automatically creates and deploys the required certificates. (From Cloudera website)

Before we begin, a few things you need to be familiar with:

MiNiFi is a java agent (a service that runs on a server) and is responsible for data collection and manipulation at the source of its creation. It is also responsible to forward that data to different destinations, like NiFi servers, for further processing.
MiNiFi is managed by the Edge Flow Management (EFM), which is a UI for managing MiNiFi agents and dataflows.

In my everyday work I encountered the need to send data to NiFi with MiNiFi. Unfortunately, I found insufficient information for the installation and configuration of MiNiFi as it is a quite new product of Cloudera.

Installing MiNiFi:
Before you begin with the installation, make sure you have the following setup:
# Java installed on the desired MiNiFi server (I have installed java 1.8 OpenJDK)
# Date and Time on the server synchronized with a NTP server or synchronized with the current time, using the command below:

$ ln -sf /usr/share/zoneinfo/{STATE}/{CITY} /etc/localtime

To install the agent, first thing you need to do is search in Cloudera website for the CEM packages and download the MiNiFi Java package (usually a tar.gz file).
Notice: you have to create a Cloudera account.
Move the package into the desired path in the server (the path you want the MiNiFi to be installed) and unzip it with the following command:

$ tar -xf /path/to/minifi/package.tar.gz

Go to /{MiNiFi_WORK_DIR}/bin directory and run the following to install the agent as a service:

$ ./minifi.sh install

Configuring MiNiFi:

In order to configure MiNiFi to communicate with secured NiFi, we need to understand a few things first.

Keystore is a file that contains the private key of the server in order to be able to establish secured connection with other secured components. On MiNiFi servers you can store it in a cert directory, inside of MiNiFi work directory.

Truststore is a file that contains certificates of other servers that need to be in contact with the MiNiFi agent (servers the MiNiFi needs to trust them). On MiNiFi servers you can store it in a cert directory, inside of MiNiFi work directory.

In my cluster, I have auto-TLS enabled, so I need my truststore to have the ROOT CA certificate. I am taking the truststore (JKS type) from the EFM configuration and copying it to the path /{MiNiFi_WORK_DIR}/cert that I created (make sure you have the truststore password).

Generating a Keystore:

generate a private key for the server:

$ openssl genrsa -aes256 -out {SERVER_NAME-key}.pem 2048 

Pay Attention: you will be prompt to insert some info about the server, keep the passphrase you choose for later.

generate a csr using the private key created:

$ openssl req -new -key {SERVER_NAME-key}.pem -out {SERVER_NAME}.csr

sign your csr on your own with openssl (if possible) or send it to the authority that signs certificates.

I have a p7b signed certificate, so I will convert it to pem file with the following command:
NOTICE: if you have a pem file, skip this part.

$ openssl pkcs7 -print_certs -in {SIGNED_CERT}.p7b -out {SERVER_NAME-cert}.pem

To create your server’s keystore, run the following:

$ openssl pkcs12 -export -in {SERVER_NAME-cert}.pem -inkey {SERVER_NAME-key}.pem -out {SERVER_NAME} -passin pass:{passphrase from previous commands} -passout pass:{passphrase from previous commands}
$ keytool -importkeystore -srckeystore {SERVER_NAME}.p12 -srcstoretype PKCS12 -srcstorepass {PASSPHRASE} -alias {SERVER_NAME} -deststorepass {PASSPHRASE} -destkeypass {PASSPHRASE} -destkeystore {SERVER_NAME-keystore}.jks

In the next step we will use the keystore we created.

Go to /{MiNiFi_WORK_DIR}/conf directory and modify the following parameters:

run.as={USER_TO_RUN_WITH}
nifi.minifi.security.keystore={/path/to/keystore.jks}
nifi.minifi.security.keystoreType=JKS
nifi.minifi.security.keystorePasswd={KEYSTORE_PASSPHRASE}
nifi.minifi.security.truststore={/path/to/truststore.jks}
nifi.minifi.security.truststoreType=JKS
nifi.minifi.security.truststorePaaswd={TRUSTSTORE_PASSWORD}
nifi.minifi.security.ssl.protocol=TLS
nifi.minifi.sensitive.props.key=”{UNIQUE_STRING_12_CHARS}”
nifi.minifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
nifi.minifi.sensitive.props.provider=BC
nifi.c2.enabled=true
nifi.c2.rest.url=https://{EFM_SERVER}:10080/efm/api/c2-protocol/heartbeat
nifi.c2.rest.url.ack=https://{EFM_SERVER}:10080/efm/api/c2-protocol/acknowledge
nifi.c2.agent.class={AGENT_CLASS}
nifi.c2.agent.identifier={AGENT_ID} # should be unique for every minifi
nifi.c2.security.truststore.location={/path/to/truststore.jks}
nifi.c2.security.truststore.password={TRUSTSTORE_PASSWORD}
nifi.c2.security.truststore.type=JKS
nifi.c2.security.keystore.location={/path/to/keystore.jks}
nifi.c2.security.keystore.password={KEYSTORE_PASSPHRASE}
nifi.c2.security.keystore.type=JKS
nifi.c2.security.need.client.auth=true

After configuring Everything, run he following command to start MiNiFi agent:

$ service minifi start

Check the logs to see if the agent has been started successfully.

On the EFM, check to see if the agent has been connected, create your dataflow and press publish to push it to MiNiFi config.yml file.