Introduction
Let me explain my accordion right away. Yes, the internet is full of manuals. Yes, full of turn-based walkthroughs. Yes, you can say that everything is chewed chewed. But specifically in my case, as it always happens, there were a handful of "buts":
There are manuals on how to set up a NiFi and NiFi Registry bundle with authentication and authorization enabled. But ... self-signed certificates are used.
There are separate manuals on how to screw a commercial certificate for NiFi; respectively for the NiFi Registry "kagbe is the same". But mutual authentication and authorization will occur with the use of Two way SSL ... and we have LDAP ... and then you will not be able to ensure the coherence of the sweet couple using only the external directory on your bare intuition.
There are manuals for connecting with LDAP for both NiFi and NiFi Registry. Nooo ... as in the previous "but", questions arise, how to get by with only LDAP, because we still have NiFi Cli, but he does not know how to do LDAP.
In other words, all manuals have a small nuance: they cover only the simplest scenarios. There is simply no documentation for complex bundles. Moreover, in the course of setting up the bundle, I encountered difficulties that are encountered only a few times in the bourgeois web and they are all either without answers, or the answers are not relevant.
What we have and what we want to get
We have:
Commercial wildcard SSL / TLS certificate;
A directory server that can do LDAP (in my case, Apache DS);
The NiFi server on which the development is taking place;
NiFi server in a grocery environment;
NiFi Registry Server.
We want:
Authentication, authorization and HTTPS. Moreover, NiFi does not allow you to enable built-in authentication mechanisms without HTTPS;
Centralized user management on a directory server;
Versioning flow and process-group in a manner similar to gith;
.
:
Apache DS;
Java Keystore Java Truststore, HTTPS;
LDAP NiFi;
LDAP NiFi Registry;
;
NiFi NiFi Registry;
NiFi NiFi Cli.
wildcard : *.love.you;
FQDN Apache DS: auth.love.you;
Apache DS: uid=admin,ou=system; : secret;
Apache DS love.you: dc=love,dc=you;
Apache DS nifiAdmin nifiUser NiFi NiFi Registry;
Apache DS nifiAdmins nifiUsers NiFi NiFi Registry;
FQDN NiFi, : nifi-deve.love.you;
FQDN NiFi Registry: nifi-re.love.you;
FQDN () NiFi: nifi.love.you;
, NiFi: $NIFI_HOME;
, NiFi Registry: $NIFI_REGISTRY_HOME;
, NiFi Toolkit: $NIFI_TOOLKIT_HOME;
, : /love/conf/cert;
PKCS12, Java Keystore Java Truststore:
sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
Apache DS
, , Apache DS . . Azure AD, " ", NiFi, AD' . Apache DS , , Apache.
10389 . Apache Directory Studio. , , .
New connection:
:
: uid=admin,ou=system secret. Check Authentication:
.
. - member objectclass inetOrgPerson. LDAP Browser' ou=schema Reload Entry:
cn=inetorgperson ou=objectClasses. m-oid=2.16.840.1.113730.3.2.2. Open With -> LDIF Entry Editor:
m-may: member Ctrl+S:
Open Configuration:
Advanced Partition configuration...
Add, ID Suffix. Ctrl+S:
: . . . :
. - LDIF . - import.ldif .
import.ldif
dn: ou=people,dc=love,dc=you
objectclass: organizationalUnit
objectClass: extensibleObject
objectclass: top
ou: people
dn: ou=groups,dc=love,dc=you
objectclass: organizationalUnit
objectClass: extensibleObject
objectclass: top
ou: groups
dn: cn=nifiUsers,ou=groups,dc=love,dc=you
objectClass: groupOfUniqueNames
objectClass: top
cn: nifiUsers
uniqueMember: cn=nifiUser,ou=people,dc=love,dc=you
dn: cn=nifiAdmins,ou=groups,dc=love,dc=you
objectClass: groupOfUniqueNames
objectClass: top
cn: nifiAdmins
uniqueMember: cn=nifiAdmin,ou=people,dc=love,dc=you
dn: cn=nifiUser,ou=people,dc=love,dc=you
objectclass: inetOrgPerson
objectclass: organizationalPerson
objectclass: person
objectclass: top
cn: nifiUser
description: A nifiUser user
sn: nifiUser
uid: nifiUser
mail: nifiUser@love.you
userpassword: password
member: cn=nifiUsers,ou=groups,dc=love,dc=you
dn: cn=nifiAdmin,ou=people,dc=love,dc=you
objectclass: inetOrgPerson
objectclass: organizationalPerson
objectclass: person
objectclass: top
cn: nifiAdmin
description: A nifiAdmin user
sn: nifiAdmin
uid: nifiAdmin
mail: nifiAdmin@love.you
userpassword: password
member: cn=nifiAdmins,ou=groups,dc=love,dc=you
dn: cn=*.love.you,ou=people,dc=love,dc=you
objectClass: person
objectClass: top
cn: *.love.you
sn: *.love.you
LDIF Import:
:
:
cn=*.love.you
, ou=people cn=*.love.you, objectClass: person:
, : , *.love.you. NiFi Registry.
Java Keystore Java Truststore
:
PEM-: fullchain.crt;
PEM-: private.key;
certificate authority (CA) PEM-: cacert.pem;
, . , .
. :
-----BEGIN CERTIFICATE----- ... ... ... -----END CERTIFICATE-----
.
, .
, , , ( ) . CA .
Java Keystore . PKCS12:
openssl pkcs12 -export -out nifi.p12 -inkey private.key -in fullchain.crt -name nifi-key
PKCS12 . .
Java Keystore PKCS12 :
keytool -importkeystore -srckeystore nifi.p12 -srcstoretype pkcs12 -srcalias nifi-key -destkeystore keystore.jks -deststoretype jks -destalias nifi-key
, . Java Keystore, .
Java Truststore:
keytool -importcert -alias nifi-cert -file cacert.pem -keystore truststore.jks
Java Truststore. . Keystore Truststore. . .
JKS NiFi, , NiFi NiFi Registry, /love/conf/cert/;
NiFi , , $NIFI_HOME/config/nifi.properties, .
nifi.properties
# Site to Site properties
nifi.remote.input.host=nifi-deve.love.you
nifi.remote.input.secure=true
nifi.remote.input.socket.port=10443
# web properties #
nifi.web.http.host=
nifi.web.http.port=
nifi.web.https.host=nifi-deve.love.you
nifi.web.https.port=9443
# security properties #
nifi.security.keystore=/love/conf/cert/keystore.jks
nifi.security.keystoreType=jks
nifi.security.keystorePasswd=sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
nifi.security.truststore=/love/conf/cert/truststore.jks
nifi.security.truststoreType=jks
nifi.security.truststorePasswd=sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
nifi.security.allow.anonymous.authentication=false
nifi.security.user.login.identity.provider=ldap-provider
: nifi.web.http.host nifi.web.http.port. , , NiFi .
21 - , .
sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
, .
NiFi , $NIFI_HOME/config/nifi.properties, .
nifi.properties
# Site to Site properties
nifi.remote.input.host=nifi.love.you
nifi.remote.input.secure=true
nifi.remote.input.socket.port=10443
# web properties #
nifi.web.http.host=
nifi.web.http.port=
nifi.web.https.host=nifi.love.you
nifi.web.https.port=9443
# security properties #
nifi.security.keystore=/love/conf/cert/keystore.jks
nifi.security.keystoreType=jks
nifi.security.keystorePasswd=sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
nifi.security.truststore=/love/conf/cert/truststore.jks
nifi.security.truststoreType=jks
nifi.security.truststorePasswd=sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
nifi.security.allow.anonymous.authentication=false
nifi.security.user.login.identity.provider=ldap-provider
# Identity Mapping Properties #
nifi.security.identity.mapping.pattern.dn=^CN=(.*?)$
nifi.security.identity.mapping.value.dn=$1
nifi.security.identity.mapping.transform.dn=NONE
: . .
21 - , .
sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
, .
NiFi Registry, $NIFI_REGISTRY_HOME/config/nifi-registry.properties, .
nifi-registry.properties
# web properties #
nifi.registry.web.http.host=
nifi.registry.web.http.port=
nifi.registry.web.https.host=nifi-re.love.you
nifi.registry.web.https.port=8443
# security properties #
nifi.registry.security.keystore=/abc/conf/cert/keystore.jks
nifi.registry.security.keystoreType=jks
nifi.registry.security.keystorePasswd=sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
nifi.registry.security.truststore=/abc/conf/cert/truststore.jks
nifi.registry.security.truststoreType=jks
nifi.registry.security.truststorePasswd=sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
nifi.registry.security.needClientAuth=false
nifi.registry.security.identity.provider=ldap-identity-provider
# Identity Mapping Properties #
nifi.registry.security.identity.mapping.pattern.dn=^CN=(.*?)$
nifi.registry.security.identity.mapping.value.dn=$1
nifi.registry.security.identity.mapping.transform.dn=NONE
: . .
16 - , .
sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
, .
LDAP NiFi
LDAP : $NIFI_HOME/conf/login-identity-providers.xml $NIFI_HOME/conf/authorizers.xml
login-identity-providers.xml . LDAP- . .
login-identity-providers.xml
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<loginIdentityProviders>
<provider>
<identifier>ldap-provider</identifier>
<class>org.apache.nifi.ldap.LdapProvider</class>
<property name="Authentication Strategy">SIMPLE</property>
<property name="Manager DN">uid=admin,ou=system</property>
<property name="Manager Password">secret</property>
<property name="TLS - Keystore"></property>
<property name="TLS - Keystore Password"></property>
<property name="TLS - Keystore Type"></property>
<property name="TLS - Truststore"></property>
<property name="TLS - Truststore Password"></property>
<property name="TLS - Truststore Type"></property>
<property name="TLS - Client Auth"></property>
<property name="TLS - Protocol"></property>
<property name="TLS - Shutdown Gracefully"></property>
<property name="Referral Strategy">FOLLOW</property>
<property name="Connect Timeout">10 secs</property>
<property name="Read Timeout">10 secs</property>
<property name="Url">ldap://auth.love.you:10389</property>
<property name="User Search Base">ou=people,dc=love,dc=you</property>
<property name="User Search Filter">(uid={0})</property>
<property name="Identity Strategy">USE_USERNAME</property>
<property name="Authentication Expiration">12 hours</property>
</provider>
</loginIdentityProviders>
, 27 29. 27 , , LDAP-. {0} - , User NiFi; uid - , , . , nifiUser.
, . ApacheDS.
29 - : (USE_USERNAME), DN (USE_DN). , , . DN .
authorizers.xml , NiFi "" . .
authorizers.xml
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<authorizers>
<userGroupProvider>
<identifier>ldap-user-group-provider</identifier>
<class>org.apache.nifi.ldap.tenants.LdapUserGroupProvider</class>
<property name="Authentication Strategy">SIMPLE</property>
<property name="Manager DN">uid=admin,ou=system</property>
<property name="Manager Password">secret</property>
<property name="TLS - Keystore"></property>
<property name="TLS - Keystore Password"></property>
<property name="TLS - Keystore Type"></property>
<property name="TLS - Truststore"></property>
<property name="TLS - Truststore Password"></property>
<property name="TLS - Truststore Type"></property>
<property name="TLS - Client Auth"></property>
<property name="TLS - Protocol"></property>
<property name="TLS - Shutdown Gracefully"></property>
<property name="Referral Strategy">FOLLOW</property>
<property name="Connect Timeout">10 secs</property>
<property name="Read Timeout">10 secs</property>
<property name="Url">ldap://auth.love.you:10389</property>
<property name="Page Size"></property>
<property name="Sync Interval">1 mins</property>
<property name="Group Membership - Enforce Case Sensitivity">false</property>
<property name="User Search Base">ou=people,dc=love,dc=you</property>
<property name="User Object Class">person</property>
<property name="User Search Scope">ONE_LEVEL</property>
<property name="User Search Filter"></property>
<property name="User Identity Attribute">cn</property>
<property name="User Group Name Attribute">member</property>
<property name="User Group Name Attribute - Referenced Group Attribute"></property>
<property name="Group Search Base">ou=groups,dc=love,dc=you</property>
<property name="Group Object Class">groupOfUniqueNames</property>
<property name="Group Search Scope">ONE_LEVEL</property>
<property name="Group Search Filter"></property>
<property name="Group Name Attribute">cn</property>
<property name="Group Member Attribute"></property>
<property name="Group Member Attribute - Referenced User Attribute"></property>
</userGroupProvider>
<accessPolicyProvider>
<identifier>file-access-policy-provider</identifier>
<class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
<property name="User Group Provider">ldap-user-group-provider</property>
<property name="Authorizations File">./conf/authorizations.xml</property>
<property name="Initial Admin Identity">nifiAdmin</property>
<property name="Legacy Authorized Users File"></property>
<property name="Node Identity 1"></property>
<property name="Node Group"></property>
</accessPolicyProvider>
<authorizer>
<identifier>managed-authorizer</identifier>
<class>org.apache.nifi.authorization.StandardManagedAuthorizer</class>
<property name="Access Policy Provider">file-access-policy-provider</property>
</authorizer>
</authorizers>
:
31 - objectClass ApacheDS, ;
35 - , ;
39 - objectClass, ;
50 - , ( identifier: <identifier>ldap-user-group-provider</identifier> <userGroupProvider>);
52 - , - "" .
NiFi nifiAdmin , ldif-.
LDAP NiFi Registry
NiFi Registry NiFi. $NIFI_REGISTRY_HOME/conf/identity-providers.xml .
identity-providers.xml
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<identityProviders>
<provider>
<identifier>ldap-identity-provider</identifier>
<class>org.apache.nifi.registry.security.ldap.LdapIdentityProvider</class>
<property name="Authentication Strategy">SIMPLE</property>
<property name="Manager DN">uid=admin,ou=system</property>
<property name="Manager Password">secret</property>
<property name="Referral Strategy">FOLLOW</property>
<property name="Connect Timeout">10 secs</property>
<property name="Read Timeout">10 secs</property>
<property name="Url">ldap://auth.love.you:10389</property>
<property name="User Search Base">ou=people,dc=love,dc=you</property>
<property name="User Search Filter">(uid={0})</property>
<property name="Identity Strategy">USE_USERNAME</property>
<property name="Authentication Expiration">12 hours</property>
</provider>
</identityProviders>
$NIFI_REGISTRY_HOME/conf/authorizers.xml .
authorizers.xml
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<authorizers>
<userGroupProvider>
<identifier>ldap-user-group-provider</identifier>
<class>org.apache.nifi.registry.security.ldap.tenants.LdapUserGroupProvider</class>
<property name="Authentication Strategy">SIMPLE</property>
<property name="Manager DN">uid=admin,ou=system</property>
<property name="Manager Password">secret</property>
<property name="TLS - Keystore"></property>
<property name="TLS - Keystore Password"></property>
<property name="TLS - Keystore Type"></property>
<property name="TLS - Truststore"></property>
<property name="TLS - Truststore Password"></property>
<property name="TLS - Truststore Type"></property>
<property name="TLS - Client Auth"></property>
<property name="TLS - Protocol"></property>
<property name="TLS - Shutdown Gracefully"></property>
<property name="Referral Strategy">FOLLOW</property>
<property name="Connect Timeout">10 secs</property>
<property name="Read Timeout">10 secs</property>
<property name="Url">ldap://auth.love.you:10389</property>
<property name="Page Size"></property>
<property name="Sync Interval">1 mins</property>
<property name="Group Membership - Enforce Case Sensitivity">false</property>
<property name="User Search Base">ou=people,dc=love,dc=you</property>
<property name="User Object Class">person</property>
<property name="User Search Scope">ONE_LEVEL</property>
<property name="User Search Filter"></property>
<property name="User Identity Attribute">cn</property>
<property name="User Group Name Attribute">member</property>
<property name="User Group Name Attribute - Referenced Group Attribute"></property>
<property name="Group Search Base">ou=groups,dc=love,dc=you</property>
<property name="Group Object Class">groupOfUniqueNames</property>
<property name="Group Search Scope">ONE_LEVEL</property>
<property name="Group Search Filter"></property>
<property name="Group Name Attribute">cn</property>
<property name="Group Member Attribute"></property>
<property name="Group Member Attribute - Referenced User Attribute"></property>
</userGroupProvider>
<accessPolicyProvider>
<identifier>file-access-policy-provider</identifier>
<class>org.apache.nifi.registry.security.authorization.file.FileAccessPolicyProvider</class>
<property name="User Group Provider">ldap-user-group-provider</property>
<property name="Authorizations File">./conf/authorizations.xml</property>
<property name="Initial Admin Identity">nifiAdmin</property>
<property name="NiFi Group Name"></property>
</accessPolicyProvider>
<authorizer>
<identifier>managed-authorizer</identifier>
<class>org.apache.nifi.registry.security.authorization.StandardManagedAuthorizer</class>
<property name="Access Policy Provider">file-access-policy-provider</property>
</authorizer>
</authorizers>
NiFi Registry nifiAdmin , ldif-.
Identity Mapping Properties nifi.properties nifi-registry.properties.
, NiFi NiFi Registry.
, "Proxy user requests". , , . , " " " ", , . , , . , . . .
? , NiFi NiFi Registry , . NiFi Registry NiFi , NiFi Java Keystore , , NiFi Registry. , wildcard , CN=*.love.you.
!
, NiFi . , , CN . . , CN cn. NiFi .
. ApacheDS cn . . . : . NiFi , : nifiUser, nifiAdmin *.love.you. cn=....
, , , , .
, , ....mapping.pattern.dn capture group, ....mapping.value.dn . CN=*.love.you *.love.you, ApacheDS.
NiFi NiFi Registry
NiFi . , , NiFi . , .
Operate:
Policies , :
, *.love.you proxy user requests:
Controller Settings:
REGISTRY CLIENTS:
, + , , :
NiFi Registry, , :
USERS :
:
Bucket Test
NiFi. process group Test. process group Version -> Start version control:
:
SAVE process group :
process group .
NiFi NiFi Cli
, process group flow Registry, () NiFi.
NiFi Toolkit NiFi .
: $NIFI_TOOLKIT_HOME/config/reg.properties $NIFI_TOOLKIT_HOME/config/nifi-prod.properties:
NiFi Registry:
reg.properties
baseUrl=https://nifi-re.love.you:8443
keystore=/love/conf/cert/keystore.jks
keystoreType=jks
keystorePasswd=sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
keyPasswd=sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
truststore=/love/conf/cert/truststore.jks
truststoreType=jks
truststorePasswd=sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
proxiedEntity=nifiAdmin
sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
, .
() NiFi:
nifi-prod.properties
baseUrl=https://nifi.love.you:9443
keystore=/love/conf/cert/keystore.jks
keystoreType=jks
keystorePasswd=sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
keyPasswd=sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
truststore=/love/conf/cert/truststore.jks
truststoreType=jks
truststorePasswd=sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
proxiedEntity=nifiAdmin
sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
, .
:
NiFi Registry ID , flow.
flow ID ID flow, () NiFi.
flow;
flow () NiFi.
$NIFI_TOOLKIT_HOME/bin :
:
./cli.sh registry list-buckets -p ../conf/reg.properties
:
# Name Id Description
- ---- ------------------------------------ -----------
1 Test 84744b22-0e2e-4342-b6ab-6658ff0e54ec (empty)
flow 84744b22-0e2e-4342-b6ab-6658ff0e54ec:
./cli.sh registry list-flows -b 84744b22-0e2e-4342-b6ab-6658ff0e54ec -p ../conf/reg.properties
:
# Name Id Description
- ------------------ ------------------------------------ ------------------
1 PG versioning test bc9a3ff4-2d9c-44b0-bd8e-d86b51d3bac6 PG versioning test
flow:
./cli.sh registry list-flow-versions -f bc9a3ff4-2d9c-44b0-bd8e-d86b51d3bac6 -p ../conf/reg.properties
:
Ver Date Author Message
--- -------------------------- --------- ------------------
1 Thu, Apr 22 2021 09:28 UTC nifiAdmin PG versioning test
flow () NiFi:
./cli.sh nifi pg-import -b 84744b22-0e2e-4342-b6ab-6658ff0e54ec -f bc9a3ff4-2d9c-44b0-bd8e-d86b51d3bac6 -fv 1 -p ../conf/nifi-prod.properties
:
f90df705-0178-1000-04fe-45ba27e76f99
- ID process group () NiFi:
, HowTo . , , (, NiFi Registry), "TRACE" NiFi NiFi Registry . , , .
As far as CI / CD is concerned, I did not draw the entire chain. Everyone has their own patterns and scenarios, and the pipeline itself would be offtopic. And the above example NiFi Cli, I think, is a good starting point for implementing your own delivery scheme.
Thanks to everyone who read it and I hope the text was helpful.