Apache HDFS (Hadoop Distributed File System) is a file system designed to store large files, block-by-block distributed among the nodes of a computing cluster.
Apache Hive is a database management system based on the Hadoop platform.
Apache HBase is an open source NoSQL DBMS project of the Hadoop ecosystem.
Apache KNOX, a REST API and application gateway for Apache Hadoop ecosystem components, provides a single access point for all HTTP connections to Apache Hadoop clusters and a Single Sign On (SSO) system for services and user interface of Apache Hadoop components.
Apache Ranger is a framework for providing, monitoring and managing end-to-end data security on the Hadoop platform
Translation of a 2015 post by Securing hdfs hive hbase knox ranger . I could not find a better and newer post.
Introduction
Apache Ranger provides a comprehensive approach to securing a Hadoop cluster. It provides centralized administration of security policies for key enterprise security requirements, including authorization, accounting, and data protection.
Apache Ranger extends core functionality for coordinated use across Hadoop workloads, including batch, interactive SQL, and real-time Hadoop.
In this tutorial, we will look at using Apache Ranger for HDP 2.3 to secure your Hadoop environment. We will cover the following topics:
Support for Knox authorization and auditing
Command line policies in Hive
Command line policies in HBase
REST API for Policy Manager
Prerequisites
Hortonworks Sandbox, SSH:
Knox LDAP
Ambari http://localhost:8080/ ( — admin admin ) Knox .
Service Actions
Start
.
Knox :
Service Actions
Knox
Start Demo LDAP
.
- LDAP :
Knox
, Ranger http://localhost:6080/ -. — admin
, — admin
.
,
sudo service ranger-admin start
sandbox_knox Knox Ranger Administration Portal
, .
Knox, global knox allow
.
Sandbox for Guest
Ranger Admin .
Sandbox for Guest
SSHd ( Sandbox) CURL WebHDFS.
curl -k -u admin:admin-password 'https://127.0.0.1:8443/gateway/knox_sample/webhdfs/v1?op=LISTSTATUS'
Ranger Policy Manager → Audit screen , () Knox.
CURL, guest
.
curl -k -u guest:guest-password 'https://127.0.0.1:8443/gateway/knox_sample/webhdfs/v1?op=LISTSTATUS'
{"FileStatuses":{"FileStatus":[{"accessTime":0,"blockSize":0,"childrenNum":0,"fileId":16393,"group":"hadoop","length":0,"modificationTime":1439987528048,"owner":"yarn","pathSuffix":"app-logs","permission":"777","replication":0,"storagePolicy":0,"type":"DIRECTORY"},{"accessTime":0,"blockSize":0,"childrenNum":4,"fileId":16389,"group":"hdfs","length":0,"modificationTime":1439987809562,"owner":"hdfs","pathSuffix":"apps","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"},{"accessTime":0,"blockSize":0,"childrenNum":1,"fileId":17000,"group":"hdfs","length":0,"modificationTime":1439989173392,"owner":"hdfs","pathSuffix":"demo","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"},{"accessTime":0,"blockSize":0,"childrenNum":1,"fileId":16398,"group":"hdfs","length":0,"modificationTime":1439987529660,"owner":"hdfs","pathSuffix":"hdp","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"},{"accessTime":0,"blockSize":0,"childrenNum":1,"fileId":16394,"group":"hdfs","length":0,"modificationTime":1439987528532,"owner":"mapred","pathSuffix":"mapred","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"},{"accessTime":0,"blockSize":0,"childrenNum":2,"fileId":16396,"group":"hadoop","length":0,"modificationTime":1439987538099,"owner":"mapred","pathSuffix":"mr-history","permission":"777","replication":0,"storagePolicy":0,"type":"DIRECTORY"},{"accessTime":0,"blockSize":0,"childrenNum":1,"fileId":16954,"group":"hdfs","length":0,"modificationTime":1439988741413,"owner":"hdfs","pathSuffix":"ranger","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"},{"accessTime":0,"blockSize":0,"childrenNum":3,"fileId":16386,"group":"hdfs","length":0,"modificationTime":1440165443820,"owner":"hdfs","pathSuffix":"tmp","permission":"777","replication":0,"storagePolicy":0,"type":"DIRECTORY"},{"accessTime":0,"blockSize":0,"childrenNum":8,"fileId":16387,"group":"hdfs","length":0,"modificationTime":1439988397561,"owner":"hdfs","pathSuffix":"user","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"}]}}
Ranger Policy Manager → Audit screen.
Ranger Knox , Knox, , Ranger.
Knox Ranger, (WebHDFS, WebHCAT . .) , / IP-
grant/revoke Hive
Ranger grant
/revoke
, Hue for Hive
. Ranger , , Hive .
Hive Ranger.
Grant
, Hive
. beeline,
beeline -u "jdbc:hive2://sandbox.hortonworks.com:10000/default" -n it1 -p it1-d org.apache.hive.jdbc.HiveDriver
GRANT
grant select, update on table xademo.customer_details to user network1;
:
Ranger Administration Portal → Audit
, it1
.
Ranger it1
. Ranger , .
beeline , .
GRANT select, update on table xademo.customer_details to user network1;
, , / Ranger Admin Portal → Policy Manager. , , .
?
Ranger GRANT/REVOKE
Hive
. Hive
(Hiveserver2
).
GRANT
REVOKE
.
grant/revoke HBase
Ranger grant/revoke
, Hbase
. Hive, Ranger Hbase .
, , HBase Ambari — http://127.0.0.1:8080 ( admin
).
, Service Actions
.
Grant Hbase.
HBase Global Allow
Ranger Administration Portal — .
HBase it1
su - it1 [it1@sandbox ~]$ hbase shell
grant, mktg1
, iemployee
.
hbase(main):001:0> grant 'mktg1', 'RWC', 'iemployee'
, :
Ranger Administration Portal→ Policy Manager , admin
it1
.
HBase.
hbase(main):006:0> grant 'mktg1', 'RWC', 'iemployee' 0 row(s) in 0.8670 seconds
HBase Ranger Policy Administration. iemployee
, .
, Ranger. HBase
hbase(main):007:0> revoke 'mktg1', 'iemployee' 0 row(s) in 0.4330 seconds
,
?
Ranger GRANT/REVOKE
Hbase . .
GRANT
REVOKE .
REST APIs
Ranger REST API. API , , .
REST API
CURL. API hadoopdev-testing-policy2
HDFS sandbox_hdfs
curl -i --header "Accept:application/json" -H "Content-Type: application/json" --user admin:admin -X POST http://127.0.0.1:6080/service/public/api/policy -d '{ "policyName":"hadoopdev-testing-policy2","resourceName":"/demo/data/test","description":"Testing policy for /demo/data/test","repositoryName":"sandbox_hdfs","repositoryType":"HDFS","permMapList":[{"userList":["mktg1"],"permList":["Read"]},{"groupList":["IT"],"permList":["Read"]}],"isEnabled":true,"isRecursive":true,"isAuditEnabled":true,"version":"0.1.0","replacePerm":false}'
hadoopdev-testing-policy2
.
URL- http://127.0.0.1:6080/index.html#!/hdfs/1/policy/26
.
CURL, API.
curl -i --user admin:admin -X GET http://127.0.0.1:6080/service/public/api/policy/26
?
REST API. API , REST API Ranger.
Hopefully, on this dizzying tour of Ranger, you have experienced the simplicity and power of Ranger for safety management.