Clever Geek Handbook

Big Data Tools EAP 10: SSH Tunnels, Application Filtering, Custom Modules, and More

A new version of the Big Data Tools plugin has just been released - a plugin for IntelliJ IDEA Ultimate, DataGrip and PyCharm that provides integration with Hadoop and Spark, allows you to edit and run interactive notebooks in Zeppelin.



The main goal of this release is to fix as many problems as possible and improve the plugin from the inside, but two important improvements are visible to the naked eye:



  • Connecting to Hadoop and Spark is now possible via SSH tunnels, created by a couple of mouse clicks;
  • Hadoop monitoring can limit the amount of data loaded when viewing the list of applications.




SSH tunnels



Often, the server we need is not directly accessible, for example, if it is located inside a protected corporate circuit or is closed by special rules on the firewall. To get inside, you can use some kind of tunnel or VPN. The simplest of the tunnels, which is always at hand, is SSH.



You can create a tunnel with a single console command:



ssh -f -N -L 1005:127.0.0.1:8080 user@spark.server


A file will help automate the process a little ~/.ssh/config, in which you save the connection parameters once and then use:



Host spark
    HostName spark.server
    IdentityFile ~/.ssh/spark.server.key
    LocalForward 1005 127.0.0.1:8080
    User user


Now it is enough to write in the console ssh -f -N spark- and the tunnel will rise by itself, without entering IP addresses. Conveniently.



.



-, - . -f -N -L? , — ? ? , , .



-, , IDE. , , , IDE .



, Big Data Tools SSH-.





, , SSH.



Enable tunneling :



  • Zeppelin
  • HDFS
  • Hadoop
  • Spark Monitoring


, , SSH. : , , , .



, . - , SSH, VPN .



BDIDE-1063 YouTrack.





, Spark Monitoring , , .





, .



Big Data Tools , . , .



, .



BDIDE-1077.



Zeppelin



Zeppelin JAR-. Big Data Tools , IDE .



Zeppelin, Big Data Tools . , , . Big Data Tools , IntelliJ IDEA.



Maven JAR-. , -, .



. "User dependencies":





BDIDE-1087.





Big Data Tools — , . , . EAP 10 , Spark Monitoring.





More articles:


All Articles