I am new to java so might be this is naive question, I have one external jar file which I want to move to Hadoop File system programatically, I have API to do so but just I need to give the path of the jar file. I don't want to go with hard coded path (both absolute and relative) maybe if I can include it along with my project or maybe something better so that same code works across different machines, its possible that same jar file might not present at the same location on different machines, if I will give hardcoded path then I need to explicitly copy the file to the same location.
I have written UDF (jar file) for Hive which is available on my disk, now to access UDF in Hive one need to add jar to HIVE classpath, so I thought to copy JAR file to HDFS from my local system and then Add jar to HIVE classpath from HDFS path.
You can put two jar files in the same directory and path would look like this
String path = System.getProperty("user.dir") + File.separator + "name.jar";
Where System.getProperty("user.dir") will return user's current working directory (from where the first jar file was run), and File.separator is the system-dependent default name-separator character.
For using a UDF on Hive, you will have to add your UDF containing jar to the distributed cache so that it becomes available to all nodes in your Hive cluster. You can do this with a syntax like the following:
add jar /path_to_jar/my_jar.jar
'/path_to_jar/' is path to the jar on your local machine from which you are running this command. The jar, if needed, can accessed from your MapReduce or Hive code by just its name after it has been added to the distributed cache.
I am new to java so might be this is naive question, I have one external jar file which I want to move to Hadoop File system programatically, I have API to do so but just I need to give the path of the jar file. I don't want to go with hard coded path (both absolute and relative) maybe if I can include it along with my project or maybe something better so that same code works across different machines, its possible that same jar file might not present at the same location on different machines, if I will give hardcoded path then I need to explicitly copy the file to the same location.
I have written UDF (jar file) for Hive which is available on my disk, now to access UDF in Hive one need to add jar to HIVE classpath, so I thought to copy JAR file to HDFS from my local system and then Add jar to HIVE classpath from HDFS path.
You can put two jar files in the same directory and path would look like this
String path = System.getProperty("user.dir") + File.separator + "name.jar";
Where System.getProperty("user.dir") will return user's current working directory (from where the first jar file was run), and File.separator is the system-dependent default name-separator character.
For using a UDF on Hive, you will have to add your UDF containing jar to the distributed cache so that it becomes available to all nodes in your Hive cluster. You can do this with a syntax like the following:
add jar /path_to_jar/my_jar.jar
'/path_to_jar/' is path to the jar on your local machine from which you are running this command. The jar, if needed, can accessed from your MapReduce or Hive code by just its name after it has been added to the distributed cache.
0 commentaires:
Enregistrer un commentaire