Skip to main content

Pig step-by-step installation with integrated HCatalog

By Dinesh
1) Download tar file "pig-0.13.0.tar.gz"

2)  Gunzip and Untar the file at /opt/ds/app/pig-0.13.0

 3) Change directory to /opt/ds/app/pig-0.13.0/conf

 4) Create log4j.properties from template file

5) Update pig.properties for HCatalog. For example:

 hcat.bin=/opt/ds/app/hive-0.13.0/hcatalog/bin/hcat

 6) Edit .bashrc

export PIG_HOME=/opt/ds/app/pig-0.13.0
export PATH=$PATH:$PIG_HOME/bin
export HCAT_HOME=/opt/ds/app/hive-0.13.0/hcatalog
export PATH=$PATH:$HCAT_HOME/bin

 7) It is assumed that you have already set  HADOOP_HOME, JAVA_HOME, HADOOP_COMMON_LIB_NATIVE_DIR, HADOOP_OPTS, YARN_OPTS
8) Optionally, you can create .pigbootup in User home directory

9) Execute command from user home directory

 > source .bashrc

10) Execute

>pig -useHCatalog
11) Say you had created a table in Hive with name "hivetesting". Now, try to load with below command to verify installation.

grunt> A  = LOAD 'hivetesting' USING org.apache.hcatalog.pig.HCatLoader();
grunt> describe A;

==========

 Some times you may get problem after installing Pig like below:-

java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected
 at org.apache.hcatalog.common.HCatUtil.checkJobContextIfRunningFromBackend(HCatUtil.java:88)
 at org.apache.hcatalog.pig.HCatLoader.setLocation(HCatLoader.java:162)
 at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:540)
 at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:322)
 at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:199)
 at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:277)
 at org.apache.pig.PigServer.launchPlan(PigServer.java:1367)
 at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1352)
 at org.apache.pig.PigServer.execute(PigServer.java:1341)




Many blogs suggest you to recompile the Pig by executing command:


    

  • ant clean jar-all -Dhadoopversion=23
    • 









or recompile piggybank.jar by executing below steps






    

  • cd contrib/piggybank/java
    • 

  • ant clean
    • 

  • ant -Dhadoopversion=23
    • 





But this may not solve your problem big time. The actual cause here is related to HCatalog. Try updating it!!. In my case, I was using Hive0.13 and Pig.0.13. And I was using HCatalog provided with Hive0.13.






Then I updated Pig to 0.15 and used separate hive-hcatalog-0.13.0.2.1.1.0-385 library jars. And problem was resolved....






Because later I identified it was not Pig who was creating problem rather it was Hive-HCatalog libraries. Hope this may help.   












































Comments











Post a Comment



















Popular posts



















Read from a hive table and write back to it using spark sql












By


Dinesh




















 In context to Spark 2.2 -   if we read from an hive table and write to same, we get following exception-   scala >  dy . write . mode ( "overwrite" ). insertInto ( "incremental.test2" )               org . apache . spark . sql . AnalysisException :  Cannot  insert overwrite into table that is also being read from .;  org . apache . spark . sql . AnalysisException :  Cannot  insert overwrite into table that is also being read from .;     1. This error means that our process is reading from same table and writing to same table.  2. Normally, this should work as process writes to directory .hiveStaging...   3. This error occurs in case of saveAsTable method, as it overwrites entire table instead of individual partitions.  4. This error should not occur with insertInto method, as it overwrites partitions not the table.  5. A reason why this happening is because Hive table has following Spark TBLProperties in its definition. This problem will solve for insertInto met




















Post a Comment









Read more

































Hive Parse JSON with Array Columns and Explode it in to Multiple rows.












By


Dinesh




















 Say we have a JSON String like below -  { "billingCountry":"US" "orderItems":[       {          "itemId":1,          "product":"D1"       }, 	   {          "itemId":2,          "product":"D2"       }    ] } And, our aim is to get output parsed like below -          itemId          product              1          D1              2          D2           First, We can parse JSON as follows to get JSON String get_json_object(value, '$.orderItems.itemId') as itemId get_json_object(value, '$.orderItems.product') as product Second, Above will result String value like "[1,2]". We want to convert it to Array as follows - split(regexp_extract(get_json_object(value, '$.orderItems.itemId'),'^\\["(.*)\\"]$',1),'","') as itemId split(regexp_extract(get_json_object(value, '$.orderItems.product'),'^\\["(.*)\\"]$',1),&




















Post a Comment









Read more

































Caused by: java.lang.UnsupportedOperationException: org.apache.parquet.column.values.dictionary.PlainValuesDictionary$PlainIntegerDictionary












By


Dinesh




















  Exception -    Caused by: java.lang.UnsupportedOperationException: org.apache.parquet.column.values.dictionary.PlainValuesDictionary$PlainIntegerDictionary  at org.apache.parquet.column.Dictionary.decodeToBinary(Dictionary.java:44)  at org.apache.spark.sql.execution.vectorized.ColumnVector.getUTF8String(ColumnVector.java:645)  at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)   Analysis - This might occur because of data type mismatch between Hive Table & written Parquet file.   Solution - Correct the data type to match between Hive Table & Parquet   




















Post a Comment









Read more

































org.apache.spark.sql.AnalysisException: Cannot overwrite a path that is also being read from.;












By


Dinesh




















  Caused by: org.apache.spark.sql.AnalysisException: Cannot overwrite a path that is also being read from.; at org.apache.spark.sql.execution.command.DDLUtils$.verifyNotReadPath(ddl.scala:906) at org.apache.spark.sql.execution.datasources.DataSourceAnalysis$$anonfun$apply$1.applyOrElse(DataSourceStrategy.scala:192) at org.apache.spark.sql.execution.datasources.DataSourceAnalysis$$anonfun$apply$1.applyOrElse(DataSourceStrategy.scala:134) at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$2.apply(TreeNode.scala:267) at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$2.apply(TreeNode.scala:267) at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:70) at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:266) at org.apache.spark.sql.catalyst.trees.TreeNode.transform(TreeNode.scala:256) at org.apache.spark.sql.execution.datasources.DataSourceAnalysis.apply(DataSourceStrategy.scala:134) at org.apache.spark.sql.execution.datasource




















Post a Comment









Read more

































Hadoop Distcp Error Duplicate files in input path












By


Dinesh




















  One may face following error while copying data from one cluster to other, using Distcp  Command: hadoop distcp -i {src} {tgt} Error: org.apache.hadoop.toolsCopyListing$DulicateFileException: File would cause duplicates. Ideally there can't be same file names. So, what might be happening in your case is you trying to copy partitioned table from one cluster to other. And, 2 different named partitions have same file name. Your solution is to correct Source path  {src}  in your command, such that you provide path uptil partitioned sub directory, not the file. For ex - Refer below : /a/partcol=1/file1.txt /a/partcol=2/file1.txt If you use  {src}  as  "/a/*/*"  then you will get the error  "File would cause duplicates." But, if you use  {src}  as  "/a"  then you will not get error in copying.




















Post a Comment









Read more