Skip to main content

Spark Hadoop EMR Cross Realm Access HBase & Kafka

By Dinesh

 

  • We had in-premise Hadoop Cluster which included Kafka, HBase, HDFS, Spark, YARN , etc.
  • We planned to migrate our Big Data Jobs and Data to AWS EMR but still keeping Kafka on in-premise CDP cluster.
  • After Spawning EMR on AWS. We tried running Spark Job connecting to Kafka on in-premise cluster.
    • We did setup all VPC connections & opened 2firewall ports between the two clusters.
    • But, since EMR and CDP (in-premise) had different KDC Server & principal, it kept on failing for us to connect to Kafka ( in-premise) from EMR.
    • Note, one can set following property to see Kerberos logs - 
      • -Dsun.security.krb5.debug=true
The easiest option for us were two - 
  • Setup Cross-Realm Kerberos trust. Such that EMR principal in-premise KDC Server to use kafka service. Refer - https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system-level_authentication_guide/using_trusts
  • Setup to Cross-Realm trust using same AD accounts and domain. Refer https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-kerberos-cross-realm.html
But, Our Security team did not agree on above options, which would have made developers life a lot easier. 

So, We copied in-premise Keytab on to EMR, and tried to use that to authenticate with Kafka service. We don't recommend doing it, but we had not other option. Steps are described as below.
  • Update krb5.conf  such that it aware about both the cluster domains, kdc servers, etc.

[libdefaults]

    default_realm = EMR.LOCAL
    dns_lookup_realm = false
    udp_preference_limit = 1
    dns_lookup_kdc = false
    rdns = true
    ticket_lifetime = 24h
    forwardable = true
    default_tkt_enctypes = aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96 des3-cbc-sha1
    default_tgs_enctypes = aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96 des3-cbc-sha1
    permitted_enctypes = aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96 des3-cbc-sha1

[realms]
    EMR.LOCAL = {
kdc = ip-90-110-43-74.ec2.internal:88
admin_server = ip-90-110-43-74.ec2.internal:749
default_domain = ec2.internal
    }

CDP.INPREMISE.COM = {
kdc = cdp42.cdp.inpremise.com:88
master_kdc = cdp42.cdp.inpremise.com:88
kpasswd = cdp42.cdp.inpremise.com:464
kpasswd_server = cdp42.cdp.inpremise.com:464
}

[domain_realm]
    .ec2.internal = EMR.LOCAL
     ec2.internal = EMR.LOCAL
    cdp42.cdp.inpremise.com = CDP.INPREMISE.COM
    .cdp.inpremise.com = CDP.INPREMISE.COM

[logging]
    kdc = FILE:/var/log/kerberos/krb5kdc.log
    admin_server = FILE:/var/log/kerberos/kadmin.log
    default = FILE:/var/log/kerberos/krb5lib.log


Note - details can be found here https://web.mit.edu/kerberos/krb5-1.12/doc/admin/conf_files/krb5_conf.html#domain-realm


  • Then we did write jaas.conf as below - 
KafkaClient{
  com.sun.security.auth.module.Krb5LoginModule required
  doNotPrompt=true
  useTicketCache=false
  principal="inpremiseaccount@CDP.INPREMISE.COM"
  useKeyTab=true
  serviceName="kafka"
  keyTab="inpremiseaccount.keytab"
  renewTicket=true
  storeKey=true
  client=true;
};
Client {
  com.sun.security.auth.module.Krb5LoginModule required
  useKeyTab=true
  doNotPrompt=true
  useTicketCache=false
  serviceName="hbase"
  keyTab="awsemraccount.keytab"
  principal="awsemraccount@EMR.LOCAL"
  storeKey=true
  client=true;
};

Note that - 
  1. KafkaClient has configuration to connect to in-premise kafka server running on CDP.
  2. Client on the other hand includes configuration to connect to EMR HBase service.
  3. As our target is to run a Spark job on EMR, which reads data from a different Kafka cluster and saves data to EMR HBase.
Once above was done then our Spark command looked like below - 

INPREMISEACCOUNT_KEYTAB_PATH= <My_path>/inpremiseaccount.keytab
JAAS_PATH=<My_path>/jaas.conf
TRUSTSTORE_PATH=<My_path>/trustore.jks
KRB5_PATH=<My_path>/krb5.conf
EMRACCOUNT_KEYTAB_PATH=<My_path>/awsemraccount.keytab

spark-shell --master yarn \
 --num-executors 2 \
 --conf "spark.dynamicAllocation.enabled=false" \
 --conf "spark.shuffle.service.enabled=false" \
 --jars $mylib \
 --conf spark.executor.extraJavaOptions=" -Djava.security.auth.login.config=jaas.conf -Djava.security.krb5.conf=krb5.conf" \
--driver-java-options " -Djava.security.auth.login.config=jaas.conf -Djava.security.krb5.conf=krb5.conf" \
--files "$INPREMISEACCOUNT_KEYTAB_PATH,$JAAS_PATH,$TRUSTSTORE_PATH,$KRB5_PATH" \
--conf "spark.yarn.keytab=$EMRACCOUNT_KEYTAB_PATH" \
--conf "spark.yarn.principal=awsemraccount@EMR.LOCAL"

Comments

Post a Comment

Popular posts

Hive Parse JSON with Array Columns and Explode it in to Multiple rows.

By Dinesh
 Say we have a JSON String like below -  { "billingCountry":"US" "orderItems":[       {          "itemId":1,          "product":"D1"       },   {          "itemId":2,          "product":"D2"       }    ] } And, our aim is to get output parsed like below -  itemId product 1 D1 2 D2   First, We can parse JSON as follows to get JSON String get_json_object(value, '$.orderItems.itemId') as itemId get_json_object(value, '$.orderItems.product') as product Second, Above will result String value like "[1,2]". We want to convert it to Array as follows - split(regexp_extract(get_json_object(value, '$.orderItems.itemId'),'^\\["(.*)\\"]$',1),'","') as itemId split(regexp_extract(get_json_object(value, '$.orderItems.product'),'^\\["(.*)\\"]$',1),...
Post a Comment
Read more




org.apache.spark.sql.AnalysisException: Cannot overwrite a path that is also being read from.;

By Dinesh
  Caused by: org.apache.spark.sql.AnalysisException: Cannot overwrite a path that is also being read from.; at org.apache.spark.sql.execution.command.DDLUtils$.verifyNotReadPath(ddl.scala:906) at org.apache.spark.sql.execution.datasources.DataSourceAnalysis$$anonfun$apply$1.applyOrElse(DataSourceStrategy.scala:192) at org.apache.spark.sql.execution.datasources.DataSourceAnalysis$$anonfun$apply$1.applyOrElse(DataSourceStrategy.scala:134) at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$2.apply(TreeNode.scala:267) at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$2.apply(TreeNode.scala:267) at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:70) at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:266) at org.apache.spark.sql.catalyst.trees.TreeNode.transform(TreeNode.scala:256) at org.apache.spark.sql.execution.datasources.DataSourceAnalysis.apply(DataSourceStrategy.scala:134) at org.apache.spark.sql.execution.dataso...
Post a Comment
Read more




Read from a hive table and write back to it using spark sql

By Dinesh
In context to Spark 2.2 - if we read from an hive table and write to same, we get following exception- scala > dy . write . mode ( "overwrite" ). insertInto ( "incremental.test2" ) org . apache . spark . sql . AnalysisException : Cannot insert overwrite into table that is also being read from .; org . apache . spark . sql . AnalysisException : Cannot insert overwrite into table that is also being read from .; 1. This error means that our process is reading from same table and writing to same table. 2. Normally, this should work as process writes to directory .hiveStaging... 3. This error occurs in case of saveAsTable method, as it overwrites entire table instead of individual partitions. 4. This error should not occur with insertInto method, as it overwrites partitions not the table. 5. A reason why this happening is because Hive table has following Spark TBLProperties in its definition. This problem will solve for insertInto met...
Post a Comment
Read more




Hadoop Distcp Error Duplicate files in input path

By Dinesh
  One may face following error while copying data from one cluster to other, using Distcp  Command: hadoop distcp -i {src} {tgt} Error: org.apache.hadoop.toolsCopyListing$DulicateFileException: File would cause duplicates. Ideally there can't be same file names. So, what might be happening in your case is you trying to copy partitioned table from one cluster to other. And, 2 different named partitions have same file name. Your solution is to correct Source path  {src}  in your command, such that you provide path uptil partitioned sub directory, not the file. For ex - Refer below : /a/partcol=1/file1.txt /a/partcol=2/file1.txt If you use  {src}  as  "/a/*/*"  then you will get the error  "File would cause duplicates." But, if you use  {src}  as  "/a"  then you will not get error in copying.
Post a Comment
Read more




Scala Spark building Jar leads java.lang.StackOverflowError

By Dinesh
  Exception -  [Thread-3] ERROR scala_maven.ScalaCompileMojo - error: java.lang.StackOverflowError [Thread-3] INFO scala_maven.ScalaCompileMojo - at scala.collection.generic.TraversableForwarder$class.isEmpty(TraversableForwarder.scala:36) [Thread-3] INFO scala_maven.ScalaCompileMojo - at scala.collection.mutable.ListBuffer.isEmpty(ListBuffer.scala:45) [Thread-3] INFO scala_maven.ScalaCompileMojo - at scala.collection.mutable.ListBuffer.toList(ListBuffer.scala:306) [Thread-3] INFO scala_maven.ScalaCompileMojo - at scala.collection.mutable.ListBuffer.result(ListBuffer.scala:300) [Thread-3] INFO scala_maven.ScalaCompileMojo - at scala.collection.mutable.Stack$StackBuilder.result(Stack.scala:31) [Thread-3] INFO scala_maven.ScalaCompileMojo - at scala.collection.mutable.Stack$StackBuilder.result(Stack.scala:27) [Thread-3] INFO scala_maven.ScalaCompileMojo - at scala.collection.generic.GenericCompanion.apply(GenericCompanion.scala:50) [Thread-3] INFO scala_maven.ScalaCompile...
Post a Comment
Read more