QueryDB

 

 

  To compile code with mixed Java & Scala classes, add following to pom.xml  <build> <pluginManagement> <plugins> <plugin> <groupId>net.alchim31.maven</groupId> <artifactId>scala-maven-plugin</artifactId> <version>4.8.1</version> </plugin> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-compiler-plugin</artifactId> <version>3.8.1</version> <configuration> <source>1.8</source> <target>1.8</target> </configuration> </plugin> </plugins> </pluginManagement> <plugins> <plugin> <groupId>net.alchim31.maven</groupId> <artifactId>scala-maven-plugin</artifactId> <executions> <execution> <id>scala-c...

  Exception -  Caused by: org.apache.phoenix.exception.PhoenixIOException: Can't find method newStub in org.apache.phoenix.coprocessor.generated.MetaDataProtos$MetaDataService! at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:138) at org.apache.phoenix.query.ConnectionQueryServicesImpl.checkClientServerCompatibility(ConnectionQueryServicesImpl.java:1652) at org.apache.phoenix.query.ConnectionQueryServicesImpl.ensureTableCreated(ConnectionQueryServicesImpl.java:1462) at org.apache.phoenix.query.ConnectionQueryServicesImpl.createTable(ConnectionQueryServicesImpl.java:1913) at org.apache.phoenix.schema.MetaDataClient.createTableInternal(MetaDataClient.java:3074) Solution -  This error is due to HBase & Phoenix Jar mismatch. Execute Spark-Submit with --verbose option for debugging.  Remove all HBase Jars from classpath. And, just add phoenix shaded Jar to classpath.

  Exception  Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:4.3.0:compile failed: An API incompatibility was encountered while executing net.alchim31.maven:scala-maven-plugin:4.3.0:compile: java.lang.UnsatisfiedLinkError: /tmp/jna-1459455826/jna3448139317501565807.tmp: /tmp/jna-1459455826/jna3448139317501565807.tmp: failed to map segment from shared object: Operation not permitted Solution -  Set -Djna.tmpdir to any other directory, other then /tmp, which has execute permissions.

  Clone Git Repo -  git clone https://github.com/dinesh028/hbase-connectors.git As of December 2022, the hbase-connectors releases in maven central are only available for Scala 2.11 and cannot be used with Spark 3.x The connector has to be compiled from source for Spark 3.x, see also HBASE-25326 Allow hbase-connector to be used with Apache Spark 3.0 Build as in this example (customize HBase, Spark and Hadoop versions, as needed): mvn -Dspark.version=3.3.1 -Dscala.version=2.12.15 -Dscala.binary.version=2.12 -Dhbase.version=2.4.15 -Dhadoop-three.version=3.3.2 -DskipTests clean package Use Jar with Spark -  spark-shell --jars ~/hbase-connectors/spark/hbase-spark/target/hbase-spark*.jar References -  https://github.com/LucaCanali/Miscellaneous/blob/master/Spark_Notes/Spark_HBase_Connector.md https://kontext.tech/article/628/spark-connect-to-hbase Similarly, do build Phoenix connector or use Cloudera Repo to download Spark3 Jar @   https://repository.cloudera.co...