Spark SQL fails to read data from a ORC hive table that has a new column added to it. Giving Exception -
java.lang.AssertionError: assertion failed
at scala.Predef$.assert(Predef.scala:165)
at org.apache.spark.sql.execution.datasources.LogicalRelation$$anonfun$1.apply(LogicalRelation.scala:39)
at org.apache.spark.sql.execution.datasources.LogicalRelation$$anonfun$1.apply(LogicalRelation.scala:38)
at scala.Option.map(Option.scala:145)
This happens when following property is set - spark.sql.hive.convertMetastoreOrc=true
Solution - Comment out property if being set explicitly or set it to false. Refer https://issues.apache.org/jira/browse/SPARK-18355
Comments
Post a Comment