We had a situation recently where-in Hive or Spark Jobs stopped working resulting in Table Lock error.
On debugging Hive Metastore logs, we found following error -
ORA-00001: unique constraint (SDL_HIVE.NOTIFICATION_LOG_EVENT_ID) violated
We figured out that its an Oracle Unique Index that was throwing exception on inserting entries into NOTIFICATION_LOG table.
- Hive’s metastore database primarily tracks partition locations and compaction work for the tables. Importantly, it also tracks the changes that occur in the DB so any federation/backup Hive clusters can successfully receive the data. This particular list of changes lives in the NOTIFICATION_LOG table.
- The ID column in this table are incremented using NOTIFICATION_SEQUENCE table.
- Somehow, new EVENT_ID(s) which were generated had already had an entry in NOTIFICATION_LOG table. Thus, we observed this failure.
Thus, to solve above issue -
- We took backup of NOTIFICATION_LOG table.
- And, Truncated NOTIFICATION_LOG table.
Comments
Post a Comment