FileAlreadyExistsException in Spark jobs

Description: The FileAlreadyExistsException error occurs in the following scenarios:

1. Failure of the previous task might leave some files that trigger the FileAlreadyExistsException errors as shown below.
2. When the executor runs out of memory, the individual tasks of that executor are scheduled on another executor. As a result, the FileAlreadyExistsException error occurs.
3. When any Spark executor fails, Spark retries to start the task, which might result into FileAlreadyExistsException error after the maximum number of retries.
4. In Spark, if data frame to be saved has different schema than the Hive table schema, generally, columns should be in sequence with partition columns being last.
5. In Spark, say your table is partitioned on  column A
1. Say you have 2 data frames
2. you union them 
3. try to save it in the table, it will result error above
4. Because, 
1. Say data frame 1 had a record with column A="valueA"
2. Say data frame 2 has a record with column A="valueA"
3. After union , this will still be part of different partitions. So, 2 executor will write data to same partition, leading error above.
4. Thus, after union, perform : df.repartition(A)
5. This will make same value of column A to come in to same partition.