Spark 3

2020

• Adaptive Query Execution (AQE):

• Dynamic Partition Pruning

• GraphX

• Pandas API on PySpark

• Python 3.6+ & Scala 2.12+ support

• GPU scheduling

• Binary File Data Source API

• Improved Scalability and Metadata Handling in HDFS

• Improved Web UI