Apache Spark Init

Creator

Seonglae Cho

Created

2024 May 4 8:19

Editor

Seonglae Cho

Edited

2024 May 4 9:6

Refs


# Install Spark
wget https://dlcdn.apache.org/spark/spark-3.5.1/spark-3.5.1-bin-hadoop3.tgz
tar xf spark-3.5.1-bin-hadoop3.tgz
rm spark-3.5.1-bin-hadoop3.tgz
mv spark-3.5.1-bin-hadoop3 /root/spark3
pyspark
spark-shell

Edit configs in config folder

workers
spark-env

Run sbin services

start-master.sh

check spark url by accessing 8080 UI

start-worker.sh spark url

Ports in usual

Master UI in 8080

Worker in 8081

History server in 18080

Spark server on 7077

Recommendations

/////////