# Deploy Spark into Kubernetes Cluster **URL:** \ **Category:** General Discussions\ **Created:** [October 3, 2018, 9:25am UTC](https://discuss.kubernetes.io/t/deploy-spark-into-kubernetes-cluster/2994 "2018-10-03T09:25:18Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![xnuxer88](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/xnuxer88/32/1546_2.png) [@xnuxer88](https://discuss.kubernetes.io/u/xnuxer88)\ **Post date:** [October 3, 2018, 9:25am UTC](https://discuss.kubernetes.io/t/deploy-spark-into-kubernetes-cluster/2994/1 "2018-10-03T09:25:18Z")
Hi, I’m newbie in Kubernetes & Spark Environment. I’m requested to deploy Spark inside Kubernetes so that it’s can be auto Horizontal Scalling. The problem is, I can’t deploy SparkPi example from official website([https://spark.apache.org/docs/latest/running-on-kubernetes#cluster-mode](https://spark.apache.org/docs/latest/running-on-kubernetes#cluster-mode)). I’ve already follow the instruction, but the pods failed to execute. Here is the explanation : 1. Already run : Kubectl proxy 2. When execute : `spark-submit --master k8s://https://localhost:6445 --deploy-mode cluster --name spark-pi --class org.apache.spark.examples.SparkPi --conf spark.executor.instances=5 --conf spark.kubernetes.container.image=xnuxer88/spark-kubernetes-bash-test-entry:v1 local:///opt/spark/examples/jars/spark-examples_2.11-2.3.2.jar` Get Error : `Error: Could not find or load main class org.apache.spark.examples.SparkPi` ![Capture3](https://us1.discourse-cdn.com/flex016/uploads/kubernetes/original/2X/b/b174790815157dd9fd0844ec6359684bc784af58.png) 1. When I check the docker image (create the container from related image), I found the file. Is there any missing instruction that I forgot to follow? Please Help. Thank You. Link : [https://stackoverflow.com/questions/52623435/deploy-spark-into-kubernetes-cluster](https://stackoverflow.com/questions/52623435/deploy-spark-into-kubernetes-cluster) ---
**Author:** ![jeefy](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/jeefy/32/35_2.png) [@jeefy](https://discuss.kubernetes.io/u/jeefy)\ **Post date:** [October 3, 2018, 1:47pm UTC](https://discuss.kubernetes.io/t/deploy-spark-into-kubernetes-cluster/2994/2 "2018-10-03T13:47:46Z")
Heya! Did you by chance see this similar issue? [https://stackoverflow.com/questions/51467082/sparkpi-on-kubernetes-could-not-find-or-load-main-class](https://stackoverflow.com/questions/51467082/sparkpi-on-kubernetes-could-not-find-or-load-main-class) Is it possible the JAR file isn’t actually present in the image? I might try using their `spark-submit` command and see if that moves you along. 🙂 ---
**Author:** ![xnuxer88](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/xnuxer88/32/1546_2.png) [@xnuxer88](https://discuss.kubernetes.io/u/xnuxer88)\ **Post date:** [October 4, 2018, 3:20am UTC](https://discuss.kubernetes.io/t/deploy-spark-into-kubernetes-cluster/2994/3 "2018-10-04T03:20:45Z")
Hi Jeffy, Still the same. Command : `spark-submit --master k8s://https://localhost:6445 --deploy-mode cluster --name spark-pi --class org.apache.spark.examples.SparkPi --conf spark.executor.instances=5 --conf spark.kubernetes.container.image=xnuxer88/spark-kubernetes-bash-test-entry:v1 --conf spark.kubernetes.authenticate.driver.serviceAccountName=default https://github.com/JWebDev/spark/raw/master/spark-examples_2.11-2.3.1.jar` ![Capture7](https://us1.discourse-cdn.com/flex016/uploads/kubernetes/original/2X/0/09d75ed0c9fb543ee59609b7953d722a5cad5259.png) Thank You. ---
**Author:** ![xnuxer88](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/xnuxer88/32/1546_2.png) [@xnuxer88](https://discuss.kubernetes.io/u/xnuxer88)\ **Post date:** [October 4, 2018, 3:21am UTC](https://discuss.kubernetes.io/t/deploy-spark-into-kubernetes-cluster/2994/4 "2018-10-04T03:21:27Z")
Additional Image : ![Capture5](https://us1.discourse-cdn.com/flex016/uploads/kubernetes/original/2X/1/1ecf635f44e817da1cfdf54ff13971733707fc85.png) ---
**Author:** ![xnuxer88](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/xnuxer88/32/1546_2.png) [@xnuxer88](https://discuss.kubernetes.io/u/xnuxer88)\ **Post date:** [October 4, 2018, 3:22am UTC](https://discuss.kubernetes.io/t/deploy-spark-into-kubernetes-cluster/2994/5 "2018-10-04T03:22:00Z")
![Capture6](https://us1.discourse-cdn.com/flex016/uploads/kubernetes/original/2X/1/18fadbc767550bdb992033eb6ad8fdaa8dc32a7e.png) ---
**Author:** ![bharath\_reddy](https://avatars.discourse-cdn.com/v4/letter/b/7ba0ec/32.png) [@bharath\_reddy](https://discuss.kubernetes.io/u/bharath_reddy)\ **Post date:** [November 8, 2018, 9:28pm UTC](https://discuss.kubernetes.io/t/deploy-spark-into-kubernetes-cluster/2994/6 "2018-11-08T21:28:42Z")
Hi , I think you should use this path local:///opt/spark/jars/abcd.jar when try to run your spark jobs in k8 cluster. The thing is the script which is present in spark installation file copies all the jars present under spark2.3.2/examples/jars/abcd.jar to /opt/spark/jars location. Also before running this thing can you build your docker image using the script given in the installation directory/bin/docker.sh ---
**Author:** ![veera\_kumar](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/veera_kumar/32/7514_2.png) [@veera\_kumar](https://discuss.kubernetes.io/u/veera_kumar)\ **Post date:** [April 28, 2021, 10:33am UTC](https://discuss.kubernetes.io/t/deploy-spark-into-kubernetes-cluster/2994/7 "2021-04-28T10:33:36Z")
Is anyone used spark operator to run the spark applications in kuberntes ? ---
**Author:** ![Rajveer\_Singh](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/rajveer_singh/32/16546_2.png) [@Rajveer\_Singh](https://discuss.kubernetes.io/u/Rajveer_Singh)\ **Post date:** [December 26, 2024, 3:16pm UTC](https://discuss.kubernetes.io/t/deploy-spark-into-kubernetes-cluster/2994/8 "2024-12-26T15:16:18Z")
Checkout this [blogpost](https://officiallysingh.medium.com/unified-framework-to-build-spark-jobs-using-spring-boot-and-deploy-locally-and-on-kubernetes-3b4ead0f6636) for complete working solution, you can simply checkout and run the Jobs on your local. - REST APIs to Start, Stop and monitor Spark Jobs on a click. - Production ready Demo Spark Batch and Fault tolerant Streaming Jobs. - Run Spark Jobs locally in IDE and Minikube. - Launch Spark Job Jars or Docker images using spark-submit - Deployment on Kubernetes or AWS EMR