Create Spark Helm Chart (latest version)

Hi, what I need is the following:

- I need a helm script (kubernetes) which installs the latest spark version as yarn cluster on kubernetes

- This setup should provide Zeppelin

- All of this has to work with sparks latest structured streaming framework and should be able subscribe a kafka topic

- Confluent is used as kafka provider (and its security mechanism - sasl.jaas.config = org.apache.kafka.common.security.plain.PlainLoginModule)

- Zeppelin has to deploy jobs to cluster and NOT execute them standalone

- The kubernetes cluster doesnt use level 4 load balancer, so we cant simply use the following helm chart (charts/microsoft/spark)

- Kubernetes Level 7 Loadbalancers are used

- Basic neccessary setup (Nodes needs to have the corresponding spark versions deployed) - Acceptance criteria: I have a structured streaming script, which we can use to check if setup works, in the meantime you can use your script for development.

- Kubernetes Version is v1.16.3. - I cant grant you access to the cluster nor to the confluent stream.

Thanks for your time.
About the recuiter
Member since Nov 11, 2022
Ankush Kostha
from Coahuila, Mexico

---

---

Offer to work on this project closes in 14 days!
Are you interested in this Opportunity?

Looking for help? Checkout our video tutorial
How to search and apply for jobs

How to apply? Do you have more questions about the Job?
See frequently asked questions