• Home
  • Docker
  • Kubernetes
  • LLMs
  • Java
  • Ubuntu
  • Maven
  • Big Data
  • Archived
Big Data | Submitting Applications: spark-submit
  1. References
  2. spark-submit command line options
  3. Spark Java simple application: "Line Count"
    1. pom.xml file
    2. Java code
  4. Running the application

  1. References
    See this page for more details about submitting applications using spark-submit:
    https://spark.apache.org/docs/latest/submitting-applications.html
  2. spark-submit command line options

    • Options:
    • Cluster deploy mode only:
    • Spark standalone or Mesos with cluster deploy mode only:
    • Spark standalone and Mesos only:
    • Spark standalone and YARN only:
    • YARN only:
  3. Spark Java simple application: "Line Count"
    1. pom.xml file
    2. Java code
  4. Running the application
    If you build the application "Line Count" (mvn package) it will produce the jar line-count-0.0.1-SNAPSHOT.jar

    Before running the application, let's create a simple text file:

    • Running the application using local mode:

    • Running the application using cluster mode (Deploy Mode: client):

    • Running the application using cluster mode (Deploy Mode: cluster):

      Using "cluster" mode, Spark will launch the driver inside the cluster. spark-deploy-mode-cluster
© 2025  mtitek