Big Data – mtitek.com

Big Data

Big Data ecosystem
Apache Hadoop
- Install and configure Apache Hadoop (single node cluster) (3.3.0)
- HDFS Commands
  - HDFS - DFS Commands
  - HDFS - DFSADMIN Commands
- ORC/Parquet/Avro Tools
  - ORC Tools (1.5.4)
  - Parquet Tools (1.9.0)
  - Avro Tools (1.9.0)
Apache Hive
- Install and configure Apache Hive (HiveServer, Hive MetaStore) (3.1.2)
- Manage Hive Databases
Apache Spark
- Install and configure Apache Spark (standalone) (3.0.0)
- Access Hive Tables using Spark SQL
- Spark Tools
  - Spark Interactive Shell (Scala): spark-shell
  - Spark Interactive Shell (Python): pyspark
  - Spark Interactive Shell (R): sparkR
  - Submitting Applications: spark-submit
  - Spark SQL CLI: spark-sql
- Spark API: RDD, DataFrame, Dataset
Apache ZooKeeper (3.8.1)
Apache Solr (9.1.1)
Install and configure Apache Kafka (3.4.0)
Install and configure MongoDB (4.2)
Install and configure Apache Nutch (2.3.1)
Python (3.11.4)