Practice Test
True or False: Apache Spark can process both batch and real-time data sets.
- True
- False
Answer: True
Explanation: Apache Spark supports processing of both batch data and real-time data making it a great tool for data processing.
What is RDD in Apache Spark?
- a. Recursive Data Definition
- b. Resilient Distributed Datasets
- c. Row Data Division
- d. Reference Data Document
Answer: b. Resilient Distributed Datasets
Explanation: RDD in Apache Spark stands for Resilient Distributed Datasets, which is a fundamental data structure of Spark and allows fault-tolerant storage of data across multiple nodes.
True or False: In Apache Spark, data processing operations are lazy.
- True
- False
Answer: True
Explanation: Apache Spark has a lazy evaluation which means no action or execution happens until an action is triggered reducing the number of passes it needs to take on data, hence improving speed.
Which of the following cannot be handled by Apache Spark?
- a. Machine Learning
- b. Graph processing
- c. Streaming data
- d. Blockchains
Answer: d. Blockchains
Explanation: Blockchains cannot be processed in Apache Spark; however, it well handles tasks related to Machine Learning, Graph processing, and Streaming data.
Apache Spark is written in which language?
- a. Python
- b. Scala
- c. Java
- d. C++
Answer: b. Scala
Explanation: Apache Spark is written in Scala but offers high-level APIs in Java, Scala, Python, and R.
True or False: Apache Spark is faster than MapReduce for large scale data processing.
- True
- False
Answer: True
Explanation: Apache Spark is known for its speed as it uses RAM for processing, unlike MapReduce which relies on a disk-based storage system.
PySpark is which of the following?
- a. A library for Python
- b. Python API for Spark
- c. Another name for Apache Spark
- d. None of the above
Answer: b. Python API for Spark
Explanation: PySpark is the Python library for Spark and it is used to interface with Resilient Distributed Datasets.
Which Spark components support Stream Processing?
- a. Spark SQL
- b. Spark Streaming
- c. MLib
- d. GraphX
Answer: b. Spark Streaming
Explanation: Spark Streaming is the component of Spark which is used to process real-time data streams.
True or False: Apache Spark can run standalone without any cluster manager.
- True
- False
Answer: True
Explanation: Spark can run standalone, on Hadoop, on Apache Mesos, on Kubernetes, or on the cloud. It is not necessarily required to have a cluster manager.
Spark’s MLlib can be used for __________.
- a. Data processing
- b. Machine Learning tasks
- c. Web development
- d. Database management
Answer: b. Machine Learning tasks
Explanation: MLlib stands for Machine Learning Library. As the name suggests, it is Spark’s scalable Machine Learning library consisting of common learning algorithms and utilities.
Which statement is NOT TRUE about Apache Spark?
- a. Spark only supports structured data.
- b. Spark runs on distributed systems.
- c. Spark has built-in modules for SQL, streaming, and machine learning.
- d. Spark can work with Hadoop’s HDFS.
Answer: a. Spark only supports structured data.
Explanation: Spark supports both structured and unstructured data. Rest all the options are correct about Apache Spark.
True or False: Transformations in Apache Spark operations are immediately carried out.
- True
- False
Answer: False
Explanation: In Apache Spark, Transformations are lazy operations that define a new RDD from an existing one.
SparkSQL is used for:
- a. Performing SQL like operations on Spark data.
- b. Connecting to SQL database.
- c. Migrating SQL database to Spark.
- d. None of the above.
Answer: a.Performing SQL like operations on Spark data.
Explanation: Spark SQL is a Spark module for structured data processing, and provides a programming interface for handling data along with the performance benefits of RDDs.
True or False: Apache Spark API is available in Ruby.
- True
- False
Answer: False
Explanation: Apache Spark APIs are available in Java, Scala, Python, and R. It does not support Ruby.
Which of the following can be used as a storage level in Spark?
- a. MEMORY_AND_DISK
- b. Disk_Only
- c. Off_Heap
- d. All of the above
Answer: d. All of the above
Explanation: In Apache Spark, storage levels decide how RDD should be stored. It can be in memory (MEMORY_AND_DISK), disk (DISK_ONLY) or off-heap etc. The level of storage can be set according to the requirement of the application and the resources available.
Interview Questions
What is Apache Spark and how is it used in AWS?
Apache Spark is an open-source, distributed computing system used for big data processing and analytics. On AWS, it is typically used with Amazon EMR (Elastic MapReduce) to process vast amounts of data quickly by distributing the computations across multiple nodes.
Name the key features of Apache Spark that makes it suitable for data processing?
Apache Spark offers a fast, in-memory data processing engine, it supports multiple languages including Java, Python, R, and Scala, it can process real-time data, it has a high fault tolerance with the use of RDD (Resilient Distributed Datasets), and it allows machine learning and graph processing.
What are Resilient Distributed Datasets (RDD) in the context of Apache Spark?
RDDs are a fundamental data structure of Spark. They are an immutable distributed collection of objects that can be processed in parallel. They allow you to cache intermediate data across nodes and are highly resilient to failures.
How do you convert a dataframe to RDD in Apache Spark?
You can convert a dataframe to RDD in Apache Spark by using the rdd() function. For example, called on a DataFrame df, it would be df.rdd().
How can Spark be integrated with AWS S3?
Spark can directly read from and write to S3 by specifying the S3 path as the file path. The data can be read with ‘spark.read.text(“s3://bucket_name/key”)’ and can be written to S3 with ‘data.write.parquet(“s3://bucket_name/key”).
What is the significance of Spark Driver program?
The Spark driver program runs the main() function of an application and creates a SparkContext. It splits the program into tasks and schedules them to run on executors.
What is the role of Spark Executor in Apache Spark?
The Spark Executor is a distributed agent responsible for the execution of tasks. Each executor has its own JVM and runs multiple tasks in separate threads.
What are transformations and actions in Apache Spark?
Transformations are operations in Spark that produce a new RDD from existing one like map and filter functions. Actions, on the other hand, are Spark operations that give non-RDD values like reducing the data to a single value or saving it to an external datastore.
How can data be ingested into Spark?
Data can be ingested into Spark through reading from a file stored on Hadoop Distributed File System or local file system, from a database via JDBC, from cloud storage like AWS S3, or by creating an RDD or DataFrame from data in your program.
What is Spark SQL and why is it significant?
Spark SQL is a Spark interface to work with structured and semi-structured data. It allows querying data via SQL-like commands as well as Apache Hive variants and it integrates relational processing with Spark’s functional programming API. It supports querying data in the form of RDD and DataFrame.
What is a DataFrame in Apache Spark?
A DataFrame is a distributed collection of data organized into named columns. It is similar to a relational database table and can be created from sources such as structured data files, Hive tables, and external databases.
What is partitioning in Spark?
Partitioning is the process of dividing your data into a logical division which is smaller parts of data stored across different nodes in a cluster. Partitioning in Spark helps to increase the parallelism when executing the application.
How does Spark achieve fault tolerance?
Spark achieves fault tolerance through an inbuilt mechanism within the Resilient Distributed Dataset (RDD) where data is audited across the multiple partitions. So, even if a partition is lost, it can be re-computed from the original transformation it was derived from.
How can you persist an RDD in Spark?
An RDD can be persisted in Spark by using persistence (caching) methods like persist() or cache() on the RDD. Once an RDD is persisted, each node stores any partitions of it that it computes in memory and reuses them in other actions on that dataset.
Explain the concept of Lineage in Apache Spark.
Lineage in Apache Spark is the sequence of transformations on data, each creating a new RDD, from the start of the application to the point where an action is called. The Spark keeps track of this lineage information to rebuild lost data if needed.
extutorials.com