Important Databricks Certified Associate Developer for Apache Spark 3.5 Exam Questions

Answer: C

Explanation:

In Spark's DataFrame API, the dropna() (or equivalently, DataFrameNaFunctions.drop()) method removes rows containing null values.

Behavior:

how='any' drops rows where any column has a null value.

how='all' drops rows where all columns are null.

Since the data scientist wants to drop records with any null field, the correct parameter is how='any'.

Correct syntax:

filtered_users = raw_users.dropna(how='any')

This will remove all records that have at least one null value in any column.

Why the other options are incorrect:

A: Uses na.drop('any') but missing parentheses context (works only as raw_users.na.drop('any'), which is equivalent to option C).

B/D: how='all' only removes rows where all values are null --- too strict for this use case.

PySpark DataFrame API --- DataFrameNaFunctions.drop() and DataFrame.dropna().

Databricks Exam Guide (June 2025): Section ''Developing Apache Spark DataFrame/DataSet API Applications'' --- covers handling missing data and DataFrame cleaning operations.

===========

Answer: A

Explanation:

In Apache Spark, broadcast variables are used to efficiently distribute large, read-only data to all worker nodes. However, broadcasting very large datasets can lead to memory issues on executors if the data does not fit into the available memory.

According to the Spark documentation:

'Broadcast variables allow the programmer to keep a read-only variable cached on each machine rather than shipping a copy of it with tasks. This can greatly reduce the amount of data sent over the network.'

However, it also notes:

'Using the broadcast functionality available in SparkContext can greatly reduce the size of each serialized task, and the cost of launching a job over a cluster. If your tasks use any large object from the driver program inside of them (e.g., a static lookup table), consider turning it into a broadcast variable.'

But caution is advised when broadcasting large datasets:

'Broadcasting large variables can cause out-of-memory errors if the data does not fit in the memory of each executor.'

Therefore, if the broadcasted DataFrame containing millions of rows exceeds the memory capacity of the executors, the job may fail due to memory constraints.

Answer: A

Explanation:

The question requires creating a dictionary where keys are region values and values are the corresponding region_id integers. Furthermore, it asks to retrieve only the smallest 3 region_id values.

Key observations:

.select('region', 'region_id') puts the column order as expected by dict() --- where the first column becomes the key and the second the value.

.sort('region_id') ensures sorting in ascending order so the smallest IDs are first.

.take(3) retrieves exactly 3 rows.

Wrapping the result in dict(...) correctly builds the required Python dictionary: { 'AFRICA': 0, 'AMERICA': 1, 'ASIA': 2 }.

Incorrect options:

Option B flips the order to region_id first, resulting in a dictionary with integer keys --- not what's asked.

Option C uses .limit(3) without sorting, which leads to non-deterministic rows based on partition layout.

Option D sorts in descending order, giving the largest rather than smallest region_ids.

Hence, Option A meets all the requirements precisely.

Answer: C

Explanation:

When using the MEMORY_AND_DISK storage level, Spark attempts to cache as much of the DataFrame in memory as possible. If the DataFrame does not fit entirely in memory, Spark will store the remaining partitions on disk. This allows processing to continue, albeit with a performance overhead due to disk I/O.

As per the Spark documentation:

'MEMORY_AND_DISK: It stores partitions that do not fit in memory on disk and keeps the rest in memory. This can be useful when working with datasets that are larger than the available memory.'

--- Perficient Blogs: Spark - StorageLevel

This behavior ensures that Spark can handle datasets larger than the available memory by spilling excess data to disk, thus preventing job failures due to memory constraints.

Answer: A

Explanation:

The Spark configuration spark.executor.cores defines how many concurrent tasks can be executed within a single executor process.

Each executor is assigned a number of CPU cores.

Each core executes one task at a time.

Therefore, increasing spark.executor.cores allows an executor to run more tasks concurrently.

Example:

--conf spark.executor.cores=4

Each executor can run 4 parallel tasks.

Why the other options are incorrect:

B (spark.task.maxFailures): Sets retry attempts for failed tasks.

C (spark.executor.memory): Sets executor memory, not concurrency.

D (spark.sql.shuffle.partitions): Defines number of shuffle partitions, not executor concurrency.

Spark Configuration Guide --- Executor cores, tasks, and parallelism.

Databricks Exam Guide (June 2025): Section ''Apache Spark Architecture and Components'' --- executor configuration, CPU cores, and parallel task execution.