Important Databricks Certified Associate Developer for Apache Spark 3.5 Exam Questions

Answer: A

Explanation:

Spark supports multiple broadcast joins in a single query plan, as long as each broadcasted DataFrame is small enough to fit under the configured threshold.

Execution Plan:

Spark broadcasts df2 to all executors.

Joins df1 (big) with broadcasted df2.

Then broadcasts df3 and performs another join with the intermediate result.

The result is efficient and avoids shuffling large data.

Why the other options are incorrect:

B: Multiple broadcast joins are supported in Spark 3.x.

C: The join condition is correct since all use id as the key.

D: broadcast() can be used inline; it's valid syntax.

PySpark SQL Functions --- broadcast() usage.

Databricks Exam Guide (June 2025): Section ''Developing Apache Spark DataFrame/DataSet API Applications'' --- multiple broadcast join optimization.

===========

Answer: D

Explanation:

Shuffling occurs in operations like groupBy, reduceByKey, or join---which cause data to be moved across partitions. The repartition() operation can also cause a shuffle, but in this context, it follows an aggregation.

In Option D, the groupBy followed by agg results in a shuffle due to grouping across nodes.

The repartition(10) is a partitioning transformation but does not involve a new shuffle since the data is already grouped.

This sequence --- shuffle (groupBy) followed by non-shuffling (repartition) --- is correct.

Option A does the opposite: the filter does not cause a shuffle, but groupBy does --- this makes it the wrong order.

Answer: B

Explanation:

Option B is the correct and idiomatic approach in PySpark to split a string column (like email) based on a delimiter such as '@'.

The split(col('email'), '@') function returns an array with two elements: username and domain.

getItem(0) retrieves the first part (username).

getItem(1) retrieves the second part (domain).

withColumn() is used to create new columns from the extracted values.

Example from official Databricks Spark documentation on splitting columns:

from pyspark.sql.functions import split, col

df.withColumn('username', split(col('email'), '@').getItem(0)) \

.withColumn('domain', split(col('email'), '@').getItem(1))

Why other options are incorrect:

A uses fixed substring indices (substr(0, 5)), which won't correctly extract usernames and domains of varying lengths.

C uses substring_index, which is available but less idiomatic for splitting emails and is slightly less readable.

D removes '@' from the email entirely, losing the separation between username and domain, and ends up duplicating values in both fields.

Therefore, Option B is the most accurate and reliable solution according to Apache Spark 3.5 best practices.

Answer: A

Explanation:

The number of tasks is controlled by the number of partitions. By default, spark.sql.shuffle.partitions is 200. If stages are showing very few tasks (less than total cores), you may not be leveraging full parallelism.

From the Spark tuning guide:

'To improve performance, especially for large clusters, increase spark.sql.shuffle.partitions to create more tasks and parallelism.'

Thus:

A is correct: increasing shuffle partitions increases parallelism

B is wrong: it further reduces parallelism

C is invalid: increasing dataset size doesn't guarantee more partitions

D is irrelevant to task count per stage

Final Answer: A

Answer: B

Explanation:

Option B uses Spark's built-in SQL function length(), which is efficient and avoids the overhead of a Python UDF:

from pyspark.sql.functions import length, col

df.select(length(col('stringColumn')).alias('length'))

Explanation of other options:

Option A is incorrect syntax; spark.udf is not called this way.

Option C registers a UDF but doesn't apply it in the DataFrame transformation.

Option D is syntactically valid but uses a Python UDF which is less efficient than built-in functions.

Final Answer: B