To minimize data shuffling when joining a large transactions DataFrame with a small customers DataFrame on the customer_id column, which join approach is best?

• A. transactions.join(F.broadcast(customers), transactions.customer_id == customers.customer_id)
• B. transactions.join(customers, transactions.customer_id == customers.customer_id).distinct()
• C. transactions.join(customers, transactions.customer_id == customers.customer_id)
• D. transactions.crossJoin(customers).where(transactions.customer_id == customers.customer_id)

Answer: (A)

When you want to minimize data shuffling in a join, broadcast the smaller DataFrame so it’s replicated to every executor. In Spark, broadcasting the small table allows the join to be performed locally on each partition of the large table, avoiding shuffles of the big dataset.

In this scenario, the large transactions DataFrame is joined with a small customers DataFrame on customer_id. Broadcasting the customers DataFrame means Spark sends it to all workers, and each worker can directly join its portion of transactions with the in-memory copy of customers. This reduces network IO and avoids moving the large transactions across the cluster, which is why this approach is the most efficient for a one-to-many join where one side is tiny.

Why not the other approaches? A regular join would shuffle the large transactions by the join key, leading to expensive data movement. Adding distinct after the join doesn’t eliminate the shuffle and adds extra work to deduplicate results. A cross join multiplies every row of transactions with every row of customers, creating an enormous intermediate result, and then filtering, which is wildly inefficient. A cross join with a where clause relies on generating that giant Cartesian product first, which is also impractical.

Remember the caveat: broadcasting works best when the small table truly fits in memory; if it’s too large, broadcasting can cause memory pressure and worsen performance.