codeIn [Spark]

Here’s a categorized list of common Narrow dependency transformations, Wide dependency transformations, and Actions available in Apache Spark:

Narrow Dependency Transformations #

Narrow dependency transformations are those where each partition of the parent RDD is used by at most one partition of the child RDD. These transformations do not require data shuffling.

1. map : rdd.map(lambda x: x * 2)
2. filter : rdd.filter(lambda x: x % 2 == 0)
3. flatMap : rdd.flatMap(lambda x: (x, x * 2))
4. mapPartitions : rdd.mapPartitions(lambda iter: (x * 2 for x in iter))
5. mapPartitionsWithIndex : rdd.mapPartitionsWithIndex(lambda index, iter: (index, x) for x in iter)
6. union : rdd1.union(rdd2)
7. sample : rdd.sample(False, 0.1)
8. pipe : rdd.pipe('cmd')
9. coalesce : rdd.coalesce(1)

Wide Dependency Transformations #

Wide dependency transformations are those where multiple child partitions may depend on multiple parent partitions. These transformations often require shuffling data across the network.

1. groupByKey : rdd.groupByKey()
2. reduceByKey : rdd.reduceByKey(lambda a, b: a + b)
3. sortByKey : rdd.sortByKey()
4. join : rdd1.join(rdd2)
5. cogroup : rdd1.cogroup(rdd2)
6. groupWith : rdd1.groupWith(rdd2)
7. cartesian : rdd1.cartesian(rdd2)
8. distinct : rdd.distinct()
9. repartition : rdd.repartition(10)

Actions #

Actions trigger the execution of transformations and return results to the driver program or write data to external storage.

1. collect : rdd.collect()
2. count : rdd.count()
3. take : rdd.take(10)
4. top : rdd.top(10)
5. takeOrdered : rdd.takeOrdered(10)
6. saveAsTextFile : rdd.saveAsTextFile("path/to/file")
7. saveAsSequenceFile : rdd.saveAsSequenceFile("path/to/file")
8. saveAsObjectFile : rdd.saveAsObjectFile("path/to/file")
9. countByKey : rdd.countByKey()
10. foreach : rdd.foreach(lambda x: print(x))
11. reduce : rdd.reduce(lambda a, b: a + b)
12. aggregate : rdd.aggregate((0, 0), (lambda acc, value: (acc[0] + value, acc[1] + 1)), (lambda acc1, acc2: (acc1[0] + acc2[0], acc1[1] + acc2[1])))
13. fold : rdd.fold(0, lambda x, y: x + y)
14. saveAsHadoopFile : rdd.saveAsHadoopFile("path/to/file")
15. saveAsNewAPIHadoopFile : rdd.saveAsNewAPIHadoopFile("path/to/file")

Spark DataFrame Transformations and Actions #

Spark DataFrames offer a higher-level API compared to RDDs. Here’s a categorized list of common transformations and actions for DataFrames:

Transformations #

1. Narrow Dependency:
   • select : df.select("column1", "column2")
   • filter : df.filter(df["column"] > 100)
   • withColumn : df.withColumn("new_col", df["existing_col"] * 2)
   • drop : df.drop("column")
   • distinct : df.distinct()
   • sample : df.sample(False, 0.1)
2. Wide Dependency:
   • groupBy : df.groupBy("column").count()
   • join : f1.join(df2, "column")
   • union : f1.union(df2)
   • repartition : df.repartition(10)

Actions #

1. show : df.show()
2. collect : df.collect()
3. count : df.count()
4. take: df.take(10)
5. head : df.head(10)
6. write : df.write.csv("path/to/file")
7. foreach : df.foreach(lambda row: print(row))

Summary #

Understanding the differences between narrow and wide dependencies, as well as the available actions, helps in designing efficient Spark applications. Narrow dependencies avoid the expensive cost of shuffling data, whereas wide dependencies often require shuffling and should be minimized when possible. Actions are necessary to trigger the execution of the transformations and obtain the results.