Essential PySpark Functions: Practical Examples for Beginners

Working with big data can initially feel overwhelming — with rows and columns stretching into millions, traditional tools often slow to a crawl. That’s where PySpark shines. It combines Python’s simplicity with Spark’s distributed power, letting you process massive datasets with ease. However, learning PySpark can feel like wandering through a giant toolbox without knowing which tools matter. You don’t need every single function to get real work done. What you need are the essentials — the ones you’ll use daily to clean, transform, and analyze data. This guide walks you through those key PySpark functions, with simple examples.

Top PySpark Functions with Examples

select()

The select() function is your go-to when you only need certain columns from a DataFrame. Instead of hauling around the whole table, you can keep just what matters.

Example:

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("example").getOrCreate()
data = [("Alice", 34), ("Bob", 45), ("Charlie", 29)]
df = spark.createDataFrame(data, ["name", "age"])
df.select("name").show()

Output:

+-------+
|   name|
+-------+
|  Alice|
|    Bob|
|Charlie|
+-------+

You can also use selectExpr() to write SQL-like expressions when selecting columns.

withColumn()

When you need to create a new column or modify an existing one, use withColumn(). You pass it the name of the column and the expression to compute.

Example:

from pyspark.sql.functions import col

df.withColumn("age_plus_10", col("age") + 10).show()

Output:

+-------+---+-----------+
|   name|age|age_plus_10|
+-------+---+-----------+
|  Alice| 34|        44 |
|    Bob| 45|        55 |
|Charlie| 29|        39 |
+-------+---+-----------+

filter() / where()

You’ll often need to work with a subset of your data. filter() or where() helps you keep only rows that match a condition.

Example:

df.filter(col("age") > 30).show()

Output:

+-----+---+
| name|age|
+-----+---+
|Alice| 34|
|  Bob| 45|
+-----+---+

Both filter() and where() are interchangeable. Use whichever feels more readable to you.

groupBy() and agg()

To summarize data, you’ll use groupBy() combined with aggregation functions. You can compute counts, averages, sums, etc.

Example:

from pyspark.sql.functions import avg

data = [("Math", "Alice", 85), ("Math", "Bob", 78),
        ("English", "Alice", 90), ("English", "Bob", 80)]
df2 = spark.createDataFrame(data, ["subject", "student", "score"])
df2.groupBy("subject").agg(avg("score").alias("avg_score")).show()

Output:

+-------+---------+
|subject|avg_score|
+-------+---------+
|   Math|     81.5|
|English|     85.0|
+-------+---------+

orderBy() / sort()

If you want your results in a specific order, use orderBy() or sort(). Both do the same thing.

Example:

df.orderBy(col("age").desc()).show()

Output:

+-------+---+
|   name|age|
+-------+---+
|    Bob| 45|
|  Alice| 34|
|Charlie| 29|
+-------+---+

You can sort by multiple columns if needed.

drop()

Sometimes you want to remove a column you don’t need anymore.

Example:

df.drop("age").show()

Output:

+-------+
|   name|
+-------+
|  Alice|
|    Bob|
|Charlie|
+-------+

distinct()

To get unique rows from your DataFrame, use distinct().

Example:

data = [("Alice", 34), ("Alice", 34), ("Bob", 45)]
df3 = spark.createDataFrame(data, ["name", "age"])
df3.distinct().show()

Output:

+-----+---+
| name|age|
+-----+---+
|  Bob| 45|
|Alice| 34|
+-----+---+

dropDuplicates()

This is like distinct(), but you can specify which columns to consider when checking for duplicates.

Example:

df3.dropDuplicates(["name"]).show()

Output:

+-----+---+
| name|age|
+-----+---+
|  Bob| 45|
|Alice| 34|
+-----+---+

join()

Combining two DataFrames is a common need. Use join() to merge on a common column.

Example:

data1 = [("Alice", "Math"), ("Bob", "English")]
df4 = spark.createDataFrame(data1, ["name", "subject"])
data2 = [("Alice", 85), ("Bob", 78)]
df5 = spark.createDataFrame(data2, ["name", "score"])
df4.join(df5, on="name").show()

Output:

+-----+-------+-----+
| name|subject|score|
+-----+-------+-----+
|Alice|   Math|   85|
|  Bob|English|   78|
+-----+-------+-----+

cache()

When working with large datasets, reusing the same DataFrame can get slow. cache() keeps it in memory for faster access.

Example:

df.cache()
df.count()  # This action triggers caching

There’s no visible output here, but future operations on df will run faster.

collect()

To get your results back to Python as a list of rows, use collect(). Be careful — if your data is huge, this can crash your driver.

Example:

rows = df.collect()
print(rows)

Output:

[Row(name='Alice', age=34), Row(name='Bob', age=45), Row(name='Charlie', age=29)]

show()

This one you’ve already seen throughout the examples. show() prints your DataFrame in a readable tabular format.

Example:

df.show()

count()

To quickly find out how many rows you have.

Example:

df.count()

Output:

3

replace()

To replace specific values in a DataFrame.

Example:

df.replace("Alice", "Alicia", "name").show()

Output:

+-------+---+
|   name|age|
+-------+---+
| Alicia| 34|
|    Bob| 45|
|Charlie| 29|
+-------+---+

fillna()

To fill in missing values with a default.

Example:

data = [("Alice", None), ("Bob", 45)]
df6 = spark.createDataFrame(data, ["name", "age"])
df6.fillna(0).show()

Output:

+-----+---+
| name|age|
+-----+---+
|Alice|  0|
|  Bob| 45|
+-----+---+

explode()

When working with columns that contain arrays or lists, you often need to turn each element of the array into its row. That’s what explode() does — it flattens an array column.

Example:

from pyspark.sql.functions import explode

data = [("Alice", ["Math", "English"]), ("Bob", ["History", "Science"])]
df7 = spark.createDataFrame(data, ["name", "subjects"])
df7.select("name", explode("subjects").alias("subject")).show()

Output:

+-----+--------+
| name| subject|
+-----+--------+
|Alice|    Math|
|Alice| English|
|  Bob| History|
|  Bob| Science|
+-----+--------+

Conclusion

Knowing which PySpark functions to focus on saves you time and makes your code much cleaner. You don’t need to memorize every single function in the library — the ones we’ve covered here are more than enough to handle most real-world tasks. They cover selecting and transforming columns, filtering and grouping data, joining DataFrames, dealing with duplicates and missing values, and working efficiently with cached data. As you get more practice, you’ll start using these almost without thinking. PySpark is powerful because of how much you can do with just a few well-chosen functions. Start with these, experiment with your datasets, and the rest will come naturally.

For more information on PySpark, you can visit the Apache Spark Documentation.