PySpark - min()

In this PySpark tutorial, we will discuss how to get minimum value from single column/ multiple columns in two ways in an PySpark DataFrame.

Introduction:

DataFrame in PySpark is an two dimensional data structure that will store data in two dimensional format. One dimension refers to a row and second dimension refers to a column, So It will store the data in rows and columns.

Let's install pyspark module before going to this. The command to istall any module in python is "pip".

Syntax:

pip install module_name

Installing PySpark:

pip install pyspark

Steps to create dataframe in PySpark:

1. Import the below modules
      import pyspark
      from pyspark.sql import SparkSession

2. Create spark app named tutorialsinhand using getOrCreate() method

     Syntax: 
     spark = SparkSession.builder.appName('tutorialsinhand').getOrCreate()

3. Create list of values for dataframe

4. Pass this list to createDataFrame() method to create pyspark dataframe
    Syntax:
    spark.createDataFrame(list of values)

Method -1 : Using select() 

min() is an aggregate function used to get the minimum value from the given column in the PySpark DataFrame.

We have to import min() method from pyspark.sql.functions

from pyspark.sql.functions import min

Syntax:

dataframe.select(min("column_name"),.............)

where, column_name is the column minimum value is  returned.

Example 1:

In this example, we created pyspark dataframe with 5 rows and three columns and will get the minimum value  from marks column.

# import the below modules
import pyspark
from pyspark.sql import SparkSession

# create an app
spark = SparkSession.builder.appName('tutorialsinhand').getOrCreate()

#create a  list of data
values = [{'rollno': 1, 'student name': 'Gottumukkala Sravan kumar','marks': 98},

        {'rollno': 2, 'student name': 'Gottumukkala Bobby','marks': 89},

        {'rollno': 3, 'student name': 'Lavu Ojaswi','marks': 90},

        {'rollno': 4, 'student name': 'Lavu Gnanesh','marks': 78},

        {'rollno': 5, 'student name': 'Chennupati Rohith','marks': 100}]


# create the dataframe from the values
data = spark.createDataFrame(values)


#import min function
from pyspark.sql.functions import min

#display minimum of  marks
print(data.select(min("marks")).collect())

Output:

We are collecting the output with collect() method.

[Row(min(marks)=78)]

Example 2:

In this example, we created pyspark dataframe with 5 rows and three columns and will get the minimum from  marks  and  rollno column.

# import the below modules
import pyspark
from pyspark.sql import SparkSession

# create an app
spark = SparkSession.builder.appName('tutorialsinhand').getOrCreate()

#create a  list of data
values = [{'rollno': 1, 'student name': 'Gottumukkala Sravan kumar','marks': 98},

        {'rollno': 2, 'student name': 'Gottumukkala Bobby','marks': 89},

        {'rollno': 3, 'student name': 'Lavu Ojaswi','marks': 90},

        {'rollno': 4, 'student name': 'Lavu Gnanesh','marks': 78},

        {'rollno': 5, 'student name': 'Chennupati Rohith','marks': 100}]


# create the dataframe from the values
data = spark.createDataFrame(values)


#import min function
from pyspark.sql.functions import min

#display minimum of  marks and rollno
print(data.select(min("marks"),min("rollno")).collect())

Output:

Minimum value from  marks and rollno columns is returned.

[Row(min(marks)=78, min(rollno)=1)]

Method -2 : Using agg() 

agg() stands for aggregation which will take dictionary in which key will be th column and value will be the min function. It will return minimum from particular column provided as key

Syntax:

dataframe.agg({'column_name': 'min',............})

where, column_name is the column minimum value is  returned.

Example:

In this example, we created pyspark dataframe with 5 rows and three columns and will get the minimum value from marks  and  rollno column.

# import the below modules
import pyspark
from pyspark.sql import SparkSession

# create an app
spark = SparkSession.builder.appName('tutorialsinhand').getOrCreate()

#create a  list of data
values = [{'rollno': 1, 'student name': 'Gottumukkala Sravan kumar','marks': 98},

        {'rollno': 2, 'student name': 'Gottumukkala Bobby','marks': 89},

        {'rollno': 3, 'student name': 'Lavu Ojaswi','marks': 90},

        {'rollno': 4, 'student name': 'Lavu Gnanesh','marks': 78},

        {'rollno': 5, 'student name': 'Chennupati Rohith','marks': 100}]


# create the dataframe from the values
data = spark.createDataFrame(values)

#display minimum of  marks and rollno
data.agg({'marks': 'min','rollno': 'min'}).collect()

Output:

Minimum value from  marks and rollno columns is returned.

[Row(min(rollno)=1, min(marks)=78)]

Gottumukkala Sravan Kumar 171FA07058
B.Tech (Hon's) - IT from Vignan's University. Published 1400+ Technical Articles on Python, R, Swift, Java, C#, LISP, PHP - MySQL and Machine Learning