PySpark - withColumn()

In this PySpark tutorial, we will discuss how to use withColumn() method applied on PySpark DataFrame. 

Introduction:

DataFrame in PySpark is an two dimensional data structure that will store data in two dimensional format. One dimension refers to a row and second dimension refers to a column, So It will store the data in rows and columns.

Let's install pyspark module before going to this. The command to install any module in python is "pip".

Syntax:

pip install module_name

Installing PySpark:

pip install pyspark

Steps to create dataframe in PySpark:

1. Import the below modules

      import pyspark
      from pyspark.sql import SparkSession

2. Create spark app named tutorialsinhand using getOrCreate() method

     Syntax:

     spark = SparkSession.builder.appName('tutorialsinhand').getOrCreate()

3. Create list of values for dataframe

4. Pass this list to createDataFrame() method to create pyspark dataframe

    Syntax:
    spark.createDataFrame(list of values)

Scenario -1 : Updating existing values in a column

It is possible to update existing values in a column using withColumn() method.

Syntax:

dataframe.withColumn("column_name",updating_value)

where,'column_name is the column in which values are updated. updating_value is the value/condition/expression in which the values are changed in column(column_name).

Example:

In this example, we will update the values in marks column by multiplying its values by 56.

# import the below modules
import pyspark
from pyspark.sql import SparkSession

# create an app
spark = SparkSession.builder.appName('tutorialsinhand').getOrCreate()

#create a  list of data
values = [{'rollno': 1, 'student name': 'Gottumukkala Sravan','marks': 98},

        {'rollno': 2, 'student name': 'Gottumukkala Bobby','marks': 89},
        {'rollno': 3, 'student name': 'Lavu Ojaswi','marks': 90},

        {'rollno': 4, 'student name': 'Lavu Gnanesh','marks': 78},

        {'rollno': 5, 'student name': 'Chennupati Rohith','marks': 100}]


# create the dataframe from the values
data = spark.createDataFrame(values)

#update marks column vakues multiplied with 56
data.withColumn("marks",data.marks*56).show()

Output: Values in marks column are updated

+-----+------+-------------------+
|marks|rollno|       student name|
+-----+------+-------------------+
| 5488|     1|Gottumukkala Sravan|
| 4984|     2| Gottumukkala Bobby|
| 5040|     3|        Lavu Ojaswi|
| 4368|     4|       Lavu Gnanesh|
| 5600|     5|  Chennupati Rohith|
+-----+------+-------------------+

Scenario -2 : Add a column from existing column

In this scenario, we are adding a new column from an existing column.

Syntax:

dataframe.withColumn("new",dataframe.column_name)

where, new is the new column and column_name is the existing column..

Example:

In this example, we are adding Percentage column from marks column.

# import the below modules
import pyspark
from pyspark.sql import SparkSession

# create an app
spark = SparkSession.builder.appName('tutorialsinhand').getOrCreate()

#create a  list of data
values = [{'rollno': 1, 'student name': 'Gottumukkala Sravan','marks': 98},

        {'rollno': 2, 'student name': 'Gottumukkala Bobby','marks': 89},
        {'rollno': 3, 'student name': 'Lavu Ojaswi','marks': 90},

        {'rollno': 4, 'student name': 'Lavu Gnanesh','marks': 78},

        {'rollno': 5, 'student name': 'Chennupati Rohith','marks': 100}]


# create the dataframe from the values
data = spark.createDataFrame(values)

#add a column - Percentage from marks column
data.withColumn("Percentage",data.marks).show()

Output:

Percentage column is added (same as marks column)

+-----+------+-------------------+----------+
|marks|rollno|       student name|Percentage|
+-----+------+-------------------+----------+
|   98|     1|Gottumukkala Sravan|        98|
|   89|     2| Gottumukkala Bobby|        89|
|   90|     3|        Lavu Ojaswi|        90|
|   78|     4|       Lavu Gnanesh|        78|
|  100|     5|  Chennupati Rohith|       100|
+-----+------+-------------------+----------+