PySpark - col()

PySpark - col()

In this PySpark tutorial, we will discuss how to use col() method on PySpark DataFrame. 


DataFrame in PySpark is an two dimensional data structure that will store data in two dimensional format. One dimension refers to a row and second dimension refers to a column, So It will store the data in rows and columns.

Let's install pyspark module before going to this. The command to install any module in python is "pip".


pip install module_name

Installing PySpark:

pip install pyspark

Steps to create dataframe in PySpark:

1. Import the below modules

      import pyspark
      from pyspark.sql import SparkSession

2. Create spark app named tutorialsinhand using getOrCreate() method


     spark = SparkSession.builder.appName('tutorialsinhand').getOrCreate()

3. Create list of values for dataframe

4. Pass this list to createDataFrame() method to create pyspark dataframe

    spark.createDataFrame(list of values)

col stands for column.

Before using this method, we have to import this from pyspark.sql.functions module.


from pyspark.sql.functions import col

col() is used to select columns from the PySpark dataframe.

In this scenario, we will select columns using col() function through select() method and display the values in the column/s.


where, column_name is the column to be displayed.


In this example, we are selecting rows from rollno and marks columns.

# import the below modules
import pyspark
from pyspark.sql import SparkSession

# create an app
spark = SparkSession.builder.appName('tutorialsinhand').getOrCreate()

#create a  list of data
values = [{'rollno': 1, 'student name': 'Gottumukkala Sravan','marks': 98},

        {'rollno': 2, 'student name': 'Gottumukkala Bobby','marks': 89},
        {'rollno': 3, 'student name': 'Lavu Ojaswi','marks': 90},

        {'rollno': 4, 'student name': 'Lavu Gnanesh','marks': 78},

        {'rollno': 5, 'student name': 'Chennupati Rohith','marks': 100}]

# create the dataframe from the values
data = spark.createDataFrame(values)

#display dataframe

#import col function
from pyspark.sql.functions import col

#select rollno and marks column"rollno"),col("marks")).collect()


Finally, we are displaying rows from rollno and marks column through collect() method.

|marks|rollno|       student name|
|   98|     1|Gottumukkala Sravan|
|   89|     2| Gottumukkala Bobby|
|   90|     3|        Lavu Ojaswi|
|   78|     4|       Lavu Gnanesh|
|  100|     5|  Chennupati Rohith|

[Row(rollno=1, marks=98),
 Row(rollno=2, marks=89),
 Row(rollno=3, marks=90),
 Row(rollno=4, marks=78),
 Row(rollno=5, marks=100)]



Would you like to see your article here on tutorialsinhand. Join Write4Us program by

About the Author
Gottumukkala Sravan Kumar 171FA07058
B.Tech (Hon's) - IT from Vignan's University. Published 1400+ Technical Articles on Python, R, Swift, Java, C#, LISP, PHP - MySQL and Machine Learning
Page Views :    Published Date : Jun 12,2023  
Please Share this page

Related Articles

Like every other website we use cookies. By using our site you acknowledge that you have read and understand our Cookie Policy, Privacy Policy, and our Terms of Service. Learn more Got it!