importing pyspark in python shell

Ads
 

importing pyspark in python shell

Hi,

I am learning to develop develop code for Spark in PySpark. Now I have installed Apache Spark but the issue is that its not coming in the python shell.

How should I do for importing pyspark in python shell?

I am experience Python developer and now looking to use my skills in writing Apache Spark program in PySpark.

Thanks

View Answers

November 22, 2018 at 9:00 AM

Hi,

You can not able to access from shell because spark home and path variables are not set correctly.

You should correctly set following variables in .bashrc file:

export SPARK_HOME="/usr/local/spark-2.4.0-bin-hadoop2.7"
export PYTHONPATH=$SPARK_HOME/python/:$PYTHONPATH
export PATH=$PATH:$SPARK_HOME/bin

Open .bashrc file with following command:

sudo gedit ~/.bashrc

Then add following lines:

export SPARK_HOME="/usr/local/spark-2.4.0-bin-hadoop2.7"
export PYTHONPATH=$SPARK_HOME/python/:$PYTHONPATH
export PATH=$PATH:$SPARK_HOME/bin

Save and exit the editor.

Now activate the settings with following command:

 source ~/.bashrc

After this you should be able to open pyspark shell with following command:

pyspark

You can learn PySpark with the help of following tutorials:

Thanks

Ads









Related Tutorials/Questions & Answers:
Importing in java . . . .
Importing in java . . . .   We all know that when we are importing a class of a package in another package , the methods which are public are only available and we can use it . But suppose the method has default access specifier
Importing hibernate project
Importing hibernate project  What are the steps to import hibernate project in myeclipse
Advertisements
Importing a vdi file in VirtualBox
Importing a vdi file in VirtualBox  Hi, How to import a .vdi file in Oracle Virtualbox? Thanks   Check following video: Thanks
Importing Your Own Package
Importing Your Own Package  I would like to know if it is possible to create a class with your own methods so that you can import them when writing programs
Importing in eclipse - Java Beginners
Importing in eclipse  I'm using eclipse for designing struts application... I'm manually copying the tld files and creating struts-config.xml file for my project... Is there the best way to import the tld files or to add
importing data and edit that data
importing data and edit that data   hello sir i have data in excel sheet which contain 2 rows and i want to import that data to ms access and edit that data like adding new columns and new rows and creating new tables
importing excel file
importing excel file  Hi All I am developing a java application whre we upload a excel file in to database and all excel value should import in to the database column. I am using below logic for inserting value. it shwoing some
importing client side web history
importing client side web history  Hello, i am just trying to import the web history of client in lan through java.... can u plz help me
Wicked Shell
Wicked Shell Wicked Shellis an Eclipse plugin providing direct access to your system's shell. To be honest, Wicked Shell is not evil at all, although it calls itself wicked... Wicked Shell is a small plugin trying to support your
Version of jpox>jpox-shell dependency
List of Version of jpox>jpox-shell dependency
Version of com.jaredrummler>android-shell dependency
List of Version of com.jaredrummler>android-shell dependency
Version of com.jrummyapps>android-shell dependency
List of Version of com.jrummyapps>android-shell dependency
Version of com.optersoft>com.optersoft.build-shell dependency
List of Version of com.optersoft>com.optersoft.build-shell dependency
windows shell scripting
windows shell scripting   Hi All, Using this knowledge make a batch file to achieve the following when the batch file is executed. Accept three parameters, a. Param1 b. Param2 c. Param3 Display the accepted parameters
windows shell scripting
windows shell scripting   Hi All, Using this knowledge make a batch file to achieve the following when the batch file is executed. Accept three parameters, a. Param1 b. Param2 c. Param3 Display the accepted parameters
Deep Learning with Python Programming
Deep Learning with Python Programming  Hi, How to learn Deep Learning with Python Programming? Thanks   Deep Learning with Python Python... Leaning, Python stands out as it provides many libraries to develop many deep
ANSI Color Codes with Python
ANSI Color Codes with Python   Use ANSI codes to play with colors and styles. Create a program that will spit out a user inputted string in a color, style, or background color that they can choose. To do this, you'll need
Does importing a package imports the subpackages as well? e.g. Does importing com.MyTest.* also import com.MyTest.UnitTests?
Does importing a package imports the subpackages as well? e.g. Does importing com.MyTest.* also import com.MyTest.UnitTests?  Does importing a package imports the subpackages as well? e.g. Does importing com.MyTest.* also import
Version of cocoon>cocoon-python dependency
List of Version of cocoon>cocoon-python dependency
Version of drools>drools-python dependency
List of Version of drools>drools-python dependency
Importing data into sql plus from a text file...
Importing data into sql plus from a text file...  How to import a text file into oracle 10g enterprise edition directly to create tables and data
Importing data into sql plus from a text file...
Importing data into sql plus from a text file...  How to import a text file into oracle 10g enterprise edition directly to create tables and data
How to run external command in Python?
How to run external command in Python?  Hi, I am writing an application in Python and there is requirement of calling the external command from the Python program. I spend some time but could not find good example of calling
LINUX SHELL SCRIPT - Date Calendar
LINUX SHELL SCRIPT   I wish to create a bash shell script that meets the requirements below. ?Write a bash shell script called how_many_to_go that calculates and prints the number of days, hours, minutes and/or seconds until
Version of maven>maven-shell-plugin dependency
List of Version of maven>maven-shell-plugin dependency
Maven Dependency jpox-shell >> 1.1.2
You should include the dependency code given in this page to add Maven Dependency of jpox >> jpox-shell version1.1.2 in your project
Maven Dependency jpox-shell >> 1.1.3
You should include the dependency code given in this page to add Maven Dependency of jpox >> jpox-shell version1.1.3 in your project
Maven Dependency jpox-shell >> 1.1.9
You should include the dependency code given in this page to add Maven Dependency of jpox >> jpox-shell version1.1.9 in your project
Maven Dependency jpox-shell >> 1.2.0-beta-1
You should include the dependency code given in this page to add Maven Dependency of jpox >> jpox-shell version1.2.0-beta-1 in your project
Maven Dependency jpox-shell >> 1.2.0-beta-2
You should include the dependency code given in this page to add Maven Dependency of jpox >> jpox-shell version1.2.0-beta-2 in your project
Maven Dependency android-shell >> 1.0.0
You should include the dependency code given in this page to add Maven Dependency of com.jaredrummler >> android-shell version1.0.0 in your project

Ads