ImportError: No module named pandas - Error in sparkshell

Ads
 

ImportError: No module named pandas - Error in sparkshell

Hi,

I am new to pyspark and learning it from beginning. I have successfully installed Spark in Ubuntu and now trying to write some program.

I am trying to convert pandas array into dataframe but it giving error:

ImportError: No module named pandas

When try to import pandas:

import pandas as pd

Full error is:

Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_\   version 2.4.0
      /_/

Using Python version 2.7.12 (default, Nov 12 2018 14:36:49)
SparkSession available as 'spark'.
>>> import pandas as pd
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named pandas
>>>

How to resolve this?

Thanks

View Answers

November 22, 2018 at 9:24 AM

Hi,

This error clearly indicates that pandas is not installed on your computer. If you install pandas then the problem will be solved.

Here is the command to install pandas:

pip install pandas

Here is simple installation process:

[email protected]:/home/deepak/spark-2.4.0-bin-hadoop2.7/bin# pip install pandas
Collecting pandas
  Downloading https://files.pythonhosted.org/packages/b7/e3/f52d484244105fa3d558ce8217a5190cd3d40536076bef66d92d01566325/pandas-0.23.4-cp27-cp27mu-manylinux1_x86_64.whl (8.9MB)
    100% |????????????????????????????????| 8.9MB 125kB/s 
Collecting numpy>=1.9.0 (from pandas)
  Downloading https://files.pythonhosted.org/packages/de/37/fe7db552f4507f379d81dcb78e58e05030a8941757b1f664517d581b5553/numpy-1.15.4-cp27-cp27mu-manylinux1_x86_64.whl (13.8MB)
    100% |????????????????????????????????| 13.8MB 103kB/s 
Collecting python-dateutil>=2.5.0 (from pandas)
  Downloading https://files.pythonhosted.org/packages/74/68/d87d9b36af36f44254a8d512cbfc48369103a3b9e474be9bdfe536abfc45/python_dateutil-2.7.5-py2.py3-none-any.whl (225kB)
    100% |????????????????????????????????| 235kB 1.1MB/s 
Collecting pytz>=2011k (from pandas)
  Downloading https://files.pythonhosted.org/packages/f8/0e/2365ddc010afb3d79147f1dd544e5ee24bf4ece58ab99b16fbb465ce6dc0/pytz-2018.7-py2.py3-none-any.whl (506kB)
    100% |????????????????????????????????| 512kB 993kB/s 
Collecting six>=1.5 (from python-dateutil>=2.5.0->pandas)
  Downloading https://files.pythonhosted.org/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl
Installing collected packages: numpy, six, python-dateutil, pytz, pandas
Successfully installed numpy-1.15.4 pandas-0.23.4 python-dateutil-2.7.5 pytz-2018.7 six-1.11.0
You are using pip version 8.1.1, however version 18.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
[email protected]:/home/deepak/spark-2.4.0-bin-hadoop2.7/bin#

After this you should be able to import pandas in your program.

You can learn PySpark with the help of following tutorials:

Thanks


November 22, 2018 at 9:29 AM

Hi,

Now its working:

SparkSession available as 'spark'.
>>> import pandas as pd
>>>

Thanks

Ads









Related Tutorials/Questions & Answers:
ImportError: No module named scipy.ndimage - Solved
ImportError: No module named scipy.ndimage - Solved  Hi, How to solve following error: ImportError: No module named scipy.ndimage This error..., in <module> import scipy.ndimage as ndi ImportError: No module named
Hibernate named queries
Hibernate named queries  Hi, What is named queries in Hibernate? Thanks
Advertisements
a class named Transactions
a class named Transactions  Write this java code for me: a class named Transactions that holds the memo and the amount of a transaction. Prompt the user for the number of transactions and create a new Transaction object
Design a class named account
Design a class named account  The class contains: § An int data field named id for the account (default 0). § A double data field named balance for the account (default 0.0) § A double data field named annualInterestRate
Database named studentinfo
Database named studentinfo  1.create design of studentinfo using swing component 2.Create a database named student info. 3.create a table that stores all the details from the form,application of basic Normalization is required
Database named studentinfo
Database named studentinfo  1.create design of studentinfo using swing component 2.Create a database named student info. 3.create a table that stores all the details from the form,application of basic Normalization is required
Database named studentinfo
Database named studentinfo  1.create design of studentinfo using swing component 2.Create a database named student info. 3.create a table that stores all the details from the form,application of basic Normalization is required
Named ? SQL query in hibernate
Named ? SQL query in hibernate  What is Named SQL query in hibernate?   You can put all the HQL into the XML mapping file rather than putting the sql queries to various places in between code. sample.hbm.xml <
Hibernate Named Query
This section contain description of named query with example
create named class - Java Beginners
create named class  Can you check my [email protected]: How to create a class named Employee with name and salary. Make a class Manager inherit from Employee with an instance field named department. Supply a method toString
Hibernate Named Native SQL Query
In this section, you will learn Named Native SQL Query in Hibernate
JPA setParameter (Named Parameter)
JPA setParameter (Named Parameter)   ... API's method setParameter as a named parameter. JPA Query API's supports two types of setParameter.  1. Named Parameters  2. Numbered Parameters
Hibernate Named HQL in Annotation
In this section, you will learn how to execute Named HQL written in Model class Annotation
Admin Module
Admin Module The admin have the privilege  to add the question, delete the question. He is responsible for maintaining the site. He can also search the question paper by question Id and by title. The Admin does the following
Version of com.fmpwizard>lift-named-comet_2.8.1 dependency
List of Version of com.fmpwizard>lift-named-comet_2.8.1 dependency
Version of com.fmpwizard>lift-named-comet_2.9.1 dependency
List of Version of com.fmpwizard>lift-named-comet_2.9.1 dependency
Version of com.fmpwizard>lift-named-comet_2.8.2 dependency
List of Version of com.fmpwizard>lift-named-comet_2.8.2 dependency
Version of com.fmpwizard>lift-named-comet_2.9.0-1 dependency
List of Version of com.fmpwizard>lift-named-comet_2.9.0-1 dependency
User Module
User Module The user first need to make registration on the website for appearing in the online examination. After the making successful registration he is eligible for online test examination. To appear in the examination he need
spring JEE module tutorial
spring JEE module tutorial  Hi, I want spring Jee module tutorial and spring ORM module tutorial plz send me
named parameters vs postional parameters - Hibernate
named parameters vs postional parameters  hi,Can you please explain why we i should go for particular parameters or positional parameter... regards, somasekhar
Hibernate Named HQL using XML Mapping
In this section, you will learn executing Named HQL written in XML Mapping file
JPA Named Parameter List
JPA Named Parameter List   ... application using JPA Named Parameter List. This is similar as named parameter but here we use the an array list as named parameter.  Named Parameter List
JPA Named Parameter
JPA Named Parameter       In this section, you will learn how to use JPA Named Parameter in your jpa application. In JPA named parameter is used where you assign names
Hibernate Named Native SQL in Annotation
In this section, you will learn how to execute the Named Native SQL written in Model class Annotation
JPA Named Queries
JPA Named Queries       In this section, you will know about the JPA Named Queries...;) }) For developing JPA Named Queries, you need the following files
spring DAO module turorial
spring DAO module turorial  how to integrate springDAO and spring webMVC
Write a program named InternetCharge_YourLastName that do the following:
Write a program named InternetCharge_YourLastName that do the following: ... per month unlimited access is provided. Write a program named InternetCharge.... Instruction: We should have two classes class named InternetService
Hibernate Named Query
Hibernate Named Query       Named Query is very useful concept in hibernate. It lets...;   Calling Named Query in the coding section You can call
Maven Dependency lift-named-comet_2.8.1 >> 0.3
You should include the dependency code given in this page to add Maven Dependency of com.fmpwizard >> lift-named-comet_2.8.1 version0.3 in your project
Maven Dependency lift-named-comet_2.8.2 >> 0.3
You should include the dependency code given in this page to add Maven Dependency of com.fmpwizard >> lift-named-comet_2.8.2 version0.3 in your project

Ads