How specify the version of python that spark-submit uses

How specify the version of python that spark-submit uses

Hi,

I have installed python 3.7 and want to use with spark.

How specify the version of python that spark-submit uses?

Thanks

View Answers

June 25, 2018 at 7:45 AM

Hi,

You have to modify the value in conf/spark-env.sh file.

This will instruction to use the spark to use the python as mentioned in the configuration.

Open the file: conf/spark-env.sh

and then add following into it:

export PYSPARK_PYTHON=/opt/pathtopyton/python

You have to give complete path of python exectutable

This will solve problem.

Thanks

Ads









Related Tutorials/Questions & Answers:
How specify the version of python that spark-submit uses
How specify the version of python that spark-submit uses  Hi, I have installed python 3.7 and want to use with spark. How specify the version of python that spark-submit uses? Thanks   Hi, You have to modify
ModuleNotFoundError: No module named 'spark-submit-app-id-wrapper'
: No module named 'spark-submit-app-id-wrapper' How to remove.... You can install spark-submit-app-id-wrapper python with following command... of spark-submit-app-id-wrapper python library, ModuleNotFoundError: No module named
Advertisements
Add jars to a Spark Job spark-submit
Add jars to a Spark Job spark-submit  Add jars to a Spark Job spark-submit I want to submit some dependency jars along with the code jar file with spark-submit. How to add jar to the Spark Job Submit process? Thanks   
Add jars to a Spark Job spark-submit
Add jars to a Spark Job spark-submit  Add jars to a Spark Job spark-submit I want to submit some dependency jars along with the code jar file with spark-submit. How to add jar to the Spark Job Submit process? Thanks   
ModuleNotFoundError: No module named 'specify'
ModuleNotFoundError: No module named 'specify'  Hi, My Python... 'specify' How to remove the ModuleNotFoundError: No module named 'specify... to install padas library. You can install specify python with following command
kibana specify elasticsearch url
kibana specify elasticsearch url  Hi, I have to connect... needs to be done in kibana for this? How we can modify kibana configuration to specify elasticsearch url? Thanks   Hi, Kibana is using
kibana specify elasticsearch url
kibana specify elasticsearch url  Hi, I have to connect... needs to be done in kibana for this? How we can modify kibana configuration to specify elasticsearch url? Thanks   Hi, Kibana is using
kibana specify elasticsearch url
kibana specify elasticsearch url  Hi, I have to connect... needs to be done in kibana for this? How we can modify kibana configuration to specify elasticsearch url? Thanks   Hi, Kibana is using
how to call the parent page from child without refreshing the parent and my parent window uses the master page
how to call the parent page from child without refreshing the parent and my parent window uses the master page   how to call the parent page from child without refreshing the parent and my parent window uses the master page
Big Data and its uses
Big Data and its uses  Hi, I am trying to understand the Big Data and its uses. Please tell me: What is Big Data? What is the uses of Big Data... are the uses of Big Data? Big Data provides tremendous opportunity to the business
Java application that uses the classes and
Java application that uses the classes and   i have this class diagram with three classes: Curriculum, Course, and Lecture. Your class diagram...) I wish Write a Java application that uses the classes and methods above
Uses of GPS Tracking System
of operation. Here we explain how to uses of GPS Tracking system that includes... GPS Tracking System has many uses in respect to the security and information of vehicles and other devices, which can be traced using GPS system
how to grayscale an image in python
how to grayscale an image in python  Hi, I have may images and it want to greyscale all with python. can it be done? how to grayscale an image in python? Thanks   Hi, You can do this with opencv. here is code
How is Python used in AI?
How is Python used in AI?  Hi, I am beginner in Data Science and machine learning field. I am searching for the tutorials to learn: How is Python... learn the topic "How is Python used in AI?". Also tell me which
how to create array in python
how to create array in python  Hi, how to create array in python? I want to create an array in Python and add some values to it. Thanks   Hi, You can create a new array with the code myArray = [] and after
Uses of Android Technology
interface based on touch or even gesture. Uses of android technology not just... or tablet can be just enough, so multifarious, stylish and smarter the uses of android technology eventually has become! Understanding the uses of android
How to install botocore library in Python
How to install botocore library in Python  Hi, How to install botocore library in Python? What is the use of botocore library in Python? Thanks
How to use list in for loop in Python
How to use list in for loop in Python  Hi, I have list in Python and I want to list down all the items using for loop. How to use list in for loop in Python? Thanks
How to install the Six module in Python
distribution. How to install the Six module in Python? Thanks...How to install the Six module in Python  Hi, I want to install the six module in Python as my program is using this. I have checked it by importing
how to move file in python script
how to move file in python script  Hi, In my Python program I have... the file from one directory to another in Python? Thanks   Hi, You can use any of the following methods for moving a file to another directory in Python
How to print like printf in python
How to print like printf in python  Hi, I am from C background and there used to be printf function. Jow to print like printf in python? The printf function is C/C++ is very useful. So, looking for some line printf in Python
How to run external command in Python?
How to run external command in Python?  Hi, I am writing an application in Python and there is requirement of calling the external command from... external command. How to run external command in Python? Thanks   Hi
how to access python dictionary elements
how to access python dictionary elements  Hi, I am new to Python...? How to access python dictionary elements? Thanks   Hello, Python... a collection object in my program. Now I want to access value of a key from python
how to access python dictionary elements
how to access python dictionary elements  Hi, I am new to Python...? How to access python dictionary elements? Thanks   Hello, Python... a collection object in my program. Now I want to access value of a key from python
how to add data into dictionary in python
how to add data into dictionary in python  Hi, I have a python API that returns Python dictionary. It contains many values. Here is simple example...} I want to add DOB to this dictionary object. how to add data into dictionary
How is Python used for data science?
How is Python used for data science?  Hi, I am beginner in Data...: How is Python used for data science? Try to provide me good examples or tutorials links so that I can learn the topic "How is Python used for data science
How to append a dictionary to a list in Python
How to append a dictionary to a list in Python  Hi, I have a loop... is to add the dictionary into the list. How I can achieve this? How to append a dictionary to a list in Python? Thanks   Hi, Here is the simple example
How to read from stdin in python
How to read from stdin in python  Hi, I am writing a program in python which is suppose to take input from user. In Java it was easy, but in Python how i can achieve the same? Thanks   HI, In Python also you
How to get yesterday date in Python?
Yesterday's date in Python - How to get yesterday date in Python? Python... API. In this article we will learn how to get yesterday's date in the Python... in adding or subtracting from the date object in Python. How to get yesterday date
What is Machine Learning and its uses now a days?
What is Machine Learning and its uses now a days?  Hi, I what... Learning and its uses now a days? Thanks   What is Machine Learning and its uses now a days? In today's world, these machines and also the robots have
What is Machine Learning and its uses now a days?
What is Machine Learning and its uses now a days?  Hi, I what... Learning and its uses now a days? Thanks   What is Machine Learning and its uses now a days? In today's world, these machines and also the robots have

Ads