Centos/Redhat
sudo yum update
sudo yum install xz-devel lzma xz-libs libXcomposite libXcursor libXi libXtst libXrandr alsa-lib mesa-libEGL libXdamage mesa-libGL libXScrnSaver wget make gcc64 g++ gcc gcc-c++ openssl-devel bzip2-devel python-devel lzma libevent-devel python3-devel libffi-devel -y
sudo yum groupinstall "Development Tools"
cd /tmp
wget https://www.python.org/ftp/python/3.7.9/Python-3.7.9.tgz
tar xzf Python-3.7.9.tgz
cd Python-3.7.9
./configure --enable-optimizations
sudo make altinstall
sudo ln -sfn /usr/local/bin/python3.7 /usr/bin/python3.7
sudo ln -sfn /usr/local/bin/pip3.7 /usr/bin/pip3.7
python3.7 -m pip install --upgrade pip
After that, we can install Python modules in the below way for datagaps user
su datagaps
python3.7 -m pip install pandas
pip3.7 install requests
Update spark-defaults.conf
add below lines to spark-defaults.conf which is generally located in /opt/datagaps/DataOpsEngine/spark/conf/
spark.pyspark.driver.python python3.7
spark.pyspark.python python3.7