How to copy a file to a remote server in Python using SCP or SSH?

I have a text file on my local machine that is generated by a daily Python script run in cron.

I would like to add a bit of code to have that file sent securely to my server over SSH.


Asked by: Walter303 | Posted: 01-10-2021






Answer 1

To do this in Python (i.e. not wrapping scp through subprocess.Popen or similar) with the Paramiko library, you would do something like this:

import os
import paramiko

ssh = paramiko.SSHClient() 
ssh.load_host_keys(os.path.expanduser(os.path.join("~", ".ssh", "known_hosts")))
ssh.connect(server, username=username, password=password)
sftp = ssh.open_sftp()
sftp.put(localpath, remotepath)
sftp.close()
ssh.close()

(You would probably want to deal with unknown hosts, errors, creating any directories necessary, and so on).

Answered by: David717 | Posted: 02-11-2021



Answer 2

You can call the scp bash command (it copies files over SSH) with subprocess.run:

import subprocess
subprocess.run(["scp", FILE, "USER@SERVER:PATH"])
#e.g. subprocess.run(["scp", "foo.bar", "joe@srvr.net:/path/to/foo.bar"])

If you're creating the file that you want to send in the same Python program, you'll want to call subprocess.run command outside the with block you're using to open the file (or call .close() on the file first if you're not using a with block), so you know it's flushed to disk from Python.

You need to generate (on the source machine) and install (on the destination machine) an ssh key beforehand so that the scp automatically gets authenticated with your public ssh key (in other words, so your script doesn't ask for a password).

Answered by: Julian184 | Posted: 02-11-2021



Answer 3

You'd probably use the subprocess module. Something like this:

import subprocess
p = subprocess.Popen(["scp", myfile, destination])
sts = os.waitpid(p.pid, 0)

Where destination is probably of the form user@remotehost:remotepath. Thanks to @Charles Duffy for pointing out the weakness in my original answer, which used a single string argument to specify the scp operation shell=True - that wouldn't handle whitespace in paths.

The module documentation has examples of error checking that you may want to perform in conjunction with this operation.

Ensure that you've set up proper credentials so that you can perform an unattended, passwordless scp between the machines. There is a stackoverflow question for this already.

Answered by: Walter723 | Posted: 02-11-2021



Answer 4

There are a couple of different ways to approach the problem:

  1. Wrap command-line programs
  2. use a Python library that provides SSH capabilities (eg - Paramiko or Twisted Conch)

Each approach has its own quirks. You will need to setup SSH keys to enable password-less logins if you are wrapping system commands like "ssh", "scp" or "rsync." You can embed a password in a script using Paramiko or some other library, but you might find the lack of documentation frustrating, especially if you are not familiar with the basics of the SSH connection (eg - key exchanges, agents, etc). It probably goes without saying that SSH keys are almost always a better idea than passwords for this sort of stuff.

NOTE: its hard to beat rsync if you plan on transferring files via SSH, especially if the alternative is plain old scp.

I've used Paramiko with an eye towards replacing system calls but found myself drawn back to the wrapped commands due to their ease of use and immediate familiarity. You might be different. I gave Conch the once-over some time ago but it didn't appeal to me.

If opting for the system-call path, Python offers an array of options such as os.system or the commands/subprocess modules. I'd go with the subprocess module if using version 2.4+.

Answered by: Carlos627 | Posted: 02-11-2021



Answer 5

Reached the same problem, but instead of "hacking" or emulating command line:

Found this answer here.

from paramiko import SSHClient
from scp import SCPClient

ssh = SSHClient()
ssh.load_system_host_keys()
ssh.connect('example.com')

with SCPClient(ssh.get_transport()) as scp:
    scp.put('test.txt', 'test2.txt')
    scp.get('test2.txt')

Answered by: Gianna398 | Posted: 02-11-2021



Answer 6

You can do something like this, to handle the host key checking as well

import os
os.system("sshpass -p password scp -o StrictHostKeyChecking=no local_file_path username@hostname:remote_path")

Answered by: Joyce603 | Posted: 02-11-2021



Answer 7

fabric could be used to upload files vis ssh:

#!/usr/bin/env python
from fabric.api import execute, put
from fabric.network import disconnect_all

if __name__=="__main__":
    import sys
    # specify hostname to connect to and the remote/local paths
    srcdir, remote_dirname, hostname = sys.argv[1:]
    try:
        s = execute(put, srcdir, remote_dirname, host=hostname)
        print(repr(s))
    finally:
        disconnect_all()

Answered by: Roland698 | Posted: 02-11-2021



Answer 8

You can use the vassal package, which is exactly designed for this.

All you need is to install vassal and do

from vassal.terminal import Terminal
shell = Terminal(["scp username@host:/home/foo.txt foo_local.txt"])
shell.run()

Also, it will save you authenticate credential and don't need to type them again and again.

Answered by: Brad466 | Posted: 02-11-2021



Answer 9

Using the external resource paramiko;

    from paramiko import SSHClient
    from scp import SCPClient
    import os

    ssh = SSHClient() 
    ssh.load_host_keys(os.path.expanduser(os.path.join("~", ".ssh", "known_hosts")))
    ssh.connect(server, username='username', password='password')
    with SCPClient(ssh.get_transport()) as scp:
            scp.put('test.txt', 'test2.txt')

Answered by: Richard800 | Posted: 02-11-2021



Answer 10

I used sshfs to mount the remote directory via ssh, and shutil to copy the files:

$ mkdir ~/sshmount
$ sshfs user@remotehost:/path/to/remote/dst ~/sshmount

Then in python:

import shutil
shutil.copy('a.txt', '~/sshmount')

This method has the advantage that you can stream data over if you are generating data rather than caching locally and sending a single large file.

Answered by: Ada642 | Posted: 02-11-2021



Answer 11

Try this if you wan't to use SSL certificates:

import subprocess

try:
    # Set scp and ssh data.
    connUser = 'john'
    connHost = 'my.host.com'
    connPath = '/home/john/'
    connPrivateKey = '/home/user/myKey.pem'

    # Use scp to send file from local to host.
    scp = subprocess.Popen(['scp', '-i', connPrivateKey, 'myFile.txt', '{}@{}:{}'.format(connUser, connHost, connPath)])

except CalledProcessError:
    print('ERROR: Connection to host failed!')

Answered by: Owen493 | Posted: 02-11-2021



Answer 12

Calling scp command via subprocess doesn't allow to receive the progress report inside the script. pexpect could be used to extract that info:

import pipes
import re
import pexpect # $ pip install pexpect

def progress(locals):
    # extract percents
    print(int(re.search(br'(\d+)%$', locals['child'].after).group(1)))

command = "scp %s %s" % tuple(map(pipes.quote, [srcfile, destination]))
pexpect.run(command, events={r'\d+%': progress})

See python copy file in local network (linux -> linux)

Answered by: Daryl792 | Posted: 02-11-2021



Answer 13

A very simple approach is the following:

import os
os.system('sshpass -p "password" scp user@host:/path/to/file ./')

No python library are required (only os), and it works, however using this method relies on another ssh client to be installed. This could result in undesired behavior if ran on another system.

Answered by: Victoria481 | Posted: 02-11-2021



Answer 14

Kind of hacky, but the following should work :)

import os
filePath = "/foo/bar/baz.py"
serverPath = "/blah/boo/boom.py"
os.system("scp "+filePath+" user@myserver.com:"+serverPath)

Answered by: Michelle854 | Posted: 02-11-2021



Similar questions

Read tar file on remote SSH server without using FTP in Python

I am creating some tar file on a remote server and I want to be able to get it to my machine. Due to security reasons I can't use FTP on that server. So how I see it, I have two options: get the file (as file) in some other way and then use tarfile library - if so, I need help with getting the file without FTP. get the content of the file and then extract it. If t...


How to login sftp remote server using python with .ppk file

I'm trying to login the remote server using ppk file with paramiko but it says not a valid RSA key. I searched on web but not able to come with correct solution. Also I'm newbee in python so may be difficult for me too.


How to read excel file from remote server using python ubuntu

I have a local machine connected with internet.From my local machine i can access different machine or remote server. On this remote server i have a excel/csv file. I want to read this excel/csv file from my local machine using python programming. How to read the excel/csv file located in different machine/remote server.


Read last line of the log file from remote server in python

I am trying to read a last line of the .log file from the remote server, But getting "No such file or directory:". I am not sure how to open the connection with server. from fabric.connection import Connection HostName='\\\\Server\xyz.log' with open(HostName,'r') as f: for line in f: pass last_line = line print(last_line)


Read tar file on remote SSH server without using FTP in Python

I am creating some tar file on a remote server and I want to be able to get it to my machine. Due to security reasons I can't use FTP on that server. So how I see it, I have two options: get the file (as file) in some other way and then use tarfile library - if so, I need help with getting the file without FTP. get the content of the file and then extract it. If t...


How to login sftp remote server using python with .ppk file

I'm trying to login the remote server using ppk file with paramiko but it says not a valid RSA key. I searched on web but not able to come with correct solution. Also I'm newbee in python so may be difficult for me too.


How to read excel file from remote server using python ubuntu

I have a local machine connected with internet.From my local machine i can access different machine or remote server. On this remote server i have a excel/csv file. I want to read this excel/csv file from my local machine using python programming. How to read the excel/csv file located in different machine/remote server.


Read last line of the log file from remote server in python

I am trying to read a last line of the .log file from the remote server, But getting "No such file or directory:". I am not sure how to open the connection with server. from fabric.connection import Connection HostName='\\\\Server\xyz.log' with open(HostName,'r') as f: for line in f: pass last_line = line print(last_line)


How to use emacs tramp to edit remote python file?

I have installed the tramp on my emacs properly. I can use it to edit the remote txt files, however, once I create an *.py file on the remote host and edit it, after I input 2 letters, the whole emacs freeze, it doesn't respond. Could anyone give me some hints for this issus?


Run Remote Bash Script in Python

What I'm wanting to do, is run a bash script on another computer, from python. My first thought was to SSH into my remote computer with python, and run it that way. Is there any other way of running a remote bash script in python? Btw, the remote bash script is on the same network as my python script.


Remote PC CPU usage using python

I am using the following code to get remote PC CPU percentage of usage witch is slow and loading the remote PC because of SSHing. per=(subprocess.check_output('ssh root@192.168.32.218 nohup python psutilexe.py',stdin=None,stderr=subprocess.STDOUT,shell=True)).split(' ') print 'CPU %=',float(per[0]) print 'MEM %=',float(per[1]) where psutilexe.py is as follows: ...


Log in to remote Linux shell in Python

I want to write script on python which could execute shell commands on a remote server. I find out that I could use something like: # set environment, start new shell p = Popen("/path/to/env.sh", stdin=PIPE) # pass commands to the opened shell p.communicate("python something.py\nexit") But I do not understand how can I login to remote Linux server and execute shell commands there?...


Run remote python script

Its perhaps not the safest thing in the world to do, but assuming risks have been accounted for, what's the simplest way to load an run a python script that is http(s) hosted, from the local machine?


How to get all data in CSV from a remote MySQL host using Python?

I know it if fairly easy to use mysqldump or SELECT *INTO OUTFILE.. method to get all the data from a MySQL database into a local directory, but how to achieve the similar using Python and a remote MySQL host. There is no server directory at the remote location hence not possible tk dump it there on the server. I need to get the csv directly to my computer.


Can't remote debug a Python web site in Azure with vs2013

I deployed a bottle website on Azure but it shows 500 (internal error) and I can’t see the log. I refer to https://github.com/Microsoft/PTVS/wiki/Azure-Remote-Debugging to debug my project but it failed. It shows the error message as below. could not attach to python.exe process on Azure web site at testpybottleapp.az...


linux - Remote shell using python

I want to do a simple remote shell. I have used sockets to comunicate and it works, in LAN. Now my friend is trying to connect to my server script from his client but he cannot do it. I have opened port 4333 public and associated it to 10001 private in my router. By the way I've replaced the IP with XX. #!/usr/bin/python # -*- coding: utf-8 -*- # Client's script import socket import sys import os...


Remote access to mySQL using Python

I have trouble connecting to my mySQL database remotely through Python. I use the following to connect to mySQL: import mysql.connector cnx = mysql.connector.connect(host='XXX.XXX.XXX.X',user='XXXX',password='XXXXXX',database='testdb') But I get the following error: 2003: Can't connect to MySQL server on '%HOST%:3306' (10060 A connection attempt failed because the c...


Read tar file on remote SSH server without using FTP in Python

I am creating some tar file on a remote server and I want to be able to get it to my machine. Due to security reasons I can't use FTP on that server. So how I see it, I have two options: get the file (as file) in some other way and then use tarfile library - if so, I need help with getting the file without FTP. get the content of the file and then extract it. If t...






Still can't find your answer? Check out these communities...



PySlackers | Full Stack Python | NHS Python | Pythonist Cafe | Hacker Earth | Discord Python



top