Network File Copy using SSH
Updated April 25, 2005
Created April 23, 2001

Autogenerated Site Map
Search this Site!:
Search this site powered by FreeFind
prev next index
Please note that &&, ||, and -, are documented at the bottom of this page.

PUSH:


PULL:

  • ssh target_address cat remotefile > localfile
  • ssh target_address dd if=remotefile | dd of=localfile
  • ssh target_address cat "<" remotefile >localfile
  • ssh target_address cat "<" remotefile.gz | gunzip >localfile

    • COMPARE:

  • ###This one uses CPU cycles on the remote server to compare the files:
  • ssh target_address cat remotefile | diff - localfile
  • cat localfile | ssh target_address diff - remotefile
  • ###This one uses CPU cycles on the local server to compare the files:
  • ssh target_address cat <localfile "|" diff - remotefile

    • Push: Push local file to remote server.
    Pull: Pull remote file from remote server to local machine.

    Of course there is always ftp, scp2, nfs, smb and other methods as well.

    The above methods make a great Ghost replacement.
    One can boot a system using standalone linux on a floppy, such as tomsrtbt and can then proceed to:
    1. backup the local hard drive to a remote server or
    2. download an image from the remote server and place it on the local hard drive.
    RSH works just the same as SSH I'm sure, it's jut that ssh or ssh should give you better security.

    Note: Compressing and then transferring data is faster than transferring uncompressed data. Use compression before sending data over the wire to achieve faster data transfer speeds.

    localfile and remotefile can be files, directories, images, hard drive partitions, or hard drives.

    Moving files around on local filesystem:

    FTP VIEW:

    FTP PUSH:

    FTP PULL:

    Pipes and Redirects:

    SSH Keys

    see http://www.cpqlinux.com/sshkeys.html

    Explanation of &&, ||, and -

    && is shorthand for "if true then do"
    || is shorthand for "if false then do"
    These can be used separately or together as needed. The following examples will attempt
    to change directory to "/tmp/mydir"; you will get different results based on whether
    "/tmp/mydir" exists or not.
    cd /tmp/mydir && echo was able to change directory
    cd /tmp/mydir || echo was not able to change directory
    cd /tmp/mydir && echo was able to change directory || echo was not able to change to directory
    cd /tmp/mydir && echo success || echo failure
    cd /tmp/mydir && echo success || { echo failure; exit; }

    The dash "-" is used to reference either standard input or standard output. The context in which the dash is used is what determines whether it references standard input or standard output.

    Netcat:

    Also be aware of nc (netcat). It is good for transferring files and a lot of other TCP/UDP usages. Examples can be found in the nc man page. Here is an example to demonstrate transferring a file.

    ( On receiver: 192.168.0.10 )
    nc -l -p 3000 >file.txt

    ( On sender )
    cat file.txt | nc 192.168.0.10 3000

    New info, double ssh command (hop) to retrieve from a 3rd box. Just escape the inside quotes.

    ssh me@gateway "ssh me@protectedbox \"ls\""
    ssh me@gateway "ssh me@protectedbox \"tar -cf - .\"" | tar -xvf -
    tar -cf - sendfile | ssh me@gateway "ssh me@protectedbox \"tar -xvf -\""

    You can also try it without the quotes, just put final quotes on the specific ssh command:

    ssh me@gateway ssh me@gw2 ssh me@gw3 ssh me@gw4 "/sbin/ifconfig | grep inet"

    Note: You will get an X11 dialog box for the 2nd (protected -- i.e. behind the gateway or firewall box) box. You must have a functioning "X11 forwarding" for this to work, unless of course you have ssh keys working.

    For Debian, you can edit /etc/ssh/ssh_config and sshd_config, enable the X11 forwarding (and X11 forwarding agent for ssh_config).

    Then to test, make sure that "set | grep DISPLAY" returns good at each hop along the way. The result should be DISPLAY=localhost:10.0 or similar.

    If you get a message similar to "cannot find /usr/bin/ssh-askpass" then install the ssh-askpass package.




    Here are 2 examples of "double" ssh, sending file(s) through and executing a command on the remote side. As mentioned previously, the 2nd ssh will ask you the password via X11, not at the command line:

    tar -czf - news\ report.wmv | ssh user@domain1.com ssh root@192.168.0.1 "\(cd /boot \&\& tar -xzvf -\; md5sum news\\\ report.wmv\)"; md5sum news\ report.wmv
    tar -czf - news\ report.wmv | ssh user@domain1.com ssh root@192.168.0.1 "\(cd /boot \&\& tar -xzvf -\; md5sum \\\"news report.wmv\\\"\)"; md5sum news\ report.wmv
    This sets up compression so if you run an X11 app on the remote side it will come across compressed.
    ssh -C user@domain1.com
    This sets up a tunnel from your local nonexistant 1444 port to 143 (imap) on the remote end (mail server).
    This basically allows you to enter username/password and view mail without direct transfer over the internet -- i.e. I mean all your stuff is encrypted.
    To use, just point your mail client to localhost:1444 and it will automatically talk to 192.168.0.1:143.
    ssh -L 1444:192.168.0.1:143 user@domain1.com
    This one just combines both into 1 string. Now with this make your connection and you decide if you want to run your mail client locally (and access through the tunnel) or run your mail client remotely (with compression).
    ssh -C -L 1444:192.168.0.1:143 user@domain1.com
    Need to run something like cups configuration on a box behind a firewall? Here is one method, I'm sure there are other workable variations of these:

    Let's assume you only have the one firewall (domain1.com) to go through. This means that 192.168.0.1 doesn't block 631 and cups is configured to listen to the network (i.e. not just localhost - 127.0.0.1):

    ssh -L 8899:192.168.0.1:631 user@domain1.com


    Ah, if it was that simple. I don't have a firewall on the cups box; however, cups by default is not configured to listen to the network, it only listens to 127.0.0.1. So we need a little improvement on the previous command:

    ssh -L 8899:localhost:8899 user@domain1.com ssh -L 8899:localhost:631 user@192.168.0.1
    user@domain1.com's password:
    Pseudo-terminal will not be allocated because stdin is not a terminal.

    Now, remember you will get the x11 popup for your 2nd password. Also since this is a double ssh then you won't have an interactive prompt to play with either.

    Now just run mozilla (locally) and point to http://localhost:8899

    Again, 8899 is a non-existant local port that is created by ssh and used for the sole purpose of creating a tunnel between that port and the correct port on the remote end.

    Maybe if I rotate the port numbers you'll see their relationships:

    ssh -L 8899:localhost:8898 user@domain1.com ssh -L 8898:localhost:631 user@192.168.0.1

    The port numbers don't really matter, just as long as they can be stringed together from this end to the final destination. So in this example, locally 8899 gets sent to 8898 on domain1, then we take 8898 from domain1 and tunnel that to 631 on 192.168.0.1.
    Ok, another brain blower:

    My normal rsync looks like the following:

    rsync -e ssh -avx --progress user@domain1.com:/var/ftp/pub/knoppix/KN* .

    Here's the updated one that reaches behind the firewall:

    rsync -avx --progress -e "ssh user@domain1.com ssh user@192.168.0.1" :/var/ftp/pub/knoppix/KN* .

    Again, this is a double ssh, you will enter the password text mode for the first one (domain1.com) then you will enter the password graphics mode for the 2nd one (192.168.0.1). Of course if you have keys set up... you don't need any passwords.
    tar -cf - `ls ml* | grep -v reg`|ssh user@domain1.com ssh user@192.168.0.1 "\(cd ml \&\& tar -xvf -\)"
    # This sets compression for running remote X11 apps and displaying locally.

    ssh -C user@domain.com

    # This forwards local port 1444 to remote port 143

    ssh -L 1444:localhost:143 user@domain.com

    # This forwards local port 1444 to port 143 of 192.168.0.1 through domain.com

    ssh -L 1444:192.168.0.1:143 user@domain.com

    # This combines x11 compression and port forwarding

    ssh -C -L 1444:192.168.0.1:143 user@domain.com

    # This forwards local port 1444 to remote port 143 on 192.168.0.1
    # through domain.com, disallows the running of a command, and
    # sticks the ssh command in the background.

    ssh -f -N -L 1444:192.168.0.1:143 user@domain.com

    # This forwards local port 1444 to remote port 143 on 192.168.0.1
    # through domain.com, runs mozilla mail, and sticks the ssh
    # command in the background.

    ssh -f -C -L 1444:192.168.0.1:143 user@domain.com xcalc
    ssh -f -C -L 1444:192.168.0.1:143 user@domain.com mozilla -mail --no-remote

    1. To do ssh keys you need to generate your private and public keys.

    2. Take your public key and stuff it (append) into ~/.ssh/authorized_keys of
    whatever account you want to get into without a password.

    3. Then you have access with your private key.

    4. Also if you want to skip the yes/no question to trust the remote host,
    take the remote /etc/ssh/ssh_host_rsa_key.pub and append that to your
    ~/.ssh/known_hosts
    Rsyncing files from behind a firewall (multiple hops with rsync/ssh) without doing extra work: 
    rsync -e "ssh -X user@mydomain.com ssh -X user@192.168.1.9" -avx --progress --relative --bwlimit=10 :/pictures/20050420 .
user@mydomain.com's password:
    You will get the above command line prompt for your first ssh password. You will get the X11 popup for each of the subsequent password prompts (you can do multiple hops). This saves me from having to copy the files to my gateway, then locally, then delete from the gateway, not to mention needing extra space o n the gateway.
    prev next index
    Search this Site!:
    Search this site powered by FreeFind
    Homepage: http://www.cpqlinux.com
    Site Map: http://www.cpqlinux.com/sitemap.html