Skip to content

Transferring data

Using the command line interface

Usage of aliases

In the following commands meluxina refers to an alias that makes it easy to access and transfer data to the system

The scp command

The most common way of transferring data to a supercomputer is by using the scp command. It works very similarly to the regular cp command and uses ssh to connect to the remote machine. As such, it is mandatory to have configured a ssh key as described in the connecting to MeluXina section. As with any Linux command, more information can be obtained by using the command man scp.

The usage is best illustrated by a few examples:

You have a terminal open on your local machine and want to transfer a file to MeluXina :

scp my_local_file meluxina:my_remote_directory

To transfer a full directory from your machine, add the -r (recursive) option. This applies to all the following examples.

scp -r my_local_directory meluxina:my_remote_directory

Transferring a remote file back to your local machine, while being logged into your local machine:

scp -r meluxina:my_remote_directory my_local_directory

As you can see there is symmetry between the source and the destination arguments. If your machine is accessible from the internet, you can naturally run the command from the MeluXina system. The general command looks like:

scp source_host:source_file destination_host:destination_file
scp -r source_host:source_dir destination_host:destination_file

The host can be omitted when referring to the machine you are issuing the command from.

The sftp command

This is a secure version of the classic interactive ftp program.

You can connect to MeluXina with sftp as such:

sftp meluxina

You will then get a sftp> prompt where you can type help to see a list of possible commands.

The rsync command

rsync implements a sophisticated algorithm that allows to transfer only missing/non-matching parts of a source file to update a target file. With this the process of transferring partially modified data may be significantly faster than simply replacing all data. Among other things, rsync also allows you to specify complex filter rules to exclude certain files or directories located inside a directory that you want to sync. As with any Linux command, more information can be obtained by using the command man rsync.

For immediate usage, let us walk over a few examples.

You have a terminal open on your local machine and want to transfer a file to MeluXina :

rsync my_local_file meluxina:my_remote_directory

To transfer a full directory from your machine, add the -az (a: similar to -r, but also preserves timestamps, permissions, ownership, and preserve symlinks and z: to compress data) option. This applies to all the following examples.

rsync -az my_local_directory meluxina:my_remote_directory

To transfer a remote file back to your local machine, while issuing the command from your local machine:

rsync -az meluxina:my_remote_file my_local_directory

Using rsync safely

Before using rsync: it is highly recommended that you use the -n option to test which changes will be made.

Including/Excluding files

With the --include/--exclude options you can specify patterns describing which files are not included/excluded from the copy process.

To exclude a specific directory:

rsync -avz --exclude "subdir1" my_local_directory meluxina:my_remote_directory

To only copy files with .txt and .md suffixes:

rsync -avz --include "*.txt" --include "*.md" --exclude "*" my_local_directory meluxina:my_remote_directory

Info

To be efficient, the following options can be used with rsync command:
- a is for archive mode
- v is for verbose mode
- z is for compressed mode

Unable to transfer files

If you are able to connect to MeluXina by SSH, but cannot transfer files using scp, sftp or rsync you may have added something to your ~/.bashrc file on MeluXina that breaks file transfers. This will happen for instance if a command in your ~/.bashrc executes echo or printf.

To resolve this issue, you may add the following code close to the top of your ~/.bashrc file, before your customizations.

if ! [[ $- == *i* ]]; then
    # This is *not* an interactive shell, don't run the rest of the script.
    return
fi

If you're setting a custom umask, set it BEFORE the above statement, or your umask will not get applied to uploaded files.