auto_process_ngs.fileops

Utility functions providing a single interface for performing various file system operations (e.g. make a directory, copy files etc) transparently on either a local or a remote system.

The following functions perform specific operations directly:

  • mkdir: create a directory

  • copy: copy a file

  • copytree: recursively copy a directory

  • set_group: set the group on a file or directory

  • set_permissions: set permissions on a file or directory

  • unzip: unpack a ZIP archive

  • rename: rename (move) a file or directory

  • listdir: list the contents of a directory

  • exists: test if a file or directory exists

  • isdir: test if a path is a directory

  • remove_file: remove (delete) a file

  • remove_dir: remove (delete) a directory

  • disk_usage: get info on disk usage for a path

These functions generate commands that can be executed e.g. via a scheduler, to perform the required operations:

  • copy_command: generate command to perform copy operation

  • copytree_command: generate command to perform recursive copy operation

  • set_group_command: generate command to perform group set operation

  • set_permissions_command: generate command to perform permissions updating operation

  • unzip_command: generate command to unpack a ZIP archive

auto_process_ngs.fileops.copy(src, dest, link=False)

Copy a file

Copies a local file to a local or remote location.

Parameters:

  • src (str) – local file to copy

  • dest (str) – destination (file or directory) on a local or remote system, identified by a specifier of the form ‘[[USER@]HOST:]DEST’

  • link (bool) – hard link files instead of copying (ignored for remote copies)

auto_process_ngs.fileops.copy_command(src, dest, link=False)

Generate command to copy a file

Creates a command which copies a local file to a local or remote location.

Parameters:

  • src (str) – local file to copy

  • dest (str) – destination (file or directory) on a local or remote system, identified by a specifier of the form ‘[[USER@]HOST:]DEST’

  • link (bool) – hard link files instead of copying (ignored for remote copies)

Returns:

Command instance that can be used to

perform the copy operation.

Return type:

Command

auto_process_ngs.fileops.copytree(src, dest)

Recursively copy a local directory tree

Recursively copies an entire local directory tree rooted at ‘src’, to a local or remote destination directory ‘dest’.

Note that if ‘dest’ already exists then ‘src’ will be copied into it as a subdirectory.

Parameters:

  • src (str) – local directory to copy

  • dest (str) – destination directory) on a local or remote system, identified by a specifier of the form ‘[[USER@]HOST:]DEST’

auto_process_ngs.fileops.copytree_command(src, dest)

Generate command to recursively copy a directory tree

Creates a command which recursively copies an entire local directory tree rooted at ‘src’, to a local or remote destination directory ‘dest’.

Note that if ‘dest’ already exists then ‘src’ will be copied into it as a subdirectory.

Parameters:

  • src (str) – local directory to copy

  • dest (str) – destination directory) on a local or remote system, identified by a specifier of the form ‘[[USER@]HOST:]DEST’

Returns:

Command instance that can be used to

perform the copy operation.

Return type:

Command

auto_process_ngs.fileops.disk_usage(path)

Get disk usage for a path

Wraps the psutil ‘disk_usage’ function for local paths, and runs the ‘df’ command for paths on remote systems. Raises OSError if the path doesn’t exist.

Parameters:

path (str) – path to directory

Returns:

NamedTuple with fields ‘total’, ‘used’,

’free’ and ‘percent’.

Return type:

NamedTuple

auto_process_ngs.fileops.exists(path)

Test if a file or directory exists

Parameters:

path (str) – path to file or directory to check existence of

Returns:

True if file or directory exists,

False otherwise.

Return type:

Boolean

auto_process_ngs.fileops.isdir(path)

Test if a path is a directory

Parameters:

path (str) – path to check

Returns:

True if path is a directory,

False otherwise.

Return type:

Boolean

auto_process_ngs.fileops.listdir(path)

List contents of a directory (including hidden files)

auto_process_ngs.fileops.mkdir(newdir, recursive=False)

Create a directory

The new directory should be identified using a specifier of the form ‘[[USER@]HOST:]NEWDIR’.

The parent directories must already exist, unless the ‘recursive’ argument is set (in which case all missing parent directories will also be created).

Parameters:

  • newdir (str) – location of the new directory (can be on local or remote system)

  • recursive (bool) – if True then also create missing parent directories (default is not to create missing directories)

auto_process_ngs.fileops.remove_dir(path)

Remove (delete) a directory

Parameters:

path (str) – path to directory to delete

Returns:

zero on success, non-zero on

failure.

Return type:

Integer

auto_process_ngs.fileops.remove_file(path)

Remove (delete) a file

Parameters:

path (str) – path to file to delete

Returns:

zero on success, non-zero on

failure.

Return type:

Integer

auto_process_ngs.fileops.rename(src, dst)

Rename (move) a file or directory

Parameters:

  • src (str) – path to file or directory to rename

  • dst (str) – path to rename ‘src’ to

auto_process_ngs.fileops.set_group(group, path)

Set the group for a file or directory

‘path’ can be a file or directory on a local or remote system; if it is a directory then it will operate recursively i.e. all subdirectories and files will also have their group changed.

‘group’ is the name of the group to change ownership to (must exist on the target system).

Parameters:

  • group (str) – name of the new group

  • path (str) – path to the file or directory to change the group of, identified by a specifier of the form ‘[[USER@]HOST:]PATH’

auto_process_ngs.fileops.set_group_command(group, path, verbose=False, safe=False)

Generate command to set group on file or directory

Creates a command which sets the group on a file or directory.

‘path’ can be a file or directory on a local or remote system; if it is a directory then it will operate recursively i.e. all subdirectories and files will also have their group changed.

‘group’ is the name of the group to change ownership to (must exist on the target system).

Parameters:

  • group (str) – name of the new group

  • path (str) – path to the file or directory to change the group of, identified by a specifier of the form ‘[[USER@]HOST:]PATH’

  • verbose (bool) – if True then output a diagnostic for every file processed (default: don’t output diagnostics)

  • safe (bool) – if True then run in ‘safe’ mode i.e. only files owned by user will be have group updated (default: try to set on all files and directories)

Returns:

Command instance that can be used to

set the group.

Return type:

Command

auto_process_ngs.fileops.set_permissions(permissions, path)

Set or update the permissions for a file or directory

‘path’ can be a file or directory on a local or remote system; if it is a directory then it will operate recursively i.e. all subdirectories and files will also have their permissions updated.

‘permissions’ must be a valid permissions string as recognised by the ‘chmod’ command.

Parameters:

  • group (str) – ‘chmod’ permissions string

  • path (str) – path to the file or directory to change the group of, identified by a specifier of the form ‘[[USER@]HOST:]PATH’

auto_process_ngs.fileops.set_permissions_command(permissions, path, verbose=False, safe=False)

Generate command to set group on file or directory

Creates a command which sets the group on a file or directory.

‘path’ can be a file or directory on a local or remote system; if it is a directory then it will operate recursively i.e. all subdirectories and files will also have their group changed.

‘permissions’ must be a valid permissions string as recognised by the ‘chmod’ command.

Parameters:

  • permissions (str) – permissions string

  • path (str) – path to the file or directory to change the group of, identified by a specifier of the form ‘[[USER@]HOST:]PATH’

  • verbose (bool) – if True then output a diagnostic for every file processed (default: don’t output diagnostics)

  • safe (bool) – if True then run in ‘safe’ mode i.e. only files owned by user will be have permissions updated (default: try to set on all files and directories)

Returns:

Command instance that can be used to

set the group.

Return type:

Command

auto_process_ngs.fileops.unzip(zip_file, dest)

Unpack ZIP archive file on local or remote system

Parameters:

  • zip_file (str) – ZIP archive file identified using a specifier of the form ‘[[USER@]HOST:]ZIP_FILE’

  • dest (str) – path to extract the archive contents to (on the same system as the ZIP archive)

auto_process_ngs.fileops.unzip_command(zip_file, dest)

Generate command to unpack ZIP archive file

The ZIP archive can be on a local or a remote system.

Parameters:

  • zip_file (str) – ZIP archive file identified using a specifier of the form ‘[[USER@]HOST:]ZIP_FILE’

  • dest (str) – path to extract the archive contents to (on the same system as the ZIP archive)

Returns:

Command instance that can be used to

perform the unzip operation.

Return type:

Command