Skip to content
Snippets Groups Projects
Commit 4ebbc3c3 authored by Gary Ruben (Monash University)'s avatar Gary Ruben (Monash University)
Browse files

Removed some assertions to better support local operation

parent 6a33e965
No related branches found
No related tags found
No related merge requests found
...@@ -3,7 +3,7 @@ A script to transfer a tree of data files from a remote/source server to a ...@@ -3,7 +3,7 @@ A script to transfer a tree of data files from a remote/source server to a
local/destination computer. This runs on a local Linux machine or the eResearch dtn, on local/destination computer. This runs on a local Linux machine or the eResearch dtn, on
which the tape archive system is mounted; in our case, this is a machine at Monash. which the tape archive system is mounted; in our case, this is a machine at Monash.
Prior to running this an ssh key pair must be shared between the systems. See Prior to running this an ssh key pair must be shared between the systems. See
https://confluence.apps.monash.edu/display/XI/Australian+Synchrotron https://confluence.apps.monash.edu/display/XI/ssh+between+MASSIVE+filesystem+and+ASCI
for details on how to do this between a Monash Linux machine and ASCI for details on how to do this between a Monash Linux machine and ASCI
(Australian Synchrotron Compute Infrastructure). Requires Python 3.7 or higher (Australian Synchrotron Compute Infrastructure). Requires Python 3.7 or higher
and uses the fabric module. and uses the fabric module.
...@@ -22,8 +22,8 @@ Note that current version creates two files in the same directory as this script ...@@ -22,8 +22,8 @@ Note that current version creates two files in the same directory as this script
Known issues Known issues
------------ ------------
Note: The current version of fabric generates harmless warnings. This issue is Note: Some versions of fabric generate a harmless warning, which can be ignored. This
discussed here: https://github.com/paramiko/paramiko/issues/1369 issue is discussed here: https://github.com/paramiko/paramiko/issues/1369
Notes Notes
----- -----
...@@ -37,9 +37,6 @@ The command to count the number of files in a tarball ...@@ -37,9 +37,6 @@ The command to count the number of files in a tarball
$ tar -tf Lamb_Lung_Microfil_CT_18011B_right_CT.tar | wc -l $ tar -tf Lamb_Lung_Microfil_CT_18011B_right_CT.tar | wc -l
75920 75920
Keyword arguments for fabric's Connection.run() are documented here:
http://docs.pyinvoke.org/en/1.2/api/runners.html#invoke.runners.Runner.run
""" """
import os import os
import re import re
...@@ -219,18 +216,9 @@ def main( ...@@ -219,18 +216,9 @@ def main(
src_file_path = src_path.split("/")[:5] src_file_path = src_path.split("/")[:5]
if src_file_path[0] == "": if src_file_path[0] != "":
# Assume full path specified
assert "/".join(src_file_path[:4]) == f"/data/{experiment_name}/asci"
assert src_file_path[4] == "input" or src_file_path[4] == "output"
else:
assert src_file_path[0] == "input" or src_file_path[0] == "output"
src_path = os.path.join(f"/data/{experiment_name}/asci/", *src_file_path) src_path = os.path.join(f"/data/{experiment_name}/asci/", *src_file_path)
# remote_login = "gary.ruben@monash.edu@sftp1.synchrotron.org.au"
# src_path = f"/data/{experiment_name}/asci/output"
# dest_path = "/home/grub0002/bapcxi/vault/IMBL_2018_Oct_McGillick/output"
sys.stdout = Logger(log_filename) # Log all stdout to a log file sys.stdout = Logger(log_filename) # Log all stdout to a log file
print(textwrap.dedent(f""" print(textwrap.dedent(f"""
......
...@@ -3,7 +3,7 @@ A script to generate and transfer md5 checksum files from a remote/source server ...@@ -3,7 +3,7 @@ A script to generate and transfer md5 checksum files from a remote/source server
local/destination computer. This runs on a local Linux machine or the eResearch dtn, on local/destination computer. This runs on a local Linux machine or the eResearch dtn, on
which the tape archive system is mounted; in our case, this is a machine at Monash. which the tape archive system is mounted; in our case, this is a machine at Monash.
Prior to running this an ssh key pair must be shared between the systems. See Prior to running this an ssh key pair must be shared between the systems. See
https://confluence.apps.monash.edu/display/XI/Australian+Synchrotron https://confluence.apps.monash.edu/display/XI/ssh+between+MASSIVE+filesystem+and+ASCI
for details on how to do this between a Monash Linux machine and ASCI for details on how to do this between a Monash Linux machine and ASCI
(Australian Synchrotron Compute Infrastructure). Requires Python 3.7 or higher (Australian Synchrotron Compute Infrastructure). Requires Python 3.7 or higher
and uses the fabric module. and uses the fabric module.
...@@ -21,8 +21,8 @@ Running this creates two files in the same directory as this script file: ...@@ -21,8 +21,8 @@ Running this creates two files in the same directory as this script file:
Known issues Known issues
------------ ------------
Note: The current version of fabric generates harmless warnings. This issue is Note: Some versions of fabric generate a harmless warning, which can be ignored. This
discussed here: https://github.com/paramiko/paramiko/issues/1369 issue is discussed here: https://github.com/paramiko/paramiko/issues/1369
""" """
import os import os
...@@ -186,12 +186,7 @@ def main( ...@@ -186,12 +186,7 @@ def main(
src_file_path = src_path.split("/")[:5] src_file_path = src_path.split("/")[:5]
if src_file_path[0] == "": if src_file_path[0] != "":
# Assume full path specified
assert "/".join(src_file_path[:4]) == f"/data/{experiment_name}/asci"
assert src_file_path[4] == "input" or src_file_path[4] == "output"
else:
assert src_file_path[0] == "input" or src_file_path[0] == "output"
src_path = os.path.join(f"/data/{experiment_name}/asci/", *src_file_path) src_path = os.path.join(f"/data/{experiment_name}/asci/", *src_file_path)
sys.stdout = Logger(log_filename) # Log all stdout to a log file sys.stdout = Logger(log_filename) # Log all stdout to a log file
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment