Notes to run MonteCarlo GSA (Morris and eFAST) for class code
————————————————————————————————————————————————————————————————
(c) R. Munoz-Carpena, ABE6933, 03/2018, carpena@ufl.edu
Please see an up to date README at https://abe.ufl.edu/faculty/carpena/software/GSUA_HPC.shtml

I. Preliminaries:

1. Login to you HPC account.
2. Uncompress the class files (.zip) from /ufrc/abe6933/share/class_GSA.zip to your HPC home directory (unzip -X /ufrc/abe6933/share/class_GSA.zip).
3. Load HPC extensions (module load ufrc)
4. Request an HPC node for 2 hrs to play around without bothering others... (srundev -t 120)
5. Load modules (HPC extensions and libraries) needed to process our class problem (module load class/abe6933)
6. If you have not done this so, your PATH (directories where programs are looked for by default regardless on what directory you are on the command line) must include the local directory you are on the command line ".". If you have not modified your /home/YOURGATORLINK/.bash_profile you must do so to add "." To the PATH line.  This line should read: 
PATH=$PATH:.:~/bin
Notice the ":.:", if it is not there add it with nano or another editor and save. If. You log out and in again you will have the capability of running programs from the directory you are on by default. This is important as many unix scripts rely on finding those where you are. If you do not want to log out your can type "source .bash_profile" and you can continue in the current session.


II. Explanation of directories under 2018class_GSA/:

A) bin/: contains unix utilities “abut” and “linex” used in sens_class. Copy directory to your home directory (cp -Rf bin ~).
B) src_java/: source code of the java class program 
C) base/: baseline execution directory for the class code. It contains:
   a) the java code executable (jar)
   b) pattern input file (input.dat.ptr)
   d) par.txt file with labels of input factors to change during the simulations
   c) sens_class script to substitute values in input ptr files (labels must be as in par.txt)
   d) test.sh to test the substitution of inputs with your base run
   e) files to automate the creation and submission of HPC jobs (make_jobs.sh, class_GSA.job.top class_GSA.job.bottom and matrix.txt)
   f) output: directory where results of simulations are written after jobs are done.
D) Outputs/: this is the directory where HPC output files will be written.


III. To test the HPC GSA class example follow these steps:

1. Change directory to “base” in your scratch directory.
2. Test the java code from the command line to make sure it runs OK (java -jar PredPrey1.jar -gsua input.dat output). It should finish in about 3-4 min and outputs should written to output/. Inspect input and output files (see README_PredPrey.txt for description).
3. Test the unix script (sens_class) to read new inputs from the command line, substitute them into the input file, run java program and save the outputs to output.all (sens_class 3.1 1.0 0.2 1.0 1.0 0.1 0.1 6.0). Check outputs again.
4. Let's create the jobs to run a typical GSA problem. Change directory down to your scratch directory (cd ..).
5. We will start with the Morris input sample matrix (morrisSU_matrix.txt). In this case we have 72 simulations (rows in that file) to run. Let's copy that matrix to another file matrix.txt used in the following steps (cp morrisSU_matrix.txt matrix.txt).
6. Edit the class_GSA.job.top by:
   a)  change where it says 
      “/home/carpena” to “/home/YOURGATORLINK” 
      "/ufrc/carpena/carpena/" to "/ufrc/abe6933/share/YOURGATORLINK/"
      "carpena@ufl.edu" to "YOURGATORLINK@ufl.edu"
      Notice that /ufrc/abe6933/share/YOURGATORLINK/ will be the "scratch" ($TMP_DIR) directory for executing the HPC jobs.
   b) check that the time you are requesting is enough to execute the number of simulations in each job (be generous!). 
   c) save the file!  
7. In this case, with 72 simulations it would take ~216 min (3.6 h) to run this problem. We wish to run this problem in ~15 min using the HPC. Since every runtimes ~3 min, I will assign 5 simulations to each HPC job and will run 15 jobs in parallel (15*5= 75 simulations, so job 15 will only have 2 jobs instead of 5, i.e. 72=14*5+2). 
8. Let's create the 15 jobs with the script make_jobs.sh (sh make_jobs.sh 72 5)
9. Submit the jobs with the submission script created by make_jobs.sh (sh class_submit.sh)
10. Now wait!! To check progress (queue -u YOURGATORLINK), the jobs will show as Q while waiting and R when running. You will receive emails as they complete. Results are in your /home/YOURGATORLINK/2018class_GSA/Outputs directory, 
11. Once the jobs are completed, we need to merge the output files in the correct order, from 1 to 15 so we end up with 72 rows or simulations in the final file. Change to the Outputs directory and run the script class_merge.sh created by make_jobs.sh (sh class_merge.sh). This will create the final file output outputs_all.txt to post process the Morris results.


