Use & Output
Starting the Program • Input Data File Description • Outputs Description • Configuration Options
Running without Consideration of Uncertainty: Running Simple Case • Outliers and Model Bias • Checking the Effect of Repeated Values • Batch Processing
Accounting for Observations and Model Uncertainty
Starting the program GUI
After starting the program (click on app icon), the program GUI opens as shown below.
The program GUI is divided into the workspaces shown in the image below (outlines and red labels). The main program options are laid out on the left of the GUI, where the model evaluation data is shown next to it, and the outputs to the right. Specific options to incorporate uncertainty in the evaluation are presented at the bottom (only visible when the Uncertainty option is selected). Note the information messages in blue that will show at times during the different program actions. Details on the options and the workspaces are provided in the sections below.
Back to Top
Input data file description
A user supplied input ASCII data file (with extension ".in") is required and must be located in a working directory selected by the user. In this documentation we will use the "Example" directory provided in the installation package (see note at the end of the previous section). The input data file (*.in) must contain at least two paired vectors or columns,
{observations, predictions [...]}
The input file may contain missing values that must be denoted as "nan". The program can handle many additional options (comparisons with benchmark data, uncertainty in observed values, uncertainty in the simulated results and combinations of these) by specifying additional columns in the data input file. See the fitevaloptions_help.pdf file and corresponding input file examples provided in the distribution directory.
Back to Top
Outputs description
After running the evaluation case, the program presents a summary of statistics and a composite figure with the graphical assessment. The goodness-of-fit evaluation results are also provided in a portable data file (pdf) and a portable network graphics file (png) with the same name as the input data file, written to the same working directory where the .in file is located. The screen output and pdf files contain the following information:
a) plot of observed vs. model computed values illustrating the match on the 1:1 line (line of perfect agreement);
b) calculation of NSE and RMSE and their corresponding 95% confidence intervals;
c) plot of the approximated NSE cumulative probability function superimposed on the NSE "pedigree" regions;
d) plot illustrating the series of the observed and computed values;
e) qualitative goodness-of-fit interpretation based on the model "pedigree" classes (Acceptable, Good and Very Good);
f) p-value representing the probability of wrongly accepting the fit (NSE≥ NSEthreshold=0.65). For p-value >α, the model fit is considered Unsatisfactory;
g) model bias when it exceeds a given threshold (>5% by default);
h) possible presence of outliers;
i) normalized root mean squared error NRMSE (in %)= 100·RMSE/(standard deviation of the observations);
j) calculation of Kling–Gupta efficiency (KGE) and corresponding 95% confidence interval.
The 1:1 and series plots help to visually inspect the similarity degree of the two series and detect which observations are best or worst predicted by the model. When uncertainty is incorporated in the evaluation, the resulting 1:1 plot shows the uncertainty boundaries as vertical (measurement uncertainty) or horizontal (model uncertainty) error bars, whereas the series plot shows both as vertical error bars. Additionally, the numerical output is stored in a ASCII text file with the same name as the input data file and extension '.out'.
The individual plots can be obtained also as separated files in the specified (as argument) graphic format ('eps', 'pdf', 'jpg', 'tiff, or 'png'). For this, select the file type from the GUI menu or leave blank for no separate figures as shown below,
Removing repeated input paired values is possible by selecting that Option at the bottom of the GUI on the left. In this case, the program output indicates the number of removals only if repeated paired values are present. Additional details are provided below in the section "Checking the effect of repeated values"
Back to Top
Configuration options
Several configuration options are offered to accommodate running FITEVAL with other threshold values, calculating Legates and McCabe (1999) modified NSE, and others. To access the options, select "Configuration" on the left of the GUI menu as shown below. Notice that this will open automatically the first time that the program is open, too.
The options available include: NSE thresholds for Acceptable, Good, Very good model "pedigrees" (with default values of 0.65, 0.80, 0.90 [Ritter and Muñoz-Carpena, 2013]); relative bias threshold value (%); bootstrapping options for the original Efron and Tibshirani (1993) or Politis and Romano (1994) block bootstrap when dealing with time series or autocorrelated data; the number of bootstrapping samples (a value of 20000 is recommended for robustness); the option for computing E1, and the graphs’ font size and markers symbol type and colours. To find different combinations of markers and colors that can be used in MatLab, please visit http://www.mathworks.com/help/matlab/ref/linespec.html for full description of options. For example, "o,b" in the screenshot above indicates "blue circle" markers.
Back to Top
Running FITEVAL GUI without consideration of uncertainty
Running a simple case
Several EXAMPLE files are included in the ZIP installation package (Examples directory). The program input file must be written in ASCII or text format (be sure to select this option when saving the file with the editor of your choice). To start the model evaluation with the GUI, click on "Load" on the options on the left hand side of the GUI and a file selection panel will open. Select an input file and click "Open". If the file is present in the working directory, it can be also loaded by typing its name in the corresponding field (on the left of “Load” button) and pressing "Enter". For example, we will open "data_ex0.in" in the image below:
After loading the file the contents will appear in the GUI (Ypred and Yobs columns) so that the user can check that the correct data are imported. Note that the data columns could also be selected from another application without opening a file by copying them in the pasteboard first from the other application, and then clicking "Paste data" in the FITEVAL GUI. The pasted data can be saved into a file by providing the new file name (with file extension .in) and pressing "Save to file" in the GUI menu. If needed, old output files for the loaded file can be deleted automatically by pressing "Delete Output" before running.
Once the data is loaded the model evaluation is run by pressing "Run" in the left menu. Note that while running (and other program operations) a message in blue appears on the bottom of the GUI to inform the user on what action is being taken. After the program finishes, the ouput results (described above) are presented as shown below:
The corresponding pdf file generated automatically in this run is also shown below (located in the same directory as the input file):
The numerical output from the model evaluation can be opened by pressing on "Numerical Output" as shown in the image below. The text output can be copied also into the clipboard and pasted as text into other applications as needed. Similarly, the graphical output file can also be opened by pressing "Zoom" in the GUI and copied and pasted in other applications.
Back to Top
Outliers and model bias
Outliers (Dixon's test) and bias (with threshold set in the configuration file above) are automatically checked when running the program to guide the user about other potential sources of differences between observed and simulated values. Information for both tests is included in the graphical (filled symbols and annotated coordinates) and text outputs. An example is provided below for the file "data_ex2.in" (included in Examples directory with the installer distribution package):
Note that the specific outlier coordinates are provided also in the numerical output window and in the output text file found in the working directory.
Back to Top
Checking the effect of repeated values
For data series that move between or around certain thresholds, often data values at the thresholds can be repeated several times. If the model has been calibrated to those thresholds, the repeated values can provide an unfair bias to the model assessment.
For example, in the graphs below, the model evaluation results for the example file "data_ex3.in" are presented. The series graph shows that the observed and simulated values move between an upper and lower threshold most of the time. By selecting the FITEVAL GUI option "Remove repeated" as presented above, the program will evaluate this potential effect. The results of the model evaluation with and without repeated values are presented below for comparison.
a) Results with all data (without removing repeated values):
b) Results with repeated data removed (24 data pairs were removed):
Back to Top
Batch processing of data files
The FITEVAL GUI includes a simple but complete batch system for executing several model evaluations automatically. An ASCII text file can be created and loaded with the multi-run information, or this can be prepared within the GUI. To access this feature, select "Multi Run" in the FITEVAL GUI and "Load" the example included in the "Examples' directory "multi_run_test.txt" as shown in the illustration below:
Once the batch file is loaded, multiple evaluations can be performed automatically by selecting "Run" in the pane as shown below. The different model evaluations will be run and shown in the GUI as they progress until the last one is completed. The blue information messages in the bottom of the GUI will show the current evaluation and the runs will be completed when the multi-run panel appears on the screen again.
Briefly, the contents and options of the multi-run shown on the last image above are explained below. Note that the uncertainty options will be presented in the next section (see also fitevaloptions_help.pdf file for details).
data_ex1.in; 0 %use data_ex1.in with no uncertainty options ("0") for a noisy model fit
data_ex4.in; 0 %use data_ex4.in with no uncertainty options ("0") for a good model fit
data_ex2.in; 0.25 %use data_ex2.in with PER=0.25 observed data uncertainty
data_ex3.in norep; 0 %use data_ex3.in removing repeated values (norep)
data_ex0.in; 3 9 oN 17 %use data_ex0.in with CF method for obs. data uncertainty with Normal error distribution and CV%=17
data_ex0.in; 4 13 pT 10 20 %use data_ex0.in with CF method for model uncertainty with Triangular error distribution
data_ex0.in; 3 11 oT 25 25 pT 15 20 %use data_ex0.in with CF method for data and model uncertainty with Triangular error distributions for both
The multi-run script can be prepared within the GUI by adding new lines when pressing "Add" in the menu and filling in the corresponding information as described above. The script can be saved into an ASCII text file by writing the new file name (with file extension .txt) in the text field where "multi-run.txt" (default) appears, and selecting "Save". All lines can also be cleared to start a new batch as needed by pressing "Clear". Any line can be deleted by leaving it blank, except for the first one.
Back to Top
Running FITEVAL GUI accounting for observations and model uncertainty
Following Ritter and Muñoz-Carpena (2020) Several options are provided in FITEVAL to consider uncertainty in observations and/or model outputs based on Harmel and Smith (2007) and Harmel et al. (2010), and extensions by the FITEVAL authors. The uncertainty options are presented in the GUI when selecting "Uncertainty" in the general program options in the left of the GUI. The options are summarized in the help file (fitevaloptions_help.pdf).
As shown in the image below, after selecting "Uncertainty" the user has to select the following uncertainty options before running the case: uncertainty method (PER with common value or individual errors, or Correction factor CF), type of uncertainty (observed data, model, data and model), and when CF is selected the type and statistics of the error distribution function.
IMPORTANT NOTE:
- When running uncertainty cases the program will run twice, the first time with uncertainty and the second without uncertainty. Please compare both sets of results (PDF files generated in working directory, with and without "U" in file name).
A) Running FITEVAL GUI accounting for uncertainty of observed data
To include uncertainty in observations when evaluating the model, the user must click "Uncertainty" in the general program menu (bottom left) and select the proper uncertainty options in the menu that unfolds at the bottom of the GUI. Note that as described above the option numbers to include uncertainty in the multi-run batch must be: 0<PER<1, 1, 2, 3 (see help file for details, fitevaloptions_help.pdf).
For example, if the user desires to specify the CF method with a Normal distribution for observed data error with CV(%)=20, the user would select the options (corresponding to option 3 and case 9 in fitevaloptions_help.pdf).) shown below:
To use these same options in multi-run batch the corresponding line should be written as: data_ex0.in; 3 9 oN 20. Note that FITEVAL computes the correction factors (CFi) and the uncertainty bounds automatically based on the selected probability distribution and saves the results in the working directory with the original filename and a "U" added at the end (with file extensions .txt, .pdf, png). In addition, to assess the effect of uncertainty in the model evaluation results, FITEVAL also computes automatically the base case without uncertainty options for the same input data and writes the results with the same filename as the original input with the corresponding file extensions. For example, evaluating the model with the input file "data_ex0.in" with the options above will produce the following result ("data_ex0U.pdf"):
compared to base case ("data_ex0.pdf"):
The user can also compare both sets of results in the GUI by selecting the "Fig. No Unc." option as shown below:
Although the recommended method for including uncertainty in the model evaluation is CF based on its robust statistical underpinning and plausibility (Ritter and Muñoz-Carpena, 2020), FITEVAL GUI contains other methods to include observation errors in the model evaluation, including a single probable error range (PER) for all data points, individual PER for each data point, or individual asymmetric PER bounds. These can be selected in the Uncertainty options menu on the bottom of the screen.
For the case of "Observation errors" an extension of the PER method is applied where the user can provide a series of individual errors for each data point (as % bounds, or individual PER) in an ASCII file with extension ".err". An example is shown below for the case of % error asymmetric bounds with the model data file "data_ex0.in" and the user supplied error series "data_ex0_UBs.err", both found in the "Examples" directory. Note that the user could add manually the individual % error bounds (or PER if that option is selected) directly in the GUI but the number of error values must match that of the original data.
B) Running FITEVAL GUI accounting for uncertainty of model results
Model errors like those generated from a Monte-Carlo uncertainty analysis, can be also included in the model evaluation based on the CF method when selecting the "Correction factor" and "Prediction Unc." options. For example, for the input file "data_ex0.in" and assuming a common error distribution for model predictions described by a triangular distribution with bounds between +10% and -20% from the predicted value, we obtain:
If individual errors are desired for each predicted point, the companion FITEVAL command line MatLab executable function (fiteval.p) can be used (see FITEVAL MatLab documentation here).
C) Running FITEVAL GUI accounting for both uncertainty of observed data and model results
Model and observations uncertainties can be also considered together in the evaluation based on the CF method. In this case, both "Observation Unc." and "Prediction Unc." options are selected and distributions assigned for each source of uncertainty. For example, for the input file "data_ex0.in" assuming a common error distribution for observations described by a Normal distribution with CV=20%, and for prediction errors described by a triangular distribution with bounds between +10% and -20% from the predicted values, we obtain:
To use these same options in multi-run batch the corresponding line should be written as: data_ex0.in; 3 11 oN 20 pT 10 20. As before, if individual errors are desired for each predicted and observed points, the companion FITEVAL command line MatLab executable function (fiteval.p) can be used (see FITEVAL MatLab documentation here)
Back to Top
