University of FloridaDepartment of Agricultural & Biological Engineering

 

FITEVAL

Program for objective assessment of model goodness-of-fit with statistical signifficance based on Ritter and Muñoz-Carpena (2013).

Program Use & Output

FITEVAL can be executed as standalone application or as a MATLAB function.

An ASCII text file located in the same directory as the application is required. This file must contain
two paired vectors or columns: the first with the observations and the second with the model-calculated values file is required containing the observed and calculated values to be evaluated. The input file may contain missing values that must be denoted as nan. The filename can be passed as an argument (default filename is "fiteval.in"). The program finishes after closing the figure.

After running FITEVAL, it performs the goodness-of-fit evaluation providing a portable data file (pdf) containing:
a) a plot of observed vs. computed values illustrating the match on the 1:1 line;
b) the calculation of NSE and RMSE and their corresponding confidence intervals of 95%;
c) the qualitative goodness-of-fit interpretation based on the established classes;
d) a verification of the presence of bias or the possible presence of outliers;
e) the plot of the Ceff cumulative probability function superimposed on the Ceff class regions;
f) a plot illustrating the evolution of the observed and computed values.

Additionally, the numerical output is stored in a ascii text file. The plots can be obtained also as separated files in the specified (as argument) graphic format ('eps', 'pdf', 'jpg', 'tiff, or 'png'). A plot label can be provided by providing a "text string" as argument after the filename. Examples of use are provided in the section below.

Removing repeated cases in the observed and calculated values is possible by passing as second argument NOREP. For example: fiteval data_ex3.in NOREP. The program indicates the number of removals only if repeated cases present.

FITEVAL can apply Efron and Tibshirani (1993) bootstrap or Politis and Romano (1994) block bootstrap when dealing with time series. The latter is the default option.

The fitevalconfig.txt file is required when you want to run FITEVAL with other threshold values or for calculating Legates and McCabe (1999) modified form of the coefficient of efficiency (E1) instead of NSE. This file contains six lines specifying: Acceptable NSEthreshold, Good NSEthreshold, Very good NSEthreshold, relative bias threshold value (%), the option for computing E1, and figures’ font size, and the option for canceling the on-screen display of the graphical output (useful when using FITEVAL for multiple series calculations within Matlab code or other scripting environments). The corresponding default values are 0.65, 0.80, 0.90, 5, 0, 10, and 0, respectively (see below an example fitevalconfig.txt).

0.65 %Acceptable NSE threshold value

0.80 %Good NSE threshold value

0.90 %Very good NSE threshold value

5 %BiasValue

0 %Compute Legates and McCabe modified Ceff (1=yes,0=no)

0 % Bootstrap method (1= Efrons' bootstrapping, 0= block boostrapping)

10 % FontSizeValue

0 % Do not display graphical output (1=accept)

0 % Take into account observations uncertainty (>0,1,2,3 or 4= yes)

ro % Color and type of marker size

4 % Sizes of the series marker

To find different combinations of markers and colors that can be used in MatLab, please visit http://www.mathworks.com/help/matlab/ref/linespec.html for full description of options. For example, "ro" in the example above means "red circle" marker. For additional options in the fitevalconfig.txt file see the help file (fitevaloptions_help.pdf) in the distribution direrctory .

Examples

Several EXAMPLE files are included in this package (fiteval_1.in through fiteval_14s.in) and fiteval.in (default). The program input file must be written in ASCII or text format (be sure to select this option when saving the file with the editor of your choice). The program is exceduted from the DOS command prompt in Windows or from the terminal unix shell in OS X. Some examples are presented below,

A) Under Windows (or inside Matlab in both Windows and OS X)
fiteval <argument_list>

fiteval
fiteval fiteval.in norep
fiteval data_ex1.in
fiteval data_ex2.in
fiteval data_ex3.in
fiteval data_ex3.in norep
fiteval data_ex1.in "Example Data Set" jpg

Example of FITEVAL numerical output

Example of FITEVAL graphical output

B) Under Mac OS X

run_fiteval.sh <directory> <argument_list>

run_fiteval.sh <directory>
run_fiteval.sh <directory> fiteval.in norep
run_fiteval.sh <directory> data_ex1.in
run_fiteval.sh <directory> data_ex2.in
run_fiteval.sh <directory> data_ex3.in
run_fiteval.sh <directory> data_ex3.in norep
run_fiteval.sh <directory> data_ex1.in "Example Data Set" jpg

<directory> is the directory where the MCR (or the MatLab) is installed (see Library Requirements below). For exampe if Matlab is installed in your computer in the directory "/Applications/MATLAB_R2011a.app", the terminal command would be:

run_fiteval.sh /Applications/MATLAB_R2011a.app data_ex1.in

If instead the runtime libraries are installed, for example in "/Applications/MATLAB/MATLAB_Compiler_Runtime/v715" the the command would be:

run_fiteval.sh /Applications/MATLAB/MATLAB_Compiler_Runtime/v715 data_ex1.in

C) Running FITEVAL accounting for uncertainty of observed data or model predictions

When selecting uncertainty several options are available controlled from the fitevalconfig.txt (see help file for details, fitevaloptions_help.pdf). Typically, to run these cases, a combination of an option value (>0-4) in line 9 of the file fitevalconfig.txt and number of columns in the data input file (i.e. fiteval.in) will be required. It is important to match the option value with the number of columns to produce the desired results.

The inputs can be simplified when a common error distribution is used for all data points. For example, in order to run FITEVAL accounting for observation uncertainty (Case 9 in fitevaloptions_help.pdf) described by a normal distribution with a coefficient of variation of 17% common to all data points, the following fievalconfig.txt file will be required.

0.65 %Acceptable NSE threshold value

0.80 %Good NSE threshold value

0.90 %Very good NSE threshold value

5 %BiasValue

0 %Compute Legates and McCabe modified Ceff (1=yes,0=no)

0 % Bootstrap method (1= Efrons' bootstrapping, 0= block boostrapping)

10 % FontSizeValue

0 % Do not display graphical output (1=accept)

3 9 N 17 % Take into account observations uncertainty (>0,1,2,3 or 4= yes)

ro % Color and type of marker size

4 % Sizes of the series marker

The input file (i.e. fiteval.in) now requires only 2 columns (Observed vs. predicted values - see example fiteval_1.in). In this case FITEVAL will automaticallycompute the correction factor (CF) and upper and lower error bounds (UB_Omin, UB_Omax) for each observed value that are used to modify the goodness of fit indicators based on Harmel et al. (2010).

If the user desires to specify the CF, UB_Omin, UB_Omax for every individual data point, line 9 in the fitevalconfig.txt file now becomes,

3 % Take into account observations uncertainty (>0,1,2,3 or 4= yes)

and the fiteval.in file requires 5 columns (Yobs, Ypred, CF, UB_Omin, UB_Omax - see file fiteval_9s.in examples), as described in Case 9 of fitevaloptions_help.pdf.

Model errors like those generated from a Monte-Carlo uncertainty analysis, can be also considered (Case 13). For example, using the simplication of a common error distribution for all predictions, for example a triangular distribution with bounds given as 10% lower and 20% up from the predicted value, line 9 in the fitevalconfig.txt file now becomes,

4 13 T 10 20 % Take into account observations uncertainty (>0,1,2,3 or 4= yes)

and the fiteval.in file now requires only 2 columns (Observed vs. predicted values - see example fiteval_1.in). However if individual distributions are desired for each predicted point, the default 5 columns for Case 13 will be used (Yobs, Ypred, CF, UB_Pmin, UB_Pmax - see file fiteval_13s.in examples) and line 9 in the fitevalconfig.txt file now becomes,

4 % Take into account observations uncertainty (>0,1,2,3 or 4= yes)

 

D) Batch processing of data files

Notice that a batch file can be used for executing many FITEVAL examples automatically (to avoid stopping the execution after each on-screen graphical output set the appropriate flag in the fitevalconfig.txt file).

An example batch processing file (test_loop.m) is included in the distribution package for two data files (Example_1.in and Example_2.in). This can be extended for any number of files and names.

Library Requirements

If Matlab is already installed in your computer no addtional libraries are needed to execute this program.If you currently do not have Matlab 7.15 (R2011a) or above installed on your computer, download and install the MCRInstaller before running the application. This is a self-extracting utility, which depends on the end user's platform:

Once downloaded follow these instructions:

  1. Download and open the file.
  2. A command window opens and begins preparation for the installation.
  3. When the MCRInstaller wizard appears.
  4. You may specify where you want to install the MCR or accept the default folder.
  5. The installation begins and can take a few minutes to complete.

Once the installation is completed, MatLab developed stand-alone applications can be executed.

Program License

This program is distributed as Freeware/Public Domain under the terms of GNU-License. If the program is found useful the authors ask that acknowledgment is given to its use in any resulting publication and the authors notified. The source code is available from the authors upon request:

© Copyright 2012  Axel Ritter & Rafael Muñoz-Carpena


Return to top

References

  • Ritter, A. and R. Muñoz-Carpena. 2013. Predictive ability of hydrological models: objective assessment of goodness-of-fit with statistical significance. J. of Hydrology 480(1):33-45. doi:10.1016/j.jhydrol.2012.12.004

Return to top

This page was last updated on February 09, 2016.