Standards for Hydrodynamic Model Skill Assessment
The National Ocean Service (NOS) is developing and implementing oceanographic nowcast and forecast modeling systems to support navigational and environmental applications in U.S. coastal waters. These prediction systems provide NOAA users with nowcasts (i.e., analysis) and forecast guidance of water levels, currents, water temperature, and salinity for the next 24 to 36 hours. NOS requires these modeling systems, whether developed within or outside NOS, be assessed for skill in adherence to NOS standards. Skill assessment is an objective measurement of how well the model nowcast or forecast guidance does when compared to observations. The Coast Survey Development Laboratory (CSDL) leads the effort to develop coastal ocean and estuarine nowcast/forecast modeling systems. Once these systems demonstrate sufficient skill they are transitioned to run operationally by NOS’ Center for Operational Oceanographic Products and Services (COOPS). Therefore, CSDL and COOPS have partnered in the development of standards to evaluate operational hydrodynamic modeling systems. Skill assessment statistics that quantify model performance are calculated from significant model variables (e.g., water levels and currents) according to prescribed routines. In order for a nowcast/forecast system to be accepted operationally the skill assessment statistics must meet prescribed targets as described in the standards documentation.
Hydrodynamic Model Skill Assessment Software
A skill assessment software package has been designed to perform skill assessment for water levels, currents, temperature, and salinity for different model systems in both tidally dominated and nontidal regions. The skill assessment statistics (see Table 1) are computed automatically using data files containing observed, nowcast, and forecast variables. The observations, such as verified water levels, currents at NOS Physical Oceanographic Real Time System (PORTS) stations, and tidal constituents, can be directly acquired via the Internet from a COOPS database. Different types of data are processed and the skill assessment results are listed in tables valid at the selected verification stations. The package’s processing routines include tidal prediction, harmonic analysis, gap filling, filtering (or singular value decomposition), and other methods. The routines also include ways of concatenating nowcast and forecast guidance, and in extracting extrema. This package can be run in Unix or Linux environments. All Fortran programs can be compiled using Fortran compilers, version 77, or above. Documentation of the skill assessment software is available which describes the analysis and programs in further detail and is available for download.
Table 1. Skill Assessment Statistics
Variable 
Explanation 
Error 
The error is defined as the predicted value, p, minus the reference (observed or astronomical tide value, r : e_{i} = p_{i}  r_{i}. 
SM Series Mean 
The mean value of a series y.
Calculated as


RMSE 
Root Mean Square Error
Calculated as 
SD 
Standard Deviation
Calculated as 
CF(X) 
Central Frequency. Fraction (percentage) of errors that lie within the limits +X. 
POF(X) 
Positive Outlier Frequency. Fraction (percentage) of errors that are greater than X. 
NOF(X) 
Negative Outlier Frequency. Fraction (percentage) of errors that are less than X. 
MDPO(X) 
Maximum Duration of Positive Outliers. A positive outlier event is two or more consecutive occurrences of an error greater than X. MDPO is the length of time (based on the number of consecutive occurrences) of the longest event. 
MDNO(X) 
Maximum Duration of Negative Outliers. A negative outlier event is two or more consecutive occurrences of an error less than X. MDNO is the length of time (based on the number of consecutive occurrences) of the longest event. 
WOF(X) 
Worst Case Outlier Frequency. Fraction (percentage) of errors that, given an error of magnitude exceeding X, either (1) the simulated value of water level is greater than the astronomical tide and the observed value is less than the astronomical tide, or (2) the simulated value of water level is less than the astronomical tide and the observed value is greater than the astronomical tide. 
References
Hess, K. W., T. F. Gross, R. A. Schmalz, J. G. W. Kelley, F. Aikman, III, E. Wei, and M. S. Vincent, 2003: NOS standards for evaluating operational nowcast and forecast hydrodynamic model systems, NOAA Technical Report NOS CS 17, 47 pp.
Zhang, AJ., K.W. Hess, E. Wei, and E. Myers, 2006: Implementation of model skill assessment software for water level and current in tidal regions, NOAA Technical Report NOS CS 24, 61 pp.
