Data Assimilation
in Climate and Energy
Systems Engineering
Data and Method
Cases
In this project, we selected and analyzed past cases of localized heavy rain in Korea where prediction failures were clearly evident. Through each case, we aimed to compare and analyze the common characteristics of localized heavy rain and the difficulty of prediction, and to identify the causes of these prediction failures.
Common Synoptic Environment
-
Atmospheric convergence formed at a stationary front or the edge of the North Pacific High
-
Abundant water vapor supply from warm and humid air influx from the Yellow Sea and southern seas
-
Strong updrafts under upper-level jet streams and troughs → Significant intensification of atmospheric instability
Precipitation Characteristics
-
Localized heavy rain concentrated in a narrow area due to strong convective precipitation bands
-
Very high spatial and temporal variability of precipitation → Potential degradation of numerical weather forecasting model accuracy
CSI (Critical Success Index)
CSI(Critical Success Index, 임계성공지수) is a quantitative representation of the ratio of cases where the prediction was correct among cases where heavy rain occurred or was predicted.
Correct negatives(CN)—that is, cases where neither the prediction nor the actual rainfall was heavy—are not included in the CSI calculation. This is important because, in rare events like localized torrential rain, there can be too many instances where the prediction "no rain" is correct. Including all such cases can create the illusion that the model accurately predicted the rainfall.
In this project, the CSI is used to quantitatively evaluate the prediction performance of localized, sudden torrential rain. The CSI is an indicator that assesses the prediction success rate based on whether the rainfall actually occurred or was predicted, and it is calculated using Hits, Misses, and False Alarms.
A Hit refers to cases where torrential rain occurred in both the prediction and the observation; a Miss refers to cases where torrential rain actually occurred but was not predicted; and a False Alarm refers to cases where torrential rain was predicted but did not actually occur.
The CSI value ranges from 0 to 1, and a value closer to 1 is interpreted as indicating higher torrential rain prediction performance. However, since CSI does not include Correct Negatives—where heavy rain was predicted not to occur and actually did not—while it is suitable for evaluating the predictive performance of rare events like heavy rain, one must also consider its limitation that it may be affected by the frequency of such events.
Analysis of localized heavy rain prediction accuracy
This method calculates CSI by aligning observations (Observation: AWS/ASOS), operational forecast fields (Background: LDAPS/KLAPS, etc.), and analysis fields (Analysis: ERA5, etc.) within the project overview based on the same time and space, and then classifying H/M/F. In practice, CSI should not be used as a standalone metric, but should be interpreted in conjunction with POD, FAR, and Bias.
Since public LDAPS/KLAPS are operational deliverables that already reflect data assimilation, it is safer not to assume they represent a pure background field prior to data assimilation. Furthermore, because ERA5 is a reanalysis reference field rather than a correct observation, the most appropriate configuration is to use AWS/ASOS as the truth for verification and ERA5 as the analysis reference field.
CSI and 2×2 Classification Table
CSI is a hit rate indicator calculated by reflecting ‘Misses (M)’ and ‘False alarms (F)’ along with ‘Correct Hits (H)’, covering only cases where heavy rain actually occurred or was predicted.
The value ranges from 0 to 1, with values closer to 1 being better. However, since it is affected by event frequency and forecast bias, interpretation must be performed in conjunction with supplementary indicators.
CSI uses only Hit, Miss, and False Alarms and does not include Correct Negatives(CN), where both prediction and observation were “non-heavy rain.” Therefore, while it is advantageous for verifying localized heavy rain, which is a rare event, it is affected by event frequency and bias.
This 2×2 contingency table separates and illustrates the nature of prediction errors. Higher H is better, M represents missed risks, and F represents over-alerts.
While CN is important for calculating overall accuracy, it is excluded from the CSI because the score can be overestimated if there are too many no-events in the verification of rare events.
Even if the CSI is high, it must be verified separately whether that value is due to high PODs, low FARs, or was obtained amidst over-forecasting (Bias > 1). This is because different H/M/F combinations can produce the same CSI.
Comparative Data Design
AWS provides minute and hour data from 554 locations, while ASOS provides minute, hour, day, month, and year data from 105 locations, along with QC information.
LDAPS currently offers 48-hour forecasts at 1.5 km and 70 layers, four times a day, and operates through its own 3D-Var analysis-forecast cycle.
The KLAPS public page explains the data as 3D analysis and forecasting of the Korean Peninsula, while the publicly available specification data presents a version at 5 km, 22 layers, and 24 times a day.
ERA5 is a reanalysis that combines observations and models through data assimilation; the CDS distribution grid is 0.25°·hourly, and the raw data is approximately 31 km and 137 layers in scale.
Analysis flow chart
1. Observation and Prediction Collection : Observations are based on AWS as the primary and ASOS as the secondary; predictions are centered on LDAPS, with KLAPS used as supplementary data when necessary. ERA5 is used as the analysis reference.
2. Resolution Alignment : Since the project assumes 10-minute precipitation, observations must be specified as 1-hour aggregation (fixed window or rolling window). While the nearest grid point is used as the basis for the model grid, it is safer to also consider bilinear or small-area averages for coastal or mountainous areas.
3. Application of Heavy Rain Determination Criteria : Once a threshold is set, it must be fixed across all comparisons. If the threshold changes, the event frequency itself changes, which alters the interpretation of the CSI.
4. H/M/F/CN Classification : For each time-point (or time-grid), the "Heavy Rain Status" is converted to yes/no and accumulated in a 2×2 table.
5. Calculation of CSI and Complementary Metrics : Calculate POD, FAR, and Bias simultaneously with CSI to isolate the direction of the error.
6. Background vs. Analysis Comparison : Compare the H/M/F patterns produced by LDAPS/KLAPS and ERA5, respectively, against Observation. However, rather than attributing the difference solely to the effect of data assimilation, interpret it as a comprehensive difference including differences in resolution, model, and system.
Proposed criteria for determining heavy rain
Option A fits well with the classification that "more than 30 mm per hour is considered 'very strong rain' in the forecast terminology of the Korea Meteorological Administration, and option C is directly linked to the current heavy rain warning standard. Option B is not an official special report standard, but a high-intensity threshold for research. Regardless of which option is used, the comparability of CSI will be maintained only when one is fixed as the main threshold and the rest are presented only as sensitivity analysis.