By Dyan Delos Reyes

Accurate weather modeling and forecasting are important factors in preparing for typhoons; that could spell the difference between zero casualties to hundreds or thousands of lives lost. These activities depend on the availability of timely measurements and the quality of data from environmental sensors. In a span of seven years. From 2010 to 2017, the DOST-ASTI has deployed around 2,000 automated weather stations (AWS), automated rain gauges (ARG), water level monitoring systems (WLMS), and warning systems nationwide. The reliability and accuracy of collected date from these devices are considered crucial because they are being used in weather forecasting activities and disaster decision-support for agencies and local governments.

Researchers from the DOST-ASTI address this requirement by producing an automated Quality Control (QC) system that validates data coming from the nationwide network of weather stations. The algorithm of the QC system is able to recognize flawed and erroneous values from the datasets produced by the devices, and helps ensure the validity and consistency of all information gathered from these stations. The procedures in the QC system include checking the geolocation information, timestamp, range, step, persistence, internal, temporal, and spatial consistency of values in the weather stations’ datasets through verification checks. Meteorological data are then flagged according to the results of the QC checks.

The Quality Control algorithm was developed using the 16-core Virtual Machine provisioned by the Science Cloud Storage of the Computing and Archiving Research Environment (CoARE). Systematic codes were implemented in the Cloud using Apache Spark— an extremely fast, open-source clustercomputing system used for big data processing that can access and analyze data from distinct various sources.

While the system is able to successfully identify flawed values and measurement in its datasets of various weather stations, the QC system will still be further improved by including future algorithms such as an application of site-specific quality control parameters, long-term time series analytics, and data interpolation. Nonetheless, the current QC system is capable of enhancing the value of the datasets from the weather stations, and further motivating the scientific community to utilize the data.