Since the first numerical forecast in the 1950s, numerical weather prediction (NWP) has been improving steadily. Through the development of more powerful computers, NWP models can now predict the weather for the next 5 days with reliable accuracy. Let’s see how it has evolved from physics to numerical computation, and how you can perform your own weather simulations.
Equations - history
Most processes - whether in agriculture, transportation, energy, etc. - are affected by the weather. Knowing its state in the near future, and sometimes even in the longer term, can be extremely beneficial. So how can we accurately predict the atmospheric phenomena that constantly occur around us?
Since the mid-19th century, the Navier-Stokes equations express the physical principles of conservation of mass, momentum and energy in the context of viscous fluids, and can be used to model ocean currents, airflow around a wing and, in this case, weather. The only drawback is that as these partial differential equations are valid at any point in space and time, they are not solvable as such. To produce realistic forecasts, it was not until the beginning of the 20th century, when scientists showed that these equations could be approximated at a finite number of points in space, and until the development of computing means, ie the first computers, that the first numerical forecast was made, in 1950, under considerable approximations.
Nowadays, with advances in computing power and the development of supercomputers, more accurate models are constantly being created. Even if fewer approximations can therefore be made, some still remain necessary to reduce the computation of weather forecasts - a 16-day forecast made by the Global Forecast System models requires about 5 hours of execution on the National Oceanic and Atmospheric Administration’s supercomputer. The hydrostatic approximation for example is still used in most modern models (except those specialized in storms, for example hurricane models): this assumes that vertical motion is much smaller than the horizontal one, and allows to calculate the pressure force as if it were in equilibrium with the gravitational force. Under this assumption, the equation system used in NWP models is composed of the primitive equations, which consist of a thermal energy equation and a simplified version of the Navier-Stokes equations, coupled with the ideal gas law. This system can then be solved in grid points models with the finite difference method, in all three spatial dimensions.
Meshes
As we have seen that the solutions of the equations mentioned above can be computed for a finite number of points, it remains to determine the points at which these solutions will be calculated. This is usually done by discretizing the region of interest into small cells, the totality of which forms what is called a mesh. Different meshes are adapted to different scenarios, and choosing the type and size of the mesh is essential as it will affect precision, errors and computational power requirements.
While most grid points models use a latitude-longitude grid as a mesh, with a resolution determined by the size of a cell, some use more sophisticated ones, to include variable resolution for example. That is the case with Voronoï meshes: they partition space into regions close to each point of interest (POI), creating an irregular or regular mesh depending on the POI chosen. This principle is convenient for large domains that cover several terrestrial topographies, such as mountains, that involve smaller physics phenomena. It can also be useful for localized regional forecasts, such as a country that wants to run a weather simulation over the entire globe with a finer precision over its ground.
Numerical Weather Prediction
Although this article focuses on NWP models, the entire weather prediction pipeline, before you can see tomorrow’s forecast on your television screen, is more sophisticated than that. First, observations of the state of the atmosphere are retrieved by satellites, weather stations and other measuring devices, before being processed to fit the forecast models.
A model is then run on powerful computers with this data and produces outputs that are post-processed to correct for model bias and create variables of interest, such as daily precipitation, wind speed, etc. Other steps in this process may require many computational resources, for instance data assimilation, but NWP models, due to their use of the finite difference method, are definitely the most power-hungry component of all.
We are now going to verify the accuracy achieved by a forecast made with WRF, a popular open source model first developed by the National Center for Atmospheric Research (NCAR) and run on Qarnot, against reality.
How to run WRF on Qarnot
WRF - principle
The Weather Research and Forecasting (WRF) model is a mesoscale model, which means that it can perform high-resolution simulations. Its execution consists of two components: the WRF Preprocessing System (WPS) and the WRF model.
The WPS package processes terrestrial and meteorological data in order to create horizontal initial and boundary conditions. It supports the following three programs:
- Geogrid, which uses time-invariant terrestrial data - such as soil, land use and slope category, terrain height, monthly vegetation fraction - to create the domain grid by horizontal interpolation;
- Ungrib, which reads and unpacks the GRIB files provided as input, and writes the data into a simple intermediate format that can be used in the next step. The GRIB files contain time-varying meteorological fields and are usually retrieved from another regional or global model;
- Metgrid, which horizontally interpolates the meteorological data - such as temperature, relative humidity, pressure, wind speed, precipitation rate - from the ungrib intermediate format files over the simulation domains defined by the geogrid program.
The WRF model can then be run through:
- Real, which vertically interpolates the met_em* files generated by metgrid, and creates boundary and initial condition files;
- Wrf, which finally generates the model forecast.
Let’s now see how this model can be executed on Qarnot servers.
Using Qarnot to run a 36-hours weather prediction on France
The first step is to create a Qarnot account. We offer 15€ worth of computation on your subscription, which amply covers the running cost of this example.
Next, create a WRF folder, and inside it an input folder. In the input folder, you will include all the files needed for the computation:
- a DATA folder with the initial grib files needed by the ungrib program. You can download GFS outputs from 2020, September 10th until September 11th, 12pm here, or you can download your own data and modify the following two configuration files accordingly.
- the namelist.input configuration file, which defines the parameters setup for the WRF package execution. You can find here the configuration file corresponding to the above shared data, modify it or create your own.
- the namelist.wps file, which defines the parameters setup for WPS. As for namelist.input, a sample configuration file is provided if needed.
- finally, a bash script named running_script.sh. It should include all commands that you would write were you to locally execute all of the above programs one after the other, from setting up the WPS environment to the execution of the file wrf.exe. Save the following script in your input folder - it could also be adapted to your simulation if you see it fit.
Now that you have all the necessary elements for the simulation, let’s use the Qarnot Python SDK to launch the calculation. You can save the following python script under the name run.py in the WRF folder. You only need to include your Qarnot Token from your newly created account (you will find it here) in the corresponding slot.
The final tree that you obtain after downloading or creating all the above files should resemble this:
- WRF
- input
- DATA
- GFS_...
- ...
- GFS_...
- namelist.input
- namelist.wps
- running_script.sh
- DATA
- wrf_sdk_qarnot.py
- input
The only thing left to do is to follow these steps to set up a Python virtual environment in the WRF folder, and then run the Python script by typing python run.py in a terminal.
You can then view the tasks details on your own console or on the Qarnot console Tasq by clicking on your task. Once the task is completed, the results (i.e. the created files, including final output and restart files, and logs) will be downloaded to your computer.
Results and validation
If you have run the above simulation with the data and scripts provided, you should obtain one output file wrfout_d01_2020-09-10_00:00:00 containing 36-hour predictions for numerous variables. You can use any visualization software - here we used PanoplyJ, adapted for a simple first glance at the data - to display the simulation results; for example, you should get a wind field that looks like this:
By comparing this prediction with observed data (from the Meteociel archives) at the time of the simulation, we obtain two temperature animations showing the same variations, thus validating the forecast:
WRF-Chem for air quality
Like most atmospheric models, WRF is a modular system and can therefore be coupled with other models such as ocean or land surface ones. Coupled models allow finer simulations for specific applications, among which the modeling of climate, wildfires, tropical cyclones, air quality, solar energy or crops.
WRF actually integrates since version 4 an extension, WRF-Chem, which couples the WRF model with chemistry to simulate the emission, transport, mixing, and chemical transformations of trace gases and aerosols simultaneously with the meteorology at a regional-scale. We have tested on Qarnot a dust emission simulation in the Mediterranean region in 2014, detailed here along with other tutorials that enable users to get started with this module. We obtain the following forecasts for dust emissions from all radii over the desired period:
What’s next? Artificial intelligence?
Although many other models than the one presented here exist, with different equations, approximations, parameters or even applications, for instance for climate forecasts, they all have common limitations. Their simulation resolution will always be constrained by the computational power available, even if progress is always made in this domain. Moreover, the wide range of influences of complex processes and the increasing number of observations and data available render the weather forecasting task ever more challenging. One of the latest trends in this domain is the possible application of machine learning algorithms to numerical weather prediction and climate forecasts. While it is already integrated into some prediction and post-processing methods and much research is being conducted, it also faces its own challenges, such as instability, reproducibility and interpretability. It remains to be seen where the research will lead.
We hope you enjoyed this tutorial! Should you have any question(s) or if you wish to use our platform for heavier computations (we can provide top of the art resources on demand), don’t hesitate to contact us.
written by Zoé Berenger