How AI will revolutionise the accuracy of the weather forecast
On a screen in the headquarters of the European Centre for Medium-Range Weather Forecasts (ECMWF), in Reading, a batch of unassuming pixels trails along a map of the United States. At first yellow, then green or blue, they form a tadpole shape as they pass above the southern states of Texas and Louisiana.
These pixels represent Hurricane Beryl, which tore through the south of the United States last week – and a major breakthrough in weather forecasting, one that won the team behind it the “Nobel prize of engineering”, the MacRobert Award, on the same day that Beryl hit.
Hurricane Beryl is one of dozens of extreme weather events that have been accurately predicted by GraphCast, a weather forecasting system developed by Britain’s own premier artificial intelligence (AI) company Google DeepMind. The programme outperforms the HRES (highest resolution) system, the current gold-standard in weather forecasting, on more than 90 per cent of accuracy measures.
The team behind GraphCast, led by the scientist Rémi Lam, were described as “heroes” by head judge Prof Sir Richard Friend when they were presented with the award last week. Also in contention for the prize were the University of Oxford and AstraZeneca, for designing the manufacturing process that saw the rapid rollout of vaccines during the pandemic, saving an estimated six million lives.
It is clear why DeepMind’s accomplishment is deemed more heroic than even this: as extreme weather events become more commonplace globally, forecasters are in a race against time to develop software that can accurately predict catastrophes before they happen and help governments to evacuate people from danger zones. Other well-known tech companies such as Nvidia and Huawei have also designed their own models, but GraphCast is regarded as the most advanced.
Forecasting the weather is all about the “butterfly effect, how very small things in the initial condition you are observing can grow and make weather unpredictable at a longer lead time,” Remi Lam explains.
Traditional methods have involved “physical models” – complex mathematical equations run first by people and later, as today, by supercomputers the size of a bus. GraphCast on the other hand can be run on a laptop, and can come up with a weather forecast in mere minutes.
The model works by making a forecast based on data it has been fed on past weather events over the last decade, and applying it to millions of grid points across the Earth, with no maths involved. It can be trialled by the ECMWF in Reading alongside other experts around the world, who have approached it with a “healthy dose of scepticism”, Lam says.
“We live in an increasingly warmer world and more extreme events are happening, so we really need more accurate measures,” says Lam. It’s for this reason that DeepMind has open-sourced its model, allowing government organisations and other companies around the world to use their work and build on it for their own purposes.
The first time GraphCast was used in real life at the ECMWF, to predict the movements of Hurricane Lee to the east of the United States in September last year, the team behind it was “blown away”, Lam says. Machine learning, the technique used to produce GraphCast, is a “very analytical” and bland way of working most of the time, Lam says. “You’re always looking at decreasing error, seeing that margin go down as you train a model.”
But seeing it visualised as a cyclone on the move was “the most striking example of what GraphCast can do,” says Lam.
“We’re not weather experts, and most of us don’t have a background in forecasting,” he adds. “So the first time we saw a real cyclone, moving in real time, everyone was like: wow.”
Yet even in June 2022, less than a year after DeepMind’s first efforts to predict the weather on a medium-term basis began, the team knew they were onto something “really promising”. And by 2023, in the case of Hurricane Lee, the model was able to predict “where it would make landfall, in Nova Scotia”.
“And we were sure it would be locked in [to that path] six days before it happened, three days earlier than the traditional approach could predict,” Lam says.
In an age where we all have access to weather predictions round-the-clock on our phones as well as on TV and in newspapers, it’s easy to forget how tricky a business all this is. Humans have been trying to predict rain, sunshine and the movement of the clouds for more than 2,000 years, since 300 BC, when philosophers in ancient Greece, China and India alike devised their own ways of tracking cloud patterns and predicting how each tiny change in the weather could evolve.
It is no wonder then that experts such as Carlos Osuna, of MeteoSwiss, Switzerland’s federal office for meteorology and climatology, have been stunned by the rapid new developments in this field. In recent years, there has been an “explosion” in the use of machine learning to predict the weather, Osuna says, after years of stagnation in the capabilities of traditional forecasting.
State agencies such as MeteoSwiss have “reached the limit of computational cost,” Osuna says, being unable to build supercomputers large enough to run all the calculations needed to run forecasts more accurately. This current system is “expensive and consumes a lot of resources,” he explains. Machine learning models, on the other hand, are cheap to run and can be performed on computers the like of which many of us have in our homes, all enabled by huge progress in AI capabilities.
Currently, MeteoSwiss uses different types of machine learning to predict pollen concentration across Switzerland, as well as to “nowcast” or predict rainfall within a very short space of time. This has long been a hard nut to crack in weather forecasting, and DeepMind has pushed progress here too. In 2021, the company’s tech could forecast the likelihood of rain within a two-hour timeframe with a high degree of accuracy, outperforming similar tools 89 per cent of the time, according to 50 meteorologists at Britain’s own Met Office.
Though the Met Office doesn’t use GraphCast to make predictions today, the organisation has used machine learning in its various forms for “decades” already, says Prof Kirstine Dale, its chief AI officer. She is working with the Alan Turing Institute to develop FastNet, the Met Office’s own machine learning model that will “complement” other available models, such as GraphCast.
Like GraphCast, FastNet will broadly be used to generate “medium-range” predictions up to a fortnight in advance, many of which will inform government briefings in the future. But the Met Office is also working on ways to deploy AI to help forecast much longer-term weather issues, on a “subseasonal” or “subtropical” range, the sorts of things that would help the country prepare for flooding, drought or other disasters well before most can see the warning signs.
Dale says that such mapping, of long and short-term weather issues, will always go hand in hand with physical models when it comes to official advice. “Any machine learning model is only as good as the data it’s trained on,” she explains. “If there were to be an event like a volcano eruption, which would release lots of aerosols into the air, you would want to rely on a physics-based model because that would be an out-of-sample event that the machine learning model wouldn’t know how to predict,” she says.
Carlos Osuna, in Switzerland, agrees. “Machine learning models don’t understand physics or maths and so might make stupid mistakes,” he explains. It’s the classic “black box” problem with generative AI, which also leaves chatbots prone to “hallucinating” or making up facts, a potential pitfall in weather forecasting too.
“But it’s fantastic at capturing all the patterns that have gone before, so it produces a really good output for things like cyclones or complex thunderstorms, which traditionally have been very difficult to predict with maths alone.”
‘Most chaotic system in the world’
The limitations of machine learning forecasts today are more a problem with human understanding of the natural world than with machine learning models themselves, says Alex Levy, the chief executive of Atmo, a California-based AI weather forecasting firm.
Such models are trained on all the data that has ever been generated through older mathematical systems, a problem as “weather is by far the most chaotic system in the world,” Levy says. “Somewhere in the process of forecasting there will always be laws of physics that we’ll bump up against, beyond which it’s impossible to know how things will develop.”
This year, Atmo took up a contract with the US Air Force to predict wind patterns at Cape Canaveral, one of America’s primary space mission launch sites. The company has also recently begun to work with the national government of the Philippines to predict weather conditions in the capital of Manila down to “city block level”.
“The Philippines receives among the most typhoons per year of any country in the world, as many as 10 or 20, and, at the same time, Manila has a vast population bigger than quite a few countries,” says Levy.
Hindered by the cap on physical computing power, “the old way of forecasting was not very precise, and it was difficult to say which parts of Manila might be affected by extreme weather”. Such precision is crucial when it comes to “emergency response, which can be poorly handled without it”, says Levy.
Similarly, the old physics-based methods had to be general, following universal laws of physics, whereas new AI models can be highly specific in their predictions. In Manila, typhoons are the extreme weather phenomenon that poses the biggest risk to life, but in other settings, the dangers can be radically different.
While it might never be possible to know with absolute certainty what the weather might throw at us tomorrow or next weekend, the days where countries can be hit with an unexpected catastrophe – such as happened in October 1987, when weatherman Michael Fish infamously told the nation not to prepare for a hurricane that did in fact hit – may be increasingly a thing of the past.
After all, machine learning systems are constantly self-improving, says Levy. So in all cases, except those of “the most strange and unusual events”, it is now much easier to prepare for the future in ways that forecasters had never thought would be possible.