Reposted from the Cliff Mass Weather Blog
The US global weather forecast, operated by NOAA / National Weather Service, ranks fourth among national centers and well behind what one would expect from the world's leading US weather research community.
The answer is clear: the large U.S. weather community is not working together effectively in developing weather forecasting models and moving research to operations.
NOAA has highly skilled and motivated weather modeling researchers in its various laboratories and in the National Weather Service. The U.S. Center for Academic Research in Meteorology and Modeling is located at the National Center for Atmospheric Research (NCAR) in Boulder, Colorado, a facility operated by the Consortium of United States Universities (UCAR: University Corporation for Atmospheric Research). The U.S. Navy also researches and develops weather forecast models in places like the Naval Research Lab in Monterey. However, with all these laboratories and researchers, the US cannot create a world-leading weather forecasting model.
Here's the problem:
NOAA / NWS has its own models and is developing its own global forecasting system called UFS (Unified Forecast System); UCAR / NCAR has its own models and developing its own global forecasting system called SIMA (System for Integrated Modeling of the Atmosphere). The military (mainly the US Navy) has its own models and is developing its own global forecasting system.
There is everyone trying to do the same. Many spend tens of millions of dollars. None is state of the art.
The separate development of the same type of global predictive model has been a disaster for the US:
NOAA / NWS lacks the innovation, manpower, and ideas of the academic community. The work of the academic community lacks sufficient resources and the effective transfer of its research to societal needs is lacking. The Navy's modeling system lags NOAA in terms of forecasting capabilities, which has obvious implications for national security.
As a result, both government and academic efforts are not up to date with global weather modeling and forecasting. Progress will be slowed down. Tens to hundreds of millions of dollars are badly spent. The costs to society are enormous. Are you interested in global warming? State-of-the-art numerical weather forecasting is a first line of defense against severe weather.
Lawn battles, ego and the inefficiency of large bureaucracies
How do I do this discreetly and without offending anyone?
Large government bureaucracies are not known for efficiency and innovation. Sure, I know exceptions to this statement, but from what I've seen, people are comfortable in safe government jobs. Jobs where excellence is not highly rewarded and where failure in a public service position does not risk job loss. Government bureaucracies tend to grow over time, with little reduction in inefficient branches or activities. In NOAA model development, responsibility for model development is divided between several often competing offices. And often they don't want to help each other or work together trying to protect their piece of cake and lawn. Let me emphasize that I have a lot of respect for NOAA researchers – they are working on a broken system.
Scientists in large government-sponsored laboratories (like NCAR) can also settle for playing in their own science sandbox as they see little reason to collaborate with others.
Then there are the human weaknesses of scientists and scientific administrators. Too many times I've heard NCAR weather modelers dissing NOAA people, and sometimes vice versa. At a meeting to discuss collaboration on a major joint effort (the EPIC Modeling Center), a key university laboratory administrator made derogatory remarks to his counterpart in the National Weather Service. Such toxic interactions make collaboration difficult.
Why is the competition not working?
When I blog on the subject, does someone inevitably bring up the issue of competition? Isn't it good if lots of US groups are trying to do the same? My answer:
There is a lot of competition in the US weather modeling community right now, and this does not seem to result in a superior approach. Global weather modeling is one of the most complex tasks our species attempts: to simulate the future using the molecule on a global scale, which requires complex data assimilation, model development and statistical post-processing. Weather models (often coupled with ocean, ice, wave, and land surface models) comprise hundreds of thousands to millions of lines of code. You need the largest computers in the world. A HUGE effort. We didn't have three Apollo programs in the 1960s. Instead, the complex problem has been shared among groups across the country. With the weather forecast, three groups tried to do something even more complex. There are certain key problems that must be solved by everyone, for example in model physics. Only when we divide the problem can we deal with all the problems. When three or more teams try to do the same task, it inevitably results in important work not getting done well or not at all. Money and science / technology talent are limited. There are many international competitions, and other groups are working on global predictions (such as the European Center)
Congress knows there is a problem. They suggested a way forward.
Problems with the global weather forecast in the US are known and have been covered extensively in the media (e.g. here) and on this blog. Congress has had several hearings on the matter (one that I testified in last year) and they have passed laws that provide funding for NOAA to support research and development on weather forecasting.
It is important that Congress, with the full support of both parties, passed new laws thatInnovation center for earth prediction"Focused on" enhancing weather modeling skills, regaining and maintaining international leadership in numerical weather forecasting, and improving the transition of research into operations. "Congress intended that the EPIC center should be a robust, independent entity that brings the whole thing together. The US weather research community is building the best weather forecasting system in the world. FINALLY, break the divisions and duplications that are crippling US efforts. And Congress gave NOAA millions of dollars to make things happen.
But, as I'll describe, some in the NOAA bureaucracy have put roadblocks in the EPIC's path.
NOAA is about to make a big decision regarding EPIC: will NOAA work with the UCAR / NCAR – the academic weather community – to create a viable center that will transform the weather forecast in the US? Or will the current stagnation continue?
Leaders in my ward speaking in Congress about weather forecasting in the United States
The big decision of NOAA
NOAA is about to make a decision that will determine whether the US numerical prediction will be second, third or the best in the world.
As noted above, Congress has authorized and funded an EPIC center designed to encourage collaboration and innovation, and gave NOAA the responsibility to achieve this.
NOAA put out a call for proposals (RFP) for EPIC, but instead of following the will of Congress to create an independent center that brings the entire community together to create the best global modeling system in the world, they changed the effort into a support service contract with NOAA. Scientific development or research is not even included in the document. Merging the community is not included in the document. Innovation is not included in the document. NOAA remained in complete control. The intent was pretty transparent. Now for many of us there was great hope that UCAR / NCAR, which represents the research community, would win the EPIC contract, thereby narrowing the gap between NOAA and the academic community (UCAR / NCAR) and bringing the two groups together. NOAA employees made EPIC as unattractive as possible to UCAR / NCAR, but wonder for wonder, UCAR / NCAR did put forward a proposal for EPIC in the hope that a small start could grow into the national center that Congress had in mind. And let me be honest, there are others in UCAR / NCAR who want EPIC to fail or go elsewhere so they can continue to do business the same way.
In the next few weeks, NOAA will decide who gets the EPIC contract – and that decision could well determine the future of US weather forecasting. UCAR is the obvious choice, with extensive experience in model development, relevant software development and support from outside users. Nobody is even close to their skills and experience.
This figure shows the capabilities of global modeling in the US compared to other main groups
When NOAA selects UCAR / NCAR, the US weather modeling community can take a new and far better path. However, if NOAA selects a bandit company or a consortium of less experienced academic institutions, the future of operational prediction in the US is not promising.
In many ways, this is NOAA's last chance to get global modeling right. Private sector companies like IBM are starting to build global modeling capabilities. The competition (like the European Center) is advancing. NOAA. should be able to create a global modeling system that is the best in the world … not for jingoistic or nationalistic reasons … but because a better weather forecast would be of great benefit to mankind and an effective tool in protecting mankind from a number of Factors represents weather threats.
HT / Cam_S