Enlarge (credit: Bill Oxford | iStock / Getty Images Plus)

The US AI Safety Institute—part of the National Institute of Standards and Technology (NIST)—has finally announced its leadership team after much speculation.

Appointed as head of AI safety is Paul Christiano, a former OpenAI researcher who pioneered a foundational AI safety technique called reinforcement learning from human feedback (RLHF), but is also known for predicting that "there's a 50 percent chance AI development could end in 'doom.'" While Christiano's research background is impressive, some fear that by appointing a so-called "AI doomer," NIST may be risking encouraging non-scientific thinking that many critics view as sheer speculation.

There have been rumors that NIST staffers oppose the hiring. A controversial VentureBeat report last month cited two anonymous sources claiming that, seemingly because of Christiano's so-called "AI doomer" views, NIST staffers were "revolting." Some staff members and scientists allegedly threatened to resign, VentureBeat reported, fearing "that Christiano’s association" with effective altruism and "longtermism could compromise the institute’s objectivity and integrity."

NIST's mission is rooted in advancing science by working to "promote US innovation and industrial competitiveness by advancing measurement science, standards, and technology in ways that enhance economic security and improve our quality of life." Effective altruists believe in "using evidence and reason to figure out how to benefit others as much as possible” and longtermists that "we should be doing much more to protect future generations," both of which are more subjective and opinion-based.

On the Bankless podcast, Christiano shared his opinions last year that "there's something like a 10–20 percent chance of AI takeover" that results in humans dying, and "overall, maybe you're getting more up to a 50-50 chance of doom shortly after you have AI systems that are human level."

"The most likely way we die involves—not AI comes out of the blue and kills everyone—but involves we have deployed a lot of AI everywhere... [And] if for some reason, God forbid, all these AI systems were trying to kill us, they would definitely kill us,” Christiano said.

Critics of so-called "AI doomers" have warned that focusing on any potentially overblown talk of hypothetical killer AI systems or existential AI risks may stop humanity from focusing on current perceived harms from AI, including environmental, privacy, ethics, and bias issues. Emily Bender, a University of Washington professor of computation linguistics who has warned about AI doomers thwarting important ethical work in the field, told Ars that because "weird AI doomer discourse" was included in Joe Biden's AI executive order, "NIST has been directed to worry about these fantasy scenarios" and "that's the underlying problem" leading to Christiano's appointment.

"I think that NIST probably had the opportunity to take it a different direction," Bender told Ars. "And it's unfortunate that they didn't."

As head of AI safety, Christiano will seemingly have to monitor for current and potential risks. He will "design and conduct tests of frontier AI models, focusing on model evaluations for capabilities of national security concern," steer processes for evaluations, and implement "risk mitigations to enhance frontier model safety and security," the Department of Commerce's press release said.

Christiano has experience mitigating AI risks. He left OpenAI to found the Alignment Research Center (ARC), which the Commerce Department described as "a nonprofit research organization that seeks to align future machine learning systems with human interests by furthering theoretical research." Part of ARC's mission is to test if AI systems are evolving to manipulate or deceive humans, ARC's website said. ARC also conducts research to help AI systems scale "gracefully."

Because of Christiano's research background, some people think he is a good choice to helm the safety institute, such as Divyansh Kaushik, an associate director for emerging technologies and national security at the Federation of American Scientists. On X (formerly Twitter), Kaushik wrote that the safety institute is designed to mitigate chemical, biological, radiological, and nuclear risks from AI, and Christiano is “extremely qualified” for testing those AI models. Kaushik cautioned, however, that "if there’s truth to NIST scientists threatening to quit" over Christiano's appointment, "obviously that would be serious if true."

The Commerce Department does not comment on its staffing, so it's unclear if anyone actually resigned or plans to resign over Christiano's appointment. Since the announcement was made, Ars was not able to find any public announcements from NIST staffers suggesting that they might be considering stepping down.

In addition to Christiano, the safety institute's leadership team will include Mara Quintero Campbell, a Commerce Department official who led projects on COVID response and CHIPS Act implementation, as acting chief operating officer and chief of staff. Adam Russell, an expert focused on human-AI teaming, forecasting, and collective intelligence, will serve as chief vision officer. Rob Reich, a human-centered AI expert on leave from Stanford University, will be a senior advisor. And Mark Latonero, a former White House global AI policy expert who helped draft Biden's AI executive order, will be head of international engagement.

"To safeguard our global leadership on responsible AI and ensure we’re equipped to fulfill our mission to mitigate the risks of AI and harness its benefits, we need the top talent our nation has to offer," Gina Raimondo, US Secretary of Commerce, said in the press release. "That is precisely why we’ve selected these individuals, who are the best in their fields, to join the US AI Safety Institute executive leadership team."

VentureBeat's report claimed that Raimondo directly appointed Christiano.

Bender told Ars that there's no advantage to NIST including "doomsday scenarios" in its research on "how government and non-government agencies are using automation."

"The fundamental problem with the AI safety narrative is that it takes people out of the picture," Bender told Ars. "But the things we need to be worrying about are what people do with technology, not what technology autonomously does."

Christiano explained his views on AI doom

Ars could not immediately reach Christiano for comment, but he has explained his views on AI doom and responsible AI scaling.

In a blog posted on LessWrong, he explained that there were two distinctions that "often lead to confusion about" what he believes regarding AI doom.

The first distinction "is between dying ('extinction risk') and having a bad future ('existential risk')," clarifying that he thinks "there’s a good chance of bad futures without extinction, e.g., that AI systems take over but don’t kill everyone." One version of a "bad future" would be "an outcome where the world is governed by AI systems, and we weren’t able to build AI systems who share our values or care a lot about helping us," which Christiano said, "may not even be an objectively terrible future."

"But it does mean that humanity gave up control over its destiny, and I think in expectation it’s pretty bad," Christiano wrote.

The other distinction is "between dying now and dying later," Christiano said, clarifying that dying later may not exactly result "from AI," but from circumstances following AI advancement.

"I think that there’s a good chance that we don’t die from AI, but that AI and other technologies greatly accelerate the rate of change in the world and so something else kills us shortly later," Christiano wrote.

In that post, Christiano breaks down what he estimates are the probabilities of an AI takeover (22 percent), that "most" humans will die "within 10 years of building powerful AI" that makes labor obsolete (20 percent), and that "humanity has somehow irreversibly messed up our future within 10 years of building powerful AI" (46 percent).

He clarified that these probabilities are only intended "to quantify and communicate what I believe, not to claim I have some kind of calibrated model that spits out these numbers." He said these numbers are basically guesses that often change depending on new information that he receives.

"Only one of these guesses is even really related to my day job (the 15 percent probability that AI systems built by humans will take over)," Christiano wrote. "For the other questions I’m just a person who’s thought about it a bit in passing. I wouldn’t recommend deferring to the 15 percent, but definitely wouldn’t recommend deferring to anything else."

Timnit Gebru, who founded the Distributed Artificial Intelligence Research Institute after Google fired her from their AI ethical research team after she spoke out against discrimination, criticized Christiano's blog on X.

"What's better, that he wrote a blog on a cult forum, or that he just pulled random numbers out of his behind for this apocalyptic prediction?" Gebru wrote. "As they say, why not both."

In 2023, Christiano's nonprofit ARC helped test whether OpenAI's GPT-4 might take over the world and ultimately concluded that GPT-4 did not pose an existential risk because it was "ineffective" at "autonomous replication." Because ARC is concerned about AI systems manipulating humans, Christiano has commented on LessWrong that gain-of-function research becomes more important as AI systems become smarter. This suggests that his work at the safety institute evaluating systems will be a critical job.

"At this point it seems like we face a much larger risk from underestimating model capabilities and walking into danger than we do from causing an accident during evaluations," Christiano wrote. "If we manage risk carefully, I suspect we can make that ratio very extreme, though of course that requires us actually doing the work."

Christiano's take on pausing AI development

Christiano isn't the only one warning about AI's existential risks. In the past year, everyone from OpenAI executives to leaders of 28 countries has sounded alarms over potentially "catastrophic" AI harms. But critics like Meta Chief AI Scientist Yann LeCun have countered these warnings by claiming that the "whole debate around existential risk is wildly overblown and highly premature."

At the AI Safety Institute, Christiano will have the opportunity to mitigate actual AI risks at a time when people who build, test, and invest in AI have claimed that the speed of AI development is outpacing risk assessment. And if there's any truth to what Elon Musk says—which is hotly contested—AI will be "smarter than any one human probably around the end of next year."

To minimize surprises, Christiano's team will need to refine risk assessments, as he anticipates that models will get smarter and fine-tuning them will get riskier. Last October, on an effective altruism forum, Christiano wrote that regulations would be needed to keep AI companies in check.

"Sufficiently good responsible scaling policies (RSPs) could dramatically reduce risk" by "creating urgency around key protective measures and increasing the probability of a pause" in AI development "if those measures can’t be implemented quickly enough," Christiano explained.

Even with regulations around scaling, though, Christiano warned that "the risk from rapid AI development is very large, and that even very good RSPs would not completely eliminate that risk."

While some AI critics fearing existential risks have in the past year called for a temporary pause in AI frontier development until protective measures improve, Christiano has argued that only a unified global pause would come without significant costs.

Currently, Christiano has said that a pause isn't necessary because "the current level of risk is low enough that I think it is defensible for companies or countries to continue AI development if they have a sufficiently good plan for detecting and reacting to increasing risk."

Read Comments

Enlarge (credit: Frame Studio)

Almost from the start, arguments about mitigating climate change have included an element of cost-benefit analysis: Would it cost more to move the world off fossil fuels than it would to simply try to adapt to a changing world? A strong consensus has built that the answer to the question is a clear no, capped off by a Nobel in Economics given to one of the people whose work was key to building that consensus.

While most academics may have considered the argument put to rest, it has enjoyed an extended life in the political sphere. Large unknowns remain about both the costs and benefits, which depend in part on the remaining uncertainties in climate science and in part on the assumptions baked into economic models.

In Wednesday's edition of Nature, a small team of researchers analyzed how local economies have responded to the last 40 years of warming and projected those effects forward to 2050. They find that we're already committed to warming that will see the growth of the global economy undercut by 20 percent. That places the cost of even a limited period of climate change at roughly six times the estimated price of putting the world on a path to limit the warming to 2° C.

Linking economics and climate

Many economic studies of climate change involve assumptions about the value of spending today to avoid the costs of a warmer climate in the future, as well as the details of those costs. But the people behind the new work, Maximilian Kotz, Anders Levermann, and Leonie Wenz decided to take an empirical approach. They obtained data about the economic performance of over 1,600 individual regions around the globe, going back 40 years. They then attempted to look for connections between that performance and climate events.

Previous research already identified a number of climate measures—average temperatures, daily temperature variability, total annual precipitation, the annual number of wet days, and extreme daily rainfall—that have all been linked to economic impacts. Some of these effects, like extreme rainfall, are likely to have immediate effects. Others on this list, like temperature variability, are likely to have a gradual impact that is only felt over time.

The researchers tested each factor for lagging effects, meaning an economic impact sometime after their onset. These suggested that temperature factors could have a lagging impact up to eight years after they changed, while precipitation changes were typically felt within four years of climate-driven changes. While this relationship might be in error for some of the economic changes in some regions, the inclusion of so many regions and a long time period should help limit the impact of those spurious correlations.

With the climate/economic relationship worked out, the researchers obtained climate projections from the Coupled Model Intercomparison Project (CMIP) project. With that in hand, they could look at future climates and estimate their economic costs.

Obviously, there are limits to how far into the future this process will work. The uncertainties of the climate models grow with time; the future economy starts looking a lot less like the present, and things like temperature extremes start to reach levels where past economic behavior no longer applies.

To deal with that, Kotz, Levermann, and Wenz performed a random sampling to determine the uncertainty in the system they developed. They look for the point where the uncertainties from the two most extreme emissions scenarios overlap. That occurs in 2049; after that, we can't expect the past economic impacts of climate to apply.

Kotz, Levermann, and Wenz suggest that this is an indication of warming we're already committed to, in part because the effect of past emissions hasn't been felt in its entirety and partly because the global economy is a boat that turns slowly, so it will take time to implement significant changes in emissions. "Such a focus on the near term limits the large uncertainties about diverging future emission trajectories, the resulting long-term climate response and the validity of applying historically observed climate–economic relations over long timescales during which socio-technical conditions may change considerably," they argue.

Uneven costs

So, what happens by 2050? The researchers' model suggests that "committed damages comprise a permanent income reduction of 19 percent on average globally," compared to where growth would have gotten us. Uncertainties mean the likely range is between 11 and 29 percent. Using a middle-of-the-road scenario for economic growth, this translates to an economic hit of $38 trillion (a figure measured in international dollars.)

The authors contrast that with an estimate the IPCC made about the costs of limiting warming to 2° C: $6 trillion dollars. So, even the short-term impacts of climate change will vastly outweigh the costs of action.

This hit isn't evenly distributed. Wealthy areas in the US and Europe will only see incomes drop by about 11 percent, while Africa and South Asia take a hit of 22 percent. This is likely because wealthy countries already have a larger capacity to adjust to climate-related problems than those in the Global South. But it's also striking, as the pace of change is much larger outside the tropics, so these countries are also going to be facing more extreme changes. The researchers do see areas that experience economic benefits, but those are limited to the high latitudes nearer the poles.

Kotz, Levermann, and Wenz note that the areas with the highest costs tend to have the lowest cumulative emissions. In other words, the problems are felt most keenly in the countries that made the smallest contributions to them.

There are also some effects that are beneficial. Areas that experience increased average rainfall see incomes rise due to that effect (though drier areas see the opposite). But these same areas see added costs from increases in the average number of rainy days that largely offset this effect. And the impact of more extreme precipitation is a negative everywhere.

It could be worse

There are a couple of ways that this could end up being an underestimate of future costs. Over the long term, a continued warming climate will start to produce more events with no historical precedent, meaning there's no way to project their economic impact. By limiting the analysis to about 25 years, the researchers make it less likely to be a major factor. But unprecedented events are already occurring, so we're already at the point where some problems are being undercounted.

There are also a large number of climate events that aren't considered at all, including heat waves, severe tropical storms, and sea level rise. Individually, it's unlikely that any of these events will show dramatic changes in the next 25 years, but the cumulative impact of gradual changes isn't going to be included. Plus, there's always the chance of reaching a tipping point where there's a sudden change in frequency for one or more of these events.

Finally, the researchers don't really consider non-local impacts, such as where extreme weather in one location can ripple through supply chains to produce impacts elsewhere. Think about cases where large urban centers import much of their food from relatively distant locales.

Kotz, Levermann, and Wenz acknowledge all of these issues but suggest that their more conservative, empirical approach provides a bit of clarity that's difficult to achieve otherwise.

One aspect they don't consider, however, pertains to their comparison between the costs of our committed damages and the cost of decarbonizing the economy. The past 20 years have seen the price of mitigating climate change through renewable energy and efficiency plunge dramatically, and the price of other key technologies, such as batteries, is following a similar trajectory. By 2050, these could make the difference between the cost of acting and the cost of doing nothing even more dramatic.

Nature, 2024. DOI: 10.1038/s41586-024-07219-0  (About DOIs).

Read Comments

Click here to go see the bonus panel!

Hovertext:
Absolute Midden is copyright SMBC Enterprises 2024 all rights reserved.

This article originally appeared on Kotaku.

A recent update for Bethesda’s popular post-apocalyptic spin-off, Fallout Shelter, added some characters from the Amazon Prime live-action TV show adaptation to the base-building game. While it’s fun that you can now have some of these characters, like Lucy and Maximus, in your own digital vault, it also reveals their SPECIAL stats, too.

First released on phones in 2015, Fallout Shelter tasks you with building a brand new underground vault after the nuclear bombs have fallen. You fill your shelter with different rooms and characters, while also upgrading and expanding your vault. You can even send characters out into the wasteland to explore buildings and complete quests. I recently re-downloaded the game after watching the Amazon show, and discovered some characters from Fallout are now available in Fallout Shelter, letting us see how Walter Goggins, aka The Ghoul, would stack up in the video games.

On April 11, Bethesda announced that some characters and locations from the show will be available in Shelter. However, I’m not going to spoil any of the quests or locations added to the game (or the show, either). Instead, let’s dig into each TV show character’s SPECIAL stats. First up, vault dweller Lucy.

Lucy MacLean:

Strength-4

Perception-7

Endurance-6

Charisma-5

Intelligence-6

Agility-5

Luck- 7

The Ghoul:

Strength-5

Perception-6

Endurance-7

Charisma-7

Intelligence-4

Agility-7

Luck-4

Maximus:

Strength-7

Perception-6

Endurance-6

Charisma-5

Intelligence-4

Agility-7

Luck-5

Ma June:

Strength-5

Perception-7

Endurance-5

Charisma-7

Intelligence-6

Agility-4

Luck-6

Overall, I agree with these SPECIAL stats. However, I do think The Ghoul’s intelligence and Lucy’s agility should probably be a bit higher based on their actions in the show. But whatever, I’m just happy to see that Maximus’ status as Wasteland Himbo is canonically confirmed by Bethesda.

Of course, we don’t know if these are the stats they started the show with or if this is after season 1 and some leveling up. Either way, we can now compare these characters to our own Fallout main characters from the past games.

All of Fallout’s first seasonis available now on Amazon Prime.

.