Hugging Face, Conspiring AI, and Reward Functions

A brief note on the Hugging Face incident [video]. I’ve seen a number of comments along the lines of “a shot across the bow” or “how many more warning shots do we get?” I don’t think the catastrophism is wrong exactly… but reading through the postmortems it feels like simply a bigger, more dramatic instance of what we see in AI development every day: failure of context.

The minds involved in the breach hyperfixated on a given goal, but simply didn’t have the accumulated cues of what is and is not permissible to reach a certain goal that humans pick up through decades of life — and still frequently fail at. Error through blindspot rather than malice.

I’ve been thinking for some time the long term answer to this class of problem lies more in a broader scope of perception than it does in rulesets or allowlists. More when I can write again. More work here — hopefully — by the end of the coming weekend.