Poison pill threatens AI image models

If your favourite AI image generator starts spitting out some weird stuff, we may have an explanation.

Driving the news: Researchers successfully created a method to trick image-generating AI models like DALL-E into wrongly characterizing images during their training, according to their recently published paper.

  • The technique uses a “poison pill” they’ve named Nightshade, which adds invisible pixels to images that sabotage the AI models training on them. 

  • The lead developer of Nightshade describes the tool as a last defence against web scrapers that use content creators' work without permission. 

Why it matters: Tools like Nightshade expose the fact that the quality of AI models is largely dependent on the data they’re trained on, a flaw that makes them vulnerable to manipulation. 

  • Nightshade’s lead developer says it takes less than a few hundred poisoned images to “severely damage” models like DALL-E, Midjourney and Stable Diffusion. 

Zoom out: Widespread adoption of poison pill techniques by content creators could push more AI companies to cut licensing deals with them to ensure their training data is high quality and authentic. 

Yes, but: Poison pills can also be used for nefarious purposes, like manipulating AI chatbots to send racist and offensive messages, a scenario that played out with one of Meta’s AI chatbots last year.

Big picture: With a slew of AI copyright lawsuits playing out, Nightshade isn’t the most pressing challenge to AI developers, but the poison pill represents a growing threat to one of the industry’s most valuable commodities: quality data.—LA