If your favourite AI image generator starts spitting out some weird stuff, we may have an explanation.
Driving the news: Researchers successfully created a method to trick image-generating AI models like DALL-E into wrongly characterizing images during their training, according to their recently published paper.
-
The technique uses a “poison pill” they’ve named Nightshade, which adds invisible pixels to images that sabotage the AI models training on them.
- The lead developer of Nightshade describes the tool as a last defence against web scrapers that use content creators' work without permission.
Why it matters: Tools like Nightshade expose the fact that the quality of AI models is largely dependent on the data they’re trained on, a flaw that makes them vulnerable to manipulation.
- Nightshade’s lead developer says it takes less than a few hundred poisoned images to “severely damage” models like DALL-E, Midjourney and Stable Diffusion.
Zoom out: Widespread adoption of poison pill techniques by content creators could push more AI companies to cut licensing deals with them to ensure their training data is high quality and authentic.
- In September, AI tech developer Nividia reached a deal with Getty Images to develop a generative AI model using only Getty’s catalogue of content, while OpenAI cut a licensing deal with the Associated Press to use its content to train its ChatGPT model.
Yes, but: Poison pills can also be used for nefarious purposes, like manipulating AI chatbots to send racist and offensive messages, a scenario that played out with one of Meta’s AI chatbots last year.
Big picture: With a slew of AI copyright lawsuits playing out, Nightshade isn’t the most pressing challenge to AI developers, but the poison pill represents a growing threat to one of the industry’s most valuable commodities: quality data.—LA