The relationship between OctoML and Civitai has come to an end following reports that the text-to-image platform was being used to create images categorized as child pornography. OctoML initially planned to work with Civitai and implement measures to prevent the creation of harmful images. However, a recent report revealed that OctoML has decided to cut ties with the platform entirely.
Internal communications indicated that some Civitai users were generating sexually explicit and nonconsensual images of real people, as well as pornographic depictions of children. OctoML responded by implementing a filter to block the generation of all NSFW content on Civitai before ultimately ending their partnership. Civitai also added new moderation methods, including an embedding called Civitai Safe Helper (Minor) to prevent the model from generating images of children if mature themes or keywords are detected.
Civitai, backed by Andreessen Horowitz, has faced previous controversy for the creation of nonconsensual deepfakes of celebrities, influencers, and private citizens, primarily women. Despite adding filters to prevent the creation of NSFW content featuring certain celebrities, OctoML, utilizing Amazon Web Services servers, has decided to terminate its business relationship with Civitai, citing a commitment to ensuring the safe and responsible use of AI.
