The Curious Case of AI Cannibalism & Possible Solutions

Nowadays, you must have heard the terms “AI is eating itself” or “the internet has become a dumping ground for AI.”

Have you ever wondered why you encounter such terms? It is because the models are being trained on data fed by their own kind. The term is called AI data cannibalism.

AI Cannibalism is a new-age dilemma threatening the performance and long-term viability of large language models (LLMs). Let’s explore possible solutions.

AI Cannibalization occurs when LLMs scour the Internet to produce responses that are primarily AI-generated content and already produced by other AI systems, resulting in low-quality output.

Recently, it has been observed that there’s an increasing shift in the sentiments of paid users of GPT-4. Several users have expressed disappointment on social media platforms regarding the dip in LLMs’ output quality. The chatbot’s response has been termed as “dumber” and “lazier” in comparison to previous versions leading to concerns regarding the total user experience. This has sparked conversations within the AI community regarding the factors leading to its decline.

Why AI Cannibaslim Cases are Growing Fast?

AI is increasingly feeding on its own outputs as the internet gets filled with AI-created content, causing a trend called AI data cannibalism. Such self-replication has implications for AI models being trained on fake data instead of human-created information, incrementally warping their perceptions of reality. Science fiction writer Ted Chiang, in a piece for The New Yorker, likened this to the JPEG effect, where copying repeatedly deteriorates quality, adding distortions along the way.

Researchers from Britain and Canada have found a similar problem known as “model collapse,” in which generative AI models, when trained on AI-generated data, lose accuracy and diversity in their outputs. Alarmingly, this process is seemingly inevitable even under the best learning conditions, and it indicates the dangers of AI moving further away from true human insights as it recursively trains on its own artifacts.

Result of AI Cannibalism

Data that are generic in nature are no longer viable for pushing the performance of AI models as per developers. According to Aidan Gomez, chief executive of $2bn LLM start-up Cohere, “If you could get all the data that you needed off the web, that would be fantastic. In reality, the web is so noisy and messy that it’s not really representative of the data that you want. The web just doesn’t do everything we need.”

Now let’s shift our focus to possible solutions to address this issue.

1. Research & Development: The AI models will require unique and sophisticated datasets for improving performance and addressing the challenges in Science, Medicine, or Business. The datasets will have to be created by experts like scientists, doctors, authors, actors, or engineers, or acquired from large companies which include pharmaceuticals, banks, and retail set-ups. There is an increased need for producing datasets that are generated and curated by humans without any help from foundation LLMs. This will ensure the delivery of input by AI models that are accurate and reliable.
   
   
2. Model Collapse: LLMs exposed to AI-generated data over time are impacted by a degenerative process called ‘model collapse.’ This process results in them forgetting the data distribution underlying it. It also has an overarching effect on the future of generative AI technology.
   
   
3. Increase in Volume of AI-Generated Content: A mechanism must ensure that there is a clear way to differentiate between real, human-authored information and AI-produced material. Failure to address this issue can result in a decline in the functionality and efficacy of AI tools like ChatGPT, which increasingly depend on human-generated data for training and content generation..

In summary

AI cannibalism raises doubts in the minds of people regarding the sustenance of AI systems and their capability to advance in a fashion that aligns with human intellect. It emphasizes the requirement for extensive research and development for mitigating risks linked with cannibalization to ensure that AI models are consistent in offering accurate, dependable, and intelligent output.

At Cogito, we endeavor to address the above challenge by:

Engaging our subject matter experts who research the Internet to gain an in-depth understanding of the topics. This will ensure that the training data is well-researched and created manually without the help of foundation models.

We ensure that our team works in an on-site supervised environment so that they can access only resources conducive to proper Research and development.