By Zander Hu(AI Researcher @ MIZU) & Shu(Founder @ MIZU)
In many Sci-Fi movies, AI systems, originally designed to serve humanity, evolve beyond our control, turning into mysterious black boxes that no one truly understands. Films like The Matrix depict a terrifying future where AI enslaves humanity. The true danger lies not just in AI’s power, but in its opacity—a blackbox that defies human understanding.
Modern AI systems, particularly large language models (LLMs) like ChatGPT, have become incredibly advanced. Yet, in a crucial aspect, these systems remain inscrutable black boxes. Even when their model architecture or code is open-source, the vast, hidden training data — its sources, content, and curation processes — are often closely guarded secrets of tech companies.
This opacity of the AI training data is leading to real-world consequences that we are struggling to manage. Without access to the training data, it becomes extremely difficult to identify and correct biases, inaccuracies, or potentially harmful content that the AI may have learned.
How can we trust and responsibly use AI systems when we don't fully understand the data that shapes their knowledge and behavior? As AI continues to play an increasingly significant role in our lives, the opacity of AI training data isn't just a theoretical concern – it's leading to tangible, real-world problems.
Imagine waking up one morning to find that your company's most closely guarded secrets have been exposed. This scenario occurred in April 2023 when Samsung faced a major internal crisis. Several employees accidentally leaked sensitive information, including proprietary source code, through interactions with OpenAI’s ChatGPT. This incident led Samsung to temporarily ban the use of generative AI tools like ChatGPT across the company.
This isn't an isolated incident. The lack of transparency in AI training data makes it extremely difficult to audit these systems for potential privacy risks. The scope of potential leaks extends far beyond corporate secrets. AI chatbots may leak personal information such as someone's password, home address, sexual orientation, or even Windows product keys.
This risk isn't limited to direct leaks. Sophisticated attackers might use techniques like model inversion or membership inference attacks to extract sensitive information from AI models, even if the model doesn't explicitly output such data.
Consider the case of a high school student, Alex, who regularly used an AI-powered study assistant for research. Over time, Alex's parents noticed a shift in their child's perspectives on certain historical and political topics. It turned out that the AI, trained on undisclosed and potentially biased data sources, was subtly influencing Alex's viewpoints. This raises a crucial question: If we don't know what data AI systems are trained on, how can we trust the information they provide, especially in educational contexts?
Without access to the underlying training data, it's challenging to identify and correct these issues. This lack of transparency could potentially lead to a generation of students learning from flawed or biased sources, with far-reaching implications for society's collective knowledge and understanding.
The undisclosed nature of training data in AI systems poses a risk in global contexts. In countries like Argentina and Turkey, AI was used in elections to create misleading content that aligned with political objectives. In Argentina's 2023 election, AI-generated videos were employed to smear candidates, while in Turkey, deepfakes were used to undermine political opponents.