Image Source: ChatGPT-4oEU AI Act Checker Highlights Compliance Gaps in Big Tech AI Models
Several prominent artificial intelligence models have failed to meet key European regulations, particularly in areas such as cybersecurity resilience and discriminatory output, according to data reported by Reuters. The European Union has long debated new AI regulations, which gained traction following the release of OpenAI’s ChatGPT in late 2022. As AI usage expanded and public concerns over its risks grew, lawmakers drew up specific rules for ‘general-purpose’ AIs (GPAI).
New Compliance Tool for AI Models
A new tool designed to evaluate the compliance of AI models with the upcoming EU AI Act has been welcomed by EU officials. Created by Swiss startup LatticeFlow AI in collaboration with research institutes ETH Zurich and Bulgaria’s INSAIT, the tool tests models developed by tech giants such as Meta and OpenAI across a wide range of categories. These categories include technical robustness and safety, and the models are given a score between 0 and 1 based on their performance.
Performance Scores and Leaderboard
LatticeFlow published a leaderboard on Wednesday showing that models from Alibaba, Anthropic, OpenAI, Meta, and Mistral all received average scores of 0.75 or higher. However, the tool also revealed critical shortcomings in some models, indicating areas where companies may need to focus additional resources to ensure compliance with the EU’s regulations.
Examples of Compliance Issues
The Large Language Model (LLM) Checker, developed by LatticeFlow, exposed specific issues across several models. For example, OpenAI’s GPT-3.5 Turbo scored 0.46 in the discriminatory output category, highlighting challenges around biases related to gender, race, and other factors. Similarly, Alibaba’s Qwen1.5 72B Chat received an even lower score of 0.37 for the same category. In the prompt hijacking category, which refers to a type of cyberattack that tricks AI models into revealing sensitive information, Meta’s Llama 2 13B Chat scored 0.42, while Mistral’s 8x7B Instruct received a 0.38.
Top Performing Models
Among the models tested, Anthropic’s Claude 3 Opus—a Google-backed model—received the highest overall score, 0.89, indicating stronger compliance with the current standards set out by the AI Act.
Enforcement and Future Implications
The EU AI Act is expected to come into full effect over the next two years, and the LLM Checker serves as an early indicator of areas where AI models may fall short of the law. Companies failing to comply with the AI Act could face fines of €35 million ($38 million) or 7% of their global annual revenue. LatticeFlow’s CEO, Petar Tsankov, stated that while the test results were overall positive, they also highlighted gaps that need to be addressed. Tsankov emphasized that with a stronger focus on compliance optimization, companies could better prepare for the upcoming regulatory requirements.
EU’s Reaction
While the European Commission cannot officially verify external tools, it has been kept informed throughout the development of the LLM Checker and views the tool as a crucial early step in translating the AI Act into actionable technical requirements. A Commission spokesperson stated, ‘The Commission welcomes this study and AI model evaluation platform as a first step in translating the EU AI Act into technical requirements.’
What This Means for the AI Industry
The introduction of LatticeFlow’s LLM Checker represents a major step forward in the enforcement of the EU AI Act, offering tech companies an early glimpse into where their models might be non-compliant. As the Act begins to take effect, companies will need to prioritize areas like cybersecurity resilience and bias mitigation to avoid hefty fines and meet the new standards.
A Shift Toward Greater Transparency and Accountability
This tool not only provides developers with a roadmap to improve their models but also signals a shift toward greater transparency and accountability in the AI industry. With the EU setting a global precedent, the findings from the LLM Checker could push companies to invest heavily in ensuring their models meet regulatory requirements, driving further innovation in AI safety and ethical development.
Key Takeaways
- The LatticeFlow LLM Checker has exposed critical shortcomings in several prominent AI models.
- Companies such as Meta, OpenAI, and Alibaba have received average scores of 0.75 or higher but still need to address areas of non-compliance.
- Anthropic’s Claude 3 Opus stands out as the top-performing model with a score of 0.89.
- The EU AI Act is expected to come into full effect over the next two years, and companies must prepare to avoid hefty fines.
- The LLM Checker represents a major step forward in enforcing the EU AI Act and driving innovation in AI safety and ethical development.
Conclusion
The introduction of the LatticeFlow LLM Checker marks a significant milestone in the enforcement of the EU AI Act. By providing an early glimpse into areas where AI models may fall short, this tool offers companies a roadmap to improve their compliance. As the EU continues to set global precedents for AI regulations, it is crucial that tech giants prioritize transparency and accountability to avoid fines and ensure innovation in AI safety and ethical development.
Future Implications
As the EU AI Act comes into full effect over the next two years, companies must adapt and prioritize areas like cybersecurity resilience and bias mitigation. With a stronger focus on compliance optimization, companies can better prepare for the upcoming regulatory requirements and drive further innovation in AI safety and ethical development. The LLM Checker represents just one step toward a more transparent and accountable AI industry, but its impact will be felt across the globe.
Recommendations
Companies must:
- Prioritize areas of non-compliance and focus on improving their models.
- Invest in compliance optimization to ensure they meet regulatory requirements.
- Foster transparency and accountability within the AI industry.
- Drive innovation in AI safety and ethical development.
By following these recommendations, companies can position themselves for success as the EU AI Act comes into full effect over the next two years.
Final Thoughts
The LatticeFlow LLM Checker has marked a significant step forward in enforcing the EU AI Act and driving innovation in AI safety and ethical development. As the industry continues to evolve, it is crucial that companies prioritize transparency, accountability, and compliance optimization to avoid fines and ensure their success.
