FrontierMath Benchmark: Did OpenAI Cheat?

The too good to be true moment.

Dear Zaiku Community,

A few weeks ago, we shared our excitement about the impressive performance of OpenAI's O3 model on the FrontierMath benchmark, where it achieved exceptional scores compared to other models. However, recent information has come to light (sources ​here ​and ​here​) suggesting that OpenAI may have had privileged access to FrontierMath data beforehand. If true, this raises the possibility that the model's performance was influenced by prior exposure to at least a subset of the dataset, potentially undermining the fairness of the benchmark comparisons.

Additionally, it appears there may be a contractual agreement between OpenAI and FrontierMath's creator, Epoch AI, that restricts or limits other foundational model developers' access to the dataset. While we understand the importance of agreements to sustain the creation of such benchmarks, we believe that critical datasets such as these should be made equally accessible to all developers to foster fairness and innovation across the AI community.

We also recognise the importance of financial contributions in supporting the development of such benchmarks, and we encourage all stakeholders to collaborate in funding these efforts to ensure equitable access for all contributors.

---

QF Academy Course Preview

We’re excited to share a sneak peek of our course, “​A Crash Course on Groups, Topology, and Manifolds.​” Below, you’ll find a preview of Lecture 1 from the Topology and Manifolds section. Our mathematical instructor, Dr. Brian Hepler, is known for his approachable and engaging teaching style. His lectures are carefully crafted to ease students into the material, blending intuition with rigour. Brian’s relaxed and pedagogical approach ensures that even complex concepts feel accessible and enjoyable from the very start.

Companies and organisations can provide their technical teams with dedicated subscriptions to this course, as well as all other courses, through ​QF Academy Business​. The content can be hosted on our Learning Management System (LMS) with a dedicated, customised space, or moved into their own internal LMS.

• ​Lie Groups with Applications to Machine Learning & Quantum Computing​

You can use the promo code Lie-4-AI-2025 to enjoy a 30% discount, valid until today @23:59 GMT!

Wishing you an absolutely spectacular week ahead! 😊

Zaiku Group team