Last month, AI founders and investors told TechCrunch that we’re now in the “second era of scaling laws,” noting how established methods of improving AI models were showing diminishing returns. One promising new method they suggested could keep gains was “test-time scaling,” which seems to be what’s behind the performance of OpenAI’s o3 model – but it comes with drawbacks of its own.
Much of the AI world took the announcement of OpenAI’s o3 model as proof that AI scaling progress has not “hit a wall.” The o3 model does well on benchmarks, significantly outscoring all other models on a test of general ability called ARC-AGI, and scoring 25% on a difficult math test that no other AI model scored more than 2% on.
Of course, we at TechCrunch are taking all this with a grain of salt until we can test o3 for ourselves (very few have tried it so far). But even before o3’s release, the AI world is already convinced that something big has shifted.
The co-creator of OpenAI’s o-series of models, Noam Brown, noted on Friday that the startup is announcing o3’s impressive gains just three months after the startup announced o1 – a relatively short timeframe for such a jump in performance.
“We have every reason to believe this trajectory will continue,” said Brown in a tweet.
Anthropic co-founder Jack Clark said in a blog post on Monday that o3 is evidence that AI “progress will be faster in 2025 than in 2024.” (Keep in mind that it benefits Anthropic – especially its ability to raise capital – to suggest that AI scaling laws are continuing, even if Clark is complementing a competitor.)
Next year, Clark says the AI world will splice together test-time scaling and traditional pre-training scaling methods to eke even more returns out of AI models. Perhaps he’s suggesting that Anthropic and other AI model providers will release reasoning models of their own in 2025, just like Google did last week.
Test-time scaling means OpenAI is using more compute during ChatGPT’s inference phase, the period of time after you press enter on a prompt. It’s not clear exactly what is happening behind the scenes: OpenAI is either using more computer chips to answer a user’s question, running more powerful inference chips, or running those chips for longer periods of time – 10 to 15 minutes in some cases – before the AI produces an answer. We don’t know all the details of how o3 was made, but these benchmarks are early signs that test-time scaling may work to improve the performance of AI models.
While o3 may give some a renewed belief in the progress of AI scaling laws, OpenAI’s newest model also uses a previously unseen level of compute, which means a higher price per answer.
“Perhaps the only important caveat here is understanding that one reason why O3 is so much better is that it costs more money to run at inference time – the ability to utilize test-time compute means on some problems you can turn compute into a better answer,” Clark writes in his blog. “This is interesting because it has made the costs of running AI systems somewhat less predictable – previously, you could work out how much it cost to serve a generative model by just looking at the model and the cost to generate a given output.”
Clark, and others, pointed to o3’s performance on the ARC-AGI benchmark – a difficult test used to assess breakthroughs on AGI – as an indicator of its progress. It’s worth noting that passing this test, according to its creators, does not mean an AI model has achieved AGI, but rather it’s one way to measure progress towards the nebulous goal. That said, the o3 model blew past the scores of all previous AI models which had done the test, scoring 88% in one of its attempts. OpenAI’s next best AI model, o1, scored just 32%.
But the logarithmic x-axis on this chart may be alarming to some. The high-scoring version of o3 used more than $1000 worth of compute for every task. The o1 models used around $5 of compute per task, and o1-mini used just a few cents.
The creator of the ARC-AGI benchmark, François Chollet, writes in a blog that OpenAI used roughly 170x more compute to generate that 88% score, compared to high-efficiency version of o3 that scored just 12% lower. The high-scoring version of o3 used more than $10,000 of resources to complete the test, which makes it too expensive to compete for the ARC Prize – an unbeaten competition for AI models to beat the ARC test.
However, Chollet says o3 was still a breakthrough for AI models, nonetheless.
“o3 is a system capable of adapting to tasks it has never encountered before, arguably approaching human-level performance in the ARC-AGI domain,” said Chollet in the blog. “Of course, such generality comes at a steep cost, and wouldn’t quite be economical yet: you could pay a human to solve ARC-AGI tasks for roughly $5 per task (we know, we did that), while consuming mere cents in energy.”
It’s premature to harp on the exact pricing of all this – we’ve seen prices for AI models plummet in the last year, and OpenAI has yet to announce how much o3 will actually cost. However, these prices indicate just how much compute is required to break, even slightly, the performance barriers set by leading AI models today.
This raises some questions. What is o3 actually for? And how much more compute is necessary to make more gains around inference with o4, o5, or whatever else OpenAI names its next reasoning models?
It doesn’t seem like o3, or its successors, would be anyone’s “daily driver” like GPT-4o or Google Search might be. These models just use too much compute to answer small questions throughout your day such as, “How can the Cleveland Browns still make the 2024 playoffs?”
Instead, it seems like AI models with scaled test-time compute may only be good for big picture prompts such as, “How can the Cleveland Browns become a Super Bowl franchise in 2027?” Even then, maybe it’s only worth the high compute costs if you’re the general manager of the Cleveland Browns, and you’re using these tools to make some big decisions.
Institutions with deep pockets may be the only ones who can afford o3, at least to start, as Wharton professor Ethan Mollick notes in a tweet.
We’ve already seen OpenAI release a $200 tier to use a high-compute version of o1, but the startup has reportedly weighed creating subscription plans costing up to $2,000. When you see how much compute o3 uses, you can understand why OpenAI would consider it.
But there are drawbacks to using o3 for high-impact work. As Chollet notes, o3 is not AGI, and it still fails on some very easy tasks that a human would do quite easily.
This isn’t necessarily surprising, as large language models still have a huge hallucination problem, which o3 and test-time compute don’t seem to have solved. That’s why ChatGPT and Gemini include disclaimers below every answer they produce, asking users not to trust answers at face value. Presumably AGI, should it ever be reached, would not need such a disclaimer.
One way to unlock more gains in test-time scaling could be better AI inference chips. There’s no shortage of startups tackling just this thing, such as Groq or Cerebras, while other startups are designing more cost-efficient AI chips, such as MatX. Andreessen Horowitz general partner Anjney Midha previously told TechCrunch he expects these startups to play a bigger role in test-time scaling moving forward.
While o3 is a notable improvement to the performance of AI models, it raises several new questions around usage and costs. That said, the performance of o3 does add credence to the claim that test-time compute is the tech industry’s next best way to scale AI models.
ADVERTISEMENT:
Hello, sobat pencinta slot! pernahkah denger istilah “raja slot? Kalau belum, bersiaplah jatuh hati sama program ini. slot demo adalah mesin slots yang selalu memberi kemenangan. Ya, mesin-mesin ini bisa disebut adalah andalannya buat membawa pulang hasil. tapi, cemana sih
tekniknya nemuin raja lot yang benar? Santai Bro and Sis, kita beri tenang aja di sini
Games tergacor waktu ini satu-satunya berada Indonesia yaitu yang menyediakan ROI terbaik
SEGERA hanya di :
Informasi mengenai KING SLOT, Segera Daftar Bersama king selot terbaik dan terpercaya no satu di Indonesia. Boleh mendaftar melalui sini king slot serta memberikan hasil kembali yang paling tinggi saat sekarang ini hanyalah KING SLOT atau Raja slot paling gacor, gilak dan gaco saat sekarang di Indonesia melalui program return tinggi di kingselot serta pg king slot
slot demo gacor
slot demo gacor permainan paling top dan garansi imbal balik hasil besar bersama kdwapp.com
akun demo slot gacor
akun demo slot gacor permainan paling top dan garansi imbal balik hasil besar bersama kdwapp.com
akun slot demo gacor
akun slot demo gacor permainan paling top dan garansi imbal balik hasil besar bersama kdwapp.com
akun demo slot pragmatic
akun demo slot pragmatic permainan paling top dan garansi imbal balik hasil besar bersama kdwapp.com
akun slot demo pragmatic
akun slot demo pragmatic permainan paling top dan garansi imbal balik hasil besar bersama kdwapp.com
akun slot demo
akun slot demo permainan paling top dan garansi imbal balik hasil besar bersama kdwapp.com
akun demo slot
akun demo slot permainan paling top dan garansi imbal balik hasil besar bersama kdwapp.com
slot demo gacor
slot demo gacor permainan paling top dan garansi imbal balik hasil besar bersama jebswagstore.com
akun demo slot gacor
akun demo slot gacor permainan paling top dan garansi imbal balik hasil besar bersama jebswagstore.com
akun slot demo gacor
akun slot demo gacor permainan paling top dan garansi imbal balik hasil besar bersama jebswagstore.com
akun demo slot pragmatic
akun demo slot pragmatic permainan paling top dan garansi imbal balik hasil besar bersama jebswagstore.com
akun slot demo pragmatic
akun slot demo pragmatic permainan paling top dan garansi imbal balik hasil besar bersama jebswagstore.com
akun slot demo
akun slot demo permainan paling top dan garansi imbal balik hasil besar bersama jebswagstore.com
akun demo slot
akun demo slot permainan paling top dan garansi imbal balik hasil besar bersama jebswagstore.com
slot demo gacor
slot demo gacor permainan paling top dan garansi imbal balik hasil besar bersama demoslotgacor.pro
akun demo slot gacor
akun demo slot gacor permainan paling top dan garansi imbal balik hasil besar bersama demoslotgacor.pro
akun slot demo gacor
akun slot demo gacor permainan paling top dan garansi imbal balik hasil besar bersama demoslotgacor.pro
akun demo slot pragmatic
akun demo slot pragmatic permainan paling top dan garansi imbal balik hasil besar bersama demoslotgacor.pro
akun slot demo pragmatic
akun slot demo pragmatic permainan paling top dan garansi imbal balik hasil besar bersama demoslotgacor.pro
akun slot demo
akun slot demo permainan paling top dan garansi imbal balik hasil besar bersama demoslotgacor.pro
akun demo slot
akun demo slot permainan paling top dan garansi imbal balik hasil besar bersama demoslotgacor.pro
slot demo gacor
slot demo gacor permainan paling top dan garansi imbal balik hasil besar bersama situsslotterbaru.net
akun demo slot gacor
akun demo slot gacor permainan paling top dan garansi imbal balik hasil besar bersama situsslotterbaru.net
akun slot demo gacor
akun slot demo gacor permainan paling top dan garansi imbal balik hasil besar bersama situsslotterbaru.net
akun demo slot pragmatic
akun demo slot pragmatic permainan paling top dan garansi imbal balik hasil besar bersama situsslotterbaru.net
akun slot demo pragmatic
akun slot demo pragmatic permainan paling top dan garansi imbal balik hasil besar bersama situsslotterbaru.net
akun slot demo
akun slot demo permainan paling top dan garansi imbal balik hasil besar bersama situsslotterbaru.net
akun demo slot
akun demo slot permainan paling top dan garansi imbal balik hasil besar bersama situsslotterbaru.net
situs slot terbaru
situs slot terbaru permainan paling top dan garansi imbal balik hasil besar bersama situsslotterbaru.net
slot terbaru
slot terbaru permainan paling top dan garansi imbal balik hasil besar bersama situsslotterbaru.net
jablay88 permainan paling top dan garansi imbal balik hasil besar bersama jablay88.biz
jadijp88 permainan paling top dan garansi imbal balik hasil besar bersama jadijp88.com
jazz88 permainan paling top dan garansi imbal balik hasil besar bersama jazz88.biz
jurutogel88 permainan paling top dan garansi imbal balik hasil besar bersama jurutogel88.net
kangbet88 permainan paling top dan garansi imbal balik hasil besar bersama kangbet88.biz
kilau88 permainan paling top dan garansi imbal balik hasil besar bersama kilau88.asia
kuningtoto88 permainan paling top dan garansi imbal balik hasil besar bersama kuningtoto88.net
lomboktoto88 permainan paling top dan garansi imbal balik hasil besar bersama lomboktoto88.org
mager88 permainan paling top dan garansi imbal balik hasil besar bersama mager88.biz
mantul888 permainan paling top dan garansi imbal balik hasil besar bersama mantul888.biz
mawartoto88 permainan paling top dan garansi imbal balik hasil besar bersama mawartoto88.asia
meriah88 permainan paling top dan garansi imbal balik hasil besar bersama meriah88.biz
moba88 permainan paling top dan garansi imbal balik hasil besar bersama moba88.org
paristogel88 permainan paling top dan garansi imbal balik hasil besar bersama paristogel88.biz
parsel88 permainan paling top dan garansi imbal balik hasil besar bersama parsel88.net
paus13 permainan paling top dan garansi imbal balik hasil besar bersama paus13.com
pay7777 permainan paling top dan garansi imbal balik hasil besar bersama pay7777.net
planetliga88 permainan paling top dan garansi imbal balik hasil besar bersama planetliga88.net
ratuslot88 permainan paling top dan garansi imbal balik hasil besar bersama ratuslot88.biz
rtp100 permainan paling top dan garansi imbal balik hasil besar bersama rtp100.net
ruby88 permainan paling top dan garansi imbal balik hasil besar bersama ruby88.org
rungkad88 permainan paling top dan garansi imbal balik hasil besar bersama rungkad88.biz
senopati88 permainan paling top dan garansi imbal balik hasil besar bersama senopati88.biz
sis88 permainan paling top dan garansi imbal balik hasil besar bersama sis88.biz
sistoto permainan paling top dan garansi imbal balik hasil besar bersama sistoto.net
sontogel88 permainan paling top dan garansi imbal balik hasil besar bersama sontogel88.com
spin98 permainan paling top dan garansi imbal balik hasil besar bersama spin98.biz
totosaja88 permainan paling top dan garansi imbal balik hasil besar bersama totosaja88.com
warung22 permainan paling top dan garansi imbal balik hasil besar bersama warung22.com
warung88 permainan paling top dan garansi imbal balik hasil besar bersama warung88.asia
winrate999 permainan paling top dan garansi imbal balik hasil besar bersama winrate999.biz
wuzz888 permainan paling top dan garansi imbal balik hasil besar bersama wuzz888.com
999jitu permainan paling top dan garansi imbal balik hasil besar bersama 999jitu.org
adu88 permainan paling top dan garansi imbal balik hasil besar bersama adu88.asia
basket88 permainan paling top dan garansi imbal balik hasil besar bersama basket88.net
batu88 permainan paling top dan garansi imbal balik hasil besar bersama batu88.biz
berita88 permainan paling top dan garansi imbal balik hasil besar bersama berita88.biz
bukalapak88 permainan paling top dan garansi imbal balik hasil besar bersama bukalapak88.net
cipit888 permainan paling top dan garansi imbal balik hasil besar bersama cipit888.asia
delta888 permainan paling top dan garansi imbal balik hasil besar bersama delta888.biz
dosen88 permainan paling top dan garansi imbal balik hasil besar bersama dosen88.org
jago17 permainan paling top dan garansi imbal balik hasil besar bersama jago17.com
jalantoto88 permainan paling top dan garansi imbal balik hasil besar bersama jalantoto88.biz
janjigacor88 permainan paling top dan garansi imbal balik hasil besar bersama janjigacor88.org
jitujp88 permainan paling top dan garansi imbal balik hasil besar bersama jitujp88.com
jokiwin88 permainan paling top dan garansi imbal balik hasil besar bersama jokiwin88.net
juragan66 permainan paling top dan garansi imbal balik hasil besar bersama juragan66.biz
kenzototo88 permainan paling top dan garansi imbal balik hasil besar bersama kenzototo88.biz
kkslot77slot permainan paling top dan garansi imbal balik hasil besar bersama kkslot77slot.com
kompas88 permainan paling top dan garansi imbal balik hasil besar bersama kompas88.biz
kopi88 permainan paling top dan garansi imbal balik hasil besar bersama kopi88.biz
kudajitu88 permainan paling top dan garansi imbal balik hasil besar bersama kudajitu88.biz
kursi88 permainan paling top dan garansi imbal balik hasil besar bersama kursi88.biz
liputan88 permainan paling top dan garansi imbal balik hasil besar bersama liputan88.net
livitoto88 permainan paling top dan garansi imbal balik hasil besar bersama livitoto88.net
lotus88 permainan paling top dan garansi imbal balik hasil besar bersama lotus88.asia
m77casino88 permainan paling top dan garansi imbal balik hasil besar bersama m77casino88.com
majujp88 permainan paling top dan garansi imbal balik hasil besar bersama majujp88.org
mamikos permainan paling top dan garansi imbal balik hasil besar bersama mamikos.org
mamikos88 permainan paling top dan garansi imbal balik hasil besar bersama mamikos88.com
masterplay88 permainan paling top dan garansi imbal balik hasil besar bersama masterplay88.asia
masterplay999 permainan paling top dan garansi imbal balik hasil besar bersama masterplay999.net
medantoto88 permainan paling top dan garansi imbal balik hasil besar bersama medantoto88.biz
medusa888 permainan paling top dan garansi imbal balik hasil besar bersama medusa888.com
meja88 permainan paling top dan garansi imbal balik hasil besar bersama meja88.biz
midasplay88 permainan paling top dan garansi imbal balik hasil besar bersama midasplay88.biz
mijit888 permainan paling top dan garansi imbal balik hasil besar bersama mijit888.net
mposun88 permainan paling top dan garansi imbal balik hasil besar bersama mposun88.net
ibisbudget88 permainan paling top dan garansi imbal balik hasil besar bersama ibisbudget88.com
mercure88 permainan paling top dan garansi imbal balik hasil besar bersama mercure88.com
hotel88 permainan paling top dan garansi imbal balik hasil besar bersama hotel88.net
sheraton88 permainan paling top dan garansi imbal balik hasil besar bersama sheraton88.com
ubud88 permainan paling top dan garansi imbal balik hasil besar bersama ubud88.asia
hardrock88 permainan paling top dan garansi imbal balik hasil besar bersama hardrock88.com
kuta88 permainan paling top dan garansi imbal balik hasil besar bersama kuta88.asia
nasigoreng88 permainan paling top dan garansi imbal balik hasil besar bersama nasigoreng88.com
sate88 permainan paling top dan garansi imbal balik hasil besar bersama sate88.com
rendang88 permainan paling top dan garansi imbal balik hasil besar bersama rendang88.asia
gadogado permainan paling top dan garansi imbal balik hasil besar bersama gadogado.org
kfc88 permainan paling top dan garansi imbal balik hasil besar bersama kfc88.net
pizzahut88 permainan paling top dan garansi imbal balik hasil besar bersama pizzahut88.net
starbucks88 permainan paling top dan garansi imbal balik hasil besar bersama starbucks88.live
sederhana permainan paling top dan garansi imbal balik hasil besar bersama sederhana.org
kopikenangan permainan paling top dan garansi imbal balik hasil besar bersama kopikenangan.asia
angkawin permainan paling top dan garansi imbal balik hasil besar bersama angkawin.net
rockygerung88 permainan paling top dan garansi imbal balik hasil besar bersama rockygerung88.com
monopoli88 permainan paling top dan garansi imbal balik hasil besar bersama monopoli88.com
kimjongun permainan paling top dan garansi imbal balik hasil besar bersama kimjongun.asia
catur88 permainan paling top dan garansi imbal balik hasil besar bersama catur88.asia
tato88 permainan paling top dan garansi imbal balik hasil besar bersama tato88.org
speaker88 permainan paling top dan garansi imbal balik hasil besar bersama speaker88.net
rajah permainan paling top dan garansi imbal balik hasil besar bersama rajah.asia
kunci gitar permainan paling top dan garansi imbal balik hasil besar bersama kuncigitar.org
gitar88 permainan paling top dan garansi imbal balik hasil besar bersama gitar88.asia
gambartato permainan paling top dan garansi imbal balik hasil besar bersama gambartato.com
gambar88 permainan paling top dan garansi imbal balik hasil besar bersama gambar88.org
maxwin888slot permainan paling top dan garansi imbal balik hasil besar bersama maxwin888slot.com
rumah permainan paling top dan garansi imbal balik hasil besar bersama rumah.asia
nada888 permainan paling top dan garansi imbal balik hasil besar bersama nada888.info
musik88 permainan paling top dan garansi imbal balik hasil besar bersama musik88.asia
sewarumah permainan paling top dan garansi imbal balik hasil besar bersama sewarumah.biz