Path: ...!news.roellig-ltd.de!open-news-network.org!weretis.net!feeder8.news.weretis.net!reader5.news.weretis.net!news.solani.org!.POSTED!not-for-mail
From: Mild Shock <janburse@fastmail.fm>
Newsgroups: comp.lang.prolog
Subject: What does "o1" mean in recent Models (Re: Ilya Sutskever: The Next
 Oppenheimer)
Date: Thu, 19 Dec 2024 10:42:10 +0100
Message-ID: <vk0pph$12ke2$2@solani.org>
References: <vjumvu$1es1e$2@solani.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Injection-Date: Thu, 19 Dec 2024 09:42:09 -0000 (UTC)
Injection-Info: solani.org;
	logging-data="1135042"; mail-complaints-to="abuse@news.solani.org"
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101
 Firefox/91.0 SeaMonkey/2.53.19
Cancel-Lock: sha1:Ea+58eotYQA8vCsN7RJjjd+aATI=
X-User-ID: eJwVwoENwDAIA7CX1kJCOaeE8f8J0yzDuKhwgo7BvF22H9OfBULVrrf9duUErU9o2DsVOWtjEXVkGRbHdD9sURXN
In-Reply-To: <vjumvu$1es1e$2@solani.org>
Bytes: 2222
Lines: 41

Hi,

Could it be that "o1" likely refers to "Optimizer 1".
And what could this include?

- Compressing weights or activations into
fewer bits can significantly reduce computation,
especially in hardware, mimicking O(1)-like
efficiency for certain operations.

- Removing redundant connections in the neural
network leads to fewer computations. Sparse matrix
operations can optimize dense workloads, making specific
inference tasks faster.

- Large models are distilled into smaller ones
with similar capabilities, reducing computational
costs during inference. If the optimized paths are
cleverly structured, their complexity might be
closer to O(1) for lookup-style tasks.

So maybe Ilya Sutskever wants to tell us, in his
recent talk when he refers to the 700g brain line:
Look we did the same as biological evolution,

we found a way to construct more compact brains.

Bye

Mild Shock schrieb:
> Hi,
> 
> I liked some videos on YouTube:
> 
> Ilya Sutskever: The Next Oppenheimer
> https://www.youtube.com/watch?v=jryDWOKikys
> 
> Ilya Sutskever: Sequence to Sequence Learning
> https://www.youtube.com/watch?v=WQQdd6qGxNs
> 
> Bye