Finbarr Timbers

Latency numbers everyone should know

Wed, 15 Apr 2026 00:00:00 +0000

There's a Google PDF with a bunch of useful latency numbers that software engineers should know. I asked ChatGPT to make it into a markdown file for me.

Latency Table

Operation	Time (ns)	Time (ms)
L1 cache reference	1
Branch misprediction	3
L2 cache reference	4
Mutex lock/unlock	17
Main memory reference	100
Compress 1 kB with Zippy	2,000	0.002
Read 1 MB sequentially from memory	10,000	0.010
Send 2 kB over 10 Gbps network	1,600	0.0016
SSD 4kB Random Read	20,000	0.020
Read 1 MB sequentially from SSD	1,000,000	1
Round trip within same datacenter	500,000	0.5
Read 1 MB sequentially from disk	5,000,000	5
Read 1 MB sequentially from 1 Gbps network	10,000,000	10
Disk seek	10,000,000	10
TCP packet round trip between continents	150,000,000	150

Derived Throughput Estimates

Sequential read from HDD: ~200 MB/s
Sequential read from SSD: ~1 GB/s
Sequential read from main memory: ~100 GB/s (burst)
Sequential read from 10 Gbps Ethernet: ~1000 MB/s

Additional observations:

~6–7 round trips per second between Europe and the US
~2000 round trips per second within a datacenter

Back of the Envelope Calculations

Quick tip: Use decimal-based approximations to simplify mental math.

Sample Calculation

Problem: What is the overall latency of retrieving 30 × 256 kB images from one server?

Naïve design: Do all work on one machine → dominated by disk seek time.

Step 1: Reads Required

30 images / 2 disks per machine = 15 reads

Step 2: Time to Read One Image from HDD

(256 KB / 1 MB) * 5 ms + 10 ms seek = 1.28 ms + 10 ms = 11.28 ms

Step 3: Total Time

15 reads * 11.28 ms = 169.2 ms

Step 4: Throughput

1000 ms / 169.2 ms ≈ 5 result pages per second

What, exactly, is the GIL?

Fri, 10 Apr 2026 00:00:00 +0000

Most ML code is Python. This is surprising to many performance oriented engineers coming from non-ML communities. Python is, notably slow and has the GIL, which forces it to only execute a single thread at a time. The GIL primarily exists to make CPython’s memory management thread-safe. It has also become a load-bearing part of Python's API, as, in a classic example of Hyrum's Law, much Python code relies on the thread safety that it has created.

It is also both critically important and a constant source of confusion for many Python developers, including myself! To refresh my understanding, I wrote this article.

In short, the Global Interpreter Lock (GIL) is a lock that exists within the CPython interpreter, guaranteeing that only one thread is executing Python bytecode at any given time. This prevents multiple threads from running Python bytecode in parallel. Note the language: "Python bytecode in parallel." Pure Python is interpreted into bytecode and executed. That cannot run concurrently.

Of course, much useful code requires concurrent execution, so this is a big problem. The GIL can be released, however, by using the Python C API, which provides primitives to explicitly release it. So the GIL is released by code which calls the relevant bits from the Python C API to release it.

A common refrain in Python is that "I/O releases the GIL." This is because all of Python's blocking I/O primitives release the GIL (i.e. call the relevant bits of the Python C API!) while waiting for the I/O block to resolve. There's nothing magical about I/O here! It's simply a (relatively) slow operation implemented in a lower-level language which releases the GIL.

There are many other slow operations implemented in a lower-level language that release the GIL.

When does Torch release the GIL?

From one of the maintainers: "We do release the GIL as soon as we get out of python code. So for pytorch ops, it’s more or less all of them. The backward, unless you implement custom Functions, will run completely out of the GIL.". If we look at a typical PyTorch training loop:

for batch, (X, y) in enumerate(train_dataloader): # Typically heavy Python code, holds the GIL.
    logits = model(X) # GIL is released during a bunch of the operations here.
    loss = loss_fn(logits, y) # GIL is released during a bunch of the operations here.

    optimizer.zero_grad() # Backend ops should release GIL.
    loss.backward() # Assuming it uses autograd, almost entirely outside of the GIL.
    optimizer.step() # Much of the ops that make up the bulk of the work here happen outside of the GIL.

    if batch % 100 == 0: # Python, holds the GIL.
        print(f"loss: {loss.item():>7f}") # Python, holds the GIL (and `loss.item()` triggers a host-device sync).

Of course, all of the Python code executing between the pytorch ops will hold the GIL.

When does Ray release the GIL?[^1]

Calls to ray.get or ray.wait will typically release the GIL, although it depends on the specifics, and it's not guaranteed. The concrete call chain for ray.get is:

public ray.get(...) in worker.py which calls worker.get_objects(...) which calls self.core_worker.get_objects(...) whose Cython implementation in _raylet.pyx wraps the underlying CCoreWorkerProcess.GetCoreWorker().Get(...) call in with nogil:.

This is a typical example of the GIL! The GIL is released when you call Python code which ends up calling native code which calls the relevant bits of the Python C API to release the GIL. There's no magic!

This is why a typical pattern that I like to employ in async RL works; I like to have a single process orchestrating my job using Ray, and I'll spin up a bunch of threads on the process to handle other operations. Because the threads are constantly making calls to ray.get, they're constantly releasing and acquiring the GIL, so they work well together and don't starve each other of resources.

When does vLLM release the GIL?

vLLM is largely a Python orchestrator around heavy accelerator operations. The GIL is released whenever threads execute:

PyTorch tensor ops (ATen / CUDA kernels)
NCCL communication calls
Custom CUDA kernels (PagedAttention, FlashAttention, etc.)

These are implemented in native code and explicitly release the GIL in the same way as Torch (because they mostly are Torch). So, conveniently, you can have multiple threads running concurrently on your vLLM server, perhaps executing tool calls.

[1]: Funnily enough, if you google "when does Ray release the GIL?" one of the top results is from the Beyblade wiki.

Making RL Fast

Fri, 03 Apr 2026 00:00:00 +0000

For Olmo 3, I was put in charge of our post-training infrastucture. We made the decision to move from a synchronous RL setup to an asynchronous one to enable us to scale. In doing that work, I was fortunate enough to find a series of optimizations which made our RL setup 4x faster. As we used roughly 250k H100 hours running RL on Olmo 3, these optimizations saved us approximately 750k H100 hours (~$1.5M) at current market prices. These changes were detailed in the paper, but I wanted to write more about them here.

There were three primary optimizations:

Continuous batching.
Inflight updates.
Better thread synchronization.

Continuous batching (+11% throughput)

For Olmo 2, our post-training code used a synchronous RL approach, in which the actors and learner all operate in lockstep. For every training step, we load a batch of prompts that we want to send to the actors to complete, and we split the batch into N chunks (where N is the number of actors), sending each to a different actor. Then, our learner waits for all the actors to complete their generations. When they're all done, the N completion chunks are reassembled into a batch of completions which is trained on.

Such a synchronous RL setup is difficult to optimize and leaves a lot of performance on the table. It does what is known as "static batching." In LLM inference, due to variable sequence length, there's a natural tendency towards inefficiency. In traditional ML inference (say: you're running inference on a ResNet that is trying to classify MNIST), there's no notion of sequence length, so all you have to worry about is your batch size. To efficiently run inference, you'd fill your GPU with a sufficiently big batch of examples and be done. With LLM inference, due to variable sequence length, this is inherently inefficient. If you follow the naive approach, you'll waste (max_sequence_length - mean_sequence_length)/max_sequence_length of your compute. On Olmo 3, we had an average generation length of 14k, and a maximum of 32k, so we would have wasted 54\% of our compute with static batching. See the diagram for an illustrated example.

Instead, to optimally use your GPUs, you should be streaming examples in and out of your GPUs as previous completions finish. This is called continuous batching, where you constantly enqueue new generations as each one finishes. Continuous batching is how most LLM inference servers operate, so it wasn't that difficult to implement, but it required refactoring our codebase to support the new streaming architecture.

(I made ChatGPT make the original version of this image in Tikz. It was remarkably difficult. The LaTeX is available if you care to see how I tortured ChatGPT.)

Inflight updates (+117% throughput)

The major disadvantage of asynchronous RL is that it makes the actors more off-policy from the learner, i.e. the learner will be training on the current version of the model, but the actors might have generated the completions with an older version of the model. To avoid this, the synchronous version of the code would synchronize the actor weights after every training step. In the asynchronous setting, what this looks like is: each actor finishes all of their enqueued generations, stops accepting new prompts to their queue, updates their copy of the weights, and then resumes processing. Note, however, that this is exactly equivalent to the static batching case above, and introduces the same idleness problem!

Instead, we implemented PipelineRL, and only pause generation, instead of waiting for all completions to finish before updating the weights. By doing this, we avoid draining the queue, and keep our nice continuous batching performance, keeping our GPUs humming. This enables a significant increase in throughput: up to 2x faster in my experiments with the same resources, without hurting accuracy.

It is, however, a bit weird. Note that we pause partway through completing a generation, and then resume it with updated weights. We don't invalidate the KV cache! This means that we are using the KV cache calculated from an older version of the weights. It works fine, and can be justified with a handwavy epsilon-delta proof as a neural network is the composition of continuous functions, but it's a bit weird. Empirically it's fine, though, and it's hard to argue with a 2x speedup.

Better threading (+39% throughput)

Finally, after all these sophisticated changes to our RL pipeline, we ran headfirst into Amdahl's law. As we made our actors more asynchronous, the synchronization points become more of a bottlenck. In particular, coordinating the actors, and specifically, the weight sync, was a bottleneck.

To fix this, we had to completely decouple the actors, allowing each one to start and stop by itself, without waiting for the rest of the actors to finish their syncs as well. Similarly, we make a large number of optimizations that were not machine learning specific, and were centered around efficiently using the CPU. For example, our initial implementation of continuous batching, for instance, was slower than static batching until we adding a background prefetch thread to our actors that constantly refilled the inference queue.

Decoupling the actors was a pure systems change that affected how we synchronize the learner and the actors. The idea is that if you have N actors and need to do a weight broadcast, the naive way is something like this:

Ask them all to stop (for i in range(num_actors): actors[i].stop())
Send the weight update to each one in turn (for i in range(num_actors): actors[i].update_weights())
Restart inference on the actors (for i in range(num_actors): actors[i].start()).

In pseudo-code:

for i in range(num_actors):
    actors[i].stop()
for i in range(num_actors):
    actors[i].update_weights()
for i in range(num_actors):
    actors[i].start()

If we do this, we have a bunch of synchronization points which kill performance in distributed systems, as the actors have to wait for all of the stragglers to finish.

Instead, we can have each actor operate independently and have each actor start/stop independently:

for i in range(num_actors):
    actors[i].stop()
    actors[i].update_weights()
    actors[i].start()

Of course, we run this concurrently, so all the actors are doing this simultaneously. So perhaps something closer to:

async def update_weight(actors: list[Actor], actor_index: int):
    actor = actors[actor_index]
    await actor.stop()
	await actor.update_weights()
	await actor.restart()

await asyncio.gather(*(update_weight(actors, i) for i in range(num_actors)))

The main bottleneck now is that we have our learner broadcasting the weight updates, which is inherently sequential. We could move to an asynchronous approach like Cursor did for Composer 2, where the weights are written to disk, but we haven't found the weight broadcast to be a bottleneck in practice. Here's the table from the Olmo 3 paper showing our results.

To benchmark the changes, we used Qwen 2.5 7B, generating a batch of 16 prompts, each sampled 4 times, with a maximum generation length of 32k tokens and a maximum prompt length of 2k tokens. We run each RL experiment for 2 hours and measure tokens/second based on the total output over the 2 h. We use 2 8x H100 nodes: one for the trainer and one for the actors. We have 8 actors running; one on each GPU.

Tokens per second	MFU	MBU	Notes	Commit
881	0.30%	12.90%	Baseline before any major changes	564f8a4
975	0.33%	14.29%	plus continuous batching	ab90d5a
1358	0.46%	19.89%	plus better threading	e320ff0
2949	1.01%	43.21%	plus inflight updates	e320ff0

The Bitter Lesson

Thu, 26 Jun 2025 00:00:00 +0000

The Bitter Lesson is an excellent essay which is overwhelmingly misunderstood. The point of the bitter lesson is that, over time, methods which scale with compute will outperform methods that do not.

It is not:

The idea that we should never incorporate human knowledge
The idea that deep learning and scale are all we need (Rich is actually relatively skeptical of deep learning)

The entire point of the essay is that, in the last 5 decades, we have seen massive increases in the amount of compute available to us as an industry and we expect to continue to see massive increases in the amount of compute available to AI research. Methods which take advantage of compute will benefit, and those that do not will suffer.

The reason the lesson is bitter is that it is often much easier and quicker to get results by incorporating human knowledge.

If you’re training an autocomplete system in 1995, you’re probably not going to get very far with next token prediction, and instead, handcoded, or statistically generated rules will do better. In 2005, N-gram models are optimal. It isn’t until the mid 2010s that we start to see deep learning dominate in NLP, and not until the late 2010s that self supervised learning becomes dominant. At each point along the way, incorporating human knowledge has been advantageous, and has been a way that you can get an advantage over your competition. But in the long term, it’s a dead end. Methods which take advantage of more compute outperform over a sufficiently long time frame. Compute is the only parameter which we can expect to increase by several orders of magnitude. Much as I wish it were otherwise, it’s unlikely that we’ll see 1000x the number of tokens we have now. But in compute, that’s almost certain.

The canonical example is computer chess. Before Deep Blue expert systems were largely used. Deep Blue showed that leveraging compute to perform extensive searches against a hand coded value function1 performed extremely well. Deep Blue was a massive win for the “scale compute”/computer search crowd, as it was much more based on scale than human heuristics, but it required an evaluation function with 8000 custom chess features created by human experts, and the evaluation function weighted them using hand selected weights. One measure of the generality of the system is how easy it would be to extend it to a different scenario. To extend Deep Blue to work in Go would be extremely challenging, as one would need to come up with a proper evaluation function by creating another 8000 custom Go features.

Computer Go is another example where human knowledge fell short. AlphaGo Zero evaluated against the then state of the art Go bots, which included Pachi, GnuGo, and CrazyStone. Pachi and CrazyStone did MCTS with heuristic value functions, and GnuGo was an expert system, with a hand created decision tree to select moves. They were good at the time! But they were all ultimately dead ends. As Rich states in the article:

The bitter lesson is based on the historical observations that

1) AI researchers have often tried to build knowledge into their agents,

2) this always helps in the short term, and is personally satisfying to the researcher, but

3) in the long run it plateaus and even inhibits further progress, and

4) breakthrough progress eventually arrives by an opposing approach based on scaling computation by search and learning.

The eventual success is tinged with bitterness, and often incompletely digested, because it is success over a favored, human-centric approach.

If you look at GnuGo’s code, it was a lot of hard work by a lot of people, and was dramatically worse than what was possible. What’s surprising is that, while GnuGo began in 1989, released continued until 2009, so the authors were undoubtedly aware of Deep Blue and the stunning victory that scaled search had, yet they continued to push forward with their expert system. Brian Lee, a former Google Brain researcher who replicated AlphaGo within Brain, offers a compelling explanation for why:

I offer another point: that these stages [of the Bitter Lesson] happen over the span of a decade or so. Over this decade, PhDs are minted, career identities built, promotion criteria set, culture defined, and org charts annealed. Much in the way that science progresses one funeral at a time, progress on difficult problems progresses one organization shutdown at a time.

Consider another scenario. You work at a LLM lab, and have to make your benchmark numbers bigger than your competition or you get fired. You have the immediate temptation to include human knowledge, which in this case might be specialized datasets for a specific benchmark.

A better approach would be to make the model generally stronger. Focusing on methods that scale with compute as a filter is a strong bet to make, as Jensen Huang is doing his best to give you multiple orders of magnitude more FLOPS. Methods like test time compute, synthetic data, or MoE models are great examples. But the problem with this approach, which when I write it down seems obvious, is that in the moment, it feels indulgent. We don’t have time for proper science, we have to beat the other labs on LiveCodeBench. That is the bitter lesson: DeepSeek focuses on general improvements, gets them working, scales them to 3.8e25 FLOPS, and is SOTA.

Articles I’m reading right now:

What comes next, by Nathan Lambert (Interconnects), discussing, among other aspects, how excellent O3 is.
Undertrained tokens in R1, by Sander Land.
The Deep Blue paper, which is worth reading.

1 Deep Blue is fascinating for a variety of reasons, including the fact that they had custom “chess chips” made to encode the evaluation function in hardware.

Request for research: Monte Carlo Tree Search for reasoning, with PUCT

Tue, 21 Jan 2025 00:00:00 +0000

In the recent wave of research studying reasoning models, by which we means models like O1 which are able to use long streams of tokens to "think" and thereby generate better results, MCTS has been discussed a lot as a potentially useful tool. However, some papers, like the DeepSeek R1 paper, have tried MCTS without any success.

I worked closely with MCTS for several years while at DeepMind, and there are a number of implementation details that I think researchers (such as DeepSeek) are either getting wrong or not discussing clearly. Here, I will discuss these, and issue a Request for Research in the hopes that some intrepid researcher will implement these so I can finally stop ~~complaining~~ suggesting that someone study this.

Use PUCT, and not UCT

In the DeepSeek paper, they had a section discussing MCTS:

In the section, the authors said "MCTS guided by a pre-trained value model." They repeated the phrase "value model" repeatedly, concluding that "while MCTS can improve performance during inference when paired with a pre-trained value model, iteratively boosting model performance through self-search remains a significant challenge." To me, the phrasing indicates that the authors are not using a learned prior function, as AlphaGo/Zero/MuZero did. If true, this is a mistake.

In classic MCTS, an action-value function ( $Q(s, a)$) is used to estimate the value of each candidate action:

\[UCT(s,a) = Q(s,a) + c \sqrt{\frac{\ln N(s)}{N(s,a)}} = Q(s, a) + c E_{\text{uct}}(s, a)\]

In the UCT equation, there are two terms: $Q(s, a)$ and the exploration term, which is the right side of the equation ($c$ is a hyper-parameter), and is solely a function of how many times the (state, action) pair has been visited. If the current node has been visited a small fraction of the times that the parent node N(s) has been visited, the exploration term is large, but it grows smaller as it is visited more.

In AlphaGo/AlphaZero/MuZero style MCTS, they use PUCT, where the exploration term is weighted by a learned prior function:

\[PUCT(s,a) = Q(s,a) + c P(s,a) \frac{\sqrt{N(s)}}{1 + N(s,a)} = Q(s, a) + c E_{\text{puct}}(s, a)\]

The weight is learned by having the network predict $N(s, a)/N(s)$, i.e. the % of times that the child node is visited during search. By using the prior, MCTS is able to go much deeper. Consider chess, which has, on average, 35 legal moves at any point in the game. If you are doing N searches, and you visit every legal move at a given state before moving on to a child state, you will only be able to explore $N^{\frac{1}{35}}$, which is tiny. Most of these moves are obviously bad, so by using the prior to prune those nodes, search goes much deeper.

Consider also the form of the exploration term. $E_{\text{uct}}(s, a)$ assigns infinite weight to a (state, action) pair that hasn't been visited before, while $E_{\text{puct}}(s, a)$ assigns it a value of 0 (or, in some implementations, different fixed values to allow for e.g. optimistic initialization).

As such, UCT will do a breadth first search, while PUCT will perform a depth-first search. Neither is superior to the other in a general sense, but in a domain that has a large number of potential actions to take, like, say, language modelling, breadth-first search will not do much of anything.

My suggestion would be to use the standard logit head as the prior and train a value head on the same embeddings that the logit head gets. Then, update this during the RL phase of training. While the experiments are inherently expensive, you can do the experiments on a small model, such as Llama 1B, to see if they help.

Parallel evaluations are required for action diversity

With MCTS, it is very easy to harm the diversity of your search if you don't search in parallel. If using sequential search, make sure that you are using virtual visit counts as done in the "Mastering Board Games by External and Internal Planning with Language Models" paper, where they state:

"Parallel MCTS implementations usually rely on virtual losses (Chaslot et al., 2008; Mirsoleimani et al., 2017) to avoid multiple threads choosing the same simulation paths down due to deterministic action choices."

Otherwise, search in parallel. The lack of parallelism is, in my mind, a big reason why the Student of Games struggled to beat AlphaZero in Chess/Go.

Generally the way that MCTS works is that you have two phases:

search
action selection

During search you have another two phases:

simulation
evaluation

An example is the OpenSpiel MCTS implementation, although it is single-threaded and synchronous. In the simulation phase, you are traversing the search tree and continuing to go down the search tree until you find a new node to add to the tree or until you reach a terminal state. In the evaluation phase you are running a neural network inference to say how good the given state is. Both of these can be done asynchronously and in parallel.

So, you have some number of threads running simulations in parallel and each of them is queuing up evaluations which themselves are evaluated in parallel by a separate threadpool. The literature has shown that the exact number of threads used for both is critical and doing these asynchronously is also critical; both should be considered hyperparameters.

When you are implementing the search tree you should make sure not to use locks. You want to use locks only when you are actually adding to the search tree. For the search tree itself, use atomics or some sort of structure that lets you add or modify the search statistics concurrently.

When you reach the action selection phase there's a bunch of different rules that you can use to choose the action that you're actually going to take (there's obviously any number of action selection rules you could use, but here are 3 I've found useful):

1) Normalize the search visits and sample from the resulting probability distribution 2) Softmax the visit counts and sample from the resulting probability distribution 3) Choose the most visited action during search.

In the approximate best response paper, my experience was that some games, particularly the stochastic games, required the softmax rule to get the best response, while others (particularly chess/go) required the "most visited" rule to get the best response.

RESP advice

Fri, 08 Nov 2024 00:00:00 +0000

A few people have asked me for RESP advice, so here is my generic answer, with the disclaimer that I'm not a financial advisor.

There are two major decisions to make for an RESP:

Decide how much you want to invest per year
Decide whether you want to self-manage your money or not

This article discusses the options for 1. It really depends on how much money you have available to invest and what your alternatives are. I'm assuming you've maxed out your RRSP/TFSA: if you haven't, you probably want to contribute the minimum $2500 per year to max the gov't RESP grant and put the rest in your RRSP/TFSA. If you contribute at least $2500 per year for 14 years, and then have an additional year where you contribute at least $1k, you'll max out the CESG, which is the gov't grant you get for contributing to the RESP.

In terms of choosing how much to invest, the mathematically optimal thing to do is to invest all of the money (~$50k) in the first year. This is, obviously, hard to do, but will result in the best outcome, as the money has the longest time horizon to grow tax-free. You don't get much gov't grants by doing this, but it's a good way to maximize your

A second option is to invest $16500 ($14k + $2500 annual contribution) upfront, and then invest an additional $2500 every year until they turn 14, at which point you've maxed out all the gov't grants, and you can top it up with $1k.

Finally, if that's not financially feasible, you can do $2500 per year, and try to catch up the remaining $14k when you have extra income. In any case, it's worth doing the minimum $2500 to max out the grants.

Self-management.

A lot of the RESPs that are actively advertised to parents are, to be blunt, scams. I'd recommend going with either a low-fee online broker, like Questrade, QTrade, or WealthSimple, or with one of the big bank brokerage accounts, like TD Direct Investing or BMO Investorline. Basically, whoever you have your RRSP/TFSA with should be able to open an RESP up with them.

The recommended strategy for investing your money is to start out with a 90/10 equities/fixed income portfolio, something like VFV/VAB, and every year, adjust the portfolio so that when they turn 18, you have 100% in fixed income. You basically make the % allocated to fixed income equal to the (kid's age + 2) * 5%. So 10% in the first year, 15% when they're 1, 20% when they're 2, etc. You can be slightly more aggressive with your schedule, but you really want to be gradually shifting the portfolio towards 100% bonds by the time they're 18 to minimize variance.

This is pretty easy to do if you're comfortable buying/selling ETFs. I have a spreadsheet that calculates this.

If you're not comfortable doing this yourself, I'd go with one of the managed funds. I like JustWealth as they provide a dedicated RESP target-date fund which will do everything I described above for you. You could also go with Questwealth or WealthSimple and adjust the aggressiveness over time. Looking at Questwealth as an example (but all the advisors have similar options), I'd start out in an Aggressive fund for the first 4 years, then Growth for the next 4 years, followed by Balanced for 4 years, then Income for 4 years, and ending in Conservative. This would be easy to do, just set a calendar reminder to lower the aggressiveness every 4 years.

Installing Docker on a new VM

Fri, 02 Feb 2024 00:00:00 +0000

I consistently run into the same issue when install Docker on a new Ubuntu VM.

I copy the command from the Docker docs, and it fails with the error:

E: The repository 'https://download.docker.com/linux/ubuntu bookworm Release' does not have a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.

There is a simple fix, but it is annoying to find it via Google, and it takes me ~20 minutes every time:

https://forums.docker.com/t/solved-cannot-install-docker-ce-on-debian-stretch/50407/5.

I have now created this page to write it down. Google, please index this.

To install Docker with GPU support:

1) Install Docker.

2) Run the Docker install fix.

3) Install the Nvidia Container Toolkit.

4) Install GPU drivers.

Five years of GPT progress --- Amii talk

Mon, 29 May 2023 00:00:00 +0000

I recently gave a talk at Amii about the history of GPT models.

The slides are now online, as is the recording.

Deriving the DALL-E lower bound

Wed, 05 Apr 2023 00:00:00 +0000

In the DALL-E paper, we want to model the joint likelihood of the model distribution over images $x$, captions $y$, and tokens $z$ for an encoded RGB image¹.

The joint likelihood is modelled using the factorization

\begin{equation} \label{eq:factorization} p_{\theta, \psi}(x, y, z) = p_\theta(x | y, z) p_\psi(y, z) \end{equation}

The paper presents this lower bound

\begin{equation} \label{eq:lower-bound} \ln p_{\theta, \psi}(x, y) \geq \mathbb{E}_{z \thicksim q_\phi(z |x)} \left( \ln p_\theta (x | y, z) - D_{KL}(q_\phi(y, z | x), p_\psi(y, z))\right) \end{equation}

It was unclear to me how this was derived (and apparently unclear to others), so I thought I’d try to derive it myself.

The evidence lower bound lets us write

\[\ln p_{\theta, \psi}(x, y) \geq \mathbb{E}_{z \thicksim q_\phi}\left[ \ln \dfrac{p_{\theta, \psi}(x, y, z)}{q_\phi(z)}\right]\]

From the factorization of the joint likelihood, we can rewrite this as

\[\begin{align*} \mathbb{E}_{z \thicksim q_\phi}\left[ \ln \dfrac{p_{\theta, \psi}(x, y, z)}{q_\phi(z)}\right] &= \mathbb{E}_{z \thicksim q_\phi}\left[ \ln \dfrac{p_\theta(x | y, z) p_\psi(y, z)}{q_\phi(z)}\right]\\ &= \mathbb{E}_{z \thicksim q_\phi}\left[ \ln p_\theta(x | y, z) + \ln p_\psi(y, z) - \ln q_\phi(z)\right]\\ &= \mathbb{E}_{z \thicksim q_\phi}\left[ \ln p_\theta(x | y, z)\right] + \mathbb{E}_{z \thicksim q_\phi}\left[\ln p_\psi(y, z) - \ln q_\phi(z)\right]\\ &= \mathbb{E}_{z \thicksim q_\phi}\left[ \ln p_\theta(x | y, z)\right] + \mathbb{E}_{z \thicksim q_\phi}\left[\ln \dfrac{p_\psi(y, z)}{ q_\phi(z)}\right] \end{align*}\]

Note that the term on the right is precisely the KL-divergence between $q_\phi$ and $p_\psi$:

\[D_{KL}(q_\psi(y, z | x) || p_\psi(y, z)) = \mathbb{E}\left[\ln \dfrac{q_\psi(y, z | x)}{p_\psi(y, z)}\right] = -\mathbb{E}\left[\ln \dfrac{p_\psi(y, z)}{q_\psi(y, z | x)}\right]\]

So we can write:

\[\mathbb{E}_{z \thicksim q_\phi}\left[ \ln \dfrac{p_{\theta, \psi}(x, y, z)}{q_\phi(z)}\right] = \mathbb{E}_{z \thicksim q_\phi}\left[ \ln p_\theta(x | y, z)\right] - D_{KL}\left(q_\psi(y, z | x) || p_\psi(y, z)\right)\]

Which, combined with Equation \eqref{eq:lower-bound}, gives us our lower bound.

Encoded using a dVAE— see the DALL-E paper for details. ↩

Five years of GPT progress

Mon, 27 Mar 2023 00:00:00 +0000

If you want to read more of my writing, I have a Substack.

In this article, I discuss the generative pre-trained transformer (GPT) line of work, and how it has evolved over time. I focus on the SOTA models, and the differences between them. There are a bunch of different articles summarizing these papers, but nothing that I’m aware of that explicitly focuses on the differences between them.

I focus on the GPT line of research as that’s what’s driving the current fever pitch of development. There’s a ton of prior work before large GPTs (eg the n-gram models from the 2000s, BERT, etc) but this post is super long, so I’m gonna save those for future articles.

GPT

Abstract

The first GPT paper is interesting to read in hindsight. It doesn’t appear like anything special and doesn’t follow any of the conventions that have developed. The dataset is described in terms of GB rather than tokens, and the number of parameters in the model isn’t explicitly stated. To a certain extent, I suspect that the paper was a side project at OpenAI and wasn’t viewed as particularly important; there’s only 4 authors, and I don’t remember it particularly standing out at the time.

The architecture is remarkably unchanged compared to GPT-3:

Decoder-only transformer, with 12 layers, 768 embedding dimension, 12 attention heads, and 3072 (4x the embedding dimensions).
They use Adam, with a warm up, and anneal to 0 using a cosine schedule.
Initialize weights to N(0, 0.02), using BPE with a vocab of 40000 merges.
Activations are GELUs.
Context of 512
117M parameters
Learned position embedding, not the sinusoidal ones from “Attention is all you need”.

The number of parameters isn’t explicitly discussed, but appears to be roughly 120M, easily enough to fit on a single V100 or a standard consumer GPU (rough estimate of 120M parameters for the model, 240M for the optimizer, for 360M parameters; assuming each is a float32, then this takes up 4 bytes * 360M = 1440MB/1.4GB.

They use the BooksCorpus dataset (~20M tokens), training for 100 epochs with a batch size of 64. 20M tokens is a very small dataset by modern standards, as is a batch size of 64.

The most surprising thing compared to modern GPTs is that they train for 100 epochs. Modern GPTs rarely ever see repeated data, and if they do, they typically only see certain datapoints a small number of times (2-4x), and the entire dataset is never repeated 100x.

GPT-2

Abstract

GPT-2 is where the language models start to get big. This is the first time that OpenAI trains a model with >1B parameters. We start to see scale as a primary concern; in GPT, the authors trained a single model, but here, the authors train a range of models, with sizes ranging from GPT to 10x GPT (which is the actual GPT-2 model).

The differences in architecture compared to GPT are as follows:

They layernorm the inputs and add an additional layernorm to the output of the final self-attention block
Weights are scaled by layer by 1/sqrt(n)
Vocabulary of ~50k (up from ~40k)
Context of 1024 (up from 512)
Batches of 512 (up from 64)
Largest model is 1.5B parameters

The dataset is much, much bigger, going from roughly 20M tokens (4GB) of data consisting of publicly available books, to 9B tokens¹ (40GB) of text scraped from the internet (WebText).

It’s unclear if they trained the model for 100 epochs as before; they say they followed the same training procedure, so presumably they did. Again, this is a significant departure from later work.

Nothing here is particularly different from GPT; most of the changes are related to making the model bigger. The only other changes are the layernorm changes and the weight scaling, which don’t seem to make a big difference (although, as always, more ablations would be nice).

GPT-3

Abstract

Here is where the era of truly large language models began, and the current AI ~~bubble~~ excitement took off. In the paper, the authors train 10 models, varying from 125M parameters (”GPT-3 Small”) to 175B parameters (”GPT-3”).

For each of the models, the architectures are identical to GPT-2 with the exception that they use “alternating dense and locally banded sparse attention patterns in the layers of the transformer.” The sparse attention here refers to the attention mechanism introduced in the Sparse Transformer, which lets attention scale proportional to $O(n \sqrt{n})$ (where $n$ is the context length). The standard dot-product attention mechanism scales proportional to $O(n^2)$, so this is a substantial gain. I would have loved a proper ablation to see what difference sparse vs dense attention makes, but alas.

I’m very curious why they used sparse attention. Reproductions and later papers uniquely use dense attention. As this paper came before FlashAttention and some of the other algorithmic innovations that make dense attention faster, maybe this was a computational bottleneck? It’s really unclear.

They don’t provide any detail about the computational architecture, i.e. how they distributed the model. The authors claim it’s because it doesn’t really matter, but I think it was restricted for competitive reasons, as it makes the paper much more difficult to reproduce. Megatron, which I’ll discuss later, was highly influential because they went into detail about how they made model parallelism work for their GPT.

What I find really interesting about the GPT-3 paper is that it was an incredible advance without a lot of novelty. They took their existing methods and “just” scaled it up! Because of the need for novelty, there are many research projects that don’t get pursued because they’re “only” engineering projects, or they “only” do hyper-parameter tuning and wouldn’t be able to get published, even if they had impressive performance improvements. That OpenAI went against the grain here is a credit to them (and they were rewarded, with GPT-3 getting a best paper reward at NeurIPS ‘20).

This is particularly problematic because we know that adding complexity to our models increases performance (see: R^2 vs adjusted R^2 for simple linear models). Because of the need for novelty, there are many research projects that don’t get pursued because they’re “only” engineering projects, or they “only” do hyper-parameter tuning and wouldn’t be able to get published, even if they had impressive performance improvements. That OpenAI went against the grain here is a credit to them.

This is a strength of OpenAI (and Stability.ai, Midjourney, basically everywhere that’s not FAIR/Google Brain/Deepmind/etc). You could alternatively frame it as a weakness of the more academic labs that have promotion/performance review policies driven by publications.

Jurassic-1

PDF

I wasn’t sure whether or not to include Jurassic-1. It’s a model from the Israeli tech company AI21 Labs. I haven’t heard a lot about them, but the paper’s cited by a bunch of the papers later on in the article; they trained a 178B parameter model that outperformed GPT-3 in a few categories, and was faster for inference. It’s impressive that they’re competing with DeepMind, OpenAI, Nvidia, etc. despite only having raised <$$10M at the time. They made a zero-shot and few-shot test suite publicly available.

Like many other papers, they don’t go into detail about the engineering details behind training a large model (178B parameters) over 800 GPUs:

The paper is remarkably sparse on details, which I suspect was done for competitive reasons, just like GPT-4.

Facebook is the only company to go into detail about their experiences training a 175B parameter model, just like Nvidia is the only company to go into detail about the computational architecture required to train a LLM over many GPUs (see: the Megatron paper, next). In both cases, the companies are commoditizing their complements and strengthening their main lines of business by making it easier to train large models.

Jurassic uses a different architecture from GPT-3, but again, doesn’t go into much detail:

76 layers (vs 96 layers for GPT-3)
They use the SentencePiece tokenizer, with a large vocabulary of 256K (vs GPT-3 which used BPE w/ ~50k tokens).

Neither of these changes are material, in my opinion. I think what we’re seeing is that there’s a relatively large degree of freedom in model architectures which produce similar results. This is borne out by their evaluation, which has results similar to GPT-3 (better in some categories, worse in others), although Jurassic-1 is faster for inference due to being shallower.

We’re starting to see a consistent pattern emerge:

Papers introduce a bunch of changes, their own dataset, and have a new SOTA
but they don’t do a proper ablation, so it’s tough to understand what was important and what drove the improvements

GPT-2, GPT-3, Jurassic-1, etc. all did this.

Megatron-Turing NLG

Megatron was a highly influential paper that introduced efficient model-parallel architectures. If you’re interviewing for a LLM job today, you’re going to be expected to be familiar with it. Megatron introduced tensor parallelism, a variant of model parallelism that splits the models to allow for intra-layer model parallelism, achieving 76% as efficient as a single GPU baseline (although the baseline is only 30% of peak FLOPS).

Prior to megatron, the published SOTA for model parallelism was to use model pipelining, e.g. GPipe. However, this was difficult to do and not well supported by code. There were attempts to support tensor parallelism, e.g. Mesh-Tensorflow, which introduced a language for specifying a general class of distributed computations in TensorFlow, but nothing had really dominated. Interestingly, the first author had just left DeepMind 1 year before this was published, so this was possibly his first project at Nvidia.

Megatron has the realization that, if you have a neural network like this:

\[Y = f(XW)\]

and you split $W = \begin{bmatrix} W_1 & W_2 \end{bmatrix}$, i.e. along the columns, then $Y = \begin{bmatrix}f(X W_1) & f(X W_2)\end{bmatrix}$, so you don’t need to do any synchronization to calculate $Y$. Consequently, the only points where you need synchronization (all-reduces) in the transformer are:

In the forward pass, to concatenate the model activations after the MLP block before adding dropout
In the backwards pass, at the start of the self-attention block.

Now, I strongly suspect this is what GPT-3 and Jurassic-1 both did, but neither went into detail about the specific parallelism models they used, other than to say (from GPT-3):

To train the larger models without running out of memory, we use a mixture of model parallelism within each matrix multiply and model parallelism across the layers of the network.

Presumably, this style of parallelism is what is meant by “model parallelism within each matrix multiply,” as I find it hard to imagine what else they could mean.

Gopher

Abstract

Gopher was a LLM trained by DeepMind. Interestingly, the lead author joined OpenAI shortly after it was published, along with a few of the coauthors. The architecture was the same as GPT-2, except:

They use RMSNorm (instead of layernorm)
Use relative positional encoding scheme from Transformer-XL (while GPT-* used a learned positional embedding)
They use SentencePiece (instead of BPE). This seems to be an Alphabet thing; many of the Alphabet papers use SentencePiece, while most of the non-Alphabet world uses BPE.

The paper was very interesting from a computational perspective, as they went into detail about how they trained their model and made it work:

They used optimizer state partitioning (ZeRO)
Megatron-style model parallelism
And rematerialization/gradient checkpointing to save memory.

These are all now the standard techniques used to train large models. To the best of my knowledge, Gopher was the first paper to put all of these together and release details about doing so publicly.

It’s interesting— often, big labs don’t include details for comeptitive reasons. Here, because DeepMind was (arguably) behind, they went into extensive detail. I think we’ll see this increase with LLM research from everyone that’s not OpenAI/Anthropic, as the others don’t live/die by the commercial success of their API, and have strong incentives to make it easier for **others** to train large models (and thereby commoditize their complements).

For the paper, DeepMind built a dataset called MassiveText, which was as follows:

Interestingly, this is much smaller than the dataset OpenAI used for GPT-3. GPT-3 had roughly 45TB of text, while MassiveText “only” had about 10.5TB.

They used this dataset to trained a large model on 300B tokens. The dataset consists of 2.343 trillion tokens, so this is only 12.8%. A much smaller subset. This is interesting to compare to the earlier GPTs, which, if you recall, used 100 epochs (so they saw each token in the dataset 100 times— while Gopher only saw 10% of their tokens once)!

The Gopher appendices have some great work; someone finally did ablations! They looked at:

Adafactor vs Adam, and found that Adafactor was much less stable
Lower-precision training, trying runs with float16, bfloat16, float32, RandRound, and using bfloat16 parameters with float32 in the optimiser state (rounding randomly). They found that using float32 parameters for optimisation updates only mitigated the performance loss, saving a substantial amount of memory.
Scaling context length; they show how performance increases as the context length increases. Improvements see diminishing returns, but consistently improve. Performance looks roughly proportionate to $\sqrt{n}$ (where $n$ is the context length).

It’s really nice to see detailed empirical work like this— it’s a welcome change from the other papers that failed to do this.

Chinchilla

Abstract

Chinchilla is an incredibly influential paper that established scaling laws. It’s one of my favorite papers from the last few years, as it actually does science in a way that physicists would agree with. One answer to “is something science” is to say, if you were to meet a historical scientist in your field, could you teach them something? And if you brought Chinchilla to researchers to, say, Radford et. al in 2017, it would advance their work by several years.

Chinchilla trained over 400 GPT-style transformers, ranging in size from 70M to 16B parameters, and fit the following equation (N is the number of parameters in the LM, and D is the number of tokens in the dataset):

\[\hat{L}(N, D) = E + \dfrac{A}{N^\alpha} + \dfrac{B}{D^\beta}\]

Choosing $A, B, E, \alpha, \beta$ to minimize

\[\sum \limits_{\text{Runs } i} \text{Huber}_{\delta=10^{-3}}(\log \hat{L}_i - \log L_i)\]

where the Huber loss is

\[\text{Huber}_\delta(x) = \begin{cases} \frac 1 2 a^2 & |a| \leq \delta,\\ \delta \cdot (|a| - \frac 1 2 \delta) & |a| > \delta \end{cases}\]

Here, we can think of E as the “irreducible loss” from the dataset, i.e. the loss if we trained an infinitely large model on an infinite stream of tokens. The authors find that the optimal model is (from nostalgebraist on into the implications of Chinchilla):

The implication here is that the model size & data size matter roughly equally, which is interesting, given how much attention & effort goes to scaling up the model, and how little attention is given to the dataset.

The authors then used this equation to determine the optimal model size for the Gopher compute budget, and trained it on more tokens— 1.4T tokens, 4.6x the number of tokens Gopher was trained on. This model, being 4x smaller, has a radically smaller memory footprint and is much faster/cheaper to sample from.

The Chinchilla paper has been highly influential. Almost every team that I’ve been talking to that is training a LLM right now talks about how they’re training a Chinchilla optimal model, which is remarkable given that basically everything in the LLM space changes every week.

The standard practice before Chinchilla was to train your model for 300B tokens, which is what GPT-3, Gopher, and Jurassic-1 all did. Chinchilla reveals how wasteful that was; basically, all of these papers made themselves more expensive to infer by training models that were too large.

Changes from Chinchilla (otherwise the same as Gopher):

AdamW instead of Adam (there’s an interesting footnote regarding the choice of optimizer: “a model trained with AdamW only passes the training performance of a model trained with Adam around 80% of the way through the cosine cycle, though the ending performance is notably better”)
Uses a modified SentencePiece tokenizer that is slightly different from Gopher (doesn’t apply NFKC normalisation)
They compute the forward + backward pass in bfloat16, but store a float32 copy of the weights in the optimizer state. They find that this is basically identically efficient to using float32 everywhere.

All of the changes are ablated extensively in the appendix. None of these are particularly novel.

PaLM

Speaking of training models that were too large- we have PaLM! Palm was really, really big. As far as I’m aware, it’s the largest dense language model trained to date, at 540B parameters, requiring 6144 TPUs to train on (this is 3 entire TPU pods, each consisting of 2048 TPUs). This is incredibly expensive! Probably only Google has the resources + infrastructure to do this.

… unfortunately, they were training PaLM at the same time chinchilla was being written. Very suboptimal.

Changes from GPT-3:

Multi-query attention. Shares K/V embeddings for each head, but has separate Q embeddings. Makes it much faster during inference time.
Uses parallel transformer blocks, which improves training time by 15%. As it was trained using 6144 TPU v4 chips for 1200 hours, the total training cost (at public prices) is between $1.45 to$3.22 per chip-hour, for a total of $10M to$22M. So this change saved $1.5M to$3M.
SwiGLU activations, rather than the GELU activation used by GPT-3
RoPE embeddings, rather than the learned embeddings
Shares the input-output embeddings
No bias vectors
SentencePiece with 256k tokens

So, a ton of changes! Again, a bunch of these are common, e.g. using the learned embeddings that GPT-3 had is very passé, and almost no one does it now.

LLaMa

Abstract

LLaMa combined a bunch of the best feartures from PaLM and Chinchilla:

Pre-normalize the input of each transformer sub-layer
RMSNorm, instead of LayerNorm, as done in Gopher
SwiGLU activation function from PaLM (but a dimension of $\frac{2}{3} 4d$ instead of $4d$, as in PaLM)
Uses rotary positional embeddings (RoPE) instead of the absolute positional embeddings, as done in PaLM
Uses AdamW, as done in Chinchilla

I think that LLaMa is the recipe to follow for the current SOTA in training large models.

Computational changes:

Uses efficient attention (Rabe & Staats, FlashAttention)
Gradient checkpointing
Interestingly, they appear to be using float32s everywhere (or at least, don’t say otherwise)

These are all similar to Gopher. The one obvious optimization they missed is to use lower precision, as Chinchilla did; I’m curious why they didn’t.

My one complaint is that I wish they would have trained the model for longer. The learning curve is very far from convergence! This paper is, in my mind, the shining example showing how well smaller models can do when trained well.

GPT-4

This is where I’d include information about GPT-4, if there was any. Unfortunately, the GPT-4 technical report contains almost no information:

GPT-4 is a Transformer-style model [33] pre-trained to predict the next token in a document, using both publicly available data (such as internet data) and data licensed from third-party providers. The model was then fine-tuned using Reinforcement Learning from Human Feedback (RLHF) [34]. Given both the competitive landscape and the safety implications of large-scale models like GPT-4, this report contains no further details about the architecture (including model size), hardware, training compute, dataset construction, training method, or similar.

As a result, I’m not going to talk about it, as there’s not much to say. Hopefully OpenAI changes their mind and releases some information about their model.²

Conclusion

This is it, as of March ‘23. I’m sure something new will come along and invalidate all of this.

What have I missed? Add comments on Substack or email me and I’ll update it.

Articles I’m reading:

Why didn’t DeepMind build GPT-3?

The paper itself doesn’t report the number of tokens, but OpenWebText, the open source reproduction, gets nine billion, using OpenAI’s tokenizer. ↩
To be clear, I highly doubt this will ever happen. ↩