Lately, I have been writing on my newsletter.
There's a Google PDF with a bunch of useful latency numbers that software engineers should know. I asked ChatGPT to make it into a markdown file for me.
Most ML code is Python. This is surprising to many performance oriented engineers coming from non-ML communities. Python is, notably slow and has the GIL, which forces it to only execute a single thread at a time. The GIL primarily exists to make CPython’s memory management thread-safe. It has also become a load-bearing part of Python's API, as, in a classic example of Hyrum's Law, much Python code relies on the thread safety that it has created.
[1]: Funnily enough, if you google "when does Ray release the GIL?" one of the top results is from the Beyblade wiki.
For Olmo 3, I was put in charge of our post-training infrastucture. We made the decision to move from a synchronous RL setup to an asynchronous one to enable us to scale. In doing that work, I was fortunate enough to find a series of optimizations which made our RL setup 4x faster. As we used roughly 250k H100 hours running RL on Olmo 3, these optimizations saved us approximately 750k H100 hours (~$1.5M) at current market prices. These changes were detailed in the paper, but I wanted to write more about them here.
The Bitter Lesson is an excellent essay which is overwhelmingly misunderstood. The point of the bitter lesson is that, over time, methods which scale with compute will outperform methods that do not.
In the recent wave of research studying reasoning models, by which we means models like O1 which are able to use long streams of tokens to "think" and thereby generate better results, MCTS has been discussed a lot as a potentially useful tool. However, some papers, like the DeepSeek R1 paper, have tried MCTS without any success.
A few people have asked me for RESP advice, so here is my generic answer, with the disclaimer that I'm not a financial advisor.
I consistently run into the same issue when install Docker on a new Ubuntu VM.
I recently gave a talk at Amii about the history of GPT models.
If you want to read more of my writing, I have a Substack.
If you want to read more of my writing, I have a Substack. Articles will be posted simultaneously to both places.
There's a Substack version of this post, if you prefer that over my amateurish artisan HTML.
I’m going to try to write a weekly summary of the most interesting papers I’ve read that week. I’d love to hear what papers you’ve been reading, if you agree/disagree about my conclusions for each paper, and/or suggestions for what papers I should read next!
I regularly reference the “s-curve”, or sigmoid, as a metaphor for progress. Here, I explain what I mean, so that I can just link to this post.
I have a Substack if you want to be notified when I write.
I realized over the weekend that, unfortunately, I didn't know how back-propagation actually works (I just relied on JAX to do it for me).
I have a file in my home folder that contains Bash oneliners that I use regularly (I'm a huge nerd, naturally). I found most of them elsewhere online; I wrote very few of these from scratch.
I'm currently reading Softwar, a book about Oracle's rise. The book is brilliant, and it descibes at length Larry Ellison's sales process. There was a passage describing a meeting that Larry had that explains far more about enterprise sales than it should:
I've had a few people ask me how to do model selection correctly. Here's a
minimal example with sklearn in Python.
I'm working through the Algorithm Design Manual to improve the efficiency of my coding.
I've been trying to parse JSON files with C++, and I've found a distinct lack of full examples on how to do so. Specifically, I've struggled to find the proper commands to actually compile the code. For future reference (and to help any beginners out), here's a full example of how to use JSONcpp in your code (N.B. You're supposed to enter all of the following code in your terminal).
I'm working through TSA, and I noticed that some of my classmates are struggling to understand the difference between an ARIMA process, an AR process, and a MA process, not to mention seasonal version of the above.
I've been studying time series through TSA. The book presents a structured approach to time series analysis, and covers the material fairly well; I was impressed with the description of what a partial autocorrelation function (PACF) is, as the book explained it more intuitively than the lecture notes did. I did find the description of how to actually solve for the PACF a bit confusing, so I wrote my own explanation.