Andrej Karpathy

The GOAT of ML

In my last post I mentioned Zero to Hero by Andrej Karpathy. I just watched Deep Dive into LLMs by him, where he does an excellent job of describing the three stages of model training: pre-training, post-training supervised fine-tuning, and post-training reinforcement learning.

I won’t try to explain the contents, you should just go and watch the video. It’s highly recommended! Here’s a copy of the image Karpathy created using Excalidraw (my favorite diagramming tool for a while now) during the video.

Here are a couple of other interesting links I fished out of the video:

• FineWeb, an alternative to the OpenWeb dataset I’ve used to train a model from scratch
• TikTokenizer, an incredible tool that translates text into the token vocabulary used by different models
• LLMs visualizations by a kiwi software engineer
• LM Arena, a tool to rank (based on user votes) models across different tasks
• Andrej Karpathy’s website

Andrej Karpathy joins Moriba Jah and Dominic Tarr in the ranks of folks I found online somewhat randomly as I dove into personal projects. I came across Dominic when I started looking into Secure Scuttlebut (which I used when building Archerlog) and decentralized applications/platforms. And Moriba is someone I came to look up to as I learned about satellite tracking and the players in the field, which include his company Privateer. It’s amazing to me that these folks are so generous with their knowledge and code - I can only hope some of the topics I’ve been spending my time on and chronicling in this blog are as useful to someone in the future.