Question

How does a machine learn a game when nobody teaches it the rules?

I rewatched the DeepMind documentary about AlphaGo. It follows the AI system that beat the world’s best player at Go, a game long thought too intuitive for machines. Before they created AlphaGo, they built a prototype to prove the concept. an Agent learning to play Atari Breakout from nothing but the raw screen and the score. No rules, no strategy, no idea what the paddle was for. Left running for a long period, it found a tunnelling trick its own creators had not thought of. I found this interesting and thought about building something similar to understand how it works.

A few days ago my internet dropped and Chrome offered me its small offline dinosaur to pass the time. I played it longer than I should admit. I got the idea that it might be similar to Breakout. One agent, one tiny world, three things it can do, and a score that only climbs if it survives. A clean agent and environment loop, simple enough to build from scratch and watch learn. So I rebuilt the Chrome dino game and used a reinforcement learning approach to help it play, in the browser. The point was to understand it through implementation, one piece at a time.

Data

In conventional supervised learning the data is the starting point. You get a dataset, and the goal is to train a model to fit it. To find a predictable pattern within the data itself. Reinforcement learning does not work that way. There is no dataset. The agent generates its own as it plays, and that data changes every time the agent changes. The data can be thought of as a flow of moments. Each moment captures four things: the state the dino was in, the action it took, the reward it received, and the state that followed.

The state is captured as seven numbers fed as input to the network: distance to the next obstacle, whether that obstacle is a cactus or a bird, its height, the dino’s own height and vertical speed, the current game speed, and whether it is standing on the ground. The reward is simple. A small plus for staying alive each step, a plus one for clearing an obstacle, a minus one for crashing.

The game keeps a running memory of the most recent thirty thousand moments, called a replay buffer. As the dino improves, the moments it stores change, and so does the data it learns from. In reinforcement learning, the model writes its own training set.

Algorithm

The algorithm is a Deep Q-Network, the same method DeepMind used on Atari. The only real difference is the eyes. They fed raw pixels through a convolutional network and let it discover the ball on its own. I hand the dino seven numbers instead. That keeps the learning fast enough to watch live in a browser, and it separates the reinforcement learning from the computer vision. The mechanism underneath is identical.

dqn

Underneath, the learning half works like training any neural network. The network makes a prediction, you compare it to a target, and the gap is the error it tries to shrink. The tricky part is where the target comes from. In most machine learning it’s the labelled answer in your data, but a DQN has none, so it builds its own from the reward it just earned plus a guess of how good the next state is. This is the Bellman target: r + gamma · max Q(s’, a’). I don’t fully follow the math yet, but the idea I do follow is that the network learns partly from its own guesses. The diagram above shows the other half: the agent playing to produce these moments, and the buffer that connects the two.

The network

At the centre of both loops is a tiny neural network. Seven inputs, one hidden layer of 64 units, three outputs. The three outputs are its guess of the future reward for each move: run, jump, duck. To play, it reads those three numbers and picks the biggest.

DINO Q-NETWORK ARCHITECTURE obstacle distance obstacle type obstacle height dino height dino velocity game speed on ground? RUN Q 4.55 JUMP Q 4.10 DUCK Q 3.98 INPUT 7 nodes HIDDEN LAYER 64 nodes, ReLU (8 shown) OUTPUT 3 nodes, linear outside the network argmax pick max Q ACTION: RUN 2% random while training The 3 outputs are Q-values (predicted future reward per action), not probabilities. There is no softmax.
Acting: explore, then exploit

When the agent plays, it does not always trust the network. Early on it acts almost at random: this is the exploration rate, epsilon, and it starts near one. As it plays, epsilon decays toward 0.02, so it leans more and more on what the network has learned. Every moment it lives, chosen by chance or by the network, is stored in the replay buffer for later.

Experience replay and the target network

The learning happens separately, on a random batch pulled back out of the buffer. Two tricks make this work, and I only understood why once I saw what broke without them. The first is that random sampling itself, called experience replay. If the network learned from each moment in the order it happened, back-to-back frames would look almost identical and the training would wobble. Pulling a random mix breaks that. The second is the target network, a frozen copy of the main network that works out the target to aim at. It gets updated to match the live network only every so often. Without it, the network is chasing a target that moves every time it changes, and the whole thing falls apart. Repeat the loop enough times and guesses that started as noise slowly settle into something that plays.

Compute

Here is the part I like most. None of this runs in a data centre. It runs in a browser tab, on your own CPU. No GPU, no server, no cloud. The same kind of agent at DeepMind trained for days on powerful hardware. The dino gets good in a minute or two, because the problem is small and the state is seven numbers instead of a whole screen. My first run reached almost 5 million training steps across nearly 11,000 episodes in less than ten minutes, all on a laptop, in the browser.

Evaluation

Approach

To test whether it was actually learning, I set up a small experiment. I ran it for 10,000 episodes with the difficulty ramping up over time. Every 1,000 episodes I saved a snapshot of the network, giving me ten checkpoints. Then I tested each one frozen, with exploration off, over 100 episodes, so every checkpoint faced the same fair test

Results

The scores had some variance. ep3000 cleared 55.9 obstacles, the middle collapsed to near zero, then ep9000 spiked to its best at 82.9 before the final checkpoint fell to 0.0. The best model wasn’t the last one, it was ep9000. This up-and-down is a known problem with plain DQN, sometimes called catastrophic forgetting: the network finds a good way to play, then drifts off it, again and again. Saving a snapshot every 1,000 episodes is what let me catch the good one instead of shipping the final weights, which ran straight into the first cactus. The skill lives in the weights at one moment, not in any lasting understanding, and the checkpoints are how I catch it at its best.

Deployment

Deployment was simple. The compute runs in the browser, as a single static HTML page.

Try it live: train your own dino

The page is interactive. You can start and pause training, speed it up, take the controls yourself to feel the task the network is learning, snapshot weights, run the frozen evaluation, and scroll the learning curve back to the first episode. The aim is to watch the dino learn and improve over time.

Conclusion

I set out to understand how reinforcement learning works through implementation. This project gave me a useful way to think about how an agent learns inside a virtual environment. Given some constraints and a clear goal, the agent learns by taking action, getting feedback as rewards, and updating a neural network. It learned its game the way I’m trying to learn this field. Not by being told the rules, but by playing, failing, and slowly turning noise into something that works.

References