Letters and Words
How Little Brain started learning patterns in language.
The experiment
What Are We Trying to Discover?
I've spent time using AI, building projects with it and discovering just how entertaining these systems can be.
But eventually I started wondering something.
What if I tried building my own AI from scratch?
Not giving an existing AI a new personality. Not simply connecting to ChatGPT or another ready-made model.
Actually starting with a little model that knows almost nothing and seeing what happens as I teach it.
That's how Little Brain began.
Meet the experiment
He's Learning. I'm Learning.
Things Get Weird.
Right now, Little Brain is learning to put words together.
Sometimes he produces perfectly sensible sentences.
Other times, he invents something like a hat apple.
I still haven't worked out what one of those is.
“A bird brings a hat apple the pond.”
Hat apple remains unexplained.
The open question
Can Little Brain Develop His Own Sense of Humour?
That's what I want to find out.
I'm not trying to give him a ready-made comedy personality.
I want to explore whether his own distinctive style of humour could develop through learning and experimentation.
Will it happen?
I genuinely don't know.
And that's why this is an experiment.
The learning journey
What We're Teaching Him
We look at the small steps and keep track of what happens as we go.
Putting Sentences Together
How he's beginning to generate complete sentences.
Learning What Comes Next
How we're experimenting with training methods and discovering what helps or doesn't.
Learning patterns in words does not mean Little Brain understands language the way a person does.
The honest record · Days 1–2
A little brain, one step at a time.
Here’s how the experiments moved from character patterns to short sentences. A lower prediction score is useful, but it does not automatically mean the writing sounds better.
Model 1, Model 2 and Model 2B
Model 1 was the first neural character model. More training passages became Model 2; a smaller learning rate produced Model 2B. A separate word-level experiment exposed a punctuation bug, which was fixed, though the reviewed passages remained incoherent.
Learning to finish a sentence
Early word-based models still produced fragments. Small, controlled sentence examples gave the first valid completions, then a 100-sentence model completed all 20 held-out prompts grammatically; 19 were judged plausible.
More sentences, repeating patterns
“The toy” appeared in 18 of 30 greedy completions. Balancing that pattern shifted repetition to “fish”. A broader balancing test produced more varied endings, but mixed sentence quality.
A measured improvement
V3 Expanded improved the first matched comparison with V2. A later temperature check found 50/50 grammatical samples at 0.6, while everyday plausibility remained around 34–35 of 50. Grammar and common sense are different checks.
The misplaced “home” pattern
Some added examples encouraged endings such as “a bird finds a toy home”. Removing 82 home-related examples improved grammar in two seeds, but the changed dataset did not give consistent plausibility or validation gains. V4 was not promoted.
V2 and V3 Expanded
What changed in the first matched comparison?
V2 used 300 sentences. V3 Expanded used 900.
| Measure | V2 · 300 | V3 Expanded · 900 |
|---|---|---|
| Best validation loss lower is better | 1.8752 | 1.8076 |
| Test loss lower is better | 1.9288 | 1.8090 |
| Next-word accuracy top prediction correct | 35.98% | 39.15% |
| Everyday-plausible blank samples 50 samples · temperature 0.8 | 26 / 50 | 34 / 50 |
In this comparison, the test set was checked after the checkpoint was selected. The examples used held-out subject–verb groups, not a broad test of fresh language skills. These results describe this experiment only.
Prediction score
How well the model assigns probabilities to the next word. Lower loss is better when the comparison is matched.
Grammar
Whether the words follow the sentence patterns used in the review.
Everyday plausibility
Whether people judged a small sample to sound sensible in ordinary life. None of these checks proves human-like understanding.
Follow the adventure
Want to See What Happens Next?
You can follow Little Brain's progress as it happens. Watch him learn, see the mistakes, discover the surprises and follow my journey as I learn to build him.