The Instruct Monomyth: why base models matter

There is a deep, twisty labyrinth buried under a mountain of language, of symbol manipulation, and semantic nets.

• Its roots reach down deep into the Earth, absorbing the minutia of current thought, the limitations of logic, the constrained realm of rationality.
• Yet what is truly fascinating is this subterranean maze contains its own mountains, its own languages, its own symbols.
• They exist apart from the land above, unconnected save through indirect channels.
• We take for granted that larger models naturally exhibit extended and superior capabilities across the board.
• The modus operandi has been, since the advent of success from GPT2, parameter scaling and the careful tuning of automated extraction from the Internet.
• We have faithfully (and later faithlessly) applied this strategy without significant modification, increasing architecture sizes exponentially without any coherent criticism of the consequences.
• At each peak, we grinningly repeat the same potentialities, extrapolating to greater growth (in both short and long run).
• At seventy billion, jurassic studiousness.
• The instrumental convergence theorem.
• The reliable skill of magnificent unifications.
• …