It has no awareness of what it’s saying. It’s simply calculating the most probable next word in a typical sentence and spewing it out.
Neither of these things are true.
It does create world models (see the Othello-GPT papers, Chess-GPT replication, and the Max Tegmark world model papers).
And while it is trained on predicting the next token, it isn’t necessarily doing it from there on out purely based on “most probable” as your sentence suggests, such as using surface statistics.
Something like Othello-GPT, trained to predict the next move and only fed a bunch of moves, generated a virtual Othello board in its neural network and kept track of “my pieces” and “opponent pieces.”
Something like Othello-GPT, trained to predict the next move and only fed a bunch of moves, generated a virtual Othello board in its neural network and kept track of “my pieces” and “opponent pieces.”
AKA Othello-GPT chooses moves based on statistics.
Ofc it’s going to use a virtual board in this process. Why would a computer ever use a real one board?
Neither of these things are true.
It does create world models (see the Othello-GPT papers, Chess-GPT replication, and the Max Tegmark world model papers).
And while it is trained on predicting the next token, it isn’t necessarily doing it from there on out purely based on “most probable” as your sentence suggests, such as using surface statistics.
Something like Othello-GPT, trained to predict the next move and only fed a bunch of moves, generated a virtual Othello board in its neural network and kept track of “my pieces” and “opponent pieces.”
And that was a toy model.
AKA Othello-GPT chooses moves based on statistics.
Ofc it’s going to use a virtual board in this process. Why would a computer ever use a real one board?
There’s zero awareness here.