• 1 Post
  • 41 Comments
Joined 1 year ago
cake
Cake day: June 11th, 2023

help-circle







  • I don’t even think it’s correct to say it’s querying anything, in the sense of a database. An LLM predicts the next token with no regard for the truth (there’s no sense of factual truth during training to penalize it, since that’s a very hard thing to measure).

    Keep in mind that the same characteristic that allows it to learn the language also allows it to sort of come up with facts, it’s just a statistical distribution based on the whole context, which needs a bit randomness so it can be “creative.” So the ability to come up with facts isn’t something LLMs were designed to do, it’s just something we noticed that happens as it learns the language.

    So it learned from a specific dataset, but the measure of whether it will learn any information depends on how well represented it is in that dataset. Information that appears repeatedly in the web is quite easy for it to answer as it was reinforced during training. Information that doesn’t show up much is just not gonna be learned consistently.[1]

    [1] https://youtu.be/dDUC-LqVrPU








  • This is only true if you ignore all the other variables. Which is, let’s say, another company hiring writers and now they’ll grow their market share in comparison with the shitty AI articles company.

    Amazon has a lot of competition in Brazil and the more they make their service worse, the better for the competition. But so far Amazon only raised the bar (with fast deliveries), making all other companies improve their own services.