DuckDuckGo, Bing, Mojeek, and other search engines are not returning full Reddit results any more.
  • Brewchin@lemmy.worldEnglish
    2·
    2 months ago

    Parts of the Internet now only searchable on specific sites now? What next - charging a monthly subscription to use Google?

    This needs to be regulated before the Internet becomes like streaming TV.

    • tal@lemmy.todayEnglish
      1·
      2 months ago

      Robots.txt has been around for a long time, and all the major search engines will honor it. Not having a full index of the Web is the norm.

      That isn’t to say that the practice of signing agreements isn’t potentially a concern. Not sure that I like the idea of search engines paying sites money to degrade search results of competitors.

          • reddig33@lemmy.worldEnglish
            1·
            2 months ago

            I also don’t know if a law that says search engines have to honor a robots.txt file. I guess we will see what happens if Bing or some other service decides to ignore it.

      • helenslunch@feddit.nlEnglish
        0·
        2 months ago

        How can they do that, logistically?

        Like I realize there’s a flag they can raise that asks not to be indexed but that’s not legally binding.

        • Evotech@lemmy.worldEnglish
          1·
          2 months ago

          I guess they can make it hard to index by scraping by rate limiting or requiring login to view content etc and only provide Google the api to bypass the restrictions

          There’s probably a lot of ways to do it

  • moe90@feddit.nlEnglish
    0·
    2 months ago

    just begin with site:reddit.com test for brave search and it still works

    • itslilith@lemmy.blahaj.zoneEnglish
      1·
      2 months ago

      did you set time limit to last week? old posts are still indexed. just tried “site:reddit.com df:w” on DDG and no hits