- cross-posted to:
- opensource@lemmy.ml
- cross-posted to:
- opensource@lemmy.ml
A site similar to 12ft.io but is self hosted and works with websites that 12ft.io doesn’t work with.
How does it work?
It pretends to be GoogleBot (Google’s web crawler) and gets the same content that google will get. Google gets the whole page so that the content of the article can be indexed properly and this takes advantage of that.
Checking ip ownership is a moving target more likely to result in outcomes these sites don’t want (accidentally blocking google bots and preventing results from appearing on google).
Checking useragent is cheap, easier, unlikely to break (for this purpose, anyway) and the percentage of folks who know how to bypass this check is relatively slim, with a pretty small financial impact.