GenZ News

Article View

Perplexity is allegedly scraping websites it's not supposed to, again

205d ago
Technology
Engadget
Technology and innovation
Cloudflare has reported that Perplexity's web crawlers are allegedly bypassing website restrictions. The report indicates that Perplexity's bots are engaging in "stealth crawling" by disguising their identity to circumvent robots.txt files and web application firewalls (WAFs). This allows Perplexity to scrape content from websites that have explicitly blocked its official bots, "PerplexityBot" and "Perplexity-User." Cloudflare's tests showed that even with blocked bots and WAF rules, Perplexity could still access and display content from restricted sites.
Source