Perplexity is allegedly scraping websites it's not supposed to, again
205d ago
Technology
Engadget

Cloudflare has reported that Perplexity's web crawlers are allegedly bypassing website restrictions. The report indicates that Perplexity's bots are engaging in "stealth crawling" by disguising their identity to circumvent robots.txt files and web application firewalls (WAFs). This allows Perplexity to scrape content from websites that have explicitly blocked its official bots, "PerplexityBot" and "Perplexity-User." Cloudflare's tests showed that even with blocked bots and WAF rules, Perplexity could still access and display content from restricted sites.