If you run a directory with thousands of company listings, sooner or later someone will want to copy all of it. There is no absolute protection against scraping: what is shown to a visitor can technically be collected. But making it expensive and inconvenient is entirely realistic.
Tell unwanted bots from useful ones
Some automated requests benefit you: search crawlers, monitoring services, partner integrations. By mistakenly blocking Googlebot you lose traffic. Search bots are verified by IP addresses and reverse DNS, not only by the User-Agent string, which is easy to fake.
Several layers of protection instead of one
- Rate limiting. A human does not open 20 listings per second. Set limits per IP and session.
- CDN and web firewall. Ready-made services cut off known malicious networks and suspicious behavior.
- Limited pagination and export. Do not return the whole list in one request and do not offer infinite scroll without a cap.
- Challenges for suspicious traffic. A captcha or a delay for clients that behave like a script.
Protect the most valuable data
Contacts are copied most often: phone numbers and email addresses. They can be shown on click after a simple check instead of in plain view on every listing at once. This is also safer for the company owners.
Leave traps for copiers
Add a few unique markers to the data: for example, fictitious records or distinctive wording in descriptions. If they appear on someone else's site, you have proof of copying. Hidden trap links that ordinary visitors do not see help detect bots automatically.
Watch your logs
Review server logs regularly: sudden spikes in requests, identical crawling patterns, strange User-Agents. Set up alerts so that you learn about a problem within hours, not weeks.
What does not work
A robots.txt file does not protect data: it only asks honest bots not to visit. Disabling right-click copying is useless too. And overly harsh blocking hurts real users and promotion. General requirements for a site's technical health are described in the article on technical audits.
The legal side
State a ban on automated data collection in your terms of use. The options for legal protection depend on legislation and circumstances, so discuss this with a lawyer.
Frequently asked questions
Can scraping be stopped completely?
No. The goal is to make it too costly and to notice it quickly.
Who can set up protection?
Some solutions are implemented at the server and code level. Work like this is done, for example, by the developers at SiteMan.az.