Bots, crawlers and your resource limits

Not all the visitors using your resources are people. Bots and crawlers - automated programs that browse the web - can generate a surprising amount of traffic, and sometimes they are the hidden reason a site keeps hitting its DirhamGuard limits.

Some bots are welcome. Search-engine crawlers like Google's need to visit your pages to list you in results. But others are not: content scrapers, spam bots probing your forms, and aggressive crawlers that request thousands of pages far faster than any human would. Each request uses a little CPU and memory, and in volume that adds up quickly.

Signs bot traffic is affecting you:

• Usage spikes that do not match your real visitor numbers.
• Sustained load at odd hours when few customers are online.
• Lots of requests to login pages, search URLs or XML files in your logs.

What you can do:

• Add a sensible robots.txt to guide well-behaved crawlers and slow their crawl rate.
• Use a security or firewall plugin to block known bad bots and rate-limit abusive ones.
• Protect login and form pages with basic anti-spam measures.
• Put your site behind a CDN or caching layer so bots hit ready-made pages instead of triggering full rebuilds.

If you suspect a bot is hammering your site but cannot pin it down, send us the times of the spikes and we will help review your access logs. Our 24/7 UAE team is on WhatsApp +971 58 553 6767 or support@dirhamaday.ae.