Note: Read Custom Robots before continuing this article.

‘Custom robots’ is a feature to crawl website with editable robots.txt to understand the behavior of actual robots.txt. If ‘Custom Robots’ is enabled; the BeeBot will crawl the website as per the directives stated in custom robots. But this behavior can be different with different user agents. For instance if the selected user agent from user ‘agent section’ under ‘spider’ menu is BeeBot and in custom robots the directives are for GoogleBot then crawler will bot read any robots. See the below images for better understandings.

custom robots with user agents

In above snapshot the selected user agent is ‘BeeBot’ and in ‘custom robots’ is defined for ‘ABCBot’. In this case no robots will be followed.

Different Cases with Custom Robots:

Case 1:

User agent Selected = *;
User Agent in Custom Robots = any string

Result:

As ‘*’ is a wild card operator; defined custom robots will be followed in any case.

Case 2:

User Agent Selected = BeeBot
Custom Robots

User-agent: Some-Bot
Disallow: /user-guide

User-Agent: Some-Other-Bot
Disallow: /some-other-user-guide

User-Agent: *
Disallow: /no-directory

Result:

As the selected user agent ‘BeeBot’ do not match with first and second section of robots; thus last section will be followed for the website i.e.

User-Agent: *
Disallow: /no-directory

Other Resources


About Ahmad Ali

Ahmad is the co-founder and CEO at Webbee Inc. He’s been working as a digital marketer for past few years and has worked with some notables names across different industries. He is also the creator of Webbee SEO spider, one of the most advanced SEO spider tool on the internet.

Leave a Comment