market.ai, one of our product, 24x7 scans the websites across the internet. However we've implemented all the measure to keep our robots access to websites naturally & would not eat up websites bandwidth, but if you still wants our robots to prevent or limit. You may continue to read further.
As a webmaster, you have the ability to limit or completely prevent access to your website by our robots. To limit the access of a specific robot to your website, you can add a "robots.txt" file to your website's root directory. This file tells our robots and also search engine robots which pages or sections of your website they are allowed to crawl and index. To exclude a specific robot from crawling your website, you can add our user agent name to the "robots.txt" file and specify that it is not allowed to access any pages on your site. Here is an example of how to limit access to our robot called "market.ai-robo" in your "robots.txt" file:
In this example, the user agent name "market.ai-robo" is specified, and the "Disallow" directive indicates that this robot is not allowed to access any pages on the site (represented by the forward slash "/").
To completely prevent our robots from accessing your website, you can add a "noindex" meta tag to your website's HTML header. This tag tells robots not to index any pages on your website, effectively removing them from our indexes. Here is an example of how to add a "noindex" meta tag to your website's HTML header:
<meta name="robots" content="noindex">
Adding this meta tag will prevent our all robots from indexing any pages on your site. However, it is important to note that this approach will also prevent your site from appearing in search engine results as well as it will also stop market.ai to get fresh updates from your website which will result in misleading or oudated details being served for many users, you'll have to update your profiles each time manually by logging into the system which may not be a desirable option if you want to attract visitors & customers through organic methods.
You may contact us for more details.