Pricebot is Pricesearcher’s web crawling bot or “spider”. Crawling is the process by which Pricebot crawls websites and indexes pricing data from that site which is then inputted it into Pricesearcher.com. We use Pricebot for the purpose of collecting relevant information for analytics and price comparison.
Pricebot is an automated computer program which uses an algorithmic process to determine which information it obtains from the websites which are indexed.
Pricebot only obtains and indexes information on websites which is publicly available. Pricebot does not access any pages which are restricted or which require any passwords or permissions to enter.
We will not be collecting any personal data. We are only interested in related information about products, namely (product title, product description, price, product URL, image) and any other relevant information specific to the product.
If you want to prevent Pricebot from crawling content on your site, you have a number of options:
- you can use robots.txt to block access to files and directories on your server (see example below); or
- you can write to us directly at email@example.com and we will ensure that Pricebot does not crawl or access any information from your website.
We will not index anything you would like to remain private. All you have to do is tell us.
Example of how to use the official robots exclusion standard:
Pricebot will check your site’s robots.txt file first before making the crawl to understand what you have decided regarding allowing bots to access your website.
The crawler identifies itself by leaving a URL (pricesearcher.com/pricebot) so that you can see who we are.
The IP addresses used by Pricebot change from time to time. The best way to identify accesses by Pricebot is to use the user-agent (Pricebot). You can verify that a bot accessing your server really is Pricebot by using a reverse DNS lookup.
You can find detailed instructions on how to block web crawlers on Wikipedia and on the robots.txt page. Make sure that you don’t inadvertently block other crawlers that you don’t want to block, for example, the Googlebot.
If you have any questions of the use of Pricebot or how it may be accessing or indexing information from your website, please write to us at firstname.lastname@example.org.