SubmitiumBot

How to recognise it

SubmitiumBot sends this user agent with every request:

Mozilla/5.0 (compatible; SubmitiumBot/1.0; +https://submitium.com/bot)

It reads raw HTML and other public files. It does not run JavaScript, fill in forms, sign in or create accounts: the submissions themselves are made by people on our team.

What it fetches

Submitium submits products to online directories by hand and follows each submission until the listing is live. SubmitiumBot does the following:

• On directories we submit to: it looks for our customers' listings (listing pages, sitemaps, search results), checks that a published listing still links to the customer's site, reads the directory's icon, and checks about once a week that the directory is still online.
• On our customers' sites: it reads the site a customer adds, to draft their listing, and their homepage, to check that their badge wall is in place. Customers ask us to do this.
• Confirmation links: when a directory emails a confirmation link for a submission, it opens that link, and only if it points to the directory's own domain.

We record the address, time and response status of each request, and keep these records for 180 days.

How often

• On directories, and when it reads a site a customer adds, SubmitiumBot first reads the site's robots.txt (kept for up to 24 hours) and follows the rules of the group for SubmitiumBot, or of the * group when there is none.
• It waits at least 6 seconds between two requests to the same host, and longer when robots.txt sets a Crawl-delay (honoured up to 60 seconds).
• For a submission waiting for its listing, it looks every 6 hours for the first 3 days, then once a day until day 30, then once a week, and stops after 90 days. A live listing's link is checked about once a month.
• The badge wall check fetches only the homepage of a customer who installed the wall, at their request or while their order waits for it.

How to block it

Add these lines to your robots.txt to keep SubmitiumBot out of your whole site:

User-agent: SubmitiumBot
Disallow: /

You can also disallow only some paths. Changes reach us within 24 hours. If you run a directory and block SubmitiumBot, we stop fetching your pages and follow our customers' submissions by email and by hand instead.

Contact

If SubmitiumBot causes you any trouble, or you want us to stop visiting your site, write to [email protected] with your domain, and we will act on it within one business day. How we handle the data involved is explained in our privacy policy.