DotsjoinedBot

The crawler behind Dotsjoined, a research project that indexes published news to show how reported events relate over time.

How to recognise it

Every request carries this User-Agent:

DotsjoinedBot/<version> (+https://dotsjoined.com/bot)

Requests currently come from 165.22.212.67 (DigitalOcean, Bangalore).

How it behaves

• It checks your robots.txt rules before every request, including each redirect, and obeys the rules for DotsjoinedBot (or *). It re-reads robots.txt at least once every 24 hours. If robots.txt cannot be reached because of a server error, it does not fetch.
• It honours Crawl-delay, and otherwise sends at most one request every 5 seconds to a site.
• It uses If-None-Match and If-Modified-Since, so unchanged pages are not downloaded again.
• It fetches only public pages. It does not log in, solve challenges or get around paywalls.
• It stops reading a response after 10 MB and follows at most 5 redirects.

What it collects

News articles your site already publishes in its feeds or news sitemaps, or that public news indexes list. It stores a compressed copy of each page and the text and details read from it: headline, date, byline, section and links. Copies are kept only as long as our review of your terms allows.

Dotsjoined shows short quotations of at most 25 words, always with a link to the original article.

How to block it

Add this to your robots.txt; it takes effect within 24 hours:

User-agent: DotsjoinedBot
Disallow: /

How to opt out or contact us

Write to bot@dotsjoined.com. When a publisher asks us to stop, we stop collecting from the site and delete the copies of its pages that we stored. Quotations of at most 25 words already used as cited evidence, each with its link, are kept.