Our crawler

We read published feeds. The crawler identifies itself honestly on every request and never pretends to be a browser.

How to recognise it

It sends a User-Agent naming itself, this page and a contact address, so you can always tell our requests apart and reach a person about them:

TechNewsBot/1.0 (+https://news.thehawkbit.com/bot; mailto:support@thecodeserver.com)

How to control it

We follow robots.txt as specified in RFC 9309, including Crawl-delay, which is not part of that standard but which we honour anyway. A rule that disallows us is applied before any request is made, and a source that disallows us is switched off rather than retried.

We fetch one page at a time per host with at least a second between requests, use conditional requests so an unchanged feed costs you almost nothing, and back off when a server asks us to.

What we fetch, exactly

Your feed, and robots.txt before it. We never fetch the articles a feed links to: what we show is the headline and the extract your feed published, and the link sends the reader to you.

Images are the one other request. We fetch an image only from a publisher we already read, under the same user agent, and serve it re-encoded from our own origin so a reader’s browser never contacts you directly. That request is not checked against robots.txt today; tell us to stop and we will remove you.

We also read two public firehoses — Hacker News and Bluesky — to see which stories are being discussed. The Bluesky connection is a WebSocket, which carries no user agent; we read public posts there and fetch nothing as a result of them.

If you would rather we did not

Disallow us in robots.txt and we will stop, usually within the hour. If you would prefer to be removed entirely, write to the address in our user agent and we will remove you — see publishers.