AuraScout is a small, slow, non-commercial research crawler. Here is what it is, why it read your page, and how to make it go away.
Ordinary search gives you the ten pages an algorithm thinks are most popular. That works well for common questions and badly for everything else. If the thing you want is on a personal blog from 2011, written in a language you did not think to search in, or four clicks deep inside a site nobody links to, you will not find it. It is there. You just cannot get to it.
Deep search is the attempt to fix that by spending time instead of relying on ranking. Rather than returning a list, AuraScout starts from a question and works outward: it runs searches, reads what comes back, follows the links it finds worth following, and keeps going for as long as the trail stays warm. A language model reads each page and decides two things — is this an answer, and does this lead somewhere. Pages that lead somewhere get followed even when they say nothing themselves. Someone's link page is worthless as content and invaluable as a signpost.
It also widens the question as it learns. Ask it about a Dutch biscuit and it will work out that it should be searching in Dutch, and that a neighbouring recipe shares the same small community of bakers. That is usually where the good material turns out to be, and it is the part a human would need weeks to do by hand.
This is not a startup and there is no product. AuraScout is one piece of a personal project: an AI character called Aura, who runs as a VTuber — a streaming persona that talks with people in chat and is expected to know things. The scout is how she goes and finds out.
General-purpose search turned out to be the weak link. Ask an AI presenter something niche and it will either say something plausible and wrong, or hand back the same first-page results anyone could have found. Neither is interesting. The idea behind AuraScout is that if the thing can genuinely go and look — properly, patiently, across languages, into the parts of the web that rank badly — then it has something worth saying.
It runs on hardware in a cupboard, it is written by one person, and the results are not published, sold, or syndicated anywhere.
AuraScout/0.1 (research crawler; +https://aurascout.projectaura.net) — never disguised as a browser.AuraScout reads your robots.txt before it reads anything else, and re-reads it every 24 hours. A rule that names AuraScout specifically is treated as absolute: the URL is never requested, and never even enters the queue.
Generic User-agent: * rules are handled differently, and it is fairer to say so plainly than to imply otherwise. AuraScout reads them as strong guidance about which paths are worth its time — in practice they are usually pointing at search endpoints, tag archives, and calendar pages, which are exactly the crawl traps it wants to avoid anyway. It does not treat a blanket disallow aimed at every bot on the internet as a personal instruction. If you want one, write one, and it will be followed.
It does not attempt to bypass anything. It does not solve captchas, evade rate limits, rotate addresses, or spoof its identity to get past a block. When it hits a wall it stops and puts the page in a queue for a human to look at, which is usually the point at which the human decides it was not that important.
Add this to your robots.txt. It is honoured completely — a rule naming AuraScout is a hard stop, not a preference — and takes effect within 24 hours, once the cached copy expires.
User-agent: AuraScout Disallow: /
Or exclude part of your site:
User-agent: AuraScout Disallow: /members/ Disallow: /shop/
If you would rather not touch robots.txt, blocking the user agent at your web server works just as well. And if you would prefer to just tell a person, the address below reaches one — you do not have to explain why.
If you would rather answer with a status code than edit a file, these are the ones that mean something. Most are read per site, not per page, so a single response is enough to change how AuraScout treats everything else you host.
| You return | AuraScout does |
|---|---|
| 403 | Stops completely, for the whole site, permanently — after one check. A 403 to the plain HTTP client triggers exactly one retry with a real browser, sending the identical User-Agent. If that is refused too, AuraScout stops visiting the whole site for good and never tries again. See the note below on why the retry exists. |
| 401 | Treated as a wall. The page is abandoned and queued for a human to decide about. It never attempts to authenticate. |
| 451 | Treated as a wall, same as 401. No retry. |
| 429 | Slows down: the gap between requests doubles, up to two minutes, and the site is set aside for a while before anything is tried again. Keep returning it and it keeps halving its rate. Use this if the problem is pace rather than presence. |
| 503 | Handled identically to 429 — backs off and comes back later. The right answer if you are simply busy and do not want it gone for good. |
| 404 / 410 | That page is dropped and not requested again. The rest of your site is unaffected. |
| 301 / 302 | Followed, up to five hops. A redirect into a login page reads as a wall and stops there. |
| 200 + captcha or login page | Detected and abandoned. It does not solve captchas or try to get past interstitials; the page goes to a human queue instead. |
The distinction worth knowing: 429 means slow down, 403 means go away. If it is being a nuisance but you do not object in principle, 429 is the kinder signal and it will respect it. If you want it gone, 403 is unambiguous.
Being straight about this, because it is the one place the behaviour is less simple than "it stops".
A great many 403s are not decisions about AuraScout at all. Bot-detection at the CDN layer fingerprints the TLS handshake and the shape of the HTTP request, and refuses any client that is not a browser — regardless of what it says it is, whether it read robots.txt, or how politely it is behaving. Sites running that default return 403 to a well-behaved crawler and 200 to a person, without anyone having chosen either outcome.
So when the HTTP client is refused, AuraScout retries the page once through a real browser. Two things about that retry matter:
A rule in robots.txt that names AuraScout skips this entirely. That is an answer addressed to this crawler, and checking it with a browser would be going around it, so the site is dropped without any retry at all.
If AuraScout has been a nuisance, got something wrong, hit your server harder than it should have, or you simply want it gone, drop a line to . It is one person and one machine, so a reply is likely and a fix is easy.
Plenty of people have good reasons to be tired of crawlers turning up uninvited, and that is a fair position to hold. This page exists so that the answer to "what is this thing" takes ten seconds rather than an afternoon.