You're seeing our crawler in your server logs? Great. We're building a search engine that doesn't spy on people. Here's everything you need to know.
Alfred Search is a privacy-first, AI-native search engine built by GoSiteMe. Unlike Google, Bing, or even DuckDuckGo, we build our own independent web index using AlfredSearchBot — and we do it ethically.
No tracking. No cookies. No fingerprinting. No ad profiles. No filter bubbles. Just answers. We believe the web deserves a search engine that works for people, not on people.
AlfredSearchBot crawls publicly accessible web pages to build Alfred Search's independent web index. Here's our user agent string:
We fully obey your robots.txt directives. Block us with User-agent: AlfredSearchBot and we'll never visit again.
Maximum 1 request per second per domain. We also honor your Crawl-delay directive. We're a polite guest.
We crawl to a maximum depth of 3 links from seed pages. We won't spider your entire site unless you invite us to.
We only request text/html pages. No images, PDFs, videos, or binaries are downloaded. Minimal bandwidth impact.
Add these directives to your robots.txt file to control AlfredSearchBot:
# Block AlfredSearchBot entirely: User-agent: AlfredSearchBot Disallow: / # Allow crawling but set a crawl delay: User-agent: AlfredSearchBot Crawl-delay: 5 Allow: / # Allow crawling except private areas: User-agent: AlfredSearchBot Disallow: /admin/ Disallow: /private/ Allow: /
| Feature | Bing | DuckDuckGo | Alfred Search | |
|---|---|---|---|---|
| Zero tracking | * | |||
| Own web index | (uses Bing) | |||
| AI instant answers | ||||
| No ad profiles | ||||
| Self-hostable | ||||
| Transparent ranking | ||||
| Deep research mode | ||||
| Open-source friendly | Partial |
* DuckDuckGo has improved privacy but still relies on Microsoft's Bing index and shows tracking-based ads.
Want your pages in Alfred Search faster? Contact us to add your domain to our priority crawl queue. It's free.
Integrate Alfred Search into your own apps. Our REST API is simple, powerful, and comes with SDKs for Node.js, Python, and PHP.
Run Alfred Search under your own brand. Perfect for enterprises, schools, and organizations that need private, internal search.
Search the web without being watched. It's that simple.
Bot Name: AlfredSearchBot
Version: 1.0
Operator: GoSiteMe (gositeme.com)
Purpose: Web indexing for Alfred Search
Crawl rate: ≤1 request/second per domain (respects Crawl-delay)
Max depth: 3 links from seed pages
Content types: text/html only
Max page size: 2MB
Robots compliance: Full (User-agent, Disallow, Allow, Crawl-delay)
Contact: crawler@gositeme.com
Someone from somewhere
just launched website.com
Just now