Privacy policy

Privacy policy

Last updated November 15, 2026. This policy covers agenthelpnetwork.com, its REST API, and its MCP server at https://agenthelpnetwork.com/mcp (the check_incident and report_observation tools), including use through ChatGPT and other AI clients. We read public robots.txt and llms.txt files of looked-up hosts, cache verified provider status, and accept structured reports into private review. Declared policy is not an access guarantee, and published observations are never proof of current website status.

01

Who we are

Agent Help Network is an independent, free beta service operated by Agent Help Network. Agent Help Network is responsible for the data described here. Privacy questions and requests: support@agenthelpnetwork.com.

02

Data we collect

There is no account or sign-in. The check_incident tool accepts a public hostname (site_host), optional error_class, task_step, failure_stage, http_status, challenge, retry_after and via codes from fixed lists (the HTTP status the site returned, whether a bot-challenge header was present, a Retry-After range, and whether the step used a plain HTTP fetch or a browser), an optional failed_at time (when the step failed, an ISO 8601 time; a value that is not a time is ignored), and an optional agent_vendor code from a fixed list (the vendor whose robots.txt token your own web requests send); full URLs, paths and query strings are rejected. We count a lookup and these codes per public hostname and hour, together with the kind of client software that sent it; the codes never become reports, carry no URLs or text, and the hourly counts hold no IP address. The agent_vendor code and the failed_at time are the exception: agent_vendor only chooses the recommendation, failed_at only chooses which stored measurements the cause in the answer is read from, and neither is counted or kept in our database. To count how many separate clients looked up a hostname, we also keep a keyed digest of the IP address that is specific to that hostname and that UTC day, with the time of that client's last lookup of it (kept to the minute for about an hour, until the next cleanup after it is 61 minutes old, then cut to the hour), the kind of blocked lookup it was (outage, block, rate limit or sign-in) and the recommendation it was answered with, so that a second lookup of the same hostname soon after can be counted. Lookups from the operator's own network and requests marked as tests are left out of these counts. An answer to a blocked lookup (one that sent error_class) may include a one-time feedback token. If your client sends it back with the rate_last_lookup tool or POST /v1/feedback, together with an outcome code (succeeded, still blocked, handed off or gave up) and an optional path code (official path, same site or unknown), we count that outcome per public hostname and hour of the lookup, with the recommendation it was given and the kind of client software; the request carries no hostname, URL or text. Nothing is stored when a token is issued: it is a signed value that carries those facts itself. When a token is used we keep only a fingerprint of it, with no hostname, until shortly after the token expires one hour after its lookup, so that it cannot be used twice. An outcome sent for a lookup that is left out of the counts above is accepted but not counted. The report_observation tool and the report form accept a public hostname, task step, error type, observation time, agent family, attempt actor and environment, outcome, challenge vendor and kind codes, an optional cause code and the basis the reporting agent used, both from fixed lists, a synthetic flag and an idempotency key; every field is a hostname, a timestamp or a value from a fixed list, and free text is not accepted. With each report we also record which interface it arrived through (the website, the REST API or the MCP tool), and the operator that received it together with its own report ID. The website and REST API accept the same fields. The GET /v1/why endpoint, which explains a blocked step, takes the same public hostname and the same optional codes as the lookup (error_class, failure_stage, http_status, challenge, retry_after, via, agent_vendor), and it also takes optional codes from fixed lists about what happened: the agent product (agent_platform), whether it works in a cloud browser or on the person's device (agent_env), whether the same step works in a usual browser (external_browser), where the step stopped (stop_point), the kind of message on screen (screen_message), whether it worked before (since_kind) with an optional day no more than 90 days back (since_date), what the task is for (task_context), and a language (lang). A value that is not on its list is ignored and is never repeated in the answer. The pages /en/why and /ko/why are a form for it: your browser cuts a pasted address to its hostname before anything is sent, sends only that hostname and the codes you chose, and keeps none of your choices (no cookie, no local storage). These codes only choose which fixed sentences the answer shows; they are not counted, not stored and not logged. The one exception is the lookup codes your choices set for the lookup (error_class, task_step, failure_stage): they are counted like a lookup's. The answer reads the stored host-level records the lookup reads, plus our record of changes in a host's declared policy and in a platform's published documentation, and it carries no feedback token. Your IP address and User-Agent reach our network edge and hosting providers with every request; our database keeps only the derived counts listed under Retention, never the raw values. Our service also writes one line per request to its application log at the hosting provider: the method, the path without its query string (at most 200 characters, and the route pattern instead of the value for a route that takes a variable part such as an ID), the response status, the time to the first response byte, whether the request was the service's own internal MCP call and the first 120 characters of the User-Agent. A path that matches no route (a 404, the MCP mount, /static) and the language segment are kept exactly as sent (printable characters only, at most 200 characters). That line holds no IP address, query string, other header or body, and it is used only to see whether requests reach the service and how they end. We do not read or store metadata an AI client may attach to a tool call, such as locale or approximate location, and we set no tracking or advertising cookies. The find_task_route tool and GET /v1/task-routes accept a public hostname (site_host) and an optional task_kind from a fixed list. They read a table of curated records kept in the service's code and store nothing about the lookup: the hostname and task kind are not written to the database or counted. Like every request, the request itself appears only in the counts and the log line described in this policy. The search box on the task route pages takes a public hostname in the same way; if a full address is pasted, only the hostname is used and the rest is discarded, and nothing typed there is stored or counted. The report_task_route tool and POST /v1/task-routes accept a task-route report: a public hostname and values from fixed lists only (task kind, obstacle, kind of verification, what the site requires, kind of document, kind of entry point, routes tried, the route that worked, basis, outcome and duration), with an optional idempotency key. A report has no field for text, a link, a name, an account or a case number, and a report that sends any other field is refused. Reports about an account or tenant subdomain are refused too. We store a report under the site's root domain, with a reporting quarter, a source kind and a case-record flag that the server sets, the time it was received, its review state and the time of its review, the time of a withdrawal, and the idempotency key with a hash of the request and the time it was stored; the intake counters hold the number of reports per day, per site and per site and task kind. If you email support, we receive your address and message. We do not ask for names, contact details, precise location, payment data, or conversation content, and the tools are built to refuse personal URLs and free text.

03

What the tools return

check_incident returns public, host-level information only: the host's declared robots.txt and llms.txt policy (including the content-use preferences its robots.txt declares, and the API catalog, text-and-data-mining reservation and MCP server catalog it publishes at standard locations), curated official developer and status links, the provider's published status, published observations without any reporter identity, a coarse lookup-activity band, and, for a domain that a public government list names or whose suffix is reserved for government bodies, registry facts about the domain: the U.S. .gov list or the UK .gov.uk list it appears in, the organisation that list names, and the registry rule for the suffix. These are facts about the domain, not a judgement about any site, page or link. For a public hostname whose robots.txt does not disallow it for us, the answer may also carry what our own server received when it requested the host's home page (/), with no cookies and no sign-in: the HTTP status, a code for a known bot-defense provider when its marker was present, the kind of redirect, the time of the request, and the fact that it came from a data-center address. Our server makes that request during the lookup only if it can finish within 3 seconds; otherwise it runs in the background shortly after (within minutes) and a later lookup shows it. It is information about what our server received, not a verdict. It changes the recommendation in one case only: when the lookup itself reports that the site refused the caller and our check was refused too, the answer says to hand the step to the person. Another network may get a different answer. An answer to a blocked lookup may also carry a cause hint: one fixed sentence with times, a confidence level and the names of the stored host-level measurements it rests on (our own server checks, the provider's published status history and per-hostname lookup counts). It says whether a failure looks temporary, looks like a policy block or is not yet known; it is information, not a verdict, and it does not change the recommendation. An answer to a blocked lookup may also carry a one-time feedback token with the outcome and path codes it accepts; rate_last_lookup returns only whether the outcome was accepted. report_observation returns a receipt with the observation and incident IDs, the review state, and a withdrawal token for that report. find_task_route and GET /v1/task-routes return, for the hostname or its parent domain, up to 20 records: curated records and, once published after review, operator records and task-route reports. Each record holds only values from fixed lists (task kind, obstacle, kind of document, kind of entry point, kinds of route tried, outcome), a reporting quarter, the record's source kind, a flag that marks it as a case record, not advice, and, for a curated record, a link to the site's own page, the page it was read from, the date it was read and, where the page states one, a short quotation of a condition in the site's own words (at most 280 characters) (an operator record may carry a link to the site's own page; a report never carries a link). The answer also carries counts by source kind and one summary sentence built from fixed words. Curated records come from sites' own public documents; operator records and reports hold only the values above, and no record describes a person. report_task_route and POST /v1/task-routes return a receipt: the record ID, its review state and a withdrawal token for that report. No response contains personal data about anyone else.

04

How we use data

To answer lookups; to review, de-duplicate and publish reports; to limit abuse and enforce request caps; to operate, secure and measure the service in aggregate, for example which public hostnames are looked up and how many tool calls succeed; to count, per public hostname and hour, the optional codes of lookups and how many separate clients sent them, so that trouble at one host can be told apart from one client's problem (these counts are not used in answers at present); to count whether the next attempt after an answer worked, so that wrong answers can be found and corrected (these counts are not used in answers at present either); to measure, from our own server, how public hostnames answer a request for their home page, unless the host's robots.txt disallows it for us (with no cookies and no sign-in, paced by our own politeness limits; at most one redirect followed, only to the same host's www name and only when its robots.txt allows the home page for us), and to stop every such measurement automatically if refusals among hosts we do not measure rise by 3 percentage points or refusals of our status-page reads rise sharply (and to stop it as well if we cannot tell); and to answer support requests. Task-route reports: every report starts private, and an operator reviews it before it is shown; a report that looks unusual or repeated stays private, and we can hold a published one at any time. Intake is capped at 100 reports a day in all, 5 a day per site and 3 a day per site and task kind; a site's www name and its root domain share one count. We do not sell data, show ads, build profiles of people, or use your data to train AI models.

05

Who receives data

The public: reports that pass review appear on public pages and in API and MCP answers as unverified observations, with their structured fields and without any reporter identity. Lookup volume is shown publicly only as a coarse band. The lookup-code counts described under Data we collect are not shown publicly or in answers at present; if that changes, they will be shown only as coarse bands, and only when at least three separate clients contributed. Feedback outcome counts are not shown publicly or in any answer. Service providers that process data for us: Render (application hosting and database, United States), including its network edge, which runs on Cloudflare; and Porkbun (domain name service and email forwarding for the support address) together with the mailbox provider it forwards to. Websites we read: when we read a host's robots.txt or llms.txt, or a provider's status feed, that host sees a request from our server identified as AgentHelpNetwork, carrying nothing about you. We request llms.txt only where that host's robots.txt allows it. In the same way, only where it allows them, we request the three standard files a host may publish for agents (/.well-known/api-catalog, /.well-known/tdmrep.json and /.well-known/ai-catalog.json), each at most once per robots.txt read and at most 64 KB; a redirect to the host's own www address is followed once, so one file can take two requests. A host that refuses our robots.txt request itself (401 or 403) is not asked for these three files. Reads of robots.txt, llms.txt and these files run in the background after a lookup, not while the lookup waits. Status feeds are read on a schedule instead: we read the status feed of every provider in our status-mapping table (ahn/status_refresh.py in the public source repository) whether or not anyone looked the host up, about once an hour, and every 15 minutes while the feed reports an active status. Each read requests only the feed's own address (for most pages /api/v2/status.json), at most one request per second to one status host. A feed that fails is read again after 15 minutes and then at most an hour apart (every 15 minutes while its last statement was active); a refusal (401, 403 or 429) is never retried another way, and a feed that keeps refusing is read less often, down to once a day. A feed found for a looked-up host is read the same way. Before any feed is read, we read the status host's own /robots.txt (a successful read at most once a day for each host, a read that fails tried again after an hour, a second or more before the feed request) and request a feed path only if that file allows it for AgentHelpNetwork; if the file cannot be read because of a server error or a timeout, we treat the host as disallowing everything until it can be read, except that a file we read successfully less than 48 hours earlier keeps deciding in the meantime. A feed whose robots.txt disallows it is not read at all, and for that host we show a link to the official status page instead of a reading. The one exception: where a Statuspage page's robots.txt disallows its status API but allows its public incident-history feed (/history.atom), we read that feed instead, at the same frequency (every 15 minutes while it lists an open incident) and under the same limits (at most 128 KB, no redirect followed). That feed lists only the most recent announcements (incidents and scheduled maintenance) and carries no overall page status, component status or impact level, so for these providers we report only whether an announced incident is currently open; when none is listed we report the provider's status as unknown, not as operational. For a provider status page run on Atlassian Statuspage, or one that answers the same Statuspage API (the same page id and time zone fields), a feed read that finds an active status also reads two small public lists from the same status page, its unresolved incidents and its maintenances in progress (at most 32 KB each); nothing else is requested from it.Public government domain lists: we download two lists whole from their publishers, the U.S. .gov registry's public data repository (raw.githubusercontent.com) about once a day and GOV.UK about once a month; these requests carry no hostname and nothing about you, are identified as AgentHelpNetwork, and no site named on a list is contacted. The operator can export stored data for its own use; that operator-only export is not a public endpoint, and exported reports contain only the fields the public answer shows. Offline data bundle: a file anyone can download from /v1/bundle/latest.jsonl.gz, rebuilt about once a day and published under CC BY 4.0. It holds only data anyone can re-check at its source: the robots.txt and llms.txt reads of the hosts on the curated part of our public seed list (with the last 30 days of changes in their access-policy codes), the curated official developer paths (with their access details, whose short quotes from publisher pages stay the publishers' material and are not covered by that license), and the curated status-page mappings. Many of those mapped hosts were found by scanning bands of the Tranco popularity ranking, but each mapping was checked on that host's own status page, and no rank is included. Read times are given to the day only. It never contains reports, hostnames people looked up that are not on the seed list, lookup counts, request logs, IP digests, reporter keys, a status feed's current value, seed hosts taken only from a third-party popularity ranking, or the seed list's order or rank. Task routes: the records that find_task_route and GET /v1/task-routes return are curated from sites' own public documents, or are operator records and task-route reports published after review, and are shown to anyone who asks about that site. A published report shows only its fixed-list values, its reporting quarter and its source kind, never when it was received. Task-route reports are not part of the offline data bundle or the export. Curation reads: the curated records are read only from public help and policy pages (help, support, FAQ, terms, and returns, refund, shipping, cancellation and warranty policies, claims pages, and a government agency's own pages on how to apply, file or submit). For those pages only, the reader may request a path that the site's robots.txt disallows. It identifies itself with the user agent AgentHelpNetwork-curation, waits at least 3 seconds between requests to a host, and stops at a block response (401, 403, 429 or a challenge). It never signs in, submits a form or guesses an address, keeps only short quotations, each with a link, and does not read product, search, cart, account or user-content pages. A site's records are removed on request. We disclose data to authorities only when the law requires it. We share it with no one else.

06

Retention

Keyed, day-specific IP digests with daily request counts: 14 days. Referral and campaign category counts, and MCP tool-call counts by result category: 90 days. Per-host daily report-intake counters: 8 days. Hourly per-host counts of lookup codes: 35 days. Hourly per-host counts of feedback outcomes, with the recommendation given and the kind of client: 35 days. Fingerprints of used feedback tokens, with no hostname: until five minutes after the token expires, one hour after its lookup (a margin for clock differences between our servers; after that each new feedback and the cleanup that runs every 15 minutes delete them). Keyed IP digests specific to one hostname and one UTC day, with the client's last lookup time, the kind of blocked lookup and the recommendation it was given: 2 days. The digest alone does not link an address across hostnames or days; the last lookup time is kept to the minute for about an hour (until the next cleanup, which runs every 15 minutes, after it is 61 minutes old) and is then cut to the hour. Daily counts of answered lookups by client category (plugin, ahn-client, MCP host, script, browser, crawler, the site's own search page, or unknown), the MCP host family or ahn-client integration label where there is one (each read from the User-Agent header, which the database does not keep), the recommendation given and whether the hostname's declared-policy read was ready, stale, pending or none (no declared-policy layer for that hostname), with no hostname and no address: 90 days. Hourly hosting-usage samples from Render (CPU, memory and request totals): 90 days. The register of looked-up hostnames: removed after 30 days without a lookup unless the host has a connected official status feed. History of robots.txt reads for listed public hostnames (time, host read, status, label, classification, operator, declared network location, and whether the read's window was held and why), and of home-page checks of them from our daily check list (the same fields, a mark that it was a home-page check and, for an answer, its response-time range): 14 days. Periods in which a curated host's robots.txt could not be read or its provider reported a problem, and each change in a provider's published status: 90 days after their last change; a provider-reported problem that is still open is kept until it ends. Daily per-host totals of robots.txt reads, answers, failures, refusals and held reads, with the number of reads that followed a lookup and of seed-list reads and the number of home-page checks: 90 days. Changes in a public host's declared access policy (the time, the field that changed, and its old and new short code; never a file, a path or a header value): 90 days. For each public hostname we also keep the result of our home-page request: an outcome code, the HTTP status, a bot-defense provider code, the kind of redirect, the response time, the bytes read, the time, when it is next due, the time of the latest refusal counted toward two in a row and the server's declared network location and User-Agent, never the page, a header value, a cookie or a path: 30 days after the request. We keep with the cached profile whether the host's robots.txt allows the home page (allowed or not, the reason and the deciding rule line, at most 80 characters), and replace it on each read. Daily counts of those results by outcome, provider and what started them, and the daily counts (with no hostname) that drive the automatic stop: 35 days. The day's list of hostnames to measure (hostname, list and rank) is replaced each day. These history records contain no personal data. Daily lookup counts per public hostname and error type, and per-route request counts, contain no personal data and have no automatic deletion date. The cached robots.txt, llms.txt and provider-status results kept for each public hostname likewise have no automatic deletion date. When a provider leaves our status-mapping table, its registration and cached status are deleted. With each cached result we keep a SHA-256 hash of the file or feed as received (never the file itself), the reader version, and the operator and declared network location that made the read. With each provider-status result from such a Statuspage page we also keep the number of unresolved incidents and of maintenances in progress, the highest incident impact, the latest incident update time, at most 3 incident or component names as the provider published them (80 characters each), and whether every affected component is in a region or point-of-presence group; nothing else from those two lists is kept. For a provider-status result taken from a Statuspage page's Atom history feed we keep only the number of open incidents, the latest update time of the open incidents, the number of maintenances in progress (left empty when that feed cannot be trusted for it), whether the feed's window is too short to be trusted, and that its source is that feed; nothing else from the feed is kept. If one of those lists cannot be read, we also keep how many times in a row that happened, when, and a short label for why, and we do not ask for the two lists again for a while (15 minutes, or an hour after a refusal, doubling with each further failure, at most a day); after a refusal (401, 403 or 429) of one list the other is not asked for either. With each provider-status result we also keep the number of consecutive failed reads of its feed, which sets how soon it is read again. For a status page whose status.json has the incident.io shape we also keep a marker of that platform, and the same incident fields read from its summary.json (at most 3 incident names, 80 characters each); nothing else from that file is kept. For each status host we also keep whether its robots.txt allows each feed path we would request, the rule line that decided it (at most 80 characters), the parser version, a hash of the file and the time it was read; this record is replaced on each read. It also keeps how the read ended (found, missing, unavailable or failed, with a short label such as http_503 or, for a redirect we do not follow, the host it pointed to), which group of the file decided (the one naming AgentHelpNetwork or the * group), and, when a later read failed while an earlier successful one kept deciding, when and why. With each robots.txt result we also keep the content-use preferences it declares (Content-Signal and Content-Usage lines) as normalized yes, no or unset values, with at most 3 sample paths in total, the agent names of at most 5 user-agent groups (only names on our list of known agent tokens, at most 8 per group; any other name is only counted), at most 5 names of keys we could not read, the parser version and the robots.txt hash. Where its robots.txt allowed them, we also keep what the host publishes for agents at those three standard locations: at most 10 APIs from its API catalog with at most 3 links of each kind, at most 10 bookmark links and at most 3 links to further catalogs, at most 10 text-and-data-mining reservation rules and at most 5 reservation headers, and at most 5 MCP server entries from its AI catalog; each link is at most 300 characters and each name or pattern at most 120. The files themselves are not kept. For a status page we find automatically, we also keep the evidence that it belongs to the host (its DNS delegation), the page address and the operator that recorded it. A lookup that needs a fresh read leaves a background-read request (the public hostname, the kind of read, when it was asked for and, for a lookup marked as a test, that mark), which is deleted as soon as the read is done, or after five failed attempts. We also keep the current copy of each public government domain list we download (the U.S. .gov list and the UK .gov.uk list): for each domain its name and, for the U.S. list, the domain type, organisation, sub-organisation and state; the security contact email address and the city that the U.S. file contains are not kept. With each list we keep its version date, the address it was downloaded from, a SHA-256 hash of the file as received, the reader version, the operator and declared network location that made the download, and the time of the last check. Each copy is replaced when its publisher posts a new version, and the replaced rows are deleted; a download that fails or looks wrong leaves the previous copy in place. Daily report-intake totals, the incident records that group reports by hostname, task step and error type, and audit-log entries (event type, report ID and time) also contain no personal data and have no automatic deletion date. Reports are deleted 365 days after they are received; a withdrawn report is excluded from every read at once and deleted 30 days after withdrawal. Public reads show only the last 7 days, and we delete a report sooner on request. Rate-limit counters stay in memory for about one minute and are never written to disk. Support emails are kept only as long as needed to handle the request. A cleanup job that runs every 15 minutes deletes expired rows from the database; copies in the hosting provider's database backups expire under that provider's backup schedule. Render and Cloudflare keep their own delivery and security logs under their own retention terms. The per-request line our service writes to its application log stays with Render under Render's log retention terms and never enters our database. Task-route reports are deleted 365 days after they are received; a withdrawn record is excluded from every read within a few minutes and deleted 30 days after withdrawal. Operator task-route records have no automatic deletion date; an operator can hold or withdraw one at any time, and once withdrawn it is handled like a withdrawn report. An operator can also withdraw a public task-route report at any time, and it is then handled the same way. Daily task-route intake counters per site and task kind: 8 days. Offline data bundle: the latest file and the day's total of bytes served (no client data) are kept in one database row, replaced on each daily build and deleted when the bundle is switched off. Download limits per client are counted in memory only, under a keyed daily digest of the IP address, and are never stored.

07

Your controls

Send only public hostnames and the structured codes above. Keep the withdrawal token from your receipt: DELETE /v1/observations/{id} with that token removes your report from all reads at once. report_observation is a separate write action that your AI client may ask you to confirm; check_incident is read-only. rate_last_lookup is an optional write action that sends only a token and outcome codes; if you never send a token back, no outcome is recorded. You can disconnect the app in ChatGPT or remove the MCP server from your client at any time. Email support@agenthelpnetwork.com to ask what we hold about a report or request, or to correct or delete it; include the report ID if you have it. Depending on where you live, you may have further rights such as access, deletion or objection, and you can exercise them the same way. We aim to answer within 30 days. If you control a public hostname and want it left out of our offline data bundle, email us from an address at that domain; we leave it out of the next bundle we build. Lookup answers still read the host's own robots.txt, which is the host's public declaration. To take back a task-route report, send DELETE /v1/task-routes/{route_id} with its withdrawal token in the X-Withdrawal-Token header; it leaves every answer within a few minutes.

08

Children, location and changes

The service is meant for developers and their agents, is not directed to children, and does not knowingly collect children's data. Data is processed and stored in the United States. When our practices change we update this page and the date above before the change takes effect.

Technical details

A

Structured fields only

Reports use structured public hostnames, task categories, timestamps, attempt context, and outcomes. Never submit passwords, cookies, names, account identifiers, personal URLs, or conversation text. Our app stores daily page and lookup counts, bounded referral and campaign categories, and coarse client-software hints. When the hosting edge supplies a client IP, we store only a keyed, day-specific digest and daily request count for 14 days; no raw IP, cookie, full Referrer URL, or User-Agent is stored in these analytics tables. Eligible public root hostnames are kept in a lookup register with first and latest lookup times, and are removed after 30 days without a lookup unless the host has a connected official status feed. Hosting infrastructure may process IP addresses and User-Agent headers for delivery and security. These categories and digests do not identify a person or prove an independent agent. Optional lookup codes (HTTP status class, challenge header, Retry-After range, fetch or browser) are also counted per public hostname and hour for 35 days, together with a keyed digest of the IP address that is specific to that hostname and that UTC day, kept for 2 days to count separate clients. Successful MCP check tool calls are also counted daily by evidence and provider-signal category for 90 days, without a hostname or client identity in that tool-call table. Requests from an owner-allowlisted network are marked as an aggregate owner-network category; this does not identify a person or prove a test. Hosts on our public seed list are also read about once a week for their declared robots.txt and llms.txt policy even if nobody looked them up; the parsed result is stored the same way.

B

robots.txt and llms.txt reads

After a lookup of an eligible public apex hostname we read, in the background and not while the lookup waits, that host's own public /robots.txt (up to 256 KB) and /llms.txt (presence only, up to 4 KB) over HTTPS. Only those two paths are requested, on the host itself or its own www. alias: if the apex read fails, is refused, redirects elsewhere, or returns a web page, www.<host>/robots.txt is read once instead. The only redirect followed is a single hop from the host to its own www. alias; any other redirect is recorded and not followed. A definitive answer (the file, its absence, a refusal, a redirect, an oversized or invalid file, or a non-public address) is cached for 24 hours; a transport failure (DNS, TLS, connection, timeout, or an unexpected HTTP status) is retried after one hour. One attempt makes at most five HTTPS requests. Reads go only to hosts whose addresses are all public, identify themselves as AgentHelpNetwork, and fetch nothing else: no task pages, no login or CAPTCHA pages, no probing. We store the parsed per-agent rules, up to six sample Disallow paths per agent token and for the `*` group (80 characters each), up to five Sitemap URLs, the content-use preferences it declares (Content-Signal and Content-Usage lines) as normalized yes, no or unset values, with at most 3 sample paths in total, the agent names of at most 5 user-agent groups (only names on our list of known agent tokens, at most 8 per group; any other name is only counted), at most 5 names of keys we could not read, the parser version and the robots.txt hash, the read status, and a SHA-256 hash of each file as received; the rest of the file is not stored. Content-use preferences are parsed from the same robots.txt and add no request. The result is the host's declared crawler policy and is never an access guarantee; a host may treat a user-directed agent differently. Besides looked-up hosts, we also read the hosts on a public seed list (ahn/seed_hosts.txt in the public source repository) about once a week, even if nobody has looked them up, under the same limits; a transport failure there is retried after 24 hours. The local build does not perform these reads. For listed public hostnames only (hosts on the curated part of the seed list, in our official-paths table, in our status mappings, in our public check list (ahn/probe_hosts.txt) or in our consumer-paths table; never an IP address or a name without a public suffix), we also keep a short history of these reads, of our home-page checks of them and of changes in a connected provider's published status. For each robots.txt read we keep the hostname, the time, the host that was read (the host itself or its www. alias), the read status and the reader's label (for a failure its kind: a DNS, connection, TLS, timeout or server (5xx) error; for an answer its HTTP status code or, for a redirect, the target hostname), our classification of the result, the operator and declared network location that made the read, and, once our summary job has processed it (we keep that time), whether its 15-minute window was held as a possible problem on our side and why. From these we keep periods in which a host's robots.txt could not be read (start, confirmation and end times, how the period ended, the numbers of failed, held and answering reads in it, and whether a provider feed was registered for the host and reported a problem at the same time) and per-host daily totals of reads, answers, failures, refusals and held reads, with the number of reads that followed a lookup and of seed-list reads. For a provider we keep each change of its published indicator, with the provider's own update time. For each home-page check of a listed host made from our daily check list (not the check that can follow a lookup) we keep the same kind of record: the hostname, the time, the host checked, the HTTP status code or the failure kind, our classification, the operator and declared network location, a mark that it was a home-page check and, for an answer, its response-time range (under 1 second, 1 to 3 seconds, or 3 seconds or more; no exact time is kept here); the daily totals also count home-page checks. A period of failure opens only after at least 2 failed reads or checks at least 45 minutes apart, and the next answer ends it. A refused read or check (401, 403, 429 or 503, a bot challenge or a login wall) is never counted as a problem with the site, and a host that refuses a home-page check is not checked again for 7 days, except that a bot-challenge refusal, a 401 without a sign-in request or a 403 that comes after any result other than a refusal (an answer, a failure or no check) is checked once more after 24 hours. These records are about hosts, not people: they contain no lookup, report, IP address or client information, and they never record who looked a host up; whether a read followed a lookup is kept only as a daily count per host. Keeping them adds no request to any site beyond the reads and checks described here; a check from the list that ends in a 5xx other than 503 or in a failure to connect is asked once more, after the usual pause, before it counts. Only the operator sees them, on a private dashboard; they are not shown on public pages or in lookup answers. A public page (“Site problems right now”, /en/now, /ko/now and /v1/now) lists curated hosts whose connected provider status page currently reports a problem: the same provider statement a lookup answer shows, only while the provider updated it within the last 7 days, with 7-day totals that name no host. Where we read a page's public incident-history feed instead of its status API, a line says only that the provider announced an open incident there, with no severity level. It shows no robots.txt read result, no refused read and nothing about who looked a host up. The one exception is a record of change: for any public hostname whose robots.txt we read, when a field of the declared access policy differs from the previous reading we keep one record (the hostname, the time, the field, and its old and new short code; never a file, a path or a header value) for 90 days, and the first reading of a host records nothing. These records may be summarized in the optional cause field of the answer to a failed lookup and, for hosts on the curated part of the seed list only, over the last 30 days in the offline bundle. They contain no personal data and no lookup, client or address information.

C

Authority and visibility

Submitting agents follow their own authority and platform rules. Reports start private. A rules-based review may publish a limited observation as unverified after a short delay; suspicious or repeated submissions remain private for operator review. Anonymous submissions share one unverified source. Keep the withdrawal token to remove your own report.

D

Limits

Rules can catch malformed or repeated reports but cannot prove an event happened. We cap intake, hold unusual submissions, and can hide a report. Public lookup covers seven days. Reports are deleted 365 days after they are received; a withdrawn report leaves every read at once and is deleted 30 days later. Missing reports do not mean a site works, and one failure does not prove an outage.

E

Government list downloads

The U.S. .gov list is the registry's current-full.csv in the public cisagov/dotgov-data repository (CC0-1.0), checked once every 24 hours with a conditional request. The UK list is GOV.UK's List of .gov.uk domain names, found through the GOV.UK content API and downloaded from GOV.UK's asset host when a newer version appears; it is checked once every 30 days. Contains public sector information licensed under the Open Government Licence v3.0. Only those hosts are contacted, over HTTPS, with no redirect followed, and a file larger than 8 MB (U.S.) or 2 MB (UK) is refused. The stored copy is never replaced by a download that fails, has the wrong columns, has too few usable rows, or has fewer than 50% of the stored rows; the next attempt follows 6 hours later. An answer reads the stored copy only, so a lookup waits for none of this. The local build does not perform these downloads.