> ## Documentation Index > Fetch the complete documentation index at: https://gcore-doc-1894.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Known Bots export const MethodSection = ({children}) => children ?? null; export const MethodSwitch = ({children}) => { const tabs = React.Children.toArray(children).map(c => { if (!c || !c.props) return null; if (c.props.id) return c; const inner = c.props.children; if (inner && inner.props && inner.props.id) return inner; return null; }).filter(Boolean); const firstId = tabs.length > 0 ? tabs[0].props.id : ""; const [active, setActive] = React.useState(firstId); React.useEffect(() => { try { const saved = localStorage.getItem("gcore_docs_method"); if (saved && tabs.find(t => t.props.id === saved)) { setActive(saved); } } catch (_) {} }, []); React.useEffect(() => { try { document.querySelectorAll("h2[id], h3[id]").forEach(heading => { const visible = heading.offsetParent !== null; document.querySelectorAll(`a[href="#${heading.id}"]`).forEach(link => { if (link.closest("h1,h2,h3,h4,h5,h6")) return; const li = link.closest("li"); if (li) li.style.display = visible ? "" : "none"; }); }); } catch (_) {} window.dispatchEvent(new Event("scroll")); }, [active]); const handleClick = id => { setActive(id); try { localStorage.setItem("gcore_docs_method", id); } catch (_) {} }; return
{tabs.map(tab => { const isActive = active === tab.props.id; return ; })}
{tabs.map(tab =>
{tab.props.children}
)}
; };

This policy group functions uniquely in WAAP. Instead of challenging or blocking incoming traffic like other policy groups, it's designed to recognize and allow traffic from legitimate automated services, such as search engine crawlers, AI agents, social media preview fetchers, monitoring tools, and payment processors.

These services access a domain to gather information and are generally beneficial or necessary for normal web activity. For each bot, configure how WAAP treats its traffic once the bot has been recognized and verified.

This policy group is available in the Pro and Enterprise plans. ## Bot categories

Known bots are grouped into intent-based categories to clarify each bot's purpose and help with policy decisions. Each bot belongs to exactly one category, with policies configured at the individual bot level.

| Category | Description | | ------------------------- | ----------------------------------------------------------------------------------- | | AI Search | Bots indexing public content for AI-powered search and answers. | | AI Training | Crawlers collecting public web data to train AI models. | | User-triggered AI | Agents that fetch or browse on behalf of a specific user request. | | Search Engines | Traditional search engine crawlers indexing pages for organic search. | | Preview Bots | Fetchers that generate link previews and metadata for social platforms. | | SEO and Data Crawlers | Third-party crawlers for SEO analysis, audience measurement, and content feeds. | | Monitoring | Automated checks for uptime, performance, and infrastructure health. | | Security Scanners | Crawlers performing security audits, certificate validation, and plugin checks. | | Integrations and Services | Webhooks, payment callbacks, and infrastructure services making automated requests. | ## Bot actions

Each known bot can be set to one of two actions:

* **Allow** — the request bypasses all WAAP processing layers, including DDoS protection, behavioral analytics, OWASP checks, and other handlers. Recommended only for fully trusted bots. * **Policy-based** — the request is inspected, but the bot-detection checks that only apply to *unknown* automation are skipped (see below). Signature-based WAF and OWASP protections, IP-reputation threat signals, and custom rules still apply.

Some bots are set to **Policy-based** by default; switch a bot to **Allow** to fully permit its traffic. AI Search and AI Training bots default to **Allow**, while User-triggered AI bots default to **Policy-based**. To restrict a bot — to enforce data-protection requirements, block an unused service, or protect paid content from scraping — adjust the action for that bot.

## Policy-based mode and other protections

Most bot-detection checks are designed to identify *unknown* automated traffic. Once a request is matched to a verified known bot, running those same checks again produces guaranteed false positives rather than added security. Cloud-hosted AI fetchers would trip **Traffic from hosting services**, search crawlers would trip **Anti-scraping** (indexing pages is scraping by definition), and any high-volume crawler would trip **Traffic anomaly**.

For this reason, when a request is matched to a known bot set to **Policy-based**, the following checks are excluded — they do not evaluate, do not generate block events, do not increment detection counters, and do not appear as triggered in the event log:

| Group | Excluded checks | | ------------- | -------------------------------------------------------------------------------------------------------------------------- | | IP Reputation | Bot traffic; Traffic from hosting services | | Bot Attacks | Unknown user agent; Automated clients; Traffic anomaly; Anti-scraping; Headless browsers; Anti-spam; Vulnerability scanner |

These protections remain fully active for known-bot traffic:

* Signature-based WAF inspection and OWASP protections * IP-reputation threat signals * Custom rules — a custom rule that targets a known-bot pattern is never suppressed Exclusions apply only to traffic that passes verification. If a request carries a known-bot User-Agent but fails verification, it is not treated as a known bot and every check evaluates normally. ## Bot verification

Identification does not rely on the User-Agent alone, which attackers can spoof. A request is treated as a known bot only when its User-Agent **and** its source match the identity published by the bot's vendor. Verification uses one or more of:

* Official vendor IP ranges, refreshed regularly * Organization ownership of the source address * Signed requests, where the vendor supports them

Only traffic that passes verification is treated as a known bot and receives the handling described above. Traffic that fails verification is not classified as a known bot: it receives no exclusions and is inspected like any other request.

## Configure Known Bots

Review and configure relevant bots in the [Gcore Customer Portal](https://portal.gcore.com/accounts/reports/dashboard):

1. Navigate to **WAAP** > **Bot Management**. 2. In the domain dropdown at the top of the page, select the needed domain. 3. Click the **Known Bots** tab to view bots grouped by category and adjust each bot's action.

Known Bots policies control how WAAP handles traffic from recognized automated services — search engine crawlers, AI agents, social media preview bots, monitoring tools, and payment processors. In the Customer Portal these bots are grouped into intent-based categories, but the API exposes them as a flat list of policies. Each bot can be set to **Allow** (bypass all WAAP processing) or **Policy-based** (inspected, with unknown-automation detection checks skipped).

An [API token](/account-settings/api-tokens) is required, along with the ID of a [WAAP-protected domain](/waap/getting-started/configure-waap-for-a-domain) and the [Python](/developer-tools/sdks/python) or [Go](/developer-tools/sdks/go) SDK installed for SDK examples. This policy group is available on the Pro and Enterprise WAAP plans. ```bash theme={null} export GCORE_API_KEY="{YOUR_API_KEY}" export WAAP_DOMAIN_ID="{YOUR_DOMAIN_ID}" ``` ## View bot modes

Retrieve the full list of Known Bots policies and their current modes for a domain.

```python theme={null} import gcore import os client = gcore.Gcore() domain_id = int(os.environ["WAAP_DOMAIN_ID"]) rule_sets = client.waap.domains.list_rule_sets(domain_id) bot_set = next( rs for rs in rule_sets if rs.resource_slug == "common-automated-services" ) for bot in bot_set.rules: mode = "Allow" if bot.mode else "Policy-based" print(f"{bot.name}: {mode} ({bot.id})") ``` ```go theme={null} package main import ( "context" "fmt" "os" "strconv" gcore "github.com/G-Core/gcore-go" "github.com/G-Core/gcore-go/waap" ) func main() { client := gcore.NewClient() domainID, _ := strconv.ParseInt(os.Getenv("WAAP_DOMAIN_ID"), 10, 64) ruleSets, _ := client.Waap.Domains.ListRuleSets(context.Background(), domainID) for _, rs := range *ruleSets { if rs.ResourceSlug == "common-automated-services" { for _, bot := range rs.Rules { mode := "Policy-based" if bot.Mode { mode = "Allow" } fmt.Printf("%s: %s (%s)\n", bot.Name, mode, bot.ID) } } } } ``` ```bash theme={null} curl -X GET "https://api.gcore.com/waap/v1/domains/${WAAP_DOMAIN_ID}/rule-sets" \ -H "Authorization: APIKey ${GCORE_API_KEY}" \ | jq '.[] | select(.resource_slug == "common-automated-services") | .rules[] | {name, id, mode}' ```

The API returns all policies in the group. `mode: true` means Allow; `mode: false` means Policy-based.

`mode: false` (Policy-based) carries the detector-exclusion behavior described in the Customer Portal tab: unknown-automation detection checks are skipped for verified known-bot traffic, while signature-based WAF, OWASP protections, IP-reputation threat signals, and custom rules still apply. ## Set bot mode

Sets a bot to Allow or Policy-based by specifying the target mode. Pass `mode: true` to set Allow and `mode: false` to set Policy-based. Use the View bot modes response to find the bot ID.

| Parameter | Required | Description | | ----------- | -------- | ------------------------------------------- | | `bot_id` | Yes | ID of the bot policy to configure. | | `domain_id` | Yes | ID of the WAAP-protected domain. | | `mode` | Yes | `true` for Allow, `false` for Policy-based. | ```python theme={null} import gcore import os client = gcore.Gcore() domain_id = int(os.environ["WAAP_DOMAIN_ID"]) # Find the bot ID by name rule_sets = client.waap.domains.list_rule_sets(domain_id) bot_set = next( rs for rs in rule_sets if rs.resource_slug == "common-automated-services" ) bot = next(r for r in bot_set.rules if r.name == "Google Bot") result = client.waap.domains.policies.toggle(bot.id, domain_id=domain_id, mode=True) print(f"Google Bot is now {'Allow' if result.mode else 'Policy-based'}") ``` ```go theme={null} package main import ( "context" "fmt" "os" "strconv" gcore "github.com/G-Core/gcore-go" "github.com/G-Core/gcore-go/waap" ) func main() { client := gcore.NewClient() domainID, _ := strconv.ParseInt(os.Getenv("WAAP_DOMAIN_ID"), 10, 64) // Find the bot ID by name ruleSets, _ := client.Waap.Domains.ListRuleSets(context.Background(), domainID) var botID string for _, rs := range *ruleSets { if rs.ResourceSlug == "common-automated-services" { for _, bot := range rs.Rules { if bot.Name == "Google Bot" { botID = bot.ID } } } } result, _ := client.Waap.Domains.Policies.Toggle(context.Background(), botID, waap.DomainPolicyToggleParams{DomainID: domainID, WaapDomainPolicySettings: waap.WaapDomainPolicySettingsParam{Mode: true}}) mode := "Policy-based" if result.Mode { mode = "Allow" } fmt.Printf("Google Bot is now %s\n", mode) } ``` ```bash theme={null} # Set BOT_POLICY_ID to the bot ID from the View bot modes response export BOT_POLICY_ID="{BOT_POLICY_ID}" curl -X PATCH "https://api.gcore.com/waap/v1/domains/${WAAP_DOMAIN_ID}/policies/${BOT_POLICY_ID}" \ -H "Authorization: APIKey ${GCORE_API_KEY}" \ -H "Content-Type: application/json" \ -d '{"mode": true}' ``` Response: ```json theme={null} {"mode": true} ```

The API returns the updated mode. `mode: true` means Allow; `mode: false` means Policy-based.