Bandwidth Spiked, GA4 Flat: The AI Crawlers Your Tag Never Sees

"Bandwidth suddenly jumped — is this a bot attack?" That's the first thing people say after opening the server control panel. But open GA4 and sessions are flat: there are no visitors to account for the increase. The reason lies in where GA4's measurement begins.

Contents

Traffic flat, bandwidth spiking — the cause sits outside GA4
Why GA4 shows nothing — measurement starts at the JS snippet
What "unlimited bandwidth" actually means before you upgrade
Block them or let them in — how far robots.txt actually goes
How RevenueScope helps
FAQ

Summary / References / Related articles

TL;DR#

In a Cloudflare tally that followed the same customer set for a year, requests from AI and search crawlers rose 18%[1]
GPTBot's requests rose 305% and its share of crawling went from 2.2% to 7.7%. ClaudeBot, meanwhile, fell 46% — the direction is not uniform[1]
GA4 measurement starts by placing the Google tag on every page of the site[4]. A crawler that never runs that tag's JavaScript leaves no session behind
Known-bot exclusion is applied by GA4 automatically. There is no setting to turn it off, and no screen that shows how many hits were excluded[3]
Japan's three major shared-hosting providers don't charge for data transfer. All three, though, state in writing that they restrict excessive load[9][10][11]

1. Traffic Flat, Bandwidth Spiking: The Cause Sits Outside GA4#

You open the server control panel on Monday morning and the bandwidth graph has climbed to nearly double last week's. No incident notice has arrived. GA4 sessions trace almost the same shape as the previous week, and revenue is as usual. Nowhere are there visitors to match the extra data volume.

The first suspect is AI crawlers. In a Cloudflare tally that followed the same customer set for a year, requests from AI and search crawlers rose 18%. The figure covers May 2024 through May 2025[1]. The increase is concentrated in particular bots. OpenAI's GPTBot saw requests rise 305%, and its share of all crawling went from 2.2% to 7.7%[1].

Not every crawler is growing at once, though. Over the same period Anthropic's ClaudeBot fell 46%, its share dropping from 11.7% to 5.4%. ChatGPT-User rose from 0.1% to 1.3%[1]. Bots that are growing and bots that are shrinking sit in the same table.

Purpose is skewed too. In a Cloudflare breakdown by purpose, roughly 80% of crawling was collecting training data, while on-the-spot fetching prompted by a user instruction was under 5%[2]. What gets fetched is the HTML and the images themselves, so bandwidth rises even when visitors don't.

If GA4 sessions are rising alongside it, that's a different story — access that does execute JavaScript may be mixed in (how to isolate a spike when GA4 sessions themselves are rising).

2. Why GA4 Shows Nothing: Measurement Starts at the JS Snippet#

Nothing shows up in GA4 because GA4's measurement starts from JavaScript.

Google's official help describes the first step of measurement as placing the Google tag on every page of the site[4]. Only once that tag's JavaScript runs is a record created as one session. Most AI crawlers fetch the HTML and read what's inside, but they don't go as far as executing the embedded JavaScript. The server has sent the files out and counted them toward bandwidth, yet GA4 has no matching row. GA4 isn't dropping anything; the conditions under which a record gets created were never met.

On top of that, GA4 has a mechanism that excludes known bots automatically. Access matching the IAB known-bot list and Google's own detection is removed from reports automatically. This exclusion can't be turned off, and there is no screen that displays how many hits were excluded[3]. Inside GA4, no clue is left for you to check.

The clue is in the server access log. The log keeps the user agent (UA — the identifier the requester declares) for every request, and the major providers publish the UAs of their own crawlers. OpenAI lists GPTBot, OAI-SearchBot, ChatGPT-User and others by purpose[5]. Anthropic publishes the names of its ClaudeBot-family crawlers, and Perplexity publishes PerplexityBot and Perplexity-User[6][7]. Google-Extended alone has no UA of its own; it works as a token for controlling Google's crawlers[8].

Look at the log and you can tell which crawler the extra bandwidth came from (how to tell whether rising traffic is human or bot). But knowing a crawler's name and knowing what is happening to revenue are two different things.

On the sample data, run a CaseStudy on the revenue left after bot exclusion

3. What "Unlimited Bandwidth" Actually Means Before You Upgrade#

SAKURA Rental Server states plainly that there is no data transfer limit. It also writes that when a program's load is high, the account will be restricted. What comes back is a "503 Service Temporarily Unavailable" warning[10]. What stops you is not the transfer volume itself but the load.

All three providers are worded in similar terms. Xserver states that there is no charge for data transfer and that it is unlimited[9]. Lolipop! also calls data transfer unlimited, and writes that it will impose restrictions when the load is heavy enough to affect other users[11].

A table comparing the data transfer policies of three Japanese shared-hosting providers. Xserver, SAKURA Rental Server and Lolipop! all treat data transfer as unlimited with no metered charge, all state in writing that they restrict excessive load, and all keep the threshold at which a warning fires undisclosed. Illustrative

One thing is not settled. None of the three publishes the threshold — how many gigabytes it takes before a warning arrives. "A warning email fires at X GB" can't be read out of the official documentation, so if you're considering a move to a higher plan, check the control panel and the terms of your current contract first.

What a bandwidth spike moves first is load and cost, a separate problem from lost revenue (how invalid click filtering differs from bot exclusion). But "it's a load story, so no need to rush" only holds once you've confirmed in numbers that human sessions and revenue are intact. Sessions and revenue themselves do appear in GA4, but whether tag-executing bots are mixed in there can't be confirmed inside GA4, since the breakdown of exclusions is never displayed.

4. Block Them or Let Them In: How Far robots.txt Actually Goes#

In a Cloudflare tally for the first week of August 2025, the ratio of Anthropic's crawling to AI-referred visits was about 50,000 to 1. OpenAI was 887 to 1, and Perplexity 118 to 1[2]. That's a different period from the year-long tally above, and all of them are averages across every industry.

Cards showing how many crawls it takes for one referred visit to come back, averaged across all industries. Anthropic is about 50,000 to 1, OpenAI is 887 to 1, and Perplexity is 118 to 1. Illustrative

The volume of data fetched rises first, and the visits that come back are very few. Here the move splits in two: refuse, or allow.

The tool on the blocking side is robots.txt. OpenAI and Anthropic both officially explain that you can refuse their crawlers by naming them in a directive[5][6]. Training use and answering use can be specified separately. Google states plainly that refusing Google-Extended does not affect your ranking in Google Search[8].

This doesn't stop everything, though. Perplexity-User fetches a page on the spot in response to a user instruction, so Perplexity itself states that as a rule it does not follow robots.txt[7]. Anthropic writes in its own documentation that blocking by IP address isn't recommended either: block it and the crawler can no longer read robots.txt, so the refusal directive itself never arrives[6]. Cloudflare reports that AI crawlers often ignore robots.txt directives as well[2]. Treat the official documentation as a starting point, and after writing your refusal, watch the log to see whether it is taking effect.

Block them and bandwidth falls — and so do the visits that arrive because AI cited you (the AI toll debate Cloudflare started). 50,000 to 1 is an all-industry average, and it decides nothing for one site. What it takes to decide is a single number: how much revenue AI-referred traffic is producing on the site itself.

RevenueScope solution

Open the traffic-source list and sessions and revenue per channel appear on one screen. There's a column for the bot-exclusion count, and a channel with a value in it means that many hits were removed by the detection. The sessions and revenue that remain after exclusion are the human-only numbers.

RevenueScope's bot detection is behavior-based, removing tag-executing bots from the human metrics. It displays the exclusion count as well, so even when sessions are 0, a value in the exclusion column tells you that traffic wasn't absent — only bots were arriving. RevenueScope specializes in measuring revenue.

In the channel list, answer-referred traffic from ChatGPT, Claude, Gemini and Perplexity is handled as its own row. Sessions, revenue and RPS (Revenue Per Session — revenue per visit) line up on that same row. How much is selling via AI can be confirmed as an amount (the number to read when traffic grows but revenue doesn't).

How It Reads in a Month When Bandwidth Rose#

Here is one month of fictional Store A, as one example.

Channel	Sessions	Revenue	RPS	Bots excluded
Search	8,000	¥2,400,000	¥300	400
Ads	2,000	¥800,000	¥400	120
Direct	1,000	¥300,000	¥300	30
Via AI	600	¥300,000	¥500	50

These four rows are figures made up for fictional Store A, not those of a real store. The demo screen reads the sample data of the sample store (updated daily), so the same values won't appear there.

Two things can be read off it. Bots are arriving, but even after exclusion, search still leaves 8,000 sessions and ¥2,400,000. And traffic via AI, with only 600 sessions, has an RPS of ¥500 — above search's ¥300. Bandwidth rose but revenue is intact, and the AI route is the most efficient entrance. The next move is to refuse only the training crawls while letting the answering ones through, then read AI-referred sessions and revenue on the same screen again next month.

In the sample store, compare AI-referred traffic and revenue period by period

FAQ#

Frequently asked questions#

Q. Will bandwidth fall if I block them with robots.txt?

A. It falls by the share of crawlers that follow what's written. But kinds like Perplexity-User, which fetch on a user's instruction, are officially explained as not following robots.txt as a rule[7]. Anthropic doesn't recommend blocking by IP address either[6]. How much it falls depends on what was arriving, so after writing the refusal, confirm the effect on the server's bandwidth graph.

Q. Can GA4 settings make the bot share visible?

A. No. GA4's known-bot exclusion is applied automatically, there is no setting to turn it off, and there is no screen that displays how many hits were excluded[3]. The only place the bandwidth breakdown can be seen is the server access log. What a measurement tool can show is the exclusion count for tag-executing bots, and the human sessions and revenue that remain after exclusion.

Q. If only bandwidth rose, can revenue be left alone?

A. Once it's confirmed that human sessions and revenue haven't moved, that month can be spent on handling the load. Leave it unconfirmed and a change in revenue happening at the same time gets missed. The order is: confirm revenue first, then deal with the load.

Summary#

When only server bandwidth rises and GA4 stays flat, the answer isn't inside GA4. Measurement starts from the Google tag placed on the site[4], so access that never runs that JavaScript leaves no record behind. Known-bot exclusion doesn't show its count either[3]. The place where crawler names are visible is the access log.

A bandwidth spike is a load-and-cost problem. The three major providers don't charge for it, while all of them state in writing that they restrict excessive load[9][10][11]. What decides whether to block or allow is not an all-industry average but the site's own AI-referred revenue.

There's one thing to do today. Before lining the bandwidth graph up against GA4, put these two within reach: human sessions and revenue after bot exclusion, and revenue via AI.