Bytespider: what it is and how to allow or block it
ByteDance’s crawler. Generally listed as an AI data crawler, with no official documentation we could find.
Last checked 2026-10-08
- Company
- ByteDance
- Used for
- ByteDance (TikTok’s parent company)
- Job
- Collects pages to train AI models
- Follows robots.txt
- Documented, but reports vary
- Name in robots.txt
- Bytespider
What Bytespider does
Bytespider is the crawler run by ByteDance, the company behind TikTok. Third-party crawler trackers generally list it as a crawler that gathers data for AI, though ByteDance itself has not documented that.
Unlike every other bot in this library, we could not find an official ByteDance page describing what it does or how to control it. Treat everything about it with more caution than the documented bots.
Should you allow it?
Most businesses have no search reason to let it in. If you are opting out of AI training, add it to your robots.txt and consider a firewall rule too.
What blocking it changes: If it follows the rule, it stops collecting your pages. Because that is uncertain, sites that need a firm block usually add a firewall or CDN rule as well.
Allow or block Bytespider in robots.txt
A robots.txt rule naming Bytespider is the standard opt-out, but independent reports disagree on how reliably it is followed.
To allow it
User-agent: Bytespider Allow: /
To block it
User-agent: Bytespider Disallow: /
robots.txt lets every bot in by default, so you only need the allow lines if another rule in your file would otherwise block it. Our robots.txt guide explains how rules for different bots combine.
How to check a visit is really Bytespider
ByteDance publishes no IP list we could find, so a visit can only be judged by its user-agent string, which anyone can copy.
Our guide to server log checks shows where to find these visits on common hosting setups.
Source: No official documentation found, checked 2026-10-08. Bots and their rules change often, so check the company’s own page before making a decision that matters.
See which AI bots your site lets in
The free AI readiness check reads your robots.txt and reports which of the main AI crawlers are allowed or blocked, along with everything else that decides whether AI can understand your site.
Check your site free