Quote:
|
Originally Posted by smudge
Scenario last night:
Google: Gimme this page. (some administration side url, fuck knows how they know to index it)
Me: 403 fuck off.
Google: Gimme this page. (same url)
Me: 403 fuck off.
Google: Gimme this page. (same url)
Me: 403 fuck off.
:: a fuckload of times::
They obviously ignore the 403. What are you supposed to do in this situation? |
Have you added the forbidden urls to your robots.txt? Googlebot is good about honouring that - if you have and it's ignoring it, then it's two options - Google gotta bug, or someone's faking a UA.