Ran into an interesting issue with a bunch of 404s for paths that never existed, but are named like the DoubleClick ad areas on the site. Apparently Googlebot is seeing the defineSlot lines in the javascript and checking to see if they are paths. I guess the do have forward-slashes in them, but that seems a little aggressive Google 🙂

In other words, this is in the javascript:


so Google is trying to open:


which doesn’t exist and 404s

Fortunately, the first number is your doubleclick ID and all the ad areas start with it. So you can exclude that path /1234567/ (using your ID) in robots.txt to prevent Googlebot from trying to open them.