Tuesday, July 05, 2005

How do I know when Google hits my site?

My favorite way to track this is to use a default 404 file not found script that tracks access to robots.txt.

My experience has been that Google sometimes visits sites without using Googlebot as the user agent. However, they always seem to check for robots.txt. I know this because I track the IP and I can reverse trace the IP address back to Google.

http://www.nirsoft.net/utils/ipnetinfo.html

What I do is create a script that will handle all 404 - file not found requests. When I detect that the requested page is robots.txt, I write this information to a file.

I can use this information for creating cloaked pages (which I think are old school and I don't do anymore) or I can use this information for learning about how google visits my sites.

17 Comments:

Blogger Jordans Nike said...

cheap jordan shoes, cheap nike jordan shoes, air jordan shoes, wholesale jordan shoes for sale, cheap jordans, wholesale jordan shoes, Cheap jordans, wholesale jordan shoes, jordan shoes for cheap, cheap jordan, cheap jordans, Cheap Nike jordan shoes, discount nike jordans

Wednesday, November 05, 2014 11:59:00 PM  
Blogger co wendy said...

wholesale jordan shoes, cheap christian louboutin, Bags Louis Vuitton, ugg, Cheap Louis Vuitton Handbags, discount nike jordans, christian louboutin, Christian Louboutin Daffodile, uggs outlet, Christian Louboutin Bois Dore, christian louboutin shoes, cheap nike jordan shoes, Discount LV Handbags, uggs on sale, Cheap LV Handbags, ugg soldes, ugg boots, ugg pas cher, ugg australia, christian louboutin remise 50%, cheap jordans , Discount Louis Vuitton, discount christian louboutin, Air Jordan 11 Gamma Blue

Thursday, January 08, 2015 1:02:00 AM  
Blogger Qing Cai said...

fred perry polo shirts
true religion outlet
jordan 11
valentino shoes
timberland shoes
adidas wings
christian louboutin outlet
louis vuitton outlet online
tiffany and co jewelry
burberry outlet online
cheap nhl jerseys
toms outlet
oakley sunglasses
fred perry shirts
swarovski jewelry
kate spade outlet
lacoste shirts
true religion jeans
michael kors outlet online
abercrombie outlet
louis vuitton handbags
coach outlet store
tiffany and co
thomas sabo
kate spade outlet
lacoste polo shirts
foamposite gold
hollister clothing store
tod's sale
karen millen dresses
cai2015402

Wednesday, April 01, 2015 9:27:00 PM  
Blogger oakleyses said...

prada outlet, tiffany and co, michael kors outlet, ray ban sunglasses, replica watches, nike air max, uggs on sale, replica watches, longchamp outlet, louis vuitton outlet, longchamp, prada handbags, michael kors outlet, louis vuitton, christian louboutin outlet, chanel handbags, louboutin, tiffany jewelry, ugg boots, burberry, louboutin shoes, ugg boots, longchamp outlet, louis vuitton, polo ralph lauren outlet, michael kors outlet, michael kors outlet, nike free, gucci outlet, ugg boots, louboutin outlet, kate spade outlet, jordan shoes, louis vuitton outlet, ray ban sunglasses, michael kors outlet, oakley sunglasses, louis vuitton, ugg boots, nike air max, tory burch outlet, nike outlet, michael kors, oakley sunglasses, oakley sunglasses, cheap oakley sunglasses

Tuesday, November 17, 2015 8:56:00 PM  
Blogger oakleyses said...

michael kors, converse pas cher, vans pas cher, tn pas cher, sac guess, coach purses, oakley pas cher, longchamp pas cher, nike roshe, michael kors, lululemon, ralph lauren pas cher, nike air max, true religion outlet, air force, nike air max, mulberry, air max, louboutin pas cher, hogan, new balance pas cher, nike free run uk, nike air max, true religion jeans, ray ban uk, hermes, ralph lauren uk, michael kors, timberland, ray ban pas cher, burberry, michael kors, nike roshe run, hollister pas cher, nike free, abercrombie and fitch, hollister, coach outlet, north face, coach factory outlet, coach outlet, true religion jeans, nike blazer, true religion jeans, north face, lacoste pas cher, sac longchamp, air jordan pas cher, kate spade handbags, vanessa bruno

Tuesday, November 17, 2015 8:58:00 PM  
Blogger oakleyses said...

nfl jerseys, herve leger, jimmy choo shoes, celine handbags, north face outlet, reebok shoes, soccer jerseys, mcm handbags, new balance, mont blanc, mac cosmetics, babyliss, iphone cases, wedding dresses, beats by dre, chi flat iron, insanity workout, iphone 6 plus cases, nike air max, ipad cases, louboutin, iphone 6s cases, lululemon, timberland boots, vans shoes, iphone 5s cases, nike huarache, s5 cases, abercrombie and fitch, ferragamo shoes, hollister, bottega veneta, iphone 6 cases, instyler, iphone 6s plus cases, ghd, asics running shoes, hollister, valentino shoes, nike roshe, baseball bats, north face outlet, oakley, nike trainers, ralph lauren, p90x workout, giuseppe zanotti, birkin bag, longchamp, soccer shoes

Tuesday, November 17, 2015 9:00:00 PM  
Blogger oakleyses said...

moncler, doke gabbana outlet, ugg boots uk, barbour jackets, canada goose, ugg pas cher, moncler, replica watches, canada goose outlet, swarovski, ray ban, converse outlet, hollister, hollister, supra shoes, ugg,uggs,uggs canada, pandora jewelry, doudoune canada goose, pandora charms, moncler, louis vuitton, moncler, montre pas cher, vans, moncler, pandora jewelry, louis vuitton, michael kors outlet online, moncler outlet, thomas sabo, swarovski crystal, coach outlet, juicy couture outlet, canada goose outlet, toms shoes, canada goose, pandora charms, moncler, sac louis vuitton pas cher, canada goose uk, louis vuitton, bottes ugg, nike air max, links of london, louis vuitton, juicy couture outlet, gucci, wedding dresses, barbour, ugg,ugg australia,ugg italia, ugg australia, converse

Tuesday, November 17, 2015 9:03:00 PM  
Blogger chenlili said...

oakley sunglasses wholesale
true religion sale
louis vuitton canada
gucci belts
coach outlet store online
timberlands
air jordan 11
denver broncos jerseys
rolex watches
uggs sale
burberry handbags
uggs on sale
christian louboutin outlet
vans sneakers
louboutin shoes
oakley outlet
christian louboutin sneakers
true religion jeans
coach factory outlet
nfl jerseys
hogan scarpe
ugg boots
christian louboutin shoes
curry 3
christian louboutin sale
uggs italia
cheap jerseys
moncler coat
ugg boots
toms wedges
nike store
oakley sunglasses
louis vuitton outlet
kevin durant shoes 7
louis vuitton outlet
canada goose jackets
michael kors handbags
chenlili20160831

Tuesday, August 30, 2016 8:19:00 PM  
Blogger John said...

coach outlet store online
coach outlet online
birkenstock
fit flops
ray ban glasses
longchamp handbags
canada goose pas cher
louis vuitton outlet store
michael kors
canada goose jackets
20161015yuanyuan

Sunday, October 16, 2016 5:14:00 AM  
Blogger dada24 Xu said...

canada goose
rolex replica watches
cheap oakley sunglasses
cheap ugg boots
michael kors bags
cheap jordan shoes
replica omega watches
nike free run
pandora charms sale clearance
michael kors outlet store
zhi20170110

Monday, January 09, 2017 9:36:00 PM  
Blogger Hua Cai said...

michael kors handbags
cheap nfl jersey
ralph lauren uk
tiffany jewellery
michael kors outlet
ugg boots
longchamp handbags
nike trainers uk
kate spade uk
nfl jerseys wholesale
20170114caihuali

Saturday, January 14, 2017 1:56:00 AM  
Blogger raybanoutlet001 said...

birkenstock sandals
bears jerseys
asics shoes
reebok shoes
chaussure louboutin
cheap michael kors handbags
nike huarache trainers
nike store uk
fitflops sale clearance
jordan shoes

Tuesday, January 17, 2017 2:20:00 AM  
Blogger chenmeinv0 said...

the north face outlet
polo shirts
burberry outlet online
mont blanc mountain pen
ray ban wayfarer
louis vuitton handbags
oakley sunglasses
ugg outlet
abercrombie and fitch
ralph lauren outlet
hzx20170120

Friday, January 20, 2017 1:41:00 AM  
Blogger Unknown said...

cheap ray bans
true religion jeans
burberry outlet
mulberry handbags
coach outlet online
ugg uk
rolex watches
replica watches
toms shoes
nike air max uk
chanyuan2017.01.22

Saturday, January 21, 2017 11:48:00 PM  
Blogger 龙大猫 said...

kobe bryant shoes
kobe basketball shoes
timberland outlet
adidas ultra boost uncaged
kobe 11
nike roshe one
tiffany and co
skechers outlet
michael kors factory outlet
adidas stan smith shoes
achang

Thursday, February 23, 2017 12:22:00 AM  
Blogger Unknown said...

jianbin0317
ralph lauren shirts
ray ban sunglasses
rolex outlet
salomon shoes
michael kors outlet online
ferragamo outlet
oakley sunglasses wholesale
christian louboutin outlet
cazal outlet
jerseys wholesale

Thursday, March 16, 2017 8:40:00 PM  
Blogger Unknown said...

ed hardy clothing
cheap oakley sunglasses
michael kors handbags outlet
oakley sunglasses wholesale
rolex uk
converse shoes
versace sunglasses
michael kors handbags
swarovski outlet
fitflops
chanyuan2017.03.29

Wednesday, March 29, 2017 2:20:00 AM  

Post a Comment

Subscribe to Post Comments [Atom]

Links to this post:

Create a Link

<< Home