Tuesday, July 05, 2005

How do I know when Google hits my site?

My favorite way to track this is to use a default 404 file not found script that tracks access to robots.txt.

My experience has been that Google sometimes visits sites without using Googlebot as the user agent. However, they always seem to check for robots.txt. I know this because I track the IP and I can reverse trace the IP address back to Google.

http://www.nirsoft.net/utils/ipnetinfo.html

What I do is create a script that will handle all 404 - file not found requests. When I detect that the requested page is robots.txt, I write this information to a file.

I can use this information for creating cloaked pages (which I think are old school and I don't do anymore) or I can use this information for learning about how google visits my sites.

11 Comments:

Blogger Unknown said...

wholesale jordan shoes, cheap christian louboutin, Bags Louis Vuitton, ugg, Cheap Louis Vuitton Handbags, discount nike jordans, christian louboutin, Christian Louboutin Daffodile, uggs outlet, Christian Louboutin Bois Dore, christian louboutin shoes, cheap nike jordan shoes, Discount LV Handbags, uggs on sale, Cheap LV Handbags, ugg soldes, ugg boots, ugg pas cher, ugg australia, christian louboutin remise 50%, cheap jordans , Discount Louis Vuitton, discount christian louboutin, Air Jordan 11 Gamma Blue

Thursday, January 08, 2015 1:02:00 AM  
Blogger oakleyses said...

michael kors, converse pas cher, vans pas cher, tn pas cher, sac guess, coach purses, oakley pas cher, longchamp pas cher, nike roshe, michael kors, lululemon, ralph lauren pas cher, nike air max, true religion outlet, air force, nike air max, mulberry, air max, louboutin pas cher, hogan, new balance pas cher, nike free run uk, nike air max, true religion jeans, ray ban uk, hermes, ralph lauren uk, michael kors, timberland, ray ban pas cher, burberry, michael kors, nike roshe run, hollister pas cher, nike free, abercrombie and fitch, hollister, coach outlet, north face, coach factory outlet, coach outlet, true religion jeans, nike blazer, true religion jeans, north face, lacoste pas cher, sac longchamp, air jordan pas cher, kate spade handbags, vanessa bruno

Tuesday, November 17, 2015 8:58:00 PM  
Blogger John said...

coach outlet store online
coach outlet online
birkenstock
fit flops
ray ban glasses
longchamp handbags
canada goose pas cher
louis vuitton outlet store
michael kors
canada goose jackets
20161015yuanyuan

Sunday, October 16, 2016 5:14:00 AM  
Blogger Unknown said...

michael kors handbags
cheap nfl jersey
ralph lauren uk
tiffany jewellery
michael kors outlet
ugg boots
longchamp handbags
nike trainers uk
kate spade uk
nfl jerseys wholesale
20170114caihuali

Saturday, January 14, 2017 1:56:00 AM  
Blogger raybanoutlet001 said...

birkenstock sandals
bears jerseys
asics shoes
reebok shoes
chaussure louboutin
cheap michael kors handbags
nike huarache trainers
nike store uk
fitflops sale clearance
jordan shoes

Tuesday, January 17, 2017 2:20:00 AM  
Blogger chenmeinv0 said...

the north face outlet
polo shirts
burberry outlet online
mont blanc mountain pen
ray ban wayfarer
louis vuitton handbags
oakley sunglasses
ugg outlet
abercrombie and fitch
ralph lauren outlet
hzx20170120

Friday, January 20, 2017 1:41:00 AM  
Blogger Unknown said...

cheap ray bans
true religion jeans
burberry outlet
mulberry handbags
coach outlet online
ugg uk
rolex watches
replica watches
toms shoes
nike air max uk
chanyuan2017.01.22

Saturday, January 21, 2017 11:48:00 PM  
Blogger Unknown said...

jianbin0317
ralph lauren shirts
ray ban sunglasses
rolex outlet
salomon shoes
michael kors outlet online
ferragamo outlet
oakley sunglasses wholesale
christian louboutin outlet
cazal outlet
jerseys wholesale

Thursday, March 16, 2017 8:40:00 PM  
Blogger eric yao said...

Coach Outlet ED Hardy Outlet Coach Outlet Store Online Kate Spade Outlet Cheap Jordans Coach Purses Coach Outlet Kate Spade Outlet Toms Outlet Louis Vuitton

Wednesday, July 05, 2017 6:46:00 AM  
Blogger Admin said...

Thank you for your support who has read our article. I'm very happy if you can share our article below

5 Tanaman Herbal Untuk Menyembuhkan Maag
Cara Meneyembuhkan Flu Singapura
Obat Dermatitis Paling Ampuh
Obat Sariawan Pada Lidah
Obat Untuk Gusi Bengkak

Monday, May 21, 2018 11:35:00 PM  
Anonymous Pengobatan Kejang Demam Pada Anak said...

If the disease is not treated immediately, then everything will result in harm to the health of the body. Thank you for sharing the information
Pengobatan Patah Tulang
Obat Ampuh Atasi Penyakit Malaria
Cara Mengobati Ginjal Bengkak
Obat Ginjal Bocor Untuk Semua Usia
Obat Wasir Tanpa Operasi
Pengobatan Diare Paling Ampuh Sampai Sembuh

Monday, August 13, 2018 3:00:00 AM  

Post a Comment

Subscribe to Post Comments [Atom]

<< Home