Web Hosting Forums

Results 1 to 14 of 14

This is a discussion on Search Engines in the Hosting Talk & Chit-chat forum
Does anyone know of a special code I can insert on my site to stop search engines from loading me into their database? Ex; google, ...

  1. #1
    Loyal Client
    Join Date
    Mar 2003
    Posts
    6

    Search Engines

    Does anyone know of a special code I can insert on my site to stop search engines from loading me into their database? Ex; google, aol, etc. etc. Thanks!

  2. #2
    Loyal Client dkadave's Avatar
    Join Date
    Sep 2001
    Location
    California, from Hawaii! Gotta love the ocean...
    Posts
    549
    I thinnk, there's a meta tag.

    Something about no-index!

    I'll do a search for you and get back to you!
    ------------------------------------------------------------------------------------------------
    http://www.dkanet.com - http://www.vi01.com
    http://www.dkahost.com
    ------------------------------------------------------------------------------------------------
    AIM: dkadave - YIM: dkadave
    E-mail: [email protected]
    ------------------------------------------------------------------------------------------------

  3. #3
    Loyal Client dkadave's Avatar
    Join Date
    Sep 2001
    Location
    California, from Hawaii! Gotta love the ocean...
    Posts
    549
    Found!

    <META NAME="ROBOTS" CONTENT="NOINDEX,NOFOLLOW">
    ------------------------------------------------------------------------------------------------
    http://www.dkanet.com - http://www.vi01.com
    http://www.dkahost.com
    ------------------------------------------------------------------------------------------------
    AIM: dkadave - YIM: dkadave
    E-mail: [email protected]
    ------------------------------------------------------------------------------------------------

  4. #4
    O_o CeleronXL's Avatar
    Join Date
    Dec 2001
    Posts
    566
    Or place a file called robots.txt in the directory you want to disallow, and put

    Useragent: googlebot
    Disallow: /

    Or something like that.. I don't think that's the right syntax at all, and I'm too lazy to search.
    "Before you critisize someone, walk a mile in their shoes. That way, when you critisize them, you're a mile away and you have their shoes."
    My Site: StarCraft Sector | My vB Forums: Forum Sector
    E-Mail: [email protected] | AIM: CeleronXL | ICQ: 118648739 | MSNM: [email protected] | YIM: celeronxl

  5. #5
    Yeah, I know a LOT! Vin DSL's Avatar
    Join Date
    Mar 2003
    Location
    Arizona Uplands
    Posts
    10,661

    Re: Search Engines

    Originally posted by Autumn
    Does anyone know of a special code I can insert on my site to stop search engines from loading me into their database? Ex; google, aol, etc. etc. Thanks!
    Here you go --- use this:

    http://submission.discoveryvip.com/metatag.htm
    DISCLAIMER Any resemblance between the views expressed above and those of the owners and operators of this system is purely coincidental. Any resemblance between these views and my own are non-deterministic. The existence of Vin DSL is questionable. The existence of views in the absence of anyone to hold them is problematic. The existence of the reader is left as an exercise in the second-order coefficient.

    No Guts, No Story! VinDSL 2010

  6. #6
    Loyal Client
    Join Date
    Mar 2003
    Posts
    6
    Thanks! I hope it will work

  7. #7
    Loyal Client
    Join Date
    Mar 2003
    Posts
    6
    Originally posted by dkadave
    Found!

    <META NAME="ROBOTS" CONTENT="NOINDEX,NOFOLLOW">
    Does that block all search engines? And do you have to put that on each page, only index, ?

  8. #8
    Loyal Client dkadave's Avatar
    Join Date
    Sep 2001
    Location
    California, from Hawaii! Gotta love the ocean...
    Posts
    549
    It does all!
    ------------------------------------------------------------------------------------------------
    http://www.dkanet.com - http://www.vi01.com
    http://www.dkahost.com
    ------------------------------------------------------------------------------------------------
    AIM: dkadave - YIM: dkadave
    E-mail: [email protected]
    ------------------------------------------------------------------------------------------------

  9. #9
    Community Leader jason's Avatar
    Join Date
    Sep 2001
    Location
    Rochester, NY
    Posts
    5,884
    Originally posted by dkadave
    It does all!
    But you need to put it on all pages...

    --Jason
    Jason Pitoniak
    Interbrite Communications
    www.interbrite.com www.kodiakskorner.com

  10. #10
    Yeah, I know a LOT! Vin DSL's Avatar
    Join Date
    Mar 2003
    Location
    Arizona Uplands
    Posts
    10,661
    Originally posted by jason
    But you need to put it on all pages...
    Yeah, you need to put them on all your pages, if you want to do it right. I don't know about your site, but looking at my stats --- probably 50% of the search engines enter through inner pages. If they only enter your site through the main page, then you don't have anything to worry about, but I think this senario would be extremely unlikely...
    DISCLAIMER Any resemblance between the views expressed above and those of the owners and operators of this system is purely coincidental. Any resemblance between these views and my own are non-deterministic. The existence of Vin DSL is questionable. The existence of views in the absence of anyone to hold them is problematic. The existence of the reader is left as an exercise in the second-order coefficient.

    No Guts, No Story! VinDSL 2010

  11. #11
    Loyal Client dkadave's Avatar
    Join Date
    Sep 2001
    Location
    California, from Hawaii! Gotta love the ocean...
    Posts
    549
    Oh, sorry. Like Vin said.

    It has to be on all pages!
    ------------------------------------------------------------------------------------------------
    http://www.dkanet.com - http://www.vi01.com
    http://www.dkahost.com
    ------------------------------------------------------------------------------------------------
    AIM: dkadave - YIM: dkadave
    E-mail: [email protected]
    ------------------------------------------------------------------------------------------------

  12. #12
    Loyal Client
    Join Date
    Apr 2006
    Posts
    40
    How about if I needed to prevent robots visiting only certain pages with sensitive contents ,
    like "thankyou" pages with login information ? It should not be passport protected but it should not appear in the Google search results.

  13. #13
    Pointy Stick Expert
    Join Date
    Nov 2002
    Posts
    141
    To prevent the crawling of your entire site by well-behaved robots, you only need a single robots.txt in your root directory with the following two lines in it:

    User-agent: *
    Disallow: /

    The above will prevent all of the major search engines from crawling your site.

    I'm not doubting Vin's observations, but if you are still getting files crawled with the above in place, the robots are not well-behaved and will likely ignore any META tags you have in place.

    More info HERE.

  14. #14
    Loyal Client the_ancient's Avatar
    Join Date
    Feb 2004
    Posts
    3,474
    Quote Originally Posted by baku View Post
    How about if I needed to prevent robots visiting only certain pages with sensitive contents ,
    like "thankyou" pages with login information ? It should not be passport protected but it should not appear in the Google search results.
    then add the Meta Tag to only these pages.
    -------------------------
    the_ancient
    MP Technology Group

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •