[Micronet] google site search boxes

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

[Micronet] google site search boxes

Phyllis Orrick
Has anyone experienced any problems recently with Google site search boxes on their sites not returning nearly the appropriate number of results?

This has started to happen recently on a site I manage that's hosted by Calwebpro and one that's on a department server.

I have a ticket with the unix folks at IST, but wondered if this was something going around (like the mumps).

 
-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.
Reply | Threaded
Open this post in threaded view
|

Re: [Micronet] google site search boxes

Phyllis Orrick
Update:

Unix suggests it's our robots.txt file that's the culprit, though it predates the problem and is the same file that was on the site when Google searched worked on the other server.

Here's the text if anyone wants to take a shot.

#==============================================================================
# ROBOTS.TXT: This is the default dev/qa robots.txt file.
# ID: "$Id: robots.txt,v 1.1 2007/04/27 16:57:31 spgleb Exp spgleb $"
#==============================================================================

User-agent: *
Disallow: /


On Fri, Oct 7, 2011 at 2:59 PM, Phyllis Orrick <[hidden email]> wrote:
Has anyone experienced any problems recently with Google site search boxes on their sites not returning nearly the appropriate number of results?

This has started to happen recently on a site I manage that's hosted by Calwebpro and one that's on a department server.

I have a ticket with the unix folks at IST, but wondered if this was something going around (like the mumps).



--
Phyllis Orrick
Communications Director
Safe Transportation Research and Education Center (SafeTREC)
University of California Transportation Center (UCTC)
Institute for Urban and Regional Development (IURD)
2614 Dwight Way
UC Berkeley
Berkeley CA 94720-7374
510-643-1779
@transsafe
@californiaUTC

 
-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.
Reply | Threaded
Open this post in threaded view
|

Re: [Micronet] google site search boxes

Scot Hacker

On Oct 7, 2011, at 3:48 PM, Phyllis Orrick wrote:

> Update:
>
> Unix suggests it's our robots.txt file that's the culprit, though it predates the problem and is the same file that was on the site when Google searched worked on the other server.
>
> Here's the text if anyone wants to take a shot.
>
> #==============================================================================
> # ROBOTS.TXT: This is the default dev/qa robots.txt file.
> # ID: "$Id: robots.txt,v 1.1 2007/04/27 16:57:31 spgleb Exp spgleb $"
> #==============================================================================
>
> User-agent: *
> Disallow: /

That configuration explicitly tells search engines not to visit any pages on your site. You want to use something like this instead:

User-agent: *
Disallow: /cgi-bin/

Where you disallow only specific directories that shouldn't be indexed. I can't explain why it wasn't a problem in the past - with that configuration, you should never have had results from any legit search engine.

./s


--
Scot Hacker, Webmaster
Knight Digital Media Center
UC Berkeley Graduate School of Journalism
http://kdmc.berkeley.edu
http://twitter.com/kdmcinfo
http://kdmc.us/like-facebook
http://journalism.berkeley.edu






 
-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.