[Micronet] Google Bot Crawling issue

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

[Micronet] Google Bot Crawling issue

Luke Rockwell
Hello Micronet,

July 2 Google suddenly reduced the number of pages it crawls daily on our site alumni.berkeley.edu. We also have a news site within our site with its own sitemap that was affected the same way. I have checked rather thoroughly and found no changes in our environment that would have caused this. Further research seems to show that this is affecting the main berkeley.edu site and all sub-domains as well. Here's what I know:

On or around July 2 something changed.
Our daily pages crawled by Googlebot dropped to a few dozen from almost 1,700 pages overnight. Our web logs confirm this.
Our Impressions are down 36% and Clicks are down 28% in the last month.
Googlebot-News last crawled our site July 4.
Googlebot still crawls daily, but only a couple dozen pages.
Webmaster Tools show no errors, except there is no site error data after July 1.
Our two sitemaps for the main site (/sitemap.xml) and news (/googlenews.xml) continue to get picked up and URLs submitted, just nothing showing up in Google searches. Both are programmatically updated.
Robots.txt is unchanged and continues to test correctly.
Some few new pages are showing up in Google searches with our IP address instead of alumni.berkeley.edu in the result.
Fetch as Google does not work. It returns a Temporarily Unavailable error.
Other search engines show no change in behavior.
Google replies and public forum postings suggest it may be a result of DNS changes
 
Any help in unraveling this mystery would be appreciated.

.....................................

Luke Rockwell
Systems Administrator
Information Technology

.....................................

Cal Alumni Association | UC Berkeley
1 Alumni House, Berkeley, CA 94720
T 510.900.8196
F 510.642.6252

.....................................

Commitment, Support, Passion

_____________________________________

alumni.berkeley.edu

_____________________________________

Facebook | LinkedIn

 
-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.
Reply | Threaded
Open this post in threaded view
|

Re: [Micronet] Google Bot Crawling issue

Jay Bryon
If DNS issues are suspected, I'd recommend opening a ticket with hostmaster to begin an investigation.  


-J


On Fri, Aug 1, 2014 at 11:51 AM, Luke Rockwell <[hidden email]> wrote:
Hello Micronet,

July 2 Google suddenly reduced the number of pages it crawls daily on our site alumni.berkeley.edu. We also have a news site within our site with its own sitemap that was affected the same way. I have checked rather thoroughly and found no changes in our environment that would have caused this. Further research seems to show that this is affecting the main berkeley.edu site and all sub-domains as well. Here's what I know:

On or around July 2 something changed.
Our daily pages crawled by Googlebot dropped to a few dozen from almost 1,700 pages overnight. Our web logs confirm this.
Our Impressions are down 36% and Clicks are down 28% in the last month.
Googlebot-News last crawled our site July 4.
Googlebot still crawls daily, but only a couple dozen pages.
Webmaster Tools show no errors, except there is no site error data after July 1.
Our two sitemaps for the main site (/sitemap.xml) and news (/googlenews.xml) continue to get picked up and URLs submitted, just nothing showing up in Google searches. Both are programmatically updated.
Robots.txt is unchanged and continues to test correctly.
Some few new pages are showing up in Google searches with our IP address instead of alumni.berkeley.edu in the result.
Fetch as Google does not work. It returns a Temporarily Unavailable error.
Other search engines show no change in behavior.
Google replies and public forum postings suggest it may be a result of DNS changes
 
Any help in unraveling this mystery would be appreciated.

.....................................

Luke Rockwell
Systems Administrator
Information Technology

.....................................

Cal Alumni Association | UC Berkeley
1 Alumni House, Berkeley, CA 94720
T 510.900.8196
F 510.642.6252

.....................................

Commitment, Support, Passion

_____________________________________

alumni.berkeley.edu

_____________________________________

Facebook | LinkedIn


-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.




--
Jay Bryon
Senior Network Engineer, U.C. Berkeley/IST/IS/Network Operations and Services 
2-5636


 
-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.
Reply | Threaded
Open this post in threaded view
|

Re: [Micronet] Google Bot Crawling issue

Drew Mason
In reply to this post by Luke Rockwell
Luke,

I'm not sure if this has any significance, but the timing coincides with the 7.x-2.0 release of the google_analytics module for Drupal, which was announced as a security release and a version upgrade.

Drew Mason | AV/IT Specialist


On Fri, Aug 1, 2014 at 11:51 AM, Luke Rockwell <[hidden email]> wrote:
Hello Micronet,

July 2 Google suddenly reduced the number of pages it crawls daily on our site alumni.berkeley.edu. We also have a news site within our site with its own sitemap that was affected the same way. I have checked rather thoroughly and found no changes in our environment that would have caused this. Further research seems to show that this is affecting the main berkeley.edu site and all sub-domains as well. Here's what I know:

On or around July 2 something changed.
Our daily pages crawled by Googlebot dropped to a few dozen from almost 1,700 pages overnight. Our web logs confirm this.
Our Impressions are down 36% and Clicks are down 28% in the last month.
Googlebot-News last crawled our site July 4.
Googlebot still crawls daily, but only a couple dozen pages.
Webmaster Tools show no errors, except there is no site error data after July 1.
Our two sitemaps for the main site (/sitemap.xml) and news (/googlenews.xml) continue to get picked up and URLs submitted, just nothing showing up in Google searches. Both are programmatically updated.
Robots.txt is unchanged and continues to test correctly.
Some few new pages are showing up in Google searches with our IP address instead of alumni.berkeley.edu in the result.
Fetch as Google does not work. It returns a Temporarily Unavailable error.
Other search engines show no change in behavior.
Google replies and public forum postings suggest it may be a result of DNS changes
 
Any help in unraveling this mystery would be appreciated.

.....................................

Luke Rockwell
Systems Administrator
Information Technology

.....................................

Cal Alumni Association | UC Berkeley
1 Alumni House, Berkeley, CA 94720
T 510.900.8196
F 510.642.6252

.....................................

Commitment, Support, Passion

_____________________________________

alumni.berkeley.edu

_____________________________________

Facebook | LinkedIn


-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.



 
-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.
Reply | Threaded
Open this post in threaded view
|

Re: [Micronet] Google Bot Crawling issue

Luke Rockwell
Drew,

We didn't apply that module till July 15th

.....................................

Luke Rockwell
Systems Administrator
Information Technology

.....................................

Cal Alumni Association | UC Berkeley
1 Alumni House, Berkeley, CA 94720
T 510.900.8196
F 510.642.6252

.....................................

Commitment, Support, Passion

_____________________________________

alumni.berkeley.edu

_____________________________________

Facebook | LinkedIn


On Fri, Aug 1, 2014 at 1:33 PM, Drew Mason <[hidden email]> wrote:
Luke,

I'm not sure if this has any significance, but the timing coincides with the 7.x-2.0 release of the google_analytics module for Drupal, which was announced as a security release and a version upgrade.

Drew Mason | AV/IT Specialist


On Fri, Aug 1, 2014 at 11:51 AM, Luke Rockwell <[hidden email]> wrote:
Hello Micronet,

July 2 Google suddenly reduced the number of pages it crawls daily on our site alumni.berkeley.edu. We also have a news site within our site with its own sitemap that was affected the same way. I have checked rather thoroughly and found no changes in our environment that would have caused this. Further research seems to show that this is affecting the main berkeley.edu site and all sub-domains as well. Here's what I know:

On or around July 2 something changed.
Our daily pages crawled by Googlebot dropped to a few dozen from almost 1,700 pages overnight. Our web logs confirm this.
Our Impressions are down 36% and Clicks are down 28% in the last month.
Googlebot-News last crawled our site July 4.
Googlebot still crawls daily, but only a couple dozen pages.
Webmaster Tools show no errors, except there is no site error data after July 1.
Our two sitemaps for the main site (/sitemap.xml) and news (/googlenews.xml) continue to get picked up and URLs submitted, just nothing showing up in Google searches. Both are programmatically updated.
Robots.txt is unchanged and continues to test correctly.
Some few new pages are showing up in Google searches with our IP address instead of alumni.berkeley.edu in the result.
Fetch as Google does not work. It returns a Temporarily Unavailable error.
Other search engines show no change in behavior.
Google replies and public forum postings suggest it may be a result of DNS changes
 
Any help in unraveling this mystery would be appreciated.

.....................................

Luke Rockwell
Systems Administrator
Information Technology

.....................................

Cal Alumni Association | UC Berkeley
1 Alumni House, Berkeley, CA 94720
T 510.900.8196
F 510.642.6252

.....................................

Commitment, Support, Passion

_____________________________________

alumni.berkeley.edu

_____________________________________

Facebook | LinkedIn


-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.




 
-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.