Quantcast
Channel: NBStore Discussions Rss Feed
Viewing all articles
Browse latest Browse all 4081

New Post: Deleted Products & Google Crawler

$
0
0
Ive just started having a very strange problem, that i cant work out.

Every time the google crawler goes through the site, its requesting a product that doesnt exist anymore and that is causing a 50% cpu load on w3wp.exe. I've tried IISPeek and it shows the urls that trigger this and the app pool, time in process etc show that its my sites app pool for sure and the urls that trigger it are those where product and category id's dont exist in the database. I.e:
/Products/SoilRock/tabid/104/ProdID/86/CatID/35/Default.aspx
/Products/SoilRock/tabid/104/ProdID/83/CatID/30/Default.aspx

Now obviously there are two issues here:
  • One is why are they crawling a non existent product id (almost certainly my fault)
  • Secondly why does it cause such a massive cpu load when a product id cant be found. surely if a product doesnt exist it shouldnt cause that kind of problems?
I'm trying to narrow down why they are crawling a product that doesnt exist first.
I am assuming they previously successfully crawled a page with that url and thus keep crawling it again.
Does anyone know how to either refresh/reset the pages google crawls.
Or stop them trying to crawl those pages?
Could my sitemap.aspx be the cause?
Is there something in the store that tells googles crawler what pages to crawl?

I will look at the stores code next and see if there is some way prevent searches for products/categories that dont exist from oblitorating the app pool cpu load.
I've validated the store under admin and that didnt help, (not that i know what it does under the hood)

Im at a loss for what to look for in the store, which is why im looking at the google side first and hoping someone with more knowledge of the store can offer some advise here in the mean time.

Thanks in advance...

Viewing all articles
Browse latest Browse all 4081

Trending Articles