Here’s a great question for the Ask an SEO series by Matt Inertia. Matt writes:
Question
“Hi Chris,
I have a question for you which I’ve been trying to figure out for a few months.
If I disallow a page in robots.txt that I don’t want (or need) in the index (terms and conditions, privacy statements, logins etc) those pages are eventually removed from the Google index and their PageRank toolbar turns gray. This would indicate that those pages are not crawled or indexed and therefore do not build a PageRank.
But, is this really the case? If I am removing pages from the index using robots.txt am I inadvertently wasting PageRank by linking to those pages? Is the only way to effectively remove pages from the index and stop them building PageRank by adding nofollows to all the disallowed pages incoming links as well? I know that pages are given a gray tool bar when disallowed in robots.txt but is this a lie?!
This leads me on to the next question! Do nofollow attributes accurately cause the pages PageRank do be redistributed to the remaining followed links on that page?
Any thoughts would be appreciated!”
Answer
Thanks Matt,
The gray in your toolbar is not a lie. Google will not serve those pages in search results if you properly noindex them in your robots.txt. I’ve personally found Google to be quite obedient of the robots.txt files. But other bots might not be as polite :-)
It is best practice to nofollow links to your privacy policy and other non-relevant pages to pass PageRank only to the more important pages. The practice is known as PageRank Sculpting, or PageRank Siloing. I would definitely add the nofollow attribute to the links to those pages if you are looking to silo your PageRank to the most important pages. It does work.
Cover both bases to be safe, but don’t bother adding nofollow to the meta tags of those pages you are looking to keep out of the index. If for some crazy reason they get an IBL (inbound link) from an external site, you still want to pass that pop on to the rest of your site.
Don’t forget guys and gals, if you need some FREE SEO ADVICE then drop me a line. It’s FREEEEEEEEE!
January 27th, 2009 on 3:27 am
Cheers Chris! From my experience Google obeys robots instructions very well. Here’s a little extra literature I found about the question: http://www.stonetemple.com/articles/interview-matt-cutts.shtml
MattC – “Now, robots.txt says you are not allowed to crawl a page, and Google therefore does not crawl pages that are forbidden in robots.txt. However, they can accrue PageRank, and they can be returned in our search results.”
So, we know that noindex pages can build PageRank from external links that they happen to pick up but what matt doesnt make clear is whether the pages can accrue PageRank from internal links?
There is also a mammoth thread about the matter here: http://www.webmasterworld.com/google/3804067.htm
Cheers again!
April 16th, 2009 on 9:52 pm
You probably know that Nofollowing links to internal pages is rather questionable practise, which many experienced SEOs do not approve, as it can affect (impeed) the link flow in often uncontrollable ways.
Besides, on large sites with 1000′s dynamic pages nofollwoing is extremely hard to employ. So essentially, even if it works fine, it’s mostly suitable for small-medium size websites.
And the last but not the least – pagerank is no longer that important for rankings, so even if you do manage to distribute PR wisely to the most important pages – it’s unlikely to give those pages substantially higher rankings.
So why bother then and waste your time on this job at all?
what ar you thoughts on this?
Regards,
Andrew
May 8th, 2009 on 12:58 am
This tip is probably the most powerful one on increasing PageRank.
Make at least three pages on your site and link them as follows:
Page 1 >>>>>> Page 2
Page 2 >>>>>> Page 3
And this is the kicker
Page 3 >>>>>> Page 1
Google will give points to page 2 from page 1, then to page 3 from page 2, and then – if you link it back to page 1 – it starts all over again. I can’t even count how many points this will end up giving you. Just don’t abuse it too much – or the big sites will complain you are taking too much PR from them.
Regards
Debajyoti Das
May 18th, 2009 on 8:43 pm
There are so many controversial debate out there about this question. I do agree that pagerank is no longer that important for rankings. But it still plays a relatively important role on it. For me, I would choose to add the nofollow attribute to the links.
June 2nd, 2009 on 3:39 pm
I read the article on Page Rank Siloing and am about to get to work on my own site. But I am worried about the comments made by Natural SEO and Qingtian, am I waisting my time? Is there something more productive I can do. I just found out about page rank last month and now I hear its not all that relevant!
September 2nd, 2009 on 10:33 am
I do agree that pagerank is no longer that important for rankings. we know that noindex pages can build PageRank from external links that they happen to pick up but what matt doesnt make clear is whether the pages can accrue PageRank from internal links?
September 25th, 2009 on 11:34 pm
Pagerank sculpting actually hurts you. Matt Cutts posted about this in his blog. If you have 5 links into your site, you have 5 votes for your page. If you have 2 links that you nofollow, you don’t pass on those votes, but they disappear. Meaning, you still only have 3 link votes left on that page, whether you nofollow or not.
Therefore, it actually hurts you.
March 21st, 2010 on 9:34 pm
I think page sculpting is a waste of time. Passing a little page rank to a no index page isn’t a big deal. It definiely won’t harm whatever SEO you are doing.
March 24th, 2010 on 6:26 pm
I’ve noticed that websites with a low page rank also get a high ranking on google, so don’t worry much about PR.
May 15th, 2010 on 2:20 pm
My experience also shows that google sometimes ignores robots.txt or it might take a lot of time for google to remove pages that have already been indexed and then blocked by robots.txt
April 25th, 2011 on 8:16 pm
Quote: “Make at least three pages on your site and link them as follows:
Page 1 >>>>>> Page 2
Page 2 >>>>>> Page 3
And this is the kicker
Page 3 >>>>>> Page 1″
I have been doing this for the last 18 months with no negative effects from Google – and even after the recent Farmer update – in fact my sites are doing even better now. I use a software programme called “SEO Link Robot” to really expedite the process.
December 3rd, 2011 on 11:05 am
Hi Chirs, i also had big problems with internal linking, this video explained me a lot http://www.seomoz.org/blog/site-architecture-for-seo
February 7th, 2012 on 6:12 pm
Depends on how you do it. If you follow the post, it shows you how to do it properly. And it does not hurt you, believe me, I know from experience.