SEO News & Search Engine Updates

Yahoo Guide Lines

2009-01-07T22:13:00.001-08:00

Pages Yahoo! Wants Included in Its Search Submit Index:

Original and unique content of genuine value
Pages designed primarily for humans, with search engine considerations secondary
Hyperlinks intended to help people find interesting, related content, when applicable
Metadata (including title and description) that accurately describe the contents of a Web page
Good web design in general

Unfortunately, not all web pages contain information that is valuable to a user. Some pages are created deliberately to trick the search engine into offering inappropriate, redundant or poor-quality search results; this is often called "spam." Yahoo! does not want these pages in the index.

Pages Yahoo! Does Not Want Included in Its Search Submit Index:

Some, but not all, examples of the more common types of pages that Yahoo! does not want included:

Pages that harm accuracy, diversity or relevance of search results
Pages dedicated to directing the user to another page ("doorway pages")
Pages that have substantially the same content as other pages on the web, including affiliate content
Pages that change user preferences, reset default home pages, resize browser windows, disable back buttons, or otherwise interfere with a user's ability to navigate.
Pages with automatic software downloads including viruses, adware, spyware, or other self-installing programs
Sites with numerous, unnecessary virtual hostnames
Pages in great quantity, automatically generated, with minimal content or of little value
Pages using methods to artificially inflate search engine ranking
Pages with text that is not easily read, e.g., text that is too small or is obscured by the background of the page, or is located in an area of the page not visible to users.
Cloaking or stealth; a technique used by some web sites to deliver one page to a search engine and a different page to all other users
Cross-linking to inflate a site's apparent popularity, including participation in link exchanges or "link farms," and the use of non-navigational links
Pages built primarily for search engines or pages with excessive or off-topic keywords
Misuse or inaccurate use of competitor or brand names
Duplication of content, either by submission of multiple pages with same content, submitting the same content from multiple domains, or submitting the same content from multiple hosts
Pages that rely heavily on content or links to content created for another web site, such as affiliate content
Pages that have pop-ups, pop-unders or exit consoles
Pages that seem deceptive, fraudulent or provide a poor user experience (Descriptions that are not relevant to the page content)
Pages that result in a 404 error
Content, domains, titles, metadata or descriptions that (i) violate any applicable law or regulation; (ii) infringes in any manner any copyright, patent, trademark, trade secret or other intellectual property right of any party; (iii) breach any duty toward or rights of any person or entity including, without limitation, rights of publicity or privacy, or have otherwise resulted in any consumer fraud, product liability, tort, breach of contract, injury, damage or harm of any kind to any person or entity; (iv) are false or misleading; and/or (v) are defamatory, libelous, slanderous or threatening.
Online Gambling Sites: Yahoo! Search Submit does not accept URLs from online gambling sites. Online gambling sites are those that have gambling as their central theme, including those that accept wagers or require payment in exchange for the chance to win prizes, as well as sites that offer both information and links related primarily to the promotion of online gambling. If it comes to our attention that a site is soliciting users to participate in online gambling (including, without limitation, through email correspondence with users), we will terminate that site's feed.
Prescription Drug Sites: Prescription drug sites are those that sell prescription drug products, as well as sites that offer information or links related primarily to the sale of prescription drugs. If a site is a prescription drug site and either sells prescription drugs or facilitates the sale of prescription drugs (via links), these sites must take part in an approved qualification program. Pharmacy Checker, an independent third-party, provides a pharmacy verification program approved by Yahoo! For more information, please see http://www.pharmacychecker.com/sealprogram/choose.asp
Tobacco products: Yahoo! Search Submit does not accept URLs for tobacco related products
Adult Sites: Yahoo! Search Submit and Search Submit Express do not accept URLs from adult products sites or any other adult content. Adult sites are accepted into Yahoo! Search Submit Pro.

The Definitive List (75+) of Link Building Techniques in 2008

2008-12-16T20:19:00.000-08:00

Below, I’ve compiled a list of techniques I personally know of being utilized in the industry. It is by no means comprehensive, but is one of the most complete lists I’m aware of. I’ve tried to keep each technique as mutually exclusive as possible.
I would like this list to become definitive, so I propose this;
if you can provide 5 techniques not already covered (at the sole discretion of SEP), we will:
a. incorporate your ideas, and give you proper attribution for all to see
b. send you a $50 gift certificate
and … the person who submits the most new link building techniques by February 1st 2009, that ultimately get incorporated into this document (again, at the sole discretion of SEP), will be awarded $250.
Some of the techniques mentioned in this document are not condoned by SEP or the search engines, but are listed only to add to the comprehensiveness of the document. Only you know your own risk tolerance, and so must make the decision regarding tactics yourself.
Each of the techniques below, assumes a basic knowledge of the use of NoFollow tag, robots.txt exclusions, and the identification of bad neighbourhoods.
The Definitive List of Link Building Techniques:
1. build useful tools, that others will talk about and link to. These may be WordPress plugins (eg. a 301 Redirect Plugin for Wordpress), Firefox extensions (eg. Dave Naylor’s multi-browser SEO Tool, and 97th Floor’s Social Media Plugin), and the like
2. build fun or helpful widgets relevant to your business or industry (ie. mortgage calculators, body mass index calculators, how much is my blog worth, etc.).
3. offer badges
4. offer awards
5. issue online press releases
6. build definitive lists or resources via your blog eg. 75+ Link Building Techniques of 2008
7. write and release an ebook eg. How to SEO Your Site in 60 Minutes by Matt McGee, The SEO Handbook - 2008 by Dave Harry.
8. reciprocal linking with suppliers, clients, and complementary businesses
9. hold a contest. Sometimes even the smallest of contests can result in large numbers of links
10. guest blog on relevant sites in your industry

Image courtesy SEOResearcher.com
11. submit to quality directories
12. submit your blog postings to search engine friendly social media
13. sponsor an event, or a cause
14. provide testimonials for industry products you use
15. vanity bait others - in your blog, include positive mentions of the blog posts of others, and link freely to their work. Often, they’ll feel compelled to acknowledge you
16. comment on the blogs of others in your industry, or those of related sites.
17. send samples of your product to industry bloggers and ask them to try the products. If they like them, they may write about them. Of course the risk is, if they don’t like them, they may write about them too.
18. participate in surveys or group projects.
19. write “how to” guides or posts
20. release unique research results
21. be the first to uncover a significant news story
22. post job postings in colleges and universities
23. offer to speak at a college or university
24. help write curriculum for a college or university
25. provide a bursury or endowment to the school
26. make a financial contribution to your college or university. Often this results in a dofollow link acknowledging the contribution.
27. study your local government site to look for opportunities to get listed (ie. news, directory, etc.)
28. use ppc to drive qualified searchers to your quality content, knowing a % will link.
29. syndicate articles
30. trade articles with other webmasters
31. search for sites with dofollow ‘trackbacks’ and link to some of their blog posts.
32. carefully select your friends on social media (eg. Digg, Stumbleupon, Delicious). If they work for media (eg. newspapers, magazines, radio stations, etc.) cater the content to their apparent interests. I’ve had content mentioned on radio shows before (and linked to from the radio’s website), merely by knowing to put a ‘conspiracy’ spin on the science content.
33. internal links are crucial and important. Ensure you use proper navigation structure
34. submit to nofollow social media too, if the content is really good. If it goes hot, it will be seen by large numbers of people potentially, and will typically result in a number of good links.
35. study competitors banklinks via Yahoo Site Explorer, and request backlinks from many of the same sources
36. maintain profiles on RELEVANT social media sites with DoFollow profile pages, and heavily link those pages with other pages from the social media site (ie. lots of friends, lots of activity).
37. buy another domain with links pointing to it, and 301 redirect it
38. always carry a camera, capture interesting photos, and post them to the site and Flickr using Creative Commons licensing (requiring links back to your site).
39. include links in the footers of your RSS, so content scaped from your site contains links back to you.
40. rent some links from companies like Text Link Brokers. Be careful though. Google will penalize both the selling and buying party for paid links.
41. strike a deal to purchase a text link ad on relevant pages of a related site. Best if the link is from within a site’s actual content.
42. offer to make a contribution to charities
43. as Aaron Wall and Andy Hagans have suggested from their post 101 Link Building Tips to Market Your Website (points 62 and 63), sue a large company (get the David vs Goliath element working in your favour) or get sued by a company that people hate. Genius!
44. interview industry personalities, and post video and text versions to your blog. Others are always interested in what they have to say, and often will find reason to link to something they’ve said.
45. speak at industry events, especially if you know they offer dofollow profiles and linkbacks for such speakers.
46. apply for awards. Even if these sites do not link back to your site, news of winning awards can often be good linkable content for local media.
47. join a number of local organizations such as the local Board of Trade. Often, these sites provide dofollow member directory links.
48. offer a discount or free product samples to others who blog about your product.
49. submit your blog to RSS feed directories
50. do something remarkable such as running a marathon or triathlon
51. do something controversial, or to infuriate the industry (eg. say SEO is dying to a group of SEOs … good one Jason Calcanis, ShoeMoney). Most will link to you, not knowing they’re helping to further promote the issue
52. create niche sites for specific segments of topics of your industry, and link back to your site
53. create an industry group or panel, complete with its own website. Often competitors will not link directly to you, but they will link to sites that link to you!
54. find pages that link to you that are not in the indexes of search engines, and are not noindexed or blocked by robots.txt, and link to those pages so spiders will follow them and find the links pointing back to your site.
55. buy sponsored reviews
56. there are a number of wiki types sites on the internet, that will provide link backs if you add good quality content to their sites.
57. start a search engine friendly affiliate program
58. provide answers via some ‘Answers’ type sites, and link to your content on the matter as proof of your answer
59. provide a link (from a specific content piece) to a particular site that you wish to receive a link from, and send them large volumes of traffic. Often, this will be enough to capture their attention, and link back to your post on the subject.
60. attend related conferences and network, network, network. People are camera crazy these days, and will post pictures of most of the people the meet, often complete with links back to your site.
61. develop relationships with local media personalities, such that you are their ‘Goto’ person when they need information on a given subject.
62. some sites have a “See Us In The Press” page. When you find such sites relating to your industry, write something about them. Often, they’ll pick it up and post it to their site … everyone like to show off their cudos.
63. forum commenting
64. search keyword “add url”, keyword “add site”, and add your site to those sites
65. be a ‘top commentator’ on search engine friendly blogs offering the plugin
66. submit videos to tv channel websites (thank you for this idea Melanie Nathan) and be sure to include relevant text links.
67. point out errors (spelling, grammatical, coding) on sites where you’d like a link from. Often, thanks is provided in the form of a link.
68. develop a WordPress plugin that automatically generates links back to the author’s site from the blog where the plugin is ‘plugged in’ (this is extremely black hat!!!)
69. list yourself with the BBB (Better Business Bureau) and other local organizations. It should be noted however, that not all BBB sites offer DoFollow links
70. submit the widgets you create to widget directories
71. create customized WordPress themes, complete with properly worded attribution links
72. search for articles, sites, or posts that mention your personal or business name, though do not link to it, and then contact them and ask them to link to your site. This is courtesy of Wiep
73. if your company owns numerous web sites, have the other websites interlink. Again courtesy of Wiep
74. find companies offering the same products or services, except in different countries, regions, or even languages, and approach them to exchange links (Wiep … you’re on a roll)
75. submit your RSS feed to RSS feed directories
76. create an industry niche directory, and link back to your site

How to make sure your affiliate program passes PageRank & SEO benefits

2008-12-14T19:49:00.000-08:00

Search engines are not quite decided on whether they class affiliate links as paid links or not. If you take the time to set up an affiliate program why not use it to generate thousands of high value links to your product pages?
This post will tell you everything you need to know about maximising the SEO value of your affiliate links.

Easy: Don’t go through a 3rd party

Search engines won’t count your affiliate links if they go via a third party affiliate network. Either go with a network that allows you to use your own links or run the program in house.

Easy: Allow deep links

Most people do this already but it’s important to make sure your affiliates are linking to your product pages not just the homepage.

Harder: Consolidate your links

Most affiliate programs have links like
http://www.site.com/category/product.html?aff=123
This causes duplicate content problems - the way to fix the issue is to set an affiliate cookie and then redirect to the normal product page http://www.site.com/category/product.html

Really clever: Don’t make it look like an affiliate program

Any URL with the parameter aff=123 clearly looks like an affiliate link. Amazon is smart and uses tag= as their parameter. Why not try some of the following as affiliate links?
http://www.site.com/page/123/
http://www.site.com/product-name/page123/
http://www.site.com/blogpost/123/product-name.html
Confuse Google by using a non-standard nomenclature for your parameters.

Really clever: Intelligent use of cookies

Do you name your affiliate cookies affid? Just because Google doesn’t accept cookies doesn’t mean it doesn’t see what cookies are being sent in the header information.
When Google sees an affid cookie being set followed by a 301 redirect to strip out parameters it’s a fair assumption the link is an affiliate link.
Try calling your cookie something random like “visitor” or even cloaking the cookie so that it isn’t sent to search engines.

Super clever: Don’t use URL parameters

A few sites have started tracking based on referrer headers, this gives a clean link and search engines have no way of knowing the links are affiliate links, unless you are stupidly telling everybody about it.
My favourite trick is to use links in the following format:
http://www.site.com/#john
http://www.site.com/product-name.html#steve
Search engines view urls with different # tags as the same page so you can have as many of these as you like without coming across duplicate content issues. The way to handle tracking is to use JavaScript to parse the # tag and use it to populate a hidden form field which is posted to your shopping cart when the “add to cart” button is pressed.

Googles Page Update Life Cycle

2008-12-09T20:28:00.000-08:00

People know (or at least they should do) that implementing a number of SEO techniques and methods on any given page can influence the search rankings in a positive way. There are plenty of resources to help explain how you can create the “perfect page” in regards to SEO but are there any clear metrics for success? What can you expect if you change or alter page content, or perhaps the Meta data?
One thing we do know is that if a good SEO gets their hands on your website or specific page you will see positive results. What I’ve been doing is benchmarking when the changes take place in Google and whether the changes are positive and negative.
The Google update test
I optimised around 50 pages of a website I own that I initially setup around 3 months ago, the Meta data, page tags and content was not optimised at all. I created a strategy to optimise these pages, the actual content of these products were products i.e. one product per page. I changed the following:
1.    Optimised the meta data
2.    Included keywords and alternative keyword phrases on page
3.    Optimised the images on the page

I had read somewhere that updating large numbers of pages on a website all at once could lead to a possible penalty, although I have never seen this I thought this test would help determine this theory.
I benchmarked data over a six week period on Google, based on individual pages and their targeted keywords, which had been optimised.
Week 1
Around 80% of the pages actually increased rankings in the first week with around 15% remaining the same and only 5% dropping rank
Week 2
In the second week there were some more keyword increases and very few positions dropped - a good week all round.
Week 3
In the third week it was the complete opposite, just over 85% of the keywords dropped below their original ranking with 5% remaining the same and 10% increasing
Week 4
Huge increase of positions, now around 70% of the pages I originally optimised are ranking well above their previous position with many on page 1 or 2. Very few position drops from original positions but there were some.
Week 5
Not much movement between keyword positions but 30% of keywords have improved from week 4, 80% remain the same with around 10% dropping slightly.
Week 6
Final week and only one page has increased from week 5 while 2 pages dropped slightly, the rest remained the same.
Time for some graphs:
The first graph is has been taken as an average from over 40 optimised pages over the period of 6 weeks so visually you can see the update life cycle.

The next graph shows five randomly selected keyword behaviours over the 6 week period

The final graph shows another 10 randomly selected keywords and their position changes

Google’s Page Update Life Cycle
Yep, think that’s what I’m going to call it! Anyway I’m aware that this lifecycle of position changes probably goes on for a bit longer but the data over the 6 week period was the most active. This is something that I’ve seen many times before but have never benchmarked for such a test. It’s also worth mentioning that you do see position changes before Google has re-indexed the optimised page.
So if you go about updating pages of your site don’t worry if they go all over the place for the first month or so, if they have been optimised correctly then you should see some kind of improvement.

Advanced Website Diagnostics with Google Webmaster Tools

2008-12-09T20:26:00.000-08:00

Running a website can be complicated—so we've provided Google Webmaster Tools to help webmasters to recognize potential issues before they become real problems. Some of the issues that you can spot there are relatively small (such as having duplicate titles and descriptions), other issues can be bigger (such as your website not being reachable). While Google Webmaster Tools can't tell you exactly what you need to change, it can help you to recognize that there could be a problem that needs to be addressed.

Let's take a look at a few examples that we ran across in the Google Webmaster Help Groups:

Is your server treating Googlebot like a normal visitor?

While Googlebot tries to act like a normal user, some servers may get confused and react in strange ways. For example, although your server may work flawlessly most of the time, some servers running IIS may react with a server error (or some other action that is tied to a server error occurring) when visited by a user with Googlebot's user-agent. In the Webmaster Help Group, we've seen IIS servers return result code 500 (Server error) and result code 404 (File not found) in the "Web crawl" diagnostics section, as well as result code 302 when submitting Sitemap files. If your server is redirecting to an error page, you should make sure that we can crawl the error page and that it returns the proper result code. Once you've done that, we'll be able to show you these errors in Webmaster Tools as well. For more information about this issue and possible resolutions, please see http://todotnet.com/archive/0001/01/01/7472.aspx and http://www.kowitz.net/archive/2006/12/11/asp.net-2.0-mozilla-browser-detection-hole.aspx.

If your website is hosted on a Microsoft IIS server, also keep in mind that URLs are case-sensitive by definition (and that's how we treat them). This includes URLs in the robots.txt file, which is something that you should be careful with if your server is using URLs in a non-case-sensitive way. For example, "disallow: /paris" will block /paris but not /Paris.

Does your website have systematically broken links somewhere?

Modern content management systems (CMS) can make it easy to create issues that affect a large number of pages. Sometimes these issues are straightforward and visible when you view the pages; sometimes they're a bit harder to spot on your own. If an issue like this creates a large number of broken links, they will generally show up in the "Web crawl" diagnostics section in your Webmaster Tools account (provided those broken URLs return a proper 404 result code). In one recent case, a site had a small encoding issue in its RSS feed, resulting in over 60,000 bad URLs being found and listed in their Webmaster Tools account. As you can imagine, we would have preferred to spend time crawling content instead of these 404 errors :).

Is your website redirecting some users elsewhere?

For some websites, it can make sense to concentrate on a group of users in a certain geographic location. One method of doing that can be to redirect users located elsewhere to a different page. However, keep in mind that Googlebot might not be crawling from within your target area, so it might be redirected as well. This could mean that Googlebot will not be able to access your home page. If that happens, it's likely that Webmaster Tools will run into problems when it tries to confirm the verification code on your site, resulting in your site becoming unverified. This is not the only reason for a site becoming unverified, but if you notice this on a regular basis, it would be a good idea to investigate. On this subject, always make sure that Googlebot is treated the same way as other users from that location, otherwise that might be seen as cloaking.

Is your server unreachable when we try to crawl?

It can happen to the best of sites—servers can go down and firewalls can be overly protective. If that happens when Googlebot tries to access your site, we won't be able crawl the website and you might not even know that we tried. Luckily, we keep track of these issues and you can spot "Network unreachable" and "robots.txt unreachable" errors in your Webmaster Tools account when we can't reach your site.

Has your website been hacked?

Hackers sometimes add strange, off-topic hidden content and links to questionable pages. If it's hidden, you might not even notice it right away; but nonetheless, it can be a big problem. While the Message Center may be able to give you a warning about some kinds of hidden text, it's best if you also keep an eye out yourself. Google Webmaster Tools can show you keywords from your pages in the "What Googlebot sees" section, so you can often spot a hack there. If you see totally irrelevant keywords, it would be a good idea to investigate what's going on. You might also try setting up Google Alerts or doing queries such as [site:example.com spammy words], where "spammy words" might be words like porn, viagra, tramadol, sex or other words that your site wouldn't normally show. If you find that your site actually was hacked, I'd recommend going through our blog post about things to do after being hacked.

There are a lot of issues that can be recognized with Webmaster Tools; these are just some of the more common ones that we've seen lately. Because it can be really difficult to recognize some of these problems, it's a great idea to check your Webmaster Tools account to make sure that you catch any issues before they become real problems. If you spot something that you absolutely can't pin down, why not post in the discussion group and ask the experts there for help?

Domain Canonicalization

2008-12-08T21:08:00.000-08:00

Pop quiz: what's the difference between the following URLs:

* http://website.com
* http://www.website.com
* http://website.com/default.php
* http://www.website.com/default.php

Give up? If you're a user, then chances you expect all of those URLs will lead you to the same page. Robots, however, are not as good at determining if pages are the same, so they often store each separately. A big part of how search engines rank pages is based on how many external links those pages have. If other sites on the web link to the different versions of your home page, then search engines may calculate the value of each URL separately, based on the number of links to each version. This can effectively diminish the potential rank your page would have if it were found (and linked to) by only one URL.

The practice of consolidating all versions of a page under one URL is referred to as "canonicalization" (because you collapse all versions under the "canonical" or true version). The four examples listed above are the most common, but there are potentially many, many URLs that lead you to the same page. By adhering to several best practices, you should be able to address 90% of common site-wide canonicalization issues on your site and consequently increase how your site ranks.
Recommendation

The solution is to be explicit about the canonical form of your URLs. Following are four best practices to achieve this, with specific code and configuration examples.

1.

Select WWW or Non-WWW, then redirect the other option to your preferred version.
The hard part is choosing if you want your site to be "www.website.com" or simply "website.com". There is no right answer for every company so you'll have to figure this out on your own (but, removing the "www." saves your customers 4 keystrokes, which really add up on a mobile device, and it makes your brand the first thing your customers see).

Once you've selected, you then need to find a way to trap all requests to your application, check which form is being used, and if it is not the correct form, initiate a 301 Redirect to the correct form. For example, if the user types in wikipedia.org, they will automatically get redirected to www.wikipedia.org.
2.

Remove the default filename from the end of your URLs.
All web servers allow you to select one or more default filenames to serve when the browser requests a directory. For example, this website is run on IIS, so when the user requests "http://janeandrobot.com" we really serve "http://janeandrobot.com/default.aspx".

In the same code you use to enforce www vs. non-www, you should also check and see if the default filename is at the end of the URL and then trim it off. So, "http://janeandrobot.com/default.aspx" would be converted to "http://janeandrobot.com".
3.

Link internally to the canonical form of your URL.
Make sure you always link to the proper canonical form of your URLs from within your site. This practice helps encourage external sites to link to the site using the correct version as well (since those linking to you often cut and paste from your pages or RSS feed.) Note there is a degree of diminishing returns here, so you don't need to spend the whole weekend hunting down every last URL. Just make sure to review your site's primary navigation, top landing pages and blog.
4.

Use Google Webmaster Tools to tell Google the correct form.
Implementing these best practices on your site are ideal, since they address the problem for all search engines and give your customers a consistent, properly branded navigation experience. But what can you do if you reviewed steps 1-3 and found that it would take six months to implement on your production site? There is something that you can do today: using Google's Webmaster Tools, you can navigate to the "Tools" section and select "Set preferred domain." Here you can specify if you'd like Google to use "www.website.com" or "website.com" in their index and search results, as well as consolidate links to both versions. Note that while this will provide you short-term benefit from Google, it does not help you in Yahoo! or Live Search.

Checking Your Website

To check your website to see if you're handling domain canonicalization correctly, you can use the Live HTTP Headers add-on for Firefox.
Open the Live HTTP Headers tool, then try all the variations of the URL at several different levels to ensure they all redirect back to the appropriate canonical form. As you're checking each variation, look at the HTTP headers using the Firefox plug-in to ensure they are all 301 redirects (and not, for instance, 302 redirects).

Here's an example test case:
Canonical URL Form Test Case Test Result
http://janeandrobot.com janeandrobot.com Success
janeandrobot.com/default.aspx Success
www.janeandrobot.com Success
www.janeandrobot.com/default.aspx Success
http://janeandrobot.com/about.aspx janeandrobot.com/about.aspx Success
www.janeandrobot.com/about.aspx Success
http://janeandrobot.com/folder janeandrobot.com/folder Success
janeandrobot.com/folder/default.aspx Success
www.janeandrobot.com/folder Success
www.janeandrobot.com/folder/default.aspx Success
http://janeandrobot.com/folder/test.aspx janeandrobot.com/folder/test.aspx Success
www.janeandrobot.com/folder/test.aspx Success
Examples

Canonicalization issues are very common and being an Microsoft employee, I don't have to go far to find an example. Check out the website for Microsoft's annual Mix conference for web developers.

I was able to generate the table below by plugging the common URL variations into Yahoo's Site Explorer to find a list of links to each variation.
URL Variation Number of Links from within website Number of Links from outside websites
http://visitmix.com 17,663 59,498
http://www.visitmix.com 9,074 22,179
http://visitmix.com/default.aspx 0 22
http://www.visitmix.com/default.aspx 0 12

Looking through these numbers yields some interesting insights:

*

Not doing "www" vs "non-www" is definitely hurting their ranking - you can tell because they have a similar number of inlinks for each version. Ranking is done on a logarithmic scale, so every additional link is more valuable than the one before. If they redirected all versions to one canonical form, search engines would see their home page has having 81,711 external links, would would be a substantial boost.
*

They are not good about using the same version of the URL within their site. If you're not cognizant of this on your site, others won't be either. It looks like they use visitmix.com about 75% of the time internally, and www.visitmix.com the other 25%.

URL Referrer Tracking

2008-12-08T21:07:00.000-08:00

There may be instance when you want to track the source of a request, and a common way of doing so is by using tracking parameters in URLs. Unfortunately, implementing referrer tracking in this way can result in significant issues with search engines. In particular, it can cause duplicate content issues (since the search engine bot finds multiple valid URLs that point to the same page) and ranking issues (since all the links to the page aren't to the same URL).

Let's say that Jane and Robot uploaded two different online training seminars to YouTube as part of a viral marketing effort to drive more traffic to our site. To gauge our return on investment from each of these seminars, we've added a tracking parameter to the link within each YouTube description that a customer can click on to learn more, here are the two URLS: http://janeandrobot.com/?from=promo-seminar-1 and http://janeandrobot.com/?from=promo-seminar-2. Each would bring the customer to our home page (the same page served by http://janeandrobot.com) and we would track the conversions based on the from parameter in the URL.

While this solution may seem to work well initially, it can result in low quality tracking data and impact our search acquisition. Here's a summary of the major problems:

1.

Duplicate content - search engines sometimes have difficulty determining if two URLs contain the exact same page (see canonicalization for more information). In this case, we're creating this problem because we've created multiple URLs for the same page. Search engines are likely to find all three URLs for the home page and store/ rank them as separate content within their index. This could cause the search engine robots to crawl the page three times instead of just once (which may not be a big deal if we are only tracking two promotions, but could become a big problem if we used similar tracking parameters for many other campaigns and URLs). Not only are the robots using more bandwidth than is necessary, but since they don't crawl a site infinitely, they could spend all the allotted time crawling duplicate pages and never get to some of the good unique pages on the site.
2.

Ranking - search engines use the number of quality links pointing to a URL as a major signal in determining the authority and usefulness of that content. Because we now have three different URLs pointing to the same page, people have three choices when linking to it. The result is a lower rank for all of the variations of the URL. Search engines generally filter out duplicates, so for instance, if the original (canonical) home page has 100 incoming links and each URL with a tracking parameter has 25 links, then search engines might filter out the two URLs with fewer links and show only the canonical URL, ranking it at position eight for a particular query based on those 100 incoming links. If all incoming links were to the same URL, then search engines would count 150 links to the home page and might rank it at position three for that same query.

Another danger is that if one of the YouTube promo videos becomes exceptionally popular, its promo URL might gain more links than the original home page URL. Using this same example, if one of the promo URLs gained 200 links, search engines might choose to display it in the search results over the original home page. This could cause a confusing experience for potential customers who are looking for your home page (http://janeandrobot.com/?from=promo-seminar-1 doesn't look like a home page and searchers might be less likely to click on it, thinking it's not the page they're looking for). It's also not ideal from a branding perspective.
3. Reporting quality - as social networking sites become more popular, we become more of a sharing culture online. Many people use bookmarks, and online bookmarking sites such as Delicious, email, and other sharing sites such as Facebook, Twitter, and FriendFeed to save and share URLs. They'll click on on a URL, and if they like it, copy and paste it from the browser's address bar. If the link they're saving/sharing happens to be one of our promotional links, then they have preserved this link for all time, and everyone who clicks through the link will look identical to someone coming through the promo. This skews the reporting numbers of who went to the site after viewing the video -- which was why we set up the tracking parameters in the first place!

Implementation Options

Unfortunately there is no perfect solution for this scenario, and what works best for you depends on your infrastructure and situation. Here we've listed several common solutions that you can choose from to improve your own implementation. We generally recommend the first solution (Redirects), but there are pros and cons to each option that you should review carefully before making your decision.
Redirects (and Cookies)

The first option strives to solve the problem by trapping all of the promotional requests, recording the tracking information, then removing the tracking parameter from the URL. This can be time consuming to implement, but it is the best all-round scenario to address the three major issues listed above.

If you wanted to get fancy, and track a user's entire session based on your referral parameter, then you can use this method as well and simply set a cookie on the client machine at the same time you trap the request. This is recommended to understand the value of traffic from different sources. In either case, here are the steps you'll need to undertake:

1. Trap the incoming request - find where you web site application's logic processes the HTTP request for your page. Trap each request at that point and check if it has a tracking parameter. If it does, record this in your internal referral tracking system. You can record this either in your server logs, or in a custom referral tracking database you maintain on your own.

* If you also would like to track the entire user's session, then you should also use this opportunity to set a cookie on the client.

2. Implement the redirect - next step is to implement a 301 redirect from the current URL to the same page without the tracking parameter (or the canonical URL). Don't for get to use the cache-control attribute in the HTTP header to ensure that all the requests come to your server and don't get handled automatically in some network-based cache. Here's what a sample redirect header might look like:

301 Moved Permanently

Cache-Control: max-age=0

Note that ASP.Net and IIS both use 302 redirects by default, so you many need to manually create the 301 response code.

The way this works is that when a search engine encounters a promotional URL (http://janeandrobot.com/?from=promo-seminar-1) it issues an HTTP GET request to the URL. The HTTP response tells the search engine that this page has been permanently moved (301 Redirect) and provides the new address (the same as the old address but without the tracking parameter). The search engine then discards the first URL (with the tracking code) and only stores the second URL (without the tracking code). And everything is right in the world.

This implementation is one of the best options, but it does have some limitations:

* One downside of this method is that it requires you to manage your own referral tracking system. Because it traps the referral parameters and removes them from the URL before the page actually loads, 3rd party referral tracking applications like Google Analytics, Omniture, WebTrends or Microsoft adCenter Analytics will not be able to track these referrals.

URL Fragment

A simple and elegant option is to simply place the tracking parameter behind a hash mark in the URL, creating a URL fragment. Traditionally, these are used to denote links within a page, and are ignored completely by search engines. In fact, they simply truncate the URL fragment from the URL.

Old URL

* http://janeandrobot.com/?from=promo-seminar-1
* http://janeandrobot.com/?from=promo-seminar-2

New URL with URL Fragment

* http://janeandrobot.com/#from=promo-seminar-1
* http://janeandrobot.com/#from=promo-seminar-1

By default Google Analytics will ignore the fragment as well, however there is a simple work around that was provided to us by Avinash Kaushik, Google's web metrics evangelist. Using the following JavaScript:

var pageTracker = _gat._getTracker("UA-12345-1");

// Solution for domain level only

pageTracker._trackPageview(document.location.pathname + "/" + document.location.hash);

// If you have a path included in the URL as well

pageTracker._trackPageview(document.location.pathname + document.location.search +

"/" + document.location.hash);

You can create a few additional variations of this if you also have additional queries in the URL you would like to track. Check with your web analytics provider to find out if you need to customize your implementation to account for using URL fragments for tracking.

Does this sound too simple and easy to be true? There are a couple downsides to this approach:

* This option fixes issues 1 (duplicate content) & 2 (ranking) listed above, but it will not address the 3rd issue of reporting. You could still encounter some reporting issues using this method if people are bookmarking or emailing around the URL.
* Typically you'll have to write some custom code to parse the URL fragment. Since it's a non-standard implementation, standard methods many not support this.

Robots Exclusion Protocol

Another relatively simple solution is to use robots.txt to ensure that search engines are not indexing URLs that contain tracking parameters. This method enables you to ensure that the original (canonical) version of the URL is always the one indexed and avoids the duplicate content issues involving indexing and bandwidth.

Assuming that all of our tracking parameters will follow a similar pattern to this:

http://janeandrobot.com/?from=

we can easily create a pattern that will match for this. Below is a robots.txt file that implements the pattern:

# Sample Robots.txt file, single query parameter

User-agent: *

Disallow: /?from=

The first line means that this rule should apply to all search engines (or robots crawling your site), and the second line tells them that they can't index any URLs that start with 'janeandrobot.com/?from=' and some type of promotional code of any length. See complete information on using the Robots Exclusion Protocol. Use this pattern if you will have multiple query parameters:

# Sample Robots.txt file, multiple query parameters

User-agent: *

Disallow: /*from=

Once you've implemented the pattern appropriate for your site, you can easily check to see if it is working correctly by using the Google Webmaster Tools robots.txt analysis tool. It enables you to test specific URLs against a test robots.txt file. Note that although this tool tests GoogleBot specifically, all the major search engines support the same pattern matching rules. In Google Webmaster Tools:

1. Add the site, then click Tools > Analyze robots.txt. (Unlike most features in Google Webmaster Tools, you don't need to verify ownership of the site to use the robots.txt analysis tool). The tool displays the current robots.txt file.
2. Modify this file with the Disallow line for the tracking parameter. (If the site doesn't yet have a robots.txt file, you'll need to copy in both the User-agent and Disallow lines.)
3. In the Test URLs box, add a couple of the URLs you want to block. Also add a few URLs you do want indexed (such as the original version of the URL that you're adding tracking parameters to).
4. Click Check. The tool displays how Googlebot would interpret the robots.txt file and if each URL you are testing would be blocked or allowed.

At this point you may be thinking, wow, I can do all this and not have to write any new code? Unfortunately, there are even more downsides to this approach than the others:

* This option will fix issue 1 (duplicate content), but not issues 2 (ranking) and 3 (reporting). This can be a good interim solution while you're implementing the more complete redirects solution, but it often isn't useful enough on its own.
* Likely this will take a little bit of extra testing to ensure you get the patterns correct in your robots.txt file and don't inadvertently block content you want indexed.

Yahoo Site Explorer

Yahoo provides an online tool designed to solve this scenario. However, the solution only helps with Yahoo search traffic. To use the Yahoo fix, simply go to http://siteexplorer.search.yahoo.com and create an account for your web site in the Yahoo Site Explorer tool. Once you've verified ownership of your web site, you can use their Dynamic URL Rewriting tool to indicate which parameters in your URLs Yahoo should ignore.

image

Simply specify the name of the parameter you use for referral tracking (in our example it is 'from'), and set the action 'Remove from URLs'. Yahoo will then remove that parameter from all of your URLs while processing them and give you a handy little report about how many URLs where impacted.

Again, this is another solution that seems too easy to be true, but again, there are some significant limitations with this approach:

* At the end of the day this is still a Yahoo-only solution. With approximately 20% market share, it is likely this will not meet all of your needs. However, if you do get some percentage of your traffic from Yahoo, there is no harm in doing this in the short term while you implement another method in the longer term.
* The other problem with this solution is that it doesn't solve issue #3 (reporting), so you are still susceptible to reporting errors due to folks bookmarking and emailing your URLs with tracking codes.

Common Pitfalls
Cloaking & Conditional Redirects

Some web sites and SEO consultants attempt to solve this by a technique called cloaking or conditional redirects. Essentially what these methods do is check if the HTTP GET request is coming from a search engine and then show them something different than normal users see. This something different could be a simple 301 redirect back to the page without the tracking parameter similar to our first solution above. The difference is that our solution implemented this redirect for all requesters, and cloaking/ conditional redirects implement it only for search engines.

The big problem with this implementation method is that cloaking and conditional redirects are explicitly prohibited in the webmaster guidelines for Google, Yahoo and Live Search. If you use this method, you risk your pages being penalized or banned by the search engines. The primary reason they prohibit this behaviors is because they want to know exactly what content they are presenting searchers using their service. When a web site shows something different to a search engine robot than to a general user, a search engine can never be sure what the user will see when they go to the web site. So, even if you're thinking of implementing cloaking for what seems to be a valid, and not deceptive, reason, it's still a technique search engines strongly discourage.

This leads to the second major problem with this implementation method - it adds significant complication and can be difficult to monitor whether or not it's working - e.g. you have to test it pretending to be each of the 3 search engines robots. When things go wrong, it is likely that you're not going to see it right away, and by the time you do, your search engine traffic may already be impacted. Check out this example when Nike ran into an issue with cloaking.
Crazy Tracking Codes

Many studies on the web that show customers prefer short, understandable URLs over long complicated ones, and are more likely to click on them in the search results. In addition, users prefer descriptive keywords in URLs. Therefore, it might be worth your time to spend a few extra minutes thinking about the tracking codes you use to see if you can make them friendlier.

Good examples

* ?from=promo
* ?from=developer-video
* ?partner=a768sdf129

Bad examples

* ?i=A768SDF129,re23ADFA,style-23423,date-2008-02-01&page=2
* ?IAmSpyingOnYou=a768sdf129&YouAreASucker=re23adfd

Testing Your Implementation

So you've implemented your new favorite method, it compiles on your dev box, and now it's time to roll it into production, right? Maybe not! The initial goal of referrer URL-based tracking was to understand where your traffic was coming from so you can use that information to optimize your business. To ensure the data your collecting is actually useful, we highly recommend that you do some testing to ensure that all the common scenarios are working the way you expect, and you know where the holes are in your measurement capabilities. As with all metrics on the web, there will be holes in your data so you need to know what they are and account for them.

The first step in testing the implementation is to try it with a test parameter, walking the full scenario through start to finish.

1. Create several phoney promotional links that reflect the actual types of links you expect. This could be on your home page, product pages or with many additional query parameters that you might encounter.
2. Place these fake promotional links in a location that won't confuse your customers but are likely to get indexed by search engines. Using a social networking site or a blog might serve this well.
3. Click through those links as a customer and verify that you get to the correct page with a good user experience. Be sure to take these into account as well:
* Redirects operating properly (if you're using them) - use the Live HTTP Headers tool in FireFox to ensure the application is providing the correct headers (301 redirect and caching).
* Major browsers all work- if you're using cookies, you should test all the major browsers to ensure that they support cookies and that your scenario works the way you might expect. Don't forget to try common mobile browsers if your customers access your site this way.
4. Check out the search engine experience to ensure that you're not running into the duplicate content or ranking issues.
* Major Engines submit URL - if you place the test URLs in the right social network or place on your blog, they should get indexed within a week or so. If they don't you can also try the "submit a URL" from Google, Yahoo and Microsoft, though they are not guaranteed to work. Essentially you want to make sure the search engines have had the opportunity to see these URLs.
* Use 'site:' command to ensure tracking URLs are not indexed - here's an example query in Google, Yahoo, and Microsoft showing that our Jane and Robot example promotional URLs are not indexed.
5. Take a look at your metrics and ensure the numbers you're recording correlate to the testing you are doing. Some additional things to consider:
* Internal referrals - you might also want to add some logic to your application to filter out (or exclude) all referrals from the development team and your own employees. This is often done by checking requests against a list of known employee or company IP addresses and scrubbing those from your tracking data.
* Caching Issues - you might also want to try out several scenarios with multiple subsequent requests. You'll want to ensure that every request is going to your server and not getting cached somewhere along the way.

Best Robots.txt Tools: Generators and Analyzers

2008-12-08T21:06:00.002-08:00

While I do not encourage anyone to rely too much on Robots.txt tools (you should either make your best to understand the syntax yourself or turn to an experienced consultant to avoid any issues), the Robots.txt generators and checkers I am listing below will hopefully be of additional help:
Robots.txt generators:

Common procedure:

1. choose default / global commands (e.g. allow/disallow all robots);
2. choose files or directories blocked for all robots;
3. choose user-agent specific commands:
1. choose action;
2. choose a specific robot to be blocked.

As a general rule of thumb, I don’t recommend using Robots.txt generators for the simple reason: don’t create any advanced (i.e. non default) Robots.txt file until you are 100% sure you understand what you are blocking with it. But still I am listing two most trustworthy generators to check:

* Google Webmaster tools: Robots.txt generator allows to create simple Robots.txt files. What I like most about this tool is that it automatically adds all global commands to each specific user agent commands (helping thus to avoid one of the most common mistakes):

Google Robots.txt generator

* SEObook Robots.txt generator unfortunately misses the above feature but it is really easy (and fun) to use:

SEObook Robots.txt generator
Robots.txt checkers:

* Google Webmaster tools: Robots.txt analyzer “translates” what your Robots.txt dictates to the Googlebot:

Google Robots.txt analyzer

* Robots.txt Syntax Checker finds some common errors within your file by checking for whitespace separated lists, not widely supported standards, wildcard usage, etc.
* A Validator for Robots.txt Files also checks for syntax errors and confirms correct directory paths.

Leveraging Webmaster tools for SEO Success

2008-12-08T21:06:00.001-08:00

Google webmaster tools panel has been around for quite some time. The Live webmaster tools joined a bit later. I get a lot of questions from colleagues and clients whether these are really helpful, and lately I found myself spending more and more time within these places finding useful information and amending my SEO strategy accordingly.
Webmaster tools 101

So why do we need it and what’s in it for me? Well, in general, the webmaster tools panel is the only legitimate and official feedback platform which the engines supply us. That on its own, is more than enough for me to say “hey, let’s take a look at that these guys have to say for themselves”.

the very basic features are:

* One place to submit my standard XML site maps directly to the engines
* Feedback on crawling errors and problems
* Feedback on duplicate content issues (Google only)
* Feedback and settings for regional and domain name issues
* Control over “sitelinks” feature (Google only)
Leveraging Webmaster tools for SEO Success

While these tool are mostly ‘nice to have’, most are not very applicable. in the following clause I will give a few pointers as to how webmaster tools research became my primary place to look for root cause of indexing and ranking issues and how they can be solved using a few examples.

* Feed the crawlers with your content
Create XML site maps, use proper formatting and put only relevant content. make sure your sitemaps update automatically so Google doesn’t have to crawl the same content again and again.

* Find & Fix Broken links: lists of broken links from the webmaster tools panel will help us fix these broken links which can easily disturb our users. Moreover, they cause the crawlers extra work which ends up with bad results, resulting in reduced indexing and credibility for the entire web site.

* Find Server Problems
the “crawl stats” panel shows last 90 of google’s crawling history on your site: How many pages were downloaded every day, how much data was downloaded and the bandwidth it took. optimizing server performance and code standards can improve these parameters significantly.

* Fix Duplicate Content issues
Google webmaster tools reports duplicate descriptions and titles under the diagnostics > content area. Google shows samples of duplications they found including the actual duplicate links. All you need to do is go back to the programmers and fix the problems and you have more spider food.

* Improve crawling Efficiency
In the past, it was a well known habit to bombard Google with as much content and pages possible. This was, to say the least a bad concept. While we do want Google to take as much content from our website, havin them index our “terms of use” over and over again is really a waste of resources, and based on the assumption that every website received a certain data and bandwidth quote from the crawlers, we want them to maximize the effectiveness of their visit.
Therefore the proper use of robotx.txt directive and nofollow links (internal too) may help to guide Google to our core content and the important pages of the web site.

* Remove unwanted content
using the manual remove you can remove certain pages and even complete websites from the google index fast and at ease. The requirement is that the pages either have a block directive in the robots.txt file, noindex meta tag or to return a 404 - then you can simply enter pages from yur site you want removed an they will be removed almost immediately.
This is good also to improve the crawling and ranking efficiency.

* Find bad outgoing links
A new feature from the Live webmaster tools panel enables reporting of pages suspected as malware as well as outgoing links pointing at malware pages. This is really cool and important and may solve a lot of problems.

* Pump up your link development strategy
Both engines show you exactly which pages are linking to your pages from within your domain and from outside your domain. Analyzing that list can help a lot to understand how engines see your incoming links, where you need to improve and how to do it.

* Improve your regional rankings
Google webmaster tools and Live webmaster tools both offer the option to set your geo-target which may contradict your actual server or IP address location. Moreover, Live webmaster tools offers a very coo feature which presents the nciming links to the domain with their regional setting therefore you can easily see where most your links come from and optimize your link strategy accordingly.

Do Search Engines Use Bounce Rate As A Ranking Factor?

2008-12-08T20:11:00.000-08:00

Your web site’s bounce rate may be a significant factor in your search engine rankings. If the bounce rate on your site is high, you could end up with lower rankings in the search engines. Correspondingly, lower bounce rates may actually offer meaningful ranking boosts. (Don’t know what a bounce rate is? Hang on—definitions below.)

Don’t believe that bounce rate is a serious ranking factor? You should. A new study by SEO Black Hat shows some significant impact in a web site’s rankings as a result of significant changes in bounce rate. It is, of course, possible that the data in the study have been affected by other factors taking place at the same time, so this one study is not be any means conclusive.

I speculated on bounce rate as a ranking factor back in August of 2007. The underlying fact is that search engines want quality sites in their search results. High bounce rates may be a very good indicator of a poor site experience, or worse still, a complete mismatch between the content of the site, and the search query entered by the user. This provides heavy motivation to look at bounce rates as a meaningful SEO factor, and how to minimize bounce rates in the hope of increasing your rankings.

Bounce rate defined

What’s bounce rate? Google Analytics defines a bounce as any visit where the visitor views only one page on the site, and then does something else. What happens next? There are several possibilities, including the user clicking on a link to a page on a different web site, closing an open window or tab, typing a new URL, clicking the “back” button to leave the site, or perhaps the user doesn’t do anything and a session timeout occurs. This is still a bit fuzzy because of the nature of how “sessions” are defined in analytics packages. Analytics software that relies on Javascript tags only know when someone loads a page of your web site (so the Javascript runs).

As a result, these analytics packages have difficulty determining what happens in the meantime. In our first scenario, user A comes to your site, views one page, and then goes to a three-martini lunch (well, hopefully not). They then come back and visit 10 other pages on your site. Because of the way that analytics packages work, this will be seen as two different visits, and the first one will be recorded as a bounce (a single page visit).

For search engines there are other possibilities. For example, a bounce could be defined as user A entering a search query, going to your web site, returning to the search engine, and clicking on another result. Another possible definition involves user A entering a search query, going to your site, and returning to the search results page in less than “x” seconds. So there are a few possibilities of these kinds for the search engines to experiment with.

Of course, the major search engines have other data available as well. For example, they have toolbars which can be a rich treasure trove of data for tracking user actions. In addition, search engines license data from major ISPs and collect additional data to track where a user goes. The possibilities go well beyond what an analytics package can do.

There are some issues, of course. What happens if the user is looking for a single bit of information, such as Abe Lincoln’s birthplace. If a search result leads to a good reference site, the user gets the answer and are done. They may still click back to the search results and search on something else, or click on another result from the original search. Even though this is a satisfactory outcome, it’s recorded as a bounce by most analytics packages.

This type of scenario will be prevalent with users who are searching for a simple answer and get their responses from almanac-like information sites. The key here is to also try and factor comparative data in the way that bounce rate is used for influencing rankings. For example, the bounce rate of almanac sites may be higher than the bounce rate of an e-commerce site, which will likely be higher than the bounce rate of a directory site (where users have a high probability of going on to another site).

My guess is that search engines look (or will look) to determine how a site’s bounce rate compares to other sites that are comparable, or other sites that it in considering returning for a user’s search query. In this latter scenario, you could imagine a run-time adjustment where the search engine comes back with the “traditional” results to a user’s query, and then make a bounce rate adjustment.

The bottom line

There is a real possibility that bounce rate is a significant ranking factor right now. Even if it isn’t, it is my opinion that it will be made a factor in the near future. Even if it is not a factor, and even if it does not become one, there are plenty of reasons to look closely a bounce rate anyway.

It speaks to the conversion potential of your site, and this already gives you plenty of reason to look at it. You can read more about this aspect of bounce rate in this excellent post by analytics guru Avinash Kaushik, titled Standard Metrics Revisited: #3: Bounce Rate.

Yahoo! Search BOSS

2008-12-08T20:10:00.000-08:00

BOSS (Build your Own Search Service) is Yahoo!'s open search web services platform. The goal of BOSS is simple: to foster innovation in the search industry. Developers, start- ups, and large Internet companies can use BOSS to build and launch web-scale search products that utilize the entire Yahoo! Search index. BOSS gives you access to Yahoo!'s investments in crawling and indexing, ranking and relevancy algorithms, and powerful infrastructure. By combining your unique assets and ideas with our search technology assets, BOSS is a platform for the next generation of search innovation, serving hundreds of millions of users across the Web.
How Do I Get Started?

1. Check out BOSS specs and mash-up examples below
2. Review the documentation
3. Get a BOSS Application ID

Using the API or Web Service
Overview

Search APIs are nothing new, but typically they've included rate limits, strict terms of service regarding the re-ordering and presentation of results, and provided little or no opportunity for monetization. These constraints have limited the innovation and commercial viability of new search solutions.

BOSS (Build your Own Search Service) is different – it's a truly open API with as few rules and limitations as possible. With BOSS, developers and start-ups now have the technology and infrastructure to build next generation search solutions that can compete head-to-head with the principals in the search industry. BOSS will grow and evolve with a focus on providing additional functionality, tools, and data for developers.
Previously Available with Yahoo! Search API Available with BOSS
Queries Per Day 5,000 Unlimited*
No Restrictions on Presentation no yes
Re-Ordering Allowed no yes
Blending of Proprietary and Yahoo! Search Content Allowed no yes
Monetization no Coming Soon!
White-Label Attribution Required yes

* BOSS offers developers unlimited daily queries, though Yahoo! reserves the right to limit unintended usage, such as automated querying by bots.
Examples

hakia

hakia, a leading semantic search engine, uses Yahoo! Search BOSS to accelerate its semantic analysis of the Web by accessing the vast amounts of web documents in the Yahoo! Search index.

Me.dium Search

Me.dium combined the BOSS API with its insight into the real time surfing activity of the crowds to build a unique "Crowd-Powered" social search engine prototype.

Daylife

Daylife To-Go is a new self-service, hosted publishing platform from Daylife. Anyone can use this platform to automatically generate 100% customizable pages and widgets. Daylife To-Go uses the BOSS API platform to power its Web search module.

Cluuz

Cluuz generates easier to understand search results through patent pending semantic cluster graphs, image extraction, and tag clouds. The Cluuz analysis is performed in real-time on results returned from BOSS API.
Revenue Sharing

In the near future, we will launch a monetization platform enabling Yahoo! and partners to jointly participate in the economics of BOSS-powered search products. Either Yahoo! sponsored search integration, with certain implementation and exclusivity requirements, or potentially a payment model, will be required above a specified query threshold.
Terms of Use

Use of this service is subject to the BOSS API Terms of Use.
Learn More
BOSS Mashup Framework

The BOSS Mashup Framework is an experimental python library that provides developers with tools for mashing up the BOSS API with other third-party data sources.
BOSS Custom

BOSS Custom is an invite-only program focused on building highly scalable next generation search products. This program is designed for consumer web businesses with unique assets (such as extensive user data or novel technologies) that want to develop truly innovative search products using Yahoo!'s core search technology.

Pagination and Duplicate Content Issues

2008-12-07T21:22:00.000-08:00

Very often with large dynamic sites webmasters just can’t do without pagination. Apart from usability here they come across another serious issue: duplicate content.
The thing is that all pages normally have identical titles and meta description and while some SEOs think it’s not an issue at all (as search engines “don’t see identical titles as duplicate content”), I do believe that this may at least result in inadequate site crawling rate and depth.

The problem may have a number of solutions none of which is unfortunately perfect:

Solution	Drawback
Add a different portion of the title/ description to the beginning. title A-G Blue Widgetse.g.	Not in each case this unique element can be sorted out with pagination (most often that will be just page # variable).
Put all the content in one html document. Then use JavaScript to create pagination without reloading the page.	Can be done only if there are not too many results to list; otherwise the page will be enormous and search engines won’t be able to crawl all of it.
Add NoIndex meta tag to the each page except the first one to keep search engines from indexing the pages but still allow them to crawl and follow the links. meta content="”NOINDEX," follow”="" name="”ROBOTS”"> /meta>	Naturally, none of the pages except the first one will ever be ranked and this still doesn’t solve the “PageRank leak” problem (i.e. to many links to follow and give weight).

Now, while each of these solutions has its pros and cons, I have a question to you: how do you handle these issues:

Do you try to avoid pagination at all?
Do you believe pagination doesn’t create duplicate content?
Can you think of any other solution not listed here?

7 Ways to Tame Duplicate Content

2008-12-07T21:17:00.000-08:00

Whether or not you believe that Google has an official duplicate content penalty, there's no question that duplicate content hurts. Letting your indexed pages run wild means losing control of your content, and that leaves search engines guessing about your site and its purpose.
Imagine that you have 25,000 pages in the Google index but only 1,000 pages of that site actually have unique content that you want visitors to see. By losing control of your duplicate content, you've essentially diluted all of those important pages by a factor of 25; for each important page, you have 24 other pages that the spiders have to sort through and prioritize. One way or another, your 25,000 pages are all competing against each other.
Of course, removing duplicate content, especially for large, dynamic sites, isn't easy, and figuring out where to focus your efforts can be frustrating at best. Having fought this battle more than once (including some penalty situations), I'd like to offer a few suggestions:
1. Rewrite Title Tags
They may not be glamorous, but HTML title tags are still a major cue for search engines, not to mention visitors, who see them on-page, in SERPs, and in bookmarks. Even in 2008, I too often see sites with one title, usually something like "Bob's Company" or, even worse, something like "Welcome." SEOs may argue about what makes a good page title, but I think most of us would agree on this: if a page is important enough to exist, it's important enough to have its own title.
2. Rewrite META Descriptions
Whether or not they impact rankings, META descriptions are a strong cue for search spiders, especially when it comes to duplication. Take the time to write decent descriptions, or find a way to generate descriptions if your site is dynamic (grab a database field and shorten it if you have to). If you absolutely can't create unique descriptions, consider dropping the META description field altogether. In some cases, it will be better to have the search engines auto-generate the description than duplicate one description across your entire site.
3. Rewrite Page Copy
This one may seem obvious, but if content isn't different, then it's not going to look different. Copy duplication often occurs when people include the same block of text on many pages or copy-and-paste to create content. If you're repeating text everywhere, consider whether it's really important enough to be repeated. If you copy-and-pasted your entire site, it's time to buckle down and write some content.
4. Lighten Your Code
Although search spiders have gotten a lot better about digesting large amounts of content, many sites still have trouble when unique content ends up pushed deep into the source code. This is especially a problem for older, non-CSS sites or for sites with a large amount of header content. Streamlining your code can be a big help; even if you can't make the jump to "pure" CSS, consider moving core elements into a style sheet. If you're repeating a lot of header content on every page, consider whether that could be reduced or if some of it could exist in one place (such as the home page). Often, large blocks of repeated content are as bad for visitors as they are for spiders.
5. Emphasize Unique Content
Consider using emphasis tags (

,

, , etc.) in your unique content, and use them sparingly in your page header. This will help spiders isolate page-specific content and tell pages apart more easily. Using headers and emphasis consistently will also help your visitors and make you a better copywriter, in my experience. 6. Control Duplicate URLs
This subject is a blog post or 10 all by itself, but I'll try to cover the basics. Dynamic sites frequently suffer from content duplicates created by multiple URLs pointing to the same page. For example, you may have 1 page with the following 3 URLs:
www.mysite.com/product/super-widget
www.mysite.com/product/12345
www.mysite.com/product.php?id=12345
Ideally, only one of those URLs would be visible and the others would operate as hidden redirects, but that isn't always feasible. If you can't use consistent URLs, pick the most descriptive format, and block the rest (nofollow or robots.txt). If one format is older and being phased out, make sure you redirect (301) the old versions properly until you can remove them.
7. Block Functional URLs
This is essentially a subset of #6, but concerns URLs that aren't really duplicates but have a functional purpose. For example, you may have something like:
www.mysite.com/product.php?id=12345
www.mysite.com/product.php?id=12345&search=super%20widget
www.mysite.com/product.php?id=12345&print=yes
These extended URLs are essentially functional directives, telling the page to take an action (like displaying a printable version) or passing along information (like a search string). Removing these completely gets pretty elaborate and isn't always possible, but these directives should definitely be blocked to search spiders. They are essentially hidden instructions that have no value to the spiders or SERPs.

Keyword Cannibalization and How to Handle It

2008-12-07T21:09:00.000-08:00

Keyword cannibalization (no matter how awfully terrifying it may sound) is a widely-spread website internal information structure problem that occurs when multiple subpages are (heavily) targeting one and the same key term.
Very often webmasters are unaware of any keyword cannibalization problems throughout the site - for example, (partial) internal duplicate content issues caused by the site CMS (e.g. pagination causes multiple pages with one and the same title) can account for it.
Others intentionally optimize several pages for one key term “to strengthen” the effect: they think the more a keyword is used throughout the site, the more important it seems for a search engine.
Keyword cannibalization is the issue not to be taken lightly because:

It causes inadequate index and crawl depth by forcing Google to choose between many pages and pick one it feels best fits the query “filtering” the rest of relevant pages;
It accounts for lower SEO effectiveness: efforts are not focused: you are spreading link power, keyword targeting, and anchor text on your site across multiple pages.
It causes internal site competition: your own pages compete with each other for a position in Google.

How to avoid or solve keyword cannibalization problems?

Get rid of internal duplicate content issues;
Organize your keyword lists to thoroughly think through the internal information structure:

Here is also a useful discussion on optimizing your site for multiple keywords.

Carefully think over your website architecture to make the most of internal anchor text and keyword prominence.

We often discuss the search network and content network distribution within Google AdWords. However, we have not explored AdWords’ third distribution channel: Google Search Partners. Today, we will discuss the these Google Search Partners are, how they effect your PPC campaign, and how can you monitor/manage them effectively.

2008-12-07T21:05:00.000-08:00

We often discuss the search network and content network distribution within Google AdWords. However, we have not explored AdWords’ third distribution channel: Google Search Partners. Today, we will discuss the these Google Search Partners are, how they effect your PPC campaign, and how can you monitor/manage them effectively.
First, let’s figure out what these Search Partners are. The Search Partners consist of sites that are within the Google Search Network. Your ads may appear alongside or above search results, as part of a results page as a user navigates through a site’s directory, or on other relevant search pages. Google’s global search network includes Google Product Search and Google Groups.
Quick note, keep in mind that your click-through rate on the search and content networks doesn’t affect your ad’s Quality Score on Google search.
You can choose whether or not you want your ads to appear on the Search partner network. Within your campaign settings tab, under the “Networks and bidding” section, you will find this information. As you can see here, you can opt-in to the Google search, Search Partner and Content networks:

If you choose to display your ads through the Search Partner network, you will be able to monitor you performance directly within the AdWords interface. At the campaign and ad group level you can now parse out your performance for each individual distribution channel. When you are running ads on all three networks, this is what your ad group interface can look like:

You can monitor your performance for all three channels here. I mention this because within AdWords report function you can not separate Google Search from Search Partners so this is the only place within your account where you can gain this visibility.
How do you effectively manage your performance within the Google Search Partners? First, you can opt in and out of this option at the campaign level, not at the ad group level. This can be irksome because within a given campaign, I have had ad groups that do well and some that don’t do well in the Search Partner network. When this occurs, you can separate your ad groups into individual campaigns. You can turn off the Search Partner network where it is not effective. And you can leave the Search Partner network on for those ad groups that do well. Basically, this is an all-or-nothing solution.
I did some research to learn how to best manage your CPC and ad position in the Search Partner network. Unfortunately, there is no direct way to adjust your bid on the solely Search Partner network without also adjusting your Google Search bid. So, if you alter a keyword bid this will effect both search networks. My suggestion is to manage your bids for the Google Search network and the Search Partner network will follow.
There is a great amount of opportunity with the Google Search Partner network. However, you have to monitor your performance closely, and you need to manage your bids in both Google search channels smartly.

Why User Experience Is A Crucial Part Of Good SEO

2008-12-07T20:57:00.000-08:00

Have you ever heard a search engine optimization (SEO) professional use the term user experience during a presentation, in an article or as part of a sales pitch? On the web, user experience (commonly abbreviated as UX or UE) is a term used to describe the overall perception, experience, and satisfaction that users have as a result of their interactions with a website.
Search engine optimization is all about the user experience, because the idea behind SEO is to get users to their desired information and destination(s) as quickly and easily as possible by using the users’ language (keywords). Searchers type in keywords at a commercial web search engine. Searchers’ expectations are validated in search results pages and, hopefully, after they click on links within those search results…a perfect, seamless user experience.
On the surface, an SEO professional’s presumed knowledge of user experience might sound impressive. However, if you do a little digging, you might discover that search professionals have their own preconceived notions as to what constitutes a positive user experience, notions that have little or nothing to do with users at all.
Users + experience = user experience
At the core of user experience is, you guessed it, users. I know this seems blatantly obvious and a little bit stupid for me to write. Nevertheless, user experience is a concept that seems to be lost on many search professionals. Here is why.
There are many different ways that both search professionals and usability professionals gather information about users. Focus groups, Web analytics data, keyword research, field interviews, and usability tests are all ways that these professionals can gather information about searcher behavior and interaction with a website. Search professionals rely heavily on keyword research tools and Web analytics data to determine how users, and search engines, interact with a website. However, as I outlined in a previous article, When Keyword Research and Search Data Deceives, keyword data can lead search marketers down the wrong path.
For example, a keyword phrase might be popular. And a site might rank well for this particular keyword phrase and its long-tail variations. But if the searchers who use this keyword phrase are not among your site’s target audience, then all of the time and expense put into the optimization and advertising for this keyword phrase is wasted. I often hear the legitimate-sounding excuse of the “positive brand experience” for appearing at the top of search results for various keyword phrases. How is appearing at the top of search results to the wrong target audience a positive brand (user) experience?
When I test search results pages for usability, I do not hear test participants say that they view these websites in a positive manner. Rather, they are quite irritated when search listings (and the corresponding web pages) do not meet their expectations. And they become increasingly irritated when the same site appears over and over again for multiple searches. They are not only irritated with the website—they are also irritated with the search engine that keeps delivering listings from the same site over and over again. Test participants usually do not blame themselves for formulating poor search queries. They often blame the website owner and the search engine.
The user experience does not come from a brand manager’s perspective, a marketing manager’s perspective, or even a search engine’s perspective. The user experience comes from the users’ perspective. Search marketers would do well to keep this point in mind when doing SEO.
User experience and interaction
Also at the core of the user experience is interaction. How do site visitors interact with a website? Are web pages sticky, encouraging site visitors to continue viewing other pages within a website? Is a page’s bounce rate extremely high? Do all web pages need to be sticky?
Web analytics data can certainly tell search marketers how site visitors use a website. But this data doesn’t tell us why site visitors do the things they do on a website, what motivates them. In my opinion, it is the combination of the how and the why that delivers the best ROI (return on investment). All too often, search marketers completely miss the boat on the why part because they limit their knowledge of site interaction to web analytics data and general website usability guidelines. They never truly interact with users.
In order to truly evaluate the how and the why of website interaction, you need to:

Put the interface in front of members of your primary target audience,
Objectively observe their behaviors, and
Ask questions about their behaviors without leading them into giving you answers that you want to hear

Focus groups cannot give you this feedback because the focus group leaders are driving the interactions. Web analytics data is not communicating users’ motivations behind their actions. So if I hear a search professional claim that he or she is all about promoting a positive user experience, my questions to them are:

How many usability tests have you done in the past three months?
What are the names of these usability tests?
Without violating client confidentiality, what did you learn from these tests?
What other types of client interaction do you measure, and how?

Of course, I am not saying that search professionals are not “pro” user experience because they do not usability test. However, if I hear the job title of User Experience Designer or User Experience Specialist, I want to know that the person with these job titles actually interacts with users to truly understand their experience.
One of my favorite usability best practices quotations came from Susan Weinschenk from Human Factors International:

“You can apply all usability guidelines to a website and have a completely unusable interface.”

The search experience is a large part of the overall user experience. I laud all search marketers who follow usability best practices during the optimization process, because usability certainly helps with a site’s link development and conversions. But I also know that reading guidelines and following them without actually observing human interaction with optimized websites can lead search marketers and website owners down the wrong path. As part of the SEO and usabality testing process, be sure not to neglect one-on-one interaction with your target audience. You won’t regret it.

Traffic Distribution by Google Ranking

2008-12-05T04:58:00.000-08:00

SEO Sales Process: Overcoming Common SEO Objections

2008-12-05T04:55:00.000-08:00

1. Search Engines Will Find Us/We Already Rank

Sure. Under what keyword terms? How much of the site are the spiders missing?
There is a big difference between arbitrary ranking in search engine listings, and ranking for focused keyword terms. Demonstrate to the client the value of appearing under a wide variety of targeted keyword terms, as opposed to this being a random process. It is like the difference between advertising where few people are looking, as opposed to appearing on a string of billboards in prominent locations.
You could do a side by side comparison between the client and a more established competitor using Compete.com graphs. If they already rank for valuable terms, try to get them to track the business derived from those rankings, and show them the upside potential of increasing rank.

2. We'll Have To Redesign Our Site. That Costs Money

Quite possibly.
Try to demonstrate to the client that the potential benefits outweigh the costs. One way to price organic search traffic is to use the PPC prices as a guide. It could also be argued that organic listings have a higher trust level amongst users, making the traffic potentially even more valuable.
So how much is that poor design costing them in terms of lost opportunity?

3. SEO is Expensive

A common objection, usually made because the client can't determine the amount of work required, or the the value added.
Break down the work into separate tasks, and outline how long each task is likely to take. If the client knows your rate per hour, then they will be more able to determine if the cost is fair.
For example:

Industry analysis - research industry sector, marketing and sales trends.
Competition analysis - conduct review of competitor sites
Keyword research - research keyword terms
Site optimization, including title tags, meta tags, copy and internal linking
Link building/directory submission/social media promotion
Monitoring and reporting

Another aspect of this objection has to do with the value proposition. Again, try printing out the PPC bid prices for the same keyword traffic, and show how your work effectively undercuts that price. If you can, try and get information about how much the client spends on other channels, and do a side by side comparison of the relative merits, costs and benefits.

4. Upper management Won't Support It

Perhaps you need to be talking to the decision maker ;)
Ask what upper-managements objections would be? Sometimes this objection is legitimate, but it is often used to avoid having to tell you "no, thanks". The client cites an authority, who isn't present, implying that any further negotiations with the client will prove fruitless.

5. Why Should We Change The Way We Write Just For Search Engines?

This objection is commonly used by copywriters and journalists.
Established writers often use methodologies that don't take into account SEO. One way to get around this objection is to request a trail run on a few test pages. Once you're demonstrated that writing effective copy can result in an increase in visitors and conversions, you'll have more sway when it comes to changing the rest of the site.
Also, appeal to the copywriters vanity. If more people see their work, isn't that a good thing?
Cite "This Boring Headline Is Written for Google", an article about how The New York Times changed their writing practices to accommodate SEO.

"We're all struggling and experimenting with how news is presented in the future," said Larry Kramer, president of CBS Digital Media. "And there's nothing wrong with search engine optimization as long as it doesn't interfere with news judgment. It shouldn't, and it's up to us to make sure it doesn't. But it is a tool that is part of being effective in this medium."

6. SEO Doesn't Work. It's A Scam!

Ask the client why they feel this way. Has the client had dealings with SEOs in the past? Seen some bad press?
Have case studies on hand that demonstrate how you've solved search marketing problems in the past. Also provide recommendations from previous clients who were happy with your work.
Reframe the debate in terms of problems and solutions.

7. We Have A Strong Brand, So We Don't Need SEO

This is true, so long as people only search on the brand.
But what about those searchers who are searching for generic product/service names?
I once had this objection from a well-known childrens' clothes retailer. I ran a few search reports on generic searches, such as kids t-shirt, babywear, etc, and showed the client the traffic numbers. I then showed the client that their site wasn't appearing under any of those terms.
But her competitors were.
Why choose one or the other when you could easily have both?

8. We Like Flash. It's Cool!

Run away. Run fast..... ;)
Seriously though, such objections usually come from designers who place a lot of emphasis on site appearance, or want to play with the latest toys.
In the past, I've approached this in one of two ways. If they want to keep designing in Flash, or other technologies that make crawling and linking difficult, then suggest workarounds that don't affect the design. For example, create a print-friendly version of the site. This is the part of the site that gets crawled and seen by search engines and search visitors, while the designers can still focus on their elaborate designs. Essentially, you create a site within a site.
Show them that their competitors outrank them, in part, by using different technology. Is Flash really worth that competitive disadvantage?
From Google AdWords Blog:

Did you know that 20% of the queries Google receives each day are ones we haven’t seen in at least 90 days, if at all? With that kind of unpredictable search behavior, it's extremely difficult to create a keyword list that covers all relevant queries using only exact match."

It's even harder to capture that traffic using Flash.
BTW: Check out this example. Here is the spider's view of McDonalds.com.

9. Are SEO Services Really That Important?

Compared to.....?
It's an effort vs reward question. Again, if you can demonstrate clear commercial benefits over and above the cost, then "hell yes!". Try to focus on the clients business problems, and be prepared to demonstrate how the SEO spend will solve those problems in cost effective ways.
Those are a few common objections. I'm sure you've heard others. What is important to understand is that not all objections are legitimate. Most are stalling tactics used to delay making a decision. That decision is difficult to make because the client will expose themselves to risk.

Simply by being pre-prepared for objections, you help negate that risk, and can quickly move the client towards make a decision.

Top Time-Saving Yahoo! Search Tips

2008-12-05T04:50:00.000-08:00

When it comes to the Web, there's nothing wrong with cutting a few corners. That's why we decided to focus today's post on some time-saving tips for your next search. Some may seem obvious; others you may already know and use. But we hope a few will help you cut through the chase. You can find a full list of shortcuts and search tips here.
1. Square Brackets, "inurl," "originurlextension," and Site Restriction
To get a more targeted search, try these tricks out:

Words within square brackets -- adding square brackets to your search makes the keyword match order dependent. So typing in '[Jack Black]' will return results such as 'jack with black' but not 'black jack.'
"inurl" -- if you want to be sure that a specific term will appear in the site's URL, use the "inurl:[query]" operator. For example: 'inurl:iPod.'
Site restriction -- to restrict your search to pages within a specific domain, use the "site:[domain]" operator, followed by your query. For instance: 'Site:Apple.com iPod.'
"orginurlextension" -- to search on specific file types, add 'originurlextension:[file format]' after your search query. For example: 'nanotechnology originurlextension:swf' OR 'nanotechnology originurlextension:pdf.'

2. Package Tracking
Did you know that you can track your packages right in Yahoo! Search? Here's How it works:

For UPS packages, simply type in your tracking number
For FedEx or the U.S. Postal Service, just add the name before the tracking number. For example: 'FedEx [tracking number]' or 'USPS [tracking number]'

3. Definitions & Synonyms
To look up the definition of a word, try adding "define" or "definition" to your search term. For example: 'quixotic definition,' 'definition of globalization' or 'define ergonomics.' Or, if you're looking for a synonym, try adding "synonym" to your search term. For example: 'humorous synonym.'
4. Exclude Terms, Either/Or and Exact Phrase Match
This one's been around for a while, but a few simple operators can be a huge time-saver:

Exclude terms -- if you want a term to be excluded from your results, use a minus sign before it. 'Simpsons -movie' returns results for "The Simpsons" TV show, books, games, etc., but not the movie.
Either/or -- by default, all of the words you use in a search are included in the results. If you want to be more flexible, try adding "OR" (note the capitalization) between two terms. For example: 'Sony laptops OR notebooks' gives you results containing either "Sony laptops" or "Sony notebooks."
Exact phrase match -- if you want results to contain an exact phrase, put quotation marks around it: "Queen Elizabeth I".

You can also combine these tricks for even more refined searches. Try: '"Sony VAIO" laptops OR notebooks.'
5. Travel
With the holidays approaching, many of us have travel on our minds. Here are a few shortcuts to get you to your destination even faster:

Flight tracker -- search for the airline and flight number and you'll get a shortcut to the flight's status. Try: 'American 83' or 'Lufthansa 421.'
Traffic -- if you're driving instead of flying, you can search for traffic before you leave. Example: 'traffic Los Angeles.' Click on the shortcut and you'll get a map with traffic alerts.
Maps -- try searching for the exact address: '1600 Pennsylvania Avenue, Washington, DC.' Don't have the address? No problem. Add "map" before the city: 'map San Francisco.' You can also search for the zip code by itself: '20502.'

You can go here to check out more handy travel shortcuts.

6. Yahoo! Services
If you're looking for a Yahoo! site, simply add an exclamation point after the site name and voila! Try it out with 'Mail!,' 'News!,' 'Sports!,' or 'Finance!'
7. Yahoo! Open Shortcuts
Yahoo! Open Shortcuts are the ultimate time-saving search feature. Add an exclamation point to the front of certain terms to instantly navigate to a URL, search a site, recall a favorite Yahoo! search, or start an application.

'!wiki queen elizabeth' takes you directly to the Wikipedia page for Queen Elizabeth.
'!wsf' gives you the Yahoo! Search results for "weather San Francisco."
'!clist' takes you to Craigslist.
'!ebay lamps' searches eBay for lamps.

Search for '!list' to see a bunch more. Those examples have already been set up for everyone to use, but the real power is that YOU can create your own customized shortcuts.

Have new ideas or suggestions for us? Let us know in the comments below. We're always looking for ways to make Yahoo! Search more efficient for you.

Second Page Poaching - Advanced White Hat SEO Techniques

2008-12-03T20:34:00.000-08:00

It is time that someone put Quadzilla in his place. Don’t get me wrong, I think there is a huge amount of innovation in the black hat and gray hat industries that simply is too risky for white-hatters to discover. Nevertheless, it is simply ridiculous to claim that white hat techniques have become so uniform and ubiquitous in their application that nothing truly “advanced” continues to exist. In today’s post I am going to talk about a technique with which many of you will not be familiar. Here at Virante we call it Second Page Poaching. But, before we begin, let me start with a brief explanation of what I believe to be “advanced white-hat SEO”.
Chances are, unless your site has thousands well-optimized pages in Google, advanced white hat techniques and many others will be useless. Advanced white hat SEO techniques tend to deal with scalable SEO solutions that bring higher search RoI for sites that seem to have reached the peak of potential search traffic. We are not talking about training a young boxer to become a contender. We are talking about turning a contender into a champion. When your PR7 eCommerce site is competing against other PR7 eCommerce sites for identical products and identical search phrases, all traditional optimization techniques (white, black, gray, blue hat, whatever) tend to fall by the wayside. Why buy links when you already have 100K natural inbounds? Why cloak when you have tons of legitimate content? This is where advanced white hat SEO kicks in. These are techniques which can bring high RoI with little to no risk when scaled properly. The example I will discuss today, “Second Page Poaching” is highly scalable, easily implemented, and offers a high Return to Risk ratio.

What is Second Page Poaching

the coordination of analytics (to determine high second-page rankings) with PR flow and in-site anchor-text to coax minor SERP changes from Page 2 to Page 1.

Why Second Page Poaching

We need to recognize that most on-site SEO techniques, especially PageRank flow, will only increase rankings by a single position or two. If you have a well optimized site, even providing a sitewide link to one particular internal page is unlikely to push it up 5 or 6 positions. Moreover, PR-flow solutions are unlikely to move a page from position 3 to position 2 or 1, where competition is more stiff. Instead, we would like to target the pages that will see the greatest traffic increase from an increase of 1 or 2 positions in the search engines.
Looking at the released AOL search data, we can determine which positions are most prime for “poaching”. We use this term because we are hunting for pages and related keywords on your site that meet certain qualifications.
Below is a graph showing the relative percentage increase of clickthroughs based on location in the top 12 in AOL’s released data. As you will see, there are spikes at moving from position 2->1, positions 11->10 and 12->11, 3->1, 11->9 and 12->10.

If we look at the data directly, you can see that the increases are in the 500% range or greater for moving from the top of the 2nd page to the bottom of the first. More importantly, as previously discussed, it is unlikely that PR-Flow methods will help you move from #2 to #1, given the competitiveness. But will that little bit of PageRank boost help you move from 11 to 10? You betcha! Simply put, if you can move hundreds of pages on your site from ranking #11 to #10, you will see a 5 fold increase in traffic for those keywords. If you were to do the same to move them from #7 to #6, you would barely see an increase at all.

Data Provided by RedCardinal
Now, as an advanced technique, it is important to realize that this becomes incredibly valuable when a site already has tens if not hundreds of thousands of pages. If you bring in 10,000 visitors a month from Page 2 traffic, you could see your traffic increase by 40,000 fairly rapidly. If you are a mom-and-pop shop and have 10 visitors from Page 2 traffic, you might only see 20 new visitors, as your site’s internal PR will be less capable of pumping up the rankings for those Page 2 listings.

Step 1: Data Collection

For most large sites, data collection is quite easy. Simply analyze your existing log files or capture inbound traffic on the fly and record any visitor from a Page 2 listing. Identify all inbound referrers that include both “google.com”, “q=” and “start=10″ Store the keyword and the landing page. Make sure your table also stores a timestamp as well, as frequency will matter when we make future considerations of which keywords to poach. If your log files store referrer data, it may be useful to go ahead and include historical data rather than starting from scratch. A suitable site should find hundreds if not thousands of potential keyword/landing page combinations from which to choose.

Step 2: Data Analysis

Because this step will be the most processor intensive, it is important to prioritize. In the Data Analysis component, we will judge the keyword/landing page combinations based on several characteristics. Because we only want to try to poach keywords for which we already rank #11 or #12, we will have to perform rank checks. In the interest of lowering what could potentially be a large computing burden, we should first consider the other metrics and then choose from that group which keywords we will check for rankings. We will consider the following characteristics”

Frequency: Number of visitors driven per month, the higher the better.
Conversion Rate: Why poach keywords for pages that convert poorly?
Sale Potential: Why poach keywords for low RoI goods?

Now, assuming we have a set of keywords ordered by highest conversion, profit and frequency, we run simple rank checking software to identify those keywords for which your site currently ranks #11 or #12. Once that data point is added to the set, we use a formula to determine which keywords are most worth targeting. I will not get into it now, it is kind of proprietary, but you will want to take into account several factors to determine the minimum number of links needed to promote a page from #11 to #10 or #12 to #10. Once complete, you should now have our list of hundreds if not thousands of “keyword/landing page” combinations worth targeting.

Step 3: PR Flow Implementation

There are many creative ways to add these links across your site, the easiest of which is a simple “Other People Searched For:” section at the bottom of internal pages that list up to 5 alternatives. Your system would then choose 5 pages from the list and add text links with the inbound keyphrase as the anchor-text pointing to these landing pages. If you want to get really crafty, you can use your own internal search to identify related pages upon which to include the different landing page links that occur in your list. Ultimately, though, you will have added a large volume of links across your site which slightly increase the PageRank focused upon these high potential pages ranking #11 or #12. As Google respiders your site and finds these links, these pages will crawl up the 1 or 2 positions needed to quadruple or better their current inbound traffic for particular keywords.

Bear in mind that you risk very little with this technique. PR Flow tends to have little impact on your high-dollar, high-traffic keywords (where inbound links rule the day). Most importantly, because the system is automated, it will allow your internal pages to drop from position 6 to 7, where little to no real traffic is lost, but will capture and restore rankings if any pages ever drop from 9 or 10 to 11 or 12 due to the new internal linking scheme. You will lose PR on pages th.at have nothing to lose, and gain PR on pages that have everything to win. Magnified across 10,000 pages, and you can see the profits from a mile away.

Step 4: Churn and Burn

This is perhaps the most important part of the process. Continued analysis.

the system needs to capture any successful poaches and make sure you continue to link to them internally. if the system drops pages once they move from 11 to 10, you have failed
the system needs to determine non-movers, pages where internal linking is not improving their position, and blacklist them so you do not continue to waste extra PR flow
the system needs to continue to replace non-movers with the next best solution, and continue to look for more keyword/landing page options

Summing Up

I know this is a long post, but I feel like it is worth reiterating. Advanced white hat SEO techniques do exist but no one wants to talk about them for the same reason no one wants to talk about advanced black hat techniques. Second Page Poaching is just one of many different options available to large-scale websites looking to gain an edge over their competitors. Many of these techniques are fully automated, easily implemented, and highly scalable. However, most of them are kept under lock-and-key.

SEO Strategy Document

2008-12-02T20:43:00.001-08:00

Below is a SEO Strategy Document I have used in the past. I decided to post it after speaking with a respected SEO Consultant who wasn’t sure how to craft the document. I hope this helps other consultants as well. Early this year, I also posted an Example of a PPC Strategy Document.
Feel free to add any comments, constructive criticism, etc.
Beginning of document…
This is the strategy document for the SEO initiative within the overall marketing plan for [CLIENT SITE]. This document will provide detail to the strategy, analysis, and optimization recommendations.
It is our recommendation to use both an SEO & PPC Search campaign. However, to initially begin with the SEO component.
The reason the sequence should be SEO before PPC is because a highly optimized webpage will produce a much more efficient paid search campaign due to a higher quality score, and result in lower CPC’s and higher rankings.
Therefore, this document focuses on the SEO initiative for the [CLIENT SITE] main site, and the [CLIENT SITE] ecommerce site.
We will be optimizing the website in its entirety, with a specific success metric on a keyword/landing page combinations.
Primary Objective:
- Increase visitor traffic to http://www.[Client Site].com
- To drive people to the specified content on [Client Site].com from pages that we optimize for specific keyword phrases.
Secondary Objective:
- To generate brand awareness for the [CLIENT SITE] brand.
Domains:
- http://www.[Client Site].com
Target Search Engine:
- Google – We will focus our optimization efforts on Google.
- Google has roughly a 60% market share of search volume.
- Once we have a good ranking on Google, it is more than likely we will have a good ranking on the other engines because the ranking factors are all very similar across the engines.

Budget:
- $[blank] monthly budget
Results Timeline:
- After implementation of the recommendations, there is a delay of at least a week or two, before seeing results in the Search Engine.
- The keyword rankings should continue to improve as the strategic inbound links age, and the SEO implementations have been indexed. On average, the optimum effect from the on-site & off-site SEO effort will be experienced in approximately 12 weeks.
Launch Date:
- January 1, 2008
Keyword Research and Analysis:
- Determine which keywords will pull in qualified traffic, how many searches do these terms/phrases receive, how should keywords & webpages be used in combination to attract search traffic.
- Analyze Keyword Difficulty
Keywords:
- We will concentrate on optimizing keyword phrases that are mapped to a specific webpage.
- 1 Keyword/Landing page combinations.
Domain Analysis:
- Analysis of the website information architecture and internal linking structure
- Analysis of HTML and page layout
Content Analysis & Creation:
- Amount of Indexable Text Content
- Quality & quantity of visible HTML text on page
- SEO Copywriting
- Additional webpage/content creation if necessary
Linking Analysis:
- Overall Internal link structure
- Internal Link Popularity of Linking Page within Host Site/Domain
- Link Popularity
- Link Popularity of Site in Topical Community
- The link weight/authority of the target website amongst its topical peers in the online world.
- Link Relevance
- Topical Relevance of Inbound Links to Site.
- The subject-specific relationship between the sites/pages linking to the target page and the target keyword.
- Deep Link Analysis
- Deep link percentage is the % of all inbound links that point at pages other than the homepage.
Link Development:
- Increase link popularity
- Implement a Strategic Link Campaign by purchasing links on websites that will provide a high SEO value.
Indexing Analysis:
- Duplicate Content
- Content very similar or duplicate of existing content in the index
- Possible duplicate Title/Meta Tags on many Pages
- Robots.txt exclusions
Baseline Reporting:
- Generate a baseline Ranking Report for the keywords. The baseline, as the name implies, gives us an indication of where the webpages are ranked on Google, and will be used to measure results.
- Baseline Traffic report for http://www.[Client Site].com
- Baseline Monthly Sales report for http://www.[Client Site].com
Implementation:
- Programming our recommendations. Once all the changes have been completed, we will run an audit to verify that the recommendation was implemented correctly and identify any changes, if needed.
Success Metrics:
- First page Search Engine rankings.
- Increase in Website Traffic to http://www.[Client Site].com
- Increase Sales volume to http://www.[Client Site].com , and generate a positive ROI.

Link Building Strategies

2008-12-02T20:29:00.000-08:00

Six and a half years ago (which is ages, in Internet years), Robin Nobles, Eric Ward, and John Alexander compiled a legendary list of 131 legitimate link building strategies. Four years later, Aaron Wall and Andy Hagans published 101 link building tips to market your website, which was inspired by the other article. Considering the furiously changing face of search engine marketing and with 2009 already ahead of us, I thought it was time to evaluate both lists and create an updated collection of link building strategies.

7 Internal link building strategies

1. Make sure that your navigation is spiderable. Either use (anchor text carrying) text based navigation, or an image based navigation with relevant alt attributes attached to each image link.
2. Breadcrumbs are a great internal linking tool. Use them for usability and anchor text differentiation.
3. In-content links not only tend to have a higher click through rate and perceived trust, but are also able to add more relevance to a link because of the surrounding text.
4. Use a sitemap. A good sitemap is useful for visitors, useful for search engines and, therefore, useful for you.
5. Link to topically relevant pages on important pages of your website. Link to important pages on every (or most) topically relevant page of your website.
6. Be consistent in linking behavior. If you link to homepage.com, always link to homepage.com, and not to homepage.com, homepage.com/index.php and homepage.com/index.php&id=123.
7. Identify your most linked-to pages, and make sure that the link juice flows to your most important pages from there, in a well-optimized way.

10 Easy link building starters

8. Optimize your existing links. Contact the webmasters of prominent websites that link to you and ask them to change ‘click here’ to an anchor text that contains relevant keywords, an anchor text that encourages clicking through, or -ideally- a combination of both.
9. Monitor your 404 statistics. Keep track of whoever links to old pages or misspelled URLs, which is data that Google provides as well. Contact those webmasters and provide them a good URL which they can link to.
10. Create a ‘link to us’ page, where you provide information about how people can link to you and which URL(s), logo and/ or anchor text they can use. Update this page regularly in order to diversify the anchor text.
11. Contact family, friends, colleagues and other people you know and let them know about your website. Some will send you useful feedback, others -who happen to have a website of their own- might link to you.
12. Do you block search engine bots from indexing certain parts of your website via robots.txt or meta-noindex? Find out if people link to this section of your site. If so, contact the webmasters of these sites and kindly ask them to link to an other page of your website.
13. Use your spell check. People will more likely link to correctly spelled articles than to content that’s full of grammar errors.
14. Search for websites that already mention your business name or URL, but haven’t linked to your website. This works excellently in Yahoo!.
15. Look for websites that mention your personal name, but currently don’t link to your site. Use Yahoo! for this as well.
16. Leave comments on the blogs you visit every day. Hey, you’re visiting them anyway, so why don’t leave a (relevant, useful!) comment?
17. Find out which website your company owns. If you work for a small company, there may possibly be several. If you work for a large company, the number will probably knock you off your shoes. Link these websites (carefully!) together, or redirect the most important and/ or relevant ones to your main website.

12 Old school link building techniques

18. Search for related websites by using relevant keywords. Filter out all interesting websites and contact them. When you did this for your main keyword(s), there are still tons of other combinations possible.
19. Check which websites link to your competitors. Try to get them to link to your website as well.
20. Check which types of websites link to websites that offer the same services or products as you, but in a different country/ language. This might result in a “I never thought of that…” feeling.
21. Either interview an expert from your field, or try to get interviewed by someone else. Don’t forget to mention your best content: readers of the interview might be willing to link to it.
22. Write guest posts for relevant websites in your niche. You could also write posts about your industry for websites that are slightly related to your niche.
23. Teach. Whether it’s a public workshop (local press), a class at a local college or University (.edu website) or at a business related event (industry links), teaching can result in authority links.
24. Use any search engine advertising program and advertise on keywords that linkerati might use. Try to convert the targeted traffic into links.
25. Use Google AdWords’ content network to determine which (relevant) websites generate traffic and conversions. Contact those websites directly.
26. Join an affiliate program. See #25.
27. Determine who’s linked to you before. Contact them again when you’re releasing an interesting new piece of content.
28. Trade links. There’s nothing wrong, with swapping links with a few, highly relevant, authority websites that can bring in extra traffic. Exchanging links with lots of irrelevant websites, however, might get you in trouble.
29. Donate to a charity. Although buying links is not allowed by Google, there are still lots of ways you can buy links (kind of) legitimately.

12 Places to submit your URL to

30. Most social media websites are only useful for promoting good content (which will get you links in return), but sites like LinkedIn still provide dofollow links with an anchor text of your choice.
31. Some general directories, such as DMOZ, the Yahoo Directory and Best of the Web are still worth submitting your website to. Make sure to submit your site to the most appropriate category.
32. High quality, niche directories can be worth considering as well. Notice the emphasis on high quality.
33. Don’t forget to submit your website to high quality, regional directories. Especially worthwhile for websites that target local markets.
34. Publish stunning, interesting, funny or beautiful images in your Flickr account, that contains a link to your website.
35. Writing an article about a relevant topic, that contains one or more links to your website, and submitting it to article directories such as eZineArticles might work for you.
36. Relevant, non-spammy links in Wikipedia articles, Yahoo! Answers or Google Groups may have nofollow attributes attached, but can lead to (dofollow) links indirectly.
37. Submit your RSS feed to important RSS directories.
38. Blog directories may be willing to link to your blog. Submit your blog to the high quality ones.
39. Use PR websites to distribute your press releases, in addition to your PR agency. Make sure that your press release contains one or more (clickable) links to your website.
40. Got a great design? Submit your site to CSS directories and/ or website design contests. Even well-designed parts of your website can result in links.
41. Twitter. Just published a new post or article? Mention it on Twitter, your followers might visit it and -if they find it interesting- link to it.

12 Ways to make people write about you

42. Send out christmas gifts or birthday gifts to bloggers (or website owners) you know.
43. Offer services or a product in exchange for a review. Don’t ask the bloggers or webmasters to link to you, they most often will do anyway.
44. Create something unique. Top 10s, top 250s, mash-ups, how-tos, best-ofs, surveys, studies, awards. Define the proper hook, create unique content and attract good links. The possibilities are infinite.
45. Try to start a hype, use a new word, get a meme going, or do something else you’re the first at.
46. Link to others. People -especially bloggers- will notice it if you link to them. If you do this several times and offer content that is or might be relevant to these bloggers, they might link to you as well eventually.
47. If you happen to have some breaking news, offer a blogger (or a select group) the scoop. Bloggers love to publish scoops.
48. Say something groundbreaking, shocking, confronting, stupid, weird or flattering. People tend to link to others who are different or act that way.
49. Create something with an amazing design. This does not necessarily have to be your website, just having an awesome business card can result in extra links.
50. Launch an extraordinary offline campaign. People will talk about this online. If you integrate this offline campaign with an online version in a perfect way, you may even receive some extra links from ‘this is how you should integrate offline and online’-articles as well.
51. Create a contest and offer give-aways for winners. This is not only a great way to get attention, but to get valuable input as well, for example when hosting a guest post contest.
52. Build useful tools and/ or plugins that are free to use.
53. Speak at an industry conference. You’ll meet lots of interesting new people, and will probably get mentioned in several conference write-ups.

12 Common business tactics

54. Add a link to your local Chamber of Commerce profile.
55. The Better Business Bureau, and any industry related association you’re a member of are interesting link targets as well.
56. Contact your (preferred) suppliers, manufacturers, other partners. Obtain links from these website if they have a partners page as well.
57. Offer to write testimonials or a quote to your suppliers, if they are willing to link back to your site in or near this testimonial.
58. Ask clients to write testimonials about your product or service that they publish on their website, in exchange for a discount, extra fast delivery or any other benefit you can provide.
59. Hire a publicist. Press agency employees usually know the right people in the right places, which can result in a higher acceptancy rate of your press release.
60. Join relevant forums. You can either link to your website on your profile page, in your signature or in your posts. Notice how this one is listed under ‘Business related tactics’ in stead of ‘Places to submit your URL to’? There’s a reason why: forums are not places to drop links, but to join discussions.
61. Sponsor something. There are tons of possibilities, such as an industry conference, a sports club, a relevant forum, a local happening, or just any offline event that happens to have a website.
62. Hire an intern. You can let him or her work on a piece of research, which you can in your link building process. Also, don’t forget the website of the University you’re intern is attending.
63. Offer awesome product or services. People love talking about great stuff they’ve bought. If your products are ‘just’ good in stead of awesome, make sure that your after sales or customer care is excellent. People love talking about companies with a great service as well. Of course, offering crappy products or a lousy service will also result in links, but I don’t think those are the links you’re after.
64. Look for companies that went out of business. Either acquire their website, or contact the website that they’re currently getting links from and ask these sites to link to you in stead.
65. Turn your colleagues into link developers. Each of them has his or her own specialty and group of contacts. This not only take works off of your hands, but is very efficient as well.

4 Important considerations

66. Hire a link builder or an expert. Either let somebody you trust manage (a part of) your campaign, or visit a link building workshop. Especially when you’ve been building links for your own site for several years, a fresh mind can bring new ideas.
67. Hang in there. Link building isn’t something you can do in just a few hours, or something that you only have to do during one week in a year. Building a brand can’t be done in a single day, the same goes for a solid link profile. It’s a continuing process that takes time. Lots of time.
68. Keep an eye on the news. Follow important and interesting different blogs, in order to keep up with the latest news, trends and tricks. I’m not just talking about link building or SEM blogs, but make sure to follow general marketing blogs, slightly different, creative blogs or industry related news websites as well.
69. If you have to ask yourself ‘is this a legitimate approach’ or ’should I be doing this’, the answer is probably no. Too much, too aggressive or too shady isn’t advisable. Don’t do things you would be ashamed of when explaining them to your mother. Or Matt Cutts.

0 Advanced link building strategies

There is no such thing as advanced link building. While this list already sums up quite a few different strategies, I’m pretty sure that you can easily come up with a dozen more, that are specifically suitable for your company or industry.
Eric Ward once said that link building is “one part marketing, two parts public relations, and three parts common sense”. I’d say that link building is 10% basic SEO knowledge, 20% business thinking, 30% creativity and 40% perseverance. Either way, there’s nothing advanced to it.

SEO Bookmarklets Collection: On-Page SEO and Domain Stats

2008-12-01T23:20:00.000-08:00

Last week I listed a few useful Google bookmarklets for SEOs. This time I am looking at some more helpful browser bookmarklets that can come in handy for on-page SEO research and a quick view at essential site stats (to install any of them just drag it to your bookmark toolbar):

Domain information:

Alexa info - view Alexa general profile for the current domain.
Robots.txt - instantly access site Robots.txt page;
Whois bookmarklet - find domain information (via Whois.DomainTools);
AttentionMeter - view Compete.com, Quantcast, Google Trends, Alexa and Technorati stats for the current domain name.

Page SEO:

Page Headers - shows all headers for a current page (by Yoast.com);
Validate HTML - instantly access current page HTML W3.org validator;
HTTP Headers - Web-Sniffer.net header checker for the current page;
Edit Page - lets you edit the current page;
Word Frequency - highlight a word and see how many times it was mentioned on a page;
Wayback newest - shows you the latest copy saved by Archive.org;
Page Freshness - find out last modified date
Make a link for this page - instantly generate a link for the current page;
Highlight headings - shows you on-page H1-H6 headings;

Highlight NoFollow - highlight links with “nofollow” attribute;
Show Meta - view page meta tags.

Sources:

Concertina pagination - getting large scale websites indexed

2008-12-01T23:10:00.001-08:00

As many of you will know getting large numbers of pages into the index of search engines is a science on its own.

image by hoveringdog
It’s clear that cross linking from other pages of the project helps massively. Not only with indexation but with ranking of the pages as well. SEO black-holes like Wikipedia are the living proof of the effectiveness of - what you can call excessive cross-linking.
The other obvious helper to high volume indexation is back-link power. A nice back-link from the yahoo.com homepage will do wonders to the deep-crawl frequency of your site and the number of pages listed.
Ideally you will get back-links not only to you homepage but on any level in the hierarchy of your site.
Time is an additional factor in this equation. It takes time to get millions of pages into the index of search engines.
Now the above is very well known common knowledge among SEO people and even the below is no ground breaking paradigm shifting revelation - but at the same time its a very effective method that’s easy to implement but still rarely used amongst sites with huge numbers of content pages and multiple levels of categories.
The method I’m talking about is a non constant based pagination. It is often referred to as concertina.
And the method is as simple as effective. When in most cases the pagination will look like this.

And the pages will show 1 to 10 on the first page 11 to 20 on the second and so on and so forth. This scheme will stay the same even if you have tens of thousands of pages.
Due to the fact however that few sites have back-links from the yahoo.com leads to the fact that search engine crawler will get “tired” after a while. This means that the crawler will only get to a certain depth of pages on your site.
To help with this it is much more effective to implement a pagination that looks like this.

or this

For the first couple of pages its identical to the example shown above. But after page ten you will get page 20 then 50 then 100 and so on.
By using this method you make sure that the crawler hits pages further down the hierarchy quicker and this does lead to a significant increase of pages indexed.
Here are a couple of sites that use various types of concertina paging for you to have a look.

Digital Point forum
Kelkoo Shopping
Flickr uses a kind of concertina as well

Date with Googlebot

2008-12-01T23:05:00.001-08:00

Our date with Googlebot was so wonderful, but it's hard to tell if we, the websites, said the right thing. We returned 301 permanent redirect, but should we have responded with 302 temporary redirect (so he knows we're playing hard to get)? If we sent a few new 404s, will he ever call our site again? Should we support the header "If-Modified-Since?" These questions can be confusing, just like young love. So without further ado, let's ask the expert, Googlebot, and find out how he judged our response (code).

Supporting the "If-Modified-Since" header and returning 304 can save bandwidth.

-----------
Dearest Googlebot,
Recently, I did some spring cleaning on my site and deleted a couple of old, orphaned pages. They now return the 404 "Page not found" code. Is this ok, or have I confused you?
Frankie O'Fore

Dear Frankie,
404s are the standard way of telling me that a page no longer exists. I won't be upset—it's normal that old pages are pruned from websites, or updated to fresher content. Most websites will show a handful of 404s in the Crawl Diagnostics over at Webmaster Tools. It's really not a big deal. As long as you have good site architecture with links to all your indexable content, I'll be happy, because it means I can find everything I need.

But don't forget, it's not just me who comes to your website—there may be humans seeing these pages too. If you've only got a very simple '404 page not found' message, visitors who aren't as savvy could be baffled. There are lots of ways to make your 404 page more friendly; a quick one is our 404 widget over at Webmaster Tools, which will help direct people to content which does exist. For more information, you can read the blog post. Most web hosting companies, big and small, will let you customise your 404 page (and other return codes too).

Love and kisses,
Googlebot

Hey Googlebot,
I was just reading your reply to Frankie above, and it raised a couple of questions.
What if I have someone linking to a page that no longer exists? How can I make sure my visitors still find what they're after? Also, what if I just move some pages around? I'd like to better organise my site, but I'm worried you'll get confused. How can I help you?
Yours hopefully,
Little Jimmy

Hello Jimmy,
   Let's pretend there are no anachronisms in your letter, and get to the meat of the matter. Firstly, let's look at links coming from other sites. Obviously, these can be a great source of traffic, and you don't want visitors presented with an unfriendly 'Page not found' message. So, you can harness the power of the mighty redirect.

   There are two types of redirect—301 and 302. Actually, there are lots more, but these are the two we'll concern ourselves with now. Just like 404, 301 and 302 are different types of responses codes you can send to users and search engine crawlers. They're both redirects, but a 301 is permanent and a 302 is temporary. A 301 redirect tells me that whatever this page used to be, now it lives somewhere else. This is perfect for when you're re-organising your site, and also helps with links from offsite. Whenever I see a 301, I'll update all references to that old page with the new one you've told me about. Isn't that easy?

   If you don't know where to begin with redirects, let me get you started. It depends on your webserver, but here are some searches that may be helpful:
Apache: http://www.google.com/search?q=301+redirect+apache
IIS: http://www.google.com/search?q=301+redirect+iis
You can also check your manual, or the README files that came with your server.

   As an alternative to a redirect, you can email the webmaster of the site linking to you and ask them to update their link. Not sure what sites are linking to you? Don't despair - my human co-workers have made that easy to figure out. In the "Links" portion of Webmaster Tools, you can enter a specific URL on your site to determine who's linking to it.

  My human co-workers also just released a tool which shows URLs linking to non-existent pages on your site. You can read more about that here.

Yours informationally,
Googlebot

Darling Googlebot,
I have a problem—I live in a very dynamic part of the web, and I keep changing my mind about things. When you ask me questions, I never respond the same way twice—my top threads change every hour, and I get new content all the time! You seem like a straightforward guy who wants straightforward answers. How can I tell you when things change without confusing you?
Temp O'Rary

Dear Temp,
   I just told little Jimmy that 301's are the best way to tell a Googlebot about your new address, but what you're looking for is a 302.
   Once you're indexed, it's the polite way to tell your visitors that your address is still the right one, but that the content can temporarily be found elsewhere. In these situations, a 302 (or the rarer '307 Temporary Redirect') would be better. For example, orkut redirects from http://orkut.com to http://google.com/accounts/login?service=orkut, which isn't a page that humans would find particularly useful when searching for Orkut***.
It's on a different domain, for starters. So, a 302 has been used to tell me that all the content and linking properties of the URL shouldn't be updated to the target - it's just a temporary page.

  That's why when you search for orkut, you see orkut.com and not that longer URL.

  Remember: simple communication is the key to any relationship.

Your friend,
Googlebot

*** Please note, I simplified the URL to make it easier to read. It's actually much more complex than that.

Captain Googlebot,
I am the kind of site who likes to reinvent herself. I noticed that the links to me on my friends' sites are all to URLs I got rid of several redesigns ago! I had set up 301s to my new URLs for those pages, but after that I 301'ed the newer URLs to my next version. Now I'm afraid that if you follow their directions when you come to crawl, you'll end up following a string of 301s so long that by the end you won't come calling any more.
Ethel Binky

Dear Ethel,
It sounds like you have set up some URLs that redirect to more redirects to... well, goodness! In small amounts, these "repeat redirects" are understandable, but it may be worth considering why you're using them in the first place. If you remove the 301s in the middle and send me straight to the final destination on all of them, you'll save the both of us a bunch of time and HTTP requests. But don't just think of us. Other people get tired of seeing that same old 'contacting.... loading ... contacting...' game in their status bar.

Put yourself in their shoes—if your string of redirects starts to look rather long, users might fear that you have set them off into an infinite loop! Bots and humans alike can get scared by that kind of "eternal commitment." Instead, try to get rid of those chained redirects, or at least keep 'em short. Think of the humans!

Yours thoughtfully,
Googlebot

Dear Googlebot,
I know you must like me—you even ask me for unmodified files, like my college thesis that hasn't changed in 10 years. It's starting to be a real hassle! Is there anything I can do to prevent your taking up my lovely bandwidth?

Janet Crinklenose

Janet, Janet, Janet,
It sounds like you might want to learn a new phrase—'304 Not Modified'. If I've seen a URL before, I insert an 'If-Modified-Since' in my request's header. This line also includes an HTTP-formatted date string. If you don't want to send me yet another copy of that file, stand up for yourself and send back a normal HTTP header with the status '304 Not Modified'! I like information, and this qualifies too. When you do that, there's no need to send me a copy of the file—which means you don't waste your bandwidth, and I don't feel like you're palming me off with the same old stuff.

You'll probably notice that a lot of browsers and proxies will say 'If-Modified-Since' in their headers, too. You can be well on your way to curbing that pesky bandwidth bill.

Now go out there and save some bandwidth!
Good ol' Googlebot

-----------

Googlebot has been so helpful! Now we know how to best respond to users and search engines. The next time we get together, though, it's time to sit down for a good long heart-to-heart with the guy (Date with Googlebot: Part III, is coming soon!).

First date with the Googlebot: Headers and compression

2008-12-01T23:04:00.000-08:00

Name/User-Agent: Googlebot
IP Address: Verify it here
Looking For: Websites with unique and compelling content
Major Turn Off: Violations of the Webmaster Guidelines

Googlebot -- what a dreamboat. It's like he knows us , , and soul. He's probably not looking for anything exclusive; he sees billions of other sites (though we share our data with other bots as well :), but tonight we'll really get to know each other as website and crawler.

I know, it's never good to over-analyze a first date. We're going to get to know Googlebot a bit more slowly, in a series of posts:

Our first date (tonight!): Headers Googlebot sends, file formats he "notices," whether it's better to compress data
Judging his response: Response codes (301s, 302s), how he handles redirects and If-Modified-Since
Next steps: Following links, having him crawl faster or slower (so he doesn't come on too strong)

And tonight is just the first date...

***************
Googlebot: ACK
Website: Googlebot, you're here!
Googlebot: I am.

GET / HTTP/1.1
Host: example.com
Connection: Keep-alive
Accept: */*
From: googlebot(at)googlebot.com
User-Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Accept-Encoding: gzip,deflate

Website: Those headers are so flashy! Would you crawl with the same headers if my site were in the U.S., Asia or Europe? Do you ever use different headers?

Googlebot: My headers are typically consistent world-wide. I'm trying to see what a page looks like for the default language and settings for the site. Sometimes the User-Agent is different, for instance AdSense fetches use "Mediapartners-Google":
User-Agent: Mediapartners-Google

Or for image search:
User-Agent: Googlebot-Image/1.0

Wireless fetches often have carrier-specific user agents, whereas Google Reader RSS fetches include extra info such as number of subscribers.

I usually avoid cookies (so no "Cookie:" header) since I don't want the content affected too much by session-specific info. And, if a server uses a session id in a dynamic URL rather than a cookie, I can usually figure this out, so that I don't end up crawling your same page a million times with a million different session ids.

Website: I'm very complex. I have many file types. Your headers say "Accept: */*". Do you index all URLs or are certain file extensions automatically filtered?

Googlebot: That depends on what I'm looking for.

If I'm indexing for regular web search, and I see links to MP3s and videos, I probably won't download those. Similarly, if I see a JPG, I will treat it differently than an HTML or PDF link. For instance, JPG is much less likely to change frequently than HTML, so I will check the JPG for changes less often to save bandwidth. Meanwhile, if I'm looking for links as Google Scholar, I'm going to be far more interested in the PDF article than the JPG file. Downloading doodles (like JPGs) and videos of skateboarding dogs is distracting for a scholar—do you agree?

Website: Yes, they can be distracting. I'm in awe of your dedication. I love doodles (JPGs) and find them hard to resist.

Googlebot: Me, too; I'm not always so scholarly. When I crawl for image search, I'm very interested in JPGs. And for news, I'm mostly looking at HTML and nearby images.

There are also plenty of extensions (exe, dll, zip, dmg...), that tend to be big and less useful for a search engine.

Website: If you saw my URL, http://www.example.com/page1.LOL111, would you (whimper whimper) reject it just because it contains an unknown file extension?

Googlebot: Website, let me give a bit more background. After actually downloading a file, I use the Content-Type header to check whether it really is HTML, an image, text, or something else. If it's a special data type like a PDF file, Word document, or Excel spreadsheet, I'll make sure it's in the valid format and extract the text content. Maybe it has a virus; you never know. If the document or data type is really garbled, there's usually not much to do besides discard the content.

So, if I'm crawling http://www.example.com/page1.LOL111 with an unknown file extension, it's likely that I would start to download it. If I can't figure out the content type from the header, or it's a format that we don't index (e.g. mp3), then it'll be put aside. Otherwise, we proceed indexing the file.

Website: My apologies for scrutinizing your style, Googlebot, but I noticed your Accept-Encoding headers say:
Accept-Encoding: gzip,deflate

Can you explain these headers to me?

Googlebot: Sure. All major search engines and web browsers support gzip compression for content to save bandwidth. Other entries that you might see here include "x-gzip" (the same as "gzip"), "deflate" (which we also support), and "identity" (none).

Website: Can you talk more about file compression and "Accept-Encoding: gzip,deflate"? Many of my URLs consist of big Flash files and stunning images, not just HTML. Would it help you to crawl faster if I compressed my larger files?

Googlebot: There's not a simple answer to this question. First of all, many file formats, such as swf (Flash), jpg, png, gif, and pdf are already compressed (there are also specialized Flash optimizers).

Website: Perhaps I've been compressing my Flash files and I didn't even know? I'm obviously very efficient.

Googlebot: Both Apache and IIS have options to enable gzip and deflate compression, though there's a CPU cost involved for the bandwidth saved. Typically, it's only enabled for easily compressible text HTML/CSS/PHP content. And it only gets used if the user's browser or I (a search engine crawler) allow it. Personally, I prefer "gzip" over "deflate". Gzip is a slightly more robust encoding — there is consistently a checksum and a full header, giving me less guess-work than with deflate. Otherwise they're very similar compression algorithms.

If you have some spare CPU on your servers, it might be worth experimenting with compression (links: Apache, IIS). But, if you're serving dynamic content and your servers are already heavily CPU loaded, you might want to hold off.

Website: Great information. I'm really glad you came tonight — thank goodness my robots.txt allowed it. That file can be like an over-protective parent!

Googlebot: Ah yes; meeting the parents, the robots.txt. I've met plenty of crazy ones. Some are really just HTML error pages rather than valid robots.txt. Some have infinite redirects all over the place, maybe to totally unrelated sites, while others are just huge and have thousands of different URLs listed individually. Here's one unfortunate pattern. The site is normally eager for me to crawl:
  User-Agent: *
  Allow: /

Then, during a peak time with high user traffic, the site switches the robots.txt to something restrictive:
  # Can you go away for a while? I'll let you back
  # again in the future. Really, I promise!
  User-Agent: *
  Disallow: /

The problem with the above robots.txt file-swapping is that once I see the restrictive robots.txt, I may have to start throwing away content I've already crawled in the index. And then I have to recrawl a lot of content once I'm allowed to hit the site again. At least a 503 response code would've been temporary.

I typically only re-check robots.txt once a day (otherwise on many virtual hosting sites, I'd be spending a large fraction of my fetches just getting robots.txt, and no date wants to "meet the parents" that often). For webmasters, trying to control crawl rate through robots.txt swapping usually backfires. It's better to set the rate to "slower" in Webmaster Tools.

The Biggest Web Site Usability Mistakes You Can Make

2008-12-01T22:41:00.001-08:00

When you built your first web site, didn’t you just want to promote it everywhere with big bold letters saying, “HEY EVERYONE! COME HERE AND LOOK AT MY WEBSITE! ISN’T IT GREAT?” Or, when you submit your web site to forums for web site reviews, what do you typically ask for? You may write, “Tell me what you think of my web site” or “Which color do you like better, blue or red?” or “Did I optimize for search engines properly?”
The worst mistake you will make as a web site owner is to ask someone to “look” at your web site. It’s like the dreaded, “Do I look fat?” question. There’s never a safe answer. For starters, someone may look slim standing up, but resemble the Buddha when sitting on a couch. You need to assign tasks to get honest answers to these tough questions.
To understand if your web site is meeting its usability requirements, ask people to take it for a spin and try it out—and more specifically, to see if they can answer the following questions:
What is the purpose of the site?
Ann Smarty wrote in Check Your Site Usability With These Fun Tools about the Five Second Test tool. It’s a fun way to explore immediate impressions and experiment by asking, “Tell me what the site is about”, to see if the site’s purpose is communicated clearly. It can’t warn if your shopping cart is broken. It doesn’t alert you if your sales lead form was invasive and turned away potential customers.
When car shopping, good sales people begin by explaining a car’s features and describe updates from previous models. They’ll walk around the car with you and demonstrate how to pack the back with groceries and squeeze in fishing poles. You won’t buy it at this point, however. For now, the sales person is spinning you a tale to help you imagine yourself inside that gorgeous expensive hunk of machine.
Sales people don’t approach you with “Do you think this car would look better in orange?”
What need does it fulfill for me?
Another area of concern that web site owners have is conversions. They’ll ask for feedback by writing, “My sales are down! Can you look at my homepage and tell me what I’m doing wrong?” Sometimes they’ll write, “We just redesigned our entire web site and our data tells us people are still leaving from the homepage. Help!”
If you’ve ever sold a home, you know that real estate agents will tell you to clean up the yard, paint the walls, empty it out so it looks roomy and place flowers around. If all we had to do was to make a house look pretty to sell it, we wouldn’t need real estate agents to show our homes. They’d sell themselves. Web sites with fresh paint on the homepage but no repairs to the information architecture, persuasive content, functionality and user experience can’t be expected to perform miracles.
A good real estate agent will bring potential buyers to a house and encourage them to turn on the water faucets, open closets, and help them visualize 50 people in the family room at Uncle Frank’s birthday party. What if you move in, get arthritis and can’t manage stairs anymore? The value proposition is not just about features and price. It’s about what benefit someone will get by purchasing your service, buying your products or experiencing your online tools. In addition, try to help visitors plan ahead and make logical choices rather than purposely pushing them into a revenue stream that will only benefit you in the long run. Why? Word of mouth marketing, the “long tail”, customer satisfaction and brand management.
Is it responsive to my emotions?
When you wrote up requirements for your online business, did you remember to include emotion? Most likely, it never crossed your mind. Do you watch how people use web sites when they’re in a hurry? Upset? Worried? Stressed? Tired? Hungry?
Google recently launched SearchWiki. Regardless of what you may think about it, what motivated them are their users. Their data shows that searchers want better ways to search, organize, save their research and quickly find favorite web sites. With user feedback, Google can create user personas to help developers understand how searchers use Google when they’ve just been informed of bad news. How do stress and exhaustion affect search behavior?
Consideration for your web site visitors’ emotional state may be vital for your web site. While testing a web site recently for a young adult rehabilitation center, I was pleased to find their content was written in a warm, caring way. The colors were soothing pastels. The pictures showed happy clients. Unfortunately, their content was all about the center and types of therapy. It was long winded, requiring time to read and digest. What wasn’t addressed with bullet point details on the homepage was proof of their claims to calm fears and concerns over ethics. Were there case studies or testimonials? Could a parent talk to other parents who sent a child there?
What if the site visitor is a father at the end of his rope? He’s tired, busy, a widower and shaking with emotion as he scrolls, clicks and reads. Was the site designed for an upset Dad? If not, what could be done to create a better user experience for him? For this site visitor, the site of a healthy looking child beaming into the camera while holding a farm animal might be an emotional hook. Someone available to speak with him “now” may persuade him to call because his feelings are raw at that moment.
Consider an insurance quote site with low conversions. It contains all the bells and whistles, content, easy navigation, and basically the same claims that every other insurance quote company places on their web site. Even if this site should present a reasonable competitive business value proposition, their form may not be converting. Why? In many cases, there is no information about how long the quote process will take. How many pages is the form? How much information is expected from the applicant? Is the follow up response by email or phone call? Is there a choice? Do they show competitive data and choices?
The biggest mistake is to believe that web site appearance matters the most. How it looks is only one part of the process. How it performs is another. What it can give back to site visitors and how effectively it conveys that information will matter even more.

Top SEO Experts Chart

2008-11-27T02:47:00.000-08:00

	Expert (Google pages) (alphabetical list)	Nickname	Website/ Blog (SEJ 2005 Rank)	Location	Favorite Forum	Style
1	2- Aaron Wall (276K) seobook (370K)	seobook	www.seobook.com (2) www.threadwatch.org	State College, PA	SEW Forums
2	Ammon Johns (11K)	Black_Knight	www.e-marketing-news.co.uk	Brighton, England	cre8asite
3	Andy Beal (259K)	-	www.marketingpilgrim.com	Raleigh, NC	-
4	Andy Hagans (52K)	Andy Hagans	www.andyhagans.com www.linkbuildingblog.com	Cincinnati, OH	-
5	Barry Schwartz (387K) rustybrick (188K)	rustybrick	www.rustybrick.com	Suffern, NY	seroundtable
6	Barry Welford (17K)	bwelford	www.cre8asiteforums.com	Montreal, QC, Can.	cre8asite
7	Ben Pheiffer (30)	Phoenix	www.ranksmart.com	Austin, TX	SEOChat
8	Bernard nolastname	Bernard	www.measuring-up.com	Friendswood, TX	SEOChat
9	Brad Callen (86K)	-	www,seoelite.com	Indianapolis, IN	-
10	Brad Fallon (79K)	-	www.bradfallon.com	Atlanta, GA	-
11	Brett Tabke (89K)	Brett_Tabke	www.webmasterworld.com	Austin, TX	wmw.com
	Bruce Clay (162K)	-	www.bruceclay.com	Simi Valley, CA	orgamic listings
12	Chris Boggs (33K)	Chris Boggs	www.avenuea-razorfish.com	Baltimore, MD	SEW Forums
13	Danny Sullivan (699K)	dannysullivan	searchengineland.com daggle.com	Salisbury, England	SEW Forums
14	David Naylor (178K)	DaveN	www.davidnaylor.co.uk	UK	SEW Forums
15	Donna Fontenot (646) dazzlindonna (40K)	dazzlindonna	www.seo-scoop.com	Franklinton, LA	SEO Refugee
16	8- Erik Dafforn (78K)	-	intrapromote.com seo speedwagon (8)	Stow, OH	-
17	5- Greg Boser (55K)	webguerrilla	www.webguerrilla.com (5)	Newhall, CA	SEO Rock
18	Jakob Nielsen (1.4M)	-	www.useit.com	Fremont, CA	-
19	Jennifer Slegg (36K) jensense (155K)	jensense	www.jensense.com	Victoria, BC, Can.	wmw.com
20	Jill Whalen (165K)	Jill Whalen	www.highrankings.com	Ashland, MA	SEW Forums
21	6- Jim Boykin (141K)	webuildpages	www.webuildpages.com www.jimboykin.blog (6)	Troy, NY	(dp).forums
22	Lee Odden (112K)	toprank	TopRank Results TopRank.blog	Mound, MN	SEW Forums seroundtable
23	1- Matt Cutts (1.8M)	matt	www.google.com www.matcutts.com/Blog (1)	Mountain View, CA	-
24	Michael Gray (854K)	Graywolf	www.wolf-howl.com	New York, NY	-
25	Mike Grehan (153K)	-	www.search-engine-book.co.uk www.mikegrehan.com	UK	-
26	Morgan Carey (23K)	seo_guy	www.seo-guy.com	BC, Canada	SEOChat
27	Nick Wilson (199K)	nick	performancing.com performancing.com/blog	Denmark	-
28	3- Rand Fishkin (100K)	randfish	www.seomoz.org www.seomoz..org/blog (3)	Seattle, WA	seroundtable SEW Forums
29	Roger Montti (2K) martinibuster (806K)	martinibuster	www.martinibuster.com	San Francisco, CA	WmW.com
30	Shari Thurow (137K)	-	www.searchenginesbook.com	Carpentersville, IL	-
31	Shawn Hogan (62K)	Shawn	www.digitalpoint.com	San Diego, CA	seroundtable
32	Todd Friesen (20K) oilman (711K)	oilman	www.oilman.ca	Victoria, BC, Can.	SEO Rock
33	4- Todd Malicoat (58K) stuntdubl (370K)	stuntdubl	www.stuntdubl.com (4)	Troy, NY	wmw.com
34	??? ???	Earl Grey	www.syndk8.net	Ireland-?	-
35	7- ??? ??? quadzilla (247K)	quadzilla	seoblackhat.com (7)	East Teknopolis-?	seoblackhat.com
20	170	90	200	150	100	50
36-51

Google XML Data Feed Optimization Tips

2008-11-26T20:25:00.000-08:00

XML Data Feed Optimization Tips

· Add products to the appropriate category

· Always include searchable keywords in product description.

· Make use of custom/optional fields for better visibility on CSEs(utilize it for additional product attributes)

· Add relevant keywords to the product title and description.

· Make sure that you have provided active & correct link of product landing page.

· Monitor competitors prices frequently & strategically decide the sale price. Conduct Price Analytics

· Provide attractive offers like Free Shipping, Discount sale, Clearance sale, coupon & rebates etc...

· Understand Data feed requirements of each CSE while creating, optimizing & submitting the product data feed to them.

· Drop down non-Performing products

· Remove out of stock products

· Conduct FDSI (Features, Description, Specifications, Images) comparison analysis with yours and other merchants in the shopping engine

· Give correct product image

· Conduct A/B testing and track which ad works best for you. Click here for A/B testing basics

· Figure out ROI from each comparison engine and turn off if any specific engine is not performing well.

· Review and submit shopping feed frequently.

· If there is high a CTR (Click through rate) from CSE with no returns/sales there might the following possible problems related to your feeds and/or landing page.

Wrong Placement of Product in the shopping engine will also yield tons of irrelevant clicks...

· A Shopper might come to the retailer site with an intention to learn about the product.

· A slight difference in the title and description will discourage users to buy products.

· Links do not take users to the correct landing page.

· Unreliable Merchant - (Look and Feel is not good, Not a secure website, Bad information structuring, invisible or wrong placement of shopping cart, etc)

· Product specification mismatch

· Price mismatch

· Shipping information mismatch

· Long checkout process

Site Usage: General but Important Terms of Analytics

2008-11-25T23:05:00.001-08:00

Let's go to know about some of important terms generally we need to see in web analytics. These terms describe about whole site position in brief summary.

Visits: When a user request from www server then it is called Visit. Therefore we can say Visits represent the total number of times people have visited your web site. There is an important point to note that When a user is not active over 30 minutes on the server, another request is recognized as an another visit.

Page Views: The total number of distinct pages seen by users with in a time frame after visiting on site. For example, if a Web site contains 100 pages and an estimated 1000 visitors per month, the site receives approximately 100000 monthly page views (100X1000=100000). Page Views is also known as page impressions.

% Pages/Visit: The average of pages seen per visit. It tells that how many pages seen by a user or per visit.

% Bounce Rate: The percentage of users or visitors who arrive at a web site entry page, then leave without visiting or seen other pages of website. Means to say user leaves the site without going into deep page of site.

Avg. Time on Site: Counted estimated time of presence of visitor on site. Approx for 30 minutes only. After idle 30 minutes it is counted new visit.

% New Visits: It shows the percentage of actual new visits of visitors on website.

Quick security checklist for webmasters

2008-11-24T23:06:00.001-08:00

In recent months, there's been a noticeable increase in the number of compromised websites around the web. One explanation is that people are resorting to hacking sites in order to distribute malware or attempt to spam search results. Regardless of the reason, it's a great time for all of us to review helpful webmaster security tips.

Obligatory disclaimer: While we've collected tips and pointers below, and we encourage webmasters to "please try the following at home," this is by no means an exhaustive list for your website's security. We hope it's useful, but we recommend that you conduct more thorough research as well.

* Check your server configuration.

Apache has some security configuration tips on their site and Microsoft has some tech center resources for IIS on theirs. Some of these tips include information on directory permissions, server side includes, authentication and encryption.

* Stay up-to-date with the latest software updates and patches.

A common pitfall for many webmasters is to install a forum or blog on their website and then forget about it. Much like taking your car in for a tune-up, it's important to make sure you have all the latest updates for any software program you have installed. Need some tips? Blogger Mark Blair has a few good ones, including making a list of all the software and plug-ins used for your website and keeping track of the version numbers and updates. He also suggests taking advantage of any feeds their websites may provide.

* Regularly keep an eye on your log files.

Making this a habit has many great benefits, one of which is added security. You might be surprised with what you find.

* Check your site for common vulnerabilities.

Avoid having directories with open permissions. This is almost like leaving the front door to your home wide open, with a door mat that reads "Come on in and help yourself!" Also check for any XSS (cross-site scripting) and SQL injection vulnerabilities. Finally, choose good passwords. The Gmail support center has some good guidelines to follow, which can be helpful for choosing passwords in general.

* Be wary of third-party content providers.

If you're considering installing an application provided by a third party, such as a widget, counter, ad network, or webstat service, be sure to exercise due diligence. While there are lots of great third-party content on the web, it's also possible for providers to use these applications to push exploits, such as dangerous scripts, towards your visitors. Make sure the application is created by a reputable source. Do they have a legitimate website with support and contact information? Have other webmasters used the service?

* Try a Google site: search to see what's indexed.

This may seem a bit obvious, but it's commonly overlooked. It's always a good idea to do a sanity check and make sure things look normal. If you're not already familiar with the site: search operator, it's a way for you to restrict your search to a specific site. For example, the search site:googleblog.blogspot.com will only return results from the Official Google Blog.

* Use Google's Webmaster Tools.

They're free, and include all kinds of good stuff like a site status wizard and tools for managing how Googlebot crawls your site. Another nice feature is that if Google believes your site has been hacked to host malware, our webmaster console will show more detailed information, such as a sample of harmful URLs. Once you think the malware is removed, you then can request a reevaluation through Webmaster Tools.

* Use secure protocols.

SSH and SFTP should be used for data transfer, rather than plain text protocols such as telnet or FTP. SSH and SFTP use encryption and are much safer. For this and many other useful tips, check out StopBadware.org's Tips for Cleaning and Securing Your Website.

* Read the Google Online Security Blog.

Here's some great content about online security and safety with pointers to lots of useful resources. It's a good one to add to your Google Reader feeds. :)

* Contact your hosting company for support.

Most hosting companies have helpful and responsive support groups. If you think something may be wrong, or you simply want to make sure you're in the know, visit their website or give 'em a call.

We hope you find these tips helpful. If you have some of your own tips you'd like to share, feel free to leave a comment below or start a discussion in the Google Webmaster Help group. Practice safe webmastering!

7 must-read Webmaster Central blog posts

2008-11-24T23:03:00.000-08:00

Our search quality and Webmaster Central teams love helping webmasters solve problems. But since we can't be in all places at all times answering all questions, we also try hard to show you how to help yourself. We put a lot of work into providing documentation and blog posts to answer your questions and guide you through the data and tools we provide, and we're constantly looking for ways to improve the visibility of that information.

While I always encourage people to search our Help Center and blog for answers, there are a few articles in particular to which I'm constantly referring people. Some are recent and some are buried in years' worth of archives, but each is worth a read:

1. Googlebot can't access my website
Web hosters seem to be getting more aggressive about blocking spam bots and aggressive crawlers from their servers, which is generally a good thing; however, sometimes they also block Googlebot without knowing it. If you or your hoster are "allowing" Googlebot through by whitelisting Googlebot IP addresses, you may still be blocking some of our IPs without knowing it (since our full IP list isn't public, for reasons explained in the post). In order to be sure you're allowing Googlebot access to your site, use the method in this blog post to verify whether a crawler is Googlebot.
2. URL blocked by robots.txt
Sometimes the web crawl section of Webmaster Tools reports a URL as "blocked by robots.txt", but your robots.txt file doesn't seem to block crawling of that URL. Check out this list of troubleshooting tips, especially the part about redirects. This thread from our Help Group also explains why you may see discrepancies between our web crawl error reports and our robots.txt analysis tool.
3. Why was my URL removal request denied?
(Okay, I'm cheating a little: this one is a Help Center article and not a blog post.) In order to remove a URL from Google search results you need to first put something in place that will prevent Googlebot from simply picking that URL up again the next time it crawls your site. This may be a 404 (or 410) status code, a noindex meta tag, or a robots.txt file, depending on what type of removal request you're submitting. Follow the directions in this article and you should be good to go.
4. Flash best practices
Flash continues to be a hot topic for webmasters interested in making visually complex content accessible to search engines. In this post Bergy, our resident Flash expert, outlines best practices for working with Flash.
5. The supplemental index
The "supplemental index" was a big topic of conversation in 2007, and it seems some webmasters are still worried about it. Instead of worrying, point your browser to this post on how we now search our entire index for every query.
6. Duplicate content
Duplicate content—another perennial concern of webmasters. This post talks in detail about duplicate content caused by URL parameters, and also references Adam's previous post on deftly dealing with duplicate content, which gives lots of good suggestions on how to avoid or mitigate problems caused by duplicate content.
7. Sitemaps FAQs
This post answers the most frequent questions we get about Sitemaps. And I'm not just saying it's great because I posted it. :-)

Sometimes, knowing how to find existing information is the biggest barrier to getting a question answered. So try searching our blog, Help Center and Help Group next time you have a question, and please let us know if you can't find a piece of information that you think should be there!

Taking advantage of universal search

2008-11-24T22:51:00.000-08:00

we unveiled exciting changes in our search results. With universal search, we've begun blending results from more than just the web in order to provide the most relevant and useful results possible. In addition to web pages, for instance, the search results may include video, news, images, maps, and books. Over time, we'll continue to enhance this blending so that searchers can get the exact information they need right from the search results.

This is great news for the searcher, but what does it mean for you, the webmaster? It's great news for you as well. Many people do their searches from web search and aren't aware of our many other tools to search for images, news, videos, maps, and books. Since more of those results may now be returned in web search, if you have content that is returned in these others searches, more potential visitors may see your results.

Want to make sure you're taking full advantage of universal search? Here are some tips:

Google News results
If your site includes news content, you can, submit your site for inclusion in Google News. Once your site is included, you can let us know about your latest articles by submitting a News Sitemap. (Note News Sitemaps are currently available for English sites only.)

News Archive results
If you have historical news content (available for free or by subscription), you can submit it for inclusion in News Archive Search.

Image results
If your site includes images, you can opt-in to enhanced Image search in webmaster tools, which will enable us to gather additional metadata about your images using our Image Labeler. This helps us return your images for the most relevant queries. Also ensure that you are fully taking advantage of the images on your site.

Local results
If your site is for a business in a particular geographic location, you can provide information to us using our Local Business Center. By providing this information, you can help us provide the best, locally relevant results to searchers both in web search and on Google Maps.

Video results
If you have video content, you can host it on Google Video, YouTube, or a number of other video hosting providers. If the video is a relevant result for the query, searchers can play the video directly from the search results page (for Google Video and YouTube) or can view a thumbnail of the video then click over to the player for other hosting providers. You can easily upload videos to Google Video or to YouTube.

Our goal with universal search is to provide most relevant and useful results, so for those of you who want to connect to visitors via search, our best advice remains the same: create valuable, unique content that is exactly what searchers are looking for.

Taking advantage of universal search

2008-11-24T22:50:00.000-08:00

Universal search and personalized search were two of the hot topics at SMX West last month. Many webmasters wanted to know how these evolutions in search influence the way their content appears in search results, and how they can use these features to gain more relevant search traffic. We posted several recommendations on how to take advantage of universal search last year. Here are a few additional tips:

1. Local search: Help nearby searchers find your business.
Of the various search verticals, local search was the one we heard the most questions about. Here are a few tips to help business owners get the most out of local search:
* Add your business to our Local Business Center
* Make sure your business is listed under the appropriate categories
* Make sure your business name, address and phone number appear on your website, and that they're accessible to search engines
* Add your business hours, images, coupons, or additional information to help searchers determine whether your business meets their needs
2. Video search: Enhance your video results.
Several site owners asked whether they could specify a preferred thumbnail image for videos when they appear in search results. Good news: our Video Sitemaps protocol lets you suggest a thumbnail for each video.
3. Personalized search basics
A few observations from Googler Phil McDonnell:
* Personalization of search results is usually accomplished through subtle ranking changes, rather than a drastic rearrangement of results. You shouldn't worry about personalization radically altering your site's ranking for a particular query.
* Targeting a niche, or filling a very specific need, may be a good way to stand out in personalized results. For example, rather than creating a site about "music," you could create a site about the musical history of Haiti. Or about musicians who recorded with Elton John between 1969-1979.
* Some personalization is based on the geographic location of the searcher; for example, a user searching for [needle] in Seattle is more likely to get search results about the Space Needle than, say, a searcher in Florida. Take advantage of features like Local Business Center and geographic targeting to let us know whether your website is especially relevant to searchers in a particular location.
* As always, create interesting, unique and compelling content or tools.
4. Image search: Increase your visibility.
One panelist presented a case study in which a client's images were being filtered out of search results by SafeSearch because they had been classified as explicit. If you find yourself in this situation and believe your site should not be filtered by SafeSearch, use this contact form to let us know. Select the Report a problem > Inappropriate or irrelevant search results option and describe your situation.

Building SEO Momentum by Using A Consistent Site Structure

2008-11-24T05:24:00.000-08:00

Change. It is a part of life, especially on the web. Evolve or die.

But some things need not change to be successful. In some cases change undermines your momentum, particularly in the field of search, where most of the traffic goes to the top couple ranked sites.

One of the biggest problems in the field of SEO for enterprise-level sites is content management. Product lines, editorial calendars, marketing, and content management systems often dictate that pieces and parts of a site are organized in a sub-optimal way and/or move locations.

Back in 1998 Tim Berners-Lee stated that Cool URIs don’t change:

There are no reasons at all in theory for people to change URIs (or stop maintaining documents), but millions of reasons in practice.

In theory, the domain name space owner owns the domain name space and therefore all URIs in it. Except insolvency, nothing prevents the domain name owner from keeping the name. And in theory the URI space under your domain name is totally under your control, so you can make it as stable as you like. Pretty much the only good reason for a document to disappear from the Web is that the company which owned the domain name went out of business or can no longer afford to keep the server running. Then why are there so many dangling links in the world? Part of it is just lack of forethought.

The 10 year old document is as relevant today as the day it was published. And perhaps even more so since search is the primary mode of navigation on the web.

Map it Out in Advance

One of the easiest ways to avoid site decay is to plan your site out in advance. When you know where something belongs and have given it proper foresight its half-life is much greater than a site built nearly at random. The following image shows an example of how you can plan out URLs, page titles, meta descriptions, on page headings, and related keyword modifiers to include in the page.

Archiving & Static Hub Sections

Many publishing based business models cover a seasonal topic, like Christmas or the Coachella music festival. Each year those publishers could create a new section with a filename like christmas2009, but doing so may make it harder to rank for core generic phrases like Christmas. When a person links to a very specific URL it does not consolidate PageRank and anchor text between years. So in a zen-like fashion changing URLs each year throws away your old anchor text and has you starting from scratch again.

A better solution is to use a filename like christmas, and each year update it to talk about the current year, while archiving the default page for the prior year as something like christmas/2009/. In doing this, the core URL (yoursite.com/christmas) keeps building PageRank and anchor text each year, building off of last year’s success. Old content remains archived, and can be easily findable by linking to archives from the core page.

If you wanted to combine many core related URLs like christmas2008 and christmas2007 into 1 URL you can do so through the use of 301 redirects. Keep in mind that you want to think through what URLs you want changed, which ones you do not want changed, and then use a server header checker to verify the proper response codes.

Reclaiming Lost Link Equity

If someone is linking to a page that no longer exists on your site you are wasting link equity and traffic. Some content management systems offer features or extensions that can be used to track 404s and other errors. Drupal offers an error log and this Wordpress redirection plug in shows you pages people attempted to visit that returned a 404 status code. Your server logs should also help locate any pages with 404 status codes.

Once you discover a well linked to URL that no longer exists you have a couple options:

* Look through your back ups and/or Archive.org to see if any copies of the page’s content are still available. If the content was of high quality and you are uncertain why it went away then consider republishing the content on the same URL.
* If the page moved or the content is no longer relevant to your site, then 301 redirect the URL to a related page.

Google offers two tools to help you reclaim misdirected traffic and link popularity that is being wasted on dead URLs. They offer a 404 widget which can be used to help site users find related content on your site, which is useful for when someone makes a typo when typing out your filenames. Google Webmaster Central allows you to find 404 pages on your site that webmasters are linking to.

Does your site have a feed? A feed can connect you to your readers and keep them returning to your content. Most blogs have feeds, but increasingly, other types of sites with frequently changing content are making feeds available as well. Some examples of sites that offer feeds:

2008-11-20T22:24:00.000-08:00

Does your site have a feed? A feed can connect you to your readers and keep them returning to your content. Most blogs have feeds, but increasingly, other types of sites with frequently changing content are making feeds available as well. Some examples of sites that offer feeds:

News sources such as the New York Times publish feeds of their latest stories
Companies like Apple publish feeds of their press releases, as well as a few other feeds.
Blogs including the Official Google Blog publish feeds with their latest posts
Shopping sites like Buy.com publish feeds with noteworthy deals

Find out how many readers are subscribed to your feed
If your site has a feed, you can now get information about the number of Google Reader and Google Personalized Homepage subscribers. If you use Feedburner, you'll start to see numbers from these subscriptions taken into account. You can also find this number in the crawling data in your logs. We crawl feeds with the user-agent Feedfetcher-Google, so simply look for this user-agent in your logs to find the subscriber number. If multiple URLs point to the same feed, we may crawl each separately, so in this case, just count up the subscriber numbers listed for each unique feed-id. An example of what you might see in your logs is below:

User-Agent: Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; 4 subscribers; feed-id=1794595805790851116)

Making your feed available to Google
You can submit your feed as a Sitemap in webmaster tools. This will let us know about the URLs listed in the feed so we can crawl and index them for web search. In addition, if you want to make sure your feed shows up in the list of available feeds for Google products, simply add a < link> tag with the feed URL to the section of your page. For instance:

link href="http://www.example.com/atom.xml" rel="alternate" title="Your Feed Title" type="application/atom+xml">link

Remember that Feedfetcher-Google retrieves feeds only for use in Google Reader and Personalized Homepage. For the content to appear in web search results, Googlebot will have to crawl it as well.

Don't yet have a feed?
If you use a content management system or blogging platform, feed functionality may be built right now. For instance, if you use Blogger, you can go to Settings > Site Feed and make sure that Publish Site Feed is set to Yes. You can also set the feed to either full or short and can add a footer. The URL listed here is what subscribers add to their feed readers. A link to this URL will appear on your blog.

More tips from the Google Reader team
In order to provide the best experience for your users, the Google Reader team has also put together some tips for feed publishers. This document covers feed best practices, common implementation pitfalls, and various ways to promote your feeds. Whether you're creating your feeds from scratch or have been publishing them for a long time, we encourage you to take a look at our tips to make the most of your feeds. If you have any questions, please get in touch.

Keyword Rich Internal Anchor Text - How Much Is Too Much?

2008-11-20T21:20:00.001-08:00

We’ve heard and seen evidence that internal anchor text over-optimization (sorry, Jill, but this has really become a generally recognizable term) can result in Google penalty. Rand Fishkin, for example mentioned this issue in PRO tips as well as on the blog.
Still, while we are pretty sure, this penalty does exist, we don’t know how many internal keyword-rich links are actually too many.
Another great WebmasterWorld thread shares some results of experiments regarding this issue and while you are free to argue the overall accuracy of such tests, I thought it should definitely be brought to discussion here.
The test involved different patterns of linking from subpages to the home page which yielded the following results:

Inter-linking pattern	Google’s reaction
Every site page (~90 pages) links to the home page in the sitewide nav bar using the same keyword.	6 page drop in Google rankings.
Every site page (~90 pages) links to the home page in-content using the same keyword.	6 page drop in Google rankings (after the short period of improved rankings).
Every site page (~90 pages) links to the home page in-content using variations of a keyword.	3 page drop in Google rankings.
The first 10 pages listed in google.com for site:domain.com links to the home page.	Increase in Google rankings (from #5 to #3).

Just a few notes:

As to the last point, I suspect that any 10 pages (not only those listed top 10 in Google) should be OK with Google (as long as these pages are crawled regularly and frequently enough).
As CainIV states in the thread, the drop in rankings was easy to cope with: as soon as he removed the keyword-rich links, rankings were back (and this probably accounts for the domain trust;
Both the effect on rankings and the penalty may largely depend on the terms: how competitive and also how “shady” they are (I imagine sudden heavy optimization for ‘poker’ related terms can result in quicker and more serious penalties).

User Generated Content, SEO & Branding… Work It!

2008-11-20T21:13:00.000-08:00

Consumer - or User - generated content (CGC / UGC) is becoming more and more important to the success of online retailers, and to businesses who rely on their website to drive brand exposure and generate contacts.

What is Consumer Generated Content?

It is content that users (or consumers) write and submit online and can be:

· Product reviews (e.g. Amazon.com)

· Service reviews (e.g. Google Local)

· Seller reviews (e.g. ebay)

· Blog comments

· News comments

· Forum posts or comments

· Video/Audio and Photo sharing sites (e.g. YouTube and Flickr)

· Social networking sites comments or posts (e.g. Twitter, SecondLife, Facebook, MySpace)

· Wikis

· Composite review sites (e.g. ConsumerReport.org)

Is Consumer Generated Content (CGC) Really Such A Big Deal?

The data is two-sided; and both sides of the coin – Consumers and Marketers – answer with a resounding ‘Yes!’:

1. Consumers are relying more and more on UGC:

· According to Rubicon, ‘online comments and reviews posted by the enthusiasts are second only to word of mouth as a purchase driver for all web users. Those personal reviews are far more influential than official reviews posted by a website or magazine, or information posted online by a manufacturer.’

· The Rubicon data also indicates that around 80% of all UGC is contributed by around 9% of the US online population. This 9% is in the majority impartial, honest and equally likely to post both negative and positive comments or opinions. In some cases, such as on Amazon, the reviews themselves are open to voting:

· As of October 2008, almost half of US online adults read ratings and reviews at least once a month, and 19% post them. Nearly twice as many read reviews compared with 2007. (The Growth Of Social Technology Adoption, Forrester, 2008)*

· Satisfaction for those who recalled customer reviews on the retailers’ site is 10% higher than those who said there were no reviews offered. (30 UK Online Retail Satisfaction Index, January 2008, ForeSee Results)*

· 83% of shoppers said online product evaluations and reviews had at least some level of influence on their purchasing decisions. (Opinion Research Corporation, an infoGROUP company, July 2008)*

· 74% agree—including 14% who strongly agree—that they choose companies and brands based on what others say online about their customer service experiences, the survey shows. (Society for New Communications Research, May 2008)*

· 70% of online consumers said they use the Internet to research everyday grocery products. (Prospectiv, 2008)*

· 91% of millionaires say they always or often look at reviews before buying luxury goods; 68% of ultra-affluent shoppers use consumer reviews. (Unity Marketing/Google study, reported in AdAge October 2008)*

2. Marketers are relying more and more on UGC:

· The Shop.org State of Retailing Online study, conducted by Forrester Research, found only 26% of the 137 top retailers surveyed offered customer ratings and reviews, but 96% of them ranked customer ratings and reviews as an effective or very effective tactic at driving conversion. (Forrester)*

· 68% of online marketers believe “media is in big trouble and will lose dollars to user-generated content.” (iMedia Connection, February 2008)*

· By 2020, 84% of marketers agree that building customer trust will become marketing’s primary objective, and 82% agree that collaboration with customers will prevail over marketing. (1to1 Media survey of the 1to1 Xchange panel, April, 2008)*

· 11% of retailers reported a 20% or more overall increase in conversions as a result of adding reviews to their sites, 21% reported an 11% to 20% increase and 5% reported a 1% to 10% increase. (eTailing Group, 2008)*

· Consumers were willing to pay between 20 to 99% more for a 5-star rated product than for a 4-star rated product, depending on the product category. (comScore/Kelsey, October 2007)*

· Reviews usage drives higher spending: 27% of users report an increase of 5-10%; almost 7% report an increase of 20%+. (Avenue A/Razorfish “Digital Consumer Behavior Study,” October 2007) *

· In a study of a major electronics retailer site of 30,000 search visitors landing on user review pages converted 60% more often, spent 50% more, and viewed 82% more pages than search visitors to other pages. (Bazaarvoice case study with major electronics retailer)*

Case Study: PETCO (2007)*:

Shoppers who browsed the site’s “Top Rated Products” page, which features products rated most highly by customers, had a 49% higher conversion rate than the site average.
Giving shoppers the ability to sort products within a category by customer rating led to a sales increase of 41% per unique visitor.
After their order, PETCO asked customers, “What online tool most influenced your purchase decision?“ The Number 1 answer was product ratings and reviews, with site search coming in a distant second.
PETCO realized a 5X increase in email click-through rates by including relevant ratings and reviews content in the campaign promotion.
Products with reviews have a 17% lower return rate than those without reviews.

…But, the impact of UGC doesn’t stop there…

UGC Influences Offline Commerce

· For every $1 in online sales, the Internet influenced $3.45 of store sales. (eMarketer, 2007)

· 24% of internet users reported using online reviews prior to paying for a service delivered offline. (comScore/The Kelsey Group, October 2007)

· More than 75% of review users in nearly every category reported that the review had a significant influence on their purchase, with hotels ranking the highest (87%). (comScore/The Kelsey Group, October 2007)

· 97% of those surveyed who said they made a purchase based on an online review said they found the review to have been accurate. (comScore/The Kelsey Group, October 2007)

· Consumers who shop online for digital cameras and TVs spend 10% more on in-store purchases than consumers who do not search online. (ChannelForce for Yahoo Search Marketing)

· 90% of those surveyed say they have a better overall shopping experience when they research products online before shopping in-store. (Harris Interactive, October 2007)

“But What About Negative UGC?”

If people are researching a product online, they are looking for what might be wrong with it. If you can be transparent on your own site about your product or service by offering reviews, then you can be sure that both positive and negative feedback (if warranted) will be included.**

1. Negative Reviews are actually of interest:

Based on the fact that folks are likely to continue searching if there is nothing negative about a product, just to make sure, especially with electronics or high-end purchases, if you don’t offer a transparent insight into your offering on your own site, by people who have actually bought from you, these researchers are very likely to go elsewhere to keep researching.**

· “As for the product(s) with negative reviews … my experience is that negative reviews do not hurt a product as long as there are also positive reviews associated with it.” - Don Zeidler, Director, Burpee.com**

2. Negative reviews provide helpful feedback to consumers and R&D:

· “For example, what’s one of the best selling products so far this decade? The Apple iPod. It gets a ton of positive reviews, but one negative comment you’ll see over and over again is that the surface of the iPod scratches easily. Customers say things that the retailer and Apple can’t say: “When you buy this, get a case”. Obviously this is not stopping customers from buying the iPod, but this constructive advice is getting them to buy a case.” – Sam Decker, VP Marketing, BazaarVoice

3. Negative Reviews provide authenticity:

Why would a well known, enormously powerful and successful online retailer continue selling a product that had far more negative reviews than positive? Should they? Yes!

· Case Study**:

· Product: Hot Wheels Slimecano

· Reseller: Amazon.com

· Some Review Titles:

o “Junk, not worth any price”

o “I wish I could give it zero stars”

o “If you want your son to cry, buy this toy!”

o “If you want your husband to cry, buy this toy!”

· Reseller Objective:

If the retailer’s mission is to be a trusted editor of its assortment, then products with overwhelmingly negative reviews will prune the assortment quickly for the best products. But the reseller needs to continue selling the product as an ‘objective provider’ and let the market forces, or in this case, parental voices, have their impact. After a year and half, Amazon stopped selling the toy, but kept the reviews, the image and the page… Good Job!

4. Positive Reviews Outweigh Negatives:

· 62 percent of brand-related talk features products in a positive light, while less than 10% of conversations feature products negatively. (KellerFay Group)

· 60 percent of online shoppers provide feedback about shopping experiences, and they are more likely to give feedback about a positive experience than a negative one. (Jupiter)

Conclusion

If you sell products and services, and if you have a web site, make sure you allow for reviews. If your products are great to mediocre, you should get far more positive than negative feedback, and a resultant positive impact on sales and revenue.

If you sell a crappy product – in the words of Sam Decker:

1. Without reviews, you keep selling the product and risk costly returns and low customer satisfaction

2. With reviews, you can use the leading indicator of negative reviews and quickly remove this product from inventory to reduce returns and improve satisfaction

3. Or, just allow the negative reviews to steer customers to a more satisfying purchase within the category. Let the best products win, and you will win.

Therefore, make sure you enable and allow feedback, optimize your review pages, and get traction for them in the search results by supporting brand evangelists, and answering detractor issues promptly and diplomatically.

Happy UGC integration!!!

Rating Tactics With The AM Marketing Cube

2008-11-20T21:12:00.001-08:00

Bottom Line First

What’s most important to you in your Internet marketing campaign? Quick results? Long lasting results? High ROI? Use the AudetteMedia Marketing Cube to identify what tactics to use:

The AM Cube Live - Try It Yourself

Then head back here for more detailed information on what went in to building Da’ Cube.

Behind the Building of the Cube

Internet marketing offers a number of powerful tactics that can be employed to maximize the attainment of campaign goals. To use them most effectively, it’s important to develop an integrated strategy in advance that does the following:
1. Defines specific goals
2. Establishes a baseline
3. Defines specific tactics
4. Defines budget allocation
5. Defines implementation milestones, metrics and measurements
Steps 1, 3 and 4 will be discussed here. Steps 2 and 5 pertain to analytics and will be discussed in another post coming soon.
Step 1: Define Specific Goals
Measurable Return on Investment (ROI)
There is always a goal to achieve some type of return on investment (ROI). But that can vary, depending on the type of the ROI desired. For example, a focus on building brand awareness will employ different tactics than a focus on building immediate bottom line sales. And it is more difficult to establish a measurable ROI with a branding campaign than with a campaign that is more direct marketing oriented.
After establishing measurable ROI goals, two other variables that should be identified are related to time:
Time to Results
This is defined as how long it takes to achieve measurable results after launch of a campaign. There are times when Time to Results is of the utmost importance, as might be the case where an initial marketing campaign is used as a pilot project with future efforts based on the early results.
Persistency of Results
Persistency is defined by how long results are produced after a formal internet marketing campaign is ended. This is an often overlooked aspect of Internet marketing tactics and can be a real added bonus to ROI. Tactics such as SEO and Content Development rank high on the Persistency scale, whereas display advertising and PPC rank low.
Although there is undoubtedly a subjective element in grading one tactic versus another using these criteria, it can still be a useful tool in designing an Internet marketing strategy tailored to match specific goals. In the AudetteMedia Cube we rate the following internet marketing tactics as to the three elements of ROI, Time & Persistency:
• Search Engine Optimization (On-page)
• Pay per Click (PPC)
• Display Advertising
• Blogging
• Content Development
• Affiliate Marketing
• Social Networking
• Online PR
• Email Marketing
In building the AudetteMedia Cube, we use a scale of 1 to 100 and rate each tactic on each of the three criteria using subjective judgment based on our experience:

Time to Results: 1 is slowest, 100 is quickest
Persistency of Results: 1 is least persistent, 100 is most persistent
ROI: 1 is lowest measurable ROI, 100 is highest measurable ROI

As mentioned, these ratings are based on our subjective judgment. We would like to hear from you if you disagree with our numbers (or even if you agree!). For example, we rate Organic SEO 25-90-90: relatively slow to produce results, results that persist for a long time, and high measurable ROI. Do you agree?
How about Display Advertising? We rate it 90-5-50: quick to produce results, little or no persistency, and medium measurable ROI.
Or PPC? We rate it 100-5-60: immediate results, little or no persistency, and relatively high measurable ROI.
What do you think?
Step 3: Define Specific Tactics
The next step is to rank the priorities of the marketing campaign on a scale of 1 to 10 and to plug them into the AudetteMedia Cube worksheet. Here’s an example of a campaign heavily focused on direct sales results and the two highest priorities of the campaign are Persistency of Results and ROI. Time to Results is of lesser importance as this is budgeted as a long-term campaign and there is no pressure to achieve short-term results:

You can see that Content Development and SEO (Organic) run a close first and second when these are the priorities.
There are times when Time to Results is of the utmost importance, as might be the case where a marketing manager is using an initial campaign as a proof of concept in order to justify a more extensive campaign. Here’s how the tactics rank under such a scenario:

Pay-per-Click (PPC) is ranked as the #1 tactic when the goal is for quick results and not much importance is placed on persistency of results. Unlike in the first example, where there wasn’t much concern about quick results, SEO and Content development are further down the scale.

The Cube Live! - Try It Yourself - Interactive Worksheet

Step 4: Define Budget Allocation
Once these priorities are set, it is possible to make an initial budget allocation of resources to the tactics that are most likely to produce desired results. This will inevitably shift over time as results are monitored and measured, but it provides an informed way to begin the implementation process.

Keyword anaysis with Advanced segments and custom reporting

2008-11-20T21:07:00.000-08:00

In the first part on Google Analytics for SEO we looked at advanced segmentation and custom reporting for search traffic analysis. This time around we’re going to look at different metrics and key indicators for keywords and phrases that are being used to reach your site.
What’s important with this is that we’re looking to adapt our SEO programs by better understanding how various keywords/phrases are performing.
Before we get going, it should once again be stressed, be creative. These are but some examples to start you down the path of crafting approaches that best work your situation.

Engagement by keyword

Right away, to get us started, navigate to ‘Traffic Sources > Keywords’ this is our main focus for this round. Now, last time we created a custom segment to measure our prime engagement visitors by segmenting those that went 3 pages or more into the site.
We can now use this segment on our keyword report;

And of course the ‘Goal Conversion’ tab –

There are a few things we’re looking at here;

Level of engagement – which terms are bringing in the visitors that engage with the site the most. You can then do some deeper analysis as to why a target is performing well or why it isn’t. This can be used to better adapt the over-all keyword targeting of the SEO program.
Engagement V. Conversions – As you can see above, some terms are going to accomplish your goals better than others. This begins to help focus future efforts as far as where you invest your resources in your SEO program.

What we can see from that above is that while some terms to have some great traction engagement wise, they don’t always ultimately lead to the best conversions. Another evident factor is that it re-affirms our desire to engage visitors on this particular site as deeper engagement does show itself to lead to higher conversions ultimately.

At a glance

Now, using the default report can give us some in-depth understanding, but I like to have an over-view; so let’s create a custom report. Go to ‘Custom Reports’;

This will give us a report which is more of an ‘at a glance’ view incorporating engagement metrics AND conversions data as well. Using our earlier custom segment we’d have this;

Once more, I normally add other conversion points, but I leave that to you. The main thing is we can look at some of our core keywords to gain insight into the engagement and conversion points related to them. If I am deciding where to concentrate my efforts ( as far as something like link building is concerned) on terms that are performing best. This is essential to an effective SEO campaign.

Geo Targeting

Last time we also created an advanced segment for our Australian visitors since this program is specifically targeting this market. So now let’s apply that to our Keyword Engagement Report – and we’d have something like this;

Once more we’re looking at not only the efficacy of the keyword targeting but also how well the geo-targeting efforts are going. I this case, it’s going well and we can drill down other ‘at a glance’ engagement metrics for each term.
By creating segments, that are important to your site, you can have a nice set of engagement metrics to better understand the performance of your core terms. Now let’s look at primary and secondary terms.

Primary and secondary keyword targeting

I don’t know about you, but we generally have primary and secondary targets with each SEO program. These are essentially groups of terms that have been identified as the money terms and secondary ones. For this we’ll be creating some new segments…

Depending if you want to track ‘just’ the primary terms, or inclusive of connectors (used in long tail) you can use either ‘matches exactly’ or ‘contains’ – it’s situational.
Now if we go to ‘Traffic Sources > Search Engines” and then activate our two keyword segments that we just created. We’ll get something like this;

This shows us that our primary terms are certainly not only pulling greater traffic, as expected, but also engaging the traffic better. It’s worth noting I am using constructors ( inclusive of primary targets). As always we also want to look at the conversion data to look for weaknesses which can be further mined.

Engagement by query type

Another area worth having as an overview on is query types, that is the general nature of a keyword/phrase. Common ones are transactional (seeking a purchase, support information and so forth), informational and geo-modifiers (location specific). So let’s go create a few more advanced analytics.
Transactional query modifiers

This time, add the following one at a time as shown above;

Find
Get
Buy
Purchase
Locate

Compare
Shop
Store
Services

…and so on. Try mining your keyword referrer data for transactional terms related to your market. Once done, we can name this (KWs Transactional) and save.
Now create one for informational queries. This time use this list;

Information
Website
Help
Resources
How to
Tutorial
Guide

Samples
Examples
Ideas
News
Tips
Learn
Site

… and mine your keyword data as always.
If we return to our ‘Traffic sources > Search engines’ report and apply our new segmentation; we have something like this;

After that? Well you can also create another segment for geographic triggers that you may have as well as one for brand related searches (company name, product names, your name..etc..). By looking a search traffic engagement and related conversions a deeper understanding of your over-all content creation and term targeting invariably emerges.

Dig in and muck about

It should be noted that you should be cross referencing your mean average rankings and geo-targeted rankings with ALL kw analytic data and analysis. Often times we find that certain search types (transactional, informational etc..) can lead to deeper engagement. This data is but starting point for deeper research.
The easy access across reports for cross referencing with conversions, make these advanced segments and custom reports quite useful. Each website and market is going to have different key indicators, by playing with the goodies we looked at here you should be able to get your own creative juices flowing into ways to best identify keyword interactions produced from your SEO efforts.
Next time we’re going to look at the content engagement from an SEO point of view. We shall then be able to wrap all our new ideas up and how each can be analyzed for adaptation of ongoing search optimization programs.

SEO Tools & Tips

2008-11-20T21:04:00.001-08:00

In my day to day SEO for my clients and for myself, I’ve come across a number of tools of the trade, some really good, some, well, not so good. Here are a few items in my geek toobox that I use daily and highly recommend.
1. Keyword Tool - I have tried them all and this is the one I always go back to for my keyword research. Not only does it give you variations on the keyword phrase you are searching for, but also provides the WordTracker count and daily estimated searches on Google, Yahoo and MSN along with shortcuts to various tools like Google Trends, Keyword Discovery and several other online tools. And, you can export the thing as a CSV file. Way to go Aaron!
2. Check Server Headers Tool - Quick and easy way to check on whether your URL is being seen and followed properly by the spiders. For instance, I recently installed a WordPress plugin which appeared to work fine in a browser, but when I checked the page URLs that it produced here, I found that those pages were producing 404 errors, meaning the web surfer could see the pages, but the spiders couldn’t. Naturally, I ditched the plugin. The site also includes a batch URL processing capability (up to 25 URLs at once).
3. Web Page Analyzer - This online tool checks the speed of your site and lets you know what the download time would be at various connection speeds. Granted, most folks have broadband these days but you still don’t want a page to take several minutes to load on a 56k dial up connection. The test gives you suggestions on ways to speed up your site for visitors and spiders. Both will go away if your site is too slow.
4. Yahoo Site Explorer -Yes, Google gives you some information on sites that link to you, but not like Yahoo’s Site Explorer, which is easy to use and just requires a Yahoo login. You can filter inbound links to see internal or external linking, number of pages Yahoo sees and more.
5. Spider Simulator - Just one of many free online tools offered by this site, I jump here when I need a quick look at what the spiders are seeing. A more comprehensive spider simulator report is available in the iBusinessPromoter client software on my PC, but this online utility serves my purpose most of the time.
6. Tweetscan - These days keeping up with what is said about you and your clients is a must. I use Tweetscan to search for references to me or my clients in Twitter for reputation management, goodwill and networking opportunities.
7. SearchStatus - This is a Firefox plugin that, among other things, allows you to highlight and see nofollow links. This comes in real handy when checking backlinks or sculpting the links on your own site. The plugin includes utilities to check backlinks, Alexa rankings and so forth, but I primarily use the nofollow highlight feature.
8. MyBlogLog - Although the community aspects of the social site are free, I do use one paid service that this Yahoo owned site offers - statistics. For about $25 per year, I can get almost real time traffic stats coming off of web sites. I can see my site traffic nearly as it happens, where surfers are coming from and where they are going. From this, I can see if there is a trend or if something is wrong on a site now, not tomorrow when my Google Analytics stats are refreshed. I mentioned this service in my post on Web Analytics. This is the only non-free tool I mention in this list, but it’s such a bargain, I had to include it.
9. Google Chrome - Though not technically a tool, Google’s first attempt at a web browser has one feature that keeps it open on one of my monitors all day - the ability to log into different Google accounts in different tabs. I keep my domain e-mail, which is hosted through Google Apps, in one tab and Google Webmaster Tools and Google Analytics for my work accounts in other tabs. Now, if Chrome would just pick up some cool plugins!
10. Google Webmaster Tools - For something that I paid little attention to when first released, Google Webmaster Tools is now also open on one of my monitors all day. It just keeps getting better. From tracking down dead URLs on my sites to testing a robots.txt file, I can locate site issues that I wouldn’t otherwise know about. Though far from perfect, it’s just about the most valuable online tool I use these days.

Using X-Robots HTTP Header To Sneakily Delete Your Link Farms

2008-11-20T21:00:00.000-08:00

Zoe asked a question this week about the best way to remove old link exchange pages. I responded on Twitter but wanted to post about it as well.
It’s well known that having loads of links pages on your website isn’t a good idea. One links page with 20 links is fine, 10 links pages all with 100 links will probably do more harm than good.
If you have been given a penalty in Google then deleting all these links pages is the first thing you need to do. But what if your site is ranking well but you want to get rid of the pages before Google gives you a penalty? Surely if you remove the pages everybody will stop linking to you and your rankings will fall?
The key is to find a way of silently removing your links pages, without alerting all your link partners to the fact that their link has gone.
My solution is to make use of the X-Robots HTTP header that Googlebot is now supporting. This allows you to add the noindex meta information to a page server side so that visitors can’t see any traces in your robots.txt or your source code.
header(’X-Robots-Tag: noindex, nofollow’, true);
This will remove your links pages from Google and probably won’t alert your link partners.
You might also like to cloak the X-Robots tag so that it only appears to Googlebot - you don’t want people to be able to run a header checker and see your sneaky plan.

Duplicate content

2008-11-12T22:04:00.000-08:00

Duplicate content generally refers to substantive blocks of content within or across domains that either completely match other content or are appreciably similar. Mostly, this is not deceptive in origin. Examples of non-malicious duplicate content could include:

Discussion forums that can generate both regular and stripped-down pages targeted at mobile devices
Store items shown or linked via multiple distinct URLs
Printer-only versions of web pages

However, in some cases, content is deliberately duplicated across domains in an attempt to manipulate search engine rankings or win more traffic. Deceptive practices like this can result in a poor user experience, when a visitor sees substantially the same content repeated within a set of search results.
Google tries hard to index and show pages with distinct information. This filtering means, for instance, that if your site has a "regular" and "printer" version of each article, and neither of these is blocked in robots.txt or with a noindex meta tag, we'll choose one of them to list. In the rare cases in which Google perceives that duplicate content may be shown with intent to manipulate our rankings and deceive our users, we'll also make appropriate adjustments in the indexing and ranking of the sites involved. As a result, the ranking of the site may suffer, or the site might be removed entirely from the Google index, in which case it will no longer appear in search results.
There are some steps you can take to proactively address duplicate content issues, and ensure that visitors see the content you want them to.

Consider blocking pages from indexing: Rather than letting Google's algorithms determine the "best" version of a document, you may wish to help guide us to your preferred version. For instance, if you don't want us to index the printer versions of your site's articles, disallow those directories or make use of regular expressions in your robots.txt file.
Use 301s: If you've restructured your site, use 301 redirects ("RedirectPermanent") in your .htaccess file to smartly redirect users, Googlebot, and other spiders. (In Apache, you can do this with an .htaccess file; in IIS, you can do this through the administrative console.)
Be consistent: Try to keep your internal linking consistent. For example, don't link to http://www.example.com/page/ and http://www.example.com/page and http://www.example.com/page/index.htm.
Use top-level domains: To help us serve the most appropriate version of a document, use top-level domains whenever possible to handle country-specific content. We're more likely to know that www.example.de contains Germany-focused content, for instance, than www.example.com/de or de.example.com.
Syndicate carefully: If you syndicate your content on other sites, Google will always show the version we think is most appropriate for users in each given search, which may or may not be the version you'd prefer. However, it is helpful to ensure that each site on which your content is syndicated includes a link back to your original article. You can also ask those who use your syndicated material to block the version on their sites with robots.txt.
Use Webmaster Tools to tell us how you prefer your site to be indexed: You can tell Google your preferred domain (for example, www.example.com or http://example.com).
Minimize boilerplate repetition: For instance, instead of including lengthy copyright text on the bottom of every page, include a very brief summary and then link to a page with more details.
Avoid publishing stubs: Users don't like seeing "empty" pages, so avoid placeholders where possible. For example, don't publish pages for which you don't yet have real content. If you do create placeholder pages, use robots.txt to block these from being crawled.
Understand your content management system: Make sure you're familiar with how content is displayed on your web site. Blogs, forums, and related systems often show the same content in multiple formats. For example, a blog entry may appear on the home page of a blog, in an archive page, and in a page of other entries with the same label.
Minimize similar content: If you have many pages that are similar, consider expanding each page or consolidating the pages into one. For instance, if you have a travel site with separate pages for two cities, but the same information on both pages, you could either merge the pages into one page about both cities or you could expand each page to contain unique content about each city.

Duplicate content on a site is not grounds for action on that site unless it appears that the intent of the duplicate content is to be deceptive and manipulate search engine results. If your site suffers from duplicate content issues, and you don't follow the advice listed above, we do a good job of choosing a version of the content to show in our search results.
However, if our review indicated that you engaged in deceptive practices and your site has been removed from our search results, review your site carefully. If your site has been removed from our search results, review our webmaster guidelines for more information. Once you've made your changes and are confident that your site no longer violates our guidelines, submit your site for reconsideration.
If you find that another site is duplicating your content by scraping (misappropriating and republishing) it, it's unlikely that this will negatively impact your site's ranking in Google search results pages. If you do spot a case that's particularly frustrating, you are welcome to file a DMCA request to claim ownership of the content and request removal of the other site from Google's index.

Google Offers SEO Starter Guide

2008-11-12T20:18:00.000-08:00

Google is getting into the SEO consulting business. Well, not quite. But, Google is now formally offering an “SEO Starter Guide” with practical advice for webmasters about improving search engine visibility and increasing traffic to a web site.

It comes in the form of a 22-page PDF announced today on the Webmaster Central blog and at the PubCon show in Las Vegas. According to the Google reps at PubCon, this is the same guide Google uses internally for its own sites (YouTube, etc.).

The guide is well written and geared toward webmasters and business owners who need a basic training in SEO. Topics covered include:

* Page Titles
* Description Meta Tag
* URL Structure
* Site Navigation
* Creating Quality Content
* Anchor Text
* Heading Tags (H1s, H2s, etc.)
* Image Optimization
* Robots.txt
* Rel=”nofollow”
* Website Promotion
* Webmaster Tools
* Analytics
* More Resources

As a longtime SEO, I’m most struck by the fact that an entire section of the guide is devoted to how to use the rel=”nofollow” attribute on individual links. Many in the SEO industry have thought this attribute is a red flag, something that tells Google that a professional SEO has been tweaking the page, and not something that an average webmaster would even know about. That’s clearly not the case anymore; rel=”nofollow” is more mainstream now thanks to a full page of explanation in this SEO guide. Have a look:

Google Explains Rel=Nofollow in SEO Guide

There are one or two other minor issues that some SEO consultants might quibble about (like the recommendation to use an XML Sitemap, something that many SEOs feel is unnecessary for most websites). But on the whole, it’s an excellent document covering many of the things you’d want in an SEO 101-type guide.

What do the robots.txt file analysis results mean?

2008-11-11T21:39:00.000-08:00

When you test a URL against a robots.txt file, you will see one of the following results:

* Allowed— Googlebot will crawl the URL.
* Blocked— Googlebot will not crawl the URL.
* Not in domain— This URL is not on the same domain as the robots.txt file and therefore, you cannot block it.
* Syntax not understood— Googlebot does not recognize this as a valid URL.

Additionally you may see the following message:

* Detected as a directory; specific files may have different restrictions— Although this directory is blocked or allowed, there may be other, more specific rules in the file that block or allow URLs in the directory, so you will want to check those as well.

If Googlebot has difficulty understanding parts of your robots.txt file, you will see one of the following parsing results, which you will want to fix:

* Accepted, but should be Disallow— You misspelled "Disallow."
* Accepted, but should be User-agent— You misspelled "user-agent."
* Accepted, but correct syntax includes a colon (Rule: path)— You forgot to put a colon between "Allow" or "Disallow" and the path.
* Rule ignored by Googlebot— This is not a rule that Googlebot follows (for example, "Crawl-delay").
* No user-agent specified— You have rules that aren't associated with a user-agent.
* Syntax not understood— Googlebot does not understand this line.
* robots.txt file does not appear to be valid— Googlebot doesn't understand any parts of this file and therefore, doesn't recognize it as a valid a robots.txt file.

Top Blog List

2008-11-11T03:41:00.001-08:00

S.No. Site Category
1 John Battelle Searchblog SEO / SEM
2 Problogger Blogging
3 SEOmoz SEO / SEM
4 Matt Cutts SEO / SEM
4 Copyblogger Internet Marketing
6 Search Engine Land SEO / SEM
7 Search Engine Watch SEO / SEM
8 Performancing Blogging
9 Shoemoney Make Money Online
10 John Chow Make Money Online
11 Entrepreneurs Journey Internet Marketing
12 Dosh Dosh Social Media
13 SEO Book SEO / SEM
14 Search Engine Journal SEO / SEM
15 Net Profits Today Affiliate Marketing
16 Daily Blog Tips Blogging
16 Top Rank Blog SEO / SEM
18 Search Engine Guide SEO / SEM
19 Pronet Advertising Social Media
20 Search Engine Roundtable SEO / SEM
21 WebProNews SEO / SEM
22 Marketing Pilgrim Internet Marketing
23 ClickZ SEO / SEM
24 Chris Garrett Blogging
25 Chris Brogan Social Media
26 Quick Sprout Social Media
27 Stuntdubl SEO / SEM
28 SEO Scoop SEO / SEM
29 Stephan Spencer SEO / SEM
30 Spark Plugging Make Money Online
31 HubSpot Internet Marketing Internet Marketing
31 Blog Storm SEO / SEM
33 SEO by the Sea SEO / SEM
34 Courtney Tuttle Internet Marketing
35 Web Ink Now Social Media
35 Bruce Clay SEO / SEM
35 Blogging Tips Blogging
35 ReveNews Make Money Online
39 Sugarrae SEO / SEM
39 Andy Beard Internet Marketing
41 The Link Spiel Link Building
41 Carl Ocab Make Money Online
41 Jim Kukral Internet Marketing
44 Small Business SEM SEO / SEM
44 Blueverse Make Money Online
46 Social Media Explorer Social Media
47 Vanessa Fox Nude SEO / SEM
47 1.00E+21 Social Media
47 Redfly Marketing SEO / SEM
50 Affiliate Tip Affiliate Marketing
50 PPC Hero SEO / SEM
50 Strategic Profits Make Money Online
50 Search Marketing Gurus SEO / SEM
50 SEO Fast Start SEO / SEM
50 Stone Temple SEO / SEM
56 Freelance Folder Copywriting
57 Techipedia Social Media
57 Dean Hunt Internet Marketing
57 Ian Fernando Make Money Online
60 Blogger Unleashed Blogging
60 SEO Black Hat SEO / SEM
60 Conversation Marketing Internet Marketing
63 Winning the Web Internet Marketing
63 SEO Design Solutions SEO / SEM
63 Andy Wibbels Internet Marketing
66 Marketing Tips Blog Internet Marketing
66 Wiep.net Link Building
66 45n5 Make Money Online
66 eXtra For Every Publisher Make Money Online
66 Graywolf SEO Blog SEO / SEM
71 Search Engine People SEO / SEM
71 Brent Csutoras Social Media
71 Michelle MacPhearson Social Media
71 Dave Naylor SEO / SEM
75 Garry Conn Make Money Online
75 Caroline Middlebrook Make Money Online
75 Local SEO Guide SEO / SEM
78 Solo SEO SEO / SEM
78 Blogsessive Blogging
80 Affiliate Watcher Affiliate Marketing
80 Balkhis Make Money Online
80 Jim Boykin Link Building
83 SEO Smarty SEO / SEM
84 aimClear SEO / SEM
84 Super Affiliate Mindset Affiliate Marketing
84 Dan Zarrella Social Media
87 Traffikd Social Media
88 Yoast SEO / SEM
88 Stephan Miller Make Money Online
88 CDF Networks Affiliate Marketing
88 5 Star Affiliate Programs Affiliate Marketing
92 Cre8pc SEO / SEM
92 John Andrews SEO / SEM
94 SEO ROI SEO / SEM
94 ViperChill Social Media
94 Grownup Geek Make Money Online
97 Self Made Minds Make Money Online
97 Slightly Shady SEO SEO / SEM
97 Cash Tactics Make Money Online
97 Eric Ward Link Building
101 Shimon Sandler SEO / SEM
101 Why Do Work Make Money Online
101 Cornwall SEO Social Media
101 SearchRank SEO / SEM
101 Who Is Andrew Wee Make Money Online
106 John Cow Make Money Online
106 Hamlet Batista SEO / SEM
108 Vinny Lingham Internet Marketing
108 Josh Spaulding Internet Marketing
110 Huomah SEO / SEM
110 MEMWG Make Money Online
110 SEO Theory SEO / SEM
113 Bill Hartzer SEO / SEM
113 SEOish SEO / SEM
115 SEO Chicks SEO / SEM
115 Thou Shall Blog Blogging
115 Google Lady Affiliate Marketing
118 SeoPedia SEO / SEM
118 Eric Lander SEO / SEM
120 Here.org.uk Affiliate Marketing
120 Sebastian's Pamphlets SEO / SEM
122 The Net Fool Make Money Online
122 Blogging Bits Blogging
122 Nate Whitehill Make Money Online
125 Collective Thoughts Social Media
125 Mason World Internet Marketing
125 Can I Make Big Money Online Make Money Online
125 Affiliate Confession Affiliate Marketing
125 Blue Hat SEO SEO / SEM
125 Aojon Affiliate Marketing
131 Muhammad Saleem Social Media
132 Pure Blogging Blogging
132 Terry Dean Internet Marketing
132 3 Dog Media Social Media
132 Small Fuel Internet Marketing
132 NowSourcing Social Media
132 Ades Blog Make Money Online
138 The University Kid Make Money Online
139 Jon Waraas Make Money Online
139 The Mad Hat SEO / SEM
139 Just Make Money Online Make Money Online
142 Scott Monty Social Media
142 Retire at 21 Make Money Online
142 Blogging Experiment Blogging
145 Social Desire Social Media
146 Blogtrepreneur Blogging
147 Collin Lahay Link Building
147 Jangro Internet Marketing
149 SEOptimise SEO / SEM
149 SEM Clubhouse SEO / SEM
151 eMonetized Make Money Online
151 Zac Johnson Affiliate Marketing
151 The Niche Store Builder Make Money Online
154 Palatnik Factor Internet Marketing
154 Dazzlin Donna Make Money Online
156 Cath Lawson Make Money Online
156 Mr Javo Internet Marketing
158 All Things SEM SEO / SEM
159 CPA Affiliates Affiliate Marketing
159 ShandyKing SEO / SEM
161 Problogineer Blogging
161 Stand Out Blogger Blogging
163 Jim Karter Make Money Online
164 Tyler Cruz Make Money Online
165 Nickycakes Affiliate Marketing
165 Net Business Blog Make Money Online
167 Scribbles and Words Blogging
167 Jonathan Volk Affiliate Marketing
167 Rajaie Talks Make Money Online
170 Slyvisions Make Money Online
170 Gather Success Make Money Online
170 Dat Money Make Money Online
170 Sabahan Make Money Online
174 IM with Joe Internet Marketing
174 Darin.cc SEO / SEM
176 Blogging Secret Blogging
176 Tim Nash SEO / SEM
178 Ask Shane Internet Marketing
179 Lost Art of Blogging Blogging
180 Preblogging Blogging
180 SEO Refugee SEO / SEM
182 Utah SEO Pro SEO / SEM
183 Uber Affiliate Affiliate Marketing
183 David Dalka Internet Marketing
185 Search For Blogging Blogging
186 Infected by Bugs Make Money Online
187 Newest on the Net Internet Marketing
187 Big Marketing Blog Internet Marketing
187 The Income Academy Make Money Online
190 Onreact SEO SEO / SEM
191 That Pam Chick SEO / SEM
191 Etienne Teo Make Money Online
191 Blog About Your Blog Blogging
191 Money Bites Make Money Online
195 Yimto Affiliate Marketing
196 Turnip of Power Make Money Online
196 Karl Ribas SEO / SEM
198 Bukiki HomeBiz Make Money Online
198 Marcus Hochstadt Make Money Online
200 Pat B. Doyle Make Money Online
201 We Build Pages Internet Marketing
201 Feed Flare Blogging
203 The Blog Entrepreneur Blogging
203 Enkay Blog Make Money Online
203 It's Write Now Copywriting
206 Search and Social SEO / SEM
207 Chris Hooley SEO / SEM
208 Blogging Fingers Blogging
209 SEO Scientist SEO / SEM
210 Richard Lee Make Money Online
210 Gonzo SEO SEO / SEM
210 One Man's Goal Make Money Online
213 Cash Quests Make Money Online
214 PQ Internet Internet Marketing
214 Christian Affiliate Marketers Affiliate Marketing
216 The Writers Manifesto Copywriting
217 Copy Brighter Social Media
218 Ask Kalena SEO / SEM
219 Affiliate Toolbox Affiliate Marketing
219 Internet Babel Make Money Online
221 Big Ben Patton Internet Marketing
222 Ryan Shamus Internet Marketing
223 Pimp My PageRank SEO / SEM
224 Fraser's Affiliate Marketing Blog Affiliate Marketing
225 Nick Wilsdon SEO / SEM
226 Cyber Cashology Make Money Online
227 The Home Business Archive Internet Marketing
228 Wingnut SEO SEO / SEM
229 Out of My Gord SEO / SEM
230 Sitemost SEO / SEM
231 Bruce Hopkins Make Money Online
232 Think Like an SOB Affiliate Marketing
233 Mubin Ahmed Make Money Online
234 James D. Brausch Internet Marketing

Canonical URL Issues

2008-11-10T03:56:00.001-08:00

There's a potential canonical URL issue that we've not touched on often, if ever. It's the kind of thing that might cause indexing issues or split PageRank into different "piles" - and even, potentially, generate duplicate URL problems.

This canonical problem comes from adding a period to the end of a domain name - http://www.example.com. - and that can trigger a cascade of potential problems. If the trailing period is at the end of the domain name and the site's navigation uses relative urls, then the extra period gets carried forward, and forward, and forward, through succeeding links.

There's a new thread in our Apache Forum that touches on the issue, and it also shares a fix - http://www.webmasterworld.com/apache/3718084.htm As moderator jdMorgan observes, even google.com. has this problem!

This kind of link can be generated innocently enough by forum software that automatically creates links for text strings that look like urls but are at the end of a sentence. And many servers will not have a problem resolving that url with an extra period.

So, for the sake of a complete reference, I'd like to collect the potential canonical url issues all in one place.

Canonical URL Issues

Different domain names serving the same content (302 redirects can make this kind of mess)
Different hostnames within one domain, such as "with-www" and "no-www" versions
With and without "index.html" for the domain root or a subdirectory root
Different protocols - https and http
Trailing period on the domain name
Double forward slash in the file path - http://example.com//page.html
Swapping the order of query string parameters
URL rewrite that allows typos for the "keyworded" virtual directory name
Any forum software or CMS that generates alternate URLs for the same content
URLs that include session parameters, click path tracking, etc.
Adding a port number to the domain name: example.com:443
URLs with unneeded query strings or extra parameters in the query string

The Problem With URLs

2008-11-02T21:34:00.000-08:00

URLs are simple things. Or so you'd think. Let's say you wanted to detect an URL in a block of text and convert it into a bona fide hyperlink. No problem, right?

Visit my website at http://www.example.com, it's awesome!

To locate the URL in the above text, a simple regular expression should suffice -- we'll look for a string at a word boundary beginning with http:// , followed by one or more non-space characters:

\bhttp://[^\s]+

Piece of cake. This seems to work. There's plenty of forum and discussion software out there which auto-links using exactly this approach. Although it mostly works, it's far from perfect. What if the text block looked like this?

My website (http://www.example.com) is awesome.

This URL will be incorrectly encoded with the final paren. This, by the way, is an extremely common way average everyday users include URLs in their text.

What's truly aggravating is that parens in URLs are perfectly legal. They're part of the spec and everything:

only alphanumerics, the special characters "$-_.+!*'(),", and reserved characters used for their reserved purposes may be used unencoded within a URL.

Certain sites, most notably Wikipedia and MSDN, love to generate URLs with parens. The sites are lousy with the damn things:

http://en.wikipedia.org/wiki/PC_Tools_(Central_Point_Software)
http://msdn.microsoft.com/en-us/library/aa752574(VS.85).aspx

URLs with actual parens in them means we can't take the easy way out and ignore the final paren. You could force users to escape the parens, but that's sort of draconian, and it's a little unreasonable to expect your users to know how to escape characters in the URL.

http://en.wikipedia.org/wiki/PC_Tools_%28Central_Point_Software%29
http://msdn.microsoft.com/en-us/library/aa752574%28VS.85%29.aspx

To detect URLs correctly in all most cases, you have to come up with something more sophisticated. Granted, this isn't the toughest problem in computer science, but it's one that many coders get wrong. Even coders with years of experience, like, say, Paul Graham.

If we're more clever in constructing the regular expression, we can do a better job.

\(?\bhttp://[-A-Za-z0-9+&@#/%?=~_()|!:,.;]*[-A-Za-z0-9+&@#/%=~_()|]

1. The primary improvement here is that we're only accepting a whitelist of known good URL characters. Allowing arbitrary random characters in URLs is setting yourself up for XSS exploits, and I can tell you that from personal experience. Don't do it!
2. We only allow certain characters to "end" the URL. Ending a URL in common punctuation marks like period, exclamation point, semicolon, etc means those characters will be considered end-of-hyperlink characters and not included in the URL.
3. Parens, if present, are allowed in the URL -- and we absorb the leading paren, if it is there, too.

I couldn't come up with a way for the regex alone to distinguish between URLs that legitimately end in parens (ala Wikipedia), and URLs that the user has enclosed in parens. Thus, there has to be a handful of postfix code to detect and discard the user-enclosed parens from the matched URLs:

if (s.StartsWith("(") && s.EndsWith(")"))
{
return s.Substring(1, s.Length - 2);
}

That's a whole lot of extra work, just because the URL spec allows parens. We can't fix Wikipedia or MSDN and we certainly can't change the URL spec. But we can ensure that our websites avoid becoming part of the problem. Avoid using parens (or any unusual characters, for that matter) in URLs you create. They're annoying to use, and rarely handled correctly by auto-linking code.

October 2008 Web Server Survey

2008-11-02T20:30:00.000-08:00

In the October 2008 survey we received responses from 182,226,259 sites, which reflects growth of 948 thousand since last month.

Apache once again shows the largest growth, gaining 463 thousand sites this month. ThePlanet.com gains 1.3 million sites this month — nearly all of which are running on Apache — but this includes a large number of 'link farm' sites that use .pl domains to propagate search terms using pornographic phrases.

Google shows the next largest growth and boosts its total by 411 thousand sites. Google now runs 10.5 million sites on its own webserver software, which is used to host its own services in addition to user-generated applications and blogs. Some server names include:

* GFE/1.3, which is used by Google's Blogger service to publish third party blogs under the blogspot.com domain, and spreadsheets and other documents under docs.google.com.
* GWS-GRFE/0.50, which runs Google Groups.
* gws. This simple, lowercase name is used by Google's main search site at google.com and Google Image Search.
* Google Frontend, which is used to run third party applications on Google App Engine (often using the appspot.com domain) and Google Mashups.

Total Sites Across All Domains August 1995 - October 2008

Total Sites Across All Domains, August 1995 - October 2008

Market Share for Top Servers Across All Domains August 1995 - October 2008

Graph of market share for top servers across all domains, August 1995 - October 2008

Top Developers
Developer September 2008 Percent October 2008 Percent Change
Apache 91,425,295 50.43% 91,888,508 50.43% -0.01
Microsoft 62,374,823 34.41% 62,766,928 34.44% 0.04
Google 10,076,405 5.56% 10,487,607 5.76% 0.20
lighttpd 3,095,928 1.71% 3,072,457 1.69% -0.02
Active Sites
Developer September 2008 Percent October 2008 Percent Change
Apache 33,719,369 46.75% 33,310,242 46.26% -0.49
Microsoft 25,155,273 34.88% 25,594,704 35.55% 0.67
Google 7,714,617 10.70% 7,645,615 10.62% -0.08
lighttpd 144,499 0.20% 134,161 0.19% -0.01
Totals for Active Servers Across All Domains
June 2000 - October 2008

Optimizing Web Site Navigation

2008-10-31T04:01:00.000-07:00

While outlining universal nodes and topical anchors, discussing link placement and supporting search engine crawls, I've dropped a few snippets of information on internal linking in a Web site's (outer) navigation. Now lets draw the whole picture by looking at the impact navigational links have on search engine placements.

Web site navigation obviously must be user friendly. User friendliness plus a few tweaks and shortcuts implemented for search engine spiders makes up a search engine friendly navigation. Laying out navigational links to lead users straight to the content they're searching for allows some fair search engine optimizing. What you never should do is tweaking the linkage for the engines when this results in a loss of usability and visitor support.

Technically, outer navigation elements are a part of the page's template (see page partitioning and link placement). Search engines can distinguish templated page areas from the body's (unique) content. As a matter of fact, they weight text and links differently depending on the page area. How much power navigational links have with regard to ranking depends on the site's architecture. On sites where the outer navigation is very repetitive, that is the menus get duplicated over and over with very few page specific items, those navigational links are treated like artificial links and their power gets downgraded next to zero. You will find this kind of flawed design at many eCommerce sites, where the static outer navigation (i.e. links to product lines and home page) is identical on most pages. The in-depth linkage is represented by the dynamicly generated inner navigation, that is links nearby or within the page body.

Since it makes no sense to deal with impotent page areas, clever developers balance the linking power by placing as many in-depth navigation as possible at the outer navigation areas. The goal is to drill down the outer navigation to the last node (e.g. product), while restricting the inner navigation to within-the-node linkage (e.g. product sizes and colors). Sometimes it's even possible to integrate a complex node's internal navigation with the outer navigation. This approach enables powerful linking in the peripheral areas, because every node comes with a different menu, that is less repetition (link duplication).

Another advantage of node specific outer navigation is, that it supports internal authority hubs. Having less than a handful of links leading to upper levels, most of the linking power gets used to strengthen on-topic (navigational) links. Additionally, a node specific outer navigation, e.g. a left handed menu, bread crumbs near the top and horizontal links at the bottom, develops enough linking power (mostly received from deep inbound links) to support the root and the main sections as well. Thus having a search engine unfriendly DHTML menu or flash based navigation at the top or right side of the page doesn't harm anymore, it may even help to establish topic authority hubs.

To demonstrate the impact keywords placed in a page's templated area can have on rankings, here is an example of a node specific outer navigation where a search engine assigns a lot of weight to navigational anchor text. The search term is "Internet Google", which is by the way a totally useless #4 spot, because nobody searches for it. I've picked it because it pulls 68.5 million results at Google, "Internet" is not closely related to the on page content (the word "Internet" appears only once in a navigational link and the URL), and at least it looks like a popular search. Here is the SERP:

Google's first SERP for 'Internet Google'

Look at the snippet and the screenshot of the page at the time of indexing. The sequence of keywords in the bread crumbs' anchor text makes (most of) the placement on the SERP.

This page is about 'Google Sitemap Validation' and has not so much to do with 'Internet'

It works fine with a few other useless keyword phrases taken from the page's bread crumbs too: Utterly Useless Keywords: smart internet business google Utterly Useless Keywords: it consulting internet google Utterly Useless Keywords: smart internet google Utterly Useless Keywords: consulting internet google ... but don't expect it's that easy to achieve top rankings in competitive markets. However, keywords within the navigation can help to define themes and topics, so you should use your most important keywords in prominent navigational anchor text.

Bread crumbs are a great way to break down a site's theme to topics and sub-topics. This You are here path to the root index page, placed near the top of each page, can act as an authority hub's sole connection to it's upper hierarchy levels. It's not even necessary to repeat the complete path to the root in the left handed menu.

By the way, consistent linking of the current node is neither lazy nor useless, because in complex nodes the current page is often different from the node's point of entry.

Other important navigation element are top level links, stored popular searches and horizontal views. The number of top level links, leading to the home page and main sections, must be kept as low as possible to avoid dilution of topical authority build around the rich nodes in deeper levels. There is nothing to say against nicely formatted top level links which aren't spiderable, e.g. in java based drop down menus, if they improve the surfing experience. From a SEO point of view they are (in most cases) pretty much useless.

Horizontal views are for example indexes of all image galleries, all tutorials related to a product line, or all articles related to a broader theme. These indexes may or may not reflect a part of the site's hierarchy, but mostly they are used as more content type oriented than theme specific layers. Like site maps, horizontal views should not contain more than 100 links per page, 15-25 links plus descriptions and/or previews are a proven limit. The content linked on a site map page or on a horizontal index should be describable with a short catchword (phrase) in a manu item. If that doesn't work, probably the collection of links is useless at all.

Methods to Support Search Engines in Crawling and Ranking

2008-10-31T03:55:00.000-07:00

Let's recap the basic methods of steering and supporting search engine crawling and ranking:

# Provide unique content. A lot of unique content. Add fresh content frequently.
# Acquire valuable inbound links from related pages on foreign servers, regardless of their search engine ranking. Actively acquire deep inbound links to content pages, but accept home page links. Do not run massive link campaigns if your site is rather new. Let the amount of relevant inbound links grow smoothly and steadily to avoid red-flagging.
# Put in carefully selected outbound links to on-topic authority pages on each content page. Ask for reciprocal links, but do not dump your links if the other site does not link back.
# Implement a surfer friendly, themed navigation. Go for text links to support deep crawling. Provide each page at least one internal link from a static page, for example from a site map page.
# Encourage other sites to make use of your RSS feeds and alike. To protect the uniqueness of your site's content, do not put text snippets from your site into feeds or submitted articles. Write short summaries instead and use a different wording.
# Use search engine friendly, short but keyword rich URLs. Hide user tracking from search engine crawlers.
# Log each crawler visit and keep these data forever. Develop smart reports querying your logs and study them frequently. Use these logs to improve your internal linking.
# Make use of the robots exclusion protocol to keep spiders away from internal areas. Do not try to hide your CSS files from robots.
# Make use of the robots META tag to ensure that only one version of each page on your server gets indexed. When it comes to pages carrying partial content of other pages, make your decision based on common sense, not on any SEO bible.
# Use rel="nofollow" in your links, when you cannot vote for the linked page (user submitted content in guestbooks, blogs ...). Do not hoard PageRank™.
# Make use of Google SiteMaps as a 'robots inclusion protocol'.
# Do not cheat the search engines.

About /robots.txt

2008-10-31T03:54:00.000-07:00

In a nutshell

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.

It works likes this: a robot wants to vists a Web site URL, say http://www.example.com/welcome.html. Before it does so, it firsts checks for http://www.example.com/robots.txt, and finds:

User-agent: *
Disallow: /

The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.

There are two important considerations when using /robots.txt:

* robots can ignore your /robots.txt. Especially malware robots that scan the web for security vulnerabilities, and email address harvesters used by spammers will pay no attention.
* the /robots.txt file is a publicly available file. Anyone can see what sections of your server you don't want robots to use.

So don't try to use /robots.txt to hide information.

See also:

* Can I block just bad robots?
* Why did this robot ignore my /robots.txt?
* What are the security implications of /robots.txt?

The details

The /robots.txt is a de-facto standard, and is not owned by any standards body. There are two historical descriptions:

* the original 1994 A Standard for Robot Exclusion document.
* a 1997 Internet Draft specification A Method for Web Robots Control

In addition there are external resources:

* HTML 4.01 specification, Appendix B.4.1
* Wikipedia - Robots Exclusion Standard

The /robots.txt standard is not actively developed. See What about further development of /robots.txt? for more discussion.

The rest of this page gives an overview of how to use /robots.txt on your server, with some simple recipes. To learn more see also the FAQ.
How to create a /robots.txt file
Where to put it

The short answer: in the top-level directory of your web server.

The longer answer:

When a robot looks for the "/robots.txt" file for URL, it strips the path component from the URL (everything from the first single slash), and puts "/robots.txt" in its place.

For example, for "http://www.example.com/shop/index.html, it will remove the "/shop/index.html", and replace it with "/robots.txt", and will end up with "http://www.example.com/robots.txt".

So, as a web site owner you need to put it in the right place on your web server for that resulting URL to work. Usually that is the same place where you put your web site's main "index.html" welcome page. Where exactly that is, and how to put the file there, depends on your web server software.

Remember to use all lower case for the filename: "robots.txt", not "Robots.TXT.

See also:

* What program should I use to create /robots.txt?
* How do I use /robots.txt on a virtual host?
* How do I use /robots.txt on a shared host?

What to put in it
The "/robots.txt" file is a text file, with one or more records. Usually contains a single record looking like this:

User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /~joe/

In this example, three directories are excluded.

Note that you need a separate "Disallow" line for every URL prefix you want to exclude -- you cannot say "Disallow: /cgi-bin/ /tmp/" on a single line. Also, you may not have blank lines in a record, as they are used to delimit multiple records.

Note also that globbing and regular expression are not supported in either the User-agent or Disallow lines. The '*' in the User-agent field is a special value meaning "any robot". Specifically, you cannot have lines like "User-agent: *bot*", "Disallow: /tmp/*" or "Disallow: *.gif".

What you want to exclude depends on your server. Everything not explicitly disallowed is considered fair game to retrieve. Here follow some examples:
To exclude all robots from the entire server

User-agent: *
Disallow: /

To allow all robots complete access

User-agent: *
Disallow:

(or just create an empty "/robots.txt" file, or don't use one at all)
To exclude all robots from part of the server

User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /junk/

To exclude a single robot

User-agent: BadBot
Disallow: /

To allow a single robot

User-agent: Google
Disallow:

User-agent: *
Disallow: /

To exclude all files except one
This is currently a bit awkward, as there is no "Allow" field. The easy way is to put all files to be disallowed into a separate directory, say "stuff", and leave the one file in the level above this directory:

User-agent: *
Disallow: /~joe/stuff/

Alternatively you can explicitly disallow all disallowed pages:

User-agent: *
Disallow: /~joe/junk.html
Disallow: /~joe/foo.html
Disallow: /~joe/bar.html

Print format

Robots Exclusion Protocol: now with even more flexibility

2008-10-31T03:34:00.000-07:00

This is the third and last in my series of blog posts about the Robots Exclusion Protocol (REP). In the first post, I introduced robots.txt and the robots META tags, giving an overview of when to use them. In the second post, I shared some examples of what you can do with the REP. Today, I'll introduce two new features that we have recently added to the protocol.

As a product manager, I'm always talking to content providers to learn about your needs for REP. We are constantly looking for ways to improve the control you have over how your content is indexed. These new features will give you flexible and convenient ways to improve the detailed control you have with Google.

Tell us if a page is going to expire
Sometimes you know in advance that a page is going to expire in the future. Maybe you have a temporary page that will be removed at the end of the month. Perhaps some pages are available free for a week, but after that you put them into an archive that users pay to access. In these cases, you want the page to show in Google search results until it expires, then have it removed: you don't want users getting frustrated when they find a page in the results but can't access it on your site.

We have introduced a new META tag that allows you to tell us when a page should be removed from the main Google web search results: the aptly named unavailable_after tag. This one follows a similar syntax to other REP META tags. For example, to specify that an HTML page should be removed from the search results after 3pm Eastern Standard Time on 25th August 2007, simply add the following tag to the first section of the page:

META NAME="GOOGLEBOT" CONTENT="unavailable_after: 25-Aug-2007 15:00:00 EST"

The date and time is specified in the RFC 850 format.

This information is treated as a removal request: it will take about a day after the removal date passes for the page to disappear from the search results. We currently only support unavailable_after for Google web search results.

After the removal, the page stops showing in Google search results but it is not removed from our system. If you need a page to be excised from our systems completely, including any internal copies we might have, you should use the existing URL removal tool which you can read about on our Webmaster Central blog.

Meta tags everywhere
The REP META tags give you useful control over how each webpage on your site is indexed. But it only works for HTML pages. How can you control access to other types of documents, such as Adobe PDF files, video and audio files and other types? Well, now the same flexibility for specifying per-URL tags is available for all other files type.

We've extended our support for META tags so they can now be associated with any file. Simply add any supported META tag to a new X-Robots-Tag directive in the HTTP Header used to serve the file. Here are some illustrative examples:

* Don't display a cache link or snippet for this item in the Google search results:

X-Robots-Tag: noarchive, nosnippet

* Don't include this document in the Google search results:

X-Robots-Tag: noindex

* Tell us that a document will be unavailable after 7th July 2007, 4:30pm GMT:

X-Robots-Tag: unavailable_after: 7 Jul 2007 16:30:00 GMT

You can combine multiple directives in the same document. For example:

* Do not show a cached link for this document, and remove it from the index after 23rd July 2007, 3pm PST:

X-Robots-Tag: noarchive
X-Robots-Tag: unavailable_after: 23 Jul 2007 15:00:00 PST

Our goal for these features is to provide more flexibility for indexing and inclusion in Google's search results. We hope you enjoy using them.

I Robot | Robots.txt Help | SebastianX of Sebastians Pamphlets

2008-10-31T03:09:00.000-07:00

Hobo - Right Sebastian! What do you think you are doing calling me out on a slight bit of “misinformation” on a post I made for a bit of branding. Just who do you think you are spamming my content with useful, original and interesting content?

Don’t you realize that @ 1,500 stumblers and Twitters visited my site as a result of this slapping?? You trying to discredit me? :)

Sebastian - Howdy Shaun, - I’m so sorry that I discredited you, that was really not my intention.

I couldn’t resist coz robots.txt is kinda pet peeve of mine. Thanks for the opportunity to spam your neat blog with my links thoughts, though. :)

Hobo: That post was about how expert SEO people were using Robots.txt - I should have put a disclaimer at the bottom saying I didn’t know a thing about Robots.xt files and that I had nicked mine some time ago from Michael Gray and forgot about it. And spam my blog all you like with that kind of content, although I’ve got Lucia’s Linky Love installed so generally Spam doesn’t get much of a foothold about these parts (actually I am not even sure if that is working properly).

OK - you seem to know what you’re on about when it comes to robots.txt. Fancy educating me and the Hobo team as to what you’ve learned and know about these often misunderstood files? You know, all that stuff that took you years to learn, Let me have it….now!

Hobo - WTF is a Robots.txt file, Sebastian, in simple idiot’s terms?

Well, the “idiot’s version” will lack interesting details, but it will get you started.

Robots.txt is a plain text file. You must not edit it with HTML editors, word processors, nor any applications other than a plain text editor like vi (Ok, notepad.exe is allowed too). You shouldn’t embed images and such, also any other HTML code is strictly forbidden.

Hobo - Why shouldn’t I edit it with my Dreamweaver FTP client, for instance?

Because all those fancy apps insert useless crap like formatting, HTML code and whatnot. Most probably search engines aren’t capable to interpret a robots.txt file like:

{\b\lang2057\langfe1031\langnp2057\insrsid6911344\charrsid11089941 User-agent: Googlebot}{ \lang2057\langfe1031\langnp2057\insrsid6911344\charrsid11089941 \line Disallow: / \line Allow: }{\cs15\i\lang2057\langfe1031\langnp2057\insrsid6911344\charrsid2903095 /}{\i\lang2057\langfe1031\langnp2057\insrsid6911344\charrsid2903095 content}{ \cs15\i\lang2057\langfe1031\langnp2057\insrsid6911344\charrsid2903095 /} …
(Ok Ok, I’ve made up this example, but it represents the raw contents of text files saved with HTML editors and word processors.)

Hobo - Where Do I put the damn thing?

Robots.txt resides in the root directory of your Web space, that’s either a domain or a subdomain, for example “/web/user/htdocs/example.com/robots.txt” resolving to http://example.com/robots.txt.

Can I use Robots.txt in sub directories?

Of course you’re free to create robots.txt files in all your subdirectories, but you shouldn’t expect search engines to request/obey those. If you for some weird reasons use subdomains like crap.example.com, then the example.com/robots.txt is not exactly a suitable instrument to steer crawling of subdomains, hence ensure each subdomain serves its own robots.txt.

When you upload your robots.txt then make sure to do it in ASCII mode, your FTP client usually offers “ASCII|Auto|Binary” - choose “ASCII” even when you’ve used an ANSI editor to create it.

Hobo - Why?

Because plain text files contain ASCII content only. Sometimes standards that say “upload *.htm *.php *.txt .htaccess *.xml files in ASCII mode to prevent them from inadvertently corruption during the transfer, storing with invalid EOL codes, etc.” do make sense. (You’ve asked for the idiot version, didn’t you?)

Hobo - What about if I am on a Free Host?

If you’re on a free host, robots.txt is not for you. Your hosting service will create a read-only robots.txt “file” that’s suitable to steal even more traffic than its ads that you can’t remove from your headers and footers.

Now, if you’re still interested in the topic, you must learn how search engines work to understand what you can archive with a robots.txt file and what’s just myths posted on your favorite forum.

Hobo - Sebastian, Do you know how search engines work, then?

Yep, to some degree. ;) Basically, a search engine has three major components:

1. A crawler that burns your bandwidth fetching your unchanged files over and over until you’re belly up.
2. An indexer that buries your stuff unless you’re Matt Cutts or blog on a server that gained search engine love making use of the cruelest black hat tactics you can think of.
3. A query engine that accepts search queries and pulls results from the search index but ignores your stuff coz you’re neither me nor Matt Cutts.

Hobo - What goes into the robots.txt file?

Your robots.txt file contains useful but pretty much ignored statements like
# Please don't crawl this site during our business hours!
(the crawler is not aware of your time zone and doesn’t grab your office hours from your site), as well as actual crawler directives. In other words, everything you write in your robots.txt is a directive for crawlers (dumb Web robots that can fetch your contents but nothing more), not indexers (high sophisticated algorithms that rank only brain farts from Matt and me).

Hobo - I say index, you say crawl. You say tomato, I say….ah! I see!

Currently, there are only three statements you can use in robots.txt:

1. Disallow: /path
2. Allow: /path
3. Sitemap: http://example.com/sitemap.xml

Some search engines support other directives like “crawl-delay”, but that’s utterly nonsense, hence safely igore those.

The content of a robots.txt file consists of sections dedicated to particular crawlers. If you’ve nothing to hide, then your robots.txt file looks like:
User-agent: *
Disallow:
Allow: /
Sitemap: http://example.com/sitemap.xml

If you’re comfortable with Google but MSN scares you, then write:
User-agent: *
Disallow:

User-agent: Googlebot
Disallow:

User-agent: msnbot
Disallow: /

Please note that you must terminate every crawler section with an empty line. You can gather the names of crawlers by visiting a search engine’s Webmaster section.

From the examples above you’ve learned that each search engine has its own section (at least if you want to hide anything from a particular SE), that each section starts with a
User-agent: [crawler name]
line, and that each section is terminated with a blank line. The user agent name “*” stands for the universal Web robot, that means that if your robots.txt lacks a section for a particular crawler, it will use the “*” directives, and that when you’ve a section for a particular crawler, it will ignore the “*” section. In other words, if you create a section for a crawler, you must duplicate all statements from the “all crawlers” (”User-agent: *”) section before you edit the code.

Now to the directives. The most important crawler directive is
Disallow: /path

“Disallow” means that a crawler must not fetch contents from URIs that match “/path”. “/path” is either a relative URI or an URI pattern (”*” matches any string and “$” marks the end of an URI). Not all search engines support wildcards, for example MSN lacks any wildcard support (they might grow up some day).

URIs are always relative to the Web space’s root, so if you copy and paste URLs then remove the http://example.com part but not the leading slash.

Allow: path/
refines Disallow: statements, for example
User-agent: Googlebot
Disallow: /
Allow: /content/
allows crawling only within http://example.com/content/

Sitemap: http://example.com/sitemap.xml
points search engines that support the sitemaps protocol to the submission files.

Please note that all robots.txt directives are crawler directives that don’t affect indexing. Search engines do index disallow’ed URLs pulling title and snippet from foreign sources, for example ODP (DMOZ - The Open Directory) listings or the Yahoo directory. Some search engines provide a method to remove disallow’ed contents from their SERPs on request.

Hobo - Say I want to keep a file / folder out of Google. Exactly what what would I need to do?

You’d check each HTTP request for Googlebot and serve it a 403 or 410 HTTP response code. Or put a “noindex,noarchive” Googlebot meta tag.
"meta name=”Googlebot” content=”noindex,noarchive” /"
Robots.txt blocks with Disallow: don’t prevent from indexing. Don’t block crawling of pages that you want to have deindexed, as long as you don’t want to use Google’s robots.txt based URL terminator every six months.

Hobo - Sebastian, thanks so much for your invaluable insights into this pesky but poweful file. Your blog was recently cited by Jim Boykin as a favourite destination of Jim’s. If I had to ask you to tell the readers 5 of your favourite posts on your own website, which ones would you pick?

Being a greedy link-whore of course I’d pick my Canonical SEO definitions. I hope you don’t mind that this link points to my sitemap (sheesh you’ve spammed the rest of my blog anyways - Hobo …. :)) that some folks have even sphunn. Ok, that’s zero, and here is the list of 5 posts that I consider somewhat useful, either because they’re interesting from a technical point of view, or because they tell something about me.

0. The anatomy of a server sided redirect: 301, 302 and 307 illuminated SEO wise
0. Shit happens, your redirects hit the fan!
0. Why proper error handling is important
1. Analyzing search engine rankings by human traffic
2. If you free-host your blog flee now!
3. Microsoft funding bankrupt Live Search experiment with porn spam
4. SEOs home alone - Google’s nightmare
5. My plea to Google - Please sanitize your REP revamps

I should have mentioned earlier that counting somewhat challenges me when it comes to limits of links lists. Of course I like a few more of my posts, but I can resist to quote my blog’s site map.

Hobo - Where online do you hang out?

At Sphinn and Google’s Webmaster Help Group. For the latter some folks call me a slimy Google groupie, but I can perfectly live with that. Google’s SEO forum is a nice place to help noobs and discuss interesting topics as well.

Hobo - Who do you read every day/week?

Oh well. That’s a very long list. Probably the OPML file would be too large to email it. I read (sometimes skim) my friend’s posts daily, when I’m swamped at least weekly. I guess the best way to get a grip of my reading preferences is my shared feed, my list of stumbles, bookmarks, and sphinns.

Hobo - Tell me who your favourite music band is? Mine is the Stone Roses, have you heard of them?

Today that’s Ten Years After, yesterday it was Bob Dylan. Stone Roses is not on my radar, maybe I missed out on a great band?

Hobo - What else are you interested in online?

Tough question. What can a lonely geek do online? Viewing porn of course. Seriously, I consume more technical stuff than smut.

Hobo - I’ll send you a couple of links complete with free passwords I confiscated off my Managing Director, Michael ;)

Can’t wait for this list. If it contains passwords from one of my adult sites I’ll sue Michael! ;)

If someone wants to know more about robots.txt, where do they go?

Honestly, I don’t know a better resource than my brain, partly dumped here. I even developed a few new robots.txt directives and posted a request for comments a few days ago. I hope that Google, the one and only search engine that seriously invests in REP evolvements, will not ignore this post caused by the sneakily embedded “Google bashing”. I plan to write a few more posts, not that technical and with real world examples.

Hobo - Can I ask you how you auto generate and mask robots.txt, or is that not for idiots? Is that even ethical?

Of course you can ask, and yes, it’s for everybody and 100% ethical. It’s a very simple task, in fact it’s plain cloaking. The trick is to make the robots.txt file a server sided script. Then check all requests for verified crawlers and serve the right contents to each search engine. A smart robots.txt even maintains crawler IP lists and stores raw data for reports. I recently wrote a manual on cloaked robots.txt files on request of a loyal reader.

Hobo - Think Disney will come after you for your avatar now you are famous after being interviewd on the Hobo blog?

Sebastian's avatarI’m sure they will try it, since your blog will become an authority on grumpy red crabs called Sebastian. I’m not too afraid though, because I use only a tiny thumbnailed version of an image created by a designer who –hopefully– didn’t scrape it from Disney, as icon/avatar. If they become nasty, I’ll just pay a license fee and change my avatar on all social media sites, but I doubt that’s necessary. To avoid such hassles I’ve bought an individually drawed red crab from an awesome cartoonist last year. That’s what you see on my blog, and I use it as avatar as well, at least with new profiles.

Hobo - What’s your day job? Who do you work for?

I’m a freelancer loosely affiliated with a company that sells IT consulting services in several industries. I do Web developer training, software design / engineering (mostly the architectural tasks), and grab development / (technical) SEO projects myself to educate yours truly. I’m a dad of three little monsters, working at home. If you want to hire me, drop me a line. ;)

Sebastian, a big thanks for slapping me about about Robots.txt and indeed for helping me craft the Idiot’s Guide To Robots.txt. I certainly learned a lot from talking to you for a day, and I hope some others can learn from this snippet article. You’re a gentleman spammer. :)

Internal Links - Only The First Link Counts in Google?

2008-10-31T03:08:00.000-07:00

I thought I would share the results of another simple test I did to see how Google treats internal links.

What does Google count, when it finds two links on the same page going to the same internal destination page.

I surmised:

1. Google might count one link, the first it finds as it indexes a page
2. Google might count them all (I think unlikely)
3. Google might count perhaps 55 characters of ALL of the available links (could be useful)

OK - From this test, and the results on this site anyways, testing links internal to this site, it seems Google only counted the first link when it came to ranking the target page.

In much the same method as my recent seo test where I tested how many words you should put in a link, I relied on the “These terms only appear in links pointing to this page” (when you click on the cache) that Google helpfully shows when the word isn’t on the page.

Again, I pointed 2 everyday words at a page that don’t appear on the page or in links to the page, and searched for the page in Google using a term I knew it would rank high for (Shaun Anderson) and added my modifier keywords. I left it for quite some time, and checked every now and again the results.

Google Cache

Searching for “shaun anderson” + “Keyword 1″ returned the page (cache shown above).

Cartoonist

Searching for the term “shaun anderson” + “keyword 2″ did not return the page at all, only the page with the actual link on it, further down the SERPS.

Fireman

Not even in a site search.

Site Search

It’s not exactly Google terrorism to identify this, so here is the actual test page where you can see the simple test in action.

So today :) on this site :) in internal links :), Google only counted the first link as far as anchor text transfer is concerned :)

How you can use to your advantage?

1. Perhaps, you could place your navigation below your text
2. This lets you vary the anchor text to important internal pages on your site, within the text content, instead of ramming down Google’s throat one anchor text link (usually high in the navigation)
3. Varying anchor text naturally optimises to an extent the page for long tail ‘human’ searches you might overlook when writing the actual target page text
4. Of course, I assume links within text surrounded by text are more important than links in navigation menus
5. It makes use of your internal links to to rank a page for more terms, especially useful if you link to your important pages often, and don’t have a lot of incoming natural links to achieve a similar benefit

Long Tail SEO

Credit - Graphic first sourced at Search Engine Land and created by Elliance, an eMarketing firm.

Works for me anyways, when I’m building new sites, especially useful on longtail searches, and there’s plenty of editorial content being added to the site for me to link to a few sales pages.

Note: I would think Google would analyse everything it finds, so it would find it easy to spot spammy techniques we’ve all seen on sites trying to force Google to take multiple link anchor text to one page.

25 Web Form Optimization Tips

2008-10-29T23:30:00.000-07:00

Stop for a moment and consider the goals of your website. Regardless of whether it’s a purchase through a shopping cart, a lead generation, white paper download, or a email opt, I’m going to bet every one of these actions requires a customer to use a web form.

With web forms playing such an important role in the completing goals, it goes without saying that we should optimize the heck out of them. Below are 25 tips for doing just that.

1. Ditch the Captchas: Captcha’s are great for blocking spam, but some evidence suggests they are just as good at blocking conversions. A little spam isn’t the end of the world, and definitely isn’t worth losing conversions over. If you must use a Captcha, make sure it’s easy to read.
2. Remove Unnecessary Fields: Do you really need to ask for your customers date of birth and gender? Even if your customers aren’t concerned about privacy issues, odds are they’re lazy and might just abandon your excessively inquisitive form. Here’s some great advice from Get Elastic on registration forms.
3. Keep It Simple: Just because we can use CSS to do all sorts of fancy things with text boxes, doesn’t mean we should. Keeping form fields simple will ensure that customers understand their purpose and won’t confuse them with design elements.
4. Clear the Clear Button: Having a clear button next to the submit button just makes it easier for customers to accidentally delete what they’ve entered. Skip this unnecessary feature.
5. Cancel the Cancel Button: In the case of long or multi-part form pages, such as checkouts, don’t give customers the option to cancel their decision. That’s equivalent to a commission driven salesperson asking, “are you sure you really want to buy this?”
6. Label Required Fields: People want to do as little as possible. For this reason, let your customers know what they are required to fill out with an asterisk or similar label.
7. Use Point of Action References: If customers are getting confused by the information you’re asking for in a particular field, include a small note with a popup link with more information. For example, one of the most common POA references is an explanation of the 3 digit CVV code found on the back of credit cards.
8. Show Formatting Examples: Some fields should have a notes showing how to format them, depending on your database requirements. For example, you might want phone number formatted in a certain way, with or without parenthesis, dashes, etc. In general though, keep these formatting requirements to a minimum in order to keep it simple for customers.
9. Make it International Friendly: Forms requiring an address can be confusing if they’re built only with US residents in mind. Check out these detailed guidelines for building international friendly forms.
10. Allow Easy Forward and Backward Movement: Customers rarely maneuver through our website the way we intend them to. In order words, they hit the back button the forward button, refresh, etc. Depending on how your forms pass data, this could cause error messages such as “this page has expired”. Make sure you test the forward and backward flow of any multiple page forms on your site.
11. Logical Tab Sequence: Don’t you hate it when you hit the tab button, and rather than going to the next field, the focus moves somewhere else on the page? This problem is likely due to the way the form is laid out with HTML tables. Make sure your forms tabs in a logical sequence to prevent customers from accidentally skipping fields.
12. Server Side Validation: Basically, there are 2 ways to ensure that your visitors are entering correct data into fields. You can use client-side scripting (such as Javascript which is browser dependent) or server side error processing. In addition to server side validation being less reliant on the user’s browser settings, it is also preferable from a security point of view.
13. Clear Error Messages: When displaying error messages when customers enter invalid data, make sure your messages are clear and well placed. This means saying “Please enter an email address” rather than something vague like “you must fill out all fields.” A best practice is taking them right back to the field with incorrect data, and displaying the error message next to it.
14. Show What’s Needed When Its Needed: It’s best to hide form fields until you know they are absolutely needed. For example, if you already know your user is from the US, you can dynamically hide the province field and show the state drop down box instead.
15. Logical List Order: When using drop down lists or radio button lists, make sure you order them in a logical way, listing items higher if they are selected more often. In other words, if 90% of your customers buy from the USA, don’t list Afghanistan as your first option, and United States at the very bottom.
16. AJAX Validation: Some sites have begun to validate form inputs as soon as the user tabs out of the field. This can be very effective, since it does not break the flow of the process. In other words, its easier to correct an error immediately after entering it rather than after the whole form is completed.
17. Remember Me Feature: For login forms, always allow customers to choose a “remember me” option, which uses a cookie to fill in login information the next time. Who wants to remember all those passwords anyway?
18. Set Focus: When a page loads containing a forms, sending the cursor to the first required field will prevent users from having to click into the field in order to start typing. This can be accomplished with a simple JavaScript function.
19. Avoid Obnoxious Password Requirements: Ever received this annoying error? “Your password must contain at least one letter, number, and be least X number of digits.” Requiring passwords to be formated in a certain way may help security, but it will likely discourage return visits since visitors must now remember a new password they are not used to.
20. Progress Indicators: For any forms that span multiple pages, make sure to include a progress indicator letting people know where they are in the process. These are most commonly seen during checkout and would include steps such as “Shipping Info > Payment Info > Receipt Confirmation.”
21. Minimize Scrolling & Pages: A good case can be made to limit the number of pages in a a multi-part form in order to prevent customers from abandoning. However, an opposing case can also be made than ridiculously long, single pages forms that require scrolling can scare off customers. There’s no sure-fire rule here, its a perfect opportunity to perform your own a/b test.
22. Strong Call to Action Buttons: Sometimes “Submit” just doesn’t cut it. In other words, be specific and action oriented with your form buttons.
23. Use External Labels: Have you ever used a form that labeled the field with text that disappears when you click into it? This can be a great space saver, but extremely confusing if a customer forgets what the field is for since the label has disappeared. Here’s a great example of why external form labels are more effective.
24. Prioritize Size and Location of Multiple Button Forms: On a form with multiple action buttons, make sure you emphasize the most important button leading to the conversion. For example, your final order confirmation screen has 2 buttons, “Finalize Order” and “Edit Order”, make sure the “Finalize Order” button is larger and more prominent.
25. Clear Confirmations: Have you ever filled out a long, tedious form, clicked submit, only to be returned to what seems like the same page with the form empty? You can do everything right with your form, but if you drop the ball on the confirmation, your customers will be helplessly confused. In addition to making a clear confirmation message, check out these other tips to prevent wasting your confirmation page.

Image Optimization Part 1: The Importance of Images

2008-10-29T23:18:00.000-07:00

This is the first in a series of posts about image optimization. In this series, I’ll explore how images affect web site performance and what can you do to your images in order to improve page loading times. (I won’t say how many posts in this series, so that I can claim later that I underpromised and overdelivered…).

When you think about improving page response time, one of the first obvious things to think about is the page weight. It’s obvious that, all things being equal, the heavier a page is the slower it will be. If we take this to the extreme, we can say that the fastest page you can possibly have is the blank page. Once you start adding stuff to the blank page, you’re only making it slower.

On a more serious note, it really is up to you how much content you want to put on a page, so let’s focus on what comes next. After you’ve settled on the content, it’s your job to make sure the content and components are as small as possible. Following our Yahoo! performance best practices, you should make sure that all plain text components (HTML, XML, CSS, JavaScript…) are sent compressed over the wire and that you minify CSS and JavaScript.

But what about the images, how can you speed them up without sacrificing quality and looks? But first, does it really matter?
How important are the images?

Before we start, let’s see if we should even bother with images. Lately we’ve been witnessing the rise of rich internet applications with lots of JavaScript — by “lots” meaning sometimes 300K or more worth of JavaScript code. In other cases, especially in advertising, Flash seems to be the weapon of choice. So, on average, how much of the page weight is images. It’s easy to answer this question by just looking at Alexa’s top 10 websites in the world (as of October 2008) and use YSlow to check what percent of the total page weight is images. The results are given below.
Percentage of page weight that goes to images, average 46.6% 1 Yahoo! 39%
2 Google 75%
3 YouTube 37%
4 Live.com 94%
5 Facebook 39%
6 MSN 59%
7 MySpace 36%
8 Wikipedia 34%
9 Blogger 28%
10 Yahoo! JP 25%

On average, 46.6% of the page weight for these popular sites consists of images, included either inline with tags or via CSS stylesheets. Other studies show that this percent can be even higher, depending on the cross section of sites you examine. The exact number is not important, because every site is unique and different from the average; for example Amazon’s home page was made of 75% images at the time of the experiment.

This is a massive percentage and it tells us one thing: There’s huge potential to improve the performance of websites if we can improve the way we handle the image payload. By focusing on images you can make a difference and delight your site visitors with a faster and more pleasant experience.
To be continued…

Over the course of the following weeks, we’ll be publishing more about image optimization. The topics for discussion include:

* different image formats and how to pick the right one
* ways to put your images on a diet without compromising quality
* optimizing generated images
* the effect of using AlphaImageLoader
* favicons
* CSS sprites
* serving images faster

The series of posts will not require Photoshop or other designers’ domain-specific knowledge, so it should be pretty easy for anyone to learn and apply these techniques. More to come soon…

Google Adds RSS Feeds For Web Search Results

2008-10-29T23:17:00.000-07:00

Google RSS Feed screenshot

As expected, Google has added an RSS feed for web search results to the Google Alerts service. As seen in the screenshot above, when creating a new alert, you can now choose to get the alert via email or RSS feed. RSS feed alerts are only available to logged-in Google account holders.

As we reported earlier this month, Google is the last major search engine to offer its web search results via RSS.

This is a good addition, but I have to agree with Google Operating System today: “The new feature from Google Alerts is useful, but Google should’ve provided an option to subscribe to feeds for each search result.”