When new websites are “born”, it make take a while for them to start getting search engine traffic. And you can’t get search traffic from a search engine until they have visited your website and “crawled” over it’s pages. It’s important for new website owners to know how to tell if and when search engines, like Google, the most important search engine, has come and crawled around their website. But how do you tell when Google has visited your website?
The best way to tell, is by using a neat feature of the Google search page. To use this, you can either go to Google.com, or use the Google search field in your browser’s toolbar. The important point here, is to use the Google search field, and not the browser’s regular URL (this is what website you’re on now) field.
In the Google search field, type in “site:” immediately followed by your domain name. No spaces.
So if I want to see the results for my own website, I would type in “site:AldebaranWebDesign.com” and hit the return button.
I’ll paste my own results below, so you can see what these look like:
A few things to highlight. First, you only see results if Google has crawled at least one page of your site. Google won’t crawl your entire site on the first visit – it might take several visits, over a period of time to index all the pages in your website.
Take a look at the results. In the above example, the first line of each snippet, is the title of the page. This is set by the web designer by utilizing the “meta tag” for title. The second line is the page description. This is also set by the meta tag for description. If these look funny or don’t make sense – contact your designer to get these fixed.
See on the last line of each snippet, how it has a link to “Cached” ? The cached version is a snapshot of the page as Google captured it during it’s last crawl. If you click on “Cached” you can see this snapshot as well as a timestamp of when the crawl occured. Fascinating stuff!
How long will it take Google to find my website and pay it’s first visit?
Good question, and the answer is: it depends. When I develop new websites, I always inform Google (and MSN and Yahoo) of the new site’s existence. If you find evidence your new website hasn’t been crawled yet, you can always use this link to submit it to Google. Submitting more than once does nothing and may annoy Google, so don’t do it repeatedly. Another way to get Google to visit your new website, is to make sure other websites have links to your site. When Google crawls them, it will follow those links to your site. I’d say if you have a new website and have submitted it to Google, it should take around a month to get crawled for the first time.
How long will it take to get my whole website fully indexed by Google?
It may take several weeks after the first crawl to get fully indexed, assuming your website is fairly small, under 50 pages. The larger your website, the more time it takes to get fully indexed.
Why is getting indexed so important to getting traffic?
Because you need to be indexed by a search engine before they will send you any traffic, just like you need to have your business published in a phone directory before folks can look you up and find your business. Indexing is a page by page activity – and most likely your home page will be indexed first. But until a page is crawled and indexed by a search engine, that search engine won’t be able to show your page in the results of searches.
What if I have a blog or an online store – will they be indexed?
If you have a blog or an online store that’s written in PHP, yes, they will be indexed as well. I don’t work in other languages, so I don’t have personal proof, but I do believe that most kinds of dynamic website pages are crawl-able and index-able by Google. To verify this, simply check yourself. If you have 100 products in your store, or 100 blog articles, they all should eventually be indexed by Google. But it’s a good question to ask your web designer to prove this to you before you invest in a particular blog or online store or other database driven online application.
What about Yahoo and MSN?
You can use the same technique to see if Yahoo and MSN have visited and indexed your website. The only difference is that like Google, MSN shows you the crawl date via “Cached Page”, while Yahoo doesn’t. Just remember you’re typing “site:yourdomain.com” in the search field.
So if you have a brand new website, or an old one that isn’t bringing you much search engine traffic, take a look and make sure your website has been crawled and indexed by the major search engines like Google, MSN and Yahoo. You can’t get free search traffic until you get crawled.Jill
Aldebaran Web Design, Seattle
Jill Olkoski has a BS in Engineering, a BS in Computer Science and an MA in Clinical Psychology. She delights in using her advanced technical and psychological skills to help small business owners develop cost-effective and successful websites.