How To Tell If Search Engines, Like Google, Have Visited Your New Website
October 13th, 2008
When new websites are “born”, it make take a while for them to start getting search engine traffic. And you can’t get search traffic from a search engine until they have visited your website and “crawled” over it’s pages. It’s important for new website owners to know how to tell if and when search engines, like Google, the most important search engine, has come and crawled around their website. But how do you tell when Google has visited your website?
The best way to tell, is by using a neat feature of the Google search page. To use this, you can either go to Google.com, or use the Google search field in your browser’s toolbar. The important point here, is to use the Google search field, and not the browser’s regular URL (this is what website you’re on now) field.
In the Google search field, type in “site:” immediately followed by your domain name. No spaces.
So if I want to see the results for my own website, I would type in “site:AldebaranWebDesign.com” and hit the return button.
I’ll paste my own results below, so you can see what these look like:
A few things to highlight. First, you only see results if Google has crawled at least one page of your site. Google won’t crawl your entire site on the first visit – it might take several visits, over a period of time to index all the pages in your website.
Take a look at the results. In the above example, the first line of each snippet, is the title of the page. This is set by the web designer by utilizing the “meta tag” for title. The second line is the page description. This is also set by the meta tag for description. If these look funny or don’t make sense – contact your designer to get these fixed.
See on the last line of each snippet, how it has a link to “Cached” ? The cached version is a snapshot of the page as Google captured it during it’s last crawl. If you click on “Cached” you can see this snapshot as well as a timestamp of when the crawl occured. Fascinating stuff!
How long will it take Google to find my website and pay it’s first visit?
Good question, and the answer is: it depends. When I develop new websites, I always inform Google (and MSN and Yahoo) of the new site’s existence. If you find evidence your new website hasn’t been crawled yet, you can always use this link to submit it to Google. Submitting more than once does nothing and may annoy Google, so don’t do it repeatedly. Another way to get Google to visit your new website, is to make sure other websites have links to your site. When Google crawls them, it will follow those links to your site. I’d say if you have a new website and have submitted it to Google, it should take around a month to get crawled for the first time.
How long will it take to get my whole website fully indexed by Google?
It may take several weeks after the first crawl to get fully indexed, assuming your website is fairly small, under 50 pages. The larger your website, the more time it takes to get fully indexed.
Why is getting indexed so important to getting traffic?
Because you need to be indexed by a search engine before they will send you any traffic, just like you need to have your business published in a phone directory before folks can look you up and find your business. Indexing is a page by page activity – and most likely your home page will be indexed first. But until a page is crawled and indexed by a search engine, that search engine won’t be able to show your page in the results of searches.
What if I have a blog or an online store – will they be indexed?
If you have a blog or an online store that’s written in PHP, yes, they will be indexed as well. I don’t work in other languages, so I don’t have personal proof, but I do believe that most kinds of dynamic website pages are crawl-able and index-able by Google. To verify this, simply check yourself. If you have 100 products in your store, or 100 blog articles, they all should eventually be indexed by Google. But it’s a good question to ask your web designer to prove this to you before you invest in a particular blog or online store or other database driven online application.
What about Yahoo and MSN?
You can use the same technique to see if Yahoo and MSN have visited and indexed your website. The only difference is that like Google, MSN shows you the crawl date via “Cached Page”, while Yahoo doesn’t. Just remember you’re typing “site:yourdomain.com” in the search field.
So if you have a brand new website, or an old one that isn’t bringing you much search engine traffic, take a look and make sure your website has been crawled and indexed by the major search engines like Google, MSN and Yahoo. You can’t get free search traffic until you get crawled.
Jill--------------
J. Olkoski
Aldebaran Web Design, Seattle
Jill Olkoski has a BS in Engineering, a BS in Computer Science and an MA in Clinical Psychology. She delights in using her advanced technical and psychological skills to help small business owners develop cost-effective and successful websites.
February 17th, 2009 at 1:39 pm
Do you have to be online at the time for google to crawl your page and cache it?
February 17th, 2009 at 1:55 pm
Hi Maureen,
No, your being online is completely irrelevant since your website files are always available from your website’s server. The only time Google couldn’t crawl your site, is if your website was down. Your own browsing is completely separate.
May 20th, 2009 at 2:19 am
Hey
I have done what you said about site:website.com and my website is showing up…that means google has crawled my page…but when i am using that particular keyword for the page and going for general search then my page doesnt appear at all….what could be the reason????
Thanks…
May 20th, 2009 at 8:32 am
Hi Mushir,
There are many, many, many factors that Google considers when ranking a page for a particular keyword. While it’s good that Google has visited and indexed your pages, that is no guarantee you’ll get good rankings for your keywords. Necessary, but not sufficient. Try reading the information on Google Webmaster Central to begin to learn about how to optimize your web pages. This activity takes time and patience.
June 5th, 2009 at 4:14 am
Thanks,you just saved me $250.00 ……..we too got the email from mplw.com.regards lorraine
June 5th, 2009 at 9:26 am
Hi Lorraine,
Telling me I just saved you $250 has made my day – it’s exactly why I write these articles – yippee!
July 2nd, 2009 at 9:59 am
My website is not getting indexed and I have meta-tags and everything! What’s the reason for that? ould it be that my site’s not finished?
July 6th, 2009 at 9:53 am
Hi Daniel,
There are many, many reasons why a website isn’t indexed. Meta tags have nothing to do with it. I recommend making sure your site has been submitted to Google and making sure you have created a sitemap.xml file to help search engines find all of your pages. Google has a bunch of webmaster tools that can also give you some assistance. But if your website is brand new, it often takes weeks or months to get crawled.
July 15th, 2009 at 7:45 am
Hello,
I was checking how G. was indexing my site and was writing the URL in the browser’s regular URL, obviously saw nothing and panic…
So I searched and found your page, very useful, thank you very much!
July 15th, 2009 at 9:24 am
Hi Pablo,
Very glad to help!
April 27th, 2010 at 9:33 am
I am just starting out in internet marketing and all of this can be a bit confusing! Thank you so much for the information you provided. I now know google has indexed my web page. Now I just need to figure out how to rank via my key words. I am using squidoo to build my page (trying to save money) so the Google webmaster doesn’t really help me. Any other suggestions would be really appreciated. Thanks so much for your help! 🙂
March 27th, 2011 at 4:27 pm
Jill, thanks for the article. I wasn’t sure if my site had been indexed but now I know a quick and easy way to check.
June 22nd, 2011 at 9:13 pm
So I looked at the site: option and I am indexed as the pages came up.
How many back links should I be getting before I begin to see my rank move up? I know it wil be differnt for each site and each website category, but there has to be some sort of guidline?
20 links? 40? 1000?
My site is http://www.belly-fat-workouts.com and I have only a few back links, but should I be posting more on my home page to work up links better in google.
This is all a little bit confusing to me
June 23rd, 2011 at 7:41 am
There’s no guideline for backlinks – the more the better. Once Google’s crawled all your pages, it might be a few weeks before your initial ranking settles down. From there, you move up by adding unique relevant content (perhaps by adding a blog to your website). As your domain ages, you might drift up a bit as well. Getting other folks to link to you is hard, blogging is the best way I’ve found to do that.
July 15th, 2012 at 6:28 pm
Thanks for this helpful tip. We went live with a new website very recently and when I search for the url I can’t bring it up, but if I do the “site:” search I get several pages of results. However, none of the search results include the “cached” link. Does that mean that the site has been submitted to search engines but not crawled yet?
August 24th, 2012 at 12:17 am
Hello,
Very good article here,but I have seen some other articles on how to know if google has indexed my site and the query included the www.(I.e site:www.sitename.com. If I use the method you said above my site shows on google,but if I use this one in my comment it shows there is no document matching my site. Which is correct way to know if my site has been indexed by google?
August 24th, 2012 at 8:53 am
Wonderful question. Most websites, if they’ve been set up correctly by the designer, has set a redirect to either the www or non www version. For my clients, I redirect all to www. For my own website I have it without www. However, I always run this test without either, with just the domain name, and it works regardless of whether the redirect is set or now.
Now, if you’re checking links TO your site, you have to take into account the www vs non www issue, but to just check indexed pages, check just the domain name.
November 7th, 2012 at 12:39 am
Thanks for this. helped me get answers quick about telling google about my new site. i love the layout. allowed me to find answers quick. cheers