Over the past there months I have been developing a new site for my wife. She came up with this great product for children, so we decided to sell the product. Naturally, I will plug the name of her site; but at a later date of course. For the time being, lets stay on topic.The reason I bring up my wife’s site is because it gave me purpose in getting crawled fast on the search engines. More specifically Google.com, Yahoo.com and MSN.com We launched her site late October of 2008 so this information is more than just ‘fresh’ in my mind.

One of the biggest problems with launching a new site is that more than likely the site is also on a new domain name. New domain names in the world of search engines are no different than newborn children in that no one really knows they exist except for those involved in the conception. Eventually a search engine will find your new site, however the time frame between launch and crawling is an unknown time frame. What we want to do in get crawled fast, in efforts to get indexed fast.

So, how do we overcome this issue of getting a site crawled as fast and soon as possible? Well, there are multiple options here, some of which are great practices however I’m going to show you how to go directly to the search engine and get a crawl within a matter of hours if need be. Keep in mind that crawling and indexing are two different things, as indexing will be discussed later.

When getting crawled for Google.com the fasted and most reliable thing you can do is submit your XML sitemap to Google. Basically, the XML sitemap is your way as a webmaster to instruct Google on which pages you feel should be crawled. I know, I know; you’ve probably heard that submitting your XML sitemap to the search engines is a waste of time. Well, let me tell you that anyone who makes that statement is either a moron or has no SEO experience at all.

After you site is live, you can do one of two option in which I recommend. First, you can easily visit XML Sitemaps at www.xml-sitemaps.com plug in your URL on the homepage and download the auto generated XML file. After you download your XML file you will want to save/upload this file in the ‘root folder’ of your site.Or, you can download this killer free software tool called G Site Crawler located at www.gsitecrawler.com Simply download the software, install the software and generate a sitemap. I prefer G Site Crawler as I manage many websites these days and I can organize my XML sitemaps accordingly. I can also create a XML sitemap for Google specially with G Site Crawler, unlike xml-sitemaps. If you only have one or two site, just use the first option above. If you use G Site Crawler, please read the manual first as it makes much more sense this way.After you have create your sitemap.xml file visit the Google Webmaster Central located at www.google.com/webmaster then click on the ‘improve traffic with Google webmaster tools’ option. If you do not have a user account, simply create one. I highly suggest going through the tutorial as well on this one.Once logged into Google Webmaster, enter your domain URL to add into Google Webmaster (ie:www.yoururl.com). You will need to verify your site as well; this is Google’s way of authorization. The easiest way to authorize your site is to generate the Meta Tag that Google provides, plug it into your home page html with the rest of your Meta Data and your good to go. After you have authorized your site you can submit your XML sitemap. Since you uploaded the XML into your root folder, you will need to tell Google where your XML sitemap is located (ie:www.yoururl.com/sitemap.xml).After you submit your sitemap it will take a day or so for Google to crawl to your site. This is truly the most reliable way for Google to crawl your site. If you have more dynamic page content such as a blog, I would suggest using a plug-in that utilizes a dynamic XML. If your page is static; meaning that you will not be adding new pages regularly you will need to generate a new sitemap and submit it to Google when you update your site.

For Yahoo you can visit https://siteexplorer.search.yahoo.com/submit What this does is tell the Yahoo crawlers to crawl your site. I would only suggest you perform this action once as the crawler will revisit your site after the initial visit once its good and ready.You can also perform the same action as Google Webmaster Central with Yahoo Site Explorer, however I find the Yahoo version is primitive and not worth the time (at this current moment).

Performing the above actions will greatly if not guarantee that your new site will be crawled. Getting your site indexed is a whole different realm. There is no way to get your site indexed and cached other than waiting. When the search engines deem your site is ready to be indexed and cached, that’s when it will happen. I can tell you that the more content in general you have, the greater your odds in getting indexed faster. For the sake of this post, we are just concerned with getting crawled fast. If for any reason you do not want to follow these steps and still have this ‘get crawled fast’ mentality I highly recommend that you post your URL on a few blogs and forums. Search engine crawlers are always crawling social media, blogs, forums etc… if you create a back link on a popular site you can literally get crawled within a matter of hours when done right. Either way make sure you submit your XML sitemap to Google.