Most of the time we want all search engines to spider our websites and index all pages. Because it is very important to search engine optimization. But There are time when we doesn’t want search engines to crowl our web sites. Times like if the site is still developing or you have made a diplicate of your original site for testing purposes you don’t want search engines to spider these web sites. The best way to stop it is using robot.txt file. The reason is, if you allow search engines to crawl these duplicate sites or pages, would be considered duplicate content by Google and other search engines

This is how to stop all search engines spidering your site. Just make a text file called robot and put these lines ;

User-agent: *
Disallow: /

How to tells all crawlers not to enter into some directories of a website

User-agent: *
Disallow: /cgi-bin/
Disallow: /images/
Disallow: /tmp/
Disallow: /test/

How to tell a specific crawler not to enter one specific directory

User-agent: Googlebot
Disallow: /DirectoryName/

How to tell all crawlers not to enter one specific file

User-agent: *
Disallow: /directory/file.html

I think that’s enough for you to get a pretty good idea about how to use robot.txt file. It can be very useful in times. Specially having a robots.txt file is much preferred over ‘nofollow’ links.

A sitemap is a little-known secret to enhancing your Web site’s position in the search engine listings. No, it’s not a killer secret that will draw in thousands of new visitors overnight, but it is an important addition to your toolset, and not hard to implement. This article will tell you why you need a sitemap, and how to create one and submit it to the search engines.

The term “sitemap” can refer to two different things. Many large, complex Web sites provide a visual sitemap that visitors can use for quick navigation, if they already know roughly where they want to go. If your site is large or complex, you should provide one of these sitemaps for your visitors.

But this article is about the other kind of sitemap: The kind that is made for the search engines, like Google, to use in indexing your site. There are several forms that these sitemaps can take, but we’ll get to that a little later.

First of all, let’s consider why you even need a sitemap. Google and the other search engines will index your site even if you don’t have a sitemap. However, there are four main advantages to having a sitemap:

1. If your site uses non-HTML links, such as Macromedia Flash menus or JavaScript menus, the search engines will not be able to follow these links, and so they will not find all of your pages. A code-driven site must use a sitemap.

2. A sitemap tells the search engines which pages on your site are more important, and which are less important. This prevents the less important pages from competing with your own pages in the listings.

3. A sitemap tells the search engines which pages on your site are updated more frequently than others. This enables the search engines to ignore your static pages, increasing the likelihood that they will have the most current data on your most dynamic pages.

4. A sitemap enables you to tell the search engines when you have added or updated your site’s content. To some extent, this puts you in control of making the search engines aware of your latest content. Of course, it doesn’t force the search engines to do your bidding, but it tends to make it easier for users to find your new pages more quickly.

So, what is a sitemap?

As mentioned above, there are many possible forms of sitemaps, but we’ll concentrate on the most useful kind, the XML sitemap format created and promulgated by sitemaps.org. This protocol, currently known as “Sitemap 0.90,” is maintained and endorsed jointly by Google, MSN, Yahoo, and Ask, so you know it is pretty much a universal standard.

An XML sitemap consists of a list of pages on your Web site, and standard information about each page. Here is an example:

< url >
< loc >http://www.freelancesubmit.com/Index.htm< /loc >
< lastmod >2008-04-07< /lastmod >
< changefreq >never
< priority >0.3
< /url >

…

< url >

< loc >http://www.freelancesubmit.com/Services.htm< /loc >
< lastmod >2008-04-07
< changefreq >weekly
< priority >0.8
< /url >

…

Don’t worry about the technical details of formatting the XML. We’ll talk about tools that will create this for you in a moment.

There are three things to notice about each entry:

1. LastMod. Tell the search engines the last date (and time) you changed this page. That will tell them which ones they ought to index right away, and which ones they can ignore.

2. ChangeFreq. In case you’re not updating your sitemap all the time, this will give the search engines a clue as to how often they ought to check each page.

3. Priority. This tells the search engines the relative importance of this page, compared to all the other pages in your site.

In assigning a value for “Priority,” on a scale of 0.0 to 1.0, determine which pages are most important and which are least important within your site. We’re not telling the search engines that this “Services” page is in the 80th percentile of all pages on the Web, but it is far more important than the “Index” page within this site. That’s where we want our visitors to end up.

It’s easy to identify pages within your site which are lowest priority. Some examples:

- Privacy Policy - “Contact us” - “About us”

Please don’t misunderstand this. It’s not that your “Privacy Policy” page is unimportant and so you might as well not have one. It’s that your “Privacy Policy” is important enough to take for granted: Your visitors will find it when they need it. But for search engine purposes, you’d rather direct them to the pages where you actually do your business.

So, how do you create a sitemap?

There are a number of software tools that will create a sitemap by reading your site’s content. You will have to adjust the results, especially the “Priority” settings, but most of these do a pretty good job. Search the Web for “sitemap generator,” or for any of the following specific free tools:

- SitemapDoc - XML-Sitemaps - AuditMyPC Google Sitemap Generator

And once you have your sitemap, what do you do with it?

There are three things to do, in sequence:

1. Place the sitemap file into the root directory of your Web server, alongside your main “index” file. And each time you update it, place the new copy there.

2. Notify the major search engines of your new sitemap file each time you update it. For Google, this means to submit it from within “Webmaster Tools.” For other major search engines, search on that search engine for “submit sitemap,” and you’ll probably find where to enter the URL of your sitemap file.

3. Place a reference to the sitemap file in your robots.txt file, as “Sitemap: http://www.freelancesubmit.com/sitemap.xml”. This will make sure that any search engine will find it, even those that you did not submit it to directly. You only need to do this once, unless you change the name or location of your sitemap file.

Once you have your sitemap created and submitted, don’t forget to maintain it. Each time you add a page to your Web site, add it to your sitemap. Each time you update a page on your Web site, update its “lastmod” setting in your sitemap. Try adjusting the “priority” of your pages from time to time to see if it improves the performance of that particular page. And each time you modify your sitemap, resubmit it to the major search engines.

Soruce - Digitalpoint forum

1. Write and submit articles to the article directories.

2. Leave comments on other people’s blogs with a backlink to your site.

3. Answer people’s questions on 3w.an*swe*rs.y*aho*o.c*om.

4. Post in forums and have a link to your site in your signature.

5. Write a press release and submit it to 3w.P*RWe*b.c*om.

6. Advertise your website in the appropriate category on 3w.Cr*aigsL*ist.c*om.

7. Give an unbiased testimonial on a product/service that you have used in exchange for a backlink to your site.

8. Start a blog and submit it to the 100’s of free blog directories.

9. Manually submit your website to the major search engines.

10. Optimize each page of your website for a particular keyword or search phrase.

11. Add a link in your email signature to your website. It’s a free and easy way to get a little more traffic.

12. Make a custom 404 error page for your website redirecting people to your home page.

13. Use PPC search engine advertising.

14. Add a “bookmark this site” link to your webpages.

15. Have a tell-a-friend form on your site.

16. Send articles to ezine publishers that includes a link to your website.

17. Hold a crazy content and make it go viral.

18. Give away a freebie (ebook, report, e-course) to keep people coming back to your site.

19. Add an RSS feed to your blog.

20. Submit your site to any related niche directories on the net.

21. Participate in a banner or link exchange program.

22. Create a software program and give it away for free.

23. Purchase the misspellings or variations of your domain name, or those of your competitors.

24. Buy a domain name related to your niche that is already receiving traffic and forward it to your site.

25. Pass out business cards with your domain on them everywhere you go.

26. Start and affiliate program and let your affiliates send you visitors.

27. Start a page on social bookmarking sites such as 3w.M*ySpa*ce.c*om.

28. Submit a viral video to 3w.Y*ouTu*be.c*om

29. Conduct and publish surveys to your website.

30. Find joint venture partners that will send you traffic.

31. Start your own newsletter or ezine.

32. Use a autoresponder or email campaign to keep people coming back to your site.

33. Purchase ads on other sites.

34. Send a free copy of your product to other site owners in exchange for a product review.

35. Sell or place classified ads on 3w.e*Ba*y.c*om with a link to your site.

36. Post free classified ads on any of the sites that allow them with a link to your site.

37. Exchange reciprocal links with other related websites.

38. Network with other people at seminars or other live events.

39. Purchase advertising in popular newsletters or ezines.

40. Advertise on other product’s “thank you” pages.

41. Create a free ebook and list in on the “free ebook” sites.

42. Buy and use a memorable domain name.

43. Do something controversial.

44. Create an Amazon profile and submit reviews for books and other products that you have read.

45. Start a lens on 3w.S*quid*oo.c*om.

46. Use a traffic exchange (low quality traffic, but can sometimes be worthwhile).

47. Get referrals form similar but non-competing sites.

48. Create and sell a product with resell or giveaway rights and include a link to your site in it o others pass it around for you.

49. Email your list. If you don’t have one, get one.

50. Submit your site to all social Bookmarking sites you find

Free Keyword Searching tools

Free Keyword discovery tools

Digital Point Keyword Research Tool
Google Suggest
Related Pages
Good Keywords
Overture Inventory

Payed Keyword Searching tools :

Wordtracker - free trial also available - Discover the best keywords to
target on your website
Keyword Discovery- free trial also available - Find the best keywords for
your website

Website Analysis tools :

Google Analytics
StatCounter
WebTrends
Omniture
OneStat.com
WebSTAT
Site Meter
Awstats
Fireclick
ClickTracks

Site ranking free tools:

www.alexa.com
www.compete.com
www.ranking.com
www.nielsen-netratings.com/

Keyword density checker tools:

www.ranks.nl
www.keyworddensity.com
www.seochat.com/seo-tools/keyword-density
http://www.seobench.com/keyword-density-analyzer/
http://www.stargeek.com/keyword_density.php
http://www.virtualpromote.com/tools/keyword-analyzer/
www.golexa.com/

Back link Checker:

http://www.backlinkwatch.com/
http://www.webconfs.com/anchor-text-analysis.php
http://www.prsearch.biz/inbx.php

Blog & RSS Feed Analyzing tools:

Pheedo
MeasureMap
Feedburner
Performancing

Link partner searching tools:

Link Partner Finder
Find Link Partners from zebonline

Tools which you may find helpful:

Link Popularity Check - quick and easy to use web tool from Marketleap.
OptiLink Software is a link analysis program that reveals how top ranked pages achieved their search engine rankings.
Zeus Internet Robot - automatic reciprocal link generator and link directory creator from Cyber-robotics.
Arelis Link Builder - reciprocal links solution from Axandra/Voget Selbach Enterprises GmbH

Search engine optimization (SEO) is a foreign field to a lot of people. Rarely does a day go by when I don’t get asked a few questions on the subject. So I’ve decided to post this FAQ article in the hopes that it will help people understand the basics, and make them a little more comfortable with the whole domain.

Q: Why are search engines important to me?

A: 85% of all website traffic is driven by search engines. The only online activity more popular than search is email. 79.2% of US users don’t go to page 2 of search results. 42% of users click on the no.1 result. For the under-40 age-group, the Internet will become the most used media in the next 2-3 years.

Q: How do search engines decide on their rankings ?

A: IMPORTANT: You cannot pay a search engine in return for a high ranking in the natural results. You can only get a high ranking if your content is seen as relevant by the search engines.
Search engines identify relevant content for their search results by sending out ¡¥spiders¡¦ or ¡¥robots¡¦ which ¡¥crawl¡¦ (analyze) your site and ¡¥index¡¦ (record) its details. Complex algorithms are then employed to determine whether your site is useful and should be included in the search engine¡¦s search results.

Q: Can’t I just pay for a high ranking ?

A: No. The biggest concern for search engine companies like Google and Yahoo is finding content that will bring them more traffic (and thus more advertising revenue). In other words, their results must be relevant. Relevant results makes for a good search engine; irrelevant results makes for a short-lived search engine.

Most search engines these days return two types of results whenever you click Search:

> Natural/Organic ¡V The ¡¥real¡¦ search results. The results that most users are looking for and which take up most of the window. For most searches, the search engine displays a long list of links to sites with content which is related to the word you searched for. These results are ranked according to how relevant and important they are.
> Paid ¡V Pure advertising. This is how the search engines make their money. Advertisers pay the search engines to display their ad whenever someone searches for a word which is related to their product or service. These ads look similar to the natural search results, but are normally labeled ¡§Sponsored Links¡¨, and normally take up a smaller portion of the window.

Q: How do I get a high ranking ?

A: There are four main steps:
Step 1 - Use the right words on your website
Step 2 - Get lots of relevant sites to link to yours
Step 3 - Use the right words in those links
Step 4 - Have lots of content on your site & add more regularly

Q: What is search engine optimization (SEO) ?

A: Search Engine Optimization (SEO) is the art of creating a website which is search engine-friendly. This means:
> using the right words in your copy
> using the right words in your HTML code
> structuring your site properly
> designing your site properly

Many people use SEO to also describe the other ingredient in a high ranking, ‘Link Popularity’.

Q: What is link popularity ?

A: Think of the search engines as a big election. All the websites in the world are candidates. The links to your website are votes. The more votes (links) a candidate (website) has, the more important it is, and the higher its ranking. Link popularity is all about how many links you have, and how you can get more.
Links to your site tell the search engines how important your site is. They assume that if it¡¦s important enough for a lot of other sites to link to, it¡¦s important enough for them to display at the top of the rankings. Links are the single most important factor in ranking. Generally speaking, the more links you have to your site from other sites, the better your ranking.

Q: Are some links better than others ?

A: Yes! The ideal kind of links are those that:
> come from relevant sites (sites which use the same keywords);
> come from important sites (have a high ranking);
> include your keyword as part of the visible link text;
> include varying link text (not the same link text each time); and
> come from a page that links to few other sites.

When a search engine sees a link which satisfies most or all of these conditions, it says, ¡§Hey, this site must be credible and important, because others in the same industry are pointing to it.¡¨

Q: How do I get lots of links back to my site ?

A: There are many possible ways to generate links. Some are dubious (like auto-generation software, and sites set up by webmasters simply to host links to their other sites) and I won¡¦t be discussing them here. Others, like those discussed below, are legitimate.
> Add your site to DMOZ & Yahoo Directories (and other free directories)
> Check where your competitors¡¦ links are coming from
> Article PR - Write and submit articles for Internet publication
> Swap links
> Partner websites
> Pay for links

Read more here