Most of the time we want all search engines to spider our websites and index all pages. Because it is very important to search engine optimization. But There are time when we doesn’t want search engines to crowl our web sites. Times like if the site is still developing or you have made a diplicate of your original site for testing purposes you don’t want search engines to spider these web sites. The best way to stop it is using robot.txt file. The reason is, if you allow search engines to crawl these duplicate sites or pages, would be considered duplicate content by Google and other search engines

This is how to stop all search engines spidering your site. Just make a text file called robot and put these lines ;

User-agent: *
Disallow: /

How to tells all crawlers not to enter into some directories of a website

User-agent: *
Disallow: /cgi-bin/
Disallow: /images/
Disallow: /tmp/
Disallow: /test/

How to tell a specific crawler not to enter one specific directory

User-agent: Googlebot
Disallow: /DirectoryName/

How to tell all crawlers not to enter one specific file

User-agent: *
Disallow: /directory/file.html

I think that’s enough for you to get a pretty good idea about how to use robot.txt file. It can be very useful in times. Specially having a robots.txt file is much preferred over ‘nofollow’ links.

Affiliate - Person or organization which earns money from their website by placing advertising (text, banners, pop-ups etc.) on it that generate actions that have been defined as commissionable by merchants

CPC - CPC stands for cost per click( through ) and refers to the cost incurred in getting one person to click on an advertisement ( banner, text or other form of advertising ) and going to the site referenced by the link.

CPA - CPA stands for Cost per Action. Generally speaking publishers that own trafficked web sites are paid by the advertiser when a visitor performs a specific action. These actions can vary, but usually include either viewing a site, filling out a form to join a newsletter. CPA ad networks are considered to be one of the most advantageous forms of advertising available online.

PPC - Pay per click (PPC) is an advertising model where advertisers only pay when a user actually clicks on an ad to visit the advertiser’s website. Both refer PPC and CPC to the same concept, but PPC refers more to the business model, and CPC to the actual price.

CPS - Cost per Sale (CPS). Pricing based on the number of sales transactions your ad generates.

PFP - Pay For Performance is same thing like CPA, way for advertisers to select how they want to pay for their advertising — by click, impression, sale, or other variable.

PPL - Pay per Lead is an online advertising payment model in which payment is based solely based on qualifying leads. The advertiser only pays for leads generated at their destination site. No payment is made for visitors who don’t sign up.

PPS - Pay per Sale is the same Cost pay Sale ( CPS ) concept .

CPM - Cost per thousand impressions. The CPM model refers to advertising bought on the basis of impression. This is in contrast to the various types of pay-for-performance advertising, whereby payment is only triggered by a mutually agreed upon activity (i.e. click-through, registration, sale).

I’m Hasitha, Free lance webmaster who spend most of the time online. Its been while since I registered this domain and finally I have started a blog with it. I guess I would write about areas related to my work like search engine optimization, Ads Networks like AdSense  and things about webmastering.