Robots.txt and what does it do

What the Robots.txt

Robots.txt and what does it do?
Have you ever think like that?
Have terbesit in your mind to know robots.txt?

This text file is often underestimated, but sanggat truly remarkable function. Why incredible? Later I will describe them. Continue to refer to this article.

Previous better if we get acquainted with small files on this one.

robots.txt is a filter for our website, or can we equate with a security guard in front of the house or an office. We know saptam useful to filter out anyone who may come into the office and also give direction where guests will it, eg to the receptionist. Security guard also gave a ban so do not go to a place undue, ie should not go into the barn, the secret chamber, and others.

Same with the security guard, robots.txt also provide direction and prohibition to guests. Well, here is a robot search engine. Say google. Google has a robot called google-bot. Well, the task of google bots are looking for and track every website listed in google webmaster. With google-bot, Google collects all things about a website, such as a recent article, login pages, specific pages, and more.

This is where the role of a small file called robots.txt earlier. (Robots.txt) will give directives and prohibitions against google-bot. For example robots.txt allow google-bot to access a folder, or prohibit google-bot accessing a folder. However, if the robots.txt file is not found on a website, of course, google-bot will se delish themselves exploring the website.

Then, klo se crawled by google-bot delish ngaruhnya what? This is obviously dangerous. Because config files, upload form, the upload folder, and other confidential files will be tracked by Google-bot and save it to a database of google. Actually this is not a problem. However, problems arise when there is a prankster looking for a website as a victim of hacking for fun in google. They will easily find the files config our website and easily outsmart.

Explain, that the robots.txt file is very useful for a website. Now we try to create a robots.txt file even though many website providers robots.txt generator.

Previously we create a scenario. owned search engine robots should not access the folder config and admin. Here's how.

First create a file with the name and extension robots txt so, his name robots.txt then place it in the main folder (if in hosting placed in public_html / www). Then type the following code (shown in red).

User-agent: * <- this means that the rules are written is intended for all robots of all search engines
Dissallow: / config / <- this means that prohibits robots of all the search engines to discover the config folder
Dissallow: / admin / <- this means that prohibits robots from all search engines to browse the folder admin

Then save. Be our robots.txt file. Simple indeed, but its function is remarkable.

Similarly, my brief article about the robots.txt file and its functions and how to manufacture robots.txt itself. I hope this helps.


Thank you for reading the article 77digitalmedia