Input! Input!

---

In the classic 1986 Film "Short Circuit" the main character is a cute Robotwith an acute thirst for knowledge known only as 'Number 5'.http://www.dvdreview.com/fullreviews/short_circuit.shtmlOne of Ro...
In the classic 1986 Film "Short Circuit" the main character is a cute Robotwith an acute thirst for knowledge known only as 'Number 5'.http://www.dvdreview.com/fullreviews/short_circuit.shtmlOne of Robot number 5s most engaging traits is that whenever he stumblesacross anything interesting he rushes off enthusiastically to collect andindex the data with cries of "Input! Input!".Robots designed to collect and index data actually exist but they are notmade of exotic metals, they are software programs and are used by SearchEngines to gather information about the relevance of your web site to aparticular search term. These Robots ( sometimes called spiders orcrawlers ) are smart but not very selective.
Unless you provide unambiguousground rules for visiting Search Engine robots, excluding them from areasyou don't want them to enter, then every file on your web site will beperceived as "Input!" and is likely to get indexed."But", I hear you ask, "I want to get indexed by Search Engines, why is thisa problem"?Indexing everything sounds superficially smart, however as part of acoherent web site promotion and Search Engine optimization strategy it has anumber of important disadvantages.* Search Engine spiders should be actively discouraged from visiting areaswhere sensitive information might be stored.* Indiscriminately indexing everything can seriously dilute the relevancy ofyour web sites overall theme and can produce a sub-optimal rank in SearchEngine listings.* Allowing a Search Engine spider to index everything can even inadvertentlylead to the perception by some of the Search Engine that your web sitecontains spam, this can lead to your site being blacklisted.* For multilingual web sites it's imperative to focus English languagerobots onto the relevant English language pages and to direct robots frominternational Search Engines, who might be looking for Spanish, German orFrench language resources, to the appropriately localized content areas ofyour site.* Search Engine robots can only "read" text. Dynamic content or graphicalcomponents cannot be read or indexed, rendering your site effectivelyinvisible to Search Engines.* Some robots "rapid fire" requests causing severe, server loading problemswhich can detract from your visitors browsing experience and ultimatelycause loss of business.The answer to this problem lies in having a Robot exclusion file on your webserver.Robot exclusion files, normally in the form of "robots.txt" are ASCII textfiles which reside in the document root directory of web servers and areused to set access permissions and control the actions of robots or spiders.Most of the major US and international Search Engines deploy spiders whichlook for a robots.txt file during their visit to a web site.
There is anagreed industry standard for robots.txt files and, in order to work asanticipated, robots.txt has to be correctly formatted and placed in theproper location on the web server. Once uploaded to your server robots.txtis utilized to notify individual spiders about which elements of a web sitecannot be visited and should not be made available on the public Internet.Used in conjunction with Search Engine optimization tools and/or servicesrobots.txt can significantly enhance your sites chances of thatall-important first page listing on the major US and international SearchEngines by focusing individual spiders on specific content.Although only a small ASCII text file, robots.txt enables a significantdegree of fine tuning to be applied to your Search Engine optimizationprogram.
Used intelligently robots.txt can do a big job, significantlyimproving your knowledge about, and control of, visiting Search Enginerobots. This is particularly the case where a web site owner either wishesto deliver specific content optimized for a particular Search Engine, or haspaid for an accelerated Search Engine listing service where if would beuseful to track the activity of the robot associated with that specificpaid-for service.Just as Robot Number 5 gathered more and more input and transformed thisdata into useful information, so web site owners can use the data generatedby the interaction between robots.txt, visiting spiders and their web logsto gain significant competitive advantage.
Article Tags: Input Input, Search Engine, Search Engines Source: Free Articles from ArticlesFactory.com .