Technological Search Engine optimization: Specialized Search engine optimization refers to Internet site and server optimization that assists the crawler in crawling, indexing and ranking functions to rank your site superior. txt file is then parsed and may instruct the robotic as to which internet pages are usually not being crawled. https://www.youtube.com/watch?v=ARvdj_mJDAY