Google have been smart in enhancing the crawling technology while it comes to modern web designing, be it mobile web design checklist or crawl related problem for smart phones, Google seems to be leading as compared to its counterparts like Bing and other search engines.

Today Google explained how it has made significant improvements as compared to technology being used in 1998 to crawl the websites when it was unable to deal with Java Scripts. While today Google technology has improved to the point where it sees web pages as modern browsers and able to render all the web page to read its contents same like modern web browsers do.

Before this, webmasters assumption was that Google read raw textual content within the page. As now Google is more capable of reading richer websites having Java Scripts, dealing with external files and CSS style sheets, Google have explained few points about how you should allow Google to crawl contents of these external files for better understanding of your web page by Google.

• CSS and External Files: Since Google wants full access of these file to understand your web page, you should not block these files via robots.txt. Now Google wants access of all files like the modern browsers do
• Web Servers: Google also want that your webs server be able to handle the crawl request, if the server is not able to handle request goes time out, Google will not be able to render your website properly and understand the contents which can impact your web site rankings.
• Java Scripts: Google says to use Java Scripts carefully, avoiding such scripts which hide contents from browsers OR scripts which are too complex to render. Google also says to provide options for those users whose browser does not support Java Scripts

So next time you plan to develop rich function website, it will help to take care of Google advice on how Google deals with these websites to avoid potential issues.