Web scraping is a computer software technique of extracting information from websites. Usually, such software programs simulate human exploration of the World Wide Web (WWW) by either implementing low-level Hypertext Transfer Protocol (HTTP), or embedding a fully-fledged web browser such as Mozilla Firefox. Web scraping is closely related to web indexing, which indexes information on the web using a bot or web crawler and is a universal technique adopted by most search engines. In contrast, web scraping focuses more on the transformation of unstructured data on the web, typically in HTML format, into structured data that can be stored and analysed in a central local database or spreadsheet. Web scraping is also related to web automation, which simulates human browsing using computer software. Uses of web scraping include on-line price comparison, contact scraping, weather data monitoring, website change detection, research, web mash-up and web data integration.
