关键词: 网络爬虫, 爬虫排除协议, 保密措施, 商业秘密, 数据, 规制

Abstract: Human society has entered the era of big data. Big data acquisition is an important part of the big data industry. Scraping data through web spider is an important way to obtain data. In order to limit the crawler's access to data, companies rely on robots exclusion protocol and take other technical measures. There have been several cases in judicial practice that the behavior of web crawlers acquiring data constitutes an infringement of the legal interests of computer data, personal information, copyright and neighboring rights. But there is no case yet in which the behavior of spiders is identified as an infringement of trade secrets. Robots exclusion protocol is a generally accepted commercial habit in the Internet and is legally binding. The strength of anti-scraping technical measures is as strong as the confidentiality measures listed in the judicial interpretation, and it is feasible to regulate the behavior of obtaining data by crawlers by trade secret.

Key words: Crawler, Robots Exclusion Protocol, Confidentiality Measures, Trade secrets, Data, Regulations