6

Reviewing the question How to scrape rugby union data from espn.co.uk, should we allow web scraping question. This is a wide topic ranging from:

  • technology used for scraping
  • navigation and selection technique to get the data
  • how to avoid anti-bot (scraping) defense from the website

Often web scraping is the last resort when open data is not available. It might be good too to discuss the legality of the technique.

This question is related to Are data extraction techniques on-topic? which was more focused on PDF format.

1
  • HI I have a similar doubt, that is whether this SE provide codes/programmes for webscrapping also ? The answer to the question How to scrape rugby union data from espn.co.uk only pointed to a method but didnt solved the problem itself. What do you think about that
    – Eka
    Apr 3, 2016 at 5:21

2 Answers 2

Reset to default
7

I think we should allow webscraping questions, because unfortunately one often has to resort to it when the data is not nicely dumped. A webpage can be considered as a form of open data, where the format is simply not so machine readable.

I wouldn't worry about the legality, since as far as I know, sharing knowledge regarding webscraping techniques is legal, at least in the vast majority of situations.

4

I think it would be a really bad idea to discuss the "legality" of the issue. Ethics sure but I'd stay away from providing any legal service or advice. I would say allow questions and/or answers that include web scrapping. It's possible that the poster is looking for a way to collect and aggregate the data to make it open.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .