I am crawling news websites and want to extract News Title, News Abstract (First Paragraph), etc
I plugged into the WebKit parser code to easily navigate the webpage as a tree. To eliminate navigation and other non-news content I take the text version of the article (minus the HTML tags, WebKit provides API for the same). Then I run the diff algorithm comparing various article's text from the same website this results in similar text being eliminated. This gives me content minus the common navigation content etc.
Despite the above approach, I am still getting quite some junk in my final text. This results in incorrect News Abstract being extracted. The error rate is 5 in 10 articles i.e. 50%. Error as in
- Suggest an alternative strategy for the extraction of pure content,
- Would/Can learning Natural Language processing help in extracting the correct abstract from these articles?
- How would you approach the above problem?
- Are there any research papers on the same?