How do I scrape specific news from news API in Python?

Question

I want to fetch the top headlines from the URL, for that, I have executed the below code:

main_url = "https://newsapi.org/v1/articles?source=bbc-news&sortBy=top&apiKey=ENTER API KEY"

bbc_page = requests.get(main_url).json() 

article = bbc_page["articles"] 

results = [] 

for arr in article: 

    results.append(arr["title"]) 

for i in range(len(results)): 

    print(i + 1, results[i])

Now, my question is how can I extract the top headlines from other news website pages searching for a list of keywords like Coronavirus or other distinct topics?

Answer 1

 For achieving your result, you can use the q attribute for searching query:

Below is the code where we have used q='corona virus':

from newsapi import NewsApiClient

# Init

newsapi = NewsApiClient(api_key='API_KEY') # fb3a5891a786455bb898f36e92b09f24

# /v2/top-headlines

top_headlines = newsapi.get_top_headlines(q='corona virus',

                                          sources='bbc-news,the-verge',

                                          category='business',

                                          language='en',

                                          country='us')

# /v2/everything

all_articles = newsapi.get_everything(q='corona virus',

                                      sources='bbc-news,the-verge',

                                      domains='bbc.co.uk,techcrunch.com',

                                      from_param='2017-12-01',

                                      to='2017-12-12',

                                      language='en',

                                      sort_by='relevancy',

                                      page=2)

Or you can use the API Key in the URL but it is better to put it in the header. Url can get saved in some logs.

import requests

API_KEY = 'fb3a5891a786455bb898f36e92b09f24'

params = {

    'q': 'corona virus',

    'source': 'bbc-news',

    'sortBy': 'top',

    'language': 'en',

    #'category': 'business',

    #'country': 'us',

    #'apiKey': API_KEY,

}

headers = {

    'X-Api-Key': API_KEY,  # KEY in the header to hide it from the URL

}

url = 'https://newsapi.org/v2/top-headlines'

response = requests.get(url, params=params, headers=headers)

data = response.json()

articles = data["articles"] 

results = [arr["title"] for arr in articles] 

for i, arr in enumerate(results, 1): 

    print(i, arr)

The output:

1 Live updates: CPAC attendee tests positive for coronavirus; Italy considers locking down the center of the outbreak

2 Trump: Not concerned at all about coronavirus exposure - CNN Video

3 NHL starts closing dressing rooms to media to prevent the spread of coronavirus

4 Two Australian Defence Force officers test positive for COVID-19

5 Princess says passenger brought coronavirus on the ship; cruise companies to change boarding protocols

6 CPAC Attendee Tests Positive For Coronavirus, Didn't Have Contact With Trump

7 Hobart man who tested positive for coronavirus ignored direction to self-isolate and went to work at the hotel

8 Health Minister says its 'not a day for criticism' as Coronavirus GP accuses health minister of 'grandstanding

9 U.S. conservative conference CPAC attendee tests positive for coronavirus

10 Italy poised to seal off north over coronavirus: Live updates

11 Elderly NSW man becomes third Australian to die with coronavirus

12 Coronavirus update: Two Australian Defence Force personnel infected

13 Live updates: CPAC attendee tests positive for coronavirus; Italy considers locking down the center of the outbreak

14 The divide between those who can buy in bulk, and those who can't

15 Live updates: CPAC attendee tests positive for coronavirus; Italy considers locking down the center of the outbreak

16 CPAC attendee tested positive for coronavirus

17 Hobart man who tested positive for coronavirus ignored direction to self-isolate and went to work at the hotel

18 Coalition pulls multibillion-dollar coronavirus stimulus plan together

19 China January-February exports tumble, imports down as coronavirus batters trade and business

20 U.S. death toll from coronavirus hits 19, New York declares emergency

Join this Python Training course now if you want to gain more knowledge in Python.