Saturday, March 6, 2021
  • Setup menu at Appearance » Menus and assign menu to Top Bar Navigation
Advertisement
  • AI Development
    • Artificial Intelligence
    • Machine Learning
    • Neural Networks
    • Learn to Code
  • Data
    • Blockchain
    • Big Data
    • Data Science
  • IT Security
    • Internet Privacy
    • Internet Security
  • Marketing
    • Digital Marketing
    • Marketing Technology
  • Technology Companies
  • Crypto News
No Result
View All Result
NikolaNews
  • AI Development
    • Artificial Intelligence
    • Machine Learning
    • Neural Networks
    • Learn to Code
  • Data
    • Blockchain
    • Big Data
    • Data Science
  • IT Security
    • Internet Privacy
    • Internet Security
  • Marketing
    • Digital Marketing
    • Marketing Technology
  • Technology Companies
  • Crypto News
No Result
View All Result
NikolaNews
No Result
View All Result
Home Data Science

Scraping eBay using BeautifulSoup in Python

September 26, 2019
in Data Science
Scraping eBay using BeautifulSoup in Python
585
SHARES
3.2k
VIEWS
Share on FacebookShare on Twitter

Even though Amazon is the leader in e-commerce marketplaces – eBay still has its fair share in the online retail industry. Brands selling online should be monitoring prices on eBay as well to gain a competitive advantage. 

Extracting data from eBay at a huge scale regularly is a challenging problem for data scientists. Here is an example of scraping eBay using python to identify prices of mobile phones. 

You might also like

A Plethora of Machine Learning Articles: Part 2

The Effect IoT Has Had on Software Testing

Why Cloud Data Discovery Matters for Your Business

Lets us imagine a use case where you need to monitor pricing of a product, say a mobile phone from eBay. Also, you want to visualise the range of price offering available on the mobile phone which you want to monitor. Moreover, you have other mobile phones under consideration so you may also want to compare their prices as well. In this blog, we will be scraping eBay to collect the prices of phones and find out the difference between their offerings on the eBay website.

Web scraping is an efficient mechanism for collecting data from the internet. You can also read about different myths about web scraping here.

Scraping eBay step by step

In this section, we will walk you through the step by step process of scraping eBay for products and their prices. 

1.Selecting the required information

The very first task in web scraping is to identify the target web page. It is the web page from which you need to extract all the required information. We will be scraping eBay for the product listings so we can just open the eBay website and type our product in their search bar and hit enter. Once page loads with all the product listing of that product, all you need to do is pull that URL out from the browser. This URL will be our target URL. In our case, the URL will be “https://www.ebay.com/sch/i.html?_from=R40&_nkw=galaxy+note+8&am…“. Notice the two parameters in this URL i.e. “nkw” (new keyword) and “pgn” (page number) parameter. These parameters in the URL defines the search query. If we change “pgn” parameter to 2, then it will open the second page of the product listings for galaxy note 8 phone and if we were to change “nkw” to iPhone X then eBay will search for iPhone X and will show you the corresponding results.

2. Finalising the tags for extraction

Once we have finalised the target web page, we need to understand its HTML layout to scrape the results out. This is the most important and critical part of the web scraping and basic HTML knowledge is a pre-requisite for this step. When on the target web page, do “inspect element” and open the developer tools window or just do CTRL+SHIFT+I. In the new window, you will find the source code of the target web page. In our case,  all the products are mentioned as list elements so we have to grab all these lists. In order to grab an HTML element, we need to have an identifier associated with it. It can be an id of that element or any class name or any other HTML attribute of the particular element. We are using the class name as the identifier. All the lists have the same class name i.e. s-item. On further inspection, we got the class names for the product name and product price which are “s-item__title” and “s-item__price” respectively. With this information, we have successfully completed step 2!

3. Putting the scraped data in a structured format
After having our extractors/identifiers we only need to extract specific portions out from the HTML content. Once this is done, we need to organise this data into a suitable structured format. We will be creating a table where we will have all the product names in one column and their prices in the other.

4. Visualising the results (optional)
Since we are to compare the price offerings on two different mobile phones, we will visualise the results too. This is not a mandatory step for web scraping but is more of a process to turn your collected data into some actionable insights. We will be plotting boxplots to understand the distribution of the price offerings on both galaxy note 8 and iPhone 8 mobile phones.

Required libraries and Installation

To implement web scraping for this use case, you will need python, pip (package installer for python) and BeautifulSoup library in python for web scraping. You will also need pandas and numpy library to organise the collected data into a structured format. 

  1. Installing Python and PIP
    Depending upon your operating system, you can follow this blog link to setup python and Pip in your system.
  2. Installing Beautiful soup library

    apt-get install python-bs4

    pip install beautifulsoup4

  3. Installing pandas and numpy

    pip install pandas

    pip install numpy

    We are done setting up our environment and now can begin with the scraping implementation using python. Implementation consists of the steps discussed in the earlier section.

Python implementation for scraping eBay

In this section, we will perform two scraping operations i.e. one for the iPhone 8 and other for the galaxy note 8 mobile phones. Implementation has been repeated for the two mobile phones for easier comprehension. A more optimised version can have two separate scrapping activity combined into one which is not required right now though.

Scrapping eBay for Galaxy Note 8 products

item_name = []

prices = []

 

for i in range(1,10):

 

    r= requests.get(ebayUrl)

    data=r.text

    soup=BeautifulSoup(data)

 

    listings = soup.find_all('li', attrs={'class': 's-item'})

 

    for listing in listings:

        prod_name=" "

        prod_price = " "

        for name in listing.find_all('h3', attrs={'class':"s-item__title"}):

            if(str(name.find(text=True, recursive=False))!="None"):

                prod_name=str(name.find(text=True, recursive=False))

                item_name.append(prod_name)

 

        if(prod_name!=" "):

            price = listing.find('span', attrs={'class':"s-item__price"})

            prod_price = str(price.find(text=True, recursive=False))

            prod_price = int(sub(",","",prod_price.split("INR")[1].split(".")[0]))

            prices.append(prod_price)

 

from scipy import stats

import numpy as np

 

data_note_8 = pd.DataFrame({"Name":item_name, "Prices": prices})

data_note_8 = data_note_8.iloc[np.abs(stats.zscore(data_note_8["Prices"]))< 3,]

Collected Data for Galaxy note 8

scraping eBay | Note 8 data

Scrapping eBay for iPhone 8 

item_name = []

prices = []

 

for i in range(1,10):

 

    r= requests.get(ebayUrl)

    data=r.text

    soup=BeautifulSoup(data)

 

    listings = soup.find_all('li', attrs={'class': 's-item'})

 

    for listing in listings:

        prod_name=" "

        prod_price = " "

        for name in listing.find_all('h3', attrs={'class':"s-item__title"}):

            if(str(name.find(text=True, recursive=False))!="None"):

                prod_name=str(name.find(text=True, recursive=False))

                item_name.append(prod_name)

 

        if(prod_name!=" "):

            price = listing.find('span', attrs={'class':"s-item__price"})

            prod_price = str(price.find(text=True, recursive=False))

            prod_price = int(sub(",","",prod_price.split("INR")[1].split(".")[0]))

            prices.append(prod_price)

 

from scipy import stats

import numpy as np

 

data_note_8 = pd.DataFrame({"Name":item_name, "Prices": prices})

data_note_8 = data_note_8.iloc[np.abs(stats.zscore(data_note_8["Prices"])) < 3,]

Collected data for iPhone 8

Scraping eBay using BeautifulSoup in Python

scraping eBay | Iphone data

Visualising the price of products

Now is the time to visualise the scraped results. We will be using the boxplots to visualise the distribution of prices of mobile phones. Box plot helps us in visualising a trend in numerical values. The green line is the median of the collected price data. The box extends from the Q1 to Q3 quartile values of the data, with a line at the median (Q2). The whiskers extend from the edges of the box to show the range of the data.

Scraping eBay using BeautifulSoup in Python

scraping eBay | Price Comparison

For iPhone 8, most of the prices lie between INR 25k-35k whereas most of the galaxy note 8 phones are available in the price range of 25k-30k. However, variation in prices of iPhone 8 is much more than galaxy note 8. iPhone 8 is available for minimum INR 15k on eBay whereas the minimum cost of galaxy note 8 on eBay is around 22-23K INR!

Conclusion

In this blog, we successfully used python for scraping eBay for two different products and their pricing. We also compared the available prices for galaxy note 8 and iPhone 8 to make a better purchase decision. Web scraping coupled with data science can be leveraged for smart decision making be it in the fortune 500 companies or in your day to day life.


Credit: Data Science Central By: Sandra Moraes

Previous Post

Disappointing Binance US and Bakkt Launches Have Something in Common

Next Post

Hackers looking into injecting card stealing code on routers, rather then websites

Related Posts

A Plethora of Machine Learning Articles: Part 2
Data Science

A Plethora of Machine Learning Articles: Part 2

March 4, 2021
The Effect IoT Has Had on Software Testing
Data Science

The Effect IoT Has Had on Software Testing

March 3, 2021
Why Cloud Data Discovery Matters for Your Business
Data Science

Why Cloud Data Discovery Matters for Your Business

March 2, 2021
DSC Weekly Digest 01 March 2021
Data Science

DSC Weekly Digest 01 March 2021

March 2, 2021
Companies in the Global Data Science Platforms Resorting to Product Innovation to Stay Ahead in the Game
Data Science

Companies in the Global Data Science Platforms Resorting to Product Innovation to Stay Ahead in the Game

March 2, 2021
Next Post
Hackers looking into injecting card stealing code on routers, rather then websites

Hackers looking into injecting card stealing code on routers, rather then websites

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Recommended

Plasticity in Deep Learning: Dynamic Adaptations for AI Self-Driving Cars

Plasticity in Deep Learning: Dynamic Adaptations for AI Self-Driving Cars

January 6, 2019
Microsoft, Google Use Artificial Intelligence to Fight Hackers

Microsoft, Google Use Artificial Intelligence to Fight Hackers

January 6, 2019

Categories

  • Artificial Intelligence
  • Big Data
  • Blockchain
  • Crypto News
  • Data Science
  • Digital Marketing
  • Internet Privacy
  • Internet Security
  • Learn to Code
  • Machine Learning
  • Marketing Technology
  • Neural Networks
  • Technology Companies

Don't miss it

Microsoft Exchange zero-day vulnerabilities exploited in attacks against US local governments
Internet Security

Microsoft Exchange zero-day vulnerabilities exploited in attacks against US local governments

March 6, 2021
Hands-on Guide to Interpret Machine Learning with SHAP –
Machine Learning

Hands-on Guide to Interpret Machine Learning with SHAP –

March 6, 2021
$100 in crypto for a kilo of gold: Scammer pleads guilty to investor fraud
Internet Security

$100 in crypto for a kilo of gold: Scammer pleads guilty to investor fraud

March 6, 2021
Revolution by Artificial Intelligence, Machine Learning and Deep Learning in the healthcare industry
Machine Learning

Revolution by Artificial Intelligence, Machine Learning and Deep Learning in the healthcare industry

March 6, 2021
Deploy AI models -Part 3 using Flask and Json | by RAVI SHEKHAR TIWARI | Feb, 2021
Neural Networks

Deploy AI models -Part 3 using Flask and Json | by RAVI SHEKHAR TIWARI | Feb, 2021

March 6, 2021
These two unusual versions of ransomware tell us a lot about how attacks are evolving
Internet Security

These two unusual versions of ransomware tell us a lot about how attacks are evolving

March 6, 2021
NikolaNews

NikolaNews.com is an online News Portal which aims to share news about blockchain, AI, Big Data, and Data Privacy and more!

What’s New Here?

  • Microsoft Exchange zero-day vulnerabilities exploited in attacks against US local governments March 6, 2021
  • Hands-on Guide to Interpret Machine Learning with SHAP – March 6, 2021
  • $100 in crypto for a kilo of gold: Scammer pleads guilty to investor fraud March 6, 2021
  • Revolution by Artificial Intelligence, Machine Learning and Deep Learning in the healthcare industry March 6, 2021

Subscribe to get more!

© 2019 NikolaNews.com - Global Tech Updates

No Result
View All Result
  • AI Development
    • Artificial Intelligence
    • Machine Learning
    • Neural Networks
    • Learn to Code
  • Data
    • Blockchain
    • Big Data
    • Data Science
  • IT Security
    • Internet Privacy
    • Internet Security
  • Marketing
    • Digital Marketing
    • Marketing Technology
  • Technology Companies
  • Crypto News

© 2019 NikolaNews.com - Global Tech Updates