How to use web scraping for marketing and product analytics

Business success depends not only on technical implementation but also on careful analytical work and planning. To avoid losing time and money, it is better to study the market at once: analyze demand, competitors, target audience and external factors so that you can make decisions based on real market opportunities rather than on your hunches.

Qualitative research opens new trading opportunities, helps to reveal the competitors’ advantages, and explores the target audience’s desires. Companies that analyze the market reach their expected revenue levels faster.

What is data-driven marketing?

Data-driven marketing is marketing based on analyzing large amounts of information about all business processes, mostly about consumers. Now marketers spend over $6 billion annually to create solutions using data management platforms and demand-side platforms to get their message out to users. More than 60 percent of companies are actively using big data to:

• monitor the sales funnel’s viability
• research commercial offers and marketing campaigns
• retain the target audience’s attention
• choose the best advertising channels
• proper allocation of the advertising budget.

Data-driven principles are simple: you make decisions based on an analysis of numbers. Intuition and personal experience take second place. Specialists must be able to interpret data and make hypotheses. In addition, you should take care of how to mine, store and visualize those numbers. This will require both meters and analytics services, as well as technology: machine learning, predictive analysis and artificial intelligence. Classic data-driven marketing helps:

• compile an accurate portrait of the target audience, followed by segmentation
• track channels for attracting traffic
• create personalized campaigns
• test advertising channels and find the most profitable ones
• predict the reaction of the audience to advertising
• expand the customer base
• work with clients by collecting and analyzing feedback, recommendations, evaluations
• improve the customer experience
• develop offers that are relevant to the audience.

By collecting statistics over periods, which you need to systematize somehow, you will find correlations and create models of audience behavior, predict sales and set KPIs. 

How do companies gather data?

Companies collect data in different ways from almost every corner of the internet. Some methods are highly technical, while others are more deductive. Information can either be collected or bought. 

Some data vendors offer already compiled compilations, but they may not contain the values you need. And you won’t have the ability to collect or augment them. 

Data collected by yourself or with the help of a third party is more complete and always up-to-date. You’ll be able to aggregate it independently according to your needs and business requests and work with it as you want. Companies use different tools to collect data:

• web scraping
• feedback form
• email form to receive newsletters
• form for registration and creation of a personal account
• questionnaire to participate in the loyalty program
• form for filling in contact data to place an order
• registration through social networks
• cookies, Google Analytics, Google Keyword Planner.

Those companies that maximize data collection and know how to work with it run almost all of their operations: marketing, sales, staffing, updates and enhancements, deliveries. The more data they collect, the more information they will have about their users, and more importantly, they understand the context of what they do.

What is web scraping?

Web scraping is an automated process of extracting data from certain site pages according to certain rules using bots. A bot is a program in any programming language whose logic of operation is aimed at retrieving data from websites. 

The bot’s task is to retrieve web page HTML code that contains information it is interested in. After receiving the code, scraper analyzes it, extracting all necessary data. This way, the bot can open all possible site pages and get information.

Web scraping can be used to gather any information useful for businesses, whether it’s data about users, products, competitors, and so on. Everything we talked about in data-driven marketing and will talk about below can be obtained through scraping.

Web scraping: is it legal?

The ultimate question is about how you plan to use the extracted data. In many countries, it is legal to collect data for public consumption and use it for analysis, for example. But it is always illegal to collect confidential information without permission for sale or to use the material as your own without identifying the source. 

Websites have their own “Terms of Use” and copyright information, which can be found on the homepage that state how their data can be used and how you can access their site. It’s a good idea to check through the website’s terms of use before beginning any scraping project.

Scraping also falls under several legal provisions. Some of them are Violation of the Computer Fraud and Abuse Act (CFAA), Violation of the Digital Millennium Copyright Act (DMCA), Copyright Infringement.

Why conduct a market analysis?

Market analysis is the collection and processing of information about the environment in which an enterprise will operate or is already operating. Usually aimed at reducing the risks of entrepreneurial activity. The results obtained allow making management decisions, which are always accompanied by risk in conditions of constant market changes and uncertainty in the behavior of both consumers and competitors. It should be conducted when you:

• need to make decisions aimed at business development
• need an objective assessment of your company’s performance
• need to launch a new line of products or services
• need to choose a company’s exit strategy from a crisis.

The final work results and company stability depend on this information. Market analysis is needed for:

• price optimization
• lead generation
• competitor monitoring
• trend monitoring
• investment decision making
• reputation monitoring.

How to conduct a market assessment

The set of trackable metrics is formed directly from the objectives. Focus on key parameters and collect data on factors that accurately or at least hypothetically influence those metrics.

Don’t collect data just in case. Data-driven marketing should help you optimize your business, not waste all of your resources on collecting useless information. 

General market activities study

To clearly define a segment and niche for a detailed analysis of the commodity market, it requires data on market capacity, dynamics, and changes.

Market capacity is the market size of a particular product or service, expressed in the total volume of product sales over a given calculation period. It is usually calculated for a specific territory and a certain period and depends on:

• price levels;
• a product’s consumption size;
• quality of goods;
• The availability of similar ways to satisfy the same need;
• marketing and advertising campaigns.

Based on the capacity indicator you can estimate the prospects for sales growth and the maximum income that can be obtained in the selected segment.

To determine the dynamics and prospects of the market, take the capacity figures for the last 5 years and determine whether there is an increase in demand for goods? If demand is increasing, then the dynamics are positive; if it is decreasing, then the dynamics are negative. If the indicator is negative, find out the reasons, assess the situation and other proposals in the market, and then make a decision – is it worth entering the market or not.

Demand and target audience analysis

The next step is to determine the category of customers who bring most of the company’s income. Consumers are divided into groups based on general characteristics: age, gender, social status, income level and others. In the course of such work, we consider not only consumer inclinations and customs, habits and preferences, but also clarify the reasons for the behavior of specific consumer groups. This makes it possible to anticipate the future structure of their interests.

The demand for the product is determined by:

• customer income and preferences
• product price
• the number of people who are looking for the product
• how much they are willing to pay
• the number of products available for purchase.

Demand for products will show how much customers are interested in the product at a particular time or are in the mood to buy. To analyze demand as accurately as possible, determine market geography and target audience. Look for data to estimate the market in:

• official statistics
• special services
• user generated content (forums, blogs, discussions, etc.)
• marketing research
• search queries.

How to analyze demand

Demand analysis for a product or service will help you understand how much demand there is for the offer and track the dynamics of users’ interests. Collect information before analyzing. It can be obtained through:

• surveys
• the trial run of advertising and research advertising competitors
• study of competitors
• monitoring demand with the help of ready-made services.

Surveys

A survey is a simple and common way to collect data that can be conducted online. Answering the questions will help determine customer expectations and reactions to the product launch, how people will position the new product, what they would like to change in existing products or expect to see in new ones and trace market-shaping patterns.

Questions are created according to the needs of the business using tools such as Google Forms, Oprosso, Examinare. Or ready-made templates from the Internet are used. These can be found on Survio, Typeform, or QuestionPro.

Advertising research

A test advertising campaign is a good starting point for testing product demand. You can place ads in Google or Bing to evaluate the interest of buyers, the value of the offer, the effectiveness of the UTP, the cost of the targeted action, the approximate amount of traffic, the size of the advertising budget, and its payback.

With Facebook Ads Library you can check the contextual advertising of competitors, what is being used in the ads to attract customers and find out how many users of social networks correspond to your audience.

Google Keyword Planner is a tool designed to collect and forecast keyword query statistics and create new advertising campaigns. The service displays average number of search queries by keyword and close variants per month; analyzes word statistics; gets forecasts for existing campaigns; shows a list of words that competitors use for advertising.

Competitor research

If you are launching a new and unique product, you shouldn’t be afraid of competition in your segment. In other cases, you need to study the advantages and disadvantages of competitors in advance.

Study both direct and indirect competitors, they are all a valuable source of information. Pay attention to those who work in the market for a long time and even in a crisis keep the bar and those who lag, to take into account their mistakes. All the information gathered will show who you will have to deal with when you enter the market. Be sure to look at:

• Site: its traffic, functionality, traffic sources, visitor demographics, offerings, promotion tools, and keywords by which users find the site.
• Assortment of products, their ratings, prices, delivery methods, and return conditions.
• Customer reviews, ratings, and ratings of customers – what attracts or discourages customers, what their choice is based on.

For a more detailed analysis of competitors, tools and services such as SimilarWeb, SEMRush, Spywords, Screaming Frog, Marketing Grader, and others are used.

Trend analysis

Analyzing trends will give you an understanding of how your business is performing and predict where current business practices will lead. They can provide information on what’s best to do and what’s not. For example, this applies to investing. 

You can use trend analysis to improve your business by identifying successful areas, tweaking the development of lagging areas, and providing evidence for decision-making. 

To find out what’s trending, use Google Trends or Twitter Trend Takeover.

Google Trends

Google Trends is used to assess general trends in demand dynamics on Google. Such information is especially useful for large stores or when starting to sell new products. The service allows you to select a region, set a period, and look at the situation in searches for images, news, products, and YouTube. Google Trends has a feature that compares trends for related queries.

Twitter trend takeover

Trends on Twitter appear in or near each user’s feed and display the latest popular topics that you can use for market research and trending.

Also, Facebook used to have a “Trending” section to keep users updated. Current topics showed headlines that people were talking about all over Facebook. Unfortunately, Facebook has removed this feature.

Data visualization

It is difficult to conclude the results of a study if the information is not organized. To make the analysis more efficient and convenient, structure the data; otherwise, all the information collected will be hard to comprehend. Present the information in graphs, charts, tables, histograms, or maps. Usually, analysis tools visualize the data at the same time. You can also use special services, such as Google Charts, Tableau, Qlik, Infogram, Orange, Power BI.

When analyzing and visualizing market trends, it is best to combine data from Google Trends and Keyword Planner with data from website scraping for a more complete picture of what is happening. For the analysis, data from 500M products from more than 1.1M stores on the Shopify platform was gathered for November 2021.

We’ll put the number of search queries from Google Keyword Planner and the number of products added to stores into relative figures, like Google Trends, where 100 is peak popularity and 0 is no data.

Example: Vitamin C serum

Vitamin C serum is currently one of the top products in demand, according to online searches. Interest in the product has gradually grown and has remained steadily popular for several years, which is also shown in the chart below. It’s coupled with a growing interest in skin health.

Example: Massage guns

Body massage guns have become very popular on the market of massagers, which has attracted the stores’ attention. In the comparative graph, you can see the positive dynamics of demand for the product and how soon online stores began to add automatic massagers to their assortment. The peak of interest came in Q1 2021 and is still not subsiding.

Example: Overshirts

Recently, there has been an increase in requests for oversize shirts, designed to be worn over other things like a jacket. That is why they are called overshirts, they are versatile. Over time, interest in the product is only increasing, as can be seen in the chart. The peak of interest came in Q3 2021, when overshirts became a fashion trend.

Example: Yoga mats

In times of pandemics, the situation has made people enthusiastic about yoga and other fitness routines. The yoga mat is a must-have for people working out in fitness clubs as well as those who do it at home. 

People’s interest peaked in Q3 2020, when there was a new wave of disease and people began to stay home again. Unfortunately, after that interest in yoga mats died down. By Q3 2021, interest has slowly started to grow again.

Closing thoughts 

It is unthinkable today for a company to exist without knowing the basic drivers of the market mechanism. Marketing analysis serves as the basis for developing a whole range of measures, including adapting goods for new target segments or markets, adjusting prices depending on competitors’ ones, and sales promotion. Competent market research significantly increases the awareness of management, reduces entrepreneurial risks, and increases the validity of management decisions.

Sergey Ermakovich is the chief marketing officer at Techvice.