Intro

I perform sentiment analysis for trading.

Problem

I want to trade popular stocks that relate to ai tech companies. I am interested in sentiment analysis.

Proposed Solution

1. Get data from different sources and store them in a data lake and in a datawarehouse.
   
2. Calculate sentiment with Bert.
3. Calculate engagement metrics.
4. Build a dashboard that displays sentiment analysis scores and engagement of each stock for each platform separately.
   

Data Sources:

1. Youtube API
2. Reddit API (Prawn)
3. Google Search
4. News Data API - https://newsdata.io/
5. News API - https://newsapi.org/
6. Market Aux API - https://www.marketaux.com/

KPIs / Metrics

We search the company name and related queries on these platforms and use the results in the following manner:

1. Engagement
   

   1. Youtube
      

      1. List Searched Videos: Avg Views
         
      2. List Searched Videos: Avg Likes
         
      3. List Searched Videos: Avg Comments
         
      4. List Searched Videos: Avg Comment Likes
         

   2. Reddit
      

      1. List Searched Post: Avg Score
         
      2. List Searched Post: Avg Upvote Ratio
         
      3. List Searched Post: Avg Comments
         
      4. List Searched Post: Avg Comment Score
         

   3. Google Search
      

      1. Search Result: Count
         
      2. Format Time
         

2. Sentiment
   

   1. Youtube
      

      1. Positive or negative sentiment
         
         1. Titles + Descriptions
            
         2. Comments
            

   2. Reddit
      

      1. Searched
         
         1. Positive or negative sentiment
            
            1. Titles + Descriptions
               
            2. Comments
               

   3. Market Aux API
      

      1. Positive or negative sentiment
         
         1. Titles + Descriptions
            
         2. API's Sentiment Score
            

   4. News API Org
      

      1. Positive or negative sentiment
         
         1. Titles + Descriptions
            

   5. News Data IO
      

      1. Positive or negative sentiment
         
         1. Titles + Descriptions
            

   6. Google Search
      

      1. Positive or negative sentiment
         
         1. Link Titles
            

Tools Used:

1. APIs listed below
2. Duckdb
3. BERT
4. Plotly Express
5. Plotly Dash
6. Pandas

How to run:

1. Configure Project:

python -m venv .localenv
.localenv\Scripts\python -m pip install -r ./requirements.txt

2. Use
   1. Extract Data

   .localenv\Scripts\python -m src.scripts.extract
   

   2. Transform Data

   .localenv\Scripts\python -m src.scripts.transform
   

   3. Dashboard

   .localenv\Scripts\python -m src.dashboard.dashboard
   

Keys

Put the .env file in ./keys/.env:

project
├───keys
    └───.env

URL=http://localhost
PORT=8080

# Youtube
YOUTUBE_API_KEY=
PATH_TO_SECRET=
YPATH_TO_TOKEN=

# Reddit
REDDIT_CLIENT_ID=
REDDIT_CLIENT_SECRET=
REDDIT_USER_AGENT=
REDDIT_USERNAME=
REDDIT_PASSWORD=
RPATH_TO_TOKEN=

# Google
GOOGLE_API_KEY=
GOOGLE_SEARCH_ENGINE_ID=

# NewsDataIO
NEWS_DATA_IO_API_KEY=

# NewsApiOrg
NEWS_API_ORG_API_KEY=

# MarketAux
MARKET_AUX_API_KEY=