ETL Pipeline for #OnThisDay Tweet Analysis

This workflow is ideal for:

• Social Media Managers: To monitor and analyze tweets related to specific topics, such as #OnThisDay, and engage with followers based on sentiment.
• Data Analysts: To gather and store tweet data in PostgreSQL and MongoDB for further analysis and reporting.
• Marketers: To track sentiment around their brand or campaigns on Twitter, allowing for timely responses.
• Developers: To integrate different data sources and automate notifications through Slack.

This workflow addresses the challenge of efficiently gathering, analyzing, and responding to social media content. It automates the process of:

• Collecting tweets containing #OnThisDay.
• Storing relevant data in both PostgreSQL and MongoDB.
• Analyzing sentiment using Google Cloud Natural Language.
• Sending notifications to a Slack channel when certain sentiment thresholds are met, ensuring timely engagement.

1. Scheduled Trigger: The workflow runs daily at 6 AM to fetch tweets.
2. Twitter Node: It searches for the latest 3 tweets containing the hashtag #OnThisDay.
3. MongoDB Node: The tweets are inserted into the MongoDB collection for storage.
4. Google Cloud Natural Language Node: Each tweet's text is analyzed to determine its sentiment score and magnitude.
5. Set Node: The sentiment score and magnitude, along with the tweet text, are prepared for further processing.
6. Postgres Node: The relevant data is stored in the PostgreSQL database.
7. IF Node: A condition checks if the sentiment score is above a specified threshold.
8. Slack Node: If the condition is met, a notification is sent to the Slack channel with details of the tweet.
9. NoOp Node: If the condition is not met, the workflow simply ends without action.