Tweets Forwarding

The Tweets Forwarding feature in Bragabot integrates Twitter activity seamlessly with Telegram groups. This implementation involves two key components: a Custom Twitter Endpoint and a Recent Search Mechanism. The Custom Twitter Endpoint is a dedicated service that runs in a separate Kubernetes pod, responsible for real-time tweet streaming and periodic searches to ensure tweets are consistently forwarded to the relevant Telegram groups.

Architecture Overview

1. Custom Twitter Endpoint: A standalone service running in a Kubernetes pod that handles real-time tweet streaming and periodic tweet searches.

2. Bragabot Backend: The core platform where groups and their settings are managed, and where the forwarded tweets are processed and stored.

3. Database: Stores configuration settings for each group, including enabled tweet types, social links, and history of forwarded tweets.

Workflow

Tweets Forwarding Worflow

Components and Implementation Details

1. Custom Twitter Endpoint

The Custom Twitter Endpoint is the core of the Tweets Forwarding feature. It handles two primary tasks:

1. Real-Time Tweet Streaming: Captures tweets in real-time based on dynamic rules configured for each group.

2. Recent Search Mechanism: Periodically searches for tweets posted within the last 15 minutes as a backup mechanism to ensure no tweets are missed.

1.1. Real-Time Tweet Streaming

The real-time streaming service listens for tweets that match the criteria set by the groups in Bragabot. It filters these tweets and forwards them to the relevant Telegram groups.

Steps:

1. Fetching Active Groups:

   • The service fetches all groups from the database where Tweets Forwarding is active.

   • For each group, it retrieves settings such as which tweet types are enabled (e.g., Tweets, Retweets, Replies).

2. Building Streaming Rules:

   • The service dynamically constructs rules for the Twitter Streaming API based on group settings.

   • Rules include filters for specific tweet types, such as excluding replies or including only tweets from the project’s Twitter handle.

   Code Example: Constructing stream rules based on group settings

def build_stream_rules(groups):
    rules = []
    for group in groups:
        if group.tweets_forwarding_active:
            base_rule = f"from:{group.twitter_handle}"
            
            if not group.include_replies:
                base_rule += " -is:reply"
            
            if not group.include_retweets:
                base_rule += " -is:retweet"
            
            if not group.include_quotes:
                base_rule += " -is:quote"
            
            rules.append({"value": base_rule, "tag": f"group:{group.id}"})
    
    return rules

# Usage
groups = fetch_active_groups_from_db()
stream_rules = build_stream_rules(groups)

1. Starting the Stream:

• The Twitter Streaming API is initiated using the constructed rules.

• As tweets are captured, they are processed and forwarded to the corresponding Telegram groups.

Code Example: Starting the Twitter stream.

def start_twitter_stream(rules):
    # Initialize the Twitter Streaming Client
    stream = TwitterStreamingClient(bearer_token=os.getenv('TWITTER_BEARER_TOKEN'))
    
    # Apply the rules
    stream.set_rules(rules)
    
    # Start the stream
    stream.filter(track=[rule['value'] for rule in rules])

# Usage
start_twitter_stream(stream_rules)

1. Processing and Forwarding Tweets:

   • Each tweet is processed to check if screenshots should be captured and whether social links should be appended.

   • The tweet, along with any additional information, is then forwarded to the designated Telegram group.

Code Example: Processing a tweet.

def process_tweet(tweet, group):
    # Check if screenshot is enabled
    if group.screenshot_enabled:
        screenshot_url = capture_screenshot(tweet)
        tweet['screenshot'] = screenshot_url
    
    # Check if social links are enabled
    if group.social_links_enabled:
        social_links = fetch_social_links(group.id)
        tweet['social_links'] = social_links
    
    # Forward the tweet to the Telegram group
    forward_to_telegram_group(tweet, group.telegram_group_id)

1.2. Recent Search Mechanism

Overview: The Recent Search Mechanism serves as a backup to the real-time stream, ensuring that any tweets missed due to stream failures are captured and forwarded.

Steps:

1. Periodic Execution:

   • The Recent Search runs every 5 minutes as a scheduled job within the Kubernetes pod.

2. Fetching Recent Tweets:

   • The service queries Twitter’s API for tweets posted in the last 15 minutes that match the group’s settings.

Code Example: Fetching recent tweets.

from bragabot_utils import TimeUtils

def build_recent_search_query(group):
    base_query = f"from:{group.twitter_handle} since:{(TimeUtils.get_current_utc_time() - timedelta(minutes=15)).isoformat()}"

    # Apply group-specific settings
    if not group.include_replies:
        base_query += " -is:reply"
    
    if group.include_retweets:
        base_query += " OR is:retweet"
    
    if group.include_quotes:
        base_query += " OR is:quote"
    
    return base_query

def fetch_recent_tweets(groups):
    recent_tweets = []
    for group in groups:
        query = build_recent_search_query(group)
        tweets = twitter_api.search_recent_tweets(query=query)
        
        for tweet in tweets:
            if not is_tweet_already_notified(tweet.id, group.id):
                recent_tweets.append((tweet, group))
    
    return recent_tweets

# Usage
recent_tweets = fetch_recent_tweets(groups)

1. Processing and Forwarding:

   • Similar to the real-time stream, the fetched tweets are processed and forwarded to the Telegram groups if they haven’t been notified yet.

2. Database Update:

   • Each processed tweet is logged in the database to ensure it isn’t forwarded again by either the stream or future searches.

2. Bragabot Backend Integration

The Bragabot Backend handles the management of group settings, such as enabling/disabling Tweets Forwarding, configuring tweet types, and managing social links. It also interacts with the Custom Twitter Endpoint by providing the necessary group settings and receiving the forwarded tweets for logging and display in the Telegram groups.

Key Functions:

• Group Settings Management: Admins can configure which tweet types should be forwarded, whether to include screenshots, and which social links to add.

• Integration with Custom Twitter Endpoint: The backend periodically updates the Custom Twitter Endpoint with group settings.

3. Handling Screenshots and Social Links

When processing tweets, the system checks whether screenshots and social links should be included. If enabled, the system interacts with the Bragabot Screenshots API and the database to retrieve the necessary data.

Steps:

1. Requesting Screenshots:

   • If screenshots are enabled, a request is sent to the Bragabot Screenshots API to capture the tweet’s appearance.

Code Example: Capturing a screenshot.

def capture_screenshot(tweet):
    response = requests.post(
        url="https://api.bragabot.com/bragabot-screenshot/screenshot/",
        json={"tweet_url": f"https://twitter.com/{tweet['author']}/status/{tweet['id']}"}
    )
    return response.json().get('screenshot_url')

# Usage
screenshot_url = capture_screenshot(tweet)

1. Fetching Social Links:

   • If social links are enabled, they are fetched from the database and formatted for inclusion in the forwarded message.

The Tweets Forwarding feature in Bragabot is a robust implementation that ensures real-time and reliable forwarding of tweets from your project’s Twitter account to Telegram groups. By utilizing a Custom Twitter Endpoint, real-time streaming, and a backup search mechanism, Bragabot guarantees that no tweet is missed, and each tweet is processed according to the group’s specific settings. The modular design, running independently in a Kubernetes pod, ensures scalability and resilience, making this feature an essential tool for community engagement.