Hour 9: Categorizing the content

June 24, 2025

Now that we've retrieved all the content, it's time to figure out how we can use that information to answer questions people are seeking answers to via Google.

Okay, the general idea here is that we're going to be converting all of the content that we have into text.

This would involve generating questions based on a given topic, finding the search volume for these questions, and assembling answers to these questions using the vector DB. The questions we generate would be based on topic clusters that we can identify using an LLM on the text content that we feed it.

This is the rough general idea that I have, and it's going to be dependent on the size of the material that we have. Arabic takes up a lot more tokens than English per word, so this is going to affect how we use LLMs with the entire corpus of data.

Rough plan:

  1. Content Processing & Storage

    • Convert all retrieved content into searchable text format
    • Set up vector database to store embeddings of the content
    • Handle Arabic text tokenization efficiently (considering higher token usage)

  2. Topic Clustering & Analysis

    • Use LLM to analyze the corpus and identify key topic clusters
    • Group related content by themes (humanitarian, political, historical, etc.)
    • Create topic hierarchies for better organization

  3. Question Generation

    • Generate relevant questions for each topic cluster
    • Focus on questions people are likely to search for on Google
    • Ensure questions cover different angles and perspectives

  4. Search Volume Research

    • Research search volumes for generated questions using keyword tools
    • Prioritize high-volume, relevant queries
    • Identify content gaps where we have answers but low search presence

  5. Answer Assembly System

    • Build system to query vector DB for relevant content
    • Create comprehensive answers by combining multiple sources
    • Ensure factual accuracy and proper attribution

  6. Content Optimization

    • Structure answers for SEO optimization
    • Create different content formats (articles, FAQs, summaries)
    • Implement content validation and fact-checking processes