Qdrant Vector Database Embedding Pipeline

This workflow is ideal for:

• Data Scientists: Those looking to embed large datasets into a vector database for semantic search and retrieval.
• Machine Learning Engineers: Professionals who need to preprocess and embed text data efficiently.
• Developers: Individuals building applications that require integration with Qdrant and OpenAI for advanced data processing.
• Researchers: Academics or analysts needing to manage and analyze large volumes of text data.
• Business Analysts: Users who wish to leverage AI embeddings for insights from unstructured data.

This workflow addresses the challenge of efficiently embedding and storing large datasets into a vector database. It automates the process of:

• Fetching JSON files from an FTP server.
• Processing each file to extract relevant text data.
• Embedding the processed data using OpenAI's API.
• Storing the embeddings in Qdrant for future semantic retrieval. This saves time and reduces manual errors in data handling and embedding.

1. Manual Trigger: The workflow starts when the user clicks ‘Test workflow’.
2. List Files: It lists all JSON files from the specified FTP directory (Oracle/AI/embedding/svenska).
3. Iterate Over Files: Each file is processed individually to ensure efficient handling.
4. Download Each File: The current file is downloaded in binary format.
5. Parse JSON Document: The downloaded JSON file is converted into a document format compatible with embeddings.
6. Split Text: The text is split into smaller chunks based on a specified separator ("chunk_id").
7. Generate Embeddings: The split text chunks are sent to OpenAI to generate embeddings.
8. Store in Vector DB: Finally, the embeddings are stored in the Qdrant vector database for semantic search.