Testing Mulitple Local LLM with LM Studio

This workflow is ideal for:

• Data Analysts: Individuals who analyze and report on model performance metrics.
• Researchers: Those conducting studies on language model outputs and their readability.
• Developers: Engineers looking to integrate local LLMs into applications for testing purposes.
• Educators: Teachers or trainers wanting to evaluate the readability of content generated by language models for educational materials.
• Product Managers: Managers overseeing AI projects that require performance tracking and reporting.

This workflow addresses the need for automated testing and analysis of language model outputs. It provides a systematic approach to:

• Retrieve and manage multiple LLMs.
• Analyze the readability and effectiveness of responses from these models.
• Track performance metrics over time, allowing for informed decision-making and model adjustments.

1. Trigger: The workflow starts manually when a chat message is received.
2. Model Retrieval: It queries the local LLM server to fetch the list of available models.
3. System Prompt Addition: A guiding prompt is added to ensure responses are concise and appropriate for a 5th-grade reading level.
4. Response Analysis: The chat input is sent to the selected models, and their responses are analyzed for various metrics, including:
   • Readability Score
   • Word Count
   • Sentence Count
   • Average Sentence Length
   • Average Word Length
5. Time Tracking: The workflow captures the start and end times of the model execution to calculate total time spent.
6. Data Preparation: The analyzed metrics are structured for reporting.
7. Google Sheets Integration: Results are saved to a Google Sheet for easy access and further analysis.