Raga Engine
The RAGA Engine handles concurrent multi-user inference with queue management, dynamic system prompt construction, and two primary output paths. This page complements the Architecture Overview with inference component details.
Engine Workflow
Engine Components
| Component | Function |
|---|---|
| MAX_VLLM_RUNNING (QUEUE 4) | User request queue, capping a maximum of 4 active requests concurrently |
| VLLM /metrics concurrent 2 | Runs model inference with a maximum of 2 parallel concurrency, with metrics monitoring |
| System Prompt Builder | Dynamically constructs the system prompt based on the model's context window (32k / 128k / xx) |
| API Context (100,000 char) | Injects external API search results into the context |
| Basic (10 items) | Baseline workspace knowledge that is always included |
| OCR (3 items) | Indexed documents via INDEX_OCR_KNOWLEDGE, INDEX_OCR_SUMMARIZE_CHUNK, INDEX_OCR_SUMMARIZE |
| RDBMS (10,000 char) | Relational data, processed via Text-to-SQL |
| Audio (20 rows) | Indexed audio transcripts via INDEX_AUDIO_KNOWLEDGE, INDEX_AUDIO_SUMMARIZE_CHUNK, INDEX_AUDIO_SUMMARIZE |
| History Prompt | Previous conversation history within the session |
| Internal Prompt | RAGA's internal system instruction (not exposed to the user) |
| Time Now | Real-time timestamp injection for temporal context |
| files_users | Files uploaded by the user within the active session |
| Personalization | User-specific preferences and context |
| Image Context | Multimodal image content sent by the user |
| Text To API | Converts a user question into a structured API call |
| Text To SQL | Converts a user question into a structured SQL query |
| LLM Provider | The inference backbone is multi-provider, selected per request via the llm parameter: local (Qwen2.5-14B-Instruct-AWQ, default) and local_v2 (Qwen2.5-32B-Instruct-AWQ), both self-hosted via vLLM, plus OpenAI-compatible external providers — sambanova (Llama-3.3-70B), groq (Llama-3.3-70B-versatile), openai (gpt-4o-mini), and alibaba (Qwen-plus) |
| Multimodal Provider | A separate path for image analysis (Image Context): local (Qwen2.5-VL-32B-Instruct-AWQ) or alibaba (Qwen-VL-Max) |
Summary
The RAGA Engine bounds concurrency through a queue (up to 4 active requests, 2 parallel inferences in VLLM), then dynamically assembles the system prompt from eleven context types — from baseline workspace knowledge to Text-to-SQL and Text-to-API results — before passing it to whichever LLM provider is configured (local Qwen-based vLLM by default, or an external provider such as SambaNova, Groq, OpenAI, or Alibaba as an alternative).