Technical — Chatbot Service

RAGA's inference & knowledge orchestration service, built on Python + FastAPI. Accepts a user question along with a list of relevant knowledge sources, gathers context from Elasticsearch and supporting services in parallel, assembles a dynamic system prompt, then calls one of several LLM providers (local or cloud) to generate an answer. Called by api-tarantula (env var CHATBOT_URL) for every incoming chat.

Repository

Key	Value
Git Remote	https://git.tlab.co.id/tarantula/service/chatbot-service.git
Active Branch	main

git clone https://git.tlab.co.id/tarantula/service/chatbot-service.git
cd chatbot-service

Tech Stack

Layer	Technology
Runtime	Python (FastAPI + uvicorn)
Search / Knowledge Index	Elasticsearch
LLM Client	openai SDK (OpenAI-compatible for every provider)
File Parsing	PyPDF2, python-docx, openpyxl
Observability	OpenTelemetry (opentelemetry-instrumentation-fastapi/requests/logging, OTLP exporter)
Secret Management	infisical_sdk (Python)
Testing	pytest + pytest-mock + pytest-cov

Folder Structure

chatbot-service/
├── app/
│   ├── main.py                 # FastAPI app, /chatbot & /health endpoints, knowledge source orchestration
│   ├── run.py                  # Entry point that runs uvicorn
│   ├── queue_manager.py        # Alternative queue manager — NOT used by main.py (see note below)
│   ├── config/
│   │   ├── settings.py         # Config loader via Infisical + LLM provider constants
│   │   ├── logging_config.py
│   │   └── telemetry.py        # OpenTelemetry setup (fail-safe, never crashes if OTel misbehaves)
│   ├── models/
│   │   └── request.py          # Pydantic request schema (ChatbotRequest, KMItem)
│   ├── services/
│   │   ├── llm_service.py      # Multi-provider LLM connection, prompt assembly, streaming, image analysis
│   │   ├── data_service.py     # Elasticsearch queries per knowledge source type
│   │   ├── sql_service.py      # Text-to-SQL: generates the query + calls Database Connect
│   │   └── api_service.py      # Text-to-API: generates the call + calls API Connect
│   └── utils/
│       └── helpers.py          # Download & extract file content from a URL (txt/md/log/pdf/docx/xlsx)
└── tests/                      # Unit tests per service

Environment Variables

Configuration is pulled from Infisical (INFISICAL_PROJECT_ID, INFISICAL_ENVIRONMENT, INFISICAL_SECRET_PATH, INFISICAL_CLIENT_ID, INFISICAL_CLIENT_SECRET, INFISICAL_HOST), falling back to os.getenv() if the Infisical fetch fails.

Elasticsearch

Variable	Description
URL_ELASTIC	Elasticsearch server URL
USER_ELASTIC	Elasticsearch username
PASS_ELASTIC	Elasticsearch password
VERIFY_ELASTIC	Verify TLS certificates (default false)

Note: this repo's app/.env.example contains example values that look like real credentials (a specific host IP and password), not generic placeholders like your_xxx_key. These example values should be replaced with plain placeholders, and any credential that may already have been committed here should be rotated.

Elasticsearch Indices

Variable	Default	Description
ES_INDEX_RDBMS_LIST	rdbms_list	List of external database connections
ES_INDEX_API_LIST	api_list	List of external API definitions
ES_INDEX_BASIC_KNOWLEDGE	basic_knowledge	Baseline knowledge (topics)
ES_INDEX_AUDIO_KNOWLEDGE	audio_knowledge	Audio metadata
ES_INDEX_AUDIO_SUMMARIZE_CHUNK	summarize_audio_chunk	Per-chunk audio transcript summaries
ES_INDEX_AUDIO_SUMMARIZE	summarize_audio	Full audio summaries
ES_INDEX_OCR_KNOWLEDGE	tarantula-ocr	Document OCR results
ES_INDEX_OCR_SUMMARIZE_CHUNK	summarize_document_chunk	Per-chunk document summaries
ES_INDEX_OCR_SUMMARIZE	summarize_document	Full document summaries
ES_INDEX_CHAT_HISTORIES	chat_histories	Conversation history

LLM Providers

Every provider has three variables: API_KEY_*, BASE_URL_*, MODEL_*.

Provider (llm)	Default Base URL	Default Model
local	http://192.168.0.25:8026/v1	Qwen/Qwen2.5-14B-Instruct-AWQ
local_v2	http://192.168.0.27:18000/v1	Qwen/Qwen2.5-32B-Instruct-AWQ
sambanova	https://api.sambanova.ai/v1	Meta-Llama-3.1-70B-Instruct
groq	https://api.groq.com/openai/v1	llama-3.3-70b-versatile
openai	https://api.openai.com/v1	gpt-4o-mini
alibaba	https://dashscope-intl.aliyuncs.com/compatible-mode/v1	qwen-plus

If base_url_llm + api_key_llm + model_llm are sent directly in the request, all three are used as-is (overriding llm).

Multimodal Model (Image Analysis)

Variable	Default	Description
MULTIMODAL_SERVER	local	Provider used for image analysis (local or alibaba)
API_KEY_LOCAL_MULTIMODAL	EMPTY	Local VLM API key
BASE_URL_LOCAL_MULTIMODAL	http://192.168.0.25:8026/v1	Local VLM base URL
MODEL_LOCAL_MULTIMODAL	Qwen/Qwen2.5-VL-32B-Instruct-AWQ	Local VLM model
API_KEY_ALIBABA_MULTIMODAL	EMPTY	Alibaba VLM API key
BASE_URL_ALIBABA_MULTIMODAL	https://dashscope-intl.aliyuncs.com/compatible-mode/v1	Alibaba VLM base URL
MODEL_ALIBABA_MULTIMODAL	qwen-vl-max	Alibaba VLM model

Integration & Concurrency

Variable	Default	Description
API_URL_SQL	http://192.168.0.25:8105/execute	Database Connect endpoint for Text-to-SQL
API_URL_GATE	http://192.168.0.25:8037/proxy	API Connect endpoint for Text-to-API
VLLM_BASE_URLS	http://192.168.0.27:18011,http://192.168.0.28:18011	Comma-separated VLLM base URLs, polled via /metrics
MAX_VLLM_RUNNING	4	Cap on running VLLM requests before new requests are queued
MAX_VLLM_WAITING	4	Cap on waiting VLLM requests before new requests are queued
MAX_CONCURRENT_SLOTS	2	Number of parallel /chatbot processing slots (threading.Semaphore)
POLL_INTERVAL	1	Polling interval (seconds) while waiting for a slot/queue
TIMEOUT_VLLM	30	Timeout (seconds) when calling VLLM's /metrics
USE_QUEUE	true	Read by queue_manager.py — see the note below

Architecture note: app/queue_manager.py defines its own queue manager but is never imported or used by main.py. The concurrency mechanism actually in effect is threading.Semaphore(MAX_CONCURRENT_SLOTS) plus manual polling of VLLM's /metrics in main.py. This module is likely leftover from a refactor that was never cleaned up.

Endpoints

Method	Path	Description
POST	/chatbot	Main endpoint: processes the question, gathers knowledge, calls the LLM
GET	/health	Health check — verifies Elasticsearch connectivity

POST /chatbot

Main request fields (see ChatbotRequest):

Field	Type	Description
question	string	Required. The user's question
internal	bool	If true, the LLM may answer using general knowledge; if false (default), it's restricted to available data
chain	bool	If true, chat history (room_chat_id) is included as context
room_chat_id	string	Room chat ID, used to fetch history
personalization	string	Custom tone/style instruction; empty = default language-matching behavior
llm	string	Provider: local, local_v2, sambanova, groq, openai, alibaba
base_url_llm / model_llm / api_key_llm	string	Manual provider override (optional)
faq	string	Additional context for FAQ mode
km	array	List of { type, topic_id[] } — knowledge sources to include (basic, ocr, rdbms, api, audio)
stream	bool	If true, the response is an NDJSON stream
source	bool	If true, include raw source data (source_data) in the response
think	bool	If true, appends the /think suffix; default is /no_think
image / image_url	bool / array	Enables image analysis via the multimodal model
files / files_url	bool / array	Enables extraction & Q&A over files from a URL

200 Response (non-stream):

{
  "response": "answer from the LLM",
  "internal": false,
  "chain": false,
  "room_chat_id": "uuid",
  "token_usage": { "prompt_tokens": 123, "completion_tokens": 45, "total_tokens": 168 },
  "source_data": { "basic": [], "ocr": [], "rdbms": [], "api": [], "audio": [] },
  "degraded_sources": ["rdbms"]
}

degraded_sources only appears if a source failed to fetch (not a fatal error — the request still proceeds with whichever sources succeeded).

200 Response (stream), media_type: application/jsonlines, one JSON object per line:

{"type": "message", "data": "chunk of the answer text"}
{"type": "metadata", "data": { "response": "...", "token_usage": {...}, "source_data": {...} }}

Concurrency & VLLM Queue Flow

Knowledge Source Gathering (Parallel)

The five knowledge source types are fetched concurrently via ThreadPoolExecutor(max_workers=5); each is wrapped in _safe_get so that one source failing doesn't fail the whole request.

Source	Function	Elasticsearch Index	Result Size
Basic (topic)	get_data	ES_INDEX_BASIC_KNOWLEDGE	10 documents
Document/OCR	get_data_ocr	ES_INDEX_OCR_KNOWLEDGE + summary indices	3 documents
Audio	get_data_audio	ES_INDEX_AUDIO_KNOWLEDGE + summary indices	10 documents
Database (RDBMS)	get_data_rdbms	ES_INDEX_RDBMS_LIST (via es.get, 1 schema document)	1 connection
External API	get_data_api	ES_INDEX_API_LIST (via es.mget)	matches number of topic_id
Chat history (chain=true)	get_history	ES_INDEX_CHAT_HISTORIES	last 20 messages

Text-to-SQL & Text-to-API

For database and API sources, the raw content is not used directly — Chatbot Service asks the LLM to compose the right query/call first, then executes it:

External API calls (when there is more than one topic_id) are processed in parallel via ThreadPoolExecutor. If any API call fails or returns invalid data, the entire source_data["api"] for that request is cleared (not just the failing endpoint).

System Prompt Assembly

process_latest() in llm_service.py assembles the system prompt from the following components, matching what's described on the Raga Engine page:

Component	Source
Current time	datetime.now()
Internal prompt	"Data-only" mode (default) or "general knowledge allowed" mode (internal=true)
Knowledge source content	Merged from basic + OCR + audio + Text-to-SQL result + Text-to-API result
Chat history	From get_history, cleaned of repeated "no information available" lines
User files	Extraction result of files_url via generate_answer()
Image description	Result of analyze_images() — a separate call to the multimodal model
Personalization	Custom tone instruction or the default (matches the user's language)

The user prompt is built as aggressive_prefix + question + suffix (/think or /no_think, controlling the model's reasoning mode).

File Extraction (files_url)

Type	Extensions
Text	.txt, .md, .log
Document	.pdf, .docx, .xlsx

Files are downloaded temporarily from the URL, their content extracted, then answered via generate_answer() before being merged into the system prompt as files_users.

External Integrations

Service	Env Variable	Description
Database Connect	API_URL_SQL	Executes the SQL query produced by Text-to-SQL
API Connect	API_URL_GATE	Executes the API call produced by Text-to-API
VLLM (local)	VLLM_BASE_URLS	Source of queue metrics for the local/local_v2 providers
Infisical	INFISICAL_HOST	Source of all runtime secrets

Health Check

GET /health only checks Elasticsearch connectivity (es.ping()) — returns 200 ("status": "ok") or 503 ("status": "degraded") if Elasticsearch is unreachable. LLM providers and downstream services (Database Connect, API Connect, VLLM) are not covered by this check.

Build & Run

pip install -r requirements.txt

# Run directly
uvicorn app.run:app --host 0.0.0.0 --port 8080

# Docker
docker build -t chatbot-service .
docker run -p 8080:8080 chatbot-service