Core Capabilities & Features
| Feature | Implementation & Description |
|---|---|
| 🎙️ Voice-First Interface | Real-time speech-to-text with Deepgram Nova-2 + ultra-expressive streaming text-to-speech with Cartesia Sonic over WebRTC via LiveKit Cloud. End-to-end latency kept under 650ms. |
| 🤖 AI Video Avatar | Tavus-powered photorealistic doctor avatar responds directly in video with synchronized lip motion, expressions, and human-like conversational presence. |
| 📅 Calendar Booking | Interactive visual calendar UI enabling patients to inspect available specialist slots, select appointment times, and confirm reservations hands-free or via click. |
| 🔁 Session Memory | Identifies callers by phone number. Returning users immediately receive personalized greetings, access their medical history, and continue uninterrupted care. |
| 📝 Auto Call Summary | Structured clinical summary generated dynamically after each session, logging symptoms, booked slots, patient preferences, and clinical timestamps. |
| 🛠️ Live Tool Feedback | Visual on-screen pulse indicators display agent tool states in real time: "Fetching available slots…", "Verifying schedule…", "Booking confirmed ✅". |
| 💬 Dual Transcript | Color-coded, real-time synchronized chat bubbles (caller on right in electric blue, agent on left in sleek charcoal glass) with auto-scroll. |
System Architecture & Data Pipeline
CareVoice AI coordinates a distributed real-time pipeline between a modern TypeScript frontend, a Python async token service, LiveKit's global WebRTC mesh, an autonomous Python Agent Worker, and a local persistence store.
┌──────────────────────┐ HTTPS ┌─────────────────────────┐
│ React Frontend │ ──────────────────────► │ Python Token Server │
│ (Vite + TS) │ │ (aiohttp :8080) │
└──────────────────────┘ └──────────┬──────────────┘
│ │
│ WebRTC (LiveKit SDK) LiveKit Cloud
│ │
└─────────────────────────────────────────────────► │
┌──────────▼──────────────┐
│ Python Agent Worker │
│ (LiveKit Agents 1.x) │
│ │
│ STT: Deepgram │
│ LLM: OpenAI GPT-4o │
│ TTS: Cartesia │
│ Avatar: Tavus │
└──────────┬──────────────┘
│ SQLite
┌──────────▼──────────────┐
│ voice_agent.db │
│ users · appointments │
│ time_slots · summaries │
└─────────────────────────┘1. Frontend Client (Vite + React + TS)
Provides zero-latency WebRTC media negotiation via the LiveKit Client SDK. Renders an adaptive UI with live audio visualizer waveforms, calendar slot picker, and interactive transcript.
2. Authentication & Token Server
An asynchronous Python microservice built on aiohttp generating cryptographically signed JWT tokens with granular room permissions and ephemeral identity management.
3. Autonomous Agent Worker
Built with LiveKit Agents 1.x, managing bidirectional audio streaming pipes. Performs Voice Activity Detection (VAD), calls tool functions for calendar manipulation, and orchestrates Tavus avatar video frames.
4. Relational State Store
A SQLite persistent store preserving callers, structured appointment records, provider schedules, and GPT-synthesized consultation notes with audit timestamps.
Engineering Highlights & Low Latency Optimizations
Sub-Second Voice Turnaround
Traditional voice agents suffer 2–3 second conversational lag. By streaming audio directly through Deepgram's websocket STT into GPT-4o's token streaming pipeline, chunked directly into Cartesia's neural audio engine, conversational interruption and turn-taking feel instantaneous.
Deterministic Function Calling
Engineered rigorous JSON function schemas for doctor schedule checking, patient identity verification, and appointment booking, preventing hallucinations while keeping responses empathetic and medical-compliant.
Audio-Visual Sync with Tavus
Synchronized streaming audio tracks with Tavus photorealistic avatar video tracks over WebRTC channels without desync or stutter under fluctuating network conditions.