AI Video Analysis Agent
The state-of-the-art AI video analysis agent.
Upload any video, ask anything. Gemini 2.5 Flash for timestamped structure, Whisper for voice, on React 19 and tRPC 11. Neural Noir design.
- TypeScript 5.9
- React 19
- Vite 7
- tRPC 11
- Drizzle ORM
- MIT License
01 / Why Ellie
Understand media without scrubbing timelines by hand.
Ellie is a video intelligence agent for people who need to understand media without scrubbing timelines by hand. Upload a video, let Gemini 2.5 Flash produce timestamped structure, then ask natural-language questions over the generated context. The experience is deliberately Neural Noir: dark, cinematic, fast, and agentic without hiding the engineering underneath.
| Product Box | What It Does | Current Implementation |
|---|---|---|
| Video Intelligence | Converts media into summaries, scenes, frames, audio events, and emotions. | analysis.analyze calls Gemini 2.5 Flash through the LLM gateway. |
| Ask Anything Chat | Answers questions against video analysis with timestamp-aware responses. | chat.send uses browser-held history plus persisted analysis context. |
| Voice Input | Turns recorded audio into text before it enters the assistant surface. | voice.transcribe calls Whisper-compatible speech-to-text. |
| Typed Product Core | Keeps client and server contracts aligned as the agent surface evolves. | React 19 consumes an Express/tRPC 11 backend with shared TypeScript. |
| Deployment Baseline | Ships with validation scripts, Docker packaging, and health probes. | pnpm ci, Dockerfile, /api/health, /api/readiness, /api/ready. |
02 / Capabilities
Six surfaces. One agent.
The README is product-forward, but the repository is not a mockup. The core routes implement upload, multimodal analysis, chat, transcription, session helpers, and operational readiness endpoints.
| Surface | Route or Module | Limits and Behavior |
|---|---|---|
| Upload | video.upload | Accepts MP4, WebM, MOV, and AVI as base64 payloads up to 100 MB. |
| Analyze | analysis.analyze | Produces structured JSON results with type, timestamp, content, and confidence. |
| Retrieve Results | analysis.results | Reads persisted analysis rows for a video from the Drizzle/MySQL data layer. |
| Chat | chat.send | Answers against stored analysis context with ephemeral browser-side chat history. |
| Transcribe | voice.transcribe | Accepts WebM, MP3, WAV, OGG, M4A, and MPEG audio up to 16 MB via Whisper. |
| System | system.* | Exposes health and readiness signals for smoke tests and deployment probes. |
03 / Architecture
Heavy client. Thin orchestration boundary.
Ellie uses a heavy-client, thin-orchestration backend model. The browser owns the immediate product feel and ephemeral conversation memory. The backend owns the safe boundary for uploads, storage, provider calls, persistence, rate limits, and readiness checks.
request flow
- browser
- trpc
- express
- gateway
- gemini
- drizzle
| Boundary | Responsibility | Why It Exists |
|---|---|---|
| Client Agent UI | Upload flow, video workspace, chat state, visual feedback. | Preserves speed and keeps short-lived interaction state near the user. |
| tRPC Contract | Typed procedure calls between browser and server. | Reduces drift between frontend assumptions and backend behavior. |
| Express Runtime | Provider calls, storage writes, DB updates, rate limits, sessions. | Keeps secrets and operational state out of the browser. |
| Data Plane | Users, videos, analysis results, conversations, and messages. | Gives the product durable entities without overbuilding the first agent milestone. |
| Provider Gateway | Gemini 2.5 Flash chat/completions and Whisper transcription. | Centralizes AI access behind environment-managed credentials. |
| Operations Surface | Health, readiness, production validation, Docker packaging. | Makes the repository deployable and reviewable by maintainers. |
04 / Stack
TypeScript across the product boundary.
Modern, but intentionally familiar. React and Vite for the browser, Express and tRPC for the API, Drizzle and MySQL for persistence, Forge-compatible helpers for Gemini and Whisper.
| Layer | Technology | Version or Track |
|---|---|---|
| Language | TypeScript | 5.9.3 |
| Frontend | React, React DOM, Vite | 19.2.x, 7.x |
| Design System | Tailwind CSS, Radix UI, framer-motion | 4.x, Radix primitives, 12.x |
| Routing + Data | wouter, TanStack Query, tRPC React Query | 3.x, 5.x, 11.6.x |
| Backend Runtime | Node.js, Express | Express 4.21.x |
| API Contract | tRPC server/client | 11.6.x |
| Database | Drizzle ORM, mysql2 | 0.44.x, 3.15.x |
| Validation | Zod | 4.1.x |
| AI + Voice | Gemini 2.5 Flash, Whisper-compatible STT | Provider-gateway configured |
| Testing + Quality | Vitest, Prettier, TypeScript compiler | 2.1.x, 3.6.x, 5.9.3 |
| Packaging | Docker, Docker Compose | Repository-defined |
| Package Manager | pnpm | 10.15.1 |
05 / Run Locally
One pnpm workflow. Six commands to a dev server.
Configure environment values first, install with the lockfile, then run the TypeScript server in development mode.
01$ gh repo clone Alexi5000/Ellie02$ cd Ellie03$ pnpm install --frozen-lockfile04$ cp .env.example .env05$ pnpm validate:env06$ pnpm dev| Command | Purpose |
|---|---|
| pnpm dev | Development server with TypeScript watch. |
| pnpm check | TypeScript no-emit gate. |
| pnpm test | Runs the Vitest suite. |
| pnpm build | Vite client and bundled server runtime. |
| pnpm validate:env | Checks required local environment configuration. |
| pnpm validate:env:production | Checks production-only environment requirements. |
| pnpm ci | Runs the full repository validation sequence. |
| pnpm start | Runs the built production server from dist/. |
06 / Operations
Prepared for review, not just screenshots.
Ellie ships with a root Dockerfile, a Compose file, CI-compatible scripts, environment validation, and liveness/readiness endpoints. Some GitHub workflow checks depend on repository-level secrets and permissions, so local validation remains the authoritative green gate until those secrets are configured.
| Gate | Command or Endpoint | Expected Signal |
|---|---|---|
| Install | pnpm install --frozen-lockfile | Lockfile-consistent dependency graph. |
| Environment | pnpm validate:env | Required local variables are accounted for. |
| Production Env | pnpm validate:env:production | Production-only variables are accounted for. |
| Types | pnpm check | TypeScript passes without emitting files. |
| Tests | pnpm test | Vitest suite completes. |
| Build | pnpm build | Vite client and server bundle are generated. |
| Full Local Gate | pnpm ci | Environment, types, tests, and build pass together. |
| Health | GET /api/health | Runtime liveness response. |
| Readiness | GET /api/readiness | /api/ready | Dependency-aware deployment probe. |
07 / Roadmap
The next step is deeper production orchestration.
Ellie's current shape is intentionally lean: strong browser experience, typed backend orchestration, persisted analysis, and real provider calls.
- 01 / Trackplanned
Durable Conversations
Persist chat threads and add source-linked answer citations.
- 02 / Trackplanned
Processing Queue
Move long-running video analysis into workers with retry telemetry.
- 03 / Trackplanned
Signed Uploads
Replace the base64 upload flow with direct-to-storage signed URLs.
- 04 / Trackplanned
Provider Observability
Track model, prompt version, latency, token usage, and failure mode.
- 05 / Trackplanned
Tenant Readiness
Add organizations, roles, audit logs, and retention policies.
- 06 / Trackplanned
Evaluation Harness
Golden-video tests for scene, transcript, frame, and chat quality.