​

System Architecture Overview

​

High-Level Architecture

​

Core Components

​

Web Application (apps/web)

• Dashboard: Aggregated brand visibility metrics across all providers
• Prompt Management: Configure and schedule prompts to test brand mentions
• Analysis Views: Detailed sentiment, position, and visibility analysis
• Workspace Management: Multi-tenant workspace configuration
• Settings: Provider configuration, scheduling, and team management

• Next.js 15 with App Router
• tRPC for type-safe API communication
• Better Auth for authentication
• React Query for data fetching
• Tailwind CSS for styling

​

Agent Worker (apps/agent)

1. Pulls a job from the provider-specific BullMQ queue
2. Launches a Playwright browser instance with proxy rotation
3. Navigates to the AI provider (ChatGPT, Claude, etc.)
4. Submits the configured prompts
5. Extracts responses and sources using provider-specific selectors
6. Stores raw responses in ClickHouse
7. Triggers analysis jobs

• Provider-specific browser automation with Playwright
• Warm browser pool for performance optimization
• Proxy rotation with health scoring
• Graceful shutdown with job re-queuing
• Per-provider sequential processing to avoid contention

workers = PROVIDER_LIST.map((provider) => {
  const w = new Worker(getQueueName(provider), handleJob, {
    connection,
    concurrency: workerConcurrency,
    lockDuration: 15 * 60 * 1000, // 15 minutes
    stalledInterval: 60 * 1000,
    maxStalledCount: 5,
  });
  return w;
});

​

Service Layer (packages/services)

• Agent Services (src/agent/):
  • queue.ts: BullMQ queue management per provider
  • jobs.ts: Job submission and progress tracking
  • redis.ts: Redis connection management
• Analysis Services (src/analysis/):
  • runAnalysis.ts: LLM-based response analysis
  • analysis.ts: Analysis result storage and retrieval
• Prompt Services (src/prompt/):
  • Prompt CRUD operations
  • Workspace prompt retrieval
• Workspace Services (src/workspace/):
  • Workspace configuration
  • Provider enablement

​

Database Package (packages/db)

• PostgreSQL: User data, workspaces, prompts, authentication
• ClickHouse: Time-series prompt responses and analysis results
• Drizzle ORM: Type-safe schema definitions and migrations

• auth.ts: Better Auth tables
• workspace.ts: Workspace, prompts, and configuration
• ClickHouse schemas defined in clickhouse-init/

​

Data Flow: Prompt Submission to Analysis

​

1. User Submits Prompt Run

​

2. Agent Worker Processes Job

​

3. Analysis Pipeline

​

4. Dashboard Display

• Time-series metrics: Visibility trends over time
• Provider comparison: Brand mentions across ChatGPT, Claude, etc.
• Sentiment analysis: Positive/negative/neutral scoring
• Position tracking: Where brand appears in responses
• Source analysis: Which sources mention the brand

​

Component Communication

​

Web App → Services

// apps/web/src/app/(auth)/prompts/_lib/mutations/prompt.mutations.ts
const { mutate: runNow } = trpc.agent.runNow.useMutation();

• Composes all domain routers (agent, analysis, prompt, workspace)
• Exports type definitions for client usage
• Integrates middleware for auth, rate limiting, timing

​

Services → Databases

import { db } from "@oneglanse/db";

const prompts = await db
  .select()
  .from(schema.prompts)
  .where(eq(schema.prompts.workspaceId, workspaceId));

import { clickhouse } from "@oneglanse/db";

const result = await clickhouse.query({
  query: "SELECT * FROM prompt_responses WHERE user_id = {userId:String}",
  query_params: { userId },
});

​

Services → Queue

export function getProviderQueue(provider: Provider): Queue {
  let q = queues.get(provider);
  if (!q) {
    q = new Queue(getQueueName(provider), {
      connection,
      defaultJobOptions: DEFAULT_JOB_OPTIONS,
    });
    queues.set(provider, q);
  }
  return q;
}

type ProviderJobData = {
  jobGroupId: string;
  provider: Provider;
  prompts: UserPrompt[];
  user_id: string;
  workspace_id: string;
  created_at?: string;
};

​

Agent → Browser Automation

// apps/agent/src/core/agentHandler.ts:31-58
const warmFactory: AgentFactory = async () => {
  const warm = await getWarmBrowser(provider).catch(() => null);
  if (warm) {
    await navigateWithRetry(warm.page, config.url, {
      waitUntil: "domcontentloaded",
      timeout: 30_000,
    });
    return { browser, context, page, proxy, cleanup };
  }
  return agentFactory();
};

​

Key Architectural Decisions

​

1. Monorepo with Turborepo

• Shared code reuse across apps (types, services, UI components)
• Atomic commits across multiple packages
• Fast incremental builds with caching
• Single dependency installation

• pnpm workspaces for package management
• Turborepo for build orchestration
• Workspace protocol (workspace:*) for internal dependencies

​

2. tRPC for Type-Safe APIs

• End-to-end type safety from server to client
• No code generation required
• Automatic TypeScript inference
• Better DX with autocomplete and error checking

• Rejected due to manual type synchronization overhead

​

3. BullMQ for Job Queue

• Robust job retry and failure handling
• Per-provider queue isolation
• Job progress tracking with Redis
• Graceful shutdown support

• Separate queue per provider to avoid rate limit cascades
• Sequential processing per provider (concurrency: 1)
• 15-minute lock duration for long-running browser jobs

​

4. Playwright for Browser Automation

• Headless browser execution
• Modern async API
• Built-in waiting and retry mechanisms
• Cross-browser support

• Puppeteer: Less active maintenance
• Selenium: Older, more complex API

​

5. Dual Database Strategy

• User accounts, workspaces, prompts
• Strong consistency requirements
• Relational integrity

• High-volume prompt responses (millions of rows)
• Time-series analysis
• Fast aggregation queries
• Column-oriented storage

​

6. Warm Browser Pool

• Reduces job execution time by 3-5 seconds
• Avoids repeated Chrome/CDP startup overhead
• Browser state reset between jobs

• Pre-warmed browsers kept alive between jobs
• Cleanup on navigation failure
• Graceful shutdown closes all warm browsers

​

7. Proxy Rotation with Health Scoring

• Avoids provider rate limits and blocks
• Automatic bad proxy detection
• Maintains high success rates

• Proxy health scores persist across jobs
• Failed attempts decrement score
• Lowest score proxies avoided in future jobs

​

8. Better Auth

• Modern, lightweight authentication
• Framework-agnostic
• Built-in social providers (Google)
• Type-safe session management

• Better Auth chosen for simpler setup and better TypeScript support

​

Performance Optimizations

1. Warm Browser Pool: 3-5s faster job execution
2. Redis Progress Tracking: Real-time job status without database polling
3. Turborepo Caching: Incremental builds skip unchanged packages
4. ClickHouse: Fast analytical queries over millions of responses
5. Concurrent Provider Jobs: All providers run in parallel
6. React Query: Automatic caching and background refetching

​

Scalability Considerations

• Horizontal Agent Scaling: Multiple agent workers can process jobs concurrently
• Provider Queue Isolation: One provider’s failures don’t block others
• ClickHouse Sharding: Can partition by workspace_id for large datasets
• Redis Cluster: Supports clustering for high-throughput job queues
• Next.js Edge Functions: Can deploy API routes to edge for lower latency

​

Security

• Better Auth: Session-based authentication with HTTP-only cookies
• tRPC Middleware: isAuthenticated middleware on protected routes
• Workspace Isolation: All queries filtered by workspace_id
• Rate Limiting: Per-user rate limits on expensive operations
• Internal Endpoints: isInternal middleware for cron jobs
• Environment Variables: Secrets stored in .env, never committed

​

Monitoring and Observability

• Structured Logging: Provider-specific loggers with context
• Job Progress Tracking: Real-time status in Redis
• Error Capture: Domain-specific error classes (@oneglanse/errors)
• Timing Middleware: Request duration tracking
• Graceful Shutdown: Clean worker termination with job re-queuing

​

Related Documentation

• Monorepo Structure - Detailed folder layout
• Tech Stack - Technology choices and versions