v3.0.0

MailGPT

A production-grade metadata-layer graph-update pipeline that derives relationship graphs from email metadata. Designed for schema evolution, idempotent graph updates, and continuous state drift prevention.

Schema EvolutionGraph UpdatesDrift PreventionIntent ExtractionState MachinesCompaction
Deep Dives

Explore the architecture

Overview

From raw metadata to a verified graph

Every email passes through four ordered stages. Each stage is an independent, replacable transformer that emits structured data for the next.

1

Ingest

Pull mail metadata from IMAP, API webhooks, or message queues. Normalize headers, strip attachments, and emit structured metadata events.

2

Process

Extract entities (people, threads, organizations) and relationships via pipeline transformers. Resolve aliases, detect reply chains.

3

Graph Update

Merge extracted entities into the graph. Apply versioned mutations, create typed edges, and update attribute vectors atomically.

4

Validate

Verify graph consistency against source metadata. Recompute checksums, detect drift, and emit reconciliation reports.

Core Concepts

Schema Evolution, Merge Mechanics, Drift Prevention

Versioned Entity Schemas

Every entity type embeds a schema_version field. The read layer reconciles versions at query time through version-aware serializers that fill defaults for missing fields. Schema changes are additive only.

Idempotent Graph Updates

Each mutation carries an idempotencyKey derived from the source message ID. The storage layer deduplicates by key, enabling safe retry after any failure. Entity resolution, attribute merging, and edge reconciliation run in a per-message transaction.

Checksum Drift Detection

Every entity stores a SHA-256 checksum of its serialized state. A background reconciler periodically replays source metadata through the pipeline and compares checksums. Mismatches trigger targeted re-merges and audit events.

graph-updater.ts (core loop)typescript
export class GraphUpdater {
  private stores: EntityStores;
  private reconciler: DriftReconciler;

  async processEmail(event: EmailMetadataEvent): Promise<GraphUpdateResult> {
    const idempotencyKey = `process:${event.messageId}`;
    if (await this.stores.hasBeenApplied(idempotencyKey)) {
      return { status: "duplicate", idempotencyKey };
    }

    const thread = await this.resolveThread(event.threadId, event);
    const sender = await this.resolvePerson(event.from, event);
    const recipients = await Promise.all(event.to.map((e) => this.resolvePerson(e, event)));

    const email: Email = {
      schema_version: 2,
      message_id: event.messageId,
      thread_id: event.threadId,
      subject: event.subject,
      sent_at: event.timestamp,
      received_at: Date.now(),
      from: event.from, to: event.to, cc: event.cc,
      in_reply_to: event.inReplyTo ?? null,
      references: event.references ?? [],
      size_bytes: event.sizeBytes,
      checksum: "",
      created_at: Date.now(),
    };
    email.checksum = computeEntityChecksum(email);

    const edges: EdgeMutation[] = [
      { from: sender.email, to: email.message_id, type: "SENT", properties: { timestamp: event.timestamp } },
      ...recipients.map((r) => ({ from: r.email, to: email.message_id, type: "RECEIVED", properties: { timestamp: event.timestamp } })),
      { from: email.message_id, to: event.threadId, type: "BELONGS_TO", properties: {} },
    ];

    await this.stores.transaction(async (tx) => {
      await tx.storeEntity("email", email);
      await tx.storeEntity("thread", thread);
      await tx.storeEntity("person", sender);
      for (const r of recipients) await tx.storeEntity("person", r);
      for (const edge of edges) await tx.storeEdge(edge);
      await tx.recordIdempotency(idempotencyKey);
    });

    this.reconciler.scheduleCheck("email", email.message_id).catch(console.error);
    return { status: "applied", idempotencyKey, edgesCreated: edges.length };
  }
}
Reference

Entity Schemas

The graph operates on four core entity types. Each is versioned, checksummed, and linked by typed edges.

person.schema.tstypescript
interface Person {
  schema_version: 3;
  email: string;
  name: string;
  domain: string;
  role: "individual" | "team" | "service";
  avatar_url: string | null;
  timezone: string | null;
  aliases: string[];
  version: Record<string, number>;
  checksum: string;
  created_at: number;
  updated_at: number;
}
email.schema.tstypescript
interface Email {
  schema_version: 2;
  message_id: string;
  thread_id: string;
  subject: string;
  sent_at: number;
  received_at: number;
  from: string;
  to: string[];
  cc: string[];
  bcc: string[];
  in_reply_to: string | null;
  references: string[];
  size_bytes: number;
  checksum: string;
  created_at: number;
}
thread.schema.tstypescript
interface Thread {
  schema_version: 2;
  thread_id: string;
  subject: string;
  participants: string[];
  message_count: number;
  first_message_at: number;
  last_message_at: number;
  is_archived: boolean;
  labels: string[];
  checksum: string;
  version: Record<string, number>;
  created_at: number;
  updated_at: number;
}
intent.schema.tstypescript
interface ExtractedIntent {
  schema_version: 1;
  intent_id: string;
  message_id: string;
  type: IntentType;
  confidence: number;
  entities: ExtractedEntity[];
  summary: string;
  action_items: string[];
  deadline: number | null;
  model: string;
  prompt_hash: string;
  checksum: string;
  created_at: number;
}

type IntentType =
  | "meeting_request"
  | "task_assignment"
  | "document_review"
  | "decision_record"
  | "status_update"
  | "follow_up"
  | "informational";

interface ExtractedEntity {
  role: "requestor" | "assignee" | "participant" | "reviewer";
  email: string;
  name: string;
}

MailGPT - Metadata-Layer Graph-Update Pipeline Specification

Designed as a reference architecture. Not a production deployment.