Structured Extraction

LLM-Guided Structured Intent Extraction

Extract actionable intents from email metadata using constrained LLM generation. Each extracted intent is typed, confidence-scored, checksummed, and committed to the graph as a first-class entity.

Extraction Pipeline

From email event to typed intent

1

Raw Email

Metadata payload from the ingest stage: headers, routing, thread context, timestamps

2

Prompt Assembly

Construct a structured extraction prompt with the email metadata and a JSON schema describing the expected output

3

LLM Inference

Call the model with constrained decoding or tool-use to produce a valid structured intent object

4

Validate & Commit

Validate the extracted intent against the schema, assign a confidence score, and commit to the graph

Schema

Intent Type System

Every extracted intent conforms to a typed discriminated union. The schema is itself versioned, allowing new intent types to be added without breaking existing extractions.

intent-types.tstypescript
// Discriminated union of all supported intent types.
// Adding a new type requires only extending this union
// and adding a template to the prompt registry.

export type IntentType =
  | "meeting_request"
  | "task_assignment"
  | "document_review"
  | "decision_record"
  | "status_update"
  | "follow_up"
  | "informational";

export type IntentSchema = {
  intent_id: string;
  message_id: string;
  thread_id: string;
  type: IntentType;
  confidence: number;  // 0.0 - 1.0
  summary: string;
  action_items: string[];
  deadline: number | null;
  participants: ExtractedParticipant[];
  entities: Record<string, string[]>;
};

export type ExtractedParticipant = {
  role: "requestor" | "assignee" | "participant" | "reviewer";
  email: string;
  name: string;
};

// Per-type metadata constraints
export const INTENT_METADATA: Record<IntentType, {
  requiredParticipants: ExtractedParticipant["role"][];
  allowsDeadline: boolean;
  maxActionItems: number;
}> = {
  meeting_request: {
    requiredParticipants: ["requestor", "participant"],
    allowsDeadline: true,
    maxActionItems: 3,
  },
  task_assignment: {
    requiredParticipants: ["requestor", "assignee"],
    allowsDeadline: true,
    maxActionItems: 10,
  },
  document_review: {
    requiredParticipants: ["requestor", "reviewer"],
    allowsDeadline: true,
    maxActionItems: 5,
  },
  decision_record: {
    requiredParticipants: ["participant"],
    allowsDeadline: false,
    maxActionItems: 5,
  },
  status_update: {
    requiredParticipants: ["participant"],
    allowsDeadline: false,
    maxActionItems: 3,
  },
  follow_up: {
    requiredParticipants: ["requestor"],
    allowsDeadline: true,
    maxActionItems: 5,
  },
  informational: {
    requiredParticipants: [],
    allowsDeadline: false,
    maxActionItems: 0,
  },
};
extraction-prompt.tstypescript
// Prompt template for LLM-based structured extraction.
// Uses a system prompt + few-shot examples + the email metadata.

export function buildExtractionPrompt(email: EmailMetadata): string {
  return `Extract structured intent from this email metadata.
Return ONLY valid JSON matching the schema below.

SUPPORTED INTENT TYPES:
- meeting_request: Scheduling a meeting or call
- task_assignment: Delegating work with explicit ownership
- document_review: Requesting feedback on a document
- decision_record: Recording a decision made
- status_update: Providing progress information
- follow_up: Checking in on prior commitments
- informational: General update with no action required

EMAIL METADATA:
Subject: ${email.subject}
From: ${email.from}
To: ${email.to.join(", ")}
CC: ${email.cc.join(", ")}
In-Reply-To: ${email.inReplyTo ?? "none"}
Timestamp: ${new Date(email.timestamp).toISOString()}

Return JSON:
{
  "type": "<intent_type>",
  "confidence": <0.0-1.0>,
  "summary": "<one-sentence summary>",
  "action_items": ["<item>", ...],
  "deadline": "<ISO date or null>",
  "participants": [
    {
      "role": "<requestor|assignee|participant|reviewer>",
      "email": "<email>",
      "name": "<name>"
    }
  ],
  "entities": {
    "<entity_type>": ["<value>", ...]
  }
}

Return ONLY the JSON object, no other text.`;
}
Validation

Confidence Scoring & Fallback

Not every extraction is equally reliable. The pipeline assigns a confidence score and applies type-specific validation rules. Low-confidence extractions are flagged for manual review or re-extraction with a more detailed prompt.

confidence-scorer.tstypescript
// Confidence scoring combines model-reported confidence with
// structural validation to produce a final reliability score.

export type ConfidenceLevel = "high" | "medium" | "low";

export interface ConfidenceResult {
  score: number;
  level: ConfidenceLevel;
  signals: string[];
}

export function scoreConfidence(
  extracted: IntentSchema,
  modelConfidence: number,
): ConfidenceResult {
  const signals: string[] = [];
  let score = modelConfidence;

  // 1. Participant completeness
  const meta = INTENT_METADATA[extracted.type];
  const hasRequiredRoles = meta.requiredParticipants.every((role) =>
    extracted.participants.some((p) => p.role === role),
  );
  if (!hasRequiredRoles) {
    score *= 0.6;
    signals.push("missing_required_participants");
  } else {
    score = Math.min(1, score * 1.1);
    signals.push("participants_complete");
  }

  // 2. Action item count check
  if (extracted.action_items.length > meta.maxActionItems) {
    score *= 0.85;
    signals.push("excessive_action_items");
  }

  // 3. Deadline consistency
  if (meta.allowsDeadline && !extracted.deadline) {
    score *= 0.9;
    signals.push("missing_expected_deadline");
  } else if (!meta.allowsDeadline && extracted.deadline) {
    score *= 0.8;
    signals.push("unexpected_deadline");
  }

  // 4. Summary length heuristic
  const wordCount = extracted.summary.split(/\s+/).length;
  if (wordCount < 3 || wordCount > 40) {
    score *= 0.85;
    signals.push("summary_length_outside_range");
  }

  const clamped = Math.max(0, Math.min(1, score));
  const level: ConfidenceLevel =
    clamped >= 0.8 ? "high"
    : clamped >= 0.5 ? "medium"
    : "low";

  return { score: clamped, level, signals };
}

// Low-confidence intents enter a re-extraction loop with
// a more explicit prompt and stricter constraints.
export async function extractWithFallback(
  email: EmailMetadata,
  maxRetries = 2,
): Promise<{ intent: IntentSchema; confidence: ConfidenceResult }> {
  for (let attempt = 0; attempt <= maxRetries; attempt++) {
    const prompt = attempt === 0
      ? buildExtractionPrompt(email)
      : buildRetryPrompt(email, attempt);

    const raw = await callLLM(prompt);
    const parsed = JSON.parse(raw) as IntentSchema;
    const confidence = scoreConfidence(parsed, parsed.confidence);

    if (confidence.level !== "low" || attempt === maxRetries) {
      return { intent: parsed, confidence };
    }
  }
  throw new Error("unreachable");
}

Confidence Signal Sources

  • Model confidence - Raw logit-based confidence from the LLM output
  • Participant coverage - Are all required participant roles present?
  • Action item hygiene - Count within per-type limits
  • Deadline consistency - Expected deadline present and valid
  • Summary quality - Length and specificity heuristics
  • Schema validation - Field types and constraints match the schema

Low-confidence fallback

When confidence falls below 0.5, the pipeline enters a re-extraction loop with an expanded prompt that includes the full email thread context and explicit type-specific examples. If the second pass also scores low, the intent is marked needs_review and surfaced in the dashboard for manual resolution.

Implementation

Intent Extractor

The extractor orchestrates prompt assembly, LLM inference, confidence scoring, and graph commitment in a single idempotent pipeline stage.

intent-extractor.tstypescript
// Full intent extraction pipeline stage. Plugs into the main
// GraphUpdater as a transformer on the Process stage.

export interface IntentExtractorConfig {
  model: string;
  maxRetries: number;
  confidenceThreshold: number;
  promptVersion: string;
}

export class IntentExtractor {
  private config: IntentExtractorConfig;

  constructor(config: IntentExtractorConfig) {
    this.config = config;
  }

  async extract(
    email: EmailMetadata,
    threadContext?: Thread,
  ): Promise<ExtractionResult> {
    const extractionId = `extract:${email.messageId}`;

    // 1. Build the prompt with optional thread context
    const prompt = threadContext
      ? buildContextAwarePrompt(email, threadContext)
      : buildExtractionPrompt(email);

    // 2. Call the LLM with structured output constraints
    const startTime = performance.now();
    const raw = await this.callModel(prompt);
    const latency = performance.now() - startTime;

    // 3. Parse and validate
    let parsed: IntentSchema;
    try {
      parsed = this.validateAndParse(raw);
    } catch (err) {
      return {
        status: "parse_error",
        extractionId,
        error: (err as Error).message,
        latency,
      };
    }

    // 4. Score confidence
    const confidence = scoreConfidence(parsed, parsed.confidence);

    if (confidence.level === "low" && this.config.maxRetries > 0) {
      return this.retryWithExpandedContext(email, threadContext, extractionId, latency);
    }

    // 5. Commit the intent to the graph
    const intent: StoredIntent = {
      schema_version: 1,
      intent_id: extractionId,
      message_id: email.messageId,
      type: parsed.type,
      confidence: confidence.score,
      summary: parsed.summary,
      action_items: parsed.action_items,
      deadline: parsed.deadline,
      participants: parsed.participants,
      model: this.config.model,
      prompt_version: this.config.promptVersion,
      latency_ms: Math.round(latency),
      checksum: "",
      created_at: Date.now(),
    };
    intent.checksum = computeEntityChecksum(intent);

    return {
      status: "success",
      extractionId,
      intent,
      confidence,
      latency,
    };
  }

  private async callModel(prompt: string): Promise<string> {
    // In production, this calls the configured LLM provider
    // with constrained decoding / response_format = "json_object".
    // For the reference architecture, the interface is:
    //
    // POST /v1/chat/completions
    // {
    //   model: this.config.model,
    //   messages: [{ role: "user", content: prompt }],
    //   response_format: { type: "json_object" },
    //   temperature: 0.1
    // }

    const response = await fetch(process.env.LLM_ENDPOINT!, {
      method: "POST",
      headers: {
        "Content-Type": "application/json",
        Authorization: `Bearer ${process.env.LLM_API_KEY}`,
      },
      body: JSON.stringify({
        model: this.config.model,
        messages: [{ role: "user", content: prompt }],
        response_format: { type: "json_object" },
        temperature: 0.1,
        max_tokens: 1024,
      }),
    });

    if (!response.ok) {
      throw new Error(`LLM call failed: ${response.status} ${response.statusText}`);
    }

    const data = await response.json();
    return data.choices[0].message.content;
  }

  private validateAndParse(raw: string): IntentSchema {
    const parsed = JSON.parse(raw);

    if (!parsed.type || !INTENT_METADATA[parsed.type]) {
      throw new Error(`Unknown intent type: ${parsed.type}`);
    }

    if (typeof parsed.confidence !== "number") {
      throw new Error("Missing confidence score");
    }

    if (!Array.isArray(parsed.action_items)) {
      throw new Error("Missing action_items array");
    }

    return parsed as IntentSchema;
  }

  private async retryWithExpandedContext(
    email: EmailMetadata,
    threadContext: Thread | undefined,
    extractionId: string,
    previousLatency: number,
  ): Promise<ExtractionResult> {
    const contextPrompt = buildContextAwarePrompt(
      email,
      threadContext,
      { includeExamples: true, strictMode: true },
    );

    const raw = await this.callModel(contextPrompt);
    const parsed = this.validateAndParse(raw);
    const confidence = scoreConfidence(parsed, parsed.confidence);

    return {
      status: "success",
      extractionId,
      intent: {
        schema_version: 1,
        intent_id: extractionId,
        message_id: email.messageId,
        type: parsed.type,
        confidence: confidence.score,
        summary: parsed.summary,
        action_items: parsed.action_items,
        deadline: parsed.deadline,
        participants: parsed.participants,
        model: this.config.model,
        prompt_version: `${this.config.promptVersion}-retry`,
        latency_ms: Math.round(previousLatency + 0),
        checksum: "",
        created_at: Date.now(),
      },
      confidence,
      latency: previousLatency,
    };
  }
}

export type ExtractionResult =
  | { status: "success"; extractionId: string; intent: StoredIntent; confidence: ConfidenceResult; latency: number }
  | { status: "parse_error"; extractionId: string; error: string; latency: number };
Prompt Engineering

Prompt Strategies

Different email contexts call for different prompt structures. The registry holds specialized prompt templates keyed by intent type and extraction phase.

prompt-registry.tstypescript
// Central registry of prompt templates, keyed by intent type
// and extraction phase (initial vs retry).

type PromptPhase = "initial" | "retry" | "context_aware";

const PROMPT_REGISTRY: Record<
  IntentType,
  Record<PromptPhase, (email: EmailMetadata, ctx?: Thread) => string>
> = {
  meeting_request: {
    initial: (email) => `Extract a meeting request from this email.
Subject: ${email.subject}
Body indicators: "let's meet", "schedule", "calendar", "free for"
Return JSON with type "meeting_request", proposed participants, and suggested times.`,
    retry: (email) => `[STRICT] Extract ONLY a meeting request.
The email subject "${email.subject}" strongly suggests scheduling.
Identify: requestor, proposed attendees, time references.
Return valid JSON only.`,
    context_aware: (email, ctx) => `Thread context: ${ctx?.message_count ?? 1} messages,
last activity: ${ctx?.last_message_at ?? "unknown"}.
Extract meeting request intent from the latest message.
Subject: "${email.subject}"`,
  },
  task_assignment: {
    initial: (email) => `Extract a task assignment from this email.
Owner keywords: "please", "can you", "your responsibility"
Deadline indicators: "by EOD", "by Friday", "ASAP"
Return JSON with assignee, task description, and deadline.`,
    retry: (email) => `[STRICT] Extract ONLY a task assignment.
Identify: who is responsible, what needs to be done, by when.
Return valid JSON only.`,
    context_aware: (email, ctx) => {
      const deadline = extractDeadlineFromContext(email, ctx);
      return `Extract task assignment. Inferred deadline: ${deadline ?? "none detected"}.
Subject: "${email.subject}"`;
    },
  },
  document_review: {
    initial: (email) => `Extract a document review request.
Keywords: "review", "feedback", "thoughts", "look at"
Return JSON with document reference, reviewer, and requested feedback type.`,
    retry: (email) => `[STRICT] Extract ONLY a document review request.
Identify reviewer, document, and feedback scope. Return JSON only.`,
    context_aware: (email, ctx) => `Review request in thread (${ctx?.message_count} messages).
Latest: "${email.subject}". Extract reviewer and document ref.`,
  },
  decision_record: {
    initial: (email) => `Extract a decision record.
Keywords: "decided", "decision", "going with", "conclusion"
Return JSON with the decision, rationale, and deciders.`,
    retry: (email) => `[STRICT] Extract ONLY a decision. Identify the decision, who made it, and why. JSON only.`,
    context_aware: (email, ctx) => `Decision in thread context. Extract the conclusion and participants.`,
  },
  status_update: {
    initial: (email) => `Extract a status update.
Keywords: "update", "progress", "status", "currently"
Return JSON with what changed and current state.`,
    retry: (email) => `[STRICT] Extract ONLY a status update. What changed, current state. JSON only.`,
    context_aware: (email, ctx) => `Status update in ongoing thread. Extract progress delta.`,
  },
  follow_up: {
    initial: (email) => `Extract a follow-up request.
Keywords: "following up", "checking in", "any update", "bumping"
Return JSON with the original context and expected response.`,
    retry: (email) => `[STRICT] Extract ONLY a follow-up. What is the prior context, what response is expected. JSON only.`,
    context_aware: (email, ctx) => `Follow-up in thread. Prior context: "${ctx?.subject}". Extract expected response.`,
  },
  informational: {
    initial: (email) => `Extract informational content.
No action keywords detected. Summarize the information shared.
Return JSON with summary only, empty action_items.`,
    retry: (email) => `[STRICT] Informational only. Short summary, empty action items. JSON only.`,
    context_aware: (email, ctx) => `Informational in thread. Summarize the update concisely.`,
  },
};
prompt-versioning.tstypescript
// Every extraction records which prompt version was used,
// enabling A/B testing and regression tracking.

const PROMPT_VERSION = "v3.2.0";

interface PromptVersion {
  version: string;
  template: string;
  features: string[];
  deployedAt: number;
}

const promptHistory: PromptVersion[] = [
  {
    version: "v1.0.0",
    template: "single-shot, no few-shot examples",
    features: ["basic extraction"],
    deployedAt: 1700000000000,
  },
  {
    version: "v2.0.0",
    template: "few-shot with 3 examples per type",
    features: ["few-shot learning", "type hints"],
    deployedAt: 1705000000000,
  },
  {
    version: "v3.0.0",
    template: "context-aware with thread history",
    features: ["thread context", "deadline inference", "role extraction"],
    deployedAt: 1710000000000,
  },
  {
    version: "v3.1.0",
    template: "strict JSON mode with retry logic",
    features: ["response_format enforcement", "automatic retry"],
    deployedAt: 1712000000000,
  },
  {
    version: "v3.2.0",
    template: "confidence-calibrated, per-type constraints",
    features: ["confidence scoring", "per-type validation", "signal logging"],
    deployedAt: 1714000000000,
  },
];

// Track extraction quality per version for regression monitoring
export async function recordExtractionMetrics(
  version: string,
  result: ExtractionResult,
): Promise<void> {
  if (result.status !== "success") return;

  await metricsClient.increment("extraction.total", {
    version,
    type: result.intent.type,
    confidence: result.confidence.level,
  });

  await metricsClient.histogram("extraction.confidence", result.confidence.score, {
    version,
    type: result.intent.type,
  });

  await metricsClient.histogram("extraction.latency_ms", result.latency, {
    version,
    model: result.intent.model,
  });
}

MailGPT - Structured Intent Extraction Specification

LLM-guided extraction pipeline. Prompt templates, confidence scoring, fallback strategies.