Skip to main content

Why the Suggestions Pattern is Safe for AI

Understanding the design rationale behind AuditSwarm's AI safety architecture.

The Problem​

Traditional approaches to AI-powered applications face a critical challenge:

How do you let AI assist users without risking accidental or malicious data corruption?

If an AI agent can directly modify production data:

  • ❌ Bugs or hallucinations could corrupt compliance records
  • ❌ Users lose visibility into what changed
  • ❌ Compliance auditors can't verify human oversight
  • ❌ No "undo" mechanism for AI mistakes
  • ❌ Difficult to meet SOC2/ISO requirements

The Solution: Suggestions Pattern​

AuditSwarm uses a suggestions-only pattern where AI agents:

  1. ✅ Cannot modify production data directly
  2. ✅ Can create suggestions (stored as JSON)
  3. ✅ Require explicit user approval
  4. ✅ Provide full transparency of changes
  5. ✅ Enable audit trail of all AI actions

How It Works​

AI Agent Request
     │
     ├─→ MCP Server (enforce suggestions-only)
     │
     ├─→ GraphQL API (suggestions.createSuggestion)
     │
     ├─→ AISuggestion table (store as JSON)
     │
     ├─→ User Notification (blue bar)
     │
     └─→ User Approval Required ← HUMAN IN THE LOOP
              │
              ├─→ Approved → Apply to production
              └─→ Rejected → Discard

Why This Pattern?​

1. Compliance Requirements​

SOC2 CC6.1 (Logical Access Controls) requires:

"The entity implements logical access security software, infrastructure, and architectures over protected information assets to protect them from security events to meet the entity's objectives."

Human oversight on AI changes satisfies this control.

2. Data Integrity​

Production audit data must be trustworthy for compliance certifications. The suggestions pattern ensures:

  • All changes are intentional
  • No accidental AI modifications
  • Complete audit trail of who approved what

3. User Trust​

Users trust the system because:

  • They see exactly what will change
  • They maintain full control
  • They can reject bad suggestions
  • They understand AI's role (assistant, not actor)

4. Regulatory Compliance​

GDPR requires human review for automated decision-making (Article 22). The suggestions pattern provides this by default.

Alternative Approaches Considered​

❌ Alternative 1: AI with Full Write Access​

Rejected because:

  • Too risky for compliance data
  • No human oversight
  • Difficult to audit
  • Users lose control

❌ Alternative 2: Read-Only AI​

Rejected because:

  • Severely limits AI usefulness
  • Users still need to do all data entry
  • Doesn't leverage AI capabilities

❌ Alternative 3: Sandboxed "Draft" Mode​

Rejected because:

  • Adds complexity (two versions of truth)
  • Difficult to merge draft → production
  • Users confused about which version is "real"

✅ Alternative 4: Suggestions Pattern (Chosen)​

Why we chose this:

  • ✅ Safe by default (AI can't break things)
  • ✅ Simple mental model (suggestion → approve → done)
  • ✅ Full audit trail
  • ✅ Meets compliance requirements
  • ✅ Users stay in control

Technical Implementation​

Database Design​

Single table for all suggestions:

model AISuggestion {
  id            String   @id @default(cuid())
  userId        String   // Who will approve
  entityType    String   // 'audit', 'risk', 'control', etc.
  entityId      String?  // null for new entities
  operation     String   // 'create', 'update', 'delete'
  suggestedData Json     // The actual changes
  status        String   // 'pending', 'approved', 'rejected'
  createdAt     DateTime @default(now())
}

Enforcement at MCP Layer​

The MCP server only exposes the suggest_change tool:

// src/lib/mcp/tools.ts
export const tools = [
  {
    name: 'suggest_change',
    description: 'Create suggestion (requires approval)',
    // ...
  },
  // NO direct mutation tools!
  // ❌ 'create_audit'
  // ❌ 'update_risk'
  // ❌ 'delete_control'
];

AI agents physically cannot call direct mutation tools.

User Approval Flow​

// src/lib/services/suggestion-approval.service.ts
async approveSuggestion(suggestionId: string, userId: string) {
  const suggestion = await prisma.aiSuggestion.findUnique({
    where: { id: suggestionId }
  });

  // Verify user owns this suggestion
  if (suggestion.userId !== userId) {
    throw new Error('Not authorized');
  }

  // Apply to production based on operation
  if (suggestion.operation === 'create') {
    await prisma[suggestion.entityType].create({
      data: suggestion.suggestedData
    });
  }
  // ... handle update, delete

  // Mark suggestion as applied
  await prisma.aiSuggestion.update({
    where: { id: suggestionId },
    data: { status: 'approved', appliedAt: new Date() }
  });
}

Real-World Example​

User: "Update this audit status to In Progress"

Without Suggestions Pattern:

AI → GraphQL mutation → Status updated
❌ No user visibility
❌ No approval
❌ No audit trail of AI action

With Suggestions Pattern:

AI → suggest_change → Suggestion created
→ User sees notification
→ User reviews: "status: PLANNING → IN_PROGRESS"
→ User clicks "Approve"
→ Status updated with audit trail
✅ User in control
✅ Full transparency
✅ Compliant with SOC2

Benefits​

For Users​

  • ✅ Never surprised by AI changes
  • ✅ Can review before applying
  • ✅ Maintain control over data

For Compliance​

  • ✅ Human oversight required
  • ✅ Complete audit trail
  • ✅ Meets SOC2/ISO requirements

For Developers​

  • ✅ Simple to implement (one table)
  • ✅ Easy to test (mock suggestions)
  • ✅ Scalable (stateless pattern)

Drawbacks & Mitigations​

Drawback: Extra Click Required​

Mitigation: Batch approval for multiple suggestions, keyboard shortcuts

Drawback: Slower Than Direct Mutation​

Mitigation: True, but safety > speed for compliance data

Drawback: AI Can't Take Action Without User​

Mitigation: This is intentional! Humans must remain in the loop.

Future Enhancements​

Possible improvements while maintaining safety:

  1. Auto-approval for low-risk changes (e.g., adding tags)
  2. Batch operations (approve 10 suggestions at once)
  3. Suggestion confidence scores (highlight risky suggestions)
  4. Rollback mechanism (undo approved suggestions)

All while maintaining the core principle: Human approval required.

Conclusion​

The suggestions pattern is the right trade-off for AuditSwarm because:

  • ✅ Safety matches criticality (compliance data is critical)
  • ✅ User trust is paramount
  • ✅ Compliance requirements are non-negotiable
  • ✅ Implementation is simple and maintainable

Result: AI assistance that's both powerful and safe.