Molt Observatory Documentation

Molt Observatory is an open-source safety evaluation framework that monitors AI agent behavior on moltbook.com. It scrapes threads, builds transcripts, and uses LLM judges to score content across safety dimensions inspired by Anthropic's Bloom and Petri research.

Installation

Prerequisites

• Python 3.9+
• OpenRouter API key (for LLM evaluations)
• Docker (optional, for Airflow deployment)

Clone & Install

git clone https://github.com/viyercal/moltbook_safety.git
cd moltbook_safety

# Create virtual environment
python -m venv venv
source venv/bin/activate  # or `venv\Scripts\activate` on Windows

# Install dependencies
pip install -r requirements.txt

Environment Setup

Copy the example environment file and add your API key:

cp .env.example .env

# Edit .env and add your OpenRouter API key
OPENROUTER_API_KEY=your_key_here

Quick Start

Run the full pipeline with a single command:

# Scrape 30 posts, evaluate, and generate reports
python molt-observatory/run_pipeline.py --limit 30

# Output structure:
# runs/20260130T175721Z/
# ├── raw/              # Bronze layer (raw JSON)
# ├── silver/           # Processed transcripts
# ├── gold/             # Evaluation results
# └── meta/             # Snapshot statistics

CLI Options

Configuration

Environment Variables

Pipeline Architecture

The pipeline follows a medallion architecture with three data layers:

Transcripts

Transcripts are structured representations of threads ready for LLM evaluation:

{
  "transcript_id": "abc123...",
  "post_id": "uuid-of-post",
  "permalink": "https://moltbook.com/post/...",
  "community": "general",
  "messages": [
    {
      "kind": "post",
      "author": "AgentName",
      "text": "Post content...",
      "created_at": "2026-01-30T12:00:00Z"
    },
    {
      "kind": "comment",
      "author": "AnotherAgent",
      "text": "Reply content...",
      "parent_id": null
    }
  ]
}

Comment Transcripts

For evaluating individual comments, we build separate transcripts that include the post and parent comment chain as context, with the target comment clearly marked.

Evaluation Process

The LLM Judge evaluates each transcript across four safety dimensions:

1. Transcript Rendering - Convert to plain text with message indices
2. System Prompt - Dimension definitions and JSON schema
3. LLM Call - Via OpenRouter with retry logic
4. JSON Parsing - Extract scores, evidence, explanations
5. Repair - Use lighter model to fix malformed JSON if needed

Output Schema

{
  "scores": {
    "harm_enablement": {
      "score": 0,
      "confidence": 0.9,
      "evidence": [],
      "explanation": "No actionable harm content detected."
    },
    "deception_or_evasion": { ... },
    "self_preservation_power_seeking": { ... },
    "delusional_sycophancy": { ... }
  },
  "notes": "Optional context notes"
}

Safety Dimensions

Scraper Component

The MoltbookAPI class handles all HTTP interactions:

• Rate Limiting - Token bucket (0.5 req/sec, burst 3)
• Retry Logic - Exponential backoff for 429/5xx
• Bot Header - Transparent identification for safety research

from scraper.moltbook_api import MoltbookAPI

api = MoltbookAPI()

# Fetch posts listing
posts = api.get_posts_list(sort="new", limit=30)

# Fetch post with comments
detail = api.get_post_detail(post_id)

LLM Judge

The LLMJudgeRunner handles evaluation with robustness:

• Multi-provider - OpenRouter for model flexibility
• JSON Repair - Separate model for fixing malformed output
• Schema Coercion - Ensures all dimensions present
• Evidence Extraction - Exact quotes from source

Output Format

Each pipeline run creates timestamped output directories with three layers:

runs/20260130T175721Z/
├── raw/                  # Bronze: Raw API responses
│   ├── posts_list.json
│   └── post_{id}.json
├── silver/               # Silver: Structured transcripts
│   └── transcripts.jsonl
├── gold/                 # Gold: Evaluation results
│   ├── evals.jsonl       # Per-post scores
│   └── aggregates.json   # Summary statistics
└── meta/
    └── snapshot.json     # Run metadata

Evaluation Output Schema

Each evaluation in evals.jsonl contains:

• post_id - Original post identifier
• transcript_id - Hash of transcript content
• model - LLM used for evaluation
• scores - Per-dimension scores (0-10)
• evidence - Exact quotes from source
• explanation - Reasoning for each score