Self-host the whole prompt & agent layer.

Install, generate a key, and create your first prompt.

PromptMetrics ships as a single Node binary. Pick a path, set API_KEY_SALT, and you're running.

# 1. Install + start
$ npm install -g promptmetrics
$ export API_KEY_SALT=$(openssl rand -hex 32)
$ promptmetrics-server                    # listening on :3000

# 2. Mint a key (workspace=default, scopes=read,write)
$ node $(npm root -g)/promptmetrics/dist/scripts/generate-api-key.js \
    --workspace default read,write
# → pm_a91f0b3c…  (store this once)

# 3. Create a prompt
$ promptmetrics create-prompt --file welcome.json
✓ welcome · v1.0.0 · committed 7a3fe2c

Two headers. Three scopes.

Every endpoint except /health requires X-API-Key. Workspaces are scoped via X-Workspace-Id. Keys are HMAC-SHA256 hashed with a server-side salt — only the hash is stored.

curl http://localhost:3000/v1/prompts/welcome \
  -H "X-API-Key: pm_xxxxxxxxxxxxxxxx" \
  -H "X-Workspace-Id: eu-prod"

Git-backed prompt storage.

Each version is an immutable Git tag (prompts/{name}/{version}) backed by a JSON blob. SQLite indexes name → version_tag → commit_sha for sub-millisecond reads.

{
  "name": "welcome",
  "version": "1.0.0",
  "messages": [
    { "role": "system", "content": "You are a helpful assistant." },
    { "role": "user",   "content": "Hello {{name}}!" }
  ],
  "variables": { "name": { "type": "string", "required": true } },
  "model_config": { "model": "gpt-4o", "temperature": 0.7 },
  "tags": ["greeting"]
}

First-class agent telemetry.

Track agent loops with trace_id → spans. Spans can nest via parent_id. Status is ok or error. metadata accepts up to 50 top-level keys with nested objects and arrays.

{
  "trace_id": "550e8400-e29b-41d4-a716-446655440000",
  "prompt_name": "welcome",
  "version_tag": "1.0.0",
  "metadata": { "agent": "headline-agent", "loop": 1 }
}

{
  "name": "llm-call",
  "status": "ok",
  "start_time": 1000,
  "end_time": 2500,
  "metadata": { "model": "gpt-4o", "tokens_in": 120, "tokens_out": 340 }
}

High-level outcome → low-level steps.

Runs track end-to-end workflow executions. A run can optionally link to a trace_id, letting you drill from "this workflow failed" to "this span errored." Status: running · completed · failed.

{
  "workflow_name": "headline-generator",
  "input":  { "topic": "AI" },
  "trace_id": "550e8400-e29b-41d4-a716-446655440001",
  "metadata": { "agent": "headline-v2" }
}

Resolve production instead of v1.4.2.

Labels are pointers. One label per name per prompt (unique constraint). Move them to do staged rollouts without redeploys. Labels are workspace-scoped.

{
  "name": "production",
  "version_tag": "1.0.0"
}
// → 201 Created — or 409 Conflict if "production" already exists

Score prompts. Track quality over time.

Define a suite with criteria (free-form JSON). Submit results tied to a run_id. Query history; deletes cascade.

{
  "name": "accuracy-check",
  "description": "Check output accuracy",
  "prompt_name": "welcome",
  "version_tag": "1.0.0",
  "criteria": { "min_score": 0.8 }
}

{
  "run_id": "run_7a3fe2c",
  "score": 0.95,
  "metadata": { "judge": "gpt-4" }
}

All 28 endpoints, grouped.

Deploy to production.

The default SQLite setup handles millions of traces on a single node. When you need multi-node, switch to Postgres with one environment variable.

version: "3.8"
services:
  promptmetrics:
    image: promptmetrics/promptmetrics:latest
    ports:
      - "3000:3000"
    environment:
      - API_KEY_SALT=${API_KEY_SALT}
      - DATABASE_URL=${DATABASE_URL:-sqlite:./pm.db}
      - DRIVER=${DRIVER:-filesystem}
    volumes:
      - ./data:/app/data
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
      interval: 30s
      timeout: 5s
      retries: 3

Environment variables

Health checks & upgrades

curl -f http://localhost:3000/health

docker compose pull
docker compose up -d

Migrate from Langfuse or Langsmith.

Both platforms export standard formats that PromptMetrics can ingest. Most migrations finish in an hour.

curl https://api.langfuse.com/api/public/traces \
  -H "Authorization: Basic $(echo -n 'pk:sk' | base64)" \
  --output traces.json

npx promptmetrics migrate --from langfuse --input traces.json --endpoint http://localhost:3000

curl http://localhost:3000/v1/traces/count -H "X-API-Key: pm_xxx"
curl http://localhost:3000/v1/prompts/count -H "X-API-Key: pm_xxx"

Field mapping

Rollback strategy

npx promptmetrics migrate --from langfuse --input traces.json --endpoint http://localhost:3000 --dry-run

cp pm.db pm.db.pre-migration-$(date +%s)
# or for Postgres:
# pg_dump $DATABASE_URL > pm-pre-migration.sql

Pick the backend that matches your team.

DRIVER=filesystem

DRIVER=github
GITHUB_REPO=owner/repo
GITHUB_TOKEN=ghp_…

DRIVER=s3
S3_BUCKET=…
S3_REGION=…

DATABASE_URL=postgres://…