Worker Capability Configuration - Spezialisierte Worker ermöglichen #265

New issue

Closed

opened 2026-01-25 11:48:23 +00:00 by jack · 5 comments

jack commented 2026-01-25 11:48:23 +00:00

Owner

Copy link

Problem

Aktuell detektieren alle Worker ihre Capabilities automatisch und identisch:

// packages/worker/src/worker-service.ts
private detectCapabilities(): WorkerCapability[] {
  const capabilities: WorkerCapability[] = [];
  
  // Agent-basiert → Alle bekommen das gleiche
  capabilities.push('observation:mistral');
  capabilities.push('summarize:mistral');
  
  // Settings-basiert → Alle bekommen das gleiche
  if (settings.VECTOR_DB === 'qdrant') {
    capabilities.push('qdrant:sync');
    capabilities.push('semantic:search');
  }
  
  // Standard → Alle bekommen das gleiche
  capabilities.push('context:generate');
  capabilities.push('claudemd:generate');
  
  return capabilities;
}

Resultat: Alle Worker sind identisch - keine Spezialisierung möglich.

Anforderung

Worker sollen mit spezifischen Capabilities gestartet werden können:

# Worker nur für Observations
worker-service --capabilities observation:mistral

# Worker nur für CLAUDE.md Generation
worker-service --capabilities claudemd:generate

# Worker für Vector-DB Operations
worker-service --capabilities qdrant:sync,semantic:search,embedding:local

Lösung: Capability-Auflösung mit Prioritäten

┌─────────────────────────────────────────────────────────────────────┐
│                    CAPABILITY RESOLUTION                            │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  Priorität (höchste zuerst):                                       │
│                                                                     │
│  1. CLI Argument                                                   │
│     --capabilities observation:mistral,summarize:mistral            │
│                                                                     │
│  2. Environment Variable                                           │
│     WORKER_CAPABILITIES="observation:mistral,summarize:mistral"     │
│                                                                     │
│  3. Worker Profile (aus Settings)                                  │
│     --profile observer → Settings.WORKER_PROFILES.observer          │
│                                                                     │
│  4. Auto-Detection (Fallback)                                      │
│     Aktuelles Verhalten - alle Capabilities                        │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘

Implementation

1. CLI Arguments erweitern

// packages/worker/src/cli.ts
interface WorkerCLIOptions {
  capabilities?: string;     // Comma-separated: "observation:mistral,summarize:mistral"
  profile?: string;          // Profile-Name: "observer"
  labels?: string;           // Comma-separated: "region=eu,gpu=true"
}

// Parsing
const caps = options.capabilities?.split(',').map(c => c.trim());

2. Settings um Profiles erweitern

// packages/shared/src/settings.ts
export interface Settings {
  // ... existing fields
  
  WORKER_PROFILES: Record<string, WorkerProfile>;
}

interface WorkerProfile {
  capabilities: WorkerCapability[];
  labels?: Record<string, string>;
  description?: string;
}

// ~/.claude-mem/settings.json
{
  "WORKER_PROFILES": {
    "observer": {
      "description": "Observation-only worker",
      "capabilities": ["observation:mistral"]
    },
    "summarizer": {
      "description": "Summarization and CLAUDE.md generation",
      "capabilities": ["summarize:mistral", "claudemd:generate"]
    },
    "vector": {
      "description": "Vector DB operations",
      "capabilities": ["qdrant:sync", "semantic:search", "embedding:local"]
    },
    "full": {
      "description": "All capabilities (default)",
      "capabilities": ["observation:mistral", "summarize:mistral", "claudemd:generate", "context:generate"]
    }
  }
}

3. Capability Resolution

// packages/worker/src/worker-service.ts
private resolveCapabilities(): WorkerCapability[] {
  // 1. CLI Argument (höchste Priorität)
  if (this.config.capabilities?.length) {
    this.logger.info(`Using CLI capabilities: ${this.config.capabilities.join(', ')}`);
    return this.config.capabilities;
  }
  
  // 2. Environment Variable
  const envCaps = process.env.WORKER_CAPABILITIES;
  if (envCaps) {
    const caps = envCaps.split(',').map(c => c.trim() as WorkerCapability);
    this.logger.info(`Using env capabilities: ${caps.join(', ')}`);
    return caps;
  }
  
  // 3. Profile aus Settings
  const settings = loadSettings();
  if (this.config.profile) {
    const profile = settings.WORKER_PROFILES?.[this.config.profile];
    if (profile) {
      this.logger.info(`Using profile "${this.config.profile}": ${profile.capabilities.join(', ')}`);
      return profile.capabilities;
    }
    this.logger.warn(`Profile "${this.config.profile}" not found, falling back to auto-detection`);
  }
  
  // 4. Fallback: Auto-Detection (aktuelles Verhalten)
  this.logger.info('Using auto-detected capabilities');
  return this.detectCapabilities();
}

4. Labels für Worker

Zusätzlich zu Capabilities können Worker Labels haben (für #263 Federation):

interface WorkerMetadata {
  // ... existing fields
  labels?: Record<string, string>;
}

// Beispiel
{
  labels: {
    "region": "eu-central",
    "gpu": "nvidia-rtx4090",
    "env": "production"
  }
}

Deployment-Szenarien

Docker Compose mit spezialisierten Workern

services:
  # Mehrere Observer (häufigste Task-Art)
  worker-observer-1:
    image: claude-mem-worker
    environment:
      WORKER_CAPABILITIES: "observation:mistral"
      
  worker-observer-2:
    image: claude-mem-worker
    environment:
      WORKER_CAPABILITIES: "observation:mistral"
      
  worker-observer-3:
    image: claude-mem-worker
    environment:
      WORKER_CAPABILITIES: "observation:mistral"
      
  # Ein Summarizer reicht meist
  worker-summarizer:
    image: claude-mem-worker
    environment:
      WORKER_CAPABILITIES: "summarize:mistral,claudemd:generate"
      
  # Vector-Worker (optional)
  worker-vector:
    image: claude-mem-worker
    environment:
      WORKER_CAPABILITIES: "qdrant:sync,semantic:search,embedding:local"

Lokale Entwicklung mit Profilen

# Terminal 1: Observer
worker-service --profile observer

# Terminal 2: Summarizer  
worker-service --profile summarizer

# Terminal 3: Full Worker (Default)
worker-service

UI-Erweiterung

Worker-Übersicht zeigt Capabilities pro Worker:

┌─────────────────────────────────────────────────────────────────────┐
│  Workers                                              [+ Spawn] [⟳] │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  worker-1  🔄 busy                                                 │
│  └─ Capabilities: observation:mistral                              │
│  └─ Current Task: observation #1234                                │
│                                                                     │
│  worker-2  🔄 busy                                                 │
│  └─ Capabilities: observation:mistral                              │
│  └─ Current Task: observation #1235                                │
│                                                                     │
│  worker-3  💤 idle                                                 │
│  └─ Capabilities: summarize:mistral, claudemd:generate             │
│                                                                     │
│  worker-4  🔄 busy                                                 │
│  └─ Capabilities: qdrant:sync, semantic:search, embedding:local    │
│  └─ Current Task: qdrant:sync #89                                  │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘

Vorteile

Aspekt	Vorteil
Ressourcen-Effizienz	Observer brauchen keine Vector-DB Deps
Skalierung	Mehr Observer, weniger Summarizer
Isolation	Vector-Worker kann eigene Ressourcen haben
Debugging	Klare Zuordnung welcher Worker was macht
Federation-Ready	Grundlage für #263 Hub-Labels

Aufwand

• CLI Options erweitern (--capabilities, --profile, --labels)
• Settings Schema erweitern (WORKER_PROFILES)
• resolveCapabilities() implementieren
• Default-Profiles in Settings-Defaults
• UI: Capabilities in Worker-Übersicht anzeigen
• Dokumentation

Abhängigkeiten

• Voraussetzung für: #263 (WorkerHub Federation) - Labels/Capabilities dort weiterverwendet
• Voraussetzung für: #264 (Unified WebSocket) - Worker Registration mit Capabilities

Verwandte Issues

• #254 - Worker-Konfigurations-Modal (UI für Profile-Auswahl)
• #256 - Auto-Spawning (sollte Profile berücksichtigen)
• #263 - WorkerHub Federation (nutzt Labels/Capabilities für Routing)

## Problem Aktuell detektieren alle Worker ihre Capabilities automatisch und identisch: ```typescript // packages/worker/src/worker-service.ts private detectCapabilities(): WorkerCapability[] { const capabilities: WorkerCapability[] = []; // Agent-basiert → Alle bekommen das gleiche capabilities.push('observation:mistral'); capabilities.push('summarize:mistral'); // Settings-basiert → Alle bekommen das gleiche if (settings.VECTOR_DB === 'qdrant') { capabilities.push('qdrant:sync'); capabilities.push('semantic:search'); } // Standard → Alle bekommen das gleiche capabilities.push('context:generate'); capabilities.push('claudemd:generate'); return capabilities; } ``` **Resultat:** Alle Worker sind identisch - keine Spezialisierung möglich. ## Anforderung Worker sollen mit spezifischen Capabilities gestartet werden können: ```bash # Worker nur für Observations worker-service --capabilities observation:mistral # Worker nur für CLAUDE.md Generation worker-service --capabilities claudemd:generate # Worker für Vector-DB Operations worker-service --capabilities qdrant:sync,semantic:search,embedding:local ``` ## Lösung: Capability-Auflösung mit Prioritäten ``` ┌─────────────────────────────────────────────────────────────────────┐ │ CAPABILITY RESOLUTION │ ├─────────────────────────────────────────────────────────────────────┤ │ │ │ Priorität (höchste zuerst): │ │ │ │ 1. CLI Argument │ │ --capabilities observation:mistral,summarize:mistral │ │ │ │ 2. Environment Variable │ │ WORKER_CAPABILITIES="observation:mistral,summarize:mistral" │ │ │ │ 3. Worker Profile (aus Settings) │ │ --profile observer → Settings.WORKER_PROFILES.observer │ │ │ │ 4. Auto-Detection (Fallback) │ │ Aktuelles Verhalten - alle Capabilities │ │ │ └─────────────────────────────────────────────────────────────────────┘ ``` ## Implementation ### 1. CLI Arguments erweitern ```typescript // packages/worker/src/cli.ts interface WorkerCLIOptions { capabilities?: string; // Comma-separated: "observation:mistral,summarize:mistral" profile?: string; // Profile-Name: "observer" labels?: string; // Comma-separated: "region=eu,gpu=true" } // Parsing const caps = options.capabilities?.split(',').map(c => c.trim()); ``` ### 2. Settings um Profiles erweitern ```typescript // packages/shared/src/settings.ts export interface Settings { // ... existing fields WORKER_PROFILES: Record<string, WorkerProfile>; } interface WorkerProfile { capabilities: WorkerCapability[]; labels?: Record<string, string>; description?: string; } ``` ```json // ~/.claude-mem/settings.json { "WORKER_PROFILES": { "observer": { "description": "Observation-only worker", "capabilities": ["observation:mistral"] }, "summarizer": { "description": "Summarization and CLAUDE.md generation", "capabilities": ["summarize:mistral", "claudemd:generate"] }, "vector": { "description": "Vector DB operations", "capabilities": ["qdrant:sync", "semantic:search", "embedding:local"] }, "full": { "description": "All capabilities (default)", "capabilities": ["observation:mistral", "summarize:mistral", "claudemd:generate", "context:generate"] } } } ``` ### 3. Capability Resolution ```typescript // packages/worker/src/worker-service.ts private resolveCapabilities(): WorkerCapability[] { // 1. CLI Argument (höchste Priorität) if (this.config.capabilities?.length) { this.logger.info(`Using CLI capabilities: ${this.config.capabilities.join(', ')}`); return this.config.capabilities; } // 2. Environment Variable const envCaps = process.env.WORKER_CAPABILITIES; if (envCaps) { const caps = envCaps.split(',').map(c => c.trim() as WorkerCapability); this.logger.info(`Using env capabilities: ${caps.join(', ')}`); return caps; } // 3. Profile aus Settings const settings = loadSettings(); if (this.config.profile) { const profile = settings.WORKER_PROFILES?.[this.config.profile]; if (profile) { this.logger.info(`Using profile "${this.config.profile}": ${profile.capabilities.join(', ')}`); return profile.capabilities; } this.logger.warn(`Profile "${this.config.profile}" not found, falling back to auto-detection`); } // 4. Fallback: Auto-Detection (aktuelles Verhalten) this.logger.info('Using auto-detected capabilities'); return this.detectCapabilities(); } ``` ### 4. Labels für Worker Zusätzlich zu Capabilities können Worker Labels haben (für #263 Federation): ```typescript interface WorkerMetadata { // ... existing fields labels?: Record<string, string>; } // Beispiel { labels: { "region": "eu-central", "gpu": "nvidia-rtx4090", "env": "production" } } ``` ## Deployment-Szenarien ### Docker Compose mit spezialisierten Workern ```yaml services: # Mehrere Observer (häufigste Task-Art) worker-observer-1: image: claude-mem-worker environment: WORKER_CAPABILITIES: "observation:mistral" worker-observer-2: image: claude-mem-worker environment: WORKER_CAPABILITIES: "observation:mistral" worker-observer-3: image: claude-mem-worker environment: WORKER_CAPABILITIES: "observation:mistral" # Ein Summarizer reicht meist worker-summarizer: image: claude-mem-worker environment: WORKER_CAPABILITIES: "summarize:mistral,claudemd:generate" # Vector-Worker (optional) worker-vector: image: claude-mem-worker environment: WORKER_CAPABILITIES: "qdrant:sync,semantic:search,embedding:local" ``` ### Lokale Entwicklung mit Profilen ```bash # Terminal 1: Observer worker-service --profile observer # Terminal 2: Summarizer worker-service --profile summarizer # Terminal 3: Full Worker (Default) worker-service ``` ## UI-Erweiterung Worker-Übersicht zeigt Capabilities pro Worker: ``` ┌─────────────────────────────────────────────────────────────────────┐ │ Workers [+ Spawn] [⟳] │ ├─────────────────────────────────────────────────────────────────────┤ │ │ │ worker-1 🔄 busy │ │ └─ Capabilities: observation:mistral │ │ └─ Current Task: observation #1234 │ │ │ │ worker-2 🔄 busy │ │ └─ Capabilities: observation:mistral │ │ └─ Current Task: observation #1235 │ │ │ │ worker-3 💤 idle │ │ └─ Capabilities: summarize:mistral, claudemd:generate │ │ │ │ worker-4 🔄 busy │ │ └─ Capabilities: qdrant:sync, semantic:search, embedding:local │ │ └─ Current Task: qdrant:sync #89 │ │ │ └─────────────────────────────────────────────────────────────────────┘ ``` ## Vorteile | Aspekt | Vorteil | |--------|---------| | **Ressourcen-Effizienz** | Observer brauchen keine Vector-DB Deps | | **Skalierung** | Mehr Observer, weniger Summarizer | | **Isolation** | Vector-Worker kann eigene Ressourcen haben | | **Debugging** | Klare Zuordnung welcher Worker was macht | | **Federation-Ready** | Grundlage für #263 Hub-Labels | ## Aufwand - [ ] CLI Options erweitern (`--capabilities`, `--profile`, `--labels`) - [ ] Settings Schema erweitern (`WORKER_PROFILES`) - [ ] `resolveCapabilities()` implementieren - [ ] Default-Profiles in Settings-Defaults - [ ] UI: Capabilities in Worker-Übersicht anzeigen - [ ] Dokumentation ## Abhängigkeiten - Voraussetzung für: #263 (WorkerHub Federation) - Labels/Capabilities dort weiterverwendet - Voraussetzung für: #264 (Unified WebSocket) - Worker Registration mit Capabilities ## Verwandte Issues - #254 - Worker-Konfigurations-Modal (UI für Profile-Auswahl) - #256 - Auto-Spawning (sollte Profile berücksichtigen) - #263 - WorkerHub Federation (nutzt Labels/Capabilities für Routing)

jack added the

priority

medium

type

feature

labels 2026-01-25 11:48:25 +00:00

jack added a new dependency 2026-01-25 11:48:30 +00:00

#263 feat: WorkerHub Federation - Verteilte Worker-Pools mit Prioritäten

jack commented 2026-01-25 12:07:54 +00:00

Author

Owner

Copy link

Betrifft auch InProcessWorker

Die Capability-Konfiguration muss auch für den InProcessWorker implementiert werden:

Betroffene Dateien:

• packages/worker/src/worker-service.ts - Standalone Worker
• packages/worker/src/in-process-worker.ts - InProcess Worker (vom Backend gespawnt)

InProcessWorker Besonderheit

Der InProcessWorker wird vom Backend via WorkerProcessManager gespawnt. Hier muss die Capability-Konfiguration durchgereicht werden:

// packages/backend/src/websocket/worker-process-manager.ts
async spawnWorker(options: SpawnOptions): Promise<string> {
  const args = [
    '--backend', this.backendUrl,
    '--token', this.authToken,
  ];
  
  // NEU: Capabilities durchreichen
  if (options.capabilities?.length) {
    args.push('--capabilities', options.capabilities.join(','));
  }
  
  // NEU: Profile durchreichen
  if (options.profile) {
    args.push('--profile', options.profile);
  }
  
  // Worker starten...
}

UI-Integration (#254)

Das Worker-Modal sollte beim Spawnen auch Capabilities/Profile auswählen können:

┌─────────────────────────────────────────────────────────┐
│  Spawn Worker                                     [X]   │
├─────────────────────────────────────────────────────────┤
│                                                         │
│  Profile:  [▼ observer          ]                      │
│            ├─ observer - Observation only              │
│            ├─ summarizer - Summarize + CLAUDE.md       │
│            ├─ vector - Vector DB operations            │
│            └─ full - All capabilities (default)        │
│                                                         │
│  -- oder --                                            │
│                                                         │
│  Custom Capabilities:                                   │
│  ☑ observation:mistral                                 │
│  ☐ summarize:mistral                                   │
│  ☐ claudemd:generate                                   │
│  ☐ context:generate                                    │
│  ☐ qdrant:sync                                         │
│  ☐ semantic:search                                     │
│                                                         │
│                              [Cancel] [Spawn Worker]    │
└─────────────────────────────────────────────────────────┘

Auto-Spawn Konfiguration

Auch das Auto-Spawning (#256) sollte Profile berücksichtigen:

// settings.json
{
  "AUTO_SPAWN_WORKERS": {
    "enabled": true,
    "profiles": {
      "observer": { "min": 2, "max": 5 },
      "summarizer": { "min": 1, "max": 2 }
    }
  }
}

## Betrifft auch InProcessWorker Die Capability-Konfiguration muss auch für den `InProcessWorker` implementiert werden: **Betroffene Dateien:** - `packages/worker/src/worker-service.ts` - Standalone Worker - `packages/worker/src/in-process-worker.ts` - InProcess Worker (vom Backend gespawnt) ### InProcessWorker Besonderheit Der InProcessWorker wird vom Backend via `WorkerProcessManager` gespawnt. Hier muss die Capability-Konfiguration durchgereicht werden: ```typescript // packages/backend/src/websocket/worker-process-manager.ts async spawnWorker(options: SpawnOptions): Promise<string> { const args = [ '--backend', this.backendUrl, '--token', this.authToken, ]; // NEU: Capabilities durchreichen if (options.capabilities?.length) { args.push('--capabilities', options.capabilities.join(',')); } // NEU: Profile durchreichen if (options.profile) { args.push('--profile', options.profile); } // Worker starten... } ``` ### UI-Integration (#254) Das Worker-Modal sollte beim Spawnen auch Capabilities/Profile auswählen können: ``` ┌─────────────────────────────────────────────────────────┐ │ Spawn Worker [X] │ ├─────────────────────────────────────────────────────────┤ │ │ │ Profile: [▼ observer ] │ │ ├─ observer - Observation only │ │ ├─ summarizer - Summarize + CLAUDE.md │ │ ├─ vector - Vector DB operations │ │ └─ full - All capabilities (default) │ │ │ │ -- oder -- │ │ │ │ Custom Capabilities: │ │ ☑ observation:mistral │ │ ☐ summarize:mistral │ │ ☐ claudemd:generate │ │ ☐ context:generate │ │ ☐ qdrant:sync │ │ ☐ semantic:search │ │ │ │ [Cancel] [Spawn Worker] │ └─────────────────────────────────────────────────────────┘ ``` ### Auto-Spawn Konfiguration Auch das Auto-Spawning (#256) sollte Profile berücksichtigen: ```json // settings.json { "AUTO_SPAWN_WORKERS": { "enabled": true, "profiles": { "observer": { "min": 2, "max": 5 }, "summarizer": { "min": 1, "max": 2 } } } } ```

jack commented 2026-01-25 12:12:11 +00:00

Author

Owner

Copy link

Abstrakte Capabilities statt Provider-spezifische

Problem

Aktuell sind Capabilities Provider-spezifisch:

// Aktuell (redundant)
capabilities: ['observation:mistral', 'summarize:mistral']

Der Provider wird aber sowieso beim Worker-Start konfiguriert (--agent mistral). Die Capability-Provider-Kombination ist also doppelt.

Lösung

Capabilities sollten abstrakt sein - der Provider ist separate Konfiguration:

// NEU: Abstrakt
capabilities: ['observation', 'summarize', 'claudemd-generate']
provider: 'mistral'  // Separate Angabe

Bereits vorhanden

Es gibt schon AbstractCapability in packages/types/src/capabilities.ts:

export type AbstractCapability =
  | 'observation'
  | 'summarize'
  | 'embedding'
  | 'qdrant-sync'
  | 'semantic-search'
  | 'context-generate'
  | 'claudemd-generate';

Worker-Registration (neu)

// Worker → Backend
{
  type: 'register',
  capabilities: ['observation', 'summarize'],  // Abstrakt
  provider: {
    llm: 'mistral',           // Für observation, summarize, claudemd
    embedding: 'local',        // Für embedding
    vectordb: 'qdrant-local'   // Für qdrant-sync, semantic-search
  },
  metadata: { ... }
}

Task-Matching (vereinfacht)

// Vorher: Capability + Provider matchen
task.capability === 'observation:mistral'
worker.capabilities.includes('observation:mistral')

// Nachher: Nur Capability matchen
task.capability === 'observation'
worker.capabilities.includes('observation')
// Provider ist Worker-intern, Task ist agnostisch

Vorteile

Aspekt	Vorteil
Einfacher	Weniger Kombinationen zu pflegen
Flexibler	Task sagt "ich brauche observation", nicht "ich brauche mistral"
Austauschbar	Provider-Wechsel ohne Capability-Änderung
Cleaner	Separation of Concerns

Migration

1. WorkerCapability Type auf AbstractCapability umstellen
2. Provider als separates Feld in Worker-Registration
3. Task-Dispatcher: Nur auf abstrakte Capability matchen
4. Legacy-Format (observation:mistral) als Fallback parsen

Settings-Profile (angepasst)

{
  "WORKER_PROFILES": {
    "observer": {
      "capabilities": ["observation"],
      "provider": { "llm": "mistral" }
    },
    "summarizer": {
      "capabilities": ["summarize", "claudemd-generate"],
      "provider": { "llm": "mistral" }
    },
    "vector": {
      "capabilities": ["qdrant-sync", "semantic-search", "embedding"],
      "provider": { 
        "embedding": "local",
        "vectordb": "qdrant-local"
      }
    }
  }
}

## Abstrakte Capabilities statt Provider-spezifische ### Problem Aktuell sind Capabilities Provider-spezifisch: ```typescript // Aktuell (redundant) capabilities: ['observation:mistral', 'summarize:mistral'] ``` Der Provider wird aber sowieso beim Worker-Start konfiguriert (`--agent mistral`). Die Capability-Provider-Kombination ist also doppelt. ### Lösung Capabilities sollten **abstrakt** sein - der Provider ist separate Konfiguration: ```typescript // NEU: Abstrakt capabilities: ['observation', 'summarize', 'claudemd-generate'] provider: 'mistral' // Separate Angabe ``` ### Bereits vorhanden Es gibt schon `AbstractCapability` in `packages/types/src/capabilities.ts`: ```typescript export type AbstractCapability = | 'observation' | 'summarize' | 'embedding' | 'qdrant-sync' | 'semantic-search' | 'context-generate' | 'claudemd-generate'; ``` ### Worker-Registration (neu) ```typescript // Worker → Backend { type: 'register', capabilities: ['observation', 'summarize'], // Abstrakt provider: { llm: 'mistral', // Für observation, summarize, claudemd embedding: 'local', // Für embedding vectordb: 'qdrant-local' // Für qdrant-sync, semantic-search }, metadata: { ... } } ``` ### Task-Matching (vereinfacht) ```typescript // Vorher: Capability + Provider matchen task.capability === 'observation:mistral' worker.capabilities.includes('observation:mistral') // Nachher: Nur Capability matchen task.capability === 'observation' worker.capabilities.includes('observation') // Provider ist Worker-intern, Task ist agnostisch ``` ### Vorteile | Aspekt | Vorteil | |--------|---------| | **Einfacher** | Weniger Kombinationen zu pflegen | | **Flexibler** | Task sagt "ich brauche observation", nicht "ich brauche mistral" | | **Austauschbar** | Provider-Wechsel ohne Capability-Änderung | | **Cleaner** | Separation of Concerns | ### Migration 1. `WorkerCapability` Type auf `AbstractCapability` umstellen 2. Provider als separates Feld in Worker-Registration 3. Task-Dispatcher: Nur auf abstrakte Capability matchen 4. Legacy-Format (`observation:mistral`) als Fallback parsen ### Settings-Profile (angepasst) ```json { "WORKER_PROFILES": { "observer": { "capabilities": ["observation"], "provider": { "llm": "mistral" } }, "summarizer": { "capabilities": ["summarize", "claudemd-generate"], "provider": { "llm": "mistral" } }, "vector": { "capabilities": ["qdrant-sync", "semantic-search", "embedding"], "provider": { "embedding": "local", "vectordb": "qdrant-local" } } } } ```

jack commented 2026-01-25 12:14:00 +00:00

Author

Owner

Copy link

Task-Prioritäten

Problem

Aktuell werden Tasks nach FIFO (First In, First Out) abgearbeitet. Aber:

• claudemd-generate sollte schnell fertig werden (User wartet auf CLAUDE.md)
• summarize kann warten (Session ist eh schon beendet)
• observation ist mittel-wichtig

Lösung: Priority-Feld für Tasks

// packages/types/src/task.ts
interface Task {
  id: string;
  type: AbstractCapability;
  priority: TaskPriority;      // NEU
  payload: unknown;
  status: TaskStatus;
  createdAt: Date;
  // ...
}

type TaskPriority = 'critical' | 'high' | 'normal' | 'low';

// Numerisch für Sortierung
const PRIORITY_WEIGHT = {
  critical: 1000,
  high: 100,
  normal: 50,
  low: 10
};

Default-Prioritäten nach Task-Typ

Task-Typ	Default-Prio	Begründung
claudemd-generate	high	User wartet auf CLAUDE.md Update
observation	normal	Standard-Verarbeitung
summarize	low	Session ist bereits beendet
qdrant-sync	low	Background-Sync
semantic-search	high	User wartet auf Ergebnis

Task-Dispatcher Anpassung

// packages/backend/src/websocket/task-dispatcher.ts
private async getNextTask(capabilities: AbstractCapability[]): Promise<Task | null> {
  // Sortiert nach: Priority DESC, CreatedAt ASC
  return this.taskQueue
    .where({ capability: { $in: capabilities }, status: 'pending' })
    .orderBy([
      { priority: 'DESC' },   // Höchste Prio zuerst
      { createdAt: 'ASC' }    // Bei gleicher Prio: Älteste zuerst
    ])
    .first();
}

Dynamische Priorität

Tasks können ihre Priorität auch erhöhen wenn sie zu lange warten (Starvation Prevention):

// Task wartet > 5 Minuten → Prio erhöhen
if (task.createdAt < Date.now() - 5 * 60 * 1000) {
  task.priority = Math.min(task.priority + 1, PRIORITY.CRITICAL);
}

UI: Task-Queue mit Prioritäten

┌─────────────────────────────────────────────────────────────────────┐
│  Task Queue                                              [⟳] [⚙]   │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  🔴 CRITICAL (0)                                                   │
│                                                                     │
│  🟠 HIGH (3)                                                       │
│  ├─ #1247  claudemd-generate  session-abc  waiting 2s              │
│  ├─ #1245  semantic-search    query-xyz    waiting 5s              │
│  └─ #1244  claudemd-generate  session-def  waiting 8s              │
│                                                                     │
│  🟡 NORMAL (12)                                                    │
│  ├─ #1243  observation  session-abc  waiting 15s                   │
│  ├─ #1242  observation  session-def  waiting 18s                   │
│  └─ ... +10 more                                                   │
│                                                                     │
│  🟢 LOW (5)                                                        │
│  ├─ #1238  summarize    session-old  waiting 2m                    │
│  ├─ #1235  qdrant-sync  batch-123    waiting 3m                    │
│  └─ ... +3 more                                                    │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘

API: Priorität beim Queuen setzen

// Normal (default)
taskService.queue({ type: 'observation', payload: {...} });

// Mit expliziter Priorität
taskService.queue({ 
  type: 'observation', 
  payload: {...},
  priority: 'high'  // Override default
});

## Task-Prioritäten ### Problem Aktuell werden Tasks nach FIFO (First In, First Out) abgearbeitet. Aber: - `claudemd-generate` sollte schnell fertig werden (User wartet auf CLAUDE.md) - `summarize` kann warten (Session ist eh schon beendet) - `observation` ist mittel-wichtig ### Lösung: Priority-Feld für Tasks ```typescript // packages/types/src/task.ts interface Task { id: string; type: AbstractCapability; priority: TaskPriority; // NEU payload: unknown; status: TaskStatus; createdAt: Date; // ... } type TaskPriority = 'critical' | 'high' | 'normal' | 'low'; // Numerisch für Sortierung const PRIORITY_WEIGHT = { critical: 1000, high: 100, normal: 50, low: 10 }; ``` ### Default-Prioritäten nach Task-Typ | Task-Typ | Default-Prio | Begründung | |----------|--------------|------------| | `claudemd-generate` | **high** | User wartet auf CLAUDE.md Update | | `observation` | **normal** | Standard-Verarbeitung | | `summarize` | **low** | Session ist bereits beendet | | `qdrant-sync` | **low** | Background-Sync | | `semantic-search` | **high** | User wartet auf Ergebnis | ### Task-Dispatcher Anpassung ```typescript // packages/backend/src/websocket/task-dispatcher.ts private async getNextTask(capabilities: AbstractCapability[]): Promise<Task | null> { // Sortiert nach: Priority DESC, CreatedAt ASC return this.taskQueue .where({ capability: { $in: capabilities }, status: 'pending' }) .orderBy([ { priority: 'DESC' }, // Höchste Prio zuerst { createdAt: 'ASC' } // Bei gleicher Prio: Älteste zuerst ]) .first(); } ``` ### Dynamische Priorität Tasks können ihre Priorität auch erhöhen wenn sie zu lange warten (Starvation Prevention): ```typescript // Task wartet > 5 Minuten → Prio erhöhen if (task.createdAt < Date.now() - 5 * 60 * 1000) { task.priority = Math.min(task.priority + 1, PRIORITY.CRITICAL); } ``` ### UI: Task-Queue mit Prioritäten ``` ┌─────────────────────────────────────────────────────────────────────┐ │ Task Queue [⟳] [⚙] │ ├─────────────────────────────────────────────────────────────────────┤ │ │ │ 🔴 CRITICAL (0) │ │ │ │ 🟠 HIGH (3) │ │ ├─ #1247 claudemd-generate session-abc waiting 2s │ │ ├─ #1245 semantic-search query-xyz waiting 5s │ │ └─ #1244 claudemd-generate session-def waiting 8s │ │ │ │ 🟡 NORMAL (12) │ │ ├─ #1243 observation session-abc waiting 15s │ │ ├─ #1242 observation session-def waiting 18s │ │ └─ ... +10 more │ │ │ │ 🟢 LOW (5) │ │ ├─ #1238 summarize session-old waiting 2m │ │ ├─ #1235 qdrant-sync batch-123 waiting 3m │ │ └─ ... +3 more │ │ │ └─────────────────────────────────────────────────────────────────────┘ ``` ### API: Priorität beim Queuen setzen ```typescript // Normal (default) taskService.queue({ type: 'observation', payload: {...} }); // Mit expliziter Priorität taskService.queue({ type: 'observation', payload: {...}, priority: 'high' // Override default }); ```

jack commented 2026-01-25 12:15:29 +00:00

Author

Owner

Copy link

Korrektur: Vollständige Capability-Liste mit Prioritäten

Die compression Capability für Endless Mode (Issue #109) war schon definiert:

Alle Abstract Capabilities

Capability	Beschreibung	Default-Prio	Begründung
compression	Tool-Output komprimieren	critical	Pre-Compact, User wartet, zeitkritisch
claudemd-generate	CLAUDE.md generieren	high	User sieht Ergebnis in IDE
semantic-search	Vector-Suche	high	User wartet auf Ergebnis
context-generate	Context generieren	high	Wird für aktive Session gebraucht
observation	Observations extrahieren	normal	Standard-Verarbeitung
embedding	Embeddings generieren	normal	Für spätere Suche
qdrant-sync	Vector-DB sync	low	Background-Job
summarize	Session zusammenfassen	low	Session bereits beendet

Compression bei Pre-Compact

// Wenn pre-compact Event kommt
hook.on('session:pre-compact', async (session) => {
  // Compression-Task mit CRITICAL Priorität queuen
  await taskService.queue({
    type: 'compression',
    priority: 'critical',  // Muss SOFORT verarbeitet werden
    payload: {
      sessionId: session.id,
      // Welche Tool-Outputs komprimiert werden sollen
      toolOutputIds: session.pendingCompressions
    }
  });
});

Worker-Profile angepasst

{
  "WORKER_PROFILES": {
    "realtime": {
      "description": "Zeitkritische Tasks (Compression, Search)",
      "capabilities": ["compression", "semantic-search", "context-generate"],
      "provider": { "llm": "mistral" }
    },
    "observer": {
      "description": "Observation extraction",
      "capabilities": ["observation", "claudemd-generate"],
      "provider": { "llm": "mistral" }
    },
    "background": {
      "description": "Background tasks",
      "capabilities": ["summarize", "qdrant-sync", "embedding"],
      "provider": { "llm": "mistral", "embedding": "local", "vectordb": "qdrant-local" }
    }
  }
}

So können dedizierte "realtime" Worker für zeitkritische Tasks (wie Compression vor Compact) bereitstehen.

## Korrektur: Vollständige Capability-Liste mit Prioritäten Die `compression` Capability für Endless Mode (Issue #109) war schon definiert: ### Alle Abstract Capabilities | Capability | Beschreibung | Default-Prio | Begründung | |------------|--------------|--------------|------------| | `compression` | Tool-Output komprimieren | **critical** | Pre-Compact, User wartet, zeitkritisch | | `claudemd-generate` | CLAUDE.md generieren | **high** | User sieht Ergebnis in IDE | | `semantic-search` | Vector-Suche | **high** | User wartet auf Ergebnis | | `context-generate` | Context generieren | **high** | Wird für aktive Session gebraucht | | `observation` | Observations extrahieren | **normal** | Standard-Verarbeitung | | `embedding` | Embeddings generieren | **normal** | Für spätere Suche | | `qdrant-sync` | Vector-DB sync | **low** | Background-Job | | `summarize` | Session zusammenfassen | **low** | Session bereits beendet | ### Compression bei Pre-Compact ```typescript // Wenn pre-compact Event kommt hook.on('session:pre-compact', async (session) => { // Compression-Task mit CRITICAL Priorität queuen await taskService.queue({ type: 'compression', priority: 'critical', // Muss SOFORT verarbeitet werden payload: { sessionId: session.id, // Welche Tool-Outputs komprimiert werden sollen toolOutputIds: session.pendingCompressions } }); }); ``` ### Worker-Profile angepasst ```json { "WORKER_PROFILES": { "realtime": { "description": "Zeitkritische Tasks (Compression, Search)", "capabilities": ["compression", "semantic-search", "context-generate"], "provider": { "llm": "mistral" } }, "observer": { "description": "Observation extraction", "capabilities": ["observation", "claudemd-generate"], "provider": { "llm": "mistral" } }, "background": { "description": "Background tasks", "capabilities": ["summarize", "qdrant-sync", "embedding"], "provider": { "llm": "mistral", "embedding": "local", "vectordb": "qdrant-local" } } } } ``` So können dedizierte "realtime" Worker für zeitkritische Tasks (wie Compression vor Compact) bereitstehen.

jack commented 2026-01-25 12:22:51 +00:00

Author

Owner

Copy link

Implementation Complete ✅

Implemented priority-based capability resolution in commit 3720ab1:

Priority Chain (highest first)

1. CLI argument (--capabilities obs:mistral,sum:mistral)
2. Environment variable (WORKER_CAPABILITIES=obs:mistral,sum:mistral)
3. Profile from settings (--profile observer or WORKER_PROFILE=observer)
4. Auto-detection (fallback based on AI_PROVIDER)

Changes

• packages/worker/src/cli.ts:

  • Added --capabilities and --profile options
  • Added config command to show configuration
  • Added profiles command to list available profiles
  • Added parseWorkerProfiles() helper function

• packages/worker/src/worker-service.ts:

  • Added resolveCapabilities() method with priority chain
  • Reordered constructor to initialize agent before capability resolution

• packages/worker/src/in-process-worker.ts:

  • Added identical resolveCapabilities() method for consistency

Usage

# Direct capabilities
claude-mem-worker start --capabilities observation:mistral,summarize:mistral

# Using profile
claude-mem-worker start --profile observer

# Via environment
WORKER_CAPABILITIES=observation:mistral,summarize:mistral claude-mem-worker start
WORKER_PROFILE=observer claude-mem-worker start

Profile Configuration

Add to ~/.claude-mem/settings.json:

{
  "WORKER_PROFILES": "[{\"name\": \"observer\", \"capabilities\": [\"observation:mistral\"]}, {\"name\": \"full\", \"capabilities\": [\"observation:mistral\", \"summarize:mistral\", \"claudemd:generate\"]}]"
}

**Implementation Complete** ✅ Implemented priority-based capability resolution in commit `3720ab1`: ### Priority Chain (highest first) 1. **CLI argument** (`--capabilities obs:mistral,sum:mistral`) 2. **Environment variable** (`WORKER_CAPABILITIES=obs:mistral,sum:mistral`) 3. **Profile from settings** (`--profile observer` or `WORKER_PROFILE=observer`) 4. **Auto-detection** (fallback based on AI_PROVIDER) ### Changes - `packages/worker/src/cli.ts`: - Added `--capabilities` and `--profile` options - Added `config` command to show configuration - Added `profiles` command to list available profiles - Added `parseWorkerProfiles()` helper function - `packages/worker/src/worker-service.ts`: - Added `resolveCapabilities()` method with priority chain - Reordered constructor to initialize agent before capability resolution - `packages/worker/src/in-process-worker.ts`: - Added identical `resolveCapabilities()` method for consistency ### Usage ```bash # Direct capabilities claude-mem-worker start --capabilities observation:mistral,summarize:mistral # Using profile claude-mem-worker start --profile observer # Via environment WORKER_CAPABILITIES=observation:mistral,summarize:mistral claude-mem-worker start WORKER_PROFILE=observer claude-mem-worker start ``` ### Profile Configuration Add to `~/.claude-mem/settings.json`: ```json { "WORKER_PROFILES": "[{\"name\": \"observer\", \"capabilities\": [\"observation:mistral\"]}, {\"name\": \"full\", \"capabilities\": [\"observation:mistral\", \"summarize:mistral\", \"claudemd:generate\"]}]" } ```

jack closed this issue 2026-01-25 13:23:32 +00:00

jonas.hanisch referenced this issue from a commit 2026-01-25 20:16:49 +00:00

feat(worker): add capability configuration system (Issue #265)

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

renovate/all-minor-patch

renovate/react

renovate/better-sqlite3-12.x

renovate/major-vite

renovate/vite

feature/ui-improvements-and-tests

v3.0.4

v3.0.3

v3.0.2

v3.0.1

v3.0.0

v2.49.7

v2.49.6

v2.49.5

v2.49.4

v2.49.3

v2.49.2

v2.49.1

v2.49.0

v2.48.1

v2.48.0

v2.47.2

v2.47.1

v2.47.0

v2.46.0

v2.45.0

v2.44.0

v2.43.2

v2.43.1

v2.43.0

v2.42.0

v2.41.2

v2.41.1

v2.41.0

v2.40.1

v2.40.0

v2.39.0

v2.38.0

v2.37.0

v2.36.0

v2.35.0

v2.34.1

v2.34.0

v2.33.1

v2.33.0

v2.32.0

v2.31.1

v2.31.0

v2.30.0

v2.29.0

v2.28.0

v2.27.0

v2.26.0

v2.25.0

v2.24.0

v2.23.0

v2.22.0

v2.21.0

v2.20.0

v2.19.24

v2.19.23

v2.19.22

v2.19.21

v2.19.20

v2.19.19

v2.19.18

v2.19.17

v2.19.16

v2.19.15

v2.19.14

v2.19.13

v2.19.12

v2.19.11

v2.19.10

v2.19.9

v2.19.8

v2.19.7

v2.19.6

v2.19.5

v2.19.4

v2.19.3

v2.19.2

v2.19.1

v2.19.0

v2.18.2

v2.18.1

v2.18.0

v2.17.3

v2.17.2

v2.17.1

v2.17.0

v2.16.0

v2.15.0

v2.14.2

v2.14.1

v2.14.0

v2.13.2

v2.13.1

v2.13.0

v2.12.0

v2.11.3

v2.11.2

v2.11.1

v2.11.0

v2.10.0

v2.9.0

v2.8.0

v2.7.1

v2.7.0

v2.6.0

v2.5.1

v2.5.0

v2.4.5

v2.4.4

v2.4.3

v2.4.2

v2.4.1

v2.4.0

v2.3.0

v2.2.0

v2.1.0

v2.0.1

v2.0.0

v1.20.4

v1.20.3

v1.20.2

v1.20.1

v1.20.0

v1.19.0

v1.18.4

v1.18.3

v1.18.2

v1.18.1

v1.18.0

v1.17.0

v1.16.2

v1.16.1

v1.16.0

v1.15.1

v1.15.0

v1.14.2

v1.14.1

v1.14.0

v1.13.0

v1.12.0

v1.11.0

v1.10.0

v1.9.0

v1.8.1

v1.8.0

v1.7.0

v1.6.0

v1.5.0

v1.4.0

v1.3.8

v1.3.7

v1.3.6

v1.3.5

v1.3.4

v1.3.3

v1.3.2

v1.3.1

v1.3.0

v1.2.2

v1.2.1

v1.2.0

v1.1.0

v1.0.4

v1.0.3

v1.0.2

v1.0.1

v1.0.0

Labels

Clear labels   

good first issue

Good for newcomers

  

has-pr

Issue has an associated pull request

  

help wanted

Community contributions welcome

  

idea

Idea or concept - not yet ready for implementation

  

priority

critical

Urgent issue requiring immediate attention

  

priority

high

High priority issue

  

priority

low

Low priority issue

  

priority

medium

Medium priority issue

  

status

blocked

Blocked by external dependency

  

status

in-progress

Currently being worked on

  

status

needs-review

Awaiting code review

  

status

ready

Ready to be worked on

  

type

bug

Something isn't working

  

type

docs

Documentation improvements

  

type

enhancement

Improvement to existing functionality

  

type

feature

New feature or request

  

type

refactor

Code refactoring without functional changes

No labels

good first issue

has-pr

help wanted

idea

priority

critical

priority

high

priority

low

priority

medium

status

blocked

status

in-progress

status

needs-review

status

ready

type

bug

type

docs

type

enhancement

type

feature

type

refactor

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Blocks

#263 feat: WorkerHub Federation - Verteilte Worker-Pools mit Prioritäten

customable/claude-mem

Reference

customable/claude-mem#265

No description provided.