The Service Watchdog Reigns Supreme

Should I ship multiple daemons as independent services? Or should I deploy a single core daemon with multiple threaded services and an internal watchdog to manage lifecycle and uptime?

That question came up constantly while I was building Vaulthalla, usually every time something broke. Depending on what had broken, the “right” answer tended to change with it.

This isn’t a universal rule. Some systems should be multi-daemon, and some absolutely shouldn’t. Vaulthalla turned out to be firmly in the single-daemon camp.

TL;DR

Vaulthalla runs as:

• one daemon
• many threaded services
• one watchdog enforcing lifecycle

The reason isn’t tidiness. The runtime depends on:

• shared hot caches
• shared dependencies
• deterministic startup
• low-latency coordination

Past a certain point, process boundaries stopped helping and started taxing the hot path.

Where multi-daemon wins

Hard isolation

Separate processes give you real containment:

• crashes don’t cascade
• memory corruption is isolated
• security boundaries are enforceable

Independent scaling

If services are truly independent, you can:

• scale components separately
• distribute them across machines
• isolate workload characteristics

Runtime flexibility

You can also pick different tools for different jobs:

• C++ core
• Go workers
• Python jobs
• Node frontends

These are all legitimate reasons to split a system up.

Where it breaks down

The model falls apart when your services aren’t actually independent.

Vaulthalla is tightly coupled around:

• filesystem state
• permissions
• caching
• real-time session context

Splitting that across daemons didn’t simplify anything. It added coordination overhead everywhere.

Why single-daemon won

Shared memory beats serialization

With everything in one process there’s no IPC, no encoding and decoding, and no translation layer. State stays hot and directly accessible.

Shared caches stay real

Several services need the same data:

• FUSE
• CLI
• WebSocket lifecycle
• HTTP previews
• sync workers

Multi-daemon → duplicated caches → sync → IPC → latency → drift

Single-daemon → one cache

Startup becomes deterministic

A single daemon eliminates:

• repeated initialization
• duplicate DB pools
• “is service X ready yet?” checks

Startup becomes one sequence:

initialize once → wire → run

The watchdog is the turning point

The watchdog is what turns a single daemon from merely simple into something powerful. You keep service boundaries without paying for process boundaries.

Each service still has:

• its own lifecycle
• failure detection
• restart capability

The difference is that all of it is coordinated internally.

Here’s a simplified version of the watchdog loop:

1while (watchdogRunning) {2    std::vector<std::string> downServices;3 4    {5        std::scoped_lock lock(mutex_);6        for (const auto& [name, service] : services_)7            if (service && !service->isRunning())8                downServices.push_back(name);9    }10 11    for (const auto& name : downServices)12        restartService(name);13 14    std::this_thread::sleep_for(kWatchdogInterval);15}

Releasing the lock before restarting matters. If you hold it while a service restarts, you risk deadlocking your own recovery path. That isn’t hypothetical. It’s exactly the kind of coordination bug that appears once the runtime becomes a system in its own right.

The real shift: control plane complexity

Eventually the complexity moves out of the business logic and into the seams:

• startup ordering
• shutdown ordering
• health monitoring
• restart semantics
• dependency wiring

Even a small runtime manager ends up being a control plane.

For example, here is ordered startup with a dependency handoff:

1for (const auto& entry : entries) {2    tryStart(entry);3    if (entry.name == "FUSE" && fuseService)4        Deps::get().setFuseSession(fuseService->session());5}

That isn’t boilerplate. It enforces the order the runtime actually depends on.

Centralized dependencies (done intentionally)

Vaulthalla uses a single runtime dependency registry:

1void Deps::init() {2    auto& deps = get();3    if (deps.initialized()) return;4 5    deps.storageManager = std::make_shared<storage::Manager>();6    deps.apiKeyManager = std::make_shared<vault::APIKeyManager>();7    deps.authManager = std::make_shared<auth::Manager>();8    deps.sessionManager = std::make_shared<auth::session::Manager>();9    deps.secretsManager = std::make_shared<crypto::secrets::Manager>();10    deps.fsCache = std::make_shared<fs::cache::Registry>();11    deps.shellUsageManager = std::make_shared<protocols::shell::UsageManager>();12    deps.httpCacheStats = std::make_shared<stats::model::CacheStats>();13}

Global state is normally a red flag. Here it marks the boundary of the system:

• one daemon
• one runtime
• one authoritative dependency graph

Splitting it further would be separation for its own sake.

What multi-daemon cost Vaulthalla

The pattern was the same every time:

1. duplicate caches
2. sync them
3. introduce IPC
4. add latency
5. increase caching pressure
6. drift and memory overhead

UNIX sockets didn’t fix it. They only moved the complexity into the hot path.

Final form

Vaulthalla became:

• one daemon
• many threaded services
• a watchdog enforcing lifecycle
• a shared dependency graph
• hot caches that never leave memory

I don’t think that design is “correct” in general. It’s the design that stopped fighting this particular system.

If your services are constantly:

• talking to each other
• sharing state
• paying coordination costs

they may not really be separate services, and you may be paying extra to pretend they are.

Knowledge is power. In Vaulthalla, it is sacred.