Skip to content

Management & Drain

This document describes the management layer of the DICOM Router: the two additional TCP listeners for zero-downtime deployment behind HAProxy, the operator commands, the config hot reload and the CLI tool. All statements are verified against the current code.

Related documentation: User Guide · Configuration · Deployment

1. Overview: zero downtime behind HAProxy

For rolling deployments the router runs two additional TCP listeners alongside the DICOM listener. Both are only started when management.enabled: true is set (wiring in App.java).

Listener Default port Default bind Purpose
Agent-check port 8404 0.0.0.0 HAProxy polls the server state (network-accessible)
Management port 8405 127.0.0.1 Operator commands (localhost-only for security reasons)

Defaults come from default-dicom-router.yaml:

management:
  enabled: true
  agent-check-port: 8404
  agent-check-bind-address: "0.0.0.0"
  management-port: 8405
  management-bind-address: "127.0.0.1"
  # allow-remote-management: false
  # token-file: /opt/imconnect/config/management.token

Agent-check port (HAProxy integration)

AgentCheckServer.java answers every connection with a single-line ASCII response and closes afterwards. The response derives directly from the RouterState state machine:

Router state Agent-check response Meaning for HAProxy
READY ready up accepts new connections
DRAIN drain no new connections, existing ones keep running
STOPPING down instance is shutting down

HAProxy configuration (example):

server dicom1 10.0.0.1:11112 agent-check agent-port 8404 agent-inter 2s

State machine (RouterState.java)

Thread-safe via AtomicReference<State>. Allowed transitions:

READY  --drain-->  DRAIN  --shutdown-->  STOPPING
  ^                  |
  +----resume--------+
  • drain() only transitions from READY to DRAIN (CAS).
  • resume() only transitions from DRAIN back to READY (CAS).
  • stop() transitions from any state except STOPPING to STOPPING.

The DICOM listener evaluates RouterState on accept: if the state is not READY, the freshly accepted connection is closed immediately (SocketHandler.java, accept loop). Existing connections keep running unchanged.

2. Management commands

The ManagementServer.java reads one line per connection in the format <command> [token], processes it via ManagementCommandHandler.java and closes after responding. Commands are evaluated case-insensitively; the optional token (see the security section below) is taken verbatim after the first whitespace. Without configured token authentication the protocol stays strict: any trailing text is ERROR unknown command (as before stage 2) — a client sending tokens to a router without auth has configuration drift, and that should be visible.

The command line is capped at 4096 characters. Anything longer is answered with ERROR command line too long and the connection is closed without the command being processed. The longest legitimate command — command word plus token — is far below that, so the cap bounds the read buffer rather than the protocol. It is needed because the socket timeout applies per read() and not to the whole line: a client that keeps sending bytes without a line ending never triggers it. The token check runs after the read, so the cap takes effect before any authentication.

Command Effect (verified against ManagementCommandHandler)
status Returns state, connections (number of active connections), uptime, the identity of the loaded config (config-hash = SHA-256 of the file bytes at the last successful load/reload, config-loaded = timestamp), the Q/R counters and the audit pipeline state (see below).
connections Table of active DICOM associations: UUID, client, calling AET, called AET, duration.
drain RouterState.drain() → no more new connections; reports remaining active connections.
resume RouterState.resume() → accepts connections again (only from DRAIN).
reload Triggers a config re-read (see section 3); reports OK config reloaded or the validation errors.
shutdown RouterState.stop(), then graceful shutdown in its own virtual thread; reports connections to be closed.

Example responses:

$ status
state: ready
connections: 3
uptime: 2d 4h 11m 8s
config-hash: 3f9a1c0e7b2d… (SHA-256 of the last successfully loaded YAML)
config-loaded: 2026-07-12T16:41:03Z
qr-untracked-moves: 0
qr-lookup-errors: 0
audit: enabled
audit-backlog: 0 events / 118 bytes
audit-shipped: 4211
audit-skipped: 0
audit-dropped: 0
audit-undecodable: 0
events-dropped: 0
events-written: 137
audit-last-ship: 2026-07-05T14:02:11.482Z

$ drain
OK drain mode activated, 3 active connections remaining

$ reload
OK config reloaded

Unknown or empty commands return ERROR unknown command: ... (only the command word is echoed back, never the rest of the line — a mistyped command must not reflect the token). Rejected state-changing commands return ERROR unauthorized.

A command never ends without a response: if resolving the connection list fails internally, connections answers with an ERROR line, and drain/shutdown still perform their state change — the connection count in the OK line then degrades to unknown. The shutdown thread starts in every case on shutdown; an empty response (connection closed without a line) is no longer a regular outcome.

Security: loopback enforcement + token for state-changing commands

Two independent layers of protection; the agent-check port is exempt from both — it is network-accessible by design (HAProxy polling) and only returns a state string.

Stage 1 — loopback enforcement (remote threat): ConfigValidator (validateManagementBindNotRemote) enforces that management-bind-address is a loopback address (127.0.0.1, ::1); any other address results in a validation error at startup. For a deliberate opt-out (e.g. management from a separate, secured network segment) there is management.allow-remote-management: true (default false). Loopback enforcement applies regardless of whether a token is configured.

The opt-out waives the loopback requirement, not authentication: since 1.17.0 the combination of a non-loopback bind address and allow-remote-management: true additionally requires management.token-file to be set, otherwise startup (and a reload) aborts with a validation error. A network-reachable port that accepts shutdown/drain/ reload from anyone who can reach it is the one combination with no legitimate use — so it fails rather than relying on a startup banner being read. If the bind address is not loopback, App.java still logs a WARN at startup so the risk decision remains visible in the log; it distinguishes the two cases: without a token the multi-line alarm block, with a token a single line naming the network-bound port and the token requirement.

Stage 2 — shared-secret token (local threat): Even loopback-only, otherwise any unprivileged local process can stop the router (printf 'shutdown\n' | nc 127.0.0.1 8405). With management.token-file the state-changing commands drain/resume/reload/shutdown require a token on the same line (<command> <token>); the read-only commands status/connections still answer without a token — status then reduces the audit-last-error output to its category (see below). Behavior in detail (TokenVerifier/FileTokenVerifier, ManagementCommandHandler):

  • Without token-file (default): today's behavior without auth, but a one-time WARN at startup ("management port accepts state-changing commands … from any local process"). Trailing text (including a token) on state-changing and unknown commands is not silently swallowed, but rejected as ERROR unknown command — visible configuration drift instead of surprisingly executed commands. The read-only commands status/connections are the exception: a token sent along is ignored there (the answer is the reduced view) — dicom-router-ctl status sends the token whenever a token file is present locally, and an error here would break monitoring and dicom-router-sync against routers without token auth.
  • Token semantics: The token is the first non-blank line of the file, trimmed (FileTokenVerifier.readTokenFromFile — the same normalization is also used by the ConfigValidator, so a file that passes validation is guaranteed to authenticate). Further lines (e.g. comments) are ignored; files over 64 KB are rejected as misconfiguration.
  • With token-file: At startup ConfigValidator validates that the file exists, is readable, ≤ 64 KB and not empty (first non-blank line present) — otherwise a startup error. The token value is read fresh from the file on every command: rotation = rewrite the file, no restart/reload needed (the file path, by contrast, requires a restart, see section 3). Read errors at runtime (file deleted/unreadable/empty) reject the command and log ERROR (fail-closed). Missing/wrong token → ERROR unauthorized + WARN with peer address. The comparison runs in constant time via MessageDigest.isEqual (no timing leak). On status the token is optional but verified when sent: with a valid token status renders the full audit-last-error line, without one only its category, and a wrong token yields ERROR unauthorized like on the state-changing commands. The corresponding WARN is throttled to one per minute on status (the rest at DEBUG): status is polled, so a stale token file in monitoring would otherwise produce one WARN line plus a router_events entry per poll; on the state-changing commands every WARN is kept. A trailing token on connections is still ignored.
  • File hygiene: If the token file is group/other-readable, FileTokenVerifier logs a WARN at startup (not an error — analogous to ssh key hygiene).

Setup:

# Generate a token, readable only by the router user (install -m 600 equivalent):
umask 077; openssl rand -hex 32 > /opt/imconnect/config/management.token
chown imconnect:imconnect /opt/imconnect/config/management.token

# dicom-router.yaml:
#   management:
#     token-file: /opt/imconnect/config/management.token

# Rotation without restart — simply rewrite the file:
umask 077; openssl rand -hex 32 > /opt/imconnect/config/management.token.new
mv /opt/imconnect/config/management.token.new /opt/imconnect/config/management.token

bin/dicom-router-ctl finds the file at the default path /opt/imconnect/config/management.token automatically (otherwise set DICOM_ROUTER_TOKEN_FILE, see section 4) and appends the token to state-changing commands — so systemctl stop (ExecStop = dicom-router-ctl drain-and-shutdown) keeps working unchanged. status sends the token too when the file is readable (full audit-last-error line); if it is not readable, status runs token-free and shows the category. If the router rejects a token that was sent (stale or foreign token file), status retries once token-free and shows the reduced view — with a notice line on stderr, so stdout parsers stay unaffected. The wait loop of drain-and-shutdown polls the status deliberately token-free — it only reads the connections: line, and a token file rotated mid-drain must not break the wait.

Q/R counters

The two qr- lines are always present, including on a node with no Q/R configuration (where they stay at 0). They count node-wide since process start and are not reset by a reload.

Line Description
qr-untracked-moves C-MOVE-RQs that left this node without a tracking entry: no MoveDestination, no matching rewrite configured, no store, or a failed allocation of a router MessageID. The retrieve then bypasses the bridge — the images go wherever the device addressed them. A growing value means: check the configuration, or the tracking space is full (see Capacity of the tracking space).
qr-lookup-errors Tracking lookups on the return path the store could not answer — database gone, pool exhausted, or no store on the connection at all. Counted per failing query, not per discarded object: after the first failure the bridge skips the store for the rest of the association and answers every further message C000. Any value > 0 therefore means images of a retrieve were discarded. Should be 0.

Audit lines in status

With the audit trail enabled (see Configuration — audit), status shows the state of the spool/shipper path; the two events- lines appear only when audit.events is enabled as well and belong to the separate events writer:

Line Description
audit Four states: enabled (writer and shipper are running), disabled (audit.enabled: false or DB_ENABLED=false), failed (switched on, but the spool could not be opened — nothing is being recorded, the cause is on audit-last-error) and ended (ran in this process and has stopped, the counters are final). failed and ended used to look exactly like disabled, so a full disk was indistinguishable from an audit trail somebody had deliberately turned off. With failed no audit counter lines are shown — they would all be 0 and claim that nothing was lost; the events- lines stay, because that writer does not hang off the spool. ended is by its nature only visible while the process still answers — in practice within the drain/stop window; after that the management port is gone.
audit-backlog Events not yet shipped to the DB (count / spool bytes). Grows during a DB outage.
audit-shipped Events the shipper took off the spool and handed to the database without it reporting an error. Not the same as "stored" — see audit-skipped.
audit-skipped Of the shipped events, those the database could not store because the row they hang off is missing (the association of a DIMSE operation, the listener of an association). This is the knock-on effect of an earlier loss or of a retention gap: the event itself left the spool intact, what is missing is its anchor in the database. Deliberately not folded into audit-dropped — the two causes sit in different places. Should be 0.
audit-dropped Events lost for good. Five ways lead here: the spool-max-mb cap; a spool write failure on disk; a full hand-off queue (the writer is stalled); an event that never reached the spool because encoding it already failed; and an event the shipper discards as "poison" after repeated individual attempts. Should be 0.
audit-undecodable Spool lines the shipper could not decode and skipped. Not a loss path in the audit-dropped sense (those lines no longer count towards the backlog either), but the direct sign of a binary rollback: an older version meeting events a newer one wrote. Should be 0.
events-dropped Only with audit.events.enabled: events the central events writer lost — a full hand-off queue, a discarded batch, an emit after the writer ended, or a failure while capturing in the appender (best-effort, never retried). Deliberately not counted: lines the re-entrancy guard suppresses — those are persistence-stack messages from inside the writer's own write path (protective behaviour, not a loss of router events). Steadily growing means the DB sink is stalled or queue-size is too small. Should be 0.
events-written Only with audit.events.enabled: events successfully written to router_events since process start.
audit-last-ship Timestamp of the last successful DB batch (never before the first).
audit-last-error During an outage window, the last sink error (e.g. connection refused); with audit: failed, the reason the spool could not be opened. One line and normalized: exception class, SQLState and the first line of the message (truncated), followed by the underlying cause (| caused by …, same form). The follow-up lines of a PostgreSQL error (Detail:/Where:/Hint:) are left out — that is where the column values of the offending row appear. The full line requires the token (status <token>): even the first message line can carry the DB user and host:port (connection failures) or the absolute spool path (spool-open failures), and the management port may be bound remotely. Without a token (or without a configured token-file) only the category renders — exception class(es) and SQLState, no message parts; the full normalized text stays unchanged in the application log. dicom-router-ctl status sends the token automatically when the token file is configured and readable.

A growing backlog with audit-last-error means a DB outage: routing keeps running unchanged, the spool buffers, and after recovery everything is shipped completely and without duplicates — even across a router restart.

3. Config reload in detail

The reload is implemented as an atomic config swap (see SocketHandler.java and ConfigDiff.java). A restart is not needed as long as only hot-reloadable fields were changed.

Flow (App.reloadConfig())

Serialized via a reloadLock:

  1. Reload is allowed in READY and DRAIN, rejected in state STOPPING.
  2. The YAML is reloaded from disk (configLoader.load()). Structurally invalid configs or I/O errors are returned as validation errors — the old config stays active. The reload is rejected as well when the config file is missing: nothing is bootstrapped from the built-in template (that happens only on the first regular start), and the operator's file is never created or overwritten.
  3. ConfigDiff.validateReloadable(oldConfig, newConfig) checks for restart-required changes; additionally, plugin configs are validated via PluginFactory.validatePluginConfigsForReload(...).
  4. Only if there are no errors: handler.setConfig(newConfig).

Any error leaves the previously active config unchanged. Since 1.17.0 that includes every key that matches no field — in a section or in a plugin's config: block. A reload carrying a typo is therefore rejected instead of silently resetting the affected setting to its default; the old config keeps running.

Atomic swap (SocketHandler.java)

The active config lives in an AtomicReference<ApplicationConfig> configRef. setConfig(newConfig) replaces it atomically. In handleClient(...) every connection snapshots configRef.get() once on accept into a local variable and works with this snapshot throughout.

Consequence: new connections after the swap see the new config; in-flight connections keep their snapshot, even if the config is swapped in the middle of the association.

Hot-reloadable vs. restart-required

The authoritative field list is in ConfigDiff.java. Only the following fields are marked restart-required (they are bound at boot: server socket binding, management server ports, AE title in the listener DB table):

Restart-required (reload is rejected with requires restart):

  • router.listener.host
  • router.listener.port
  • router.listener.aet
  • management.enabled
  • management.agent-check-port
  • management.agent-check-bind-address
  • management.management-port
  • management.management-bind-address
  • management.allow-remote-management
  • management.token-file (the path; the token value in the file is read fresh per command and rotates without a restart)
  • audit.enabled
  • audit.spool-path
  • audit.spool-max-mb
  • audit.batch-size
  • audit.flush-interval
  • audit.events.enabled
  • audit.events.level
  • audit.events.queue-size
  • store: of a CMoveTrackingStage action (memory ↔ database) — the two tracking stores are two separate bodies of state; switching at runtime would strand every C-MOVE tracked so far. Reported as actions.<name>.store: requires restart; the message names the affected entry by its returnDestination, because one action can carry several CMoveTrackingStage entries. The key is deliberately that identifying property rather than the position in the list: a store switch combined with a reordering cannot escape the restart rule that way, and conversely neither an entry inserted in front nor one switched on raises a false alarm for the others. Restart-required as well as soon as an entry disappears (renamed or removed): a newly appeared entry with the same store then has to take its place, one for one. A disappeared entry left uncovered — or a new entry with a different store arriving in the same reload — is restart-required, because a rename cannot be told from a removal and a sibling entry carrying the same store would otherwise cover for a renamed one that switched. Pure additions (nothing disappears) stay hot: both stores exist from boot on.
  • store: of a type: qr-bridge destination — the same reasoning from the reading side: a switch would strand every lookup for retrieves already being tracked. Reported as destinations.<name>.store: requires restart. Restart-required as well as soon as a bridge disappears without a newly appeared one taking its place with the same store, one for one — the same rule as above, including its conservative reading when a bridge disappears and one with a different store arrives in the same reload. All remaining bridge fields (calling-aet, allowed-calling-aet, presentation-contexts, max-drain-bytes, timeouts.connect/.response/.release) are hot-reloadable — they are read per incoming association from the snapshot.

A pure addition with store: database is only hot on nodes that actually have a database-backed tracking store: if the node has none (database disabled, or the JDBC wiring failed at boot), the reload is rejected instead of silently degrading to the in-memory store — the store instances are boot-bound, so either switch the entry to store: memory or fix the database and restart the node. The same rule has refused to boot with store: database and no database since 1.16.0.

Additionally: if the entire management block is added or removed, ConfigDiff reports management.enabled: requires restart (section added or removed); likewise for the audit and the audit.events block.

Hot-reloadable (everything else, not checked in ConfigDiff):

  • router.listener.socketTimeout — read per connection from the snapshot (clientSocket.setSoTimeout(...)) and thus applies automatically to new connections. Since 1.17.0 the same value also bounds a single blocking write toward the client or the destination (SO_TIMEOUT covers reads only).
  • router.listener.connectTimeout — the budget for name lookup plus TCP handshake when dialling a forward destination; read per connection from the snapshot (ConnectionHandler.connect).
  • router.listener.tcplog
  • router.memory-threshold / router.materialization-threshold / router.max-elements — the streaming object model thresholds; read per connection from the snapshot (DicomObjectPipeline / DicomStage) and thus apply to new connections.
  • router.max-pending-assemblies / router.assembly-idle-timeout / router.max-object-bytes — the limits for half-open P-DATA assemblies and the cap per assembled Data Set; likewise read per connection from the snapshot (DicomStage).
  • actions (plugin instances and their config: parameters) — with one exception: the store: switch of CMoveTrackingStage is restart-required (see above)
  • filter (routing/filter rules)
  • destinations (forward/reject/qr-bridge targets, incl. return-port/return-vip) — with one exception: the store: of a qr-bridge destination is restart-required (see above)
  • haproxy-return (global default VIP and port-range for HAProxy return paths, see configuration.md)
  • proxyprotocol (mappings)
  • audit.capture-patient-name — read per capture from the connection snapshot (the remaining audit fields bind spool writer/shipper at startup and are restart-required, see above; audit.events.* additionally binds the writer thread and the Logback appender attached at startup)

Note: socketTimeout, connectTimeout and tcplog live under router.listener but are deliberately not part of the restart-required list — ConfigDiff compares under router.listener only host, port and aet.

4. CLI tool bin/dicom-router-ctl

Bash wrapper that sends a single-line command to the management port.

For the connection the script prefers bash's built-in /dev/tcp, which needs no extra package on the host. If net redirections are disabled in the bash build, it falls back to nc and passes only -w, the one switch every variant understands (netcat-openbsd, nmap-ncat, busybox). An installed netcat is therefore no longer a prerequisite, and the variant on the host no longer matters.

Environment variables:

Variable Default Purpose
DICOM_ROUTER_HOST localhost Management host
DICOM_ROUTER_MGMT_PORT 8405 Management port
DICOM_ROUTER_TIMEOUT 5 Seconds to wait for connection and answer
DICOM_ROUTER_POLL_INTERVAL 2 Poll interval (drain-and-shutdown only)
DICOM_ROUTER_TOKEN_FILE /opt/imconnect/config/management.token (if present) Token file for state-changing commands and the full audit-last-error line of status (see section 2)

Commands:

dicom-router-ctl status              # state, connection count, uptime
dicom-router-ctl connections         # list active DICOM connections
dicom-router-ctl drain               # stop accepting new connections
dicom-router-ctl resume              # resume accepting connections
dicom-router-ctl reload              # re-read config (listener/management/audit → restart required)
dicom-router-ctl shutdown            # stop IMMEDIATELY — in-flight transfers are disconnected
dicom-router-ctl drain-and-shutdown  # stop without data loss: drain, wait for 0 connections, then shutdown

Exit codes of the state-changing commands (drain, resume, reload, shutdown): 0 = done, 1 = router unreachable or the answer was unusable, 3 = the router answered and refused — a rejected reload, a missing token or one the router rejected, a shutdown while already stopping. A configured but unreadable token file ends with 1 instead — the router was never asked. (With no token file at all, the command goes out token-free; a token-protected router then refuses with ERROR unauthorized → 3.) Up to 1.17.1 the 3 cases ended with 0, so an automation reading the exit code saw success while the old configuration kept running (#343). Two cases stay at 0 because the requested state is already in effect: a drain against an already draining router and a resume against an already ready one — so repeating either in a retry loop does not turn into a failure.

3 rather than 1 so that "refused" stays distinguishable from "unreachable": dicom-router-sync needs exactly that distinction to keep treating a purely restart-required answer as a legitimate outcome.

shutdown vs. drain-and-shutdown: shutdown terminates the router immediately — existing DICOM associations (including a C-STORE transfer currently in progress) are closed in the process. drain-and-shutdown stops without data loss: first no new connections (the HAProxy agent check reports drain, the load balancer takes the instance out of rotation), then wait until all in-flight transfers have finished, only then the stop. For maintenance and deployments, drain-and-shutdown is the right way.

Teardown budget. What happens after the shutdown command has a fixed budget: the router closes all remaining connections in parallel and counts the waits against one shared deadline of 30 seconds — not per connection. The tap shippers, the drain of the audit queue into the spool and the events writer follow, a few seconds each. So the teardown costs about a minute in the worst case, regardless of how many connections are stuck. The unit's TimeoutStopSec (300 s) bounds the whole ExecStop instead, and that is mostly the drain waiting for in-flight associations. Raise it if your associations run longer than five minutes; a smaller value does not speed up the teardown, it only cuts the drain short.

Restart: There is deliberately no restart command — the CLI talks to the running process via the management port and cannot start a new one (that is the process manager's job, and DICOM_ROUTER_HOST can point to a remote host). Under systemd: systemctl restart dicom-router — the unit automatically drains first (ExecStop=dicom-router-ctl drain-and-shutdown) and then starts the new process. A deliberate dicom-router-ctl shutdown, on the other hand, stays down (Restart=on-failure only kicks in on failures).

connections is passed through 1:1 to the server and never touches the token file; status appends the token when the token file is configured or present at the default path and readable (full audit-last-error line, see section 2) — if it is not readable, status runs token-free and shows the category, which is not an error (both commands therefore keep working for users without read access to the 0600 file); drain/resume/reload/shutdown get the token appended when a token file is configured or present at the default path (without a token file, as before without a token — for setups without auth). The token is read fresh from the file per command (first non-blank line, trimmed — like the server), so a rotation during a long drain wait does not break the final shutdown. A set but unreadable DICOM_ROUTER_TOKEN_FILE aborts state-changing commands with an error.

If the router is not reachable, every command reports that on stderr ("router not reachable at <host>:<port>") and exits 1 — instead of printing nothing at all. drain-and-shutdown is orchestrated client-side:

  1. sends drain (tolerates "already draining"); if the router does not answer here, nothing has been drained → error message + exit 1,
  2. polls status in POLL_INTERVAL-second steps until connections: 0. From here on a router that is no longer reachable counts as a completed shutdown; an answer without a usable connections: line counts as an error (exit 1), so a truncated status does not spin until systemd's stop timeout,
  3. sends shutdown and checks the response: ERROR ... (e.g. unauthorized) → error message + exit 1; no longer reachable → the shutdown counts as done.

Cluster distribution: bin/dicom-router-sync

For multi-host operation, dicom-router-sync distributes the config/dicom-router.yaml maintained on the editing host to all hosts from config/cluster-hosts and reloads it — with automatic return-port allocation (--allocate-return-ports, replaces return-port: 0 markers before snapshot and verification; a no-op without markers, a hard abort when markers exist but no Java/JAR is at hand), a local history snapshot (config/config-history/), preflight validation without a router (--validate-config, best-effort when Java/JAR is available), a canary reload on the local router (validation errors abort before any other host is touched), an atomic copy via ssh, a reload via the ctl of the respective target host (the management port stays loopback-only) and two-stage verification: SHA-256 of the file plus — where the router reports it — a comparison against the config-hash from status (proves the loaded state, the table then shows MATCH (loaded)). Order: allocate → snapshot → --validate-config → canary → distribution — history and SHA verification thus always capture the allocated state. Modes: --check (drift check only), --no-reload (distribute only, allocation runs here too). Operator view: user-guide.md, section 4.

The canary has four outcomes: OK → distribute and reload everywhere; an ERROR unauthorized → abort with its own pointer at the token file (a credentials problem, not a config problem — nothing is distributed); a response whose error lines all read : requires restart → distribute, but trigger no reload (the rolling-restart instructions are printed instead); anything else → abort before another host is touched. The check looks at every error line individually, because "restart-required" distributes the file to all hosts: a rejection caused by a mistyped plugin class name also says requires restart (inside the prose of its message), and a mixed response — a genuine restart-required change next to a broken plugin entry — is a rejection, not a restart-required config.

One edge case on heterogeneous clusters (nodes with and without a database): the canary reload only checks the local router. A config with store: database can be valid there while a target node without a database rejects the reload — the already-distributed file stays on that node, and since startup refuses store: database without a database just like reload does, it would prevent the next restart from booting. Therefore distribute configs with store: database only to clusters whose nodes all have a database.

Validate-only: --validate-config

The configuration can be checked without a running router — with exactly the same validator chain as start and reload (YAML parse, core validation, plugin config validation), without side effects (no listeners, no DB; a missing file is never created from the template):

cd /opt/imconnect
java -jar dicom-router-core-*.jar --validate-config config/dicom-router.yaml

Run this from the install root (or set chdir to the install directory in automation such as Ansible): external plugin classes are loaded from a plugins/ directory resolved relative to the working directory — the same reason start.sh does a cd and the systemd unit sets WorkingDirectory=. From any other directory, the check yields false negatives (ClassNotFoundException) for configs that reference external plugins.

The check judges the file, not the machine it runs on: it needs neither a database nor the JDBC environment, and is therefore meaningful on any workstation — including for a config that will be distributed to a different node. store: database is consequently always valid here; whether this node actually has a database is decided at startup and at reload (both refuse otherwise, see Configuration).

Exit codes: 0 = valid, 1 = invalid (error list on stderr), 2 = file not found. dicom-router-sync uses this automatically as a preflight (and starts the JVM in the install root itself, no matter where it is invoked from).

Generating HAProxy return paths: --render-haproxy

The router generates the HAProxy configuration for the configured return paths (haproxy-return + return-port on the destinations) directly from its own YAML. The subcommand only generates the file — the distribution to the HAProxy hosts including the reload is handled by bin/dicom-router-haproxy-sync (next section) or, as before, by the operator:

cd /opt/imconnect
java -jar dicom-router-core-*.jar --render-haproxy config/dicom-router.yaml [output-file]

Without an output file, the output goes to stdout, and the rendered config is what lands there. Two sources that used to get in front of it are gone: for the duration of that run everything the logging would write to stdout is detached (#346) — before, the INFO record of loading the config was the first line of the redirected file — and Logback's own status output goes to stderr instead of stdout (#363). That status output appears as soon as something goes wrong in the logging configuration itself: an account without write access to LOG_DIR calling the command directly via java -jar fails the file appender, and the status output plus stack trace then stood in front of the first config line. HAProxy rejected both with "unknown keyword". Diagnostics stay visible on stderr — the skip notes do anyway, and a console appender already configured to stderr is left alone.

For automation the form with an output file remains the more reliable one: it writes the config directly and does not depend on stdout at all — not even when an explicit -Dlogback.statusListenerClass deliberately puts the status output back on stdout. That is the form dicom-router-haproxy-sync uses.

The config is first loaded and checked with the same validator chain as --validate-config; the install-root caveat above applies equally (external plugins are resolved relative to the working directory). Exit codes: 0 = generated, 1 = config invalid or the name of a rendered destination is not a valid HAProxy proxy name (allowed: A-Za-z0-9-_.: — names are never silently sanitized), 2 = file not found. Skipped destinations (see below) are not an error.

The output is deterministic (sorted by destination name, no timestamp — identical input yields a byte-identical file, diff-friendly for config management), one listen block per rendered destination, each with an explicit mode tcp (the file is meant for an additional -f; the target HAProxy's defaults are unknown) and without a health check (TCP probes cause log spam and false alarms on DICOM devices):

# generated by dicom-router --render-haproxy — do not edit
# source: dicom-router.yaml

listen dicom_return_ct_a
    mode tcp
    bind 10.0.0.10:5013
    server ct_a 192.168.10.5:104

What gets rendered: the port-range as the boundary

A forward destination is only rendered when its return-port lies inside haproxy-return.port-range. The range separates the return paths managed by the router from the ones the operator keeps in their own HAProxy config: the generated file is overwritten completely on every sync run, and a hand-maintained listen block with the same bind would collide (cannot bind socket at HAProxy startup).

A return-port outside the range stays valid configuration and is dialed at runtime via vip:return-port as before — only its listen block belongs in the operator's HAProxy config. Skipped destinations show up in two places, with different wording:

In the generated file they appear as a comment block (part of the file, so it is distributed along and counts towards the checksum):

# skipped (return-port outside port-range 5000-5999 — manual HAProxy entry):
#   ct_b  bind 10.0.0.10:4711  -> 192.168.10.9:104

On stderr there is an additional one-line summary. It is not part of the file — which keeps it visible when the config goes to stdout, and keeps the file itself byte-identical for the drift check. dicom-router-haproxy-sync captures stderr of the render call and prints the line indented:

NOTE: 1 destination(s) skipped (return-port outside port-range 5000-5999) — these return paths are manual HAProxy entries: ct_b

The check is purely port-based and VIP-independent (the port-range is defined globally). With no port-range configured there is no boundary and every forward destination with a return-port is rendered. The proxy-name check does not apply to skipped destinations — they never become an HAProxy proxy name; their name appears in the comment conservatively sanitized instead (anything outside A-Za-z0-9-_.: becomes ?). The VIP and host on the same line are sanitized as well, there [, ] and % are additionally allowed (IPv6 brackets and zone IDs — the same charset validation requires for addresses anyway).

Without a single rendered destination, only the header plus a comment starting with # no destinations with return-port is generated (exit 0) — with the suffix inside port-range 5000-5999 when destinations with a return-port exist but all of them lie outside the range.

Distributing HAProxy return paths: bin/dicom-router-haproxy-sync

Distributes the generated return-path config to the HAProxy hosts and reloads them — after the proven pattern of dicom-router-sync: distribute, check, reload, verify. Flow: render locally (--render-haproxy, JVM in the install root; a missing JVM/JAR is a hard abort — without the render there is nothing to distribute; the deterministic output makes the SHA drift check byte-exact) → drift check per host (SHA-256; unchanged → copy and preflight are skipped, the reload still runs: "file matches" does not mean "config is loaded" — a previous run may have failed the reload or run with --no-reload, and the seamless reload is idempotent and cheap; only --no-reload skips it here too) → copy via ssh into a dotfile temp next to the target (when a conf.d/ directory is loaded, HAProxy ignores dotfiles — a half-written or invalid temp can never be picked up, while the atomic rename stays possible) → preflight on the target host against the temp file (haproxy -c -f /etc/haproxy/haproxy.cfg -f <temp>); only on success is it atomically renamed over the target — on failure it is removed and the target stays untouched on its previous state, so even a later reboot or manual reload can never load an unchecked file. Preflight precondition: the HAProxy unit loads exactly haproxy.cfg plus the target file/conf.d — additional -f files of the unit are not seen by the preflight. → reload (default sudo -n systemctl reload haproxy, seamless reload: existing TCP connections and running associations survive) → SHA verification with a result table (exit 0 only when every host is OK). Modes: --check (drift check only, read-only), --no-reload (distribute + preflight, no reload).

The target hosts live in config/haproxy-hosts (template: share/haproxy-hosts.example) — deliberately not in the router YAML: the YAML is load-strictly validated runtime config, synced byte-identically to all router nodes and config-hash-monitored; the HAProxy deployment topology is operations infrastructure the router never needs at runtime. Format: one line per target host [target-path] (user@host allowed, comments/blank lines ignored), default target path /etc/haproxy/conf.d/dicom-return.cfg. Environment overrides: DICOM_ROUTER_HAPROXY_HOSTS (hosts file), DICOM_ROUTER_HAPROXY_TARGET (default target path), DICOM_ROUTER_HAPROXY_RELOAD_CMD (reload command — setups differ, conf.d vs. EXTRAOPTS -f, the script never enforces a layout).

The script manages exclusively the one generated file — never the main haproxy.cfg. It prints the rollout order as a hint (never enforced): if the target lacks listen blocks the YAML newly has → run before dicom-router-sync (a new block is harmless until someone dials it; the other way round the router would dial a port HAProxy does not know yet); if the target has blocks the YAML no longer knows → run after dicom-router-sync.

The hint tells a third case apart from those two: when a return-port moves out of the port-range, its block disappears from the generated file while the router keeps dialing the destination via vip:return-port. On the target host this looks exactly like a removed destination — the script separates the two using the # skipped (…) block of the freshly rendered file: if the block name is listed there, the destination is still configured. Those blocks get a hint of their own: the listen block belongs in the operator's HAProxy config, and until it is there the return path is unreachable. Such a block is explicitly not stale — removing it leaves the return path permanently unreachable. The hints are not mutually exclusive: if one run covers both a removed and a moved-out destination, both appear side by side, each about its own blocks.

The time reference depends on the mode. --check changes nothing, so the hint names the next regular run as the point in time. With --no-reload the target file has already been overwritten but is not loaded yet — there the next reload or restart is the point in time. In a regular run, overwrite and reload have already happened by the time the summary appears; it says so and names the way back (put the block into your own config, reload).

One residual case the comparison does not cover: it works on block names. If a destination with a return-port outside the range is renamed, the skip block carries the new name while the target still has the old one — the old block is reported as stale even though its bind is still being dialed. The hint for stale blocks therefore points explicitly at the skip block before anything is removed by hand.

A rendered output without any listen blocks is detected and still distributed with a note (legitimate, e.g. to remove return paths) — since the port-range boundary this also covers the case where destinations with a return-port do exist but all of them lie outside the range; the note names both cases. Prerequisites on the target hosts (ssh, write access, sudo rule): deployment.md; operator view: user-guide.md, section 4.

Allocating return ports automatically: --allocate-return-ports

Replaces every return-port: 0 marker in the YAML with the smallest free port of the haproxy-return.port-range — with respect to the destination's effective VIP (return-vip override or global vip; uniqueness per (vip, port) pair). Already-assigned ports are never recomputed (sticky):

cd /opt/imconnect
java -jar dicom-router-core-*.jar --allocate-return-ports config/dicom-router.yaml

The config is first loaded and checked with the same validator chain as --validate-config — tolerating only the markers themselves; any other error aborts (nothing is allocated on a broken config); the install-root caveat above applies equally. The YAML is written back textually and comment-preserving: only the affected return-port: 0 lines change (atomically via temp file + rename in the same directory). The assignment is printed per destination:

ct_b: return-port 5014
OK allocated 1 return-port(s) in: config/dicom-router.yaml

Idempotent: without markers, a no-op with exit 0. Exit codes: 0 = allocated/nothing to do, 1 = config invalid, range exhausted or marker without a port-range, 2 = file not found. This is the only subcommand with a side effect on the config file — --validate-config and --render-haproxy stay strictly read-only. dicom-router-sync runs the allocation automatically before snapshot and preflight (see above).

5. Dependency note

The management package has no imports from the core package. ManagementCommandHandler works exclusively with callbacks (Supplier/Runnable) and the decoupled records (ConnectionInfo, ConfigValidationError). The wiring between management and core happens exclusively in App.java:

  • toConnectionInfoList(handler) is the only place where core types (ConnectionHandler) are mapped to management types (ConnectionInfo).
  • Shutdown and reload callbacks (shutdown(...), App::reloadConfig) are injected there as lambdas into the ManagementCommandHandler.

Relevant classes:

  • dicom-router-core/.../management/RouterState.java — state machine (READY/DRAIN/STOPPING)
  • dicom-router-core/.../management/AgentCheckServer.java — HAProxy agent-check listener
  • dicom-router-core/.../management/ManagementServer.java — command listener
  • dicom-router-core/.../management/ManagementCommandHandler.java — command processing (no core deps)
  • dicom-router-core/.../management/TokenVerifier.java / FileTokenVerifier.java — token auth for state-changing commands; constructed from the config exclusively in App.java
  • dicom-router-core/.../management/ConnectionInfo.java — immutable connection snapshot (record)
  • dicom-router-core/.../config/ConfigDiff.java — reload diff (restart-required fields)
  • dicom-router-core/.../App.java — wiring & reloadConfig()
  • dicom-router-core/.../core/SocketHandler.java — atomic config swap, drain enforcement

6. Logging (brief overview)

Two independent, easily confused mechanisms:

Mechanism Configured via What it writes
Application logs logback.xml regular SLF4J output (INFO/WARN/ERROR/…)
DICOM PDU dumps YAML router.log-folder + filter logLevel 5/6/7 binary PDU captures via PduLoggingStage

The two mechanisms are independent — router.log-folder does not control the application logs.

Enable PDU logging only temporarily. PDU dumps (logLevel 5/6/7) are a diagnostic tool: they contain the complete patient data (PHI) and are not rotated or deleted automatically — retention and deletion are the operator's responsibility. After the analysis, reset the logLevel and remove the dumps in router.log-folder. For permanent traceability, the database audit trail (configuration section audit) is the tool, not PDU dumps.

Logback discovery order (first match wins)

  1. -Dlogback.configurationFile=<path> — set by bin/start.sh to config/logback.xml (operator-overridable).
  2. logback-test.xml on the classpath — used automatically in test runs.
  3. logback.xml on the classpath — internal default in dicom-router-core/src/main/resources/logback.xml.

Internal logback.xml (verified)

Ships sensible defaults so that even a direct java -jar start gets file rotation:

  • CONSOLE appender (ConsoleAppender) + FILE appender (RollingFileAppender).
  • Rolling policy: maxFileSize 100 MB, maxHistory 30 days, totalSizeCap 5 GB.
  • File: ${LOG_DIR}/dicom-router.log, rotated to dicom-router-%d{yyyy-MM-dd}.%i.log.gz.
  • Logger levels: ch.immeditech INFO, org.hibernate WARN, org.jboss.weld WARN, root INFO.
  • org.hibernate.engine.jdbc.spi.SqlExceptionHelper is set to OFF: on SQL errors this logger emits the raw driver message at ERROR, and PostgreSQL renders constraint violations with the complete affected row (Detail: Failing row contains (…) — patient identifiers included) verbatim into that message. The audit trail, the Q/R tracking and the central events writer write their own normalized, SQLState-carrying line to the log for their database failures — no diagnostic information is lost there; the remaining database paths keep reporting their errors through their own error handling.

Logback status messages

Logback records its own configuration in an internal status list and prints it as soon as a warning or an error occurred while doing so — a file appender that cannot write to its directory, for instance. The router routes that output to stderr (StderrStatusListener, installed before the first logger call); without the redirect Logback writes it to stdout and pollutes the output of the offline modes. Only messages of WARN and above are printed. An explicitly set -Dlogback.statusListenerClass=… takes precedence and is never overwritten.

LOG_DIR

Environment variable, default /var/log/imconnect (the same directory as the other deployment defaults — systemd ReadWritePaths, router.log-folder), overrides the log directory in the internal and the external config. The bundled systemd unit sets it explicitly via the env file (EnvironmentFile=, see dicom-router.env.example).