Monitoring¶
Three mechanisms, from the inside out: a health check of the process itself, a ping to an external monitor on every cycle, and notifications about what an instance does.
Health check¶
The daemon writes a status file per instance on every completed cycle (successful or not: a failing provider is still activity, already reported through logging and backoff), by default under $TMPDIR/dnspatch-health (DNSPATCH_HEALTH_DIR overrides it).
dnspatch healthcheck re-reads the config to learn each instance's own interval, checks that every status file is fresh (at most twice the instance's interval old, at least 30s), and exits non-zero otherwise. It needs no shell or curl, which a distroless image does not have. Both the lightweight and the full images already run it as their HEALTHCHECK.
If the container's filesystem is read_only, mount /tmp (or wherever DNSPATCH_HEALTH_DIR points) as tmpfs, as compose.yml does; otherwise every write fails, and healthcheck reports the daemon as stuck even though it is working fine. A write failure never affects the DNS updates themselves, only the health check.
Monitoring pings¶
ping_url, set on an instance, is called on every completed cycle: a GET request on success, and the same URL with /fail appended on failure. It is compatible with Healthchecks.io and Uptime Kuma push monitors. Unlike the health check above, this reaches an external service: it works as a dead man's switch, alerting when the ping stops arriving even if dnspatch's own process and container stay up.
ping_url works in every build.
Notifications¶
A [notify.<name>] definition describes a message broker, and an instance that publishes to it sends events to it. By default that is one kind of event, a flip of the instance's status between success and failure; the events key chooses others (see Event types). The default is deliberately quiet, since a notification channel is for what deserves a human's attention, unlike the ping above, which needs a heartbeat on every cycle to work as a dead man's switch.
[notify.alerts]
type = "redis"
address = "${REDIS_URL}" # a redis:// URL; carries auth and the database index
events = ["status", "ip_change"] # optional; the default is ["status"]
Connecting instances to notifiers¶
Notifiers are defined the way retrievers and providers are, and each instance names the ones it publishes to in its notify list. An instance with no notify key publishes to every notifier the file defines, and notify = [] gives it none. Each definition is its own broker connection, shared by the instances that list it, with its own optional topic_prefix, so several can be of one type:
[notify.alerts]
type = "redis"
address = "${REDIS_URL}"
[notify.backup]
type = "redis"
address = "${REDIS_URL_BACKUP}"
topic_prefix = "backup.dnspatch."
[[instance]]
name = "home"
notify = ["alerts"] # only alerts
[[instance]]
name = "office"
notify = ["alerts", "backup"]
[[instance]]
name = "lab" # no notify key: alerts and backup
Event types¶
A notifier publishes the types listed in its events; without the key that is ["status"], a flip of an instance between success and failure. The types are:
status: the instance as a whole started failing or recovered.provider_status: one provider started failing or recovered.retriever_status: one retriever started failing or recovered.ip_change: an address was written to a provider and differs from the previous one.cycle: a cycle finished.lifecycle: an instance started or stopped.
Which types an instance gets, how to override them for one instance, the fields of every message and the situations in which each event arrives are on Notification event types.
What is published¶
The brokers publish a small JSON event to the channel dnspatch.events.<instance> (override the prefix with topic_prefix), and whatever is subscribed to it, a bot you write or a small relay service, decides what to do next. This keeps adding a new notification channel a change on the listener's side only, with dnspatch's config and binary untouched. The topic is the same for every type of event; tell them apart by the event field of the message. The shoutrrr notifier is the exception: it words the event as a message and sends it to a chat or push service itself, see Shoutrrr.
Redis¶
The redis notifier takes a redis:// URL as address. Redis Pub/Sub is fire-and-forget: a subscriber that is not connected when an event is published misses it, which is fine for state changes, since the next one (or the next ping_url or health check cycle) still gets through; with cycle or lifecycle a missed event is simply gone.
RabbitMQ¶
The rabbitmq notifier (address is an amqp:// or amqps:// URL) publishes to a durable topic exchange, dnspatch by default (exchange changes it), with dnspatch.events.<instance> as the routing key. Bind a queue to the exchange with the pattern you want (dnspatch.events.# for everything) and events wait there while the consumer is away; with no queue bound, the broker drops them. The daemon connects at startup and logs an error if the broker cannot be reached or refuses the credentials, but it keeps running: the connection is re-opened on the next event. A connection the broker or the network drops later is logged at once, as an error with the notifier's name, and not when the next event fails to go out.
MQTT¶
The mqtt notifier takes a mqtt://, mqtts:// (TLS), tcp://, ssl://, ws:// or wss:// URL as address, with the user name and password in it when the broker wants them. Every event is a message on the topic dnspatch/events/<instance>: MQTT separates topic levels with a slash, so the dots of topic_prefix and of the instance name become slashes. Subscribe to dnspatch/events/# for everything. qos (1 by default) sets the delivery guarantee, retain = true makes the broker keep the last event of each topic for a subscriber that joins later, and client_id is random unless you set it. The daemon connects at startup and logs an error if the broker cannot be reached or refuses the credentials, but it keeps running: the connection is re-opened on the next event. A connection the broker or the network drops later is logged at once, as an error with the notifier's name, and not when the next event fails to go out.
Shoutrrr¶
The shoutrrr notifier sends each event as a text message to chat and push services, with no bot or relay of your own. urls is a list of Shoutrrr service URLs, one per destination, and every event goes to all of them:
[notify.phone]
type = "shoutrrr"
urls = [
"telegram://${TELEGRAM_BOT_TOKEN}@telegram?chats=@channel-name",
"ntfy://ntfy.sh/${NTFY_TOPIC}",
]
Only the secret part of a URL needs to be in the environment: ${NAME} is replaced inside a string, so the rest of the URL (the service, the chat) stays readable in the file. The variables come from the environment of the process, which is what env_file: .env in Docker Compose sets up; see Docker.
The title is dnspatch: <instance> and the body says what happened, for example cf ipv4: 1.1.1.1 -> 2.2.2.2 for ip_change; topic_prefix has no use here. Every service of Shoutrrr is available, among them telegram, discord, slack, ntfy, matrix, gotify, pushover, smtp (e-mail) and generic (a webhook); the list and the URL format of each are in the Shoutrrr documentation. A URL is checked at startup, so a wrong scheme or a missing parameter stops the daemon with a message, and the URLs, which carry tokens, are cut out of the errors that reach the log.
The notifier is in the full build only, where it adds about 10 MB to the binary.
URL examples¶
Write the secret of a URL, the token or the password, as ${NAME} and keep its value in the environment or in a file (${file:./secrets/telegram_token.txt}), and the rest of the URL in the file.
| Service | URL |
|---|---|
| Telegram | telegram://${TELEGRAM_BOT_TOKEN}@telegram?chats=@channel-name,<chat-id> (channels is accepted for chats) |
| ntfy.sh | ntfy://ntfy.sh/${NTFY_TOPIC}: on the public server the name of the topic is all that protects it |
| ntfy on your own server | ntfy://ntfy.example.com:8080/dnspatch?scheme=http&token=${NTFY_TOKEN}, or user:${NTFY_PASSWORD}@ in front of the host when it wants a login |
| Discord | discord://${DISCORD_WEBHOOK_TOKEN}@<webhook-id>, the two parts of the webhook URL that Discord gives you |
[notify.phone]
type = "shoutrrr"
urls = [
"telegram://${TELEGRAM_BOT_TOKEN}@telegram?chats=@channel-name",
"discord://${DISCORD_WEBHOOK_TOKEN}@123456789012345678",
] # every event goes to both
events = ["status", "ip_change"]
[[instance]]
name = "home"
notify = ["phone"]
A character that means something in a URL (/, ?, #, @, a space) has to be percent-encoded in the value of the variable, such as %2F for a slash. The tokens of the services above do not have any.
What a message looks like¶
The title is dnspatch: <instance>; the body depends on the event:
| Event | Body |
|---|---|
ip_change |
cf ipv4: 1.1.1.1 -> 2.2.2.2, one line per changed address (none for the old one when the record had no address) |
status |
status: failure or status: recovery, and error: ... on a failure |
provider_status |
provider cf: failure, and error: ... |
retriever_status |
retriever ipify: failure, and error: ... |
cycle |
cycle succeeded or cycle failed, and error: ... |
lifecycle |
started 0.5.0 or stopped 0.5.0 |
If one service refuses a message the others still get it, and the error, with the secrets cut out, is logged.
Build requirements¶
A notifier needs a build that has its backend compiled in: the redis, rabbitmq, mqtt or shoutrrr tag for these, or full for every backend and every other plugin (the full binary and image use it); see Building from source. The lightweight build rejects a config whose instances use a notifier; a definition that no instance uses is ignored, and dnspatch --check-config shows the notifiers each instance publishes to. A build that lacks the backend a definition names says which tag brings it, and an error in one definition is reported by its name (notify "backup" (redis): ...).
Adding another backend is a plugins/notifiers/<backend> package that implements plugin.Notifier and registers itself in init, like a provider does; go generate gives it a build tag. See Writing a plugin.