Module 10 · Operating Portainer
Module 10 of 10

Operating Portainer.

The final module is a reference. Once a customer has Portainer running in production, the operator's job is a small set of recurring tasks (backups, patches, license, DR) plus a well-known list of failure modes. This module walks each one, then hands you the triage playbook you will come back to for years.

12 chapters Reference Come back to this
01

The operating model

Portainer is a control plane, and control planes have a small operating footprint. A running Portainer instance needs: a durable place to keep its data, a schedule for backing that data up, a plan for patching itself, a license that stays current, and a person or team who knows what to do when it stops behaving. Everything in this module falls under one of those five headings.

The other property to internalize is that a Portainer outage is not a workload outage. Managed environments continue running whether Portainer is up or down; Portainer is not in the request path for anything the environments run. If the Portainer server is down for an hour, applications keep serving, edge agents queue up their next poll, and nothing user-visible breaks. Management is unavailable during that hour; workloads are not.

This makes the operating model less panicky than it looks. You still want Portainer up, and you still respond to alerts, but you do not treat a Portainer restart as a P1. You treat it as a control-plane issue with the recovery time you would give any control plane.

02

Backup: what and where

A Portainer backup captures the datastore (the BoltDB file, all users, teams, environments, RBAC, policies, licenses, add-on records, alert rules, edge configurations, everything Portainer needs to remember) plus the TLS and SSL material. The archive can be password-protected; if it is, the password becomes required for restore.

Three backup targets are supported: local disk, S3 (or any S3-compatible endpoint including MinIO, Wasabi, Backblaze B2), and Azure Blob. Each target has its own scheduler; you can run several simultaneously (nightly to S3 and weekly to Azure Blob for cross-cloud redundancy, for example) rather than picking one.

The reference-architecture guidance is the 3-2-1 rule: three copies of the data, on two different media, with at least one off-site. Portainer's own local backup satisfies "one copy on the same host"; anything you would call production needs at least one target on different infrastructure. A local-only backup on the same VM as Portainer is a lab pattern, not a production one; if the VM dies, both copies die together.

Gotcha

Portainer does not rotate backups on its target. A daily backup to S3 will produce 365 archives after a year unless you configure a bucket lifecycle policy or a similar retention rule on the target side. Set that up when you set up the backup; do not discover it in year two.

In your lab

Configure a scheduled backup to S3 (MinIO on one of your VMs is a fine lab stand-in). Take an on-demand backup to test the credentials. Deliberately break something in Portainer (delete a test team, add and remove an environment). Restore. Confirm the state is what you expected.

03

Restore, and what it does not touch

Restore replaces the Portainer datastore. It triggers an in-process server rebuild, which is why the restore endpoint runs even when the server is otherwise unable to boot cleanly; you can restore from a broken state without a manual recovery dance.

What restore does not touch is external state. Registry credentials the agents already have baked in, edge keys already deployed to devices, installed add-on Helm releases in the cluster, GitOps repository state on the git server, workload state in the managed environments; all of that is external to Portainer's datastore. A restore rolls back Portainer's memory of what exists; it does not roll back the world.

This is what "restore succeeded but users report weirdness" almost always is. The customer restored a backup from before an environment was added; the environment still exists in the cluster, the workloads are still running, but Portainer does not know about it. Fix: re-add the environment, do not repeat the restore.

04

Upgrading Portainer manually

Manual upgrades are simple: pull the new image, restart the container (Docker) or roll the pod (Kubernetes). The persistent data directory carries over; on boot, the new binary runs the migration flow against the existing datastore.

Sequence to follow: take a fresh backup before the upgrade (not the scheduled one; a specific pre-upgrade one you can reach for if you need it); pull the target image; restart with the new image; watch the logs. The first line ("starting Portainer" with version, build number, image tag, Go version) tells you what actually loaded. If migration was needed, the log will show it; if migration failed, the log will show the failure and the automatic restore.

Cross-major-version upgrades follow the release notes for that version. Cross-minor upgrades are usually straightforward but check the notes; things like the 2.44-to-2.45 line moved GitOps polling intervals from stacks to sources, and that is a behavior change worth knowing about before an upgrade rather than during.

05

Auto-patch on the LTS line

2.45.0 is an LTS release: APIVersion is 2.45.0, APIVersionSupport is "LTS," and the auto-patch service is what keeps an instance current on its patch line without manual intervention. Auto-patch never crosses a minor. A 2.45.0 instance can auto-patch itself to 2.45.1, 2.45.2, and so on; it will never move to 2.46 automatically.

Configure through Settings.AutoPatchSettings.Enabled plus Settings.AutoPatchSettings.PatchCron. The default schedule is 0 4 * * * (04:00 UTC daily); customize it to a maintenance window that suits the customer. Empty or invalid cron falls back to the default.

How it works. The service ticks hourly and asks the OCI registry patch finder for the latest same-minor patch (registry-1.docker.io by default, repository portainer/portainer-ee, or a Portainer-registered registry). If none is found, nothing happens; if one is, the next scheduled apply time comes from the cron. When the schedule fires, the service takes a datastore backup first (in addition to the migration flow's own pre-migration backup), executes the platform-appropriate self-update, and records the outcome in the autoupdate dataservice. History is available at GET /api/auto_updates.

For air-gapped instances, register a Portainer registry that mirrors portainer/portainer-ee and point auto-patch at it. The patch finder will look at your registry instead of Docker Hub.

Sharp edge

Auto-patch enabled but nothing happening is one of three things: the cron next-fire time is in the future (check Next()), there is no patch available for the running minor version (the finder only looks at the same major.minor), or the registry is unreachable. Rule those out in that order.

06

Migration behavior

On every boot with a schema newer than the datastore, Portainer runs its migration flow. Understanding the flow is the difference between "the upgrade broke" and "the upgrade rolled back cleanly and now I have a defect to file."

The flow: if a previous migration left the IsUpdating flag set (meaning a previous attempt crashed mid-flight), boot fails with "database is updating"; you either restore the pre-migration backup Portainer took automatically, or use --force-rollback. Otherwise, an automatic backup of the DB file is taken before any migration; the migrations run inside a fail-safe wrapper that clears IsUpdating on success and automatically restores the pre-migration backup on failure.

What this means operationally: an upgrade that fails to migrate rolls itself back to the pre-upgrade state cleanly, and the customer is running the old version with a failure logged. That failure is the defect to escalate; do not attempt a workaround before you understand what the migration was doing. The log message "migration failed, restoring database to previous version" is the tell.

Rollback flags: --rollback steps one version back (CE-defined). --force-rollback forces restoration of the previous backup (EE, for genuinely broken states). --rollback-to-ce is the BE-to-CE downgrade path.

MIGRATION FLOW · CLEAN ROLLBACK ON FAILURE Boot portainer starts IsUpdating set? from prior crash Auto backup of DB file Run migrations fail-safe wrapper sets/clears IsUpdating Success clear flag run new version Restore backup roll back log defect · run old "database is updating" restore prior backup or --force-rollback no yes success fail
Every boot with a newer schema takes a backup first, migrates inside a fail-safe wrapper, and rolls back cleanly on failure. A migration failure leaves the customer on the previous version, with a clear defect to file.
07

License and node counting

License management is a recurring operational task. Portainer's license service starts with a check-in on startup and re-syncs on an interval, sending metadata to the licensing server: node counts, endpoint type distribution, CPU counts, auth method in use, and whether server-level Edge ID enforcement is on. This is the licensing telemetry channel.

Node counting matters for Free-tier licenses; only Free-tier abuse triggers enforcement. When the used node count exceeds the licensed count, an OveruseStartedTimestamp is recorded; if usage drops back under the limit, the timestamp resets. The grace period is 10 days; after that, ShouldEnforceOveruse() returns true.

Where enforcement bites. Non-admin login refuses with HTTP 402 "Node limit for this license has been exceeded, please contact your administrator" (both internal and LDAP paths). New environment provisioning is blocked. Administrators can always log in; that is the deliberate safety valve. If a customer reports "all my users are locked out but I can still log in," this is the diagnosis; walk the environment list, delete stale environments, and enforcement resets.

Multiple licenses can stack. Subscription licenses stack with subscription licenses; V2 Free stacks with V2 Subscription; other combinations conflict, and adding a conflicting license fails with the conflict listed (or force-replaces if you pass force). Air-gapped adds work because local validation still runs against the key; a network check-in failure alone is tolerated.

08

DR posture

Because Portainer's datastore is single-writer BoltDB (Module 3, chapter 07), the DR strategy is durable storage plus backup, not clustering. Two site failure scenarios are worth walking through explicitly.

Same-site DR. The Portainer host or pod fails; the scheduler restarts it on another node with the persistent volume following. Recovery time is the reschedule time (single-digit minutes on Kubernetes with a healthy PVC and node pool). No data loss; agents reconnect on the next poll after the server comes back.

Cross-site DR. The entire primary site is unavailable. Recovery is: at the secondary site, install Portainer at the same version, restore the most recent backup. Point DNS or a load balancer at the new server. Agents whose edge keys pointed at the original URL will keep polling the old URL until DNS resolves them elsewhere; if you can update DNS, agents follow. If you cannot (some corporate configs pin DNS at the agent side), you re-issue keys, which is manual work per environment.

The RTO and RPO depend on backup frequency and DNS mechanics, not on Portainer itself. A backup every 6 hours means RPO ~6 hours; DNS TTL determines how fast agents move. Design for the customer's actual DR requirements; do not oversell Portainer's DR properties.

09

Common failure modes

This is the concentrated form of the triage playbook. Symptom on the left, most likely subsystem on the right. Use this as a starting point; the earlier modules have the depth.

Symptom → subsystem
SymptomMost likely cause
All non-admin users get 402 on loginLicense overuse enforcement (Free tier past 10-day grace). Walk the environment list; delete stale environments to reset.
All non-admin users get 403 "License is not valid"License expired, revoked, or absent. Admins can still log in; use that to add or renew.
Edge device never appears in PortainerEdge key URL wrong, or tunnel address wrong. Regenerate with the correct --edge-tunnel-server-address.
Edge device shows online but tunnel operations fail with "Unable to open the tunnel"Tunnel port not reachable from the device, or a websocket-blocking proxy in between. Check --tunnel-port exposure and HTTP_PROXY/HTTPS_PROXY on the agent.
Agent stopped by itself ~3 days after installAGENT_SECRET_TIMEOUT (72h). Set AGENT_SECRET before deployment or associate with the server before the timeout.
Auto-update/webhook stopped workingStack author user missing (offboarded), change window suppressing deploys, or no commit change detected. Check in that order.
Policy shows NotSupported on some environmentsAgent below the policy type's minimum version (resource-patch types need 2.44.0+) or wrong environment type.
Policy never reaches async devicesasync-policies feature flag is off; without it, this is by design.
User's Kubernetes access changed but they still see stale permissionsPer-user Kubernetes client cache. Re-login clears it.
Add-on page redirects to login or 403User is not authenticated at the gateway, or the add-on's per-add-on access list does not include them.
Boot loop with "database is updating"Interrupted prior migration. Restore the automatic backup or --force-rollback.
Client IPs in auth activity logs are the load balancer, not the users--trusted-proxies not set to include the proxy.
Duplicate multi-severity alerts for one incidentAlertmanager inhibition label matching; custom labels break the canonical inhibition rules.
OneUptime logs endpoint returns auth errors on cluster-wide queriesCluster-wide queries require admin or edge-admin, or namespace-wide access; deliberate.

For anything more subtle than this, the reference file the internals skill contains for that subsystem is the next step.

10

Triage: what to collect from the customer

When escalating a real issue, either to your own team or to Portainer support, the collection list below covers 90% of what will be asked for. Collect it before opening the ticket; you save round-trips.

  1. Server boot log lines: the "starting Portainer" line has version and build number. If auto-patch is on, also include the last few entries from GET /api/auto_updates.
  2. Debug logs around the incident: toggle debug at runtime with PUT /api/support/debug_log; pull a bundle with POST /api/support/download. Do not ask the customer to restart with --log-level DEBUG; that loses the incident state.
  3. Agent logs (LOG_LEVEL=DEBUG) and the agent's full environment (redact secrets).
  4. Exact endpoint type and edge mode, and edge key provenance (when it was generated, what URL it points at).
  5. For license issues: node count from the license page, license type, whether admins can log in.
  6. For tunnel issues: the one-shot agent connectivity check output (EDGE_CONNECTIVITY_CHECK=1 with a bounded EDGE_CONNECTIVITY_CHECK_TIMEOUT).
  7. For GitOps issues: the workflow detail (the git_sync_error field on it), the source's interval, and the source's access list.
  8. For auto-patch issues: GET /api/auto_updates output and the AutoPatchSettings values from public settings.
11

The support bundle and runtime debug logging

Two operator tools worth naming out loud, because you will use them for every real triage.

The runtime debug log toggle (PUT /api/support/debug_log) turns verbose logging on or off without a restart. This is what lets you capture debug logs at the moment of an issue rather than after the fact. Turn it on when the customer reports the problem starting, capture, turn it off. Do not leave it on; debug logs get large fast.

The support bundle download (POST /api/support/download) produces a bundle containing logs, configuration snapshots, feature-flag state, and other diagnostic material Portainer support will want. This is the artifact to attach to any support ticket. Generate it fresh each time; a stale bundle from before the incident is not useful.

12

Where to go from here

You have reached the end of Lets Learn Portainer. If you have worked through every module and every lab, you can install and configure Portainer Business Edition in a real environment, onboard multiple environments across Docker and Kubernetes families, tune it for the operating model in front of you, and keep it running through backups, patches, license renewals, and the failure modes above. That is a competent Portainer operator, and it is what this course was written to produce.

Three suggestions for what to do next. First, run through the labs a second time on someone else's Portainer install (a colleague's, a partner's, a POC environment). The second time is where the mechanics become instinct rather than instruction. Second, put the course on the shelf as a reference; the middle modules (5 through 8) will be what you come back to when a real question lands and you want to answer with confidence. Third, when you find a claim in this course that has drifted with a Portainer release, or a section that is thinner than it should be, tell us. This course is versioned against Portainer 2.45.0 LTS and will move forward with the LTS line; every correction lands in the next version.

Thanks for reading. Now go build something real.