System Health

Administration → System Health → Overview shows on one page what the installation knows about itself: the health checks, version and uptime, the inSCADA process and the machine it runs on, the last 24 hours of resource history, the days left on the certificate and the licence, the cluster, the connections that are not connected, scripts, jobs, the most frequent errors and diagnostic files for support.

It is a status screen, not a setting: it changes nothing. The page does not refresh on its own; every reading runs every health check on the server, so it is read when it opens and when you press Reload. Checked at in the top right says when.

On a cluster each node answers for itself

The page shows the node that answered the request; the cards name it. Another node’s JVM, machine and checks show when the page is opened at that node’s own address.

Who may open it

Permission	What it allows
VIEW_SYSTEM_HEALTH	The page itself. Notification mails go to the accounts that hold it, too.
VIEW_SYSTEM_DIAGNOSTICS	The Diagnostics card: the server log, a thread dump, the support bundle.

Both are installation-wide and held by the default space’s administrator role; one process, one JVM, one certificate and one licence answer for every space, so this is the installation’s view. A tenant space’s administrator does not hold them, and sees their own space’s state where they already do (Job Monitor, the connection status screens).

Diagnostics is a permission of its own because the server log carries every space’s lines, account names and device addresses; VIEW_SYSTEM_HEALTH alone does not open it.

Status and “Checks that are not up”

The top of the page shows the installation’s overall status and, when it is not good, the list Checks that are not up:. The statuses, worst first:

Status	Meaning
Down	A check fails: a data store does not answer, the disk is practically full, the licence is invalid… HTTP 503 to an outside monitor.
Out of service	A check reports itself out of service. HTTP 503 to an outside monitor.
Warning	Something is heading for trouble, but the system works. Still HTTP 200 to an outside monitor.
Up	All is well.
Unknown	The check has no figure to read (not measured yet, not reported on this platform, not configured). It does not change the overall status and sends no mail.

The overall status is the worst of the checks. Connections, Jobs, Scripts, the resource history and the most frequent errors do not change it: they are shown, not judged.

Notifications

The line under the status says whether health changes can be e-mailed:

• “Health changes are e-mailed to N address(es).”
• “Notifications cannot be sent: the application e-mail is not configured.” — with an Open App Email settings link.
• “Notifications cannot be sent: no one with System Health access has an e-mail address.”
• Accounts with no address are counted: “N account(s) with System Health access have no e-mail address and are not notified.”

When, and what, is mailed is below: E-mail notifications.

Health checks

The checks show as tiles grouped by what they watch; each carries its status and the one detail that says most, and Show details opens the rest.

Group	Check	Warning	Down
Infrastructure	Database (configuration) — PostgreSQL, as this node uses it	—	No answer within 3 seconds, a connection failure, the connection pool exhausted, or not checked for 60 seconds
	Redis (live values) — live values, sessions, alarm state; on a cluster the leadership lease	—	The same rule
	InfluxDB (logged values) — the logged values	—	The same rule. A wrong password or a missing database shows under InfluxDB write loss, not here.
Machine	Disk space	—	Free space below the Disk full threshold (5 GB by default): PostgreSQL, InfluxDB and the log are about to stop writing
	Free disk space	Less than 10 % of the disk free	Never
	Machine memory in use	Above 90 % at every sample for 15 minutes	Never
	Machine CPU	Above 90 % at every sample for 15 minutes	Never
Security	TLS certificates	—	The served certificate is invalid (expired or not valid yet)
	Proxy security	—	Reverse-proxy mode, but the server does not read the forwarded headers
	LDAP connection (while LDAP is on)	—	The directory does not answer; Unknown when not configured
	License	—	The licence is invalid or expired
Cluster (on a cluster)	Cluster	A Sentinel or the arbiter does not answer	No leader
	Cluster replication	Nothing is being sent, or the standby has long been absent	A record permanently failed to replicate
Thread pools	Job thread pool, Script thread pool	The pool is 80 % full	The pool is 95 % full or shutting down
Data	InfluxDB write loss	Logged values were dropped before reaching InfluxDB	—
Web server	HTTP ports — the application’s own ports, tried from inside every 30 seconds. It does not look at device connections.	A port other than the main one (the sandbox, say) does not answer	The main port does not answer

A check this list does not name — one an installation adds — shows under Other.

The machine’s early warnings

Free disk space, Machine memory in use and Machine CPU say a shortage is coming, while there is still time to act. They answer Warning only, never Down: mail goes out, but nothing that routes on the response code takes the node out. For memory and CPU, “sustained” means every sample: one sample at or below the threshold in the window holds the warning back, and no warning is given before the window has filled — ten minutes after a start, say.

The thresholds and the window are under Server Settings → General → System Health and take effect at the next restart. On a disk smaller than 50 GB the 5 GB Disk full threshold comes before the 10 % warning; lower the threshold on a small installation.

This node and the JVM

The This node card: Node, Version, Build time, Started at, Uptime.

The JVM (inSCADA process) card is the inSCADA process alone: Heap (used of max), Heap committed, Non-heap used, Threads (live · peak · daemon), Processors, Load average (1 min) (none on Windows) and the Java version. It is a single reading; how these move over time is the Resource history.

Machine

The Machine card is the whole computer the node runs on, with every program on it, the databases too. PostgreSQL, InfluxDB and Redis usually run on the same machine, and the memory or CPU they take does not show on the JVM card.

Field	Description
Machine CPU	The load over all of the machine’s processors, over the last one-minute interval.
Machine memory in use	The total less the available memory. On Linux the kernel’s own estimate of available memory is read, page cache included — so a healthy server does not read as “almost no free memory”.
Swap in use	“No swap on this machine” on a machine without swap. Not reported on Windows.
Disk (data and logs)	The free space of the disk the Disk space check watches.

A figure that cannot be read is shown with its reason, never as zero (“Not reported on this platform.”, “Not measured yet: the first reading comes about two minutes after start.”). Inside a container the figures are the container’s limits, not the host’s; where the databases run in containers of their own, what they take is not visible from here.

Resource history (last 24 hours)

A snapshot cannot show a leak. So every node samples itself once a minute and keeps the last 24 hours in memory; the page draws each series with its change per hour (/h) over the Last hour and the Last 24 hours.

Group	Series	A steady rise means
inSCADA process (JVM)	Heap after GC	A memory leak: what stays held after each collection keeps growing
	Metaspace, Loaded classes	A class leak
	Code cache	The area for compiled (JIT) code is filling up
	Thread count	A thread leak
	inSCADA process CPU	If it stays high, inSCADA itself keeps the processors busy
	Time in GC	Memory pressure
	Database / Redis / InfluxDB response time	The data store is slowing down
Machine	Machine CPU	If it stays high, the machine is busy — inSCADA or a database
	Machine memory in use	Something on the machine is taking more and more memory
	Swap in use	The machine is short of memory and moving it to disk
	Free disk space (data and logs)	A fall: logs or data are filling the disk

The change is the slope fitted to every sample in the window, so no single sample decides it. Without data covering at least half the window it reads “Not enough data yet”. The curves are shown, not judged: apart from the machine’s memory and CPU warnings, no series raises a warning or a mail.

A restart empties the history

The history is in memory and a restart empties it — and a restart is usually exactly how a leak ends. So every five minutes the latest sample is also written to the server log as one JVM trend: line; the 30-day log then answers “since when has it been growing” after the restart.

Certificate and licence

The Certificate card shows the name of the certificate the server serves, Valid until and the days left; orange from 30 days, red once expired. When the server serves no TLS of its own (behind a proxy) it says so; the proxy’s certificate is then on the Certificate card of Server Settings (Open Server Settings).

The License card shows Type, Expires and the days left; orange from 30 days, red from 7. A licence that does not expire reads Does not expire.

Cluster

On a single server, “This node runs standalone.” On a cluster: this node’s role (Leader / Standby), the Active server, the Nodes and any expected node not seen (”… absent for …”), Replication lag, Failed records, Conflicts. Replication in detail is under System Health → Replication Consistency (Open Replication Consistency).

Connections

The connections an operator started and did not stop — across every space: Started, Connected, Not connected. Those not connected are listed by space, project and name; the State column:

State	Meaning
Could not start	Its start failed; it is retried at growing intervals, up to once a minute.
Connecting	A start is under way, or the first attempt to reach the device has not ended.
Link down	The connection runs, but its link to the device is down.

There is no “since when”: the product does not track when a link went down.

Last resume of started connections shows how the started connections were resumed when the service started (or, on a cluster, when leadership was taken): the Trigger (At start-up / On taking leadership), Finished at and the Result (how many were resumed, failed, skipped, unresolved). While it is still under way, the page says a connection may not have been started yet.

Scripts (last 24 hours)

How this node’s script runs ended: Runs, Failed runs — including Timed out (stopped by the Script time limit), Rejected (pool full) and Statement limit exceeded —, Scripts that failed, and Scripts that failed most (at most 5, with Last failed). Saved scripts, callApi calls from screens, frame expressions and report datasources are all counted; debug sessions are not.

Failing now lists, by name, the expressions failing in the connections right now: Value expressions, Log expressions, and Frame scripts that fail as a whole — with Connection, Message and Failing since. A single variable whose expression fails inside a frame script that ran shows nowhere else: the variable keeps its last value and no alarm fires.

The counts are kept in this node’s memory; a restart starts them again. On a cluster nearly every script runs on the leader.

Jobs

The freshness of the jobs scheduled on this node, every project at once. Three tabs — a job can be on more than one:

Tab	When
Failing	Its last executed run failed and it has not succeeded since.
Skipping	It skipped a trigger in the last 24 hours: the previous run was still running, or the job pool was full. A skip is not a failure — a slow job is not a broken one. Skipped and Skip % are counted since the job was scheduled on this node.
Stale	A periodic or fixed-delay job with no successful run for longer than 10 periods (and at least 5 minutes). A run that hangs and never ends records neither a success nor a failure; this label is what catches it. Daily, monthly and once jobs get no stale verdict.

The columns use the same figures as Job Monitor (Last success, Last failure, Stale since…).

On a standby

Connections, jobs and expressions run on the leader only. On a cluster a standby says “This node is a standby: … run on the active server, not here.” in these sections rather than an empty list, which would read as “nothing is wrong”.

Errors and diagnostics

Most frequent errors (last 24 hours)

The ten errors repeated most in the last 24 hours in the event log — the one the Log screen shows — across every space. Lines whose messages differ only in numbers, quoted values or ids are counted together (Pattern); each group shows Count (at least), First seen, Last seen, in how many spaces it appeared and its newest sample line. Counts are lower bounds. The answer is kept for a minute.

Diagnostics

The evidence of a fault, taken from the application while the problem is still there (needs VIEW_SYSTEM_DIAGNOSTICS):

• Server log — the last lines of the server’s own log file (inscada.log); choose how many Lines. This is not the project event log the Log screen shows.
• Log files — the current file and its rotated pieces, with their size and last change, each with Download.
• Download thread dump — what every thread of the JVM is doing at that moment.
• Download support bundle — one zip to send to support: the System Health report, the resource history, the most frequent errors, the node’s effective configuration (secrets masked), a thread dump and the last 24 hours of server logs (up to 100 MB). Older log files are downloaded one by one from the list.

Take them before restarting

A restart ends the threads a thread dump would show. With a hang or a slowdown, download the thread dump or the support bundle first, then restart.

On a cluster, each node’s logs and thread dump are taken at that node’s own address. Every download leaves a line in the server log saying who downloaded what.

E-mail notifications

Nobody watches a screen all day. So each node evaluates itself once a minute and mails the administrators when something changes.

Who is written to, and through what:

• E-mail only, through the installation’s App Email account (Settings → Login → App Email). A space’s own mail settings and SMS are never used.
• The enabled accounts that hold VIEW_SYSTEM_HEALTH — in practice the default space’s administrators and anyone its roles grant it to. An account with no address is counted and not written to; two accounts with one address get one mail.
• When the application e-mail is not configured, or nobody has an address, nothing is sent and nothing is queued; a change that happens meanwhile is not mailed later. The notification line at the top of the page says so.

What is mailed:

Event	Rule
The checks that are not up change — a check goes Warning, Down or Out of service, moves between them, or comes back	Once the new status has held for 5 minutes, each check on its own clock. A change that reverts within those 5 minutes sends nothing. A second check failing while the overall status is already Down is mailed too, as is one recovering; when nothing is left, the mail is the all-clear.
The served certificate’s days left reach 30, 7 and 1	Once per threshold; a restart does not mail the same one again, and a renewed certificate starts over.
A cluster node is not seen, and is seen again	Once it has lasted 5 minutes; mailed by the leader only, so a cluster of three does not mail one loss twice.
Jobs turn stale, and recover	In one mail, at most 20 of each with the true count — an outage that stales 200 device polls sends one mail.

Unknown is not mailed. At most one mail of each kind every 10 minutes; a held change is not lost, because the next evaluation compares what holds now with what was last mailed. A sustained load is mailed about 20 minutes after it began (15 to warn, 5 to settle).

The mail’s subject carries the overall status and the node; its body lists what changed and since when (in the plant’s time zone). When the Application address is set to an address reachable from elsewhere, the mail also names the installation and links to System Health. It is written in the installation’s language, else English, and carries no details, stack trace or secret.

Watching it from outside

This page and its mails run inside the process they report on. Three failures they cannot report, because the reporter goes with them:

• Death — a process that exits sends no mail. The Windows service manager restarts it, but a node that crashes on every start restarts in a loop, and nothing says so.
• A hang — a JVM that stops answering but has not exited is not restarted by the service manager, and the notifier’s thread is as stuck as the rest.
• Network loss — a node that is fine but cannot be reached from where the operators are, or cannot reach its mail server, is silent either way.

So a monitoring tool outside the product (PRTG, Zabbix, Uptime Kuma, any HTTP monitor) has to ask the node on a schedule. Probe each node directly (https://<node>:8082 by default), not the address users reach through a proxy: on a cluster the proxy sends everything to one node, and the other is the one you would not hear about.

/actuator/health/readiness — availability

This is the probe; its status code is the verdict:

State	HTTP	Body (no credential)
Healthy	200	{"status":"UP"}
Degraded: a check is at Warning, none is down	200	{"status":"WARNING"}
A check is Down or Out of service	503	{"status":"DOWN"} (or OUT_OF_SERVICE)
Dead, hung, unreachable	none — the connection is refused or times out	—

Warning deliberately answers 200: a monitor that reads only the status code treats a degraded node as up. PostgreSQL, Redis and InfluxDB are each critical: any one of them Down makes readiness answer 503.

/actuator/health — a second, softer monitor

Raw health also catches what readiness lets through on purpose: any Warning, and replication loss on a cluster. Alarm, at a lower severity, when its body is not "status":"UP". On a cluster, do not use raw health for availability or load balancing: when Cluster replication goes Down over one record, raw health answers 503 although the node is serving the plant perfectly well; readiness leaves that check out.

On a cluster, /actuator/leader says which node is active (200 on the leader, 503 on the standby). What is worth an alarm is no node answering 200; with a proxy in front, also probe the plant’s own address as a third monitor.

The credential

/actuator/health, /actuator/health/readiness and /actuator/leader are read without a credential and answer the status line only. With the Monitoring user and Monitoring password (Server Settings → General) every check’s status and details come too — to put the failing check’s name into the alarm text. The cost: a request carrying a wrong password is refused even on health (401), so a password changed on the node and not in the monitor reads as an outage. A monitor needs no credential.

Recommended settings

	Availability (readiness)	Degraded (raw health)	Through the proxy (cluster)
Interval	60 s	5 min	60 s
Timeout	30 s	30 s	30 s
Alarm after	3 failures in a row	3 failures in a row	3 failures in a row
Expect	HTTP 200	HTTP 200 and "status":"UP" in the body	HTTP 200

Three minutes rides out a restart and a long pause without paging anybody, and still says “down” well before an operator would notice a frozen screen. The timeout is wide on purpose: a node whose peer or directory has gone silent answers late but with the verdict, and a 10-second timeout would turn that into a false “unreachable”.

The node’s certificate is often the installation’s own (self-signed, or from a private CA). Give the monitor that CA, or switch off certificate verification for these monitors; the certificate’s expiry is mailed by the product.

Send the monitor’s alerts somewhere else

Point every monitor’s notification at something other than the server it watches: a mail sent through the node’s own mail server is the one thing a dead node cannot do.

Related

• Server Settings — the System Health thresholds, the monitoring user, the certificate
• App Email — the account notifications are sent through
• Job Monitor — job figures, per space
• Log — the project event log
• Redundant Architecture — the cluster and what the load balancer probes