Hi,
No problems, and running pvesh from the agent works fine. On the second question the choice isn't quite what it looks like though, because with NetXMS the REST API route is itself an agent feature.
pvesh works from the agent for a reason worth knowing about: it invokes API functions directly without going through the REST/HTTPS server, and it requires root. The packaged agent runs as root - the systemd unit does not drop privileges, and the agent changes uid only if you pass -u/-g or set UserId/GroupId in nxagentd.conf. So it works with no extra setup, which also means anyone who can add a DCI can make the agent fork a root command on the hypervisor. Worth deciding that deliberately rather than finding it out later.
What will actually bite you is process spawning. ExternalParameter (ExternalMetric in current naming) forks a process per metric per poll, and pvesh is a Perl program that loads the PVE API modules on every invocation. One metric per guest on a 60 second interval turns into a continuous stream of interpreter startups on the host. The default execution timeout is 5 seconds (ExecTimeout, or ExternalMetricTimeout for these specifically), and a slow pvesh call will hit it and get killed.
Use ExternalDataProvider instead - added in 5.2.0. It runs the command once per polling interval, caches the output, and serves any number of metrics out of the cached document with jq queries, evaluated inside the agent so there is no jq binary to install:
[ExternalDataProvider/PVE]
Command = ["/usr/bin/pvesh", "get", "/cluster/resources", "--type", "vm", "--output-format", "json"]
PollingInterval = 60
Timeout = 30000
[ExternalDataProvider/PVE/Metrics]
PVE.Guest.Status(*) = .[] | select(.vmid == $1) | .status
PVE.Guest.CPU(*) = .[] | select(.vmid == $1) | .cpu
PVE.Guest.CPU(*).dataType = float
PVE.Guest.Memory(*) = .[] | select(.vmid == $1) | .mem
PVE.Guest.Memory(*).dataType = uint64
[ExternalDataProvider/PVE/Lists]
PVE.Guests = .[] | .vmid
That is one pvesh call a minute for the whole cluster no matter how many DCIs you build on top, and PVE.Guests gives you instance discovery over vmid. Two details: Timeout here is in milliseconds, and metrics with parameters ($1) need agent 6.0 or newer - on 5.2 define each metric with a fixed query instead.
On agent versus REST: there is no version of the REST route that doesn't involve an agent. Web service DCIs are executed by an agent acting as web service proxy - the server never makes the HTTP request itself. If the node has no web service proxy configured it falls back to the zone proxy, and finally to the management server node, so "just using the REST API" in practice means "the server's own agent makes the call". The choice is which agent and where it sits, not whether there is one.
Consequences:
Responses are cached per URL for the retention time set on the web service definition, so every DCI reading out of the same document costs one HTTP request. Against /cluster/resources that is a single call per interval covering every guest in the cluster.
The extraction is identical either way - web service DCIs and ExternalDataProvider share the same jq-based extractor, so the queries you write don't change with the route.
Only an agent on the PVE host can reach 8006 on localhost, which matters if that port isn't open to the server. Poll from an agent elsewhere and you install nothing on the hypervisor at all.
So the real trade is narrow. pvesh saves you managing an API token, at the price of forking a root-privileged Perl process on the hypervisor. The web service route costs you a token and gives you a read-only credential in exchange (PVEAuditor is enough), one HTTP call per interval, and the freedom to poll from an agent that isn't on the PVE host. I'd take the REST API as the default and keep pvesh for anything the HTTP API doesn't expose.
One thing has changed since my post above. The OTLP route I said didn't work now does - the receiver accepted protobuf only, which is why PVE 9's OpenTelemetry metric server couldn't reach it, and that is fixed in 6.2.3 (https://github.com/netxms/netxms/issues/3482), released on 9 August. On 6.2.3 or later you can point Datacenter - Metric Server - OpenTelemetry straight at the NetXMS web API and have PVE push guest metrics on its own: no collector in between, no polling, and no agent on the host. It needs an auth token passed through the otel-headers option, and the receiver is loaded as a server module (Module = otlp in netxmsd.conf). None of the ingestion side is documented yet - https://github.com/netxms/netxms-doc/issues/67 covers that.
