ZFS FileSystem

[rainerh]

Can I watch the ZFS filesystem like
# zpool ststus
I want to get the status, ih the ZFS FileSytsem is degarded or online or in a healthy way
Thank you very much
Rainer

[Filipp Sudanov]

Currently you can configure something like
ExternalMetric=zpoolState:zpool status | grep "state: .*"
in agent configuration file. This will add zpoolState metric, which would return e.g. " state: DEGRADED", your add something after grep to cut "state: " part.
It's also possible for the metric to have a parameter, which will be given to the executed command
ExternalMetric=zpoolState(*):zpool status $1 | grep "state: .*"
this way you can get value of zpoolState(POOL_NAME)

In the future - ZFS v 2.3. will support json output (https://github.com/openzfs/zfs/issues/2626). And in v. 5.2. of NetXMS we will have new external parameter mechanism that parses json, xml and plain text, so all that can be used to get zfs state.

[rainerh]

Thank you for the tip.
I have done in nxagent.conf:

ExternalMetric=zpoolCOUNTER:zpool list | tail -n +2 | wc -l
ExternalMetric=zpoolStateRAID(*):zpool status $1 | grep -E "raid|mirror" | awk '{print $2}'

2 problems:
(1)
line 2 resolves not only ONLINE, but in console it will work
(the complete line is shown)

(2)
can I use zpoolCOUNTER  für Agent List in Instance Discovery?

Thank You
Rainer

[Filipp Sudanov]

By the way, can you show what ouput zpool_influxdb command produces on your system? It's written that it has machine-readable format, but I was not able to find an example of output.

[rainerh]

console output:
root@pve110:~# zpool status rpool | grep -E "raid|mirror" | awk '{print $2}'
ONLINE
root@pve110:~# zpool status datastore01 | grep -E "raid|mirror" | awk '{print $2}'
ONLINE

NetXMS output:
zpoolStateRAID(datastore01) =   raidz2-0                                      ONLINE      0    0    0
zpoolStateRAID(rpool) =   mirror-0                                          ONLINE      0    0    0

[rainerh]

Sorry about the next lines
It´s a lot.

root@pve110:~# zpool_influxdb
zpool_stats,name=datastore01,state=ONLINE,vdev=root alloc=7922084216832u,free=7436718833664u,size=15358803050496u,read_bytes=9190572863488u,read_errors=0u,read_ops=1743624457u,write_bytes=75531624361984u,write_errors=0u,write_ops=2847279442u,checksum_errors=0u,fragmentation=0u 1736494698741419432
zpool_stats,name=datastore01,state=ONLINE,vdev=root/raidz-0 alloc=7922084216832u,free=7436718833664u,size=15358803050496u,read_bytes=9190572863488u,read_errors=0u,read_ops=1743624457u,write_bytes=75531624361984u,write_errors=0u,write_ops=2847279442u,checksum_errors=0u,fragmentation=14u 1736494698741419432
zpool_stats,name=datastore01,state=ONLINE,path=/dev/disk/by-id/ata-KINGSTON_SEDC600M1920G_50026B728202A8DA-part1,vdev=root/raidz-0/disk-0 alloc=0u,free=0u,size=0u,read_bytes=1149090598912u,read_errors=0u,read_ops=217997528u,write_bytes=9441822150656u,write_errors=0u,write_ops=355221542u,checksum_errors=0u,fragmentation=0u 1736494698741419432
zpool_stats,name=datastore01,state=ONLINE,path=/dev/disk/by-id/ata-KINGSTON_SEDC600M1920G_50026B728202A966-part1,vdev=root/raidz-0/disk-1 alloc=0u,free=0u,size=0u,read_bytes=1148579872768u,read_errors=0u,read_ops=217868646u,write_bytes=9441236426752u,write_errors=0u,write_ops=355597227u,checksum_errors=0u,fragmentation=0u 1736494698741419432
zpool_stats,name=datastore01,state=ONLINE,path=/dev/disk/by-id/ata-KINGSTON_SEDC600M1920G_50026B728202A984-part1,vdev=root/raidz-0/disk-2 alloc=0u,free=0u,size=0u,read_bytes=1149143625728u,read_errors=0u,read_ops=218012588u,write_bytes=9441759522816u,write_errors=0u,write_ops=355689495u,checksum_errors=0u,fragmentation=0u 1736494698741419432
zpool_stats,name=datastore01,state=ONLINE,path=/dev/disk/by-id/ata-KINGSTON_SEDC600M1920G_50026B728202A99A-part1,vdev=root/raidz-0/disk-3 alloc=0u,free=0u,size=0u,read_bytes=1148693905408u,read_errors=0u,read_ops=217894259u,write_bytes=9441218277376u,write_errors=0u,write_ops=355668995u,checksum_errors=0u,fragmentation=0u 1736494698741419432
zpool_stats,name=datastore01,state=ONLINE,path=/dev/disk/by-id/ata-KINGSTON_SEDC600M1920G_50026B7686B80675-part1,vdev=root/raidz-0/disk-4 alloc=0u,free=0u,size=0u,read_bytes=1149027926016u,read_errors=0u,read_ops=218037401u,write_bytes=9441705701376u,write_errors=0u,write_ops=356111426u,checksum_errors=0u,fragmentation=0u 1736494698741419432
zpool_stats,name=datastore01,state=ONLINE,path=/dev/disk/by-id/ata-KINGSTON_SEDC600M1920G_50026B7686B806C5-part1,vdev=root/raidz-0/disk-5 alloc=0u,free=0u,size=0u,read_bytes=1148363948032u,read_errors=0u,read_ops=217934364u,write_bytes=9441133146112u,write_errors=0u,write_ops=356187147u,checksum_errors=0u,fragmentation=0u 1736494698741419432
zpool_stats,name=datastore01,state=ONLINE,path=/dev/disk/by-id/ata-KINGSTON_SEDC600M1920G_50026B7686B8080E-part1,vdev=root/raidz-0/disk-6 alloc=0u,free=0u,size=0u,read_bytes=1149131272192u,read_errors=0u,read_ops=217975482u,write_bytes=9441662730240u,write_errors=0u,write_ops=356371097u,checksum_errors=0u,fragmentation=0u 1736494698741419432
zpool_stats,name=datastore01,state=ONLINE,path=/dev/disk/by-id/ata-KINGSTON_SEDC600M1920G_50026B7686B8080F-part1,vdev=root/raidz-0/disk-7 
...
...
zpool_io_size,le=4194304,name=rpool,path=/dev/disk/by-id/nvme-eui.00000000000000008ce38e1001c168c3-part3,vdev=root/mirror-0/disk-1 sync_read_ind=0u,sync_write_ind=0u,async_read_ind=0u,async_write_ind=0u,scrub_read_ind=0u,sync_read_agg=0u,sync_write_agg=0u,async_read_agg=0u,async_write_agg=0u,scrub_read_agg=0u,trim_write_ind=208u,trim_write_agg=0u,rebuild_write_ind=0u,rebuild_write_agg=0u 1736494698744262723
zpool_io_size,le=8388608,name=rpool,path=/dev/disk/by-id/nvme-eui.00000000000000008ce38e1001c168c3-part3,vdev=root/mirror-0/disk-1 sync_read_ind=0u,sync_write_ind=0u,async_read_ind=0u,async_write_ind=0u,scrub_read_ind=0u,sync_read_agg=0u,sync_write_agg=0u,async_read_agg=0u,async_write_agg=0u,scrub_read_agg=0u,trim_write_ind=218u,trim_write_agg=0u,rebuild_write_ind=0u,rebuild_write_agg=0u 1736494698744262723
zpool_io_size,le=+Inf,name=rpool,path=/dev/disk/by-id/nvme-eui.00000000000000008ce38e1001c168c3-part3,vdev=root/mirror-0/disk-1 sync_read_ind=0u,sync_write_ind=0u,async_read_ind=0u,async_write_ind=0u,scrub_read_ind=0u,sync_read_agg=0u,sync_write_agg=0u,async_read_agg=0u,async_write_agg=0u,scrub_read_agg=0u,trim_write_ind=8060u,trim_write_agg=0u,rebuild_write_ind=0u,rebuild_write_agg=0u 1736494698744262723
zpool_vdev_queue,name=rpool,vdev=root sync_r_active=0u,sync_w_active=0u,async_r_active=0u,async_w_active=0u,async_scrub_active=0u,rebuild_active=0u,sync_r_pend=0u,sync_w_pend=0u,async_r_pend=0u,async_w_pend=0u,async_scrub_pend=0u,rebuild_pend=0u 1736494698744262723
root@pve110:~#

[Filipp Sudanov]

@rainerh said in ZFS FileSystem:

console output:
root@pve110:~# zpool status rpool | grep -E "raid|mirror" | awk '{print $2}'
ONLINE
root@pve110:~# zpool status datastore01 | grep -E "raid|mirror" | awk '{print $2}'
ONLINE

NetXMS output:
zpoolStateRAID(datastore01) =   raidz2-0                                      ONLINE      0    0    0
zpoolStateRAID(rpool) =   mirror-0                                          ONLINE      0    0    0

looks like awk part did not work at all. How is it specified in agent config file, could be some issue with quotes.
You can also do something like this:

zpool status datastore01 | grep -E "raid|mirror" | tr -s " " | cut -d" " -f 4

Another option is to have all the command in a separate .sh file and just call that.

For zpool_influxdb, may be adding --no-histogram will print less lines. Also looks cumbersome to parse, you can try something like this:

zpool_influxdb | "vdev=root/raidz-0\ " | cut -d',' -f 3 | cut -d'=' -f 2

If you can extract names of HDDs from it's output, you can use that as external list (https://netxms.org/documentation/adminguide/agent-management.html#externallist) - this way you could create DCIs automatically for HDDs.

[rainerh]

Thank you so much 
The following in /etc/nxagentd.conf will work fine:

ExternalMetric=zpoolCOUNTER:zpool list | tail -n +2 | wc -l
ExternalMetric=zpoolNAME(*):zpool list | tail -n +2 | sed -n "$1p" | cut -d' ' -f1
ExternalMetric=zpoolStateSTORE(*):zpool status $1 | grep -E "state" | tr -s " " | cut -d" " -f 3
ExternalMetric=zpoolStateRAID(*):zpool status $1 | grep -E "raid|mirror" | tr -s " " | cut -d" " -f 3

with result:
zpoolCOUNTER = 2
zpoolNAME(1) = datastore01
zpoolNAME(2) = rpool
zpoolStateRAID(datastore01) = ONLINE
zpoolStateRAID(rpool) = ONLINE
zpoolStateSTORE(datastore01) = ONLINE
zpoolStateSTORE(rpool) = ONLINE

Can you please help me, when I want to use  zpoolCOUNTER = 2 for instance discovery in

Metric
zpoolStateRAID({instance})

Display name
ZFS RAID Status of Pool {instance-name}

Result schould be
ZFS RAID Status of Pool datastore01 = ONLINE
 ZFS RAID Status of Pool rpool = ONLINE

How to configure Instance Discovery?
Instance discovery methode:
Agent List (???)

List Name
zpoolCOUNTER (???)

But this will not work

Thank you very very much
Rainer

[rainerh]

Here is the final solution to Watch Linux ZFS Filesystem in Proxmox Server
It can be used for ZFS mirror and raid systems

Configuration of agent´s config file (/etc/nxagentd.conf):
add these lines
ExternalList=zpoolNAME:zpool list | tail -n +2 | cut -d' ' -f1
ExternalMetric=zpoolStateSTORE(*):zpool status $1 | grep -E "state" | tr -s " " | cut -d" " -f 3
ExternalMetric=zpoolStateRAID(*):zpool status $1 | grep -E "raid|mirror" | tr -s " " | cut -d" " -f 3

Here is the configuration of ZFS RAID status:

Metric
zpoolStateRAID({instance})

Display name
ZFS: RAID Status of Pool {instance}

no Transformation

Instance discovery methode
Agent List

List name
zpoolNAME

Threshold:
Last polled value not like ONLINE

Here is the configuration of ZFS STORE status:

Metric
zpoolStateSTORE({instance})

Display name
ZFS: STORE Status of Pool {instance}

no Transformation

Instance discovery methode
Agent List

List name
zpoolNAME

Threshold:
Last polled value not like ONLINE

Output:
ZFS: RAID Status of Pool datastore01 = ONLINE
ZFS: RAID Status of Pool rpool = ONLINE
ZFS: STORE Status of Pool datastore01 = ONLINE
ZFS: STORE Status of Pool rpool = ONLINE

In my case datastore01 is a raid2-0 system and rpool a mirror-0 system

This works fine for me
Thank you for helping
Rainer