History management : store averaged values ?

[touton]

Hello,

I'm testing netxms, first tests are quite promising

I'm doing some setup on the DCI History, I would like to keep data for at least 3 years. But it is useless to keep some data like CPU utilisation with a minute resolution.

Is there a way to keep after a defined period (ie: 1 month) an average value instead of a spot one ?

ie: I have a 60 seconds resolution of the last 30 days, then resolution is reduced to 30 minutes, after 1 year it is reduced to 1 h...

Thanks for your feedback

[sperlm]

Hello, not sure if this is the right way but I would create dummy DCI and transform collected values from the CPU (like once per week do a weekly average).

This is interesting functionality, will try to follow this idea and let you know if this is the way to go.

[victor]

Hi!

Yes, this is the way to go - create hierarchy of DCIs with source Internal/Dummy and transformation script reading average value from previous DCI in hierarchy. For example, for System.CPU.Usage parameter transformation script to calculate 10-minutes average will look like this:

now = time();
return GetAvgDCIValue($node, FindDCIByName("System.CPU.Usage"), now - 600, now);  // 600 is number of seconds

Best regards,
Victor

[touton]

I see, but it will be a huge job to do this for mostly all the DCI...

How the retention job is working ? Is it some kind of daily Schedule ? If data are modified here, it will allow all the reporting feature to work as-is, and also it will be esier to handle the dozen (hundreds) of DCI per node ?

[victor]

I agree that this is a workaround, not a full solution. I've added a feature request for built-in DCI data aggregation - should not be that hard to implement.

There are housekeeper job that starts at fixed intervals (controlled by server configuration parameter "HouseKeepingInterval"). It deletes all data that is older then retention time configured for DCI.

Best regards,
Victor

[sperlm]

now = time();
return GetAvgDCIValue($node, FindDCIByName("System.CPU.Usage"), now - 600, now);  // 600 is number of seconds

Hello, just informing you about results from testing this functionality.
I could not get results using FindDCIByname function because there simply is not "name" in DCI.
Following your explanation from the other topic...

"name" was the old universal name for what you now see as "Parameter" for agent DCI, "OID" for SNMP DCI, etc.

...does not seem right either. That would mean FindDCIByName looks into parameter and that is not really intuitive.

However, using FindDCIByDescription function with this functionality works like a charm.
I find using FindDCIByDescription better by far because I can set up the description myself (even in automatically created DCIs) and use it in scripts not relying on specific "name" of the DCI (if the "name" really means specific OID path), which can vary from device to device.

With Regards

Milan Sperl

[touton]

Today we use zenoss to store all the info from our machines. It stores about 3 or 4 years of data which helps us to evaluate trends or forecasts.

Zenoss also use a template mechanism to create DCI (and associate a correct  polling interval). For instance, for each interface, we have in/out speed / packets /Errors, you can imagine how much it can be with 48 ports switches (or worst). Zenoss, just resample the data automatically, so you have a hour view, then daily, weekly and yearly. Of course we don't have a seconds resolution when we browse data 2 years old, but it is far enough to see trends.

I'm waiting for the snmp table implementation to prepare templates for switches, computers, printers, sensors (I'll share all the template), but storage optimization will be required to limit  the impact.

Is it easy to select all the DCI values with date filter and re-compute them ? (ie : take x values, compute average and replace the values by only one )
We could be something like :
-take the last week values, and compute average per hour
-take the last year and compute average per days (or half day)