jeudi 7 septembre 2017

compress measurement data for charts, stored in MySQL

problem: My application collects ~1000 datasets of different measurements every 5 minutes. so every hour I collect 12.000 new datasets.

I need to export these data for charts: 1h, 24h, 1 day, 1 week, 1 month, 3 months, 6 months, 1 year, alltime

so safe some storage and for better performance, I want to compress these data.

Idea: After 1 day, I can breakdown these data. Therefor I want to calc the avg of the 12 "5 min. measurements" to get a value for 1 hours.

After maybe 1 week I can do the same for all "1h measurements" and calc the avg of 24 of them to get the avg value of the day.

I'm working in laravel (5.4). The database looks like:

id | identifier (unique) | value (float) | created_at (date)

now I'm looking for a performant algorithm to handle the big amount of data.



via Chebli Mohamed

Aucun commentaire:

Enregistrer un commentaire