Author Topic: online data logging system  (Read 8372 times)

0 Members and 1 Guest are viewing this topic.

GaryGary

  • Sr. Member
  • ****
  • Posts: 411
    • Build-It-Solar
online data logging system
« on: May 22, 2013, 10:54:47 AM »
Hi,
Just wondering if anyone is aware of a simple and inexpensive system that will put logged data online in the form of charts?

There used to be a nice simple service called Pachube, but they seem to have morphed into to commercial product that is rather pricey.

The wish list would be: opensource, easy to setup, and not expensive.

The idea would be to be able to report ongoing results from projects like this one: http://www.builditsolar.com/Experimental/OffShelfDHW/OffShelfSolarDHWV2.htm

Thanks -- Gary



Dave B

  • Hero Member
  • *****
  • Posts: 1014
  • Country: 00
    • DCB Energy Systems
Re: online data logging system
« Reply #1 on: May 22, 2013, 05:18:11 PM »
Hello Gary,

  I am currently working on the same solution. Your site has given me many ideas and I would like to work together on this to help promote this to anyone interested. Initially so as not to clutter the board please email me at:  bruggelog at netsync dot net if you get a chance. My website is www.dcbenergy.com  Thank you Gary,  Dave B.
DCB Energy Systems
http://dcbenergy.com/

GaryGary

  • Sr. Member
  • ****
  • Posts: 411
    • Build-It-Solar
Re: online data logging system
« Reply #2 on: May 22, 2013, 09:54:19 PM »
Will do Dave -- Thanks
Gary

SparWeb

  • Global Moderator
  • Super Hero Member Plus
  • *****
  • Posts: 5452
  • Country: ca
    • Wind Turbine Project Field Notes
Re: online data logging system
« Reply #3 on: May 23, 2013, 03:54:38 PM »
I don't see why this has to be "private".  Discussing ideas is what the forum is for.  Not "clutter" to me.

The basic system goes something like this:
One host computer is running, that takes most of the action.
The host computer may passively receive data from the datalogger, or actively request it periodically.
When the data is received, it is stored by the host computer.
The host also has a separate program running which reads the stored file, prepares an update, and sends that to a remote internet server.
I think that is done by FTP.  It doesn't have to be complicated, but firewall on the host computer needs to be aware of what's going on.
The internet server's job is to host the data file and a script file that periodically reads the data file. 
If the script file finds an update, it prepares a refreshed HTML page, and overwrites the old copy of the HTML file.
The HTML file will refer to any numbers or graphs that were prepared by the program(s) on the host computer, and included in the update message.
A visitor who is viewing the HTML file will see the last update. 
Buried in the HTML code is a Java command that tells the visitor's web browser to check for updates periodically. 
When the time elapses, that visitor's browser will refresh with the updated HTML file.  Up pop the new graphs and statistics.

My weather station came with a suite of little programs that can do all this.  I've experimented with them but I haven't loaded the scripts onto my website.  I chose to use the other auto-update feature in my weather station's software instead, which doesn't require my home computer to be running all the time.

If I was to use it, it would look like this:
http://www.rigel.ca/Wx/Current_Vantage_Pro_Plus.htm

The page isn't complete or updated any more.  It doesn't even show the time & date of the last update for some reason.  Anyway it serves as an example of the simple HTML page that needs to be generated by the script, if you right-click to "view source".
No one believes the theory except the one who developed it. Everyone believes the experiment except the one who ran it.
System spec: 135w BP multicrystalline panels, Xantrex C40, DIY 10ft (3m) diameter wind turbine, Tri-Star TS60, 800AH x 24V AGM Battery, Xantrex SW4024
www.sparweb.ca

SparWeb

  • Global Moderator
  • Super Hero Member Plus
  • *****
  • Posts: 5452
  • Country: ca
    • Wind Turbine Project Field Notes
Re: online data logging system
« Reply #4 on: May 23, 2013, 03:59:54 PM »
I may not have emphasized the point I meant to make.  All of the programs needed to orchestrate this are common, some in the public domain, and not too difficult for someone already familiar with common programming languages to understand.
No one believes the theory except the one who developed it. Everyone believes the experiment except the one who ran it.
System spec: 135w BP multicrystalline panels, Xantrex C40, DIY 10ft (3m) diameter wind turbine, Tri-Star TS60, 800AH x 24V AGM Battery, Xantrex SW4024
www.sparweb.ca

frackers

  • Sr. Member
  • ****
  • Posts: 435
  • Country: nz
  • Picard spits "Hello"
Re: online data logging system
« Reply #5 on: May 23, 2013, 10:23:50 PM »
I'd recommend 'cacti'
Easy web setup, text input files of data to log, history of data can stretch back years.
A typical line of log data from my wind turbine controller:
23-05-13 15:20:00 E:0 L:1 S:0 F:9 D:0 T:17.97 C:999 V:26.19 A:-37.61 P:-985 R:173 r:260 H:143 Y:830 h:-1129 y:-1151 I:8265 O:7921

The meaning of all these fields can be found at https://github.com/g8ecj/turbine/blob/master/docs but for all that is readable without all the graphs!!

Example plots at http://gilks.ath.cx/cacti/graph_view.php?action=tree&tree_id=5 with a few interruptions due to our recent earthquake repairs ;D

Robin Down Under (Or Are You Up Over)

GaryGary

  • Sr. Member
  • ****
  • Posts: 411
    • Build-It-Solar
Re: online data logging system
« Reply #6 on: May 24, 2013, 09:33:52 PM »
I'd recommend 'cacti'
Easy web setup, text input files of data to log, history of data can stretch back years.
A typical line of log data from my wind turbine controller:
23-05-13 15:20:00 E:0 L:1 S:0 F:9 D:0 T:17.97 C:999 V:26.19 A:-37.61 P:-985 R:173 r:260 H:143 Y:830 h:-1129 y:-1151 I:8265 O:7921

The meaning of all these fields can be found at https://github.com/g8ecj/turbine/blob/master/docs but for all that is readable without all the graphs!!

Example plots at http://gilks.ath.cx/cacti/graph_view.php?action=tree&tree_id=5 with a few interruptions due to our recent earthquake repairs ;D


Thanks -- had a look through it, and I guess I was hoping for something simpler.
At my skill level it looks like a month long learning curve :)

Gary