Author Topic: Datalogger update (Read 5974 times)

SparWeb · « **on:** December 21, 2021, 06:20:30 PM »

My datalogger has been ticking along for about 5 years happily.

With a refresh on the turbine, I also wanted to refresh the logger.

The biggest change is eliminating the extraneous current sensors. When I first built it, I was ambitious and gave it the capacity for 4 current sensors, theorizing that I could monitor electric current being generated by solar and wind, and used by the inverter and diversion load simultaneously. Later I realized that "ring" current sensors aren't good at measuring current that isn't continuous DC, and didn't bother.

So now it only needs 1 current sensor, wired directly to the box through a new connector.

It was also an opportunity to look through all the code (>500 lines) carefully and prune out stuff no longer needed.

Once that was done, I went for the "nice to have" feature that I've always wanted. So now, in addition to just reporting measurements, the Arduino calculates the cumulative Watt-hours to display on the LCD. I'm looking forward to watching that number grow and grow!

DamonHD · « **Reply #1 on:** December 22, 2021, 01:35:47 PM »

Deleting obsolete stuff from code can be satisfying!

Rgds

Damon

Bruce S · « **Reply #2 on:** December 22, 2021, 01:46:31 PM »

Quote from: DamonHD on December 22, 2021, 01:35:47 PM

Deleting obsolete stuff from code can be satisfying!

Rgds

Damon

But only after you've made a triple check and triple backups .
HTML and notepad++ is my "friend" :-)

Bruce S

DamonHD · « **Reply #3 on:** December 22, 2021, 02:29:18 PM »

Or a good version control system so that you can go back to any of the previous 42987 edits that you checked in over the last decade or two.

(Yes, that really is the live revision count for the repo that includes my earth.org.uk site for example: all manual commits.)

Oh, and a backup or two also, offsite!

Rgds

Damon

SparWeb · « **Reply #4 on:** December 22, 2021, 03:07:35 PM »

A lot of the "prehensile" code got commented during my edits; still there but inactive. The program ran a few times in the Arduino before those bits were wiped.
There's still a lot of commented code in it.

Bruce S · « **Reply #5 on:** December 22, 2021, 03:32:33 PM »

SparWeb & Damon
Do y'all ever cull the commented out code? Not the stuff put in there for specific reason, say"descriptors" but old code that is commented out once a quicker or more eloquent coding? AND if so do you notice the code using less space and therefore less power to the Arduino?

I can follow OperHouse's stuff easy enough

I'm more of a firmware person

.

Thanks
Bruce S

SparWeb · « **Reply #6 on:** December 22, 2021, 04:05:54 PM »

Yes, I "clean house" but only after the extra stuff has been in the closet for a while. This edit shed about 100 lines; a fairly big purge, but I also added several dozen lines to make the energy calculation work.

Comments can burden the compiled code? Most compilers should just ignore it, right? Anyway, every time I compile for an upload to the AtMega, it tells me that it's only using 9% of program space. While this program is a biggie for me, at 600 lines, it's peanuts to the capacity of an Atmel Mega 2560 ARM.

SparWeb · « **Reply #7 on:** December 22, 2021, 04:07:58 PM »

The hardware side.
All the short green "X" wires are a result of my drastic simplification of the datalogger.

DamonHD · « **Reply #8 on:** December 23, 2021, 02:48:52 AM »

If code is truely redundant (eg not debugging statements that I may want to use once in a while) and unlikely to be used again then I kill it, to make the rest easier to read and navigate. When maintaining 10s or 100s of thousands of lines, removing the truely dead stuff helps the compiler in my head even if it makes essentially no difference to the the target. (Compilers can skip comments VERY fast.)

I often find myself in suitations where I am bumping along the top of available space, and to add a new feature I have to delete an old one and its resources, eg when I shoehorned several servers' worth of stuff into a single Raspberry Pi. And so yes, in that case I really can see the benefit of culling the old stuff.

Rgds

Damon

Mary B · « **Reply #9 on:** December 23, 2021, 12:06:50 PM »

I wrote a rudimentary word processor back in 1979(senior in high school), ran to around 20k lines of basic... LOTS of comment lines so others could follow it and the teacher semi had a clue(well he was clueless but he could read comments as to what the code did... he didn't understand it) what each bit did... That program was then used by the U of MN(I used their mainframe on a dial up connection) to actually create a word processor, one of their graduate students grabbed it and I got partial credit on it for the original code and idea. The grad student asked what the heck all the comments were for and when I told him he laughed for 10 minutes. We ended up deleting 90% of them over the summer(I got paid!) and I walked him through my thinking. It also counted towards my first degree in programming.

Comments are great to understand WHY a section of code is doing something. I still use them so I don't come back later and ask myself why in the heck did I write this section this way???

SparWeb · « **Reply #10 on:** December 23, 2021, 04:01:46 PM »

I've never attempted anything so complex. This datalogger is my most complicated program and, stripped of the comments, introduction notes, and disabled segments of code, it's only 400 lines or so.

I did write a solar-system simulator, with a graphic of the planet's orbits and a buttons to control the rate of time passing. And in school I wrote a jet engine performance calculator as a thermodynamics project. Those did not need hundreds of lines of code, however! Just enough to appreciate the value of comments in my code so that I can come back to it later. It still takes me a while to remember what's going on, including this week since I hadn't touched this for a few years.

SparWeb · « **Reply #11 on:** December 23, 2021, 04:08:27 PM »

Now that the logger is installed again, and I've finished some tweaks to get nice-to-have things like temperature compensation of the current sensor working, I'm getting data that looks reasonable. I still have to calibrate it more rigorously before trusting anything it tells me.

What I'm going to enjoy a lot from now on, is the new spot for the battery shed is in a better line of sight for WiFi to work. I can leave the laptop outside in the shed and it doesn't get disconnected from the router, so I can collect the logger's data from the comfort of my chair in the house.

The wind is coming up and the temperature is dropping, so rather than fuss about with calibrating numbers, I'm just going to let it log the data, good or bad.
I can do some sanity checks on the data - see if it's good enough to just need calibration tweaks, or whether something is so out of whack that I have to re-think the code.

SparWeb · « **Reply #12 on:** December 23, 2021, 11:27:26 PM »

It is SOOO nice watching the data come in while sitting warm in the house.

Now I can't resist the urge to write a dashboard program!

DamonHD · « **Reply #13 on:** December 25, 2021, 06:16:03 AM »

That's the sort of hedonism the world can cope with more of!

Rgds

Damon

SparWeb · « **Reply #14 on:** December 28, 2021, 11:47:44 PM »

A little more pruning and what a silly thing to be removing, it turns out.

I have fussed about with the calibration of the current sensor for quite a while. It was a "to-do" thing for a while when I first built the datalogger, and I got it "close enough" and then dropped it for a while. When I came back to the project this month I chopped out hardware and tweaked the conversion from 10-bit digital to the analog current measurement, but I left it messy. I promised myself to fix it later.

"Later" came along today, and I spent some time looking at what the math is actually doing. On a hunch I had the datalogger dump raw digital values to the serial port for a while until I was convinced my hunch was right.

I should also note at this point that it's been below -20C for several days and today was a toasty -14C in the sunshine. My laptop doesn't like me.

Anyway, I basically eliminated all of the temperature compensation and strange adjustment factors in the current sensor calculation. I needed to TRUST the voltage regulation on the Arduino board. Which I should have been doing all these years in the first place. All that faffing about to find the temperature compensation factor was probably caused by an earlier error in a linear slope converting the 10-bit number to a proper voltage.

Did another logging run this afternoon, and I was rewarded by a pile of data that hugs the trend-line a little closer than it used to.

SparWeb · « **Reply #15 on:** December 28, 2021, 11:55:20 PM »

Interesting "lobes" in the raw data. Still follows a straight trend-line, but not sure where they come from.

Edit: Vertical axis is power in Watts, horizontal axis is speed in RPM.

Bruce S · « **Reply #16 on:** December 29, 2021, 09:17:50 AM »

SparWeb;
Might just be the fact that I don't have my coke-bottle glasses on, but, those raw data lobes bits look kinda Sinusoidal.

Bruce S

Mary B · « **Reply #17 on:** December 29, 2021, 01:32:26 PM »

If this is powered from an AC source you are seeing power supply hum! I would add filtering to all DC lines... second possibility is a ground loop... all grounds should originate at one point. So if you are ground sensors in one spot and the arduino is grounded elsewhere and you run a ground wire between them plus you have a battery ground coming back from the battery bank you now have 2 grounds. Sensor grounds should be at the arduino. Ground loops are something I chase a lot with ham radio... with the PC I never know for sure if a hardware failure in it is causing a ground loop, audio card has done this 3 times!, or if the USB hardware has a failure... modern ham radios all are USB now for data to and from the radio. And I have a zillion ferrite chokes around here LOL on EVERYTHING. Right now dealing with a video hardware failure that is creating a ton of hash on the ham bands.

SparWeb · « **Reply #18 on:** December 29, 2021, 06:59:17 PM »

I can rule out an AC supply signal - it's all DC - quickly. That leaves sneaky DC circuits that could be playing tricks.
Another possible source is the data itself. The current sensor resolution is not very high.

I am redrawing the system schematic in case any mistakes in the grounding show up.

SparWeb · « **Reply #19 on:** December 30, 2021, 09:38:30 AM »

Randall has some advice for me:

DamonHD · « **Reply #20 on:** December 30, 2021, 11:40:38 AM »

You can't entirely rule out AC unless you can promise me that (for example) there is no (even tiny) mains or other AC pickup on your cable runs (including from the turbine) that gets rectified along with anything else.

Just sayin'

Rgds

Damon

joestue · « **Reply #21 on:** December 30, 2021, 01:55:12 PM »

Given that the lobes in the data are equally above and below the average, i would presume that your sampling rate is too low. it needs to be either synchronized to the ac current being measured (peak measurement) or a few dozen samples need to be made and an rms calculation made to determine the value of the current.

SparWeb · « **Reply #22 on:** December 31, 2021, 03:27:32 PM »

I can't promise anything yet. The inverter is the only 60Hz AC in that building, but being an inverter it's a noisy pig and it could be doing something sneaky. The turbine AC is rectified to DC and probably has a ripple in it. The current is being measured at the battery terminal. There are some "maybes" that I have to rule out.

Today is -25C so I'm not tempted to go out and try measuring anything.

Sampling rate is also interesting - the datalogger's program flow is still largely controlled by the triggering signal given by each phase's sine wave, which drives the RPM measurement. Currently, the program outputs a data line every 1/4 second, and the current is measured 10 times in that interval. I'm switching that to 7 times, just in case a prime number will wipe out some aliasing. Either way if it's 40 samples per second or 28 samples per second, the sampling rate is 2 or 3 times higher than the turbine AC frequency.

taylorp035 · « **Reply #23 on:** December 31, 2021, 10:37:11 PM »

I love the comic!

I can tell you from my surface finish work that it takes 5 data points per sinewave to identify it. Anything less and you miss it. Not sure if that helps

Mary B · « **Reply #24 on:** January 01, 2022, 11:47:58 AM »

Quote from: SparWeb on December 31, 2021, 03:27:32 PM

I can't promise anything yet. The inverter is the only 60Hz AC in that building, but being an inverter it's a noisy pig and it could be doing something sneaky. The turbine AC is rectified to DC and probably has a ripple in it. The current is being measured at the battery terminal. There are some "maybes" that I have to rule out.

Today is -25C so I'm not tempted to go out and try measuring anything.

Sampling rate is also interesting - the datalogger's program flow is still largely controlled by the triggering signal given by each phase's sine wave, which drives the RPM measurement. Currently, the program outputs a data line every 1/4 second, and the current is measured 10 times in that interval. I'm switching that to 7 times, just in case a prime number will wipe out some aliasing. Either way if it's 40 samples per second or 28 samples per second, the sampling rate is 2 or 3 times higher than the turbine AC frequency.

Inverter could be throwing hash back onto the DC lines, especially if it is one of the cheap ones, they tend to have minimal filtering

SparWeb · « **Reply #25 on:** April 18, 2022, 12:22:36 AM »

I got back to this project and wrote a simple web page that I can view with any computer logged into my router:

Note that the current is not calibrated at the moment and drifts with temperature. There are always little things to fix like these...

SparWeb · « **Reply #26 on:** April 18, 2022, 12:48:16 AM »

Here's how it looks today:

Fun with the boot screen on the LCD:

Mary B · « **Reply #27 on:** April 18, 2022, 01:54:06 PM »

Quote from: SparWeb on April 18, 2022, 12:22:36 AM

I got back to this project and wrote a simple web page that I can view with any computer logged into my router:

(Attachment Link)

Note that the current is not calibrated at the moment and drifts with temperature. There are always little things to fix like these...

I would edit that pic to lose the IP and SSID... open invitation to a hacker...

JW · « **Reply #28 on:** April 18, 2022, 03:10:26 PM »

Yes the hackers are out there...

I have an entertainment system based on a Dell 3620. Anyway they installed a program/file disabling the audio coming from that machine. So after they did this they said there's no way he can fix this without help. It really didn't bother me but the network interface cost me some money. Anyway watching true crime all the time got boring (three weeks). So I tracked the origin of the code they used to disable the sound network (whatever) I messed with it for around 4 hours and removed what they had done on my own. I told them I didn't do this to "defeat you" I just got bored... My system is back up and running and that's that...

JW · « **Reply #29 on:** April 18, 2022, 03:55:55 PM »

Here's one of the videos I produced

Front 242 Junkdrome

https://www.youtube.com/watch?v=PR0N8twpwGg

probably off topic but what the hell

if you have a good sound system its cool

SparWeb · « **Reply #30 on:** April 20, 2022, 11:22:35 PM »

The info is there just for diagnostic and "try it and see" purposes. Now that it's working, yeah, I really don't need to display that stuff any more.

Fortunately I live in such an isolated spot that it would be an extraordinary device that could both find, access, and break the encryption on my router, without driving up to my house in an obvious manner. I can't see any of my neighbours' routers either. Different story if I lived in the suburbs, I suppose.

SparWeb · « **Reply #31 on:** December 04, 2022, 06:41:30 PM »

The Watt-hour display that I added to my data logger earlier this year has been a real treat to watch.

As the numbers counted upward, however, it started to look a little off...

That's when I realized I had underestimated just how much energy this turbine will produce, and how big the number will get. While setting up this display item in the datalogger many months ago, I merrily set up the total Watt-hours to add up into the thousands. Shouldn't that be plenty!? Well... no. Now the numbers are overflowing the screen and scrunching out the "Wh" until only the "W" is visible. In a couple of weeks it would tip over >100,000 Whr and then I wouldn't see the unit at all.

Time to face the music and update the logger to show KiloWattHours like it should. Limited space on the screen so I kept it to "kWh" for short. There is space for it to read up to "999.99" before overflowing its place on the screen. Should take a year or two before that happens.

While editing the code, I noticed that I had made space for 4 digits on the RPM scale. Umm that implies >1000 RPM which makes for very fast blades! I wonder what I was thinking when I wrote that in the code (it would have been years ago). Trimming down the display units to only 3 digits covers the entire safe range of speeds any WT should have. And it made a bit more space for my sprawling kWhr values.

PS
If anyone is wondering why the 2nd picture says "STOP", that's a little snippet of code that stops the RPM from wandering around in very small values, or getting locked into the last speed measurement before actually stopping, which would leave it reading "12RPM" or something, while the blades are clearly not moving. So anything less than 20 RPM just gets replaced with "STOP".
The OTHER question you might ask about that screen is why the current it "0.1A" when clearly the blades are stopped. My current sensor isn't very accurate, and sensitive to temperature. If I get it zero'ed on a warm day, it's not zero'ed on a cold day.

Bruce S · « **Reply #32 on:** December 05, 2022, 08:54:16 AM »

I was only going to ask why the green instead of pretty blue

. But those are good answers too.
I was going to ask if the "STOP" meant the blades were in stop mode due to winds or a storm.

Cheers
Bruce S

News:

Author Topic: Datalogger update (Read 5974 times)