Captain’s Log
September 1, 2021
LoCARB is going through its death throes…at least that’s what I think is happening.
If you can read this, you’re amazing. Read it from left to right.
Starting at 7:55 am August 31, LoCARB started randomly rebooting and logging motion events (sensor is installed up top in the brain box so its dry). I am pretty certain that the tilt sensor does not throw false positives unless the physical contact in the sensor has been closed (I never observed it logging false positives unless the sensor was in a precarious position during testing) so I’m confident the boat is getting some love out there in the ocean somehow.
The boat is also logging strange voltage and humidity fluctuations. I believe water is coming into contact with the secondary circuit board within the motor pod, shorting out the 5v, saturating the humidity sensor, and causing the Arduino to reboot. The Arduino is programmed to do an initial satellite send/receive on boot, and then resume a 30 minute update interval in the event of a reset. Unfortunately, this will use up a ton of credits if the Arduino ends up in a boot loop, but the plus side is that I can observe each random reset.
I hope it doesn’t keep on rebooting and eating up all my satellite credits while I sleep!
SHARKS!?!?
So the resetting and motion events keep happening. Weather and wave events don’t really make sense since the weather reads very calm winds (which probably mean calm waters as well). On a whim I decided to see if there was any research concerning the area of water LoCARB is currently navigating in relation to sea animals, and what I found was fascinating!
Wind Speed where LoCARB is currently located
Are sharks taking a liking to the boat? To read more about the White Shark Cafe, head here
Apparently LoCARB is in an area of the Pacific Ocean where Great White Shark activity has been tracked, they even call it the White Shark Cafe. If this is indeed what LoCARB is encountering, it must mean that the sharks are really hitting the boat with quite a bit of force to cause the electronics to reset (and/or enough to get water on the secondary circuit board which could short the Arduino…). It also means the paint scheme of the boat isn’t doing much to keep the sharks from playing with it 🙂 I do know that to trigger the motion sensor, it requires quite a bit of sudden force so these are not gentle nudges. This is all very interesting indeed!
A series of resets and motion events which have been happening since Aug 31st.
September 2, 2021
The last update from LoCARB was at 10:20 pm September 1st. Prior to that, the boat had been silent for 11 hours before resetting and sending back some concerning information.
LoCARB had logged 64 motion events after the reset within a 30 minute time interval (2 per minute). This means the motion sensor was on its side and not in a vertical position (normal upright position). To confirm this was not an Arduino issue, I sent a power cycle to the Pixhawk and a reboot command (allowing the watchdog to trigger the reset button) to the Arduino. After the Arduino reset (it has 2 minutes after a reset before sending a satellite update message) the initial boot message showed 4 motion events, which means the sensor was still in a horizontal position. The boat was telling me that it was floating on its side.
It could be that the sensor has popped out of its support due to a collision of sorts, or the brain box somehow came off the hull, but these are very unlikely situations. Another unlikely situation is that the motor pod has detached from the struts, or the struts have broken. If the motor pod (acting as self-righting ballast) had detached from the hull/struts, this would likely have severed the wires connecting the battery to the brain box and thus we would not have an update. Likewise, the struts were reinforced with steel rebar, and if those have broken I would assume there would be stress on the wires themselves. Maybe the hull area which supported the struts have failed? I don’t know. I cant actually fathom a way for the boat to orient itself in a horizontal sideways position for a prolonged amount of time.
I think a more likely situation would be that the Arduino has gone bonkers due to a short in the circuitry located within the motor pod. The erratic resets, the inconsistent message update intervals, and strange voltage behavior probably just means that the Arduino has finally shorted out due to water ingress. I cant rule out catastrophic damage caused by shark attacks now though 🙂
Hopefully the boat will report back at some point, but as of now, the boat has been silent for 14 hours (the longest its been quiet since the start of the voyage).
September 3, 2021
LoCARB is dead. The boat hasn’t sent an update since 10:20 pm September 1st.
Did it sink? I doubt it. The hull is comprised of foam and wood so if damaged its probably just in really bad shape, but still floating.
How did the electronics die? I think water ingress in the motor pod finally took its toll on either the power system or the Arduino itself. It may be in a state of constant short circuit because of the secondary circuit board. I hope that with a little more time, corrosion will work its magic and remove the short on its own, allowing the Arduino to reset and function again.
It was an amazing run though! LoCARB spent 94 days out on the ocean, traveled 1160 miles from launch, traversed a total of 1540 miles of ocean (including all the wonky zig zags and backwards motion), braved and survived bad weather with winds up to 24 kts (also made it farther than the solo-kayaker, and solo wing guy), and ended up 1609 miles from Hawaii!
Maybe one day in the future the boat (or what’s left of it) will make its way to Hawaii. One can hope anyways.
September 6th, 2021
LOCARB IS ALIVE! Well, at least for now that is. The boat sent an update today at 7:22 am, 5 days after going silent. This blows my mind, I haven’t heard of another autonomous boat going silent and recovering. With hope waning every passing day, I for sure believed that the boat would never ping back. How long this will keep up I’m unsure, but I believe my theory about the secondary circuit board being the cause of a short circuit is correct. I believe water that is/had come into contact with the secondary circuit board located in the motor pod is/was causing corrosion and shorting out the Arduino, so maybe the short cleared and allowed the Arduino to function correctly today.
Either way, I believe the problem is limited to the Arduino and not the Pixhawk autopilot, so I sent some updates to the boat to head directly to Hawaii in case the boat goes silent again. If the Pixhawk continues to function (as long as it has sufficient voltage to do so) it should still make its way in that direction, despite the Arduino short circuiting. The problem with the Pixhawk is that it needs to be power cycled every week or else it gets confused and/or sometimes loses control of its rudder.
LoCARB has risen from the dead!
Today LoCARB has my full respect.
Sent Pixhawk power cycle, Arduino reset, final waypoint, update interval set to 4 hours.
September 10, 2021
The boat has pinged back two times today, albeit not consistently.
Voltage and GPS data are 0, which means the Autopilot and Arduino are not communicating correctly, or the Autopilot has lost GPS lock (this is typical after being powered on for over 1 week with no reset).
I don’t believe the sent satellite messages are being parsed correctly by the Arduino, as I have sent previous reset messages with no result (based on expected sat message behavior). On a whim I’ve sent both a Mavlink soft-reboot command to the Autopilot and a hard power cycle command for the autopilot (also reboots the Arduino) to see if it corrects the communication problem between the Arduino and Pixhawk.
If I see the Arduino respond to the reset, but still have no communication with the Pixhawk autopilot, it may mean something has happened to the power supply going to the autopilot or the Arduino serial pins are experiencing issues.
Either way, I did program redundancy into the tracking map to allow for tracking the boat using iridium data. It isn’t as accurate but its good enough for determining the approximate location of LoCARB. Hopefully one of these update intervals I can get the Pixhawk and Arduino communicating again. Im just happy the boat is still somewhat alive!
September 11, 2021
LoCARB seems to ping back two times at a time then go silent for whatever reason. Either way, during those update intervals LoCARB was able to correct the issue with the Autopilot and regain communication. LoCARB is still heading in the right direction (and has traveled 93nm since it went silent Sept 1st), but I sent updated waypoint messages two more times just to make sure that the waypoint is updated in the Autopilot during the next random updates. All I can hope for is for the winds to cooperate, which seems to be what this week has in store (according to the windy app).
LoCARBs position with forecasted wind speed today at 4pm
September 13th, 2021
I believe I now have enough data to determine that LoCARB has indeed been damaged.
Ever since the beginning of the random resets and motion events, I have seen the brain box humidity slowly but steadily increasing. Prior to this, the brain box humidity had always hovered within 30-39% humidity. The fact that it is now increasing past this level could mean a few things; the struts have been compromised, allowing moisture into the PVC piping which the wires run within and thus allowing moisture into the brain box, and/or the brain box is being submerged and allowing some moisture in from where the lid meets the box. I believe that LoCARB’s solar platform mounting may have been compromised enough to be in a tilted configuration (think of an X vs a T), which could provide enough buoyancy (if in contact with water) to also skew the orientation of the entire boat. If this tilt somehow submerges the brain box in water, impeding a good line of sight to the sky for the satellite modem (by being wholly or partially submerged), I could expect communications behavior akin to what is now being observed (random and intermittent updates).
A possible explanation of how the brain box is being submerged in water…
I believe the random updates are not fully due to the secondary circuit board shorting out the Arduino and causing resets (which does happen) since for the past few days the updates are not always showing that the boat has reset (RPM will show 0 if sending an initial boot message if I have not turned the motor off manually). The only reason for the boat to skip an update interval if the Arduino is functioning correctly is if the operation of the satellite modem is compromised. This could be from Arduino to Modem, or Modem to satellite. Could the serial communication wires be failing between Arduino to modem? Sure, but if we take the motion events and the increasing brain box humidity into consideration, it could mean the box is being submerged in water impeding the messaging updates.
I could be completely off since I’ve learned that my past assumptions about the boat have consistently been proven wrong…Still, it’s fun to play data detective and see what kinds of theories I can come up with.
With each passing update though, my desire to see the boat and open up its compartments to run forensics keeps growing.
September 14, 2021
Today LoCARB was considerate enough to begin resuming normal updates (although it reset a few times) to let me know that there is now a new problem with its power supply.
LoCARB, you are the most considerate boat ever.
Despite resuming mostly normal messaging behavior, the boat reset a few times today and reverted back to its 30 minute update intervals. This was all good because it gave frequent enough data to help me observe the voltage drop (despite turning off the motor) much lower than its normal threshold throughout the day. This could mean one of four things; the battery is losing access to more of its cells, the BMS has lost its ability to charge another row of cells, the solar panel is not working/disconnected, or the buck converter (solar charge controller) is kaput.
Worst case scenario is the solar panel has disconnected or the buck converter has died. Without either of those two key components, there would be no more power generation. As long as the voltage remains above 5-6v however, I believe the boat can still function correctly.
If the solar panel platform had indeed been twisted in a position to cause the boat to change its rotational axis and submerge the brain box (blocking the satellite updates), could the breaking off of the solar panel platform (which would allow the boat to fully right itself) from the boat sometime after 1:12 pm explain why the humidity and voltage had dropped so dramatically (as shown above) and explain why the messaging capabilities resumed its normal behavior (despite the resets)?
Will know more in the upcoming days ahead.
September 23, 2021
LoCARB hasnt pinged back since the dramatic drop in voltage 9 days ago, so I assume the power system is fully shorted out. Lets hope LoCARB washes ashore someplace exotic and someone who finds it gives us an update in the near future! If it does wash ashore in Hawaii, I would anticipate a February-March 2022 timeframe.
