Waitt Institute

The Waitt Institute implements innovative technologies in the field through collaborations with world-renowned scientific institutions, synthesizing global expertise and accelerating groundbreaking research. More »

Critical Failure

Mary Ann desperately needs one of the parts on its way to ‘trade’ for the part Ginger needs…

STATOR FAILURE; NO VEHICLES IN THE WATER!

THURSDAY, 02.26.09

1247: To Ted Waitt
Ted, early this morning Mary Ann aborted with what we thought may have been a software reset. In reality, it was a very quick reading by the vehicles sensors that either the recovery float or ascent weight was not in place. It happened so fast that our log files on the surface don’t have a record of it, but the vehicles does; that is an abort condition. When the vehicle came to the surface, we found we did not have thrusters. Joe Lepore had the deck, and recovery went okay. When the boys ripped into it, they found the thruster unit had a serious problem. We’re not sure how or why yet, but we had a failure in the worst place possible, the tail assembly.

They tell me the encapsulated (meaning in the middle of a bunch of rubber like material) encoder failed on the main propulsion unit. This thing acts like a timing belt to control the firing of the motor phases in the feedback loop. They tested, re-tested and confirmed their testing with a couple guys checking results and with a lot of help from the rear. Then Greg Packard went to Ginger’s damaged encoder from the other night’s incident to see if it would be possible to tear into Mary Ann to try and repair it. The results are not good at all: they cannot fix it. When they tore into the potting, it ripped up the wires below. Greg is stunned. There is only one failure of this unit on record. This is not depth-related as there was no intrusion to the housing. We need this part to be replaced.

Jim Sullivan of HBOI (left) and Matt Tocchini of WID inspect the thruster unit.  The failed stator can be seen just inside the part in Jim's hands; a cylinder topped with a black ring, just inside the housing.

(L-R) AUV Operators/Technicians Jim Sullivan from Harbor Branch (HBOI) and Matt Tocchini inspect the thruster.

The problem is the only one we currently have is on the ship with you. If you recall, we thought these parts were needed to make a build to replace the loaner. We checked and there is another motor on the shelf, and they have agreed to use theirs for a build if the loan goes through. That bit of information is the most critical break we have received to date. Absent that or another motor (which Purcell is checking on now), our best case scenario until we get that part is one vehicle.

We are cautiously told the deal, which for about an hour this morning looked totally dead, has revived and is now on the verge of approval. But we are dead in the water until we get the part from you. We are working paths to build the housing at Woods Hole and at Cameron’s shop if the deal falls through, but absent another motor we will not be able to rebuild it. Purcell is scouring the world for another option.

So, regarding the thruster motor you have on board for the build of the loaner, we are discussing options for getting it back. We can run to you for part so you can stay on station. Meet halfway. Not sure yet. Meeting just started, figured I’d finish this off so you can be in the room.

We need an answer on the replacement part to make the right decision. More soon.

1303: From Ted Waitt
This is like a bad movie. So, let me get this straight. We’ve had two major failures of the same part that we never had a replacement for. Didn’t I say to make sure we had replacements, and what did we spend all that money on? Hopefully someone is machining the housing, as I’m still doubtful about getting a full replacement part.

1312: Response To Ted Waitt
I know. One failure and one damage-inflicted (AUV). We do have the one spare part, but we didn’t buy two because the part has never failed before. Some things we have four of because they are known issues. It’s the unknown unknowns that have screwed us. We can’t catch a break anywhere. Lee Frey is double checking a potential repair for the current failed one. I am waiting on word from Purcell on everything now, availability of another thruster spare from the manufacturer, status of his machine shop and titanium stock.

Status of plans and status of loaner: we know we can use the one you have to get Mary Ann back up and running, but we need info to decide what the best way to get it is, run to you or meet halfway. If the loaner is a wash, then we meet you and get the part and everything comes to Pago next Thursday for a run to us on your ship. It’s a nightmare. I am waiting on an assessment from the captain on fuel impacts, from Lee Frey on a triple check of the current failure and from Purcell on everything else.

1544: Update On Bad Thruster
Lee is still working on the thruster, he won’t give up, insists he can make it work, like a dog with a bone. He’s just about got me convinced. We might get Mary Ann working again. Don’t know what they’re up to, but want to let them roll with it, not bug ‘em. Will keep you all posted.

FRIDAY, 02.27.09

0644: Spare Part, Important: Read ASAP
The box containing spare parts in return for the loan of the thruster unit has a part inside that we no longer need to send, and that we need out here. This is from the gray pelican case I delivered to you the other day.

Please open up the case and remove the part shaped like a big steel and black donut. Then, send the case back to the States with all due speed. It should go in with Tom Sharp when he flies in for the pick up on Tarawa. As soon as it gets Stateside, it should be sent to Purcell.

AGAIN, please remove this part and bring back to RVSJ with other parts.

MOST IMPORTANT.

Thanks all.

0702: Day 10 (Yesterday, February 24)
This morning at 0244, Mary Ann aborted her mission. When she came to the surface, her thruster motors did not activate after the recovery float was deployed and the command sent for the vehicle to back down. The grapple was fired over the vehicle and with some fancy line handling she was recovered safely to the deck. When her log files were checked, there was an indication that the ascent weight or float had been released, which is an abort condition. The emergency board cycles much faster than the vehicle log, so while we did not see the indication on the surface, the records in the vehicle indicate this is what happened. That was the reason for the abort and the issue is being looked into right now (could be a jiggly ascent weight. They are made of lead and the attachment bolts are not uniform, so coulda been a tad loose). The thruster failure was another matter altogether and unrelated to the abort (although had it happened on survey, it would certainly have caused one).

When the thruster was taken apart on the surface, it was determined the encapsulated stator failed on the main propulsion unit. This thing acts like a timing belt to control the firing of the motor phases in the feedback loop. They tested, re-tested and confirmed their testing with a couple guys and with help from the rear. Greg performed surgery on Ginger’s encoder, damaged from the other night, to see if he could tear into Mary Ann’s to try and repair it. They basically started hacking off the potting and the results were not good, just more damage. Initially they thought it irreparable. We then started talking about immediate actions.

The replacement spare we have is currently on Ted Waitt’s ship and was promised as a partial trade for a thruster assembly. At the time we did not have an answer on the thruster, so we were in a very bad spot. We had one vehicle down from the vehicle/ship interface, and now we had another down due to an inexplicable motor failure — and our spare is three days away. I don’t mind saying some very dark thoughts entered my stream of consciousness. At that point, it looked like the best we could hope for was one vehicle for the duration of the mission and the replacement deal had just become irrelevant. If we needed the spares aboard the ship, then the choice in front of us was really which vehicle do we want working?

We caught a small break. We were informed we could get the part (stator) we needed, which would allow us to retain ours and still contemplate the loan. Now we had another conundrum. What to do next? Did we go to Ted to pick up our spare and lose the run time and fuel? Did we have them run the part out to us and then lose assets in running the loaner out to us should that deal be approved? Should we all sit and do nothing for the time being? Strange as it sounds, the last option was the best. We needed to sit on our hands on both sides and wait for more information.

As anxious as this made us, I have no doubt Ted Waitt’s frustration far outweighed ours. While we could keep working our Deep Ocean Transponder (DOT) field and do maintenance to keep busy, he had to sit in open water a hundred miles from Tarawa, and listen to us describe how his dream was melting away from under him. It had to be infuriating. I guarantee you, had I been he, someone would have to pay the price. We got nothing like that from him. I have to say, missions like this are never easy and there are always problems. But this is, as he described it, “like a bad movie.” We needed some joy and we needed it badly.

And it came. Lee was awakened and spent the next four hours troubleshooting the failed motor; lo and behold, he got it working again. He does not know why or how, but somehow the thing is spinning again. I hate these fixes. Through troubleshooting and tweaking, some connection got tightened or something awry was put back into place and the thing works. It’s scary; you can never trust that part again, but in this case it was a Godsend. I will take inexplicable over inoperative anytime. Mary Ann was put back in the water at 2252 and is now on survey working away. What a dear girl she is. And that was not the end of our good fortune.

Mary Ann in the water

Mary Ann is launched as the failed stator comes back online.

The loaner was approved today as well. While we did not receive the part in-hand quickly enough to get it on a plane to Honolulu today for delivery to Tarawa before the sun set on their unlighted runway, it is in our possession. It will arrive on Tarawa tomorrow at sunrise for transport out to us aboard Ted’s boat. As I write this, 3:29 a.m. on Feb 27th, that part is in the air and headed our way.

Current status: Mary Ann is down on survey. The parts to get Ginger back running are en route and should be to us sometime early next week. By Tuesday, I hope to be back on station with two vehicles. We are tracking a number of issues right now, but they can wait for a slow news day.

Thank your deities, people, and request they look over us. We are only a fraction of the distance and the lady ain’t shown herself yet.

Image Gallery