TUESDAY, 03.17.09

1150: From Ted Waitt
It seems like 12b has been cursed as we’ve dove on that like three times now.

1160: Response To Ted Waitt
Yeah, this software issue is similar to one we had before when they used a utility to check the sonar data during the pre-dives. Somehow it caused the data to be dumped into the sub bottom profiler data stream. The fix is to use a different utility to check out the sonar and that takes more time, but it’s what we’re gonna do from now on unless Hydroid can write a patch to re-set everything properly before diving. Everybody got real quiet when I got up at 2 a.m. this morning, and came back in to find them still working on this. People started leaving the room when I asked if it was operator error. Seems like it was, although I do not have a clear vision of how this happens — but I do understand it’s not as simple as just wiping a drive memory. One earlier operator error was just that, failure to wipe the system OS mission drive, but this one is different. Also, there is the possibility we might be able to retrieve this data. We have it now, but we just can’t convert it into a file type to read it on the analysis software. They are still working on that so I did not have them send the vehicle into the same box. We sent it to the next one up the line just in case we could get the data back and readable. I need to sit in on the next two vehicle checks to get a better understanding of this problem. We keep running into these software bugs. It’s so aggravating — all stuff people should have known. Same thing with the transducers. By the way, box 8b was the winner of the bad dive box, took four to get that one done (as long as I’m shining up, too little too late and bad knowledge).

1246: From Ted Waitt
Having worked with some of the smartest people in the world for nearly 20 years, and continuously pushing the envelope on technology while routinely navigating between the ‘leading edge’ and the ‘bleeding edge,’ I’ve learned one thing always holds true — nothing beats good old-fashioned common sense.

1354: Days 29 & 30 (March 15 & 16)

Greetings, I’m making a little adjustment to my dates so as to sync up with the daily reports that Andy Sherrell puts out. I have no idea how we got separated, but then again, I don’t really care. Call it a leap day or something like that.

Mary Ann is still working with her 230/540 transducers in place. She launched on the 14th at 0622 and was recovered on the 15th at 0628 after a successful survey of the remainder of boxes 11a/12a. She was on the surface for six hours while the techs worked on the issue of noise in her starboard transducer. Greg Packard did some experimenting, trying to figure out where this noise was coming from and why it shows up the way it does — some four to six hours into the mission and then fading well before the vehicle leaves the bottom. He deduced the only thing changing on the vehicle during that time was the voltage level on the batteries, and so he did some testing on the surface. Using a direct power source, a meter and the sonar imagery, he simulated battery power states and found the noise did indeed vary with the power level provided to the power supply in the main bottle. This may not be a totally accurate depiction, but the discovery he made seems to be. Apparently, the isolation of that power supply is not perfect and at that certain state, there is a frequency interference with the sonar transducers. The manufacturers pointed out to us we were pushing the sonar range past its recommended distance, and said this noise would likely disappear if we worked it within the stated range of 250 meters. When she came back after having been adjusted from 400 meters to 350 meters, we did see a decrease in noise. We have adjusted it to 325 meters to dial down the noise one last increment, hoping to maximize our potential while yet still planning surveys based on numbers giving us the best possible coverage with overlap. This seems to be working out OK.

Mary Ann is still working with her 230/540 transducers in place. She launched on the 14th at 0622 and was recovered on the 15th at 0628 after a successful survey of the remainder of boxes 11a/12a. She was on the surface for six hours while the techs worked on the issue of noise in her starboard transducer. Greg Packard did some experimenting, trying to figure out where this noise was coming from and why it shows up the way it does — some four to six hours into the mission and then fading well before the vehicle leaves the bottom. He deduced the only thing changing on the vehicle during that time was the voltage level on the batteries, and so he did some testing on the surface. Using a direct power source, a meter and the sonar imagery, he simulated battery power states and found the noise did indeed vary with the power level provided to the power supply in the main bottle. This may not be a totally accurate depiction, but the discovery he made seems to be.

Apparently, the isolation of that power supply is not perfect and at that certain state, there is a frequency interference with the sonar transducers. The manufacturers pointed out to us we were pushing the sonar range past its recommended distance, and said this noise would likely disappear if we worked it within the stated range of 250 meters. When she came back after having been adjusted from 400 meters to 350 meters, we did see a decrease in noise. We have adjusted it to 325 meters to dial down the noise one last increment, hoping to maximize our potential while yet still planning surveys based on numbers giving us the best possible coverage with overlap. This seems to be working out OK.  Mary Ann went in the water to cover the rest of box 12a on the 15th at 1238, and was recovered on the 16th at 1230. She went back in to finish up 12a and start 13a that same day at 1518 after a two-hour, 48-minute turnaround.

On Saturday, Ginger went in the water at 1532 and was recovered on Sunday the 15th at 1435. It was determined her starboard transducer was suffering from the same depth-related failure that claimed four of her predecessors. Her transducer showed degraded performance on the way to the bottom when she reached 4,770 meters, and the sonar started working again at 2,938 meters during her ascent. I wish I could say this came as a surprise to us, but we had been expecting it. We switched out her transducers to confirm this was a transducer failure, and put her back in the water. When she came up yesterday, there was a problem with the data. One of the pre-flights requires we check the sonar data and we had been using a utility that apparently makes it difficult to ensure the system is re-set to log data.

The techs are still trying to figure out a workaround for this, which will take more time and use different software to check the sonar. As well, they have asked for a software patch from Hydroid to re-set the system, but it has to be pointed out this is operator error. We will write another new procedure to ensure this does not happen again, and it will be discussed in the changeover meeting. This is doubly troubling because the team performing the pre-flight achieved the record for shortest time on the surface by turning the vehicle around in an hour and 46 minutes. While I appreciate quickness, I have spoken to the teams repeatedly about making sure we do not sacrifice accuracy for speed. It does me no good to save half an hour on the surface when the result is a wasted 24-hour sortie — and this appears to be what happened. We still have hopes this mission will be readable as the data was stored in a sub bottom profiler directory and is still there. When we sent the vehicle back down, we sent it into the next box in the hopes we can retrieve and convert the data.

As far as all the parts we have gone through, the Woods Hole team has been working hard to get us as many replacements as they can. We have a new transducer coming out to replace, on one vehicle, the long-range sets that have failed. This is a 120/410 system that will not have the 600-meter range of the 75/410 system, but it should give us a better resolution at 400-500 meters. We will still have to run one vehicle with the high frequency, short-range set for the time being. When we go to Samoa in just over a week we will have a replacement set of 75/410 transducers and hopefully, two more sets of the 120/410s. The 120/410s are from the same manufacturer that made the failed units. There is fear we are going to see the same trend in performance.