The Trouble with Rover [Spirit] is Revealed
By Ron Wilson, EE Times, February 20, 2004 (6:32 p.m. EST)
URL: http://www.eetimes.com/story/OEG20040220S0046

SAN MATEO, Calif. — When the Mars rover Spirit went dark on Jan.21 a Jet Propulsion Laboratory team undertook to reprogram the craft's computer only to find themselves introducing an unpredictable sequences of events.

The trouble with the Mars rover Spirit started much earlier in the mission than the day the craft stopped communicating with ground controllers. "It was recognized just after [the June 2003] launch that there were some serious shortcomings in the code that had been put into the launch load of software," said JPL data management engineer Roger Klemm. "The code was reworked, and a complete new memory image was uploaded to the spacecraft and installed on the rover shortly after launch." That appeared to fix the problems that had been identified with the initial load. But what no one at JPL could have anticipated was that the new load also made possible a totally implausible sequence of events that would, many months later, silence Spirit.

The Spirit rover has a radiation-hardened R6000 CPU from Lockheed-Martin Federal Systems at the heart of the system. The processor accesses 120 Mbytes of RAM and 256 Mbytes of flash. Mounted in a 6U VME chassis, the processor board also has access to custom cards that interface to systems on the rover. The operating system is Wind River Systems' Vx-Works version 5.3.1, used with its flash file system extension. In operation, the real-time OS and all other executable code are RAM-resident. The flash memory stores executable images that are loaded into RAM at system boot. Separately, about 230 Mbytes are used to implement a flash file system that stores “data products,” or data files that are created by the rover's subsystems and held for transmission to Earth. Among the data products are the images created by the rover's cameras.

“Part of my responsibility in the data management team is to keep track of the data files that are created, transmitted and deleted on the rover during the mission,” Klemm explained. “We recognized early in the planning process that the flash file system had a limited capacity for files. It is not just a limitation in the flash itself but also in the directory structure." Klemm explained that as data is collected by Spirit, files are created and stored in the flash file system until a communications window opens — an opportunity to transmit the data either directly to Earth or to one of the two orbiters circling the Red Planet. Then the files are transmitted. They are still held in the flash system until retrieved and error-corrected on Earth. If data is missing, requests are sent for retransmission. If the data is intact, a command is sent to delete the received files.

"But there were also directories of files already placed into the file system in the launch load," Klemm said. "When we uploaded a new image to the rover, we recognized that those files would have to be deleted, because they were being replaced by a new set using different directories." Accordingly, on Martian day 15 (or “sol 15”) of rover operation, a utility was uploaded to the rover to find and delete the old directories.

Murphy strikes on Mars
But the transmission that uploaded the utility was a partial failure: Only one of the utility program's two parts was received successfully. The second part was not received, and so in accordance with the communications protocol it was scheduled for retransmission on sol 19. Thus was the fuse lit on a software hand grenade. The data management team's calculations had not made any provision for leftover directories from a previous load still sitting in the flash file system. As Murphy would have it, earlier, sol 19 Spirit attempted to allocate more files than the RAM-based directory structure could accommodate. That caused an exception, which caused the task that had attempted the allocation to be suspended. That in turn led to a reboot, which attempted to mount the flash file system. But the utility software was unable to allocate enough memory for the directory structure in RAM, causing it to terminate, and so on.

Spirit fell silent, alone on the emptiness of Mars, trying and trying to reboot. And its human handlers at JPL seemed at a loss to help, unable to diagnose a system they could not see. Luckily, early in the process of proposing failure scenarios, someone remembered the earlier failure to upload the second piece of the utility. The scenario was modeled, and it was discovered that a VxWorks flag that causes a task to be suspended on a memory allocation failure was set in the existing image. "The irony of it was that the operating system was doing exactly what we'd told it to do," Klemm lamented.Working on the theory that the rover was in fact listening and rebooting, the team commanded Spirit to reboot without mounting the flash file system.

The team then uploaded a script of low-level file manipulation commands that worked directly on the flash memory without mounting the volume or building the directory table in RAM. Using the low- level commands, about a thousand files and their directories — the leftovers from the initial launch load — were removed. "At that point we mounted the flash file system and ran a checkdisk utility," Klemm said. To everyone's enormous relief, the mount was successful.

"As we had anticipated, there was some corruption from the event, so that was corrected," Klemm added. "In the process of going through the contents of the file system, we discovered a system log in which the problem was documented, step by step, right up to the allocation request that failed." Klemm said that with the leftover directories and their files removed, the system is now functioning well. But just in case, the team is working on an exception-handler routine that will more gracefully recover from an allocation failure.
...

- posted by Andy @ 4:45 PM

 
Opportunity Sees Tiny Spheres In Martian Soil
February 04, 2004. JPL

This magnified look at the martian soil [imagelink] near the Mars Exploration Rover Opportunity's landing site, Meridiani Planum, shows coarse grains sprinkled over a fine layer of sand.

NASA's Opportunity has examined its first patch of soil in the small crater where the rover landed on Mars and found strikingly spherical pebbles among the mix of particles there. "There are features in this soil unlike anything ever seen on Mars before," said Dr. Steve Squyres of Cornell University, Ithaca, N.Y., principal investigator for the science instruments on the two Mars Exploration Rovers.

For better understanding of the soil, mission controllers at NASA's Jet Propulsion Laboratory, Pasadena, Calif., plan to use Opportunity's wheels later this week to scoop a trench to expose deeper material. One front wheel will rotate to dig the hole while the other five wheels hold still. The spherical particles appear in new pictures from Opportunity's microscopic imager, the last of 20 cameras to be used on the two rover missions. Other particles in the image have jagged shapes. "The variety of shapes and colors indicates we're having particles brought in from a variety of sources," said Dr. Ken Herkenhoff of the U.S. Geological Survey's Astrogeology Team, Flagstaff, Ariz. The shapes by themselves don't reveal the particles' origin with certainty. "A number of straightforward geological processes can yield round shapes," said Dr. Hap McSween, a rover science team member from the University of Tennessee, Knoxville. They include accretion under water, but apparent pores in the particles make alternative possibilities of meteor impacts or volcanic eruptions more likely origins, he said.

A new mineral map of Opportunity's surroundings [imagelink] , the first ever done from the surface of another planet, shows that concentrations of coarse-grained hematite vary in different parts of the crater. The soil patch in the new microscopic images is in an area low in hematite. The map shows higher hematite concentrations inside the crater in a layer above an outcrop of bedrock and on the slope just under the outcrop.

Hematite usually forms in association with liquid water, so it holds special interest for the scientists trying to determine whether the rover landing sites ever had watery environments possibly suitable for sustaining life. The map uses data from Opportunity's miniature thermal emission spectrometer, which identifies rock types from a distance. "We're seeing little bits and pieces of this mystery, but we haven't pieced all the clues together yet," Squyres said.

Opportunity's Mössbauer spectrometer, an instrument on the rover's robotic arm designed to identify the types of iron-bearing minerals in a target, found a strong signal in the soil patch for olivine. Olivine is a common ingredient in volcanic rocks. A few days of analysis may be needed to discern whether any fainter signals are from hematite, said Dr. Franz Renz, science team member from the University of Mainz, Germany. To get a better look at the hematite closer to the outcrop, Opportunity will go there. It will begin by driving about 3 meters (10 feet) tomorrow, taking it about halfway to the outcrop. On Friday it will dig a trench with one of its front wheels, said JPL's Dr. Mark Adler, mission manager.
- posted by Andy @ 12:17 AM

 
'Turbulator' Technology Reduces Drag, Improve Swimmers' Time
Science Daily News 2/3/2004
Buffalo, NY---The difference between finishing first and coming in second in competitive swimming is measured in milliseconds, so when a swimmer's technique and fitness is as good as it gets, a coach turns to one remaining variable to sharpen the competitive edge -- the swimsuit. In that scenario, scientists in the Center for Research and Education in Special Environments at the University at Buffalo (UB) may be a coach's best friends. They have a patent pending on a structural element that can improve a swimmer's time by decreasing the force water exerts on swimmers, called "drag," by 10 percent when incorporated into the swimsuit design. The new element, which the researchers call a turbulator, alters the fluid dynamics of water as it flows over and around the swimmer. How drag acts on a body moving through water plays an important role in the amount of energy a competitor must exert to cover a specific distance: less drag, less energy required, quicker finish. Trials of suits incorporating the turbulator into their fabric, conducted at UB over two years, showed that adding the element could improve a swimmer's time by 3 percent, said David Pendergast, Ed.D., UB professor of physiology and biophysics. Pendergast and Joseph Mollendorf, Ph.D., UB professor of mechanical and aerospace engineering, were senior researchers on the project. TYR, the company that has licensed the technology and named it "Aqua Shift(tm)," will introduce its new line of competition suits incorporating turbulators to the swimming world today (Jan. 30, 2004) at the FINA World Cup Swimming meet. The competition is being held at the Nassau (N.Y.) Aquatics Center.

The team of UB inventors who developed the technology will be at the launch of the new suit to discuss the underlying science and the performance trials conducted in the UB center's facilities. The turbulator's science is grounded in the research team's earlier work in fluid dynamics and its success in decomposing drag, breaking it into its component forces. "No one else had done that before," Pendergast said. "We discovered there are three types of drag. (1) Friction drag, the force of water molecules as they pass over the body, is dependent on how long the body is. (2) Pressure drag, the strongest force, results from pushing the water out of the way. (3) Wave drag occurs at relatively high speeds and is the force exerted by waves created." When the researchers broke drag into its three components, they found that pressure and friction drag exerted the highest influences, said Pendergast. Their next question was: How can drag be reduced?

Their first inclination was to change the surface of the swimsuit fabric, but that approach didn't reduce drag significantly. Enter the turbulator, a strategically placed fabric-encased flexible tube that introduces a raised ridge on the suit. Pendergast describes how this element improves the fluid dynamics of a swimmer. "When water hits the shoulders of a swimmer, it separates from the body, which creates drag. By adding a turbulator, we cause water to follow the body instead of separating from it. This change increases friction drag, but reduces pressure drag. We found that placing a turbulator on the front and back of a suit significantly reduced pressure drag, overcoming the increased friction drag and adding a competitive advantage" Meanwhile, TYR had approached Albert (Budd) Termin, II, UB's swimming coach, whose swimmers compete in the company's suits, about working on reducing drag. Termin has collaborated extensively with Pendergast and Mollendorf on improving swimming efficiency. Over a two-year period, the team tested 20 suit models incorporating the turbulator for TYR at the Center for Research and Education in Special Environments. The trials took place in the center's special annular (doughnut-shaped) pool designed for conducting a variety of specialized research, including measuring drag and other hydrodynamic properties, and in UB's competition pool. "The work was part theory and part practice," said Pendergast. "It turned out the size of the turbulator was crucial. We'd predict how a certain size and placement would respond, TYR would build the suit, and we tested and retested." The final design incorporates a series of turbulators positioned on the suit front, across the shoulders and across the hips. (UB research on suit design had shown that suits that cover the swimmer from shoulder to knee or ankle produce less drag than suits with less coverage.) Yana Klochkova, who won two gold medals for Ukraine in the 2000 Olympics and is sponsored by TYR, will model the new suit at the Jan. 30 unveiling.
***
- posted by Andy @ 5:59 PM

 
The Mars Mission Continues
(In this Q&A with James Gosling, find out more about the role of Java in the Mission to Mars and download your own version of the Maestro application. )
Sun MicroSystems, Friday Jan 16, 10:00 AM PT

Q:What exactly is Java's role in the Mars Rover mission?
A: The places where NASA scientists have used Java for this mission is all on the ground side right now. They have created this collaborative command and control system called Maestro, which does this combination of data visualization, collaboration, command and control. It lets them look at images and create 3-D reconstructions of terrain. It allows various experimenters to look at the scenes and topography, browse the image databases and take part in all the participation they need to do. And to do it in a remote, distributed and collaborative kind of way - so they could actually have scientists at institutions all over the world who are not only looking at the data, but also collaboratively deciding on the way the mission should proceed. One of the nice things that the JPL guys have done is that they've made a "cleaned up for civilians" version of this application available that's called Maestro.

Q:How is the Java assisting in controlling the Rover from earth?
A: There's a Java API called Java Advanced Imaging, that's used for the images captured by the panoramic camera - the one that producing images with excruciating detail. Those panoramas are being created by combining images from two different cameras onboard the Rover, so with the two lenses they get two images - just like you've got two eyes - so you can do a stereoscopic mapping where your brain is able to figure out how far away things are. Because they've got these stereoscopic images, they can go through a process that's called stereo-image correlation, so they can calculate for each pixel in the image how far away that picture element really is.

With this information, the JPL scientists can calculate how far away each rock is, each picture element, for all of the millions of pixels in one of these large images. So you can get the depth of the image at every point. That's what a stereo camera gives you. When you've got the depth information, you can then actually build a 3-D model, the actual model of the terrain. And then you can actually map the image back onto the 3-D model. So then what you've got is a colored, three-dimensional model of the world around you.

Q:Are they actually commanding the Mars Rover with Java?
A: For the command and control system, big parts of it are this rather large Java application. There are a lot of parts involved in this. The Rover itself has a computer onboard. There's no Java in that computer now. But on the ground-side, there are a number of parts of the whole command and control chain that goes out to the Rover that's done in Java. It's not like every last piece of every subsystem is based on the Java code. Great big pieces of it are. In particular, all the data visualization, user interface front-end stuff and I believe a whole lot of the database stuff is.

Q:How does the public version of the Maestro application work?
A: If you go to the Maestro website (http://mars.telascience.org), You will find that they've got two sets of downloads. One is the Maestro application itself, and the other is a first teaser set of data from Mars. There are different versions for different platforms. There's one for Solaris, a version for Linux, there's a version for Windows and more. The fact that they've got all those versions just shows you how portable Java is, how cross-platform it is. It's exactly the same program in all of those, they are just packaged differently.

So when you download the first set of data. There's a script that walks you through looking at some of the data. Using the 3-D model they have there, and using your mouse, you can actually manipulate the 3-D model and you can get a view as if you're standing off to the side of the landing looking back at it. You can actually wander around the landing site. You can see the rocks. You can see one of the places where one of the air bags didn't deflate completely. All of this 3-D, walk-through visualization is using standard Java APIs like Java 3-D API, Java Advanced Imaging API, Java networking APIs and the user interface APIs.

Using the Maestro version they are distributing, not only do you see a 3-D model of the terrain, you see a 3-D model of the Rover. You can actually drive the Rover on this simulated terrain. It has this "video game" aspect to it. Except that you're actually driving it on a terrain model that's real. It's real Mars data that's constructing this terrain. It's not like playing Dune, where you're going through this maze that's completely fictitious.

Q:How has it been to work with the JPL scientists on this project?
A: I've spent a good amount of time down there with JPL, not only interacting with some of them, but I'm also on one of their advisory boards. In terms of talent density, IQ points per square meter, it's just an amazing place. Plus, they are doing things that most people would think of as science fiction. Most people read science fiction stories about driving dune buggies on Mars. These guys actually build them. They actually know how to fly between the planets. I've spoken to some of the guys that do interplanetary navigation, and that is really spooky stuff. You actually have to pay attention to relativity, the fact that time is not a constant - the faster you go, the slower things are. They function in a world where relativity actually matters. They are way outside of Newtonian mechanics.

JPL is a place to go to have your mind blown - partly, because they are really charged up about what they do. This is a crowd of people who are living a dream. What they are doing is so out there, so wonderful. They are doing something that is very heroic, noble, exploratory and exciting. They are the only part of the U.S. government that I can really get excited about. NASA has this incredible public outreach program, because they know that they are loved and it's a tremendous public service. They do lots of stuff with schools. The fact that they put this stripped-down version of Maestro out there is a wonderful piece of public outreach.

Q:What is it about Java that makes it so attractive to this type of application?
A: The answer is there's a bunch of things, not just one thing. One of the aspects of Java that was really important to them is that it runs on a lot of platforms. If you look at JPL, they've got Solaris boxes, Linux boxes, Windows boxes, Apple boxes - it works on all of them. If you look at the standard available API libraries available for Sun, there's a huge toolkit of things that you bring to bear. There are things like the 3-D modeling APIs, the Advanced Imaging APIs, and all the user interface APIs and networking APIs. The JPL guys used all of them. They were able to leverage all of these standard tools.

Plus, there's been a lot of experience with Java where folks have measured things like developer productivity. For example, if you compare how long it takes someone to develop a piece of software in Java versus C or C++, essentially all of the measurements show it to be about twice as effective. So if it takes some team of engineers ten months to do something in C++, it will take them five months to do it in Java. For an outfit like JPL that does everything on a shoestring budget, that also means you can trim it down five months - or still do it in ten months, but trim it down to half as many people.

There are also a number of aspects to Java that are all about building more reliable applications. It is a lot easier to build things that are more reliable - that break less often - so you don't have to worry about blowing your machine to bits. There's also a lot in Java that's all about security. So when you've got something like their large databases - whose integrity is something they need to be careful about - security is important. So, it's a whole bunch of things that all swirl together. Whenever you talk to Java developers, you get a different answer to the "why did you do it Java" question. Although, there tends to be a standard set of themes.

Q:Is this the first time that Java has been used for this type of application - being beamed out into space?
A: Actually, I don't think so. I've talked to people that have been doing various things with satellite ground station control and people that control systems for giant telescopes. People are doing stuff like this all over the place. The JPL project is what people tend to talk about because it's too damn cool for words.

Java Technology and the Mission to Mars
By Dr. James Gosling, CTO, Developer Programs
Sun Microsystems, Monday Jan 5, 4:00 PM PT

This weekend I saw what has to be the coolest Java(TM) app ever. We talk about "Java everywhere," which usually means Java(TM) technology in vehicle navigation, imaging, control consoles and things like that. This time "everywhere" means all of that and then some, because Java technology is playing a big role in NASA's latest, highly successful, Rover Mission to Mars.

On Saturday night I watched as a control room full of immensely tense geeks explode with joy as they successfully landed the six-wheeled Mars Rover "Spirit" on the planet's surface. From there, scientists at NASA's Jet Propulsion Laboratories in Pasadena will get to use their powerful, Java technology-based, ground side control system to maneuver Spirit on the Martian terrain in what has to be the most amazing network "video game" in history.

Java 3D and Java Advanced Imaging technology are also key to the software JPL is using to render and interpret realtime images captured by the Rover. NASA has even made a stripped down version of the software that you can download so you can view a simulated 3D landscape and drive the Rover around in it.

There are all kinds of reasons JPL is using Java technology for control and imaging systems for the Rovers. NASA engineers had access to hundreds of specialized APIs and network protocols that they needed to bring this off. They got great productivity and reliability. The data they're collecting through this program is part of a distributed, collaborative network of scientists and engineers, and the ability of Java technology to run on any platform enables a secure, reliable global dialogue within NASA's scientific community. Now that Java has helped us get to Mars, who knows what "Java Everywhere" will mean in the future?
- posted by Andy @ 12:04 AM