A patch was sent to Voyager 2 yesterday

Still less challenging than me updating my rolling GNU/Linux distribution daily, with an internet connection not particularly faster than a snail…

A must-read for V’Ger fans:

Prior to the start of VIM [Voyager Interstellar Mission], major FSW [Flight Software] modifications were done in all the computers (AACS, CCS and FDS). All the non-essential routines, for example, related to video cameras or encounters, were deleted and some of the codes were relocated to make the memory more available and contiguous, so new functions for VIM could be added for a 30-year long-term mission.

During the prime mission and early VIM, Voyager had been using the JPL-developed software programs called SEQTRAN (to generate sequences) and COMSIM (to simulate sequences and CCS FSW changes). They ran on now-antiquated UNIVAC mainframe computers. Shortly after the start of VIM, these programs were converted over to more modern UNIX-based SEQTRAN and High Speed Simulator (HSSIM). They were rewritten to maintain the same functionality of the old SEQTRAN and COMSIM, and tailored for VIM. Rewriting and testing required significant effort from the developers and the project personnel; however, the end result is much improved speed and efficiency.

In the section II.B. “Aging Hardware”, you will learn about all the hardware degradations and failures in the two probes. Then about the other challenges:

Disappearing expertise, in personnel and documentation, is also something that makes the operations challenging. Many of the personnel who designed and built the spacecraft have passed on. Losing expertise is also a challenge for the science team not only in operating the instruments but also in reviewing and analyzing the data.

To make the best use out of such limited memory spaces, different programmers used all kinds of tricks in maintaining and adding patches over the life of the mission, resulting in extremely unstructured code that is prone for mistakes. It is crucial to validate all the sequence products and any kind of FSW changes thoroughly in HSSIM.
Both the AACS and FDS use assembly language. The CCS uses assembly language and Voyager-unique pseudo code (interpreter). As a result, it is difficult to attract younger programmers to join the project.

The section III.A. is listing all modifications made in the Flight Software. Interestingly, as the Radioisotope Thermoelectric Generator power is decaying, instruments are switched off and their routines can be deleted from memory and replaced by new useful routines.

And many instruments have heaters to operate in cold space. As they are switched off, it is necessary to model the thermal effects on the spacecrafts:

It was decided to build a new desktop model to analyze the thermal effects of turning off subsystems in order to conserve spacecraft power and model the effects on the propellant lines. The model has been completed and is being validated by the project.

III.D Other efforts:

In addition, the team employs some unusual approaches, such as keeping “ex-Voyagers” informed as much as possible and maintaining good rapport with retirees.

5 Likes

Archeological work in progress… :artificial_satellite:

I have found interesting archives in that page of a site of the Goddard Space Flight Center (GSFC) dedicated to VOYAGER Cosmic Ray Subsystem: VOYAGER LIBRARY
and found a few FORTRAN occurrences by exploring some PDF files.

Page 8 of the PDF:

The data processinq and analysis system developed at GSFC provides automated data reduction and archival, as well as some automated data analysis for these experiments. The system was developed using IBM assembly language and FORTRAN, and resides primarily in the Science and Applications Computer Center (SACC) IBM S/360 systems at GSFC.

But no source code is shown in the report.


Starting from page 8 of the PDF, many FORTRAN subroutines are described (dated from 1977-78). No FORTRAN code shown, but pseudo-code describing each subroutine. That document is indeed the description of what we would nowadays call an API:

The FORTRAN calling sequence for BAKLOG is as follows:
CALL BAKLOG (QSATID,ITYPE)
The following list describes the arguments in the calling list

Page 73, we can see a reference to what could be specific flavors of the language (or compilers?):

4. Language:
FORTRANG or FORTRANH level 21.6 360/91/75 OS/MVT

also referenced in that title: System 360 Operating System FORTRAN IV (G and H) Programmers Guide
And in that Wikipedia page: “the versions implemented by the MFT/MVT compilers known as FORTRAN G, FORTRAN H and PL/I F.”

They seem in fact to be an optimizing compiler and a debugging compiler:

  • “the IBM FORTRAN H compiler allowed the user to specify no optimization, optimization at the registers level only, or full optimization.” Optimizing compiler - Wikipedia
  • “The OS/360 Fortran G compiler has a debug packet feature.” COMEFROM - Wikipedia

The third column probably refers to IBM 360 machines:

1 Like

Voyager 1 is still a bit dead, but some hope remains…

The latest problem with Voyager 1 lies in the probe’s Flight Data Subsystem (FDS), one of three computers on the spacecraft working alongside a command-and-control central computer and another device overseeing attitude control and pointing.

In November, the data packages transmitted by Voyager 1 manifested a repeating pattern of ones and zeros as if it were stuck, according to NASA. Dodd said engineers at JPL have spent the better part of three months trying to diagnose the cause of the problem.

So far, the ground team believes the most likely explanation for the problem is a bit of corrupted memory in the FDS. However, because of the computer hangup, engineers lack detailed data from Voyager 1 that might lead them to the root of the issue. “It’s likely somewhere in the FDS memory,” Dodd said. “A bit got flipped or corrupted. But without the telemetry, we can’t see where that FDS memory corruption is.”

In the next few weeks, Voyager’s ground team plans to transmit commands for Voyager 1 to try to isolate where the suspected corrupted memory lies within the FDS computer.

2 Likes

The most faraway memory dump of all times brings some hope:
https://blogs.nasa.gov/sunspot/2024/03/13/nasa-engineers-make-progress-toward-understanding-voyager-1-issue/

The team will compare this readout to the one that came down before the issue arose and look for discrepancies in the code and the variables to potentially find the source of the ongoing issue.
This new signal resulted from a command sent to Voyager 1 on March 1. Called a “poke” by the team, the command is meant to gently prompt the FDS to try different sequences in its software package in case the issue could be resolved by going around a corrupted section.

(If you remember what is a poke, you are probably more than 45 years old! :slight_smile: )

2 Likes

And if you know what the complimentary peek command is you definitely are over 45.

2 Likes

Voyager 1 on the road to recovery?

“It’s a part failure on one of the memories and they’re looking for a way to move a couple hundred words of software from one region to another in the flight computer,”

https://blogs.nasa.gov/voyager/2024/04/04/engineers-pinpoint-cause-of-voyager-1-issue-are-working-on-solution/

The team suspects that a single chip responsible for storing part of the affected portion of the FDS memory isn’t working. Engineers can’t determine with certainty what caused the issue. Two possibilities are that the chip could have been hit by an energetic particle from space or that it simply may have worn out after 46 years.

1 Like

The title reminded me of this story of a Lisp programmer at NASA debugging code on a satellite that was reported to be about 150 million miles away from earth when it suffered from a race condition that was supposed to be “impossible” based on a formal proof. I scanned the thread and did not see it mentioned explicitly.

1 Like

“The problem was that there was, in fact, a race condition. Which was supposed to have been impossible.” Unfortunately, one of Garret’s coders had called a lower-level Lisp function — which had inadvertently created “an end-run around the safety guarantees” of their carefully-customized language. (Garret blames himself for not explaining this more clearly to the coder.)
The team decided to “manually” trigger the event — which got the software running again.

Interesting. I did touch just the basics of Lisp a very long time ago. I would like to start again playing with it, but my days are 24 hours… Maybe in another life.

1 Like

The critical insight into Lisp:

  1. Lisp programs are also data structures in the language. Code can also be interpreted as data, which the computing scientists call “homoiconic”.
  2. Conventional Lisp program are essentially abstract syntax tree. So you are really programming a compiler.
  3. You can turn Lisp into a more conventional appearing language, without losing any of the benefits of the code as data representation, by using David A Wheeler’s readable S expression syntax:
    https://readable.sourceforge.io/

Traditional Lispers seem to hate it, but I suspect some new generation of Lisp hackers will adopt it.

1 Like

@vmagnin - Thank you for the useful information you and others have scattered throughout this thread. I began attempting to debunk the Voyager-Fortran 5 claim in early February. I belatedly discovered this thread and found you were doing similar research in parallel with me. (Obviously not in parallel as you did yours a few months before me!) Some of the resources here were ones I had already found and a good number were new to me. (As well as @ivanpribec’s separate thread, “Data General Fortran 5 is a Real Pig”!)

Anyway, the results of my research …

The Voyager spacecraft were NOT programmed in Fortran 5 (which ran on Data General minicomputers). The source of this claim, the 2013 Wired “Interstellar 8-Track” article, was actually referring to ground system “control and analysis” software. (The article’s “control and analysis” qualifier is almost always dropped when the claim is repeated elsewhere on the web.)

The article was based on an interview with Suzanne Dodd, a Voyager sequence engineer in the 1980s who returned to Voyager in 2010 as its program manager. A sequence is, simply speaking, a sequence of operations to be performed on a spacecraft during a time interval. For example, point the camera at Jupiter’s Red Spot, turn on the 8-track tape recorder, turn on the camera, turn off the camera, and stop the 8-track.

Sequences were developed in the 1970s and 1980s using tools written in UNIVAC FORTRAN V running on UNIVAC 1100-series mainframes at JPL. My guess is that the interview with Dodd was by phone and she said “Fortran Five” for FORTRAN V. The evolution to Fortran 77 and C was typical for ground systems, as the mainframes were replaced by minicomputers and then PCs or Unix workstations.

The flight software running on the Voyagers’ onboard computers was and is written in assembly language. See this 2016 paper also referenced by @vmagnin elsewhere in this thread, “Voyager Interstellar Mission” (PDF: https://arc.aiaa.org/doi/pdf/10.2514/6.2016-2415), p. 6, by Sun Kang Matsumoto, a long-time and current Voyager flight software engineer. There would have been 2 or 3 assembly languages for the three sets of computers. (2 or 3 because the AACS CPU was a modified CCS CPU and I don’t know if the modifications affected the instruction set.)

Matsumoto’s paper, p. 4, also discusses the original use of Univac mainframes for the ground software. Other sources identify the mainframes as the 1108. Univac’s FORTRAN V became available in the mid-1960s, so it would have been the standard Univac compiler for Voyager in the 1970s.

More sources and a more detailed examination of the Fortran 5 claim can found on my web page: http://www.geonius.com/writing/other/voyager.html


In my research, I did encounter something that might be of interest to the Fortran Discourse group. For the two-years-earlier Mars Viking mission, Martin Marietta Corporation (MMC) had the contract for developing the Viking Lander ground system software. MMC wrote and tested the software on a 60-bit CDC 6500 mainframe at their Denver, Colorado facility. An automatic conversion process then converted the software for compilation and running on the target 32-bit IBM 360 and 36-bit UNIVAC 1108 mainframes at JPL’s mission control center in Pasadena, California. This was done for good reasons (insufficient computing power at JPL) and with a lot of planning. Ultimately it was successful, although the IBM computers were a major headache. See “Clash of Viking Mainframes” on my web page under “Miscellaneous Information”. (I’m a new user and limited to 2 links per comment!)

5 Likes

Thanks a lot @alexm for these precisions and for sharing your huge work on your page. I will print it tomorrow and read it eagerly those next days and even next week, as it is ~36 pages long!

1 Like

I can understand that completely. Especially when it is about the Voyagers. NASA’s best program ever.

1 Like

Thanks @alexm for the research! It’s great to have accurate historical account.

1 Like

As a side note, Space Shuttle also followed a similar path. While Fortran played a big part in the design of the shuttle and later on the ground based software used in mission planning and trajectory design, about 85% of the shuttle on board avionics software was written in the HAL/S language developed for shuttle to run on the IBM onboard avionics computers. See the following for a description of HAL/S (High-order assembly language/Shuttle).

My first engineering job was working on a large Fortran code called Space Vehicle Dynamic Simulation (SVDS) for a NASA contractor in the late 1970s prior to the first Shuttle flight. I was responsible for implementing ascent and on-orbit guidance algorithms (including abort simulations) into SVDS. The SVDS program served several purposes including validation of the algorithms as well as mission planning etc. SVDS ran on the Johnson Space Center Univac 1108 systems (64K of memory) so you learned how to program overlays from the start. On one ascent simulation I ran, I missed a decimal point on the main engine throttle setting which triggered an abort due to all the SSME fuel being burned up prior to SRB seperation. I basically wiped out a large portion of Orlando :smile:

4 Likes

Thank you!

1 Like

I thought it was getting long, but I didn’t realize it was 36 pages long! Sorry! For other folks, it looks like the main subject is covered in about the first 10 pages (sections 1-5), including pictures to break up the monotony a little bit. I hope that helps.

1 Like

Well, I used another printer and it was finally 44 pages. I am now at page 37, and will finish reading it this evening. Thanks for that detailed enquiry. Of course each one will be more interested by such or such section, but it is a pleasant reading for computers and space fans!

Wow… A huge step toward total recovery! :grinning: :open_mouth: :face_with_peeking_eye:

2 Likes