OPERATIONS STAFFING
The science – and art - of satellite mission operations is now more than 50 years old. As the field enters its second half-century, current practices represent advances in capability that early mission controllers could never have imagined, and emerging technologies hold forth the promise of further re-defining this field. Yet, one constant that remains as valuable today as in the earliest days of mission operations is the human presence, as necessary for robotic missions as for human spaceflight. True autonomy in mission operations remains beyond the reach of current flight experience even as advances continue to be made in this area. For now and for the foreseeable future, men and women will continue to define and shape the practice of satellite mission operations.
This paper expands upon principles originally laid down in 2003, which focused primarily on operations staffing for robotic missions. While retaining that basis, the scope is extended to include additional background material, as well as the evolving Constellation Program, NASA’s plan to return to the Moon in response to the President’s Vision for Space Exploration. That architecture, which includes multiple vehicles, multiple missions, and multiple crews, all operating simultaneously between Low Earth Orbit, the Moon, and bases on the Moon, calls for a radical re-assessment of current staffing processes and practices lest the costs and technical barriers engendered by our present-day thinking lead to undesirable consequences. An assessment of staffing considerations for this program promises to considerably broaden the current understanding of operations staffing.
FORCE FIELD ANALYSIS
There is a permanent pressure to provide safe and successful operations for less money. We live in an era in which "low-cost" is essential to survival in the aerospace industry. In such a climate of cost constraints, the focus has turned to life-cycle costs, as it should.
Originally, satellite operations were designed with dedicated operators, engineers, support staff, and managers stationed in dedicated, specially-built control centers, using dedicated, custom-built, expensive one-of-a-kind equipment. The traditional control room concept originated in the age where computers were very bulky, power-hungry, and very expensive, and where networking was - at best - slow, serial, and dependent upon unreliable dial-up lines or expensive fixed lines over the public telephone network. The software was proprietary and expensive to design, operate and maintain. These centers were staffed 24 hours per day, seven days per week by rotating crews stationed on-console monitoring telemetry to ensure the health and safety of the satellites.
While highly successful in terms of managing spacecraft and collecting data, even up until relatively recently, that paradigm of operations is also expensive and inefficient. Its viability is fading due to budgetary reductions, evolving ground system and range architectures, and the increased number and complexity of missions.
Depending on the level of sophistication of the satellite and/or the ground system, varying levels of automation have been introduced with largely positive results. Such an implementation frees the full-time crew from monitoring telemetry, thereby enabling them to perform more complex tasks, and typically allows a reduction in the number of hours where an operator is on console - permitting, for example, a control center to be staffed for 12 hours/day, 8 hours/day, or even only Monday-Friday – with obvious cost benefits.
But this is a double-edged sword. While cost savings assume a greater importance in today’s economic situation, lights-out operations must not significantly degrade the quality of operations just to save money. The challenge is to find an optimal balance between three warring influences: budget, skill, and risk.
Satellite operations – and the attendant satellite monitoring systems - are becoming more complex as technology advances and the scope of mission objectives increase. The number of telemetry points that need to be monitored has increased dramatically. While this presents no significant challenge to even a modest automation approach, it is the control center personnel – drawing upon their knowledge, insight, and experience – who must recognize the relationship and inter-connectedness of subsystems, the significance of a reported violation, and the immediate response to it that must be taken. The additional telemetry allows the operator to have more insight, but adds an element of complexity to the system and requires satellite operators to develop a more in-depth and detailed understanding of the satellite. Such ability translates to increased ‘required knowledge-base’ sought in contemporary satellite control center engineers, which increases the costs of engineering staff.
Over time, enhanced satellite on-board computing capabilities have resulted in addition of software embedded to enable the day-to-day operations. Older satellite platforms used to have very little or no on-board software – most of the operations were mechanical. Now, virtually all operations are software driven. Again, this facilitates the controllers’ work, but it also increases the training and skill requirements and hence the costs of operations staff.
The major components of satellite operations costs are those for staff and the infrastructure (hardware and software) for spacecraft monitoring and control. Ignoring for purposes of this paper the infrastructure costs, the costs for typical staff functions and responsibilities can be grouped as follows:
Function / Responsibility· Ongoing spacecraft control / · Lower cost controllers
· Planning routine maneuvers / · Higher cost orbital analysts and engineers
· Planning and execution of anomaly responses and special campaigns / · Higher cost staff
· Training of newly hired engineers, analysts, and controllers / · Lower cost controllers for OJT
· Higher cost staff for background and perspective
· Training when new operating hardware/software is installed / · Higher cost staff
It is clearly desirable, from a cost perspective, to have a satellite design that facilitates operations and maintenance by lower-cost controllers, with the involvement of higher-cost analysts and engineers reduced as much as practicable. However, such an approach is only feasible when routine operational tasks (in the post L&EO and commissioning phases) are of such a nature that they can be done without too much knowledge of the satellite and its payload. However, as noted earlier, this runs contrary to the prevailing evolving design of satellites and payloads where aggressive mission requirements often dictate greater complexity.
EXPLORATION OF ISSUES
Through the combination of more capable satellites, advanced ground systems, and intelligent monitoring systems, an era of “lights-out” operations for unmanned satellites has begun to emerge over the past decade. During “lights-out” operations, the ground segment is run autonomously during most routine operations. Members of the satellite control team need to be involved only in those tasks that are too costly or too risky to automate. The rest of the time, the team members are on-call in case of anomalies or emergencies. When the automated ground system detects an anomaly, it immediately alerts the appropriate on-call team members and provides them with the most current information so they can assess the potential implications and, through the use of appropriate technologies (secure remote user access, intelligent paging systems, wireless connectivity, etc.) to ensure a time-critical response, start to resolve the anomaly remotely.
A typical NASA mission using an approach similar to that outlined above generally needs only one operator working a single shift 8 hours/day, 5 days/week. Prior to lights-out automation, a typical satellite operations center employed 2 - 3 operators per 8-hour shift, 3 shifts/day, and 7 days/week. Thus, it is easy to see that the proper use of automation can translate into significant savings in operations costs alone.
But these cost savings extract a price in other areas.
The set-up of the reduced size flight operations team works without much redundancy on the personnel side. While the consequences of this are plain, there are other – more subtle – issues that should be recognized. For example, mission knowledge now often resides solely in the person, not also on paper as was customary in the past. This poses the attendant risk that the comprehensive documentation of some older missions is not maintained to the same degree. This gap in institutional knowledge – particularly from experience gained in pre-launch testing and integration activities – raises the potential that subsystem linkages and behaviors might be misinterpreted or not recognized at all which, in the event of an anomaly, could result in an operations team taking an inappropriate action or failing to take an appropriate one.
The trend in spacecraft mission operations is to use autonomy to the greatest extent possible, as it has been shown that increased autonomy results in reduced operations costs. Balanced against this savings, however, is an increase in software costs to develop and implement the desired degree of autonomy. Despite encouraging trends in automated code generation, the cost of software development is still very high, as software is extremely personnel intensive to develop.
Given the assumption that mission safety and reliability will not be compromised by the introduction of autonomy, it is essential that all software be rigorously developed and validated to minimize risk. An emphasis on design reviews, documentation, and Independent Validation and Verification (IV&V) efforts, which are essential elements of a highly reliable software development program, create additional burdens that increase the software development cost.
Consequently, there are practical limits to the application of autonomy to spacecraft mission operations. As staffing levels continue to be reduced, it becomes increasingly difficult to eliminate more operators without increasing risk. Offsetting this risk with autonomy software requires an increasingly intensive design and test program. At some level, it becomes more costly to develop a fully robust autonomy capability than it is to use a small operations team augmented by selected automation capabilities. The goal is to find the point where the sum of staffing costs and software development costs is minimized.
Furthermore, an over-dependence on autonomy can ultimately result in a lack of operator preparedness in the event of a spacecraft emergency. The operator's situational awareness and familiarity with the spacecraft and its subsystems’ behaviors must be maintained; indeed, it can be argued that the implementation of automation requires greater diligence, as it is so easy to fall into the trap of trusting the software to always do the right thing. It is imperative that the operators remain close enough to the operation of the spacecraft that they can respond in a timely and appropriate manner when a spacecraft contingency or emergency arises. Ultimately, the assessment of a situation remains in the hands of the human operator, whose familiarity – or lack thereof – with current on-orbit conditions, expected and telemetered behaviors, and available options may determine success or failure at that moment.
CONSTELLATION PROGRAM CONSIDERATIONS
[THIS SECTION WILL NEED MORE DETAILED INPUT FROM THE GROUP.]
Constellation has operational needs that are not addressed by automation as implemented in support of unmanned space missions. Current development efforts are not focused on, and in some cases not applicable to, Constellation needs. That approach is not scalable to support the Constellation architecture for space exploration, which includes multiple vehicles, multiple missions, and multiple crews, all operating simultaneously between Low Earth Orbit (LEO), the Moon, and bases on the Moon. These elements will require a suite of core, adjustable mission operations technologies that work together to provide a seamless electronic process that cross mission and vehicle lines, and use identical tools and processes. Otherwise, the consequences will be high costs for mission operations due to proliferation of disparate tools and person-intensive operations practices, high barriers to automation due to lack of tool interoperability, and high costs to retire safety concerns as the tools evolve in response to changing mission needs.
1.0 Operations Staffing Best Practices
Operational staffing is dependent on many variables, including (in no particular order of importance):
The available skill set of personnel
The available budget
The complexity imposed by the spacecraft design, ground system design, and other mission requirements (e.g., maneuvers, onboard reconfigurations, etc.)
Any required support of interactive payload operations
The degree of automation that has evolved
The number of spacecraft supported by a single control center
Any network elements beyond the control center that are managed by the team
Any support functions (e.g., data processing, orbit determination, etc.) performed by the team within the control center.
Once upon a time, just one shift of an operations team might have been comprised of 5-7 (or even more) personnel, and there would have been shifts scheduled to provide 24/7/365 support. Added to these numbers would have been off-line engineers, analysts, technicians, and other supporting personnel. And that’s not even bothering to mention managers and administrative support! Such large numbers were necessary due to the complexity of the spacecraft and ground system, the reliability and maturity of the on-orbit, network, and ground system elements, basic mission requirements, and a host of other factors.
But times have changed, technology has advanced, and what once was deemed risky is now considered routine.
From a functional point of view, we now have Flight Controllers (who typically manage both on-orbit activities and the ground system) directly supported by Spacecraft Operations Engineers and Mission Planners/Schedulers. Ground System Engineers, Flight and Ground Software Engineers, Systems/Database Administrators, and Flight Dynamics Analysts perform supporting functions, which may vary in frequency and intensity depending upon the mission design and requirements. Expanded explanations of the typical roles and responsibilities of these functions are presented below. It should be noted that, in some instances, certain of these functions can be assumed by several or even all members of the operations team – this illustrates the high skill level demanded of those who now serve in this capacity.
2.0 Functional Descriptions
2.1 Direct Operations Functions
Flight controllers are responsible for managing the spacecraft commanding and data capture activities for assigned spacecraft contacts., In this sense, ‘managing’ does not necessarily translate to actually being in the control center when scheduled contacts occur; rather, it often consists of fabricating and verifying command loads and the planned schedule of pass activities in advance, ensuring identified constraints and restrictions are addressed, and being prepared to respond to anomalous indications or unforeseen interruptions to the schedule. Under certain conditions (e.g., critical maneuvers, anomaly recovery operations, etc.), this responsibility will dictate real-time interactive command and control of the on-orbit asset but, under routine conditions, this is often performed by the automated system (if one exists).