Runtime Code Modification for Avoidance of License Server Authentication

Runtime Code Modification for Avoidance of License Server Authentication

Gregory Deych and Gokul Nadathur

Computer Sciences Department

University of Wisconsin – Madison

{gdeych,gokul}@cs.wisc.edu

December 6, 1999

A general strategy for avoiding authentication by license servers has been developed. The strategy involves the modification of the execution path of the licensing process at run time without the process recognizing that such a modification has taken place. The necessary tools for such an exercise were created using the Procfile system interface provided by Solaris OS. These tools have the following function: stopping a process at call points to system calls, forcing a process to trace certain signals and faults and modifying the address space of the process at runtime. The strategy was tested on Adobe Systems Framemaker and the licensing process was subverted.

1Introduction

This paper chronicles our investigation of the license server – client interaction. Our project concentrates on bypassing or removing the elements of licensing that relied on communication with the license server. It is possible that some aspects of licensing depend on other methods, in which case they are considered to be outside our purview for the purposes of this project.

Our strategy relies on steering the execution of the program at runtime. We do not make any changes to the static object code, which means that our mutator must be executed every time the program is run. The strategy was tested on Adobe Systems Framemaker. We launch two processes: a clean Framemaker process, with licensing code intact and a Code Changing Tool that will create all the modifications to the Framemaker process. The tool process terminates itself after modifications have been carried out, leaving only the Framemaker process with network licensing functions disabled or bypassed.

We begin by describing the general licensing server paradigm in Section 2. In Section 3, we describe our strategy for avoiding the licensing process. In Section 4, we discuss the tools we have built to implement our strategy. In section 5, we describe our experiments on Framemaker as well as other programs. We finish the paper by describing additional enhancements that could be added to our tool set to achieve additional functionality, as well as how application makers may safeguard their programs from the techniques we have utilized.

2Mechanism behind using a License Server

A license server is supposed to keep applications from running, unless it is satisfied that the user requesting the application possesses a proper permission to run that application. This permission may be a personal license for that particular user or a site license specifying a number of applications that can be run at the same time. Various license servers implement a particular license-managing model, or implement several to be chosen at the convenience of the site administrator. The application programmer may develop the license server together with the client application, or the server may be an external product integrated with the application. A diagram of a license server operation is depicted in Figure 2. The application submits a request to be executed to the license server, together with some identifying information. The particulars of the information depend on the license in question, but commonly include the user name and location of the machine on which the request is to be executed. The license server evaluates the request, records it in the license database and sends the release to the application. Upon completion, the application typically notifies the server that its use of the license is finished.

3General Strategy for Avoiding the License Checking

The client application must contact the license server to determine if it is in fact licensed. If it is unable to do so, it defaults into unlicensed mode. Our general approach to the problem is two-fold: to modify the runtime process to steer the execution path around the licensing checks and correct any error flags generated due to such a modification. While the communication with license server will not affect our approach, its absence validates our result. Since we had decided to do all of our modifications dynamically, we needed a set of tools that would allow us to modify the process in the memory, as it is loaded and executing.

By stopping the process and disassembling its functions, we can obtain information vital to our strategy. First, we need to define a few terms:

• Trace point – the virtual address of the instruction that would need to be modified, as well as what instruction would need to overwrite this address.
• Trace of a process - the set of all trace points of a particular process

Our task is to analyze the runtime binary in order to determine its trace. An intuitive step would be to stop the

process when it executes a connect call. The process can then be loaded into a debugger, and the process stack examined. This strategy enables us to build a partial process call graph, constructed from the bottom up. Once we have created this graph, we are ready to examine the program’s behavior as we mutated the graph by pruning some of its links.

Our mutation strategy employed one or more of the following schemes:

• Remove a particular function from the execution tree.
• Eliminate the detection of the removal of the function.
• Branch to points that would be executed if the licensing was carried out successfully.

If our mutation of the trees caused an error condition, the exit points would lead us to the location of the error. Once the location of the error is uncovered, we either bypass the error detection code or disabled the error exit code, which prevented the process from properly responding to the lack of licensing information. By applying this method, we are able to disable a large portion of the licensing functionality.

4Tools

4.1GDB debugger

We use GDB primarily as a disassemble tool, breakpointing on an entry to a function we want to investigate and disassembling it. This allows us to see the assembly code of the function, which gives us some insight into its function as well as strategies for its modification. Unfortunately, due to the lack of debugging information in the program binary, use of GDB as a debugger is impossible.

4.2Tools Using Proc File System

We had to create a number of tools to allow us to implement the changes to the runtime program code. For this, we have chosen to use the Proc file system, which is used to access the state of the process in the Solaris operating system.

/proc - the process file system

/proc is a file system that provides access to the state of each process and light-weight process (lwp) in the system. The name of each entry in the /proc directory is a decimal number corresponding to a process-ID. These entries are themselves subdirectories. Additional files contained within each subdirectory provide access to Process State. Standard system calls like open, close, read and write are used to access the files in the /proc directory. The following is the description of files in the /proc directory that are relevant for creating the tools.

Structure of the /proc/pid (pid - process id)

as -This file contains the address-space image of the process. It can be opened for both reading and writing.

ctl -A write-only file to which control messages can be written, directing the system to change some aspect of the process’ state or to control its behavior in some way. These control messages include stopping a process at the entry and exit points of specific system calls and forcing the process to trace certain signals and faults.

status- The status file contains state information about the process and the main thread of the process. When the behavior of the process changes in certain predefined ways such an execution of a system call, reception of signals or when the state of the process changes, appropriate information is written to the status file.

Tools created using /Proc:

The tools created using the Proc file system had the following functions.

1. A process could be stopped at the entry or exit point of a system call by writing the control messages PCSENTRY/PCSEXIT to the ctl file of the process. When it is stopped at the entry to the system call, the parameters have already been loaded. The values of these parameters are stored in specific fields in the status file. So for every particular system call that the process emits it is possible to examine the values it passes. This can be used to get information about the domain (TCP or UDP) of its socket calls, information about the server it connects to and the byte sequence it sends and receives.

2. A process could be made to trace certain signals and faults and stop when such signals or faults occur. The signals or faults might be triggered by certain operations inherent to the process or due to certain control messages written to its ctl file by another process.

3. Modifying the Address space of the process:

(a)The address space of the process can be modified by writing to the as file in the /proc directory of the process. We synthesized machine instructions for branching (conditional or unconditional), no – op instructions and for moving data between registers. After these instructions have been synthesized, the address space was modified at certain addresses as dictated by the trace points. By modifying those addresses, we were able to steer the execution of the program according to our strategy.

(b)To examine the results of certain conditions or the return values of certain functions, temporary modifications were made to the address space in the form of breakpoints at arbitrary instruction addresses. This is beyond the capacity of GDB, since it recognizes only points like function entry or exits if debugging information is not built into the executable code of the process. After a breakpoint has been set, there are two more steps to be done. The breakpointing process has to be forced to trace the relevant faults, which can be done using the tools in step (2). Secondly, there has to be a mechanism for the external tool to identify whether the process that it’s monitoring has faulted on the breakpoint or not. This can be accomplished by polling on the file descriptor of the status file. When a fault occurs, the reason for the occurrence is written to the status file. By polling the status file for normal or high priority write operation, the tool recognizes the execution of the breakpoint instruction. After the process stops on the breakpoint instruction and has been examined, the original instruction (replaced by the breakpoint) is written back and the process is continued.

4. Watching the process:

Another useful operation is a watch on the address space of the process. A section of the address space of a process can be watched for read, write or execute operations. The watch operation forces the process to fault when it does the specified activity on the watched area of the address space. By forcing the process to trace the fault, the process can be stopped and its state can be examined. This is useful is when the address of the data where some manipulation would take place is known, but not the instruction that would perform the manipulation. When a process receives data from the remote server, the buffer in which the process places the data can be determined by examining the parameters of the recvfrom() call. The points in the code of the process that would manipulate this buffer can be found out by watching this buffer for a read or a write access. The point can then be used as the root of the control flow tree that essentially traces how the licensing is done.

5. Monitoring multiple processes:

Another interesting problem lies in developing mechanisms to monitor multiple processes. This is useful when licensing is done in one of the child process of the process that is being monitored. An example of this situation is cited in section 5.2. The Proc file system offers a simple and elegant solution to this case. The idea is to setup the proc state for which the child should respond in the parent process itself and turn on the "inherit on fork mode" in the parent process. This causes the child to inherit the proc state of its parent including the signals, faults and system calls it should trace. However, it should be noted that watch areas are not inherited.

5Applying the Strategy

5.1Framemaker

Adobe Systems’ Framemaker is a desktop publishing application for the UNIX environment. It utilizes FLEXlm license server to control access to itself. Framemaker is started by invoking a series of script files which set up environmental variables, determine the machine architecture the client is to be run on, and execute an appropriate binary.

During startup Framemaker executes a number of functions related to general initialization of its state, as well as some directly related to the licensing process. The functions named Flm* control the general licensing process. Other functions, of type Nl* control network licensing and communication with the license server. In fact, much of the licensing is accomplished through the function NlInit, executed from the main routine. NlInit performs license checking on startup of Framemaker. Another license check is performed when a user opens a file. In addition, the client application contacts the server at the end of the user session to notify the server that the license has been released. By applying our strategy, all contact between license server and Framemaker is eliminated.

The flowchart (figure 4) depicts the structure of the main() function of the Framemaker. The first branch is the connection to the printer server. The connect calls performed in the NlInit are the primary connections to the license server. As we have found out, if those are disabled Framemaker tries to perform the licensing one more time on Open File and New File commands. In addition, it also contacts the server on Exit. As pointed out in the flowchart all the licensing calls in these cases flow through the function CheckOutLicense(). The actual licensing was done NlGetLicense() that returned a value 1 on success and 0 on failure.

The mutation done to CheckOutLicense() is as follows:

• Erase call to NlGetLicense .
• Change the register where the return value was stored to one.

Identifying that the license mechanism has been bypassed

Two ways were used to verify our traces.

1. The connect calls emitted by the process and the points at which they were emitted was traced.
   After applying our trace to the Framemaker, we eliminated all connect calls which were not related to licensing, while preserving the XServer and Print Server connection. The results are presented in Table 2.

1. We have further tested our theory by supplying an incorrect license server to Framemaker. This was done by removing the initialization to the Environment variable FM_FLS_HOST . This initialization was done statically in a script file that was executed before the actual Framemaker executable was overlaid on the process space. The default license server was named stimpy. When Framemaker was run using the modified script, it defaults to demo mode giving a message that license was not installed. When our traces were applied, Framemaker executed with full functionality.

5.2 Experiences with Purify

This section illustrates a situation different from one in Framemaker and cites mechanisms for dealing with it. Purify, a runtime error detection tool from Rational Software, also uses a server to perform license authentication. Since Purify itself launches several processes, the problem of multiple processes being involved is inherent and cannot be avoided. During the preliminary experiments we conducted using Dyninst (yet another powerful tool for parallel debugging), we found that Purify spawns two processes. One is the process it monitors and the other is a process called rtslave. The process rtslave verifies license and hence has to be monitored by Dyninst. Dyninst did not have support for monitoring multiple processes, so we had to write a tool using Proc file system that linked Dyninst with rtslave. The problem could be solved by setting up the proc state for rtslave in the parent process and force rtslave to inherit the proc state. In this way, rtslave can be made to stop when it tries to execute a connect call. Afterwards, the strategy used in Framemaker can be applied. We would like to emphasize that this section has been written merely to illustrate the use of our tools in handling multiple processes. We have not conducted thorough experiments on Purify and the question of avoiding licensing in Purify remains open.

6Ways to defeat our scheme

There are a number of ways in which a developer can hinder our scheme:

• Since we rely on the access to the Proc file system, if the process locks its own directory in the Proc file system our approach will not be available
• Likewise, should the process use the setuid program to raise its privileges, we may have difficulty altering its process space. However, this is not a big problem, since we can stop the process after it has execed its image and prevents the setuid call.
• The license checking can employ some functions on a probabilistic basis, executing some elements of license checking only occasionally. This will increase the amount of time needed to profile the control flow of the license checking mechanism, since not all functions will be executing every time.
• Another measure that may delay our progress is naming all functions with names not indicative of their function. In Framemaker, we were able to quickly zero in on routines performing licensing since their names clearly indicated their functions.
• The most promising approach appears to be executing critical portions of the client application via an RPC call to the license server. Since our approach relies on removing the communication links between the server and the client, this would preclude the application from executing. However, this has effect of increasing the necessary server-client communication, with the concomitant result of increasing start up time.
• An admittedly extreme approach would be to add hardware encryption on the memory level. If we cannot read the memory state of the process, we would not be able to get any information about how to modify it. Of course, this involves additional cost and inconvenience, and is unlikely to be used for a consumer application.

7Conclusion