Socket Swapping for Efficient Distributed Communication Between Communicating Processes

SOCKET SWAPPING FOR EFFICIENT DISTRIBUTED COMMUNICATION BETWEEN COMMUNICATING PROCESSES

By

PRAVEEN RAMANAN

B.E., MADRASUNIVERSITY, 2000

A REPORT

Submitted in partial fulfillment of the

Requirements for the degree of

MASTER OF SCIENCE

Department of Computer and InformationScience
College of Engineering

KANSASSTATEUNIVERSITY
Manhattan, Kansas

2002

Approved by:

Major Professor

Dr. Daniel Andresen

ACKNOWLEDGEMETS

I would like to express my thanks and gratitude to Dr Daniel Andresen, my major professor for giving me constant encouragement, advice throughout this whole project as well as during my Masters in Computers Science at KansasStateUniversity. I also thank him for patiently clearing my questions and giving me good leads during this project work.

I would like to thank Dr Gurdip Singh and Dr Mitchell Neilsen for serving in my committee as well as for their valuable cooperation during this project.

I would like to thank my project partner Mr Sendhil Annamalai for his contribution towards this project as well as for his invaluable help throughout this project.

I would like to thank Mr David Bacon for his help in setting up, configuring and installing User Mode Linux (UML). I would also like to thank Mr Matthew W Hoosier as well as Jesse R Greenwald for solving some of my installation problems. I also thank Mr Rengakrishnan Subbramanian for giving me test results for MOSIX bandwidth, latency etc.

I would also like to thank my friends and roommates Mr Jagannathan Nambi, Mr Somasundaram, Mr Nitesh and Mr Girish for their constant moral support during the course of this project. I would also like to thank Mr Venkatesh Prasad, Mr Sadanand Kota and Mr Mahesh Kondwilkar for giving me some important leads in this project.

ABSTRACT

Sockets are the mechanism that allows programs to communicate, either on the same

machine or across a network. A program/process that wishes to communicate with another program/process asks the operating system to create a socket and establish communication. In this project the program/process that wants to communicate establishes a virtual socket network with the other program/process without actually talking directly with each other. This mechanism of virtual networking comes in handy for the MOSIX wherein each process distributes itself among various MOSIX enabled workstations/servers and when those process themselves need to communicate with each other. This process of virtual socket networking is done by writing a user defined system call within the linux kernel operating system. The system call tries to swap one of the existing sockets involved in communication, with a new one and assigning them back to the same process without the knowledge of both the communicating process. This system call was implemented inside the kernel of User-Mode Linux. User Mode Linux is a virtual linux operating system and a safe secure way of running linux versions and linux processes without damaging the main Linux kernel. The socket swapping mechanism was tested through java objects called aglets. Aglets are java programs/objects and have the capability of moving between machines or from one host on the internet to the other while they are communicating with each other. So the system call was called after the aglet moved from one host to the other and then the new communication channel was then established through the mechanism of socket swap.

Table of Contents

1. INTRODUCTION

1.1 Overview of Process Migration and MOSIX

1.2 MOSIX RELATION WITH SOCKET SWAP

2. User Mode Linux (UML)

2.1 Why UML?

2.2 Advantages of UML

2.3 UML Kernel

2.4 Running UML inside Linux

2.5 Logging in inside UML

2.6 Host File Access in UML

3. Introduction to System Calls

4. Socket System Call and Its Implementation

4. Socket System Call and Its Implementation

4.1 What are Sockets?

4.2 Socket Related System Calls

5. Internal Representation of Socket Data Structures

5.1 Socket Data Structure

5.2 Data Structures inside the kernel

5.3 Creating a New Socket

6. Writing the System Call

6.1 Implementing a new System Call inside UML

7. Introduction to Aglets

7.1 Aglets relation with the Implemented System Call

7.2 Aglets and Tahiti Server

7.3 Aglet Operations and Aglet Life Cycle

8. Overview of Java Native Interface (JNI)

9. Integration of UML with Aglets, JNI and User Implemented C Library

9.1 Running Aglets inside UML

9.2 User Defined C Library

9.3 Integrating Aglets with C Library through JNI

9.4 Results and Metrics

10. CONCLUSION

11. ENHANCEMENTS AND RELATED RESEARCH

12. REFERENCES

1.INTRODUCTION

1.1 Overview of Process Migration and MOSIX[1]

Process Migration is the act of transferring a process from one machine to another during its execution. So process migration involves capturing the state of the running process, transferring the state to the new host, and recreating the process on this host from the point at which it is halted. With the growth of distributed computing in the recent years, process migration has become much more useful in this industry. Process Migration has several advantages such a load balancing, fault tolerance etc.

Process Migration Criteria

• Process migration requires that the migrating process resume execution from the place where it was stopped before it migrated. The process should not start from the beginning after migration.
• There should be some interface which records the state of the process before it migrates.
• The migrated process should be accessible by the same mechanism after the migration as it was before.

Process Migration Difficulties

• Difficult if the participating hosts involved in socket migration have different architecture or different OS.
• Only limited applications us process migration. Scientific or long running applications generally use process migration. Since these make up only a small percentage of today’s applications there is a lack of applications supporting process migration.
• The issue of Socket Migration is not addressed during process migration. Hence we see that the candidates for process migration are limited to those processes that do not have network connections.

Hence we see the necessity/importance of socket migration during process migration.An example of a process migration tool is called MOSIX.

MOSIX (Multi Computer Operating System for UNIX) is a software that can transform a Linux cluster of x86 based workstations and servers to run almost like an SMP. When one or more processes are created under MOSIX, MOSIX has the ability to distribute (and redistribute) your processes (transparently) among the nodes so that it obtains the best possible performance from the processes. The important part of the MOSIX is its ability to come up with adaptive management algorithms which continually monitor the activities of the processes versus the available resources. This algorithm has the ability to respond to uneven resource distribution and to take advantage of the best possible resources.

The algorithms of MOSIX provide process migration such as

• Automatic work distribution for parallel processing or to migrate processes from slower to faster nodes.
• Load balancing for even work distribution.
• Memory ushering migrate process from a node that run out of main memory, to avoid swapping or thrashing.
• High I/O performance and parallel I/O.

1.2 MOSIX RELATION WITH SOCKET SWAP[2]

Since MOSIX distributes and redistributes processes among various nodes, the necessity for kernel socket swapping arises when the distributed processes want to communicate with each other. When a process, which has been distributed to some other node, wants to communicate, it communicates only through the base node. When a process migrates in MOSIX the migrating process is divided into contexts. They are user context, that can be migrated and the other one is system context called UHN (Unique Home Node). UHN can also be called as Base node which is the node where the process was originally created. The user context called the remote contains the program code, stack, data, memory maps and registers of the process. The remote encapsulated the process when it is running in the user level. The system context also called as the deputy contains description of the resources that the process is attached to, and kernel-stack for the execution of system code on behalf of the process. So the deputy encapsulates the process when it is running in the kernel. It holds the site dependent part of the system context of the process;

Hence it must remain in the UHN of the process.

Each process has a Unique Home Node where it was created. Normally this is the node to which the user has logged in. While the process can migrate many times between different nodes, the deputy is never migrated. Since creating a socket involves a system call, the remote site’s link layer first intercepts them. If the system call is site independent it is executed by the remote locally. Otherwise, the system call is forwarded to the deputy, which executes the system call on behalf of the process in the UHN. The deputy returns the result back to the remote site, which then continues to execute the user’s code.

Suppose two processes in machine A and B are communicating with each other. Then a processin machine Awhich is migrated to machine C wants to continue communication with process in machine B,the communication in MOSIX will actually take place via A from C to B. This might make the overall speed of socket communication in MOSIX a little slow as it takes an indirect path. So in order to avoid this unnecessary traversal, this project does a socket swapping inside the kernel. This internal socket swapping creates a new socket in machine B and actually swaps the socket that was created in machineB before process migration. This ensures that the nodes which were distributed by MOSIX can communicate directly with each other instead of going back to the base node, thereby increasing the speed of communication. Thus kernel swapping increases the performance of this MOSIX cluster by increasing the speed of communication between various processes distributed among various nodes.

This mechanism of internal socket swapping is implemented by writing a user defined system call within the Linux kernel. This was tested under the Virtual Linux operating system called UserModeLinux. UserModeLinux is virtual linux kernel which performs like a normal linux operating system and runs inside linux. So any changes in the kernel can be made in this user mode linux and can be tested under this mode.

2. User Mode Linux (UML)[3]

User Mode Linux is a safe secure way of running Linux versions and Linux processes. User Mode Linux is a virtual linux which runs inside our home linux machine. It is possible for us to run any software or process the same way that we will be running under normal linux. It has the additional advantage where in we can run buggy software, experiment with new linux kernels or distributions, and poke around in the internals of Linux, all without risking the main setup of our original linux. It acts like a virtual machine and probably might have more hardware and software virtual resources than the main machine. Disk storage for the virtual machine is entirely contained inside a single file on the physical machine where we are running user mode linux.

2.1 Why UML?

Since we have to write our own system call which has to be implemented inside the kernel, the best way to debug it, test it and run it would be through this user mode linux. During testing it is possible for the user to write some code that would someway affect the internals of the kernel. This buggy code might damage the kernel of the machine if we are the using the main linux machine to write our system call. In order to avoid that mess, nowadays there is a general trend for linux users (beginners especially) to work with UML rather than the main machine. The following gives other uses of UML many of which arepracticed in the computing industry.

2.2 Advantages of UML

Virtual Hosting

UML can run basically anything that the host can, so its possible to split a physical system into a bunch of independent virtual machines. For example the UML that we are running for this project uses the file system of the CIS machine in the lab.

Kernel Development and Debugging

This kernel allows developers to write their own code and debug them using any normal process-level tools, like gdb, gprof, and gcov. Also, the users who are away from their normal environment (from the main machine) can carry on their development, using a UML virtual machine as their kernel testing box.

Process Debugging

UML can also be useful for debugging user level processes. For example if there is an odd error that was caused due to a system call, then the user can insert printf/debugging statements inside the system call and try to debug the problem. The user can fire up UML, set a breakpoint on the system call in question, then run his program and can find out why the call is failing very quickly.

Education

UML is a great teaching tool, especially in courses where students need a dedicated machine in order to get the most out of it. Students can understand completely about OS internals and networking using UML. Organization of file system, order of execution of linux kernel and other related information about the kernel can be learnt pretty quickly.

2.3 UML Kernel

UML is a fully-functional Linux Kernel. It runs its own scheduler and VM system, relying on the host kernel only for the equivalent of hardware support. The virtual machine is completely configurable through the command line, allowing its memory, available devices, and their configuration to be specified.

Hardware Support

UML hardware support comes in the form of virtual devices which uses resources on the host. It supports all the usual types of devices that a physical system does.

• Block Devices- UML block devices are normally associated with a file on the host which contains a filesystem. This file is made to look like a block device inside the virtual machine, and the filesystem on it is mounted just as a filesystem on a physical disk is mounted in a native kernel. A block device file may also contain a swap signature in which case UML will be willing to swap onto it.
• Consoles and serial lines-These are almost identical sharing most of their code and differing only where they need to present different interfaces to the rest of the kernel. They can be attached to different interfaces on the host, including file descriptors, ptys, ttys, pts devices, and xterms. By default the main console is attached to file descriptors 0 and 1, the other consoles come up in xterms and the serial lines attach themselves to ptys.
• Network Devices-Similarly the network driver can use a variety of host interfaces to exchange pockets with other virtual machines, the host and the rest of the network. To construct a completely virtual network, with no access to the host networking, UML can use a hub daemon to forward packets from one virtual machine to another or it can set up a multicast network and have the machines send packets directly to each other.

2.4 Running UML inside Linux

1)The first step in running UML is to download the latest version of Linux Kernel.

2)Download the latest UML patch and apply the patch to the linux kernel that was downloaded in step 1.

3)Follow the steps that are usually done for compiling and running linux kernel such as xconfig, make dep etc. The only change is to append the sentence “arch=um” which informs that the linux kernel is running under the architecture of UML.

4)The final step is to run uml. Just by saying linux will try to mount the file root_fs in the current directory. If root_fs is not used the command to run uml is

Host%. ./linux ubd0=”Name of FileSystem” devfs = nomount

Ubd0 assigns a block device to the new filesytem. This filesystem can be downloaded from the official user mode linux website

Additional parameters such as memory to be used, UML-UML communication etc can mentioned in the command line if necessary.

5)On successful running of the above command the kernel will boot up and present with a login prompt. The file system that we used our project opened up six virtual console windows through which we can work with.

2.5 Logging in inside UML

The downloadedfile system usually has a account with username root and password user. Once the user logs in the UML, he will find himself inside a virtual machine. The downloaded file system has several key commands and utilities installed. There a couple of other ways to log in too. One is by the way of virtual console wherein each virtual console comes up in its own xterms. The file system we downloaded was Debian-3.0r0.ext2. On running and booting UML this filesystem opens up 6 different virtual consoles to play around with. The second way to log in is by serial line where the user attaches his favorite terminal program to the corresponding tty. The final method is to telnet to the virtual machine and log onto it. The network should be running in order to use this telnet method. After using the UML, typing the command halt in any one of the virtual consoles will bring the kernel down and the process will exit.