System Management Console
Technical Report
Team Stamina Turbo
Marc Baumbach / Sean McClellandJohn Newfield / Pat O'Hara
Sponsor
Collin Krepps (Northrop Grumman)
Faculty Mentor
Dr. Mark Ardis
Table of Contents
Project Overview......
Basic Requirements......
Constraints
Development Process
Project Schedule: Planned and Actual
System Design
Process and Product Metrics
Product State at Time of Delivery
Project Reflection
Project Overview
The System Management Console (SMC) was designed to simplify the taskof browsing multiple log files spread out over a file system. The SMCis able to track any number of log files and follow them as theyare continuously updated by their parent program. The file contentsare displayed verbosely, with every detail of the program'soutput displayed, or it can be filtered to only show lines which meeta certain criteria. The system is also able to export selected logmessages into various formats, including XML and CSV. The SMC provides the ability to create conditions under which users will be sentalerts via email and/or visually within the program.
In addition to the standard log parsing functionality, the SMC provides extended functionality to assist administrators and developers with getting the most from their log files. Primarily, the SMC provides the ability to retrieve information about system status via a web service. This service allows developers and administrators to know whether a particular program is up and healthy using a small query, without having to run the SMC themselves.
The SMC is also able to retrieve information from an Oracle database, which includes extended error or diagnostic data that is not available in the log files. Using a simple interface, users are able to manually construct SQL statements, send them to the database, view the information returned and be alerted to any exceptions or other errors that the database outputs.
Basic Requirements
The SMC shall:
Display logs from multiple folders: The SMC displays the contents of log files specified by the user. (Priority = High)
Filter log entries by keyword: The SMC allows the user to filter or restrict the display of log messages. (Priority = High)
Export selected log messages to CSV and XML formats: The SMC will allow the user to export specified log messages to multiple formats. (Priority = Medium)
Deliver email and visual alerts to system administrators: The SMC will optionally deliver email or visual alerts to selected users of the system based on predefined conditions. (Priority = Medium)
Provide the functionality for users to query the database: The SMC will allow users to query an Oracle database for simple status, statistics, and other database records. (Priority = Medium)
Web Service: The SMC provides a web service that will deliver data and statistics. (Priority = Medium)
Graphical Display of Data: The SMC will provide the ability to graphically display log file information as specified by the user. (Priority = Low)
Constraints
The SRS defined three performance requirements which were to be constraints on our system.
PE-1: The SMC server shall handle up to 15 consecutive connections. (Priority = High)
PE-2: The SMC client shall be able to view up to 10 log files simultaneously. (Priority = High)
PE-3: The SMC shall take no more than 30 seconds to complete any user request. (Priority = Medium)
For each of these requirements, the actual performance was exceeded. Performance testing showed that the SMC Server is able to handle more than 40 consecutive connections where each connection has requested 10 log files. The actual number of log files available for each client is largely dependent on the amount of memory available to the SMC Client as each log file on average tends to use up about 5 megabytes of memory.
Development Process
An incremental approach to the Software Development Cycle was used for the completion of this project. Specifically, a Spiral approach was used. This lifecycle was chosen due to its risk mitigation focus and the increased chances of client acceptance. The versatility of this approach allowed for features to be added at later stages of the software development than a traditional Waterfall process would have supported. The Spiral process was not mandated by our sponsor, but was approved by our sponsor and ended up being beneficial in mitigating risks that arose due to unforeseen or not well understood requirements.
Project Schedule: Planned and Actual
The schedule was designed up front to be a 10 week planning phase followed by three month long milestone releases. The feature set to be included in each milestone was also decided up front, and was largely followed. There were times when the up front specification of all the requirements to a specific milestone did not fit perfectly into the actual development cycles, and some modifications were needed. Thankfully the Spiral approach allowed use to do so with relative ease, and at the end of each milestone we were able to review the requirements left to be completed, and change the completion dates as necessary. It was also easy to add requirements as necessary, which we had to do a few times due to features we believed to be beneficial or when we had poorly scoped requirements in our original SRS.
Planning: End of February 2008
Requirements and architectural work is completed along with project planning to a point where the project may proceed with minimal risk.
Milestone 1: End of March 2008
Design and implementation of critical requirements is completed along with a preliminary prototype.
Milestone 2: End of April 2008
Design and implementation of near feature complete system is completed.
Final Milestone: May 2008
Final design and implementation is completed and passed off to client for approval.
System Design
The SMC was designed using a Client/Server architecture. The primary decision that needed to be made was if there would be a solitary server or if each client could serve as a server as needed. Allowing the clients to act as servers would have increased portability and possibly help distribute network load more evenly. In the end it was decided that clients would not be allowed to take on the role of servers themselves. Since the log files were hosted on a single machine, a server would have to be running on that machine no matter what in order to distribute the log files. Due to this there would have been little actual gain to allowing the clients to take on the role of server. A clear separation of the client from the server helped reduce the complexity of the design and implementation, as well as make testing easier. This decision was made during the design phase, and only the final approach was implemented.
SMC Server
The SMC Server can be broken up into five major classes, and many more minor classes. The major classes are outlined below:
SMCServer: This is the main class for the SMC Server. The SMC Server is responsible for managing client connections and setting up the parser to monitor log files specified by the clients. For each client that connects, a ConnectionThread instance is created to handle all I/O to that client.
ConnectionThread: The connection thread handles all incoming and outgoing communication to one specific client. The connection thread creates an instance of both the ConnectionInputHandler and the ConnectionOutputHandler, which are described below.
ConnectionInputHandler: The connection input handler is responsible for receiving command objects from the specific client that it is monitoring. These command objects represent requests the client has for the server. In the standard command pattern fashion, the server is able to execute these requests simply by calling the execute method on the command object.
ConnectionOutputHandler: The connection output handler is responsible for sending the results of a request back to the client that sent them.
ParserManager: The parser manager does the work of managing the log files and converting them into SMCLogRecords that will be passed to the requesting clients. The tailing and rolling of the log files is handled by the parser manager, but the actual reading of the individual log files is supplied by a helper class created by Apache called the log file pattern receiver.
The SMC Client is responsible for providing an interface to the user that allows them to view and manipulate log files as desired. User requests are encapsulated in command objects that are then sent to the server. The results of the requests are sent back to the client, and manipulated as desired.
SMCClient: The main class responsible for I/O between the client and the server. The client instantiates a ClientInputHandler and ClientOutputHandler class and also manages an instance of the Log Manager, which keeps track of the SMCLog files.
ClientInputHandler: Takes care of receiving input from the client, passes that input to the appropriate source for delegation.
ClientOutputHandler: Manages the sending of command objects to the server.
LogManager: Keeps track of SMCLogs so that the SMCClient and the UI can access them as necessary.
SMCLog: An object representation of a log file. Keeping the log file as an object allows for easy accomplishing of tasks such as filtering and sorting.
Process and Product Metrics
The following metrics were collected throughout the scope of the project.
Name / Description / FrequencyTime/Effort / The time and effort spent working on the project will be recorded for both individuals and for the team as a whole. / Weekly
Risk Mitigation / The number of open risks shall be recorded and compared to the number of risks mitigated. / Weekly
Defect Tracking / The number of open risks shall be recorded and then compared to the number of closed bugs. / Weekly
Performance / For each code release, the max number of log entries able to be read per minute shall be tracked. / Upon milestone
Time/Effort
The following table and graph show the estimated and actual hours spent by the entire week over a sampling of the weeks throughout the project.
Risk Mitigation
The risk mitigation metric ended up not proving to be useful as a metric. This was due to the fact that we left risks open for most of the two quarters in order to guarantee that we revisit the risk and verify our strategy for mitigation would still be valid should the risk occur. One of the major risks we did mitigate early was the backup of all source-controlled documents to several servers, including a nightly backup to another server on Rochester Institute of Technology’s campus.
Defect Tracking
The following table and graph shows the relationship of defects opened during a milestone and closed during that milestone. As the data suggests, there was zero slippage of defects from one milestone to the next with no known defects left in the system.
Performance
Performance metrics were tracked via three performance requirements. One of the requirements stated that all user actions should complete in less than 30 seconds. Due to a redesign in the system which included paging of large log files, the maximum number of records sent per request was typically on the order of 30,000 log records, which the system consistently was able to complete in less than 15 seconds in the worst case scenario.
Product State at Time of Delivery
All requirements except for the graphical displaying of log files were successfully accomplished. Graphically displaying log files was considered a low priority task, and unfortunately we just did not have time to get to it. Some work was done early in the project to find an Eclipse plugin that could be used for this task, but no such plugin was ever actually found, and creating one by hand would have taken more time than we had available.
We did end up accomplishing a few tasks that were not specifically in the requirements but we believed were very beneficial to the application as a whole.
Originally, requesting a log file to be monitored was done by manually typing in the location of the log file on the server, which was a less than ideal process. It was decided that a far more useful approach would be to allow the server administration to specify directories that contained log files, and then create a wizard on the client side that would list all of the log files on the server in those directories.
Project Reflection
Overall the project was a major success. Our use of the Spiral process assisted the team in addressing the major risks of the project at the beginning of project. This process allowed the team to deliver the product in increments, which in turn allowed our sponsor to provide feedback with each delivery. Another thing that went well for the team was the implementation of providing daily builds to our sponsor. Because our sponsor was not local, and we were not allowed to test on the environment on which our application would be deployed, the daily builds gave our sponsor a chance to test the latest build and see the team’s progress. The use of Eclipse RCP also went well for the team. Eclipse RCP provided functionality that would have taken many more hours to develop, and provided a framework that has been proven and thoroughly tested. The learning curve and the time it took to learn the technology was minimal compared to the functionality and extensibility the framework provided.
Although the project ran relatively smooth, there were still aspects of the system which could have used improvement. One such area is the team’s lack of updating of the documentation on a regular interval. Specifically, the team originally planned to update the risk management document weekly, but the document was typically updated bi-weekly. Another area that could have used improvement was expressing our ideas and decisions made on paper.Typically the team made project-important decisions in the meetings. These decisions, or alternatives were not recorded, and as a result the same topic would sometimes arise in a later meeting.
As stated earlier, the project was a major success; both in terms of the final product and the experience gained while taken part in such a project. The team delivered a reliable application that would be used to help alleviate the difficult task of getting an overall system status. This project provided the team with the experience of working with a technology that was unknown at the start of the project, and working in a team atmosphere to develop a solid application. System Management Console provided the team with the experience that they would receive while in the workforce, and improved their abilities to handle real-life issues pertaining to the development of software, such as memory management.
System Management Console Technical Report / 1