Open Source Enterprise Automation

© 2011 Brian M. Kelly WHITE PAPER

Open Source Enterprise Automation

With Net::FullAuto Software:

 extreme ease

 extreme power

 extreme control

 extreme security

 extreme flexibility

 extreme cost savings

Contents:

01Introduction (The Net::FullAuto Vision)

08Something Un-Expect-ed(Really!)

13Follow the Money (Use Cases and Cost Effectiveness)

14Skip the Scheduler (Reduce Job SchedulingDependency)

15Getting Started is EASY (Simple to Setup)

16Back to Basics (Net::FullAuto Use Cases and Examples)

17Need to Know Basis (Role Based Security)

18No Blind Alleys (Robust Error Checking/Handling/Reporting)

19One Button Everything (Full Enterprise Automation)

20The Network is the Computer (At Last!)

21The Future of Net::FullAuto (UNLIMITED!)

INTRODUCTION

In 2000, Brian Kelly, the author of Net::FullAuto, was working as an independent consultant for a large insurance company. As a Software Configuration Manager, it was his job to manage source control and code movement for a large multi-million dollar customer portal project. What continually frustrated him was how much of his job was manual, and how tools that promised to automate tasks, often failed – often spectacularly. Tools that did exist were outrageously expensive, required dedicated personnel, long lead times, huge learning curves, and even then it never took long to find a nuance or use case that even the pricey “software giant” itself failed to handle adequately. The reason more often than not, that otherwise simple manual tasks resist full automation, has to do with the reason most software in this world is written in the first place. Proprietary software is not written to solve problems, it is written to MAKE MONEY. But no one is going to purchase expensive software that does “nothing”, so instead some problem needing a solution is selected, and that becomes the new money magnet around which whole organizations rally. In some cases, more time is spent on insuring the software is not copied, or used beyond its licensing period, than on the actual problem it was originally conceived to solve. In fact, one multi-billion dollar software company charges a license fee for each job run on its software. Now you know just one of the reasons it’s a multi-billion dollar a year organization! Many of course simply call that “success”, and any criticism of that success as sour grapes – and that is hard to argue. But from the perspective of someone just trying to solve technical problems as part of their daily occupational duties, such “success” is often the “problem” to get away from – as fast as possible.

Such was the Net::FullAutoauthor’s experience. Organizations he consulted with always wanted to avoid the expensive solutions whenever possible, and constantly looked to him for creative alternatives. But the simple fact was, no viable open source solutions to robust enterprise automation existed. And so, with a mountain of optimism, and an equal amount of naiveté, the author of Net::FullAuto set out to fill the void. At the time, he figured it would take six months to ready Version 1.0. That was well over ELEVEN years ago. It took more than eleven years and thousands and thousands of hours of development, but Version 1.0 is now a reality.

What took so long? Stubbornness! Theauthor simply refused to compromise his vision. What the author wanted, was a programmable tool that worked just like a user working in a command shell environment – the environment most commonly used by IT personnel. At the time, secure shell was in its infancy, and telnet was the principle means through which activities on remote hosts were performed. What he wanted was a tool that would work against an unknown host, without any knowledge of the host beyond an IP address and login ID and password - much the same way a human user works with telnet (and today – secure shell or SSH). A human user logging into a remote host via SSH for the first time does not need to know what operating system she is accessing, what the command prompt is, or what shell language is set as the default. However, it turned out that software programsaspiring to work like humans needed that information (until Net::FullAuto that is).

So what? How difficult is it for an administrator or programmer to find out what the default shell is and simply hard code it or create a configuration file for the automation code to use?

Answer: it’s not difficult at all – for one host. But what about a THOUSAND hosts? Now suddenly, the problem is not so trivial. This is even truer of something like “command prompts” and “login banners”. In fact, it is now more often the case that command prompts are dynamically generated with such items as the hostname, username and directory location. Here is an example of the default prompt with a fresh installation of Cygwin:

Note the presence of the username: KB06606; the hostname: CHG211134 and the current directory: ~ (tilde or the home directory). Now change to a new directory:

Now consider the Cywin login banner:

Obviously, these mnemonics where created for HUMAN users, not programs. Yet programs that aspire to work just like humans have to be intelligent enough to cope with such dynamic and unexpected data.

Many computer scientists have considered this an impossible task. Consider the opinion of Randall L. Schwartz, expert programmer and author of numerous books and manuals,on the perlmonks.orgcollaboration site:

I don't believe there's any way that a program that sits on a pty[1](like Expect.pm) can ever distinguish where the data is coming from. This is the blessing and curse of Unix I/O.

So, no, I think you're just gonna have to consider the various things to wait for, some of which will be error-like messages coming at unexpected times.

Randall L. Schwartz,

To date, and to the author’s knowledge, no program has successfully gotten past this problem – that is, until Net::FullAuto.

Again: So what? The author has created a solution in search of a problem, the critics will say. WRONG – if that was truly the case, then no one would have to shell out a hundred dollars just to add a new job for nightly execution from software that they’ve already paid thousands of dollars to acquire in the first place! Yet, all over the world, money flows for these fees every single day. If there was a viable open source alternative, this would surely not be the case. (More on schedule based automation in a later chapter.)

Consider that every computer job that was ever automated was in its genesis run and tested manually. What that indicates, assuming there was some means by which those manual steps could be translated into automated steps(with no more effort or expertise than was needed to create the original job, task, process, etc.), is that such automation would be the preferred way to proceed.

In fact, what the author truly envisionedwas a tool where the automation would essentially be “automatically” created at the same time and in the same scripting space as the business logic itself. The result would be code that could be run either manuallyorautomated – depending on the needs of the organization – without any additional expense or development time or resources needed.

How cool would that be?

That tool NOW exists – and its name is Net::FullAuto.

SOMTHING UN-Expect-ED …

Critics would say that an Open Source command shell automation tool already exists, and its name is Expect.

True enough.

The homepage of Expect makes a pretty bold claim:

Expect can make easy all sorts of tasks that are prohibitively difficult with anything else. You will find that Expect is an absolutely invaluable tool - using it, you will be able to automate tasks that you've never even thought of before - and you'll be able to do this automation quickly and easily.

as modified April, 2010

Expect has been around for years, and indeed, is in use all over the globe.Most IT professionals have heard of Expect. But, most IT professionals don’t actually use Expect. An article published in the Linux Gazette gives us some insight into why this might be:

In the design of automated systems in Expect, one of the more difficult hurdles many programmers encounter is ensuring communication with ill-behaved connections and remote terminals.

David Fisher, Advanced Programming in Expect: A Bulletproof Interface, Linux Gazette, Issue 48, 1999

That is a bit of an understatement. The fulcrum of automation in the enterprise is precisely communication with remote terminals (or systems/hosts).It is precisely communication with remote systems, and the desire to automate processes involving them that prompts IT professionals to experiment with Expect in the first place! When such an individual (or team) discovers that Expect is anything but a silver bullet, they more often than not fall back on what they know best – manual processes.

Expect expert David Fisher goes on to provide a little more detail about the specific problems Expect has with remote communications:

If you're reading this article, though, I suspect that you've encountered some of the problems I have when programming in Expect, and you're looking for the solution here. If you're just reading out of interest, the problems arise when automating a session on a machine off in a lab, or on the other side of the world. Strange characters pop up overtheconnection, and the terminal you're connected to does weird things with its echo, but everything is working. It becomes very difficult to determine if what was sent was properly received when you have noise on the connection, terminal control codes inserted in the echo, and even server timeouts between the automation program and the remote session.

David Fisher, Advanced Programming in Expect: A Bulletproof Interface, Linux Gazette, Issue 48, 1999

Fisher goes on to provide brilliant workarounds for many of these issues. Indeed Expect is used successfully by large firms all over the globe for almost countless mission critical tasks. But he also cautions:

The code provided in this article is executable, but needs to be incorporated into any system in which it is to be used.

David Fisher, Advanced Programming in Expect: A Bulletproof Interface, Linux Gazette, Issue 48, 1999

So in other words, even today, twelve years after he wrote that article, if you play with Expect and don’t have Mr. Fisher’s articles and code, you’re simply going to run into the same problems thousands have before. In fact, he even suggests as much in another article he wrote just this past July regarding spawned ids with Expect:

Given the number of times I’ve had to fix this issue in existing code, and the number of times I’ve had to explain this simple solution in interviews and in courses I’ve taught in automation, it seems beneficial to share it with the entire community.

David Fisher, Advanced Programming in Expect: sHellspawn – Demons in the Daemons, Linux Gazette, July, 2011

It’s a bit disturbing to think that thousands of individuals have wasted thousands and thousands of hours (or days) running into the same stumbling blocks over and over again – especially when solutions exist, and have for quite some time. But this phenomenon doesn’t just exist between organizations, but even more frightening to managers everywhere, it happens within them just as much.

The Wikipedia article on Expect, points out a couple additional “limitations” with Expect:

Expect inherits the same syntax convention as Tcl, which may seem unfamiliar if accustomed to other script languages. Compared to languages such as bash, csh, and Perl, Expect has a different twist. It is sometimes challenging to remember when a variable must be prefixed with a "$", and when it must not. …

Another limitation is the difficulty in porting Expect scripts between platforms. For example, an Expect script that was written to use several Unix-based tools, might not be suitable if migrated to a Windows platform. If possible, the programmer must find counterpart command-line applications that provide the same information, and this will probably require changing the send/expects, which can be a major part of the script.(Emphasis added.)

modified on 12 November 2011 at 13:02

It’s tempting to think, that the words “Expect has a different twist” really mean “The Expect syntax is difficult, and non-intuitive. Creating automation solutions with Expect is HARD - a bit contrary to the claims on the Expect home page.”

Indeed – harder than it should be. This is precisely why most IT professionals have heard of Expect, but don’t actually use it. But in fairness to the creators of Expect, automation itself is HARD, and it is not unreasonable to expect (play on words) that a program created to accomplish it, would also demand a lot of skill from its users.

The second limitation, which has added emphasis to highlight it, underscores a very important point: That being - that with Expect, each command, each element of business logic, or code instruction, would have to be accompanied or “wrapped” with its own individual set of “send/expects, which can be a major part of the script”. This works against any real desire to combine business logic with automation code in any truly manageable sense – that is, until Net::FullAuto.

Expect utility’s very name communicates it’s methodology for parsing pseudo-terminal data; it expects, what it is toldto “expect”. A fresh default installation of Expect is not capable of dynamic discovery of the host it is working against. It needs to be told what to “expect”. Here is a typical, simple example of Expect using SSH to connect to a remote host, and running the ‘uname –a’ command:

#!/usr/bin/expect
set timeout 1
set cmd {uname -a}
spawn sshbkelly@localhost
expect_aftereof { exit 0 }
## interact with SSH
expect "yes/no" { send "yes\r" }
expect "password:" { send "password\r" }

expect "# "
send "$cmd\r"
expect "$cmd\r"
expect "(.*)\r"
send "exit\r

Note the existence of no less than FIVE “expect” statements, just to run the ‘uname –a’ command on the localhost (in this case a laptop) via SSH.

Now let’s see the samething with Net::FullAuto:

$localhost->cmd(‘uname –a’);

Believe it ornot, that’s it!! Everything else is handled automatically by Net::FullAuto.

If I want to connect to a different host:

( $remotehost ) = connect_ssh( ‘Remote_Host’ );
$remotehost->cmd( ‘uname –a’ );

Net::FullAuto handles ALL the other details for you, so you can focus on the business task – not the automation one.

Lest ye thinks the author doth protest too much, let it be said that that Expect is an important tool, and that until now, was indeed about the only truly practical way to automate almost countless tasks. In fact, there is an Expect module for Perl that can be used right inside a Net::FullAuto script. For some tasks, it will still be the best means for providing an automation solution. But for most tasks, a default installation of Net::FullAuto will provide more ease, more power, more control, more security, more flexibility and more cost savings than any ambitious Enterprise Automation solution using Expect alone.

Net::FullAuto truly is un-Expect-ed: in more ways than one. Not only does it function differently, and in the process eliminate most of Expect’s limitations, but indeed, most of the IT industry does not see it coming. In fact, it appears that many think automation utilities are about as advanced as they are ever going to get. Consider the opinion of a commentator in response to a wiki.tcl.tk topicpage entitled It seems like Expect is not actively developed:

I think people are just having a hard time accepting that Expect in its current form is complete. Does what it is supposed to. No need for a new version.

Tykling, as posted at Nov 2007

All that can intelligently be said in response - is that it is a good thing “Tykling” is an anonymous poster!Obviously, such an individual would find Net::FullAuto to be not just a surprising innovation, but indeed, perhaps a shocking one.

And finally, a quote used in this White Paper’s introduction is so “spot on”, that it bears repeating here:

I don't believe there's any way that a program that sits on a pty (like Expect.pm) can ever distinguish where the data is coming from. This is the blessing and curse of Unix I/O.

So, no, I think you're just gonna have to consider the various things to wait for, some of which will be error-like messages coming at unexpected times.

Randall L. Schwartz,

Randall? Tykling?

Prepare to be amazed …

FOLLOW THE MONEY

Because Enterprise Automation is HARD (until Net::FullAuto), it follows logically that it would also be VERY expensive; and it is. Enterprise computing typically encompasses a number of “concerns”, but by far the most important is the quality of the end user experience. Whether that user is accessing a public internet site, or an internal intranet site, organizations focus overwhelmingly on optimizing functionality for the end user. In service to that goal, IT departments around the world employ armies of specialists dedicated to activities from database development, to website image design, to server administration. Depending on the size of the organization, the number of hosts used to service all these activities can number from dozens to thousands.