Developing Dynamic, Database-Driven
Web Pages by Demonstration
David Wolber, Yingfing Su, Yih Tsung Chiang,
University of San Francisco
2130 Fulton St., San Francisco, CA. 94117
(415) 422-6451
ABSTRACT
This paper describes an interactive development environment for the end-user development of dynamic web pages that connect to databases. The system combines techniques from query by example (QBE), programming by example (PBE), and the spreadsheet paradigm. In combination, these techniques allow a designer to visually specify web pages that display and manipulate persistent data.
INTRODUCTION
Many HTML WYSIWYG editors allow designers to create static web pages without writing code. With the proliferation of dynamic pages, and specifically those that allow access and manipulation of server databases, these editors are not sufficient.
Some recent tools allow the presentation of dynamic web content to be specified in a similar way to how static pages are constructed. The designer creates a template HTML file which specifies the formatting and layout of sampledata. This template can then be sent to a translator, such as Enhydra’s XMLC[], which generates a Java code representation of the HTML file. A programmer can then manipulate the content of the page by replacing the sample data with dynamic data, e.g., data from a database. This modified HTML is then sent to the browser.
Tools like XMLC improve the development process by separating designer and programmer concerns—designers work on presentation using the HTML medium, and programmers work on dynamic content using a programming medium (e.g., Java).
Unfortunately, such tools do not help designers, build complete dynamic web pages. They are still restricted to presentation concerns only, and must rely on a programmer to implement the “dynamic” part of the page.
Macromedia's UltraDev 4 (formerly Allaire's Cold Fusion) is one system that does allow non-programmers to visually specify more than just presentation. UltraDev 4 provides dialogs, similar to those found in a DBMS, which allow a designer to map a visual component to a data source. Without any programming, a designer can create a complete dynamic web page that displays live database data. The tool also provides dialog-based methods of specifying dynamic web pages that allow end-users to manipulate that data.
UltraDev4 dialog-based specification eliminates coding syntax and thus increases the pool of designers capable of developing dynamic web pages. But the system does not leverage important end-user programming techniques that have been applied in other domains and that can dramatically ease development compared to dialogue based specification.
Our research has focused on applying some known end-user programming techniques to the domain of dynamic web page development. We have built a system, DemoWeb, that combines techniques from query by example (QBE), programming by example (PBE), and spreadsheet applications.
Programming in the WYSIWYG Interface
One popular method of end-user programming involves programming in the interface, a method made popular in seminal PBE systems such as SmallStar[], Eager[], and Mondrian[]. These systems are extensions of macro-recording—the user tells the system to watch, then demonstrates the “program” using the same operations that are used in normal operation of the application. The system generalizes the demonstrated macros and creates a program that extends the original application.
Programming in the WYSIWYG interface is slightly different. Here, the goal is to create a brand new program, not extend an existing one. The designer still “programs” by recording operations, but he performs this recording in a facsimile of the “target” interface, i.e., in a WYSIWYG editor. The operations at his disposal are not the same ones that will be available at run-time. Instead, a mapping is defined from the operations available in the WYSIWYG editor to operations in the target application. The designer demonstrates operations using WYSIWYG editing operations in order to specify the corresponding operations in the target application.
The difficulty of this approach is that the mapping between demonstrated operations target operations must be clear to the designer—he must know the semantics of his demonstrations.
But the approach offers some advantages—because demonstrations are not performed directly in the target interface, the demonstrational interface can contain auxiliary elements to help guide the demonstrations. These range from labels on visual components that guide demonstrations, to dialogs that ask for information from the designer.
Of course the more auxiliary elements provided, the less WYSIWYG the editor, and the less concrete the program specification. For instance, UltraDev4 is based on a WYSIWYG editor (Dreamweaver), but much of the programming involves dialogs and other elements
WebDev is a programming in the WYSIWYG tool that maps HTML editing operations and dialogue-choice operations to persistent data operations.
Informal user studies suggest that these techniques improve the process of developing dynamic web pages.
OVERVIEW
DemoWeb isbased on a WYSIWYG HTML editor. Designers can use it to create static pages, or they can add dynamic components to a page, in which case the system generates a servlet to render the page at run-time.
Dynamic Components
Dynamic components are HTML visual components that are mapped to database data.
One of the most common dynamic components is the dynamic table. The designer inserts it into a page just like any static component. The designer can then use a dialog to map some or all of its columns to data in a pre-existing database:
Alternatively, the designer can enter sample data into the table, then specify that DemoWeb should create a new database table with fields matching the sample data columns (see Section X).
The granularity of the mapping is not dynamic table to database table, but dynamic table column to database table column. Thus, a dynamic table component can display data from various database tables. As will be discussed shortly, it can also contain “formula” columns not mapped to a database table.
Formatting By Example
When a dynamic table is mapped to a database table, the system loads a live row of data into the WYSIWYG editor. The designer then specifies the formatting of this data by example. The designer can apply fonts, colors, and other formatting to the data in each column. At run-time, all of the rows of the database table are displayed with the formatting the designer has demonstrated. For instance, if the designer italicizes the name “Wolber”, as in Figure 1:
Table: STUDENTS
Name Test1 Test2
Wolber
/ 87 / 90the rendered page, created by the system generated servlet, displays all names in the first column with italics.
Query By Example
QBE was developed in 1971 [] and some form of it is now included in most DBMSs. Instead of requiring the explicit specification of a query, the system allows the developer to enter sample data, with an optional relational operator, within a table cell. The system uses the contextual information-- the cells where the data is entered-- to formulate a formal SQL query. The designer need not know SQL syntax and is allowed to work with concrete data instead of abstract variable names.
In WebDev, the designer specifies queries by example in the second non-header row of the dynamic table. The designer enters a boolean expression (e.g., ">85") in one or more cells.
Table: STUDENTS
Name Test1 Test2 Average
Wolber
/ 87 / 90>85
The system generates an SQL query (in this case, "Select * From STUDENTS WHERE TEST2>85") so that, at run-time, only selected rows of the mapped table are displayed in the table.
Spreadsheet Functionality
If a column in a dynamic table component is not mapped to a database, the designer maps it to a spreadsheet-like formula. The formula can reference any of the other columns in the row. For example, a designer could specify "=AVERAGE(C2:C3)" for column 4:
Table: STUDENTS
Name Test1 Test2 Average
Wolber
/ 87 / 90 / =Average(C2:C3)>85
At run-time, the fourth column of each row will display the average of the dynamic database data in the previous two columns of that row.
Programming By Example
The QBE and spreadsheet techniques described above allow for the development of web-based database views, but they don't provide a means for developing pages that manipulate a database.
Consider, for instance, a simple form page that allows the end-user to add a record to an existing database:
Figure X. A web page with
UltraDev4 …
DemoWeb provides a PBE technique for creating such a page. The designer first enters sample data in the components of the form. The designer then selects the form's submit button and chooses Record Response from the DemoWeb toolbar. Record Response is similar to the macro recorders in common desktop applications. However, in this case, the purpose of recording operations is not so this new "macro" can be replayed in the development environment. Instead, the designer is specifying the operations that should be executed at run-time when a submit button is selected. Also, the recorded operations will not be executed verbatim-- the system will infer generalized operations.
After selecting record mode, the designer enters new data in the dynamic table component for which the record should be added. For this example, let's assume the designer enters data in the table that matches the data in the form (he might even copy and paste). When the designer chooses "Stop Recording", the system compares the before and after states of the interface to identify the changes. Since the designer demonstrated entering the new data in a dynamic table, the system interprets the operation as a not just a change to the display, but a change to the underlying persistent store (database), i.e. a Table.AddRecord(d1,d2..) operation. And, since the designer entered data in the table that matches the data in the form, the system generates a generalized AddRecordcommand where d1,d2,etc. are the names of form elements.
The PBE technique described eliminates the need for some of the abstraction necessary if programming or even dialog-based specification is used. From the designer's perspective, no name is necessary to refer to the AddRecord operation, and names are not necessary for the parameters of that operation (i.e. the names of the form elements). Instead, the designer demonstrates the AddRecord operation by providing an example of what should occur when the submit button is clicked. The system performs the pattern matching and generates code for AddRecord and its parameters. The designer need only view this code if modification of it is desired.
Generated Code
WebDemo generates an HTML document and a Java servlet from the designers WYSIWYG and visual specifications. The HTML document is funneled through Enhydra's XMLC compiler. XMLC creates Java code from an HTML file. When executed within a servlet, the code renders the same exact page as if the HTML file was sent directly to a browser.
Of course the purpose of XMLC is to create a basis data structure so that a programmer can easily add dynamic content to the page. The Java code created by XMLC is based on a tree data structure fitting the Document Object Model (DOM) standard. A programmer, or in this case, the WebDemo code generator, creates code that manipulates the DOM tree to insert dynamic content.
WebDemo generates a servlet that contains both JDBC code to access any mapped databases, and DOM manipulation code that puts that data into the HTML sent to the browser.
Servlet including JDBC and DOM manipulation code
HTMLJava DOM code
Figure 3. Architecture of Generated Code
For example, if a dynamic table component is mapped to a database table, WebDemo generates code like in the following example:
RELATED WORK
WebDemo combines ideas from numerous areas of computer science: the human-computer interface field (in particular PBE), the database field, those studying applications and extensions to spreadsheets, and those interested in web development.
PBE
Numerous PBE systems have been developed and documented in two collections of research papers[]. One fundamental idea behind this research is that of programming in the interface[], that is, allowing the designer to create programs in the same interface in which the program is run. Instead of formulating abstractions like variables, the designer demonstrates operations on sample data. The system records the examples then infers or deduces a general program. PBE has been applied to numerous applications, including text and graphics editors[] and spreadsheets. Some research efforts have also focused on application-independent PBE.
A number of systems have applied PBE principles to dynamic web pages, mostly focusing on the induction of wrapper programs from examples. A Wrapper program is a web page that displays data from one or more other web pages. It generally collects input from the user, then sends it a set of one or more web pages, e.g. a number of airline pricing pages. The wrapper program then identifies the desired data on the resultant page(s) and displays this collection of dataon the wrapper program page.
Wrapper induction systems eliminate coding by allowing the designer to point out the pieces of data in an existing page that should be collected by the wrapper program. The system then infers a description of the data, often using contextual and landmark information so that, if the existing page changes slightly in the future, the wrapper will still work.
As far as we know, PBE has not previously been applied to the development of new web pages that connect directly to databases. However, examples have been used in databases for years, as the following section explains.
QBE
Back in 1971, Zloof presented QBE [], a query by example system. With QBE, the designer enters data or an expression in a column of a table, and the system uses the "example" to formulate a query. For instance, in Figure 1, the designer has entered "39" in column 2 and ">56" in column 4. The system generates the query:
Select * from Table Where age>39 and x>56
Some form of QBE is now included in most Database Management Systems (DBMS). This idea of example-based database specification has been extended in numerous research efforts, including efforts focused on Query by Templates[] and Triggers by Example[]. It has not, however, been extended to allow database manipulation by example, as is the purpose of WebDev.
Spreadsheet Technology
Spreadsheets are the most popular form of programming by example. A designer can enter a formula for a particular cell, then copy that cell to other cells in the same column (row). The system then generalizes the example (the copied cell) and generates formulas that compute data in the pasted row (column).
Research efforts such as [] have extended the spreadsheet paradigm to non-tabular interface components, and have provided more complex generalization techniques, than that described above, to handle complex non-tabular components.
Commercial Web Development
The efficient development of dynamic web pages is of critical importance to all types of companies, and new tools are emerging daily to help the process.
A major concern has been the separation of designer and programmer concerns. For instance, in the Java world, servlets alone were first used to create dynamic pages. With straight servlets, both the presentation and content of a page are specified in Java. Presentation must thus be specified by a programmer, or at least a designer that is code-savvy enough to enter HTML code within Java code.
Java Server Pages (JSP) then emerged. JSP documents are HTML based but allow Java code to be interlaced with the HTML. JSP eases the designers task somewhat, but a designer is still faced with Java code, and the designer and programmer must still work on the same document.
Template systems such as WebMacro [] provided the next generation tool. A template document is stored separately from the Java code that adds the dynamic content, so designer and programmer can work on their parts separately. WebMacro provides a simple scripting language that allows the designer to mark the places in the template where dynamic content should be put.
WebMacro library code then processes the script so that the Java programmer need only associate code with the names of the placeholders specified by the designer.
Though WebMacro provides a template, it doesn't quite allow the designer to format sample data. The scripting language introduces variables (any identifier preceded with a '$') and for loops, which are difficult for some designers. The designer can supply sample values for the script variables, however, so a sample page can be viewed.
Enhydra's XMLC provides what we felt was the best solution for WebDemo's underlying mechanism, mainly because it requires no scripting language other than normal HTML. With XMLC, the designer creates normal HTML, including regular tables with sample data in the cells. To allow the page to be dynamic, the designer need only provide id tags for important HTML elements. The XMLC compiler then parses the HTML and provides manipulation methods for the identified elements. The programmer can then easily modify elements or add new elements, such as the rows of a table.
With XMLC, the designer need not learn any special scripting other than HTML. It is also conducive to WYSIWYG development since users are comfortable with using property sheets to name items, but not so comfortable writing for loops and switching between a visual and non-visual environment.