Evaluation of Native XML DatabaseChapter 7. Testing Procedures

Chapter_6_Search_ArchitectureIndexofOriginalDocsChapter_8_Findings

Chapter 7. Testing Procedures

Testing Philosophy

General Approach

Testing Architecture

XQuery Adaptations

Timing Issues

Test 1: Storage Footprint

Test 2: Do We Get Out What We Put In?

Test 3: Storage Speed

Test 4: Queries for Accuracy

Test 5: Timed Benchmark Queries

Test 6: Working with Nearly Full Core Indices

Test 7: Extensibility Test

Test 9: Non-indexed Substring Searches

Testing Philosophy

(1). We are interested only in that testing which would shed light on the performance of the patented features described in Chapters 3 through 6 above. Specifically, we are not really interested in finding bugs or idiosyncracies which are likely to be fixed in future versions. And in particular we are not interested in testing the broader XMS which surrounds the NeoCore database. So, for example, we are not evaluating the performance or ease of use of the NeoCore XMS console; we are not interested in login/logout procedures; we are not interested in whether the NeoCore C++ API works efficiently; we are not interested in transaction management; we are not interested in locking/unlocking idiosyncracies; we are not interested in archiving or disaster recovery.

We are interested in the claimed small footprint, i.e., the amount of disk storage, that the patented features are supposed to allow. We are interested in whether XML documents retain their syntactical structure when they go through the flattening process and are later reconstructed. We are interested in the speed at which documents can be stored. We are interested in the claim that indexed searches of non-duplicate data items are near-equally rapid whether in an empty database or in a nearly full database. Given the amount of flattening, icon generation, map-file population, and indexing that has to take place on a store, we want to see how quickly stores can be done. We are especially interested in the key NeoCore claim: that a whole series of documents can be stored and searched efficiently, even when there is no pre-determined DTD or external schema for the series of documents and even when the internal structures contained in the series of documents have nothing to do with each other. That is, we are especially interested in testing NeoCore’s ability to deal with the extensibility of XML—the principle feature where NeoCore claims superiority over its competitors.

(2). We are not going to evaluate the architecture’s ability to support large amounts of data—at least not for this current test. NeoCore states that the current version of the NeoCore XMS is not yet configured for huge databases. NeoCore claims that release 3.0 will support larger databases. For this reason, for each series of testing we perform, we will do so with documents whose size totals only about 100 megabytes. Note that this is our limitation, not NeoCore’s. We did initially verify that the XMS can be configured to store far more. But there ends up being nothing to gain by testing the larger sizes, given that NeoCore acknowledges that the currently version is not designed for huge databases and that this limitation will be dealt with in a future release.

(3). Because of NeoCore’s unique architecture, the performance of the database will be affected significantly by the type of data that is input. Text-heavy documents with few tags should show processing characteristics totally different than we would get from documents with large tag structures but little data. In general then, we will do our testing several times—each time with documents of a different sort. Here are the kinds of documents we will use:

  • Text only. These are documents with a single tag, “<text_document>”, and large amounts of text data.
  • Benchmark, mixed configurations. These are documents prepared with the XOO7 Benchmark Generation Tool, using several different configuration files, so that we get a variety of document types being stored. There is a fair amount of duplication, as might be considered typical in business-to-business scenarios, where the same parts are purchased on several occasions from the same vendors or where there are thousands of customers living in the same city.
  • Benchmark, single configuration. Instead of using several different configuration files to generate benchmark data, we use one configuration. The amount of duplication in the tag structures will be quite high, although the data will show the same random variation as we get in the mixed configuration.
  • Benchmark, one file. Here we store the same benchmark document many times until we have the desired 100 megs. Not only will the tag structures be heavily duplicated, the data will be heavily duplicated also.
  • All Dupe. Here we will manually edit a benchmark file so that all tag names and all data items will have the same value. This provides an extreme challenge for the NeoCore XMS, particularly for its duplicate indices; this is for our testing purposes only, as it is not likely to mirror any typical data.
  • No Dupe. These documents are at the opposite extreme—they will cause the duplicate indices to be largely bypassed, but will challenge the core indices. These documents will have no duplicates for any tag and any data item across the entire 100 meg set, except for the metadata tags that are automatically added by the XMS.
  • Extensibility. Here each document has a randomly structured tag set as well as random data items. The tag structure does not comply with any DTD or DOCTYPE declaration. The database management system has to figure out on its own how it is going to handle the random structures. This kind of document set is designed to evaluate the database’s ability to handle new or unexpected structures that might appear in incoming XML documents.

Top of Chapter

General Approach

The testing tools described in this chapter were developed and themselves tested on a fairly typical Windows XP Home Edition platform. Once proven in that development environment, they were installed in the Computer Science laboratory at the University of Colorado, Colorado Springs, where all testing could be carried out in a more controlled environment on a more robust platform. Final testing in the UCCS laboratory was on a Windows XP Professional Edition platform with a Pentium 4, 3 GHz CPU and 1.50 GB of RAM.

Both client and server processes were run on the same single platform. This would not be considered a typical configuration. It was done in this way partly for our own convenience—we were taking only one and not two platforms away from other uses. In addition, and more significantly, by working on one platform we could minimize communication delays—as far as possible we wanted the times to reflect database performance rather than other functionality. There were sufficient CPU speeds and RAM so that we believed the client and server processes would not seriously impede one another. (Unfortunately, this belief was wrong—see our note on RAM usage in Chapter 8.)

Top of Chapter

Testing Architecture

There are several components to the testing architecture:

  • The NeoCore database being tested—we are using version 2.6.
  • The NeoCore XMS Console—a client-side GUI, running on the client’s browser, that is packaged with the NeoCore database. It is used for database administration and for convenient manual input/output.
  • Store Tool—written for this project in C++; functions as our primary interface with the database for Store commands; used for timed testing, load testing, or when there are several documents to be stored. Uses portions of #include files provided by NeoCore. Calculates execution time and gathers and records numerous statistics about the stored document and the state of the database at the time of storing. Source code, without the NeoCore #include files, is at Appendix B.
  • Query Tool—written for this project in C++; functions as our primary interface with the database for Query commands; used for timed testing, load testing, or when there are multiple queries to make. Uses portions of #include files provided by NeoCore. Gathers and stores statistics similar to those recorded by the Store Tool.
  • XET (XML Extensibility Test) Document Generation Tool. Generates XML documents with unique tag structures and unique tag names to supplement the benchmark documents. (See below, “Extensibility Testing”.) Source code is at Appendix C.
  • XOO7 benchmark generation tool—UNIX C++ source files were downloaded from the XOO7 website and then modified to work in the Windows environment; it produces valid XML documents to store in the database.
  • No-Dupe Document Generation Tool. Produces XML documents of designated size and depth, where every tag and every data item has a unique value across all documents produced. Used for tests in which the core indices and the dictionaries are used heavily and the duplicate indices are completely bypassed.
  • Text-heavy Document Generation Tool—generates a file of random lower-case characters, space-delimited into “words” of one to 10 characters, fifty characters to a line. The size of the file is provided by the user in a command-line argument. The text is surrounded by a simple “<text_document>”…”</text_document>” tag pair.
  • XOO7 queries—plain text XQuery commands were downloaded from the XOO7 website. All the queries had to be modified/adapted to work with the NeoCore database structures; see “XQuery Adaptations,” below. Once they were modified as needed and proven to work properly, as in Test 4, below, each modified query was stored in a plain text file, for use in timed testing.
  • A variety of short KornShell utilities/tools. We used the U/WIN KornShell for Windows package to run much of the testing.
  • MS Visual Studio—for creating and editing C++ source code and generating various executables.
  • Standard text editors—vi, notepad, or wordpad, depending on what we needed to edit.
  • Standard spread sheet—MS Excel; used to organize and graph statistics generated by our testing.

Here is how the testing structures are used:

  • The NeoCore XMS is installed using its default ports (7700 and 7701). Port 7701 is used by the NeoCore XMS Console, primarily for administration. Port 7700 is used by the Store Tool and the Query Tool. The two dictionaries were created initially to be five times the default sizes; the indices and the Map Store were initially 25 times the default sizes. This initial configuration was changed many times as needed by specific tests. We did not turn on the “Full Data Indexing Option”; none of the XOO7 benchmark queries required this feature, and keeping it off eliminated the need for the Data Duplicate Index. Because the XOO7 benchmark XML documents are quite large when using the default configuration files (over 4.5 megabytes for the smallest), the sizes of the database buffers were increased; buffer level 9 was increased to 15 8-meg buffers and level ten was increased to 15 16-meg buffers. Also, we increased timeout parameters to an hour just to avoid dealing with them—they served no purpose for our testing.
  • Communication with port 7700 was through the two C++ tools—the Store Tool and the Query Tool. Besides communicating with the database on Port 7700, these tools gathered or calculated a variety of statistics to be used to analyze database performance. (Statistics were output to space-delimited flat files for later evaluation using standard spreadsheet and graphing tools.) The source file for the Store Tool is included in Appendix B; the Query Tool is quite similar and is not included simply to avoid redundancy. (Appendix B does not contain the NeoCore-provided #include files; these are considered proprietary and, under the terms of our license, may not be divulged.)
  • Input to the C++ Store and Query Tools is handled in the following way. The tools look for specific filenames in a “work” directory. The Store Tool looks for files of the format “neoNNNN.xml”, where “NNNN” is an integer, typically in the range 1001 to 9999. The Query Tool looks for files of the format “neoNNNN.n.qml”, where “n” is an integer in the range 1 to 23, reflecting the number of the XOO7 query being processed. If there are no such filenames in the Work directory, the tools simply sleep for 5 seconds and then look again. On finding a correctly-named file, the tools start a timer, send the appropriate command to the database, receive the reply from the database, stop the timer, gather information and write it to a statistics file, and then delete the input file from the work directory. An assumption of the tools is that any Store file contains a single syntactically sound XML document and that each Query file contains a single syntactically sound XQuery command which uses only that XQuery subset which is recognized by the NeoCore XQuery parser. So, if we were building a series of XML documents to store in the database, we would first run the Store Tool more-or-less continuously in a separate DOS window. We would then build a workfile in, or copy a workfile to, the Work Directory and then rename it[1] “neoNNNN.xml”, at which time it would be processed by the Store Tool. Several KornShell scripting tools were developed to make such chores convenient and to make sure that each “NNNN” in the filenames is unique.
  • C++ sourcecode for the XOO7 Benchmark generation tool was downloaded from the XOO7 website. Created in a UNIX environment, it needed several minor modifications before it would compile, link, and run in MS Visual Studio. Running the program generates well-formed XML benchmark documents. Through the use of configuration files, output documents can be generated in several sizes and with some limited variations in the XML tag structure. For many tests, we used the “small3” configuration file provided on the XOO7 website; that seemed to be the configuration file used most frequently when the XOO7 developers did their initial benchmark testing. Note that the “small” in the “small3” configuration file is misleading—it generates an XML document of over 4.5 megs.
  • Query commands downloaded from the XOO7 website needed some significant adaptations before they could be used in the NeoCore environment. These adaptations are discussed in the next section. We initially tried out our adaptations manually, using the XMS Console. Once the queries were proven to work properly, each was saved as a simple text file; once copied and renamed as described above, the text files served as input for the Query Tool.

Top of Chapter

XQuery Adaptations

None of the XOO7 queries worked in the NeoCore environment without modifications, in part because the XOO7 queries use an early version of XQuery and in part because NeoCore supports only a subset of the complete XQuery language. Appendix D shows the original and the modified versions. Of course, modifying the queries raises significant questions about whether we can legitimately compare our results with the results of the original benchmark testing. The following discussion explains the problems that were encountered, the solution we used, and a rationale for using that solution. It should be noted that XMS version 2.6, the version we are using in this test, was the first NeoCore release to support XQuery; previous releases supported XPath only. Not all XQuery functionality is supported in version 2.6; we assume that eventually, in future releases, the entire XQuery syntax will be supported.[2]

Lack of Support for Upper-Case FLWR Keywords.

Problem: XOO7 used as keywords “FOR”, “LET”, “IN”, “WHERE”, and “RETURN”. Current XQuery syntax requires these keywords be lower case.

Workaround: Change uppercase keywords to lower case.

Discussion: This workaround should make no difference whatsoever in performance.

Extra “ND” tag for NeoCore XML Documents

Problem: NeoCore inserts an <ND> tag at the beginning of every XML document. As a result, the document() function selects specified documents at the “/ND” level. This means that where the XOO7 benchmark calls for

for $d in document(“small31.xml”)/ComplexAssembly

NeoCore instead must use

for $d in document(“small31.xml”)/Module1/ComplexAssembly

Workaround: Insert the required “/Module1” node designator in all queries.

Discussion: Given the NeoCore architecture, this should make little difference, since both “ND>Module1>” and “Module1>” sequences are explicitly indexed. Whatever performance penalty might theoretically arise from having to parse and process the extra eight characters is in fact justified—it is one of the costs associated with using the NeoCore architecture.

NeoCore XMS, v. 2.6, does not support the obsolete XQuery “shallow()”function.

Problem: The XQuery “shallow()” function returns the target node but none of its children. Using NeoCore’s subset of current XQuery, there does not seem to be an alternate way of returning the identical result.

Workaround: Where “shallow($a)” might return