A Query Broker is a very comfortable, though expensive, form of decoupling logical and physical views in a database access layer.
Consider the running example of an invoice . The OrderInvoiceView models the logical data structure while the OrderPhyiscalView and the CustomerPhysicalView model the corresponding physical tables.
You have decided to use the Relational Database Access Layer. You use Hierarchical Views as interface to the application kernel and Physical Views to encapsulate database access.
How to connect the Hierarchical Views, that make the Logical Access Layer and the Physical Views, that make the Physical Access Layer for reading and writing?
Cost versus flexibility: The cheapest way to connect two layers is hard-wired coupling via function calls: A ConcreteView knows which PhysicalViews it has to call. You can generate the corresponding calls using compact table descriptions. This works fine as long as both layers are stable. However, if one layer is unstable you should use some form of decoupling. In the access layer we have stable Hierarchical Views on top of an unstable Physical View layer. If the system is small enough, you may use a program generator to couple both layers. Still, this approach will produce extensive costs in terms of compilation and software distribution if the system lives for several years. Consider you have to distribute megabytes of database access software to thousands of clients for every change in the physical database model.
Reusability: Though the Physical Views may change rapidly, they reflect the physical structure of the database. Therefore it as likely that several applications use the same Physical Views but different Hierarchical Views. Writing a separate coupling mechanism for every single application nullifies the gains you get from reusing the Physical Views.
Complexity: Since the hard-coded solution is not flexible enough, you need a more complex solution. However, extra complexity makes the system more expensive and harder to maintain again.
Use a Broker [Bus+96] to connect the layers. The Hierarchical Views form the client side of the Query Broker, the Physical Views constitute the server side. Describe services using directed acyclic graphs (DAG) and use a tree matching algorithm to find best matches. Let the Query Broker assemble the Physical Views and deliver the result in a Query Result container.
A Broker is a standard technique for decoupling. Use the standard structure and adapt it to the Database Access Layer framework:
the most significant difference between Query Broker and a standard broker is that it usually takes more than one server to handle a request. The mapping to servers is not one to one but one to many.
Figure 8 : The Query Broker is a Broker adapted to the Database Access Layer Framework.
usually Brokers use symbolic names to identify services. As we have a 1:n relation between service requests and servers that fulfill them, this is not appropriate here. Therefore the Query Broker uses semantic descriptions (DAGs) to describe requested views. Consider Figure 9. The left side shows the description of the request for the OrderInvoiceView. The right side depicts the corresponding services. To assemble the ConcretePhysicalViews the Query Broker matches the keys, tagged with white ellipses.
Figure 9 : Tree matching to resolve requests to the Query Broker. The left side shows the OrderInvoiceView of our order processing system, the right side depicts the corresponding two Physical Views. If the Query Broker gets the left presentation as request, it figures out that the Physical Views on the right side are the best way to satisfy the request.
In
the following scenario an application kernel object creates an OrderInvoiceView
causing a database read().
The QueryBroker
handles the read()
and matches a view
description against available services via the matchServices()
method. The QueryBroker
forwards the request to two different ConcretePhysicalViews:
the OrderPhysicalView
and the CustomerPhysicalView.
These two read()
the data from the database and deliver the results by packing the data into
result containers. The Broker has to merge both result containers to deliver one
result to the OrderInvoiceView.
The OrderInvoiceView
unpacks the data into its instance variables.
Figure 10 : Retrieving data via a Query Broker.
Flexibility: The Query Broker completely decouples the Hierarchical and the Physical Views. At runtime, new Physical Views may register and the associations to Hierarchical Views may change.
Complexity: The tree structured result containers, the request descriptions, and a tree matching algorithm make the Query Broker complex to design. However, the Broker is well encapsulated, restricting complexity to a single subsystem.
Reusability: Since the Query Broker is independent of the views it connects, you may implement it as part of a framework. This is even better than reusing only the Physical Views or generators.
Cost: The complexity of the Query Broker makes it expensive to implement. A run time dictionary increases the cost. Implemented in a reusable framework and used in more than one application, the Query Broker will pay off. Hard-wired coupling is cheaper to build but makes optimizations more expensive and causes nightmares when you think about software distribution among several thousand client sites.
Server Registration: All ConcretePhysicalViews have to register at the QueryBroker prior to the first database access. You have to take care for it during system initialization. You may use a runtime dictionary, some other form of registry or language specific initialization techniques.
Responsibility for Casting Data Types: There are two choices, where to cast raw database types to application data types and vice versa. For example, you have to convert a CHAR(20) into an OrderKeyType and vice versa. You may assign the task of casting to the ConcreteViews as well as to the PhysicalViews. A runtime dictionary may support both alternatives, if it contains the logical data model and the knowledge which attributes to cast into which application data types.
Tree matching: Matching the DAGs is similar to code generation in compilers, where you have to find good assembler code for a program. So you may use the corresponding algorithms [Aho+86, chapter 9.2]. The Broker may even find several query plans for a request, differing in speed. The matching algorithm has to deal with ambiguous derivations and has to find the fastest solution. You have similar problems in optimizing compilers, which deal with ambiguous grammars for code generation purposes [Kel91].
Query descriptions and result containers: You have to find a good representation for the DAGs, used to describe queries. On one hand the Views should be able to specify their requests and services easily. On the other hand the presentation should conform to the requirements of the matching algorithm. An easy-to-parse textual presentation is a good choice.
A further refined variant of a Query Broker allows a very flexible development process. A Query Broker that is also enabled to generate on-the-fly dynamic SQL queries can be used to substitute the Physical Access Layer during early development stages. This works fine for clean database models but becomes hard with Overflow Tables.
Later, you add more and more Physical Views using static SQL. If even this proves to be too slow, the Physical View implementations can be made even faster using the tuple interface of the database. The Query Broker shields the application kernel from such tuning. It responds to request with the fastest services he can find in his registry.
[Bus+96] contains a comprehensive discussion of Brokers in general.
Brown and Whitenack describe a Broker [Bro+96] on a per class basis. Note, that Query Broker is more general.
The Query Broker is a compilation of various best practices:
The VAA data manager [VAA95] defines views in terms of the logical data model. It uses a generated hard wired coupling of layers. Our experiences at HYPO-Bank [Kel+96a, Col+96] taught us to use dynamic descriptions wherever possible. Two projects at sd&m used other important parts of the approach.
sd&m’s LSM project used dynamic SQL, migrated to static SQL and ended up with a tuple interface. The idea was born from bad experiences with a slow database server. The project makes extensive use of a runtime data dictionary and bridges dynamic queries and pre-compiled queries completely.
The Fall/OK project for the German police uses tree matching. The software copes with queries by example on a large data model. The data model changes very rapidly.
CORBA Persistent Object Service [Ses96] also use a Broker. Application kernel objects write their instance data to streams and a Broker (Persistent Object Manager) forwards the stream to some Persistent Object Service (database or other). Persistent Object Services may be arbitrary databases not known to the object. This is also the simple a case of a one to one mapping between service requests and servers that fulfill them. Note that this design is now (2000) replaced by another one.