XRX: XQueries in eXist

By Jeni Tennison
July 3, 2008 | Comments: 10

I want to explore the XRX (XForms, REST, XQuery) architecture, so I’ve set myself the challenge of implementing a social bookmarking web service modelled on del.icio.us as described in Chapter 7 of Leonard Richardson’s and Sam Ruby’s RESTful Web Services, using Orbeon Forms and eXist.

This is the first post in a series that will look at how the implementation works, and any problems I run into along the way. In this post, I’ll be looking at setting up eXist with some basic data and queries that will support the rest of the application. eXist is an open-source native XML database; you might want something a bit more heavy-weight if you’re looking to do this with enterprise-level quantities of content, but eXist has the basic capabilities that you’d expect from an XML database and is more than sufficient for exploring the XRX principles.

eXist comes bundled with Orbeon; you can download the latest standalone version but the URLs that I’ll be referencing here are all based on the bundled version. Orbeon works on top of a number of web application servers; I’m going to be using Tomcat here. I won’t repeat the Tomcat and Orbeon installation instructions here. Suffice to say that you need to install them and start Tomcat, and that when you’ve done so, http://localhost:8080/orbeon/ should take you to an Orbeon welcome page.

To business. The XML that I’m going to be using is pretty rudimentary, and I’ll probably amend it as I go along. For now, here’s the compact RELAX NG schema:

default namespace = "http://www.example.org/bookmarks"
datatypes xs = "http://www.w3.org/2001/XMLSchema-datatypes"
element user { 
    element name { xs:Name },
    element bookmarks {
        element bookmark {
            attribute href { xs:anyURI },
            attribute hash { xs:hexBinary },
            attribute date { xs:dateTime },
            element title { text },
            element notes { text }?,
            element tag { text }*
        }+
    },
    element bundles {
        element bundle {
            element name { text },
            element tag { text }+
        }
    }
}

I’ve created a bunch of documents in this markup language so that I can test things out. The first thing to do is to load them into an appropriate collection within eXist. A collection is any set of documents that you might want to search. I imagine that the bigger the collection the longer it will take to search, such that large applications might want to design their collection spaces in intelligent ways, but for my simple purposes, I’m just going to dump all the XML documents in one collection.

To make a collection and put documents in it, you really need to open the eXist Java UI, which is most easily done using the Webstart Client. When the client starts up, the crucial URL (since I’m using the version bundled with Orbeon) is:

xmldb:exist://localhost:8080/orbeon/exist/xmlrpc

Enter the URL and hit OK, and you should see the eXist Admin Client shown in this screenshot.

eXist Web Admin.jpeg

You can do all sorts of things from this client, including managing users, changing permissions on documents, and managing indexes and triggers. I’ll probably come back to these later. Right now, all I want to do is create a new collection and load some files into that collection. I navigate into the orbeon collection, select Create collection from the File menu, enter the name bookmarks and hit OK. Then I navigate into bookmarks and choose Store files\directories from the File menu, navigate to the files I want to import, select them, and wait while they get loaded.

Now the documents are loaded, I can access them through the eXist REST interface. If I navigate to:

http://localhost:8080/orbeon/exist/rest/db/orbeon/bookmarks/drench.xml

then I get back that XML file. Super.

Now I’ve structured the XML so that if you want a user’s bookmarks and tag bundles, then that kind of URL is all you need. But of course if you want a different view on the data, you need something more. There are a dozen other views that I need to support to cover the social bookmarking application described in RESTful web services. I’m going to split them into five groups:

bookmark query
  • an individual bookmark (as created by a single user)
  • all a user’s bookmarks
  • a list of users who have bookmarked a particular URI
recent query
  • a user’s recent bookmarks
  • a user’s recent bookmarks filtered by tag
  • all recent bookmarks
  • all recent bookmarks filtered by tag
calendar query
  • a report on the number of bookmarks generated on each date for a user
  • a report on the number of bookmarks generated on each date, filtered by tag, for a user
tag query
  • all a user’s tags
bundle query
  • all a user’s bundles
  • a particular bundle (as defined by a single user)

To support these, I’m going to create some queries. Here’s the neat thing: if you store an XQuery in eXist and then GET it through the REST interface, the query will execute and the results will be returned. You can pass query parameters within the URI; these are accessible within the XQuery using the request:get-parameter() function. For example, the following query (bookmark.xq) can be used to return a particular bookmark for a user, all a user’s bookmarks, or a list of users who have bookmarked a particular URI, depending on the parameters that are passed into it:

01: declare namespace exist = "http://exist.sourceforge.net/NS/exist";
02: declare namespace request = "http://exist-db.org/xquery/request";
03: declare namespace bk = "http://www.example.org/bookmarks";
04: 
05: declare option exist:serialize "method=xml media-type=application/xml";
06: 
07: declare variable $bookmarks as document-node()* := collection('/db/orbeon/bookmarks');
08: 
09: declare variable $user as xs:string := request:get-parameter('user', '');
10: declare variable $hash as xs:string := request:get-parameter('hash', '');
11: 
12: if ($user eq '') 
13: then
14:   <users xmlns="http://www.example.org/bookmarks">{
15:     for $u in $bookmarks/bk:user,
16:         $b in $u/bk:bookmarks/bk:bookmark[@hash = $hash]
17:     order by xs:dateTime($b/@date) descending
18:     return <user>{$u/bk:name, $b}</user>
19:   }</users>
20: else 
21:   let $user-bookmarks as element(bk:bookmarks)? := 
22:     $bookmarks/bk:user[bk:name = $user]/bk:bookmarks
23:   return if ($hash = '') 
24:          then $user-bookmarks
25:          else $user-bookmarks/bk:bookmark[@hash = $hash]

The first three lines are namespace declarations, the first for eXist’s namespace so that I can use the exist:serialize option; the second for eXist’s request namespace, so that I can use the request:get-parameter() function; and the third for the bookmarks namespace that the XML uses. Line 5 declares the exist:serialize option which determines how the result of this query gets serialized — in this case as XML with a media type of application/xml. Line 7 declares a global variable called $bookmarks which holds all the bookmark XML documents in the collection. Lines 9 and 10 declare variables for the two parameters to this query: the username and the bookmark hash.

Lines 12 to 25 do all the work. If no user is specified, then it lists users who have bookmarked the bookmark specified by the hash, ordering them based on when they bookmarked it and returning the name of the user and their actual bookmark (including tags and whatnot). If a user is specified, the query finds that user’s bookmarks; if a hash is specified too, it returns that bookmark, otherwise it returns all of them.

For example, when I use the URL:

http://localhost:8080/orbeon/exist/rest/db/orbeon/bookmarks/queries/bookmark.xq \
  ?hash=d3346bda2344033cf77460a4ef7363d8&user=drench

I get back a 200 OK response with the bookmark XML:

<bookmark xmlns="http://www.example.org/bookmarks"
         href="http://www.w3.org/TR/xslt" 
         hash="d3346bda2344033cf77460a4ef7363d8" 
         date="2008-06-03T22:37:06Z">
    <title>XSL Transformations (XSLT)</title>
    <tag>w3c</tag>
    <tag>xsl</tag>
    <tag>xslt</tag>
    <tag>xml</tag>
    <tag>specs</tag>
    <tag>docs</tag>
    <tag>reference</tag>
    <tag>work</tag>
</bookmark>

with the headers:

Server: Apache-Coyote/1.1
X-XQuery-Cached: true
Last-Modified: Mon, 16 Jun 2008 20:49:39 GMT
Content-Type: application/xml;charset=UTF-8
Content-Length: 327
Date: Thu, 03 Jul 2008 19:13:03 GMT

Similarly, the URL:

http://localhost:8080/orbeon/exist/rest/db/orbeon/bookmarks/queries/bookmark.xq \
  ?hash=d3346bda2344033cf77460a4ef7363d8

returns a 200 OK response with:

<?xml version="1.0"?>
<users xmlns="http://www.example.org/bookmarks">
    <user>
        <name>jthake</name>
        <bookmark href="http://www.w3.org/TR/xslt" 
                  hash="d3346bda2344033cf77460a4ef7363d8" 
                  date="2008-06-12T07:58:20Z">
            <title>XSL Transformations (XSLT)</title>
            <tag>xslt</tag>
        </bookmark>
    </user>
    <user>
        <name>pispis4</name>
        <bookmark href="http://www.w3.org/TR/xslt" 
                  hash="d3346bda2344033cf77460a4ef7363d8" 
                  date="2008-06-11T11:08:50Z">
            <title>XSL Transformations (XSLT)</title>
            <tag>dpms</tag>
        </bookmark>
    </user>
    [and so on]
</users>

So that’s the first group of requests. The other groups have their own queries, which follow a very similar pattern, as you’d imagine.

And that’s the foundation of the GET queries. There’s more to do, of course: better URIs, and supporting the request methods other than GET, but this post is quite long enough already, so I’ll leave those for a different post.


You might also be interested in:


10 Comments

Jeni,

Thanks for the tutorial. I wanted to give it a try but do not see how to install the Orbeon that "comes bundled" with eXist (as opposed to the standalone Orbeon). Can you explain? I am using recent eXist war (1.2.3-rev:7866-20080610) under Tomcat.

Thanks,
Morgan

I said "eXist comes bundled with Orbeon", not the other way around! :) You can get Orbeon (and eXist bundled with it) from http://www.orbeon.com/forms/download.

But there's nothing here that's Orbeon-specific except the URLs, so there's no need to get Orbeon unless you want the extra URL-mapping, pipelining, or XForms support that it gives you. (About which more soon, hopefully.)

Jeni

Jeni,

Great post. I didn't know that eXist could run a query stored in the database with a simple GET on the query itself. Now I am wondering: how does eXist know that this is a query to execute, and not some a file that I want to retrieve?

Alex

Jeni,

Thanks for this post. It is nice to see more people blogging about XRX . I am looking forward to future posts!

- Dan

Jeni, Please ignore my previous comment. I see now that you meant "install Orbeon and you will be provided with a copy of eXist". That done, the tutorial worked as expected. I look forward to next installments.

Thanks, Morgan

Thanks Alex.

Reading the documentation about stored XQueries, I think that eXist uses the mime type of the XQuery document to determine whether to execute it or return it from a GET or POST request. So you could save the XQuery with a something other than application/xquery if you wanted to actually GET it instead.

That's not altogether satisfactory (since you might want to do both things with the same query), but I guess you'd usually store XQueries that you want to make generally available somewhere else in your website: they're not XML, after all, so why put them in your XML database?

Jeni

I am so impressed with this XRX business and very pleased to find that Jeni is a few steps ahead and willing to sketch out the process.

Thanks Jeni, and thanks Dan too, especially for your writing on XForms.

Hello,

I would like to encapsulate the Xquery request in http request.
The first objective is to make a test of load with many interactive users.

can you explain to me how to do that ?

Thank you in advance for your answers,

Breizho

Hi,

Been working with eXist and XQuery for a couple months and just started looking at Orbeon and XForms a week or so ago. So your blog series looks absolutely great for me thanks for doing it.

The only question I have is, it appears you're writing it in tutorial style, and I do have both eXist and Orbeon installed, but I don't see the sample xml files or XQuery files here to download?

An I overlooking them? Or was that not your intent?

This is pretty old info on eXist, XQuery and XForms but its still pretty good too -Daniel Wozniak

Popular Topics

Archives

Or, visit our complete archives.

Recommended for You

Got a Question?