- Applications.

<?xml version="1.0"?>
<ApplicationSet name="AppSetName" verbosity="error" logpath="/dev/pts/2">
   <Database name="orders" connect="connection string"/>
   <Database name="audit" connect="another connection string"/>
   <Exit path="/AppSetName/thankyou.htm" />
   <Error path="/AppSetName/errpage.htm"/>
   <Variable name="orgname" value="ACME Industries Inc"/>
   <Variable name="copyright" value="Copyright (C) 2000 - 2009"/>
   <Application name="app1" state="hxml" xmlvar="__DCat_xml">
      <Exit path="/AppSetName/AppName/thankyou.html"
      <Error path="/AppsetName/Appname/errpage.html"/>
      <PIOPath path="/AppsetName/Appname/bin"/>
      <Database name="oldorders" connect="yet another connection string"/>
      <Variable name="MaxQueryRows" type="string" value="50"/>
      <Variable name="Version" type="string" value="1.9.1"/>
      <Page ... />
      <Page ... />
   </Application>
   ...
</ApplicationSet>

- Pages.

<?xml version="1.0"?>
<ApplicationSet name="AppSetName" verbosity="error" logpath="/dev/pts/2">>
   <Database name="orders" connect="connection string"/>
   <Database name="audit" connect="another connection string"/>
   <Exit path="/AppSetName/thankyou.htm"
   <Error path="/AppSetName/errpage.htm"/>
   <Variable name="orgname" value="ACME Industries Inc"/>
   <Variable name="copyright" value="Copyright (C) 1999 - 2002"/>
   <Application name="app1" state="hxml" xmlvar="__Dcat_xml">
      <Exit path="/AppSetName/AppName/thankyou.htm"
                  error="/AppsetName/Appname/errpage.htm"/>
      <Database name="orders" connect="yet another connection string"/>
      <Variable name="MaxQueryRows" type="string" value="50"/>
      <Variable name="Version" type="string value="1.9.1"/>
      <Page name="page1" template="page1" phases="pci" options="si" next="+" >
         <Variable name="foo" value="bar" />
         <Variable name="quality" value="horrid" />
      </Page>
      <Page name="page2" template="page2.dsp" pio="page2.o" phases="pr" options=""
                            next="next? page1: page3" />
      <Page name="page3" phases="r" />
   </Application>
   ...
</ApplicationSet>

- Flow of control in a DCat application.

http://www.DCatserver.com/dcat/appname/pagename

http://www.DCatserver.com/dcat/appname/pagename?somevar=somevalue& . . .

http://www.DCatserver.com/dcat/appname

http://www.DCatserver.com/dcat/appname/pagename/_nextpage_
http://www.DCatserver.com/dcat/appname/_start_

1. Preamble (p) - general processing before the next page can be generated,
2. If the page specifies a template - a .dcat file, check the modification date of the template against any existing translated version, and if necessary, repeat the translation process, or translate the .dcat file for the first time.
3. If the phases attibute specifies insertion point processing (i), then the corresponding method in the PIO is called. Otherwise a standard proceedure is used for substitution.
4. Set cookies (c) - set up the values for any required cookies,
5. Extra headers (h) - set up the strings for any HTTP headers other than the standard ones,
6. Load the compiled .fragments file and select its generate method, and call that to output the page content, or if there is no template, generate the content using the appropriate method in the PIO.
7. Send the headers and content back to Apache via the connector.
8. Call the postamble of any PIO if specified (a).

• Get/post request data symbol table,
• Cookies symbol table,
• State data,
• Application set object variables.

struct DCatContext
{
   const(string[string]) requestdata;
   const(string[string]) cookies;
   const(TinyXML) state;
   const(ApplicationSet) appset;
   bool delegate(string objectName, Object object) holdObject;
   Object delegate(string objectName) retrieveObject;
   void delegate(string objectName) releaseObject;

}

interface PIO
{
   string returnProc(string prevpage, string nextpage, DCatContext* context);
   void preamble(DCatContext* context);
   string interpretIP(string ipname, DCatContext* context);
   string doInsertions(FragmentList() fl, DCatContext* context);
   string[] generateHeaders(DCatContext* context);
   string generateContent(DCatContext* context);
   string[] setCookies(DCatContext* context);
   void postamble(DCatContext* context);
}

- Application State Processing Sequence.

<form name=appstate action="/appset/app1/page1" method=POST>;
<input type=hidden name=__Dcat_xml value='<%=__Dcat_xml%>'>
<input type=submit value=" Submit ">;
</form>;

- ApplicationSet Tag.

<ApplicationSet logpath="/path/to/log/file"
                [name="AppSetName"] 
                [verbosity="error | warning | info | debug"]
                [version="versionstring"]
                [xmlvar="subst_here"]>
    <Application . . . />+
    <Database . . . />*
    <Error . . . />?
    <Exit . . . />?
    <MySQL . . . />?
    <Namespaces . . . />?
    <PIOPath . . ./>?
    <Template . . ./>?
    <Variable . . . />*
    <XML . . ./>?
</AppSetName>

- Application Tag.

<Application name="AppName" [state="hxml | cxml | form"] [xmlvar="subst_here"]>
    <Database . . . />*
    <Error . . . />?
    <Exit . . . />?
    <Page . . . />+
    <PIOPath . . . />?
    <Template . . . />?
    <Variable . . . />*
</Application>

- MySQL Tag.

<MySQL [none = "none"]
         host="hostname"
         name="database name"
         user="user name"
         epw="encoded password"
         [preload="Y | N"] />

- Namespaces Tag.

<Namespaces    [appset="appsetNSQ"]
               [app="appNSQ"]
               [cookies="cookiesNSQ"]
               [page="pageNSQ"]
               [state="stateNSQ"] />

<h2><%=AS::company%> <%=AP::product%> (<%=PG::version%>)></h2>

- Page Tag.

<Page name="PageName" [phases=phaseSet]
                      [options=optionSet]
                      [next=nextSpec]
                      [xheads=extraHeaders]
                      [mimetype="mimetype"] />
    <PIOPath . . ./>?
    <Template . . ./>?
    <Variable . . . />*
</Page>

Return processing	- r
Preamble processing	- p
Cookie processing	- c
Custom insertions	- C
Extra headers		- h
Insertion points	- i
Postamble processing	- a

Generate all after headers	- g
Generate all				- G
Initialise state at this page	- i
no caching of template			- n
No cacheing of PIO				- N
Use this page as the start page	- s

"."	- Repeat the same page,
"+"	- Go to the next page as defined in the ASD,
"-"	- Go to the previous page as defined in the ASD,
"^"	- Go back to the first page of the application,
"pageX"	- Go to the named page,
"*varX"	- Go to the page named by variable varX,
"cond? page2, page3"	- Ternary conditional, if cond true ("1", "y", "Y"), page2, else page3,
"option# page2, page3, page4"	- Jump to a page depending on the integer zero based value of variable "option".

Content length			- c
Kill the connection		- k
Don't cache this response	- n

Cache-control: no-cache
Pragma: no-cache
Expires: Sat, 01 Jan 2000 00:00:00 GMT

Connection: close

- PIOPath tag.

<PIOPath path="/path/to/PIO-files" />

- Template Tag.

<Template path="/path/to/templates" />

- Variable Tag.

<Variable name="varName" value="varValue" />

- XML Tag.

<XML path="/path/to/xml" />

- Template page syntax.

<%=PG:varname%>

<table>
<%
for (var i = 0; i < limit; i++) {
    response.write("...");
}
%>
</table>

<table>
<%ipname%>
</table>

- Run time representation of templates.

struct Fragment
{
   byte type;	// 0 for plain text, 1 for <%=xxx%>, and 2 for <%yyy%>
   uint pos;	// offset from the start of the in-memory file
   uint len;	// length of the fragment - in types 1 and 2 this is the
				// length between % and %.
}

class MemoryTemplate
{
   this(string pageName)
   {
      string filename = pageName ~ ".dcat";
      if (exists(pageName ~ ".frags"))
      {
         d_time cr;
         d_time ac;
         d_time md;
         getTimes(pageName ~ ".frags", cr, ac, md);
         d_time fragtime = md;
         getTimes(filename, cr, ac, md);
         if (md > fragtime)
         {
            scanTemplate(filename);
            writeFrags(pageName);
         }
         else
            loadFiles(pageName);
      }
      else
      {
         scanTemplate(filename);
         writeFrags(pageName);
      }
   }

   void scanTemplate(string filename) { ... }
   void loadFiles(string pageName) { ... }

   typedef string delegate(string) sds;

   char[] instantiate(sds vp, sds ip)

   void writeFrags(string pageName) { ... }
   uint readFrags(string pageName) { ... }
}

- Compiled template file format. When a template is processed it is written to a .frag file so that the scanning process does not have to be repeated unless the timestamp of the template changes. The file format consists of simple records: type fragment offset 4 bytes fragment length 4 bytes |-------|-------|-------|-------|-------|-------|-------|-------|-------|

PIO Interface.

- The interface. To reiterate, the interface that a PIO must implement is as follows: interface PIO { string returnProc(string prevpage, string nextpage, DCatContext* context); void preamble(DCatContext* context); string interpretIP(string ipname, DCatContext* context); string doInsertions(FragmentList() fl, DCatContext* context); string[] generateHeaders(DCatContext* context); string generateContent(DCatContext* context); string[] setCookies(DCatContext* context); void postamble(DCatContext* context); } The context data is a pointer to a struct containing references to the various entities that can provide context data: struct DCatContext { const(string[string]) requestdata; const(string[string]) cookies; const(TinyXML) state; const(ApplicationSet) appset; bool delegate(string objectName, Object object) holdObject; Object delegate(string objectName) retrieveObject; void delegate(string objectName) releaseObject;

- The returnProc method. string returnProc(string prevpage, string nextpage, DCatContext* context); The returnProc method is given three arguments. The first two provide the framework's view of the name of the page that submitted the request, and the page that should follow as specified by the ASD. This latter may be empty. The third argument is the context. This method returns a string, which will generally be zero length. If an actual string is returned, the framework will take it as a 'goto' - an arbitrary page name. This method probably has either nothing to do, or most of the important work of the application. It may have to write the results of several previous pages from a form or from the state XML to a database. Otherwise, it's probably only going to apply some rather more complex criterion for what page should follow. If it encounters any problem during this process, it should throw an exception of type PIOException, which will be caught by the framework.

- The preamble method. void preamble(DCatContext* context); In many cases, an empty implementation will suffice here. An important exception will be the case where the XML state representation is built progressively as the application works through pages. Such additions to the XML should probably be made at this point. If it encounters any problem during this process, it should throw an exception of type PIOException, which will be caught by the framework.

- The interpretIP method. string interpretIP(string ipname, DCatContext* context); As noted previously, a meaningful implementation of this method must be provided if there are insertion points of the form <% ipname > in the corresponding page template. It receives the name of the insertion point, and must return a string to be substituted at that point. If it encounters any problem during this process, it should throw an exception of type PIOException, which will be caught by the framework.

- The doInsertions method. string doInsertions(FragmentList() fl, DCatContext* context); The automatic behavior of the framework for variable substitution, and the interpretIP method between tham can fully instantiate a template. The doInsertions method is included in case for some reason you wish to take full control of this process, as indicated by the 'i' flag in the page phases information. If it encounters any problem during this process, it should throw an exception of type PIOException, which will be caught by the framework.

string[] generateHeaders(DCatContext* context);

- The generateHeaders method. string[] generateHeaders(DCatContext* context); This method should generate any exotic headers required by the application's next page. Note that some extra headers can be generated automatically as per the xheads options in the ASD for the page. It may well be an empty implementation. If it encounters any problem during this process, it should throw an exception of type PIOException, which will be caught by the framework.

- The generateContent method. string generateContent(DCatContext* context); This method must have an implementation in those cases where you specify 'g' or 'G' in the page options. If you specify 'G' you must generate all the headers, including the content-length if that is required. If it encounters any problem during this process, it should throw an exception of type PIOException, which will be caught by the framework.

- The generateCookies method. string[] setCookies(DCatContext* context); This is similar to the generateHeaders method except that you only have to generate the text for the actual cookies, each prefixed by a string such as "+20y!", "+3m!", "+10w!", "+30d!", "+12h!", or "-!" to govern its expiration. The "-!" form will put it in the past. If it encounters any problem during this process, it should throw an exception of type PIOException, which will be caught by the framework.

- The postamble method. void postamble(DCatContext* context); This method should do any cleaning up that was engendered by prvious PIO methods. See "Communication between PIO stages"

- PIO exceptions. The exception class PIO Exception differs from a plain Exception in that it contains two separate messages. One of these - the regular toString() message, contains the error message that will be written to the application log file. The other, accessible through the displayMessage() property is the one that will be used in the generation of the error page. class PIOException : Exception { string _polite; this(string vulgar, string polite) { super(vulgar); _polite = polite; } string displayMessage() { return _polite; } }

- Communication between PIO stages. Three delegates are pased to PIO methods via the context information. The first bool delegate(string objectName, Object object) holdObject; can be used to persist information that may be required by later stages of processing. The object supplied to it will be parked somewhere by the framework. The second Object delegate(string objectName) retrieveObject; can retrieve such a parked object. The third void delegate(string objectName) releaseObject; tells the framework that you have no further need of the object and that it may thus dispose of its reference to it and leave it to the mercy of the garbage collector. The framework will in any case, at the end of page processing, release such references.

Cacheing.

- What is cached. The ASD for a Dcat installation is cached when the application is started, and remains in memory throughout its lifetime. PIOs and processed templates are cached by default, but you can explicitly state than they should not be cached in the Page definition. You might want to do this for pages that are infrequently used. Temporary objects used for communication between PIO phases are released as soon as the page processing is completed. [TODO: Implement timing out of caching on a MRU basis controlled by an attribute on the application.]

Logging.

- Log4D. Logging is done via module bevutils.log4d (.../dcat/bevutils/log4d.d). This also is work-in-progress, but has served well so far, being the primary debugging tool used during the construction of DCat. There are two methods of concern in modifying or extending the DCat code void log(int level, string msg); void logF(A...)(int level, string fmt, A args); Levels are: ERROR: 0, WARNING: 1, INFO: 2, VERBOSE: 3, DEBUG: 4 By default DCat maintains up to 10 current and rolled over log files of maximum size 1000000. Before the request has been parsed to any extent, logging goes to .../dcat/dcat....log. After the request URI is discovered, logging switches to .../dcat/appname/appname....log. [TODO: Add file name and line number information to Log4D in DEBUG mode. Find a way to make log() and logF() both log().]

Creating a PIO.

- Boilerplate implementation. A generic starting point for implementation of a PIO is as follows: module dcat.pagename; import std.string; import std.stdio; import std.stream; import bevutils.log4d; import dcat.memtemp; import dcat.asdreader; import dcat.loader; /* struct DCatContext { string[string] postdata; string[string] cookies; TinyXML state; ApplicationSet appset; Application app; Page page; bool delegate(string objectName, Object object) holdObject; Object delegate(string objectName) retrieveObject; void delegate(string objectName) releaseObject; } */ class PIOImpl : dcat.loader.PIO { Log4D _log; this(Log4D log) { _log = log; _log.log(4, "PIOImpl ctor done"); } string returnProc(string prevpage, string nextpage, dcat.loader.DCatContext* context) { return ""; } void preamble(DCatContext* context) {} string interpretIP(string ipname, DCatContext* context) { return ""; } string doInsertions(MemoryTemplate mt, DCatContext* context) { return ""; } string[] generateHeaders(DCatContext* context) { return null; } string generateContent(DCatContext* context) { return ""; } string[] setCookies(DCatContext* context) { return null; } void postamble(DCatContext* context) {} } PIO makePIO(Log4D log) { PIOImpl pio = new PIOImpl(log); return pio; }

- Caveats on dynamic loading of PIOs. At present, the dynamic loading sub-system is minimal, because minimal is likely to be quick. As such, it is quite possible that when you create a PIO and it is loaded, the load-time linker will spew out a list of unsatisfied external names. This could be dealt with programatically by having the linker load and link the module from Phobos that contained the unsatisfied external, and at some point it may be. However, it goes against my grain to have an application do some complex operation every time it is started that could be sorted out when it was built. For the present, it is the programmers responsibility to provide fixups for such unsatisfied externals that will ensure that the build-time linker pulls them in when it creates the DCat executable. This is done by editing the module dcat.fixups (/dcat/dcat/fixups.d). The primary trick for pulling in externals is as follows: extern (C) { extern int D9invariant12_d_invariantFC6ObjectZv; int* p = &D9invariant12_d_invariantFC6ObjectZv; extern int D3std4conv7__arrayZ; int* q = &D3std4conv7__arrayZ; } Note that the log file will report the unsatisfied external as e.g. "_D3std4conv7__arrayZ". The objects referenced are not neccessarily ints, that's just a C programmers choice of default type - all we need is an address. If your PIO implementation uses templates, you should also define these in fixups.d, otherwise you will get a slew of unsatisfied externals.

Building and Installing DCat.

- Building. Currently, your choices of environment are Windows, or Windows. I have a box ready to install Linux and try to port DCat to D1.x, but I haven't got around to doing that yet. If you unzip the provided file onto your Windows D drive, then you will not have anything else to do. If you unzip it elsewhere, you will have to edit the file ...\dcat\basic.asd to reflect where you have installed it (Appset/base). To build, open a command prompt, change directory to ...\dcat, and enter: make -fdcat.mak You don't need to do this initially, since the executable is included in the zip. To test it, you'll need to enter: dcat -i which should install the service, and then: net start apache2 net start dcat This of course asumes that you have Apache2 installed on your machine. For Windows, this is not difficult - see apache.org. Look at the next section of this document for configuration details.

Configuring Apache.

- Setting up httpd.conf. Some small modifications are required to the as-installed Apache httpd.conf file. Make sure that the loaded modules section includes LoadModule jk_module modules/mod_jk.so Then add some sections at the end of httpd.conf so you have JkWorkersFile d:/dcat/workers.properties JkLogFile d:/dcat/mod_jk.log JkLogLevel debug <VirtualHost localhost> ServerName localhost JkMount /dcat/* ajp13 </VirtualHost> Modify the JkWorkersfile and JkLogFile entries according to your installation preferences.

Testing.

- Run the provided test application. In your browser address bar (currently only tested with Firefox), enter: http://localhost/dcat/app1 When it doesn't work, you can email me.