OmniMark Tips

OmniMark design principles

• The streaming paradigm
• Rules-based programming
• Hierarchical markup parsing model
• Powerful pattern matching
• Referents

The streaming paradigm

The streaming paradigm is an approach to programming that concentrates on describing the process to be applied to a piece of data, and on processing data directly as it streams from one location to another. In the streaming model, the use of data structures to model input data is eliminated, and the use of data structures to model output is greatly reduced. This keeps to a minimum the use of system resources when processing large volumes of data. As a side-effect, because the processing requirements are consistent, system performance on larger data sets can be predicted with a great deal of accuracy. A program will run with equal success on a 2 kilobyte file or a 2 gigabyte file.

OmniMark has an abstracted streaming model which allows a stream to be attached to different sources of input and output — files, databases and messages — with a minimum of effort. This abstraction also allows code processing the content itself to be dissociated completely from the problems of managing or even knowing about details of the input or output type, with obvious productivity and code simplification benefits.

An application may have multiple input streams open to permit data integration. Multiple output streams may also be used to feed different targets. For instance a complex application may be taking a stream fed from a file, integrating that input with a stream fed by accessing a database and outputting the data to multiple systems (potentially in different formats).

Rules-based programming

OmniMark incorporates a declarative scripting language. This means that an application is constructed of rules for dealing with events which are triggered by the recognition of patterns of data coming into the program from a stream. In dealing with content the individual pieces of content are well known, the order in which they occur is not. This arbitrariness of content is one of its basic properties and rules are the best mechanism for dealing with it. OmniMark’s rules may be triggered by data events generated by the two types of built-in processors, the pattern processor and the markup processor. The markup processor is tightly coupled with markup parsing.

The two processors may be used in conjunction to process a single piece of content to create powerful hybrid applications; where XML is being processed and complex pattern matching is used upon the content in the markup – the text in the elements. The pattern processor may also be used ahead of the markup processor to prepare content for parsing – converting non-XML into XML for instance. The output stream of the pattern processor is fed in as the input stream of the markup processor.

All of these features are implemented in an elegant framework, which results in applications consisting of well-delineated functional code blocks both in terms of readability and actual functionality, thus producing an easily maintained application.

Hierarchical markup parsing model

Many people concerned with XML will be familiar with, or will at least have heard of, SAX and DOM models for processing. SAX is an event-based model and DOM is tree-based. OmniMark employs a third model – hierarchical. Like SAX, OmniMark leverages an event-based model, but where SAX would generate three events for an element (the start, the content and the end) OmniMark generates only one, treating the occurrence of the whole element as a single event to activate a rule. Since elements can be nested, a hierarchy of activated rules will be created, modeling the structure of the content. This simple model is easy to understand and the resulting process flow is clear, concise and thus easy to maintain.

The event-based parsing model fits neatly with the streaming approach to processing content, with the markup processor receiving a stream of data and triggering events as the elements are found, without needing to buffer or decompose the input. Therefore this model supports the design considerations for OmniMark of remaining scalable and performant when processing massive data sets or receiving high volumes of content.

In conjunction with the triggering of the rules, the markup processor maintains the current element context for the set of elements being processed at any instant. This allows the application to query and make decisions based on data about that context including the attribute values associated with the elements being processed and their parents. This mechanism has been shown to handle the vast majority of content processing requirements. However, by using other features of OmniMark this may be augmented should it be necessary – for instance a tree of all elements accessed may be constructed in the application for later manipulation.

Powerful pattern matching

The OmniMark pattern processor implements a pattern matching language that is both powerful and easy to use. Based upon an optimized regular expression mechanism, it has many other features, including:

• Maintaining context. The same pattern may have different meanings in different contexts. Therefore context needs to be maintained to allow different rules to fire in different situations.
• The ability to lookahead for patterns without actually processing the values. This allows program flow to be changed, before the pattern is reached, to allow the pattern to be processed in the right context.
• Complex pattern matching procedures (i.e., independently-called functions). This allows sophisticated pattern matching to be encapsulated and reused.
• Nested pattern matching (matching a pattern within a pattern).

The pattern processor will activate the associated rule when a pattern defined has been matched. These features are encapsulated in a language that is very English-like, making it clear, easy to comprehend application functionality, which simplifies both development and maintenance.

Referents

Often the order in which content is received as input is not the order in which it is required for output. OmniMark’s patented referent mechanism allows a placeholder to be inserted in the output stream and its value supplied later when it is available. The streaming mechanism handles the buffering of output containing unresolved placeholders. The whole referent mechanism may be scoped and nested so that buffering is kept to a minimum. Code that is processing content needs no knowledge of the mechanism; a referent is just like any other target. The major benefit of this mechanism is that it maintains the efficiency of the streaming model while enabling powerful re-ordering functionality that would otherwise be severely constrained. Referents are a key innovation within the OmniMark language and it is one reason why OmniMark is so successful at blending power and performance.

by Helen Owens-Pope

Pattern matching in OmniMark

Writing better patterns addresses two goals: making the patterns do more for you, and getting them to run fast. The former is always more important than the latter — there’s no point having a program running fast if it isn’t doing what you want it to — but they are not incompatible, and very often more effective patterns will run faster than less effective ones, because they are more to the point.

There are three main principles discussed in this paper:

• Fail Fast — write patterns that don’t waste time,
• Succeed Slow — write patterns that do their job efficiently, and
• Divide and Conquer — build patterns to cover all the cases.

Fail Fast, Succeed Slow

All OmniMark programmers, at one time or another, ask themselves if their “find” rules could or should be running faster. Most of the time it doesn’t matter. If you’re not waiting around for your OmniMark program to run, there isn’t a problem. But if your thumbs are getting tired of twiddling, it might be time to take a look at your find rules.

Perhaps the most important thing that should be kept in mind about using find rules is that they spend most of their time failing to produce results. At most, only one find rule will end up capturing a piece of text, so all the rules ahead of it are going to fail. This leads to the first principle of writing find rules: “fail fast” — or, spend as little time as possible on find rules that are failing.

Most find rules already fail quite fast; the first character in the find rule isn’t the one being looked at, most of the time, and OmniMark takes advantage of this in the way it chooses the find rules to look at. One thing that you should avoid, if you can, is a find rule that starts with any, and which usually fails later in the pattern — i.e. an any match that isn’t a “catch-all” at the bottom of the program.

The second principle of writing find rules is “succeed slow” — once you are in a find rule, or in a repeat scan match, pick up as much with it as you can. It is often the case in word processor formats, for example, that commands come in bunches. So pick up bunches, not just single commands. This cuts down the number of find rules that are performed.

With these two principles in mind, let’s visit an old friend:

find any

This little rule ends up sitting at the bottom of many an OmniMark program. It can sit by itself as above, in which case it means “please throw away anything you haven’t yet recognized.” Or it can do something simple, as in the following, where it copies the otherwise unrecognized character to the current output:

find any => one-character
   output one-character

In both cases, this little rule usually provides an excellent opportunity to speed up the OmniMark program. First of all, on the “fail fast, succeed slow” principles, it makes sense to pick up as long a run of characters as possible that will not be recognized by any other rule.

What these characters are depends very much on other find rules at work. For example, if other rules recognized text starting with any of the seven-bit ASCII graphic characters, then a rule such as:

find [\ " " to "~"]+

will “fail fast,” if the character being looked at is a seven-bit ASCII graphic, as well as “succeed slow,” because it will spend the time it takes to find all the following characters. And the nice thing about such a rule is that it can be put anywhere in the program — it’s not getting in the way of other find rules.

Just be careful to pick up any leftover characters that are sometimes, but not always picked up by other find rules. Leaving the good old “find any” rule in the program, after the “gobbler,” does this, or it can be combined as follows:

find [\ " " to "~"]+ | any

Note that the any doesn’t have a “+” sign. If it did, it would consume the rest of your input, without giving the other find rules a chance to examine it.

Finally, remember that OmniMark copies characters that are unrecognized by any find rule to the current output, and does it in a very efficient way. So if you simply want all otherwise unrecognized characters copied, don’t do the copying yourself — not doing anything is the best way to fail.

Alternatively, if you want to discard all characters not otherwise captured by find rules, you should make your default output be #suppress, and explicitly output to #main-output or wherever the output is going. Doing so makes unmatched characters go to #suppress — efficiently discarding them.

Divide and Conquer

A common need in pattern matching is to pick up everything up to, but not including, a known closing delimiter. For example, a match part that picks up the text in a quoted literal can do so as follows, with a quote character ending the picked-up text:

match "%"" [\ "%""]* => text "%""

For other than single-character closing delimiters, a simple “any except” character set doesn’t do the job. The following shows why:

match "--" [\ "-"]* => text "--"

The problem is that a single “-“, not part of a “–“, will terminate the matching done by the “any except” but won’t be matched by the final “–“. That’s where the lookahead used in the upto macro comes into play — it makes sure that the whole of the terminating delimiter is present.

Here’s a common solution to this problem, using the handy lookahead:

match "--" ((lookahead not "--") any)* "--"

What’s going on here is that the pattern repeatedly “looks ahead” to make sure that the terminating condition has not yet been met, and if it hasn’t, consumes another character. When the termination condition is found, the “*” loop exits and finds the following match, the final “–“.

The lookahead formulation does a good job of picking up text. But for those wanting to “fail fast, succeed slow”, it’s really unsatisfactory, because it examines every character in the text twice — once in the lookahead to ensure it’s not part of the closing delimiter, and a second time in the any gobbler. A better approach is one that “divides and conquers” — examining only once any character that isn’t a “-” using the “any except” form, and only doing a lookahead when a “-” is encountered:

match "--" ([\ "-"]+ | "-" lookahead not "-")* => text "--"

The divide and conquer approach to writing patterns comes in handy even when lookahead isn’t required. For example, the following match part picks up a C-like string, gobbling everything but quotes and “\” quickly, and handling “\” separately (so that a quote can be put in the text using “\””):

match "%"" ([\ "%"\"]+ | "\" any)* => text "%""

When the patterns start to become large and more complex, divide and conquer is the real winner. Here’s how to write a divide and conquer pattern in general:

1. For each character that is only sometimes matched, based on the character or sequence of characters that may precede it (such as characters preceded by “\” in C-like strings), construct an alternative for all possibilities starting with the first character of the preceding characters. For example:

   "\" any ; for escaped characters in C-like strings
   "%" any ; for escaped characters in OmniMark strings

2. For each character that is only sometimes matched, based on the character or sequence of characters that may follow it (such as the “-” possibly followed by another “-” in XML-like comments), construct an alternative that matches that character with a “lookahead” or “lookahead not” that excludes the non-matching cases. For example:

   "<" lookahead not [letter | "/!?"] ; for illegal uses of "<" in XML

3. Pick out all the characters that must always be matched, but are not matched by one of the previously constructed alternatives, and match them using a character set matcher with a “+” on it. For example:

   [" " to "~" \ "%"\"]+ ; for what's allowed "as is" in C strings
   [\ "%"%%"]+ ; for what's allowed "as is" in OmniMark strings

4. Take the partial patterns from the first three steps, connect them with “|” (or), putting the most likely alternatives first (for speed only). For example:

   ([" " to "~" \ "%"\"]+ | "\" any) ; for C string text

5. Append an “*” (repeat zero or more times) or a “+” (repeat one or more times) to the connected partial patterns, depending on whether or not the text as a whole can consist of zero characters. The result will look something like:

   ([" " to "~" \ "%"\"]+ | "\" any)*

6. Recursively apply divide and conquer to any of the constructed alternatives that itself matches delimited text.

As an example of divide and conquer, here’s a find rule that matches an XML start tag:

find "<" (letter [letter | digit | ".-_"]*) => element-name
   ([\ "%"'<>/"]+ |
      "%"" [\ "%""]* "%"" |
      "'" [\ "'"]* "'")* => attributes
   ("/" => empty-element)? ">"?

The core of the pattern (that matches the attributes) is a three-way alternative that picks up everything except a quote or apostrophe, and then tries each of the two types of quoted text. (Quoted text generally needs to be specially recognized because it can contain things that would be recognized as delimiters outside of quotes.)

** and ++

If you’ve used the “**” and “++” pattern operators introduced in OmniMark version 6, you’ll maybe be wondering where they fit into all of this. What “**” and “++” do is take the principles discussed above and apply them in a number of common and useful cases.

“**” and “++” take a (preceding) character set and (following) pattern, and match everything within that character set up to and including the pattern. (They differ only in that “++” fails if it does not encounter at least one character prior to the pattern.) For example, a convenient way of writing the XML comment matcher shown earlier in this paper is:

match "--" any** "--"

This is certainly shorter than the previous formulations. More importantly, it is easier to read and it runs efficiently.

OmniMark uses the divide and conquer principle on the pattern following “**” or “++” and builds a loop that only stops when it needs to look further, in the same way that the divide and conquer rewrite did. It saves the programmer the trouble, and is able to do things that can be hard for the programmer.

Although they’re very useful, and should get a lot of use, “**” and “++” don’t deal with all pattern matching problems. It’s good for the programmer to understand the principles described in this paper when they have to be applied explicitly.

by Helen Owens-Pope

Migrating legacy OmniMark programs

OmniMark has undergone a tremendous evolution from its earliest days as a very simple rule-based SGML scripting language, to its current state as a general-purpose programming language with modern software-engineering features.

During the course of this evolution, great pains have been taken to maintain backwards compatibility. Still, some changes to the core language have been necessary. The requirements of a general-purpose programming language, suitable for engineering complex high-performance systems, are very different from those of a simple narrowly-targeted scripting language.

Even so, many older programs do not need to be modified to work with current versions of OmniMark. The only programs that do need modification are those that are written with a version 2 (v2) coding style.

For most programs that do require modifications, they can be updated automatically in only a few seconds, using the migration script provided with this article. A few programs will require additional hand-editing, generally taking just a few minutes more.

This article will walk you through the migration process, and help you troubleshoot those few programs that require further modification.

This process can be run with any version of OmniMark from version 6 to the current version, and the results will be compatible with all of these releases.

You can download the conversion scripts mentioned here in zip format for Windows. These scripts are provided in source code form to make it easier for you to customize them for your particular code base.

This article assumes the use of the OmniMark Studio for Eclipse. In versions 8 and above, this will be the OmniMark Studio for Eclipse. In version 7, you might have either the Studio for Eclipse or the standalone Studio. In version 6 you will be using the standalone version of Studio. The program provided will work in all of these versions. The procedures for running the program vary slightly. The steps for using Studio for Eclipse are listed first, and the steps for standalone Studio follow. If you have a lot of files to convert, you may wish to compile the migration programs so that they can be run from a batch script. See the OmniMark Studio documentation for more information on compiling programs, and the OmniMark Engine documentation for information about running compiled programs.

If you are working with OmniMark version 6 in a Unix environment, you will have to compile the migration programs and use an OmniMark Engine to execute them.

Running the Migration Program

This step will take a few seconds for each file you need to migrate.

Procedure for OmniMark Studio for Eclipse

Run the to-six.xom program to upgrade the syntax.

1. Unzip the program files into a suitable directory.
2. In OmniMark Studio for Eclipse, create a project for running the migration.
   1. From the file menu, choose New -> project
   2. Expand the OmniMark option and choose OmniMark Project.
   3. Enter a name for your project.
   4. Uncheck “Use default” and navigate to the directory where you have placed your files.
3. Open to-six.xom
4. Create a launch configuration for the program to-six.xom by pulling down the “Run” menu and clicking on “Run as OmniMark Program”. Don’t worry about the error messages.
5. In the Parameters tab, under Arguments, enter
   1. “include=” to specify the directory containing any include files used by the program you are upgrading, for example include=C:\MyPrograms\OmniMark\xin
   2. the path to the program you want to migrate
   3. the path to the output destination. Don’t overwrite your input file. Either give the new file a different name or place it in a different directory

      -of newfile.xom

   4. the path to a log file to capture any error messages

      -log logfile.xom

   5. If you are using OmniMark version 8 or newer, you may want to consider adding the command-warning-ignore deprecated here, to avoid seeing numerous warnings about obsolete syntax. The obsolete syntax has been retained in this program so that it continues to work in versions 6 and 7.
6. Click Run at the bottom of the launch configuration screen.
7. Examine the result.
   1. You will find that all of the lines have been moved over by a few spaces. Whenever the program changes a line, it inserts the original line in front, as a comment beginning with “;pre-6”. Spaces are added to the front of the rest of the lines to keep everything lined up. (These extra spaces and comment lines will be removed in a later step.)
   2. You should examine the changes to make sure that the new lines are correct. You may examine the changes by searching for the text “;pre-6”.
   3. Open the log file to examine the error messages. If you see any errors, you will have to correct the code by hand. There should be very few (if any) errors. You may see warnings. Warnings about deprecated syntax can be ignored. For other warnings, you should examine the referenced lines in the output file and make sure that the code is correct. If not, you should correct the output file by hand.

Procedure for standalone Studio (OmniMark 6)

1.     Run the to-six.xop project to upgrade the syntax.

a.   In OmniMark Studio, open the project to-six.xop.

b.   Select the input file, the output file, and the log file by editing the project options. If you are migrating a program file and it includes files from a different directory, then you also specify the include path at this step.

i.   Specify the directories to be searched for include files if any. Click on the “Arguments” tab. Type “include=” followed by the name of a directory that contains include files. (Do not leave spaces around the “=” sign.) Do this once for each directory that contains include files. For example, to search the folder “C:\MyPrograms\OmniMark\xin” enter:

include=C:\MyPrograms\OmniMark\xin

ii.   Click on the “Arguments” tab, and browse to the include or program file that you want to migrate, and add it to the argument list.

iii.   Click on the “Output” tab, and type in the name of the output file. This will be a new version of your original include or program file, compatible with OmniMark 6 and above. Do not overwrite your input file. Either give your output file a new name or place it in a different folder.

iv.   Also in the “Output” tab, type in the name of a log file to capture any error messages.

c.   Save the project.

d.   Pull down the “Run” menu, and select “Execute Project”.

e.   When you see “Hit <ENTER> to continue.”, press the enter key to return to the Studio.

2.     Examine the result to make sure it is correct.

a.   Open the output file. You will find that all of the lines have been moved over by a few spaces. Whenever the program changes a line, it inserts the original line in front, as a comment beginning with “;pre-6”. Spaces are added to the front of the rest of the lines to keep everything lined up. (These extra spaces and comment lines will be removed in a later step.)

b.   You should examine the changes to make sure that the new lines are correct. You may examine the changes by searching for the text “;pre-6”.

c.   Open the log file to examine the error messages.

If you see any warnings, you should examine the referenced lines in the output file and make sure that the code is correct. If not, you should correct the output file by hand.

Once you have finished converting your program files and your include files, you should try running the programs with the newer version of OmniMark.

Procedure for OmniMark Studio for Eclipse

1. In OmniMark Studio for Eclipse, bring your updated program into Eclipse as before.
2. Select the program in the navigator window and choose Run… from the Run menu to open a launch configuration. Specify the options you want, and click “run”

At this point you should have very few syntax errors if any. Correct any of the remaining errors by hand, and then run your programs again. Make sure they produce the same results as your old programs under your old version of OmniMark. If you have any problems, or just want to understand this process better, see Appendix C: What the Migration Process Does.

Part 4: Cleaning Up

At this point, you have successfully upgraded your programs to work with the newest versions of OmniMark. Now you probably want to get rid of the comment lines added during this process. This step will take a few seconds for each file you are upgrading.

Procedure for OmniMark Studio for Eclipse:

1. Open clean.xom in OmniMark Studio for Eclipse.
2. In the Launch Configuration, specify the output file from to-six.xom as the input, and save the output to a new file.
3. Run the program.

Procedure for standalone Studio (OmniMark 6)

1. Open the project clean-six.xop.
2. Edit the project options:
   1. The input file should be the should be the output file from to-six.xop.
   2. Save the output to a new file.
3. Save the project file.
4. Pull down the “Run” menu, and select “Execute Project”.
5. When you see “Hit <ENTER> to continue.”, press the enter key to return to the Studio.

You can compare this output file to your original (pre-OmniMark 6) file with any line-by-line comparison utility. You will see that the only changes are the ones necessary to upgrade the syntax.

     omnimark --
     OmniMark Error xxxx on line 1179 in file my-prog-1.xom:
     Syntax Error.
     ...  The keyword 'SDATA' wasn't recognized because there is a
     variable, function, or opaque type with the same name.

WARNING: Quoted variable name (stream "my-var") -
  replacing with v7 syntax.

     find digit+ => id ":" any-text+ => value "%n"
       local counter id
       set id to pattern id
       ...

by Helen Owens-Pope