This is the third installment in the series of posts about designing a Command Line Interface (CLI) parser for C++. The previous posts were:

• Project Introduction
• Problem and Terminology

Today I would like to explore the solution space and get an idea about what the ideal solution might look like.

Using the terminology introduced in the previous post, an application may need to access the following three objects that result from the command line parsing: commands, options, and arguments. Both commands (or, usually, just one command) and arguments are homogeneous arrays of strings. It is normally sufficient to present them to the application as such, either directly in the argv array by identifying their start/end positions or as separate string sequences. Options, on the other hand, are a more interesting problem.

If we start thinking about the form in which we could make the parsed options information available to our applications, several alternatives come to mind. In a very simple application we might have a variable (global or declared in main()) for each option. The CLI parser then sets these variables to the values provided in the command line. Something along these lines:

bool help = false;
bool version = false;
unsigned short compression = 5;
 
int main (int argc, char* argv[])
{
  cli::parser p;
  p.option ("--help", help);
  p.option ("--version", version);
  p.option ("--compression", compression);
  p.parse (argc, argv);
 
  if (help)
  {
    ...
  }
}

The major problem with this approach is that it does not scale to a more modularized design. In such applications each module may have a specific set of options. For example, in the XSD and XSD/e compilers the compiler driver, frontend, and each code generator has a unique set of options. Placing the corresponding variables all in the global namespace is cumbersome. They are more naturally represented as member variables in the corresponding module classes.

Of course, nothing prevents us from parsing directly into member variables using the above solution. However, it requires that all the classes that hold option values be instantiated before command line parsing can begin. This creates a chicken and egg problem since these classes often need the option values in their constructors. The only way to resolve this problem with the above approach is to first parse the options into temporary variables which are then used to initialize the modules. Here is an example:

struct compressor
{
  compressor (unsigned short level);
};
 
int main (int argc, char* argv[])
{
  bool help = false;
  bool version = false;
  unsigned short compression = 5;
 
  cli::parser p;
  p.option ("--help", help);
  p.option ("--version", version);
  p.option ("--compression", compression);
  p.parse (argc, argv);
 
  compressor c (compression);
}

Another drawback of this approach is the need to repeat each option name twice: first as the variable name (e.g., help) and then as the option name (e.g., "--help"). Furthermore, in the case of global variables, there are two distinct places in the source code where each option must be recorded: first as the variable name and then as the call to option(). In non-trivial allocations the global option variables would most likely also be declared as extern in a header file so that they can be accessed from other modules. This brings the number of places where each option is recorded to three.

The alternative approach to storing the option values in individual variables is to have a dedicated object which holds them all. The application can then query this object for individual values. Logically, such an object is a heterogeneous map of option names to their values and we can use the map interface to access individual option values. Here is how this might look:

int main (int argc, char* argv[])
{
  cli::parser p;
  p.option<bool> ("--help");
  p.option<bool> ("--version");
  p.option<unsigned short> ("--compression", 5);
 
  cli::options o (p.parse (argc, argv));
 
  if (o.value<bool> ("--help"))
  {
    ...
  }
}

There are a number of drawbacks with this interface. The first is the use of strings to identify options. If we misspell one, the error will only be detected at runtime. The second drawback is the need to specify the value type every time we access the option value. Then we have the verbosity problem as in the previous approach. Option names and option types are repeated in several places in the source code which makes it hard to maintain.

The alternative interface design would be to have an individual accessor for each option. Something along these lines:

struct options: cli:options
{
  options ()
    : help_ (false), version_ (false), compression_ (5)
  {
    // The option() function is provided by cli::options.
    //
    option ("--help", help_);
    option ("--version", version_);
    option ("--compression", compression_);
  }
 
  bool help () const;
  bool version () const;
  unsigned short compression () const;
 
private:
  bool help_;
  bool version_;
  bool compression_;
};
 
int main (int argc, char* argv[])
{
  cli::parser<options> p;
  options o (p.parse (argc, argv));
 
  if (o.help ())
  {
    ...
  }
}

While we have solved all the problems with accessing the option values, the declaration of the options class is very verbose. For each option we repeat its name five times plus we have to manually implement each accessor, initialize each option variable with the default value, as well as register each option with cli:options. We could automate some of these step by using functor objects to store the option values as well as implement the accessors, for example:

struct options: cli:options
{
  options ()
    : help (false), version (false), compression (5)
  {
    option ("--help", help);
    option ("--version", version);
    option ("--compression", compression);
  }
 
  cli::option<bool> help;
  cli::option<bool> version;
  cli::option<unsigned short> compression;
};

We could also get rid of the explicit calls to the option() function by making the cli::option object automatically register with the containing object (we would need to use a global variable or a thread-local storage (TLS) slot to store the current containing object). Here is how the resulting options class could look:

struct options: cli:options
{
  options ()
    : help (false, "--help"),
      version (false, "--version"),
      compression (5, "--compression")
  {
  }
 
  cli::option<bool> help;
  cli::option<bool> version;
  cli::option<unsigned short> compression;
};

With this approach we have reduced the number of option name repetitions from five to three.

How does the above approach address the issue of modularized applications that we brought up earlier? One alternative would be to have the corresponding member variables added manually to module classes and then initialized with values from the options object. For example:

struct compressor
{
  compressor (unsigned short level)
    : level_ (level)
  {
  }
 
private:
  unsigned short level_;
};
 
int main (int argc, char* argv[])
{
  cli::parser<options> p;
  options o (p.parse (argc, argv));
 
  compressor c (o.compression ());
}

Alternatively, we could use the options object directly by inheriting the module class from it. For that, however, we would also need to split the options object into several module-specific parts, for example:

struct compression_options: virtual cli:options
{
  compression_options ()
    : compression (5)
  {
    option ("--compression", compression);
  }
 
  cli::option<unsigned short> compression;
};
 
struct compressor: private compression_options
{
  compressor (const compression_options& o)
    : compression_options (o)
  {
  }
};
 
struct options: compression_options
{
  options ()
    : help (false, "--help"),
      version (false, "--version")
  {
  }
 
  cli::option<bool> help;
  cli::option<bool> version;
};
 
int main (int argc, char* argv[])
{
  cli::parser<options> p;
  options o (p.parse (argc, argv));
 
  compressor c (o.compression ());
}

At this point it appears that we have analyzed the drawbacks of all the practical approaches and can now list the properties of an ideal solution:

1. Aggregation: options are stored in an object
2. Static naming: option accessors have names derived from option names
3. Static typing: option accessors have return types fixed to option types
4. No repetition: the option name and option type are specified only once for each option

With these properties figured out, next time we will examine the drawback of the existing solutions, namely the Program Options library from Boost as well as my previous attempt at the CLI library which is part of libcult. As usual, if you have any thoughts, feel free to add them as comments.

This is the second installment in the series of posts about designing a Command Line Interface (CLI) parser for C++. The previous post was CLI in C++: Project Introduction. Today I would like to talk about the problem that we are trying to solve as well as establish the terminology.

Command line interface is the most universal way of passing information from a caller to a program. The concept of a command line is part of most operating systems and programming languages including C++. However, the model and terminology for command line processing vary significantly among different platforms.

The Single UNIX Specification includes Utility Argument Syntax Conventions and Guidelines which document basic terminology for command line processing. The Single UNIX Specification model is the least common denominator for different UNIX implementations. It is minimal and targets system utilities rather than a wider range of applications and their possible requirements. Another de-facto command line processing model is the GNU Standard for Command Line Interfaces which generally encourages conformance to the Single UNIX Specification but adds a few extensions and uses different terminology. Our CLI parser will need to handle a wider range of use cases than those covered by these two standards. We would therefore need to establish a more complete model and associated terminology for command line processing.

Command line is an array of character strings and not just a string with spaces between words as is sometimes incorrectly assumed. Each string in a command line array is referred to as argument. The first argument usually contains the name of the executable.

The interpretation of arguments is completely up to the program logic, however, conventions exist that vary among different systems. Usually some arguments are translated into higher-level objects such as commands and options. These objects form a model for command line processing and are defined below.

Command is usually a word or a single letter that represents a command to the program logic. Neither the Single UNIX Specification nor the GNU Standard for Command Line Interfaces has the notion of a command. Other terms for command include action and function. Command is usually (but not necessarily) the first argument after the executable name. Here are a few examples:

tar x

Here we have a one letter command x (extract). In GNU tar manual it is called functional letter.

tar xvf

Here we have three commands encoded as a single letter each. Semantically, only x is a command while v (verbose) and f (read from a file) are options.

openssl req

Here we have a word command req (operations with certificate requests).

cvs checkout foo

Here we have a word command checkout and command argument foo.

tar --help

Even though --help is usually considered an option, semantically it is a command.

Option consists of option name and, optionally, one or more option values. Options are normally optional. Non-optional options are better represented by commands or arguments.

An option name takes up one argument. Option names usually start with a prefix, for example, --compile-only, -c or /c. This helps distinguish them from commands and arguments. Option names may have aliases, for example, for option name --output-dir there could be the -o alias.

An option without a value is always optional and represents an option with an implied binary value ({0, 1} or {false, true}). Such an option is sometimes called flag.

An option can be associated with a program or a command. Thus the concept of an option can be further refined to program option and command option. Program options alter behavior of the program as a whole while command options are only affecting and may only be valid for a particular command. For example:

g++ -o hello.o hello.cpp

Here we have an option with name -o which has value hello.o. hello.cpp is an argument.

ls -l

Here we have a flag with name -l.

cvs -z 6 checkout -P foo

Here we have a program option with name -z and value 6 (set compression level to 6). checkout is a command. -P is a command flag (prune empty directories). foo is a command argument.

Argument usually represents an input value or a parameter and can be mandatory or optional. The interpretation of arguments is application-specific. The same as with the options, the concept of an argument can be further refined to program argument and command argument.

Note that above we are using the term argument to mean both an element in the command line string array as well as the input value to the program that is distinct from commands and options. From the operating system point of view every item that is passed to a program via command line is an argument. It is up to the program to interpret them as commands, options, or arguments proper. The special -- argument is often used to indicate that all the following arguments must be treated as arguments proper.

It may seem premature to establish such a complete model for the initial version of the CLI parser that we are designing, especially because most applications will only use the basic subset of this model (options and arguments). I, however, prefer to think things through on the conceptual level even if there are no immediate plans to support them in the code. This way when designing the first version I can make sure that I at least understand how the complete model will fit or can be supported in the future versions without a complete redesign.

In its simplest form the task of parsing a command line boils down to determining if one or more options are specified in the command line string array and presenting this information to the rest of the application in a convenient way. This process is complicated by the fact that options can normally appear in arbitrary order. Some options may also have values, in which case they need to be extracted and converted into suitable data types (for example, the compression level probably needs to be converted to an integer). While most option values will use simple types such as integers and strings, it is plausible that conversions to application-specific types may be required. The parsing code also needs to perform reasonable error handling, such as detecting unknown options, missing option values, and value conversion failures.

The application may also need to set the default values for some options. These values are then used by the program logic in case the corresponding options were not specified.

Handling of commands and arguments is usually quite a bit simpler. Once the options have been parsed, the starting positions of a command and arguments in the command line array become known and they can be accessed by the application directly.

There is also the related problem of producing command line documentation, such as program usage information and man pages. Some applications, especially with a large number of options, may also want to allow their users to specify command line arguments in one or more files in addition to the command line proper.

And that’s it for today. If you have any thoughts, feel free to add them as comments. Next time we will try to understand what an ideal solution to the CLI parsing problem might look like. We will also analyze the shortcomings of some of the existing implementations. For that I would like to consider the Program Options library from Boost as well as my previous attempt at the CLI library which is part of libcult. We will also briefly examine whether any new features planned for C++0x could be used to address these shortcomings.