ODB 2.1.0 was released today.

In case you are not familiar with ODB, it is an object-relational mapping (ORM) system for C++. It allows you to persist C++ objects to a relational database without having to deal with tables, columns, or SQL, and manually writing any of the mapping code. ODB natively supports SQLite, PostgreSQL, MySQL, Oracle, and Microsoft SQL Server. Pre-built packages are available for GNU/Linux, Windows, Mac OS X, and Solaris. Supported C++ compilers include GCC, MS Visual C++, Sun CC, and Clang.

This release packs a long list of new features (note to ourselves: if the NEWS entries are over a page long — time to release). The major ones include the ability to use accessor and modifier functions/expressions to access data members, the ability to declare virtual data members, the ability to define database indexes, as well as support for mapping extended database types, such as geospatial types, user-defined types, and collections. There are also notable additions to the profile libraries. The Boost profile now includes support for the Multi-Index and Uuid libraries while the Qt profile now supports the QUuid type. Furthermore, there is a number of improvements in individual database support, especially for SQLite (see below).

We have also added Visual Studio 2012 and Clang 3.1 to the list of compilers that we use for testing. Specifically, all the runtime libraries, examples, and tests now come with project/solution files for Visual Studio 2012 in addition to 2010 and 2008. As always, below I am going to examine these and other notable new features in more detail. For the complete list of changes, see the official ODB 2.1.0 announcement.

Accessors and Modifiers

ODB now supports multiple ways to access data members in persistent objects, views, and value types. Now, if the data member is not accessible directly, the ODB compiler will try to automatically discover suitable accessor and modifier functions. By default the ODB compiler will look for names in the form: get/set_foo(), get/setFoo(), get/setfoo(), as well as just foo(). You can also add custom name derivations with the --accessor-regex and --modifier-regex options. Here is an example:

 
#pragma db object
class person
{
public:
  const std::string& name () const;
  void setName (const std::string&);
 
private:
  std::string name_; // Using name() and setName().
 
  ...
};
 

If the ODB compiler was unable to find a suitable accessor or modifier function, then we can specify one explicitly with the new get and set pragmas. For example:

 
#pragma db object
class person
{
public:
  const std::string& get_full_name () const;
  std::string& set_full_name ();
 
private:
  #pragma db get(get_full_name) set(set_full_name)
  std::string name_;
 
  ...
};
 

In fact, it doesn’t have to be just a function. Rather, it can be an accessor or modifier expression. Here is a more interesting example:

 
#pragma db object
class person
{
  public:
    const char* name () const;
    void name (const char*);
 
  private:
    #pragma db get(std::string (this.name ())) \
               set(this.name ((?).c_str ()))
    std::string name_;
 
  ...
};
 

For more information on automatic discovery of accessors and modifiers, refer to Section 3.2, “Declaring Persistent Objects and Values” in the ODB manual. For details on how to specify custom accessor/modifier expressions, see Section 12.4.5, “get/set/access” as well as the access example in the odb-examples package.

Virtual Data Members

A virtual data member is an imaginary data member that is only used for the purpose of database persistence. A virtual data member does not actually exist (that is, occupy space) in the C++ class.

At first, the idea of a virtual data member may seem odd but if you think about it, it’s a natural extension of the accessor/modifier support discussed above. After all, if we have an accessor/modifier pair, why do we have to have a physical data member to tie it to? Probably the best way to illustrate this idea is to show how we can use virtual data members to handle the C++ pimpl idiom:

 
#pragma db object
class person
{
public:
  const std::string& name () const;
  void name (const std::string&);
 
  unsigned short age () const;
  void age (unsigned short) const;
 
private:
  struct impl;
 
  #pragma db transient
  impl* pimpl_;
 
  #pragma db member(name) virtual(std::string)
  #pragma db member(age) virtual(unsigned short)
 
  ...
};
 

Besides the pimpl idiom, virtual data members can also be useful to aggregate or dis-aggregate real data members and to handle third-party types for which names of real data members may not be known.

Note also that virtual data members have nothing to do with C++ virtual functions or virtual inheritance. Specifically, no virtual function call overhead is incurred when using virtual data members.

For more information on virtual data members, refer to Section 12.4.13, “virtual” in the ODB manual as well as the access and pimpl examples in the odb-examples package.

Database Indexes

ODB now supports defining database indexes within the pragma language. If all you need is a simple index on a particular data member (simple or composite), then all you have to do is specify either the index (for non-unique index) or unique (for unique index) pragma. For example:

 
#pragma db object
class person
{
  ...
 
  #pragma db unique
  std::string name_;
 
  #pragma db index
  unsigned short age_;
};
 

It is also possible to define an index on more than one member as well as to give it a custom name:

 
#pragma db object
class person
{
  ...
 
  std::string first_;
  std::string last_;
 
  #pragma db index("name_i") unique members(first_, last_)
};
 

ODB also supports database-specific index types, methods, and options. Here is an example of a more involved PostgreSQL-specific index definition:

 
#pragma db object
class person
{
  ...
 
  std::string name_;
 
  #pragma db index                            \
             type("UNIQUE CONCURRENTLY")      \
             method("HASH")                   \
             member(name_, "DESC")            \
             options("WITH(FILLFACTOR = 80)")
};
 

For more information on defining database indexes, refer to Section 12.6, “Index Definition Pragmas” in the ODB manual.

Mapping Extended Database Types

Besides the standard integers, strings, and BLOBs, most modern database implementations also provide a slew of extended SQL types. Things like geospatial types, user-defined types, collections (arrays, table types, etc), key-value stores, XML, JSON, etc. While ODB does not support such extended types directly (it would take years to cover all the types in all the databases), it now includes a mechanism which, with a bit of effort, allows you to map pretty much any extended SQL type to any C++ type.

This is a really big and powerful feature. As a result, I wrote a separate post that is dedicated just to Extended Database to C++ Type Mapping. It provides much more detail and some cool examples. There is also Section 12.7, “Database Type Mapping Pragmas” in the ODB manual.

Profile Library Improvements

Both Boost and Qt profile libraries now include persistence support for their respective UUID types. By default, these types are mapped to a UUID SQL type if the database provides such a type (e.g., UUID in PostgreSQL and UNIQUEIDENTIFER in SQL Server) or to a suitable 16-byte binary type otherwise. As a result, you can now use boost::uuids::uuid and QUuid in your persistent classes without any extra effort:

 
// Boost version.
//
#pragma db object
class person
{
  ...
 
  boost::uuids::uuid id_;
};

// Qt version.
//
#pragma db object
class Person
{
  ...
 
  QUuid id_;
};
 

For more information on UUID support in Boost, refer to Section 19.6, “Uuid Library” in the ODB manual. For Qt, see Section 20.1, “Basic Types”.

In addition, the Boost profile now includes support for the Multi-Index container library. While there are some interesting implementation details about which I am planning to write in a separate post, from the user perspective, multi_index_container can now be used in persistent classes just as any standard container. For example:

 
namespace mi = boost::multi_index;
 
#pragma db object
class person
{
  ...
 
  typedef
  mi::multi_index_container<
    std::string,
    mi::indexed_by<
      mi::sequenced<>,
      mi::ordered_unique<mi::identity<std::string> >
    >
  > emails;
 
  emails emails_;
};
 

For more information on Multi-Index container support, refer to Section 19.3, “Multi-Index Container Library” in the ODB manual.

Combined Database Schema

The ODB compiler now supports the generation of the combined SQL file from multiple header files. For example:

 
odb ... --generate-schema-only --at-once --output-name schema \
employee.hxx employer.hxx
 

The result of the above command will be the schema.sql file that contains database creation code (DLL statements) for persistent classes defined in both employee.hxx and employer.hxx headers.

A combined SQL file can be easier to work with, for example, send to a DBA for review. It can also be useful when dealing with circular dependencies, as discussed in Section 6.3 “Circular Relationships” in the ODB manual.

C++11 std::array to BLOB Mapping

ODB now includes built-in support for mapping C++11 std::array<char, N> and std::array<unsigned char, N> types to BLOB/BINARY database types. For example:

 
#pragma db object
class person
{
  ...
 
  #pragma db type("BINARY(1024)")
  std::array<char, 1024> pub_key_;
};
 

SQLite Support Improvements

On Windows, SQLite ODB runtime now supports persistence of std::wstring. You can also pass the database name as std::wstring in addition to std::string. The odb::sqlite::database class constructors have also been extended to accept the virtual filesystem (vfs) module name. Finally, the default SQLite mapping for float and double now allows the NULL value since SQLite treats NaN values as NULL.

In my work on ODB I periodically run into a need to define a set container that uses only a subset of data members from a class. Occasionally I also need to have several such sets for a single class, each containing the same elements but using different subsets of the data members. Here is a motivating example (I am omitting accessors and instead making all the data members public for brevity):

 
struct person
{
  person (const std::string& email,
          const std::string& name,
          unsigned short age);
 
  std::string email;
  std::string name;
  unsigned short age;
};
 

Let’s say we want to have std::set of person elements that only use the email data member for ordering. We can create such a set fairly easily by defining a custom comparator:

 
struct email_comparator
{
  bool operator() (const person& x, const person& y) const
  {
    return x.email < y.email;
  }
};
 
typedef std::set<person, email_comparator> person_set;
 

The limitations of this approach, however, become obvious as soon as we try to use find(). We cannot just pass the email to this function. Instead, we have to pass a person instance with the desired email:

 
person_set s;
...
auto i (s.find (person ("john@doe.com", "", 0)));
 

This is inelegant. And potentially inefficient, if creating a dummy person instance is expensive. While we cannot do much about inefficiency, we can hide the ugliness by providing a custom version of the find() function:

 
struct person_set: std::set<person, email_comparator>
{
  typedef std::set<person, email_comparator> base;
 
  iterator find (const std::string& email) const
  {
    return base::find (person (email, "", 0));
  }
};
 

Things get worse if we need to be able to find elements using different ordering criteria. In our example, besides email, we may also want to lookup people based on their name. Standard set doesn’t provide this functionality and what people normally resort to is maintaining multiple sets each containing all the elements, perhaps using shared_ptr in order to avoid duplication.

If you are familiar with Boost multi_index_container then you are probably jumping out of your seat screaming that this is exactly the problem this container is here to solve. And you would be absolutely right. With Boost multi_index_container our person_set can be defined like this:

 
namespace mi = boost::multi_index;
 
typedef mi::multi_index_container<
  person,
  mi::indexed_by<
    mi::ordered_unique<
      mi::member<person, std::string, &person::email>
    >
  >
> person_set;
 

While the declaration is quite a bit more involved, in return we get a perfect find() signature that takes email as its argument:

 
person_set s;
...
auto i (s.find ("john@doe.com"));
 

It is also easy to extend the above person_set container to include another ordering criteria (called index):

 
struct by_email {};
struct by_name {};
 
typedef mi::multi_index_container<
  person,
  mi::indexed_by<
    mi::ordered_unique<
      mi::tag<by_email>,
      mi::member<person, std::string, &person::email>
    >,
    mi::ordered_unique<
      mi::tag<by_name>,
      mi::member<person, std::string, &person::name>
    >
  >
> person_set;
 
person_set s;
...
auto i (s.get<by_email> ().find ("john@doe.com"));
auto j (s.get<by_name> ().find ("Jane Doe"));
 

Now, if your project is already using Boost or if you need functionality beyond simple insert()/find(), then by all means use multi_index. On the other hand, if you are not using Boost, then adding this dependency for a simple use-case like this definitely sounds like an overkill. Furthermore, some projects, for various reasons, may not be allowed to add this dependency. For example, while you can use ODB with Boost (support for Boost and Qt is provided as add-on profile libraries), we choose to maintain the ODB compiler itself as dependency-free as possible.

So if we cannot use multi_index, are we forever relegated to using ugly and inefficient hacks discussed above? As it turns out, we can emulate functionality offered by multi_index using just the standard containers. And it will be pretty efficient and convenient to use, too.

Let’s start with the first case where we want a set that indexes on a specific data member (email in our case). Our goal is to get rid of the requirement to create a dummy person instance during lookup. Which standard container would allow us to associate a single value (email) with the whole object (person)? std::map seem to fit the bill. So, here is our first draft of a solution:

 
typedef std::map<std::string, person> person_set;
 

While this solves our immediate problem (we can now call find() with just the email), it added a couple of new ones. Let’s start with the fact that we now have two copies of the email stored for each entry: one as the map key and the other in the person object. That is wasteful. What can we do about it? Since both copies contain exactly the same value and their lifetime is exactly the same, can’t we make one just point to the other? Making person::email point to the map key is probably a bad idea. After all, person instances can exist outside our container. But the other way around sounds reasonable. That is, we make the map key point to the string stored in the value:

 
typedef std::map<const std::string*, person> person_set;
 

There is still a little problem with this approach. When we insert an element into a map (as a key and value pair), we don’t know the address of the email member in the value that is to be created (remember std::map will make a copy of the passed pair). Or, in other words, when we call insert(), we have to pass the address to something that will only be created inside this insert(). Bummer! Maybe we should add that Boost dependency after all…

For those who are still with me, the way we can work around this limitation is to pass a temporary pointer to insert() and change it to point to the inserted value after insert() has completed. To be able to do this we first have to circumvent the const-ness of the key in the map. And for that we use a little helper class template called key:

 
template <typename T>
struct key
{
  mutable const T* p;
 
  key (const T* v = 0): p (v) {}
  bool operator< (const key& x) const {return *p < *x.p;}
};
 

The key’s only purpose is to give us a mutable pointer that we can update. While at it, it also conveniently provides operator<. Ok, let’s look at our next draft (we will attend to that ??? in a second):

 
struct person_set
{
  typedef std::map<key<std::string>, person> email_map;
 
  std::pair<???, bool>
  insert (const person& v)
  {
    auto r (email_map_.insert (email_map::value_type (&v.email, v)));
 
    if (r.second)
      r.first->first.p = &r.first->second.email;
 
    return std::make_pair (r.first, r.second);
  }
 
private:
  email_map email_map_;
};
 

Ok, that wasn’t as bad as it sounded. We basically insert a key/value pair into the map using the passed value’s email member as a temporary pointer. If the insertion succeeded, we update that pointer with the email member from the inserted value. From the map’s perspective nothing changed since the pointed-to string is the same. That was the trickiest part of the whole thing. The rest is just bringing it home.

The other problem with using std::map as a base of our set implementation is that the iterator no longer points to person. Instead, we have std::pair containing the key and the value. This is not terribly convenient and fairly easy to fix with another little helper called map_iterator_adapter. Essentially it takes an std::map iterator and turns it into an iterator that iterates over map’s values while ignoring the keys:

 
template <typename I>
struct map_iterator_adapter: I
{
  typedef const typename I::value_type::second_type value_type;
  typedef value_type* pointer;
  typedef value_type& reference;
 
  map_iterator_adapter () {}
  map_iterator_adapter (I i): I (i) {}
 
  reference operator* () const {return I::operator* ().second;}
  pointer operator-> () const {return &I::operator-> ()->second;}
};
 

Ok, let’s put everything together:

 
struct person_set
{
  typedef std::map<key<std::string>, person> email_map;
  typedef map_iterator_adapter<email_map::const_iterator> iterator;
 
  std::pair<iterator, bool>
  insert (const person& v)
  {
    auto r (email_map_.insert (email_map::value_type (&v.email, v)));
    iterator i (r.first);
 
    if (r.second)
      r.first->first.p = &i->email;
 
    return std::make_pair (i, r.second);
  }
 
  iterator
  find (const std::string& email) const
  {
    return email_map_.find (&email);
  }
 
  iterator begin () const {return email_map_.begin ();}
  iterator end () const {return email_map_.end ();}
 
private:
  email_map email_map_;
};
 

For our purposes, this version of person_set can be used just like the one based on Boost multi_index:

 
person_set s;
s.insert (person ("john@doe.com", "John Doe", 29));
s.insert (person ("jane@doe.com", "Jane Doe", 27));
auto i (s.find ("john@doe.com"));
 

What if we want to add another index, say for name? The idea is to add another map with the iterator from the first map as its value. The highlighted with <-- fragments correspond to the changes necessary to add support for another index:

 
struct person_set
{
  typedef std::map<key<std::string>, person> email_map;
  typedef map_iterator_adapter<email_map::const_iterator> iterator;
 
  typedef std::map<key<std::string>, iterator> name_map;           // <--
 
  std::pair<iterator, bool>
  insert (const person& v)
  {
    // First check that we don't have any collisions in
    // the secondary indexes.
    //
    {
      auto i (name_map_.find (&v.name));                           // <--
      if (i != name_map_.end ())                                   // <--
        return std::make_pair (i->second, false);                  // <--
    }
 
    auto r (email_map_.insert (email_map::value_type (&v.email, v)));
    iterator i (r.first);
 
    if (r.second)
    {
      r.first->first.p = &i->email;
      name_map_.insert (name_map::value_type (&i->name, i));       // <--
    }
 
    return std::make_pair (i, r.second);
  }
 
  iterator                                                         // <--
  find_email (const std::string& email) const                      // <--
  {                                                                // <--
    return email_map_.find (&email);                               // <--
  }                                                                // <--
 
  iterator
  find_name (const std::string& name) const
  {
    auto i (name_map_.find (&name));
    return i != name_map_.end () ? i->second : end ();
  }
 
  iterator begin () const {return email_map_.begin ();}
  iterator end () const {return email_map_.end ();}
 
private:
  email_map email_map_;
  name_map name_map_;                                              // <--
};
 

What about multi-member (composite) indexes? Say we change our person class to store the first and last names separately but would still like to do lookup based on these two members:

 
struct person
{
  person (const std::string& email,
          const std::string& first,
          const std::string& last,
          unsigned short age);
 
  std::string email;
  std::string first;
  std::string last;
  unsigned short age;
};
 

Boost multi_index can handle this quite easily. We can also support composite indexes in our emulation if we extend our key helper to handle multi-member keys:

 
template <typename T, typename... R>
struct key: key<R...>
{
  typedef key<R...> base;
 
  mutable const T* p;
 
  key (): p (0) {}
  key (const T* v, const R*... r): base (r...), p (v) {}
 
  void assign (const T* v, const R*... r) const 
  {
    p = v; base::assign (r...);
  }
 
  bool operator< (const key& x) const
  {
    return *p < *x.p || (!(*x.p < *p) && base::operator< (x));
  }
};
 
template <typename T>
struct key<T>
{
  mutable const T* p;
 
  key (const T* v = 0): p (v) {}
  void assign (const T* v) const {p = v;}
  bool operator< (const key& x) const {return *p < *x.p;}
};
 

Based on this improvement we can now implement a set with first and last members as a key:

 
struct person_set
{
  typedef key<std::string, std::string> name_key;
  typedef std::map<name_key, person> name_map;
  typedef map_iterator_adapter<name_map::const_iterator> iterator;
 
  std::pair<iterator, bool>
  insert (const person& v)
  {
    auto r (name_map_.insert (
              std::make_pair (name_key (&v.first, &v.last), v)));
    iterator i (r.first);
 
    if (r.second)
      r.first->first.assign (&i->first, &i->last);
 
    return std::make_pair (i, r.second);
  }
 
  iterator
  find (const std::string& first, const std::string& last) const
  {
    return name_map_.find (name_key (&first, &last));
  }
 
  iterator begin () const {return name_map_.begin ();}
  iterator end () const {return name_map_.end ();}
 
private:
  name_map name_map_;
};
 

Given a choice one should probably prefer Boost.MultiIndex to the approach shown above. However, there is one potential advantage of our custom solution compared to multi_index which has serious limitations on the mutability of elements. On the other hand, because we store elements in a map, it is fairly straightforward to allow mutation provided we are careful not to modify any members that are involved in indexes.

ODB is dual-licensed under the GPL and a proprietary license (there is both a free and a commercial version of the latter). As discussed in the ODB Licensing FAQ this model caters fairly well to other open source projects that use the GPL (or a similar license), private closed source projects, as well as commercial closed source projects. However, there is a significant drawback in this dual-licensing model when it comes to other open source projects that use a more liberal (than the GPL) license. In this post I would like to discuss the nature of this problem in more detail as well as explain what we did to allow more liberally-licensed projects to still use and benefit from ODB.

In case you are not familiar with ODB, it is an object-relational mapping (ORM) system for C++. It allows you to persist C++ objects to a relational database without having to deal with tables, columns, or SQL, and manually writing any of the mapping code. ODB natively supports SQLite, PostgreSQL, MySQL, Oracle, and Microsoft SQL Server. Pre-built packages are available for GNU/Linux, Windows, Mac OS X, and Solaris and were tested with GNU g++, MS Visual C++, Sun CC, and Clang.

Before we go into the details of the ODB situation, let’s first get a quick overview of the state of the open source software licensing. All open source software licenses can be broadly divided into two categories, the so-called copyleft and non-copyleft licenses. Copyleft licenses give us the right to distribute copies and derivative works of software but require that the same rights be preserved in the derivative works. In other words, it is not possible to take a copyleft-licensed project, say a library, use it in our own project (thus forming a derivative work) and then release our project as closed source. In contrast, non-copyleft licenses allow us to base proprietary, closed source works on the non-copyleft software. The most popular copyleft license is the GNU General Public License (GPL). For non-copyleft, several licenses are widely used, including BSD, MIT, and the Apache License.

A quick note on the term derivative used in the preceding paragraph. Many people mistakenly assume it to mean only modifications to the original software. In reality, derivative work is a much broader concept. In our context, it is more useful to think of it as covering any functional dependency between two pieces of software. For example, if we use a GUI library in our application to display some information, then our application is a derivative work of that GUI library.

It is also useful to understand philosophical grounds of the copyleft vs non-copyleft debate. Proponents of the copyleft licenses want to preserve the freedom of every end-user to be able to distribute and modify the software. And if derivative works can be released closed source, then such a freedom would be denied. In contrast, the non-copyleft camp believes that the choice to release derivative works open or closed source is as essential a freedom as any other. There is also a third group, let’s call them the quid pro quo camp, which prefers the copyleft licenses because they ensure that those who benefit from their work are also required to give something back (i.e., make their work open source). Dual-licensing software under a copyleft license as well as a proprietary license on the commercial basis is then a natural extension of the quid pro quo idea. This way, companies that are unable or unwilling to make their derivative works open source have the option to instead pay a fee for a proprietary license. These fees can then be used to fund further development of the original project.

A copyleft project, say GPL-licensed, can normally use non-copyleft software, say BSD-licensed, without imposing any further restrictions on its users. This is not the case, however, in the other direction, for example, a BSD-licensed project using GPL-licensed software. While releasing the derivative work under BSD will satisfy the requirements of the GPL, the resulting whole now carries the restrictions of both BSD and the GPL. And that means that the users of the project no longer have the freedom to use it in proprietary derivative works.

If we are proponents of the copyleft licenses for the sake of preserving the end-user freedoms, then this outcome is exactly what we would want. However, if we use a copyleft license as a way to implement the quid pro quo principle, then this is not quite what we had in mind. After all, the author of the project is giving something back by releasing his software under an open source license. Further restricting what others can do with this software is not something that we should probably attempt.

And that’s exactly the problem that we faced with ODB. We are happy to let everyone use ODB in their projects as long as they make them open source under any open source license, copyleft or non-copyleft. However, as we have just discussed, the standard terms of the GPL make ODB really unattractive to non-copyleft projects.

So what we decided to do is to offer to grant a GPL license exception to any specific open source project that uses any of the common non-copyleft licenses (BSD, MIT, Apache License, etc). This exception would allow the project to use ODB without any of the GPL copyleft restrictions. Specifically, the users of such a project would still be able to use it in their closed source works even though the result would depend on ODB.

You may be wondering why didn’t we just grant a generic license exception that covers all the open source projects? Why do we need to issue exceptions on the project-by-project basis? The reason for this is because a simple generic exception would be easy to abuse. For example, a company wishing to use ODB in a closed source application could just package and release the generated database support code without any additional functionality as an open source project. It could then go ahead and use that code in a closed source application without any of the GPL restrictions. While it is possible to prevent such abuse using clever legal language, we found that a complex license exception text will only confuse things. Instead we decided to go with a very straightforward license exception text and to offer it to any open source project that looks legitimate.

In fact, the other day we granted our first exception. It was for the POLARIS Transportation Modelling Framework developed by Argonne National Laboratory (U.S. Department of Energy) and released under the BSD license.

So what do you need to do if you want an ODB license exception for your project? The process is actually very easy. Simply email us at info@codesynthesis.com the following information about your project:

1. Project name
2. Project URL
3. License used (e.g., BSD, MIT, etc)
4. Copyright holder(s)

Once we receive this, we will send you the license exception text that will be specific to your project. You can then add this text as a separate file to your source code distribution along with the LICENSE, COPYING, or a similar file. Or you can incorporate the exception text directly into one of these files. We also store each exception granted in our publicly-accessible ODB source code repository.