One of the major new features in the upcoming ODB 2.0.0 release is support for C++11. In this post I would like to show what is now possible when using ODB in the C++11 mode. Towards the end I will also mention some of the interesting implementation-related issues that we encountered. This would be of interest to anyone who is working on general-purpose C++ libraries or tools that have to be compatible with multiple C++ compilers as well as support both C++98 and C++11 from the same codebase.

In case you are not familiar with ODB, it is an object-relational mapping (ORM) system for C++. It allows you to persist C++ objects to a relational database without having to deal with tables, columns, or SQL, and manually writing any of the mapping code.

While the 2.0.0 release is still a few weeks out, if you would like to give the new C++11 support a try, you can use the 1.9.0.a1 pre-release.

While one could use most of the core C++11 language features with ODB even before 2.0.0, what was lacking is the integration with the new C++11 standard library components, specifically smart pointers and containers. By default, ODB still compiles in the C++98 mode, however, it is now possible to switch to the C++11 mode using the --std c++11 command line option (this is similar to GCC’s --std=c++0x). As you may remember, ODB uses GCC as a C++ compiler frontend which means ODB has arguably the best C++11 feature coverage available, especially now with the release of GCC 4.7.

Let’s start our examination of the C++11 standard library integration with smart pointers. New in C++11 are std::unique_ptr and std::shared_ptr/weak_ptr. Both of these smart pointers can now be used as object pointers:

#include <memory>
 
class employer;
 
#pragma db object pointer(std::unique_ptr)
class employee
{
  ...
 
  std::shared_ptr<employer> employer_;
};
 
#pragma db object pointer(std::shared_ptr)
class employer
{
  ...
};

ODB now also provides lazy variants for these smart pointers: odb::lazy_unique_ptr, odb::lazy_shared_ptr, and odb::lazy_weak_ptr. Here is an example:

#include <memory>
#include <vector>
 
#include <odb/lazy-ptr.hxx>
 
class employer;
 
#pragma db object pointer(std::shared_ptr)
class employee
{
  ...
 
  std::shared_ptr<employer> employer_;
};
 
#pragma db object pointer(std::shared_ptr)
class employer
{
  ...
 
  #pragma db inverse(employer_)
  std::vector<odb::lazy_weak_ptr<employee>> employees_;
};

Besides as object pointers, unique_ptr and shared_ptr/weak_ptr can also be used in data members. For example:

#include <memory>
#include <vector>
 
#pragma db object
class person
{
  ...
 
  #pragma db type("BLOB") null
  std::unique_ptr<std::vector<char>> public_key_;
};

It is unfortunate that boost::optional didn’t make it to C++11 as it would be ideal to handle the NULL semantics (boost::optional is supported by the Boost profile). The good news is that it seems there are plans to submit an std::optional proposal for TR2.

The newly supported containers are: std::array, std::forward_list, and the unordered containers. Here is an example of using std::unordered_set:

#include <string>
#include <unordered_set>
 
#pragma db object
class person
{
  ...
 
  std::unordered_set<std::string> emails_;
};

One C++11 language feature that comes really handy when dealing with query results is the range-based for-loop:

typedef odb::query<employee> query;
 
transaction t (db->begin ());
 
auto r (db->query<employee> (query::first == "John"));
 
for (employee& e: r)
  cout << e.first () << ' ' << e.last () << endl;
 
t.commit ();

So far we have tested C++11 support with various versions of GCC as well as VC++ 10 (we will also test with Clang before the final release). In fact, all the tests in our test suite build and run without any issues in the C++11 mode with these two compilers. ODB also comes with an example, called c++11, that shows support for some of the C++11 features discussed above.

These are the user-visible features when it comes to C++11 support and they are nice and neat. For those interested, here are some not so neat implementation details that I think other library authors will have to deal with if they decide to support C++11.

The first issue that we had to address is simultaneous support for C++98 and C++11. In our case, supporting both from the same codebase was not that difficult (though more on that shortly). We just had to add a number of #ifdef ODB_CXX11.

What we only realized later was that to make C++11 support practical we also had to support both from the same installation. To understand why, consider what happens when a library is packaged, say, for Ubuntu or Fedora. A single library is built and a single set of headers is packaged. To be at all usable, these packages cannot be C++98 or C++11. They have to support both at the same time. It is probably possible to have two versions of the library and ask the user to link to the correct one depending on which C++ standard they are using. But you will inevitably run into tooling limitations (e.g., pkg-config doesn’t have the --std c++11 option). The situation with headers are even worse, unless your users are prepared to pass a specific -I option depending on which C++ standard they are using.

The conclusion that we came to is this: if you want your library to be usable once installed in both C++98 and C++11 modes in a canonical way (i.e., without having to specify extra -I options, defines, or different libraries to link), then the C++11 support has to be header-only.

This has some interesting implications. For example, initially, we used an autoconf test to detect whether we are in the C++11 mode and write the appropriate value to config.h. This had to be scraped and we now use a more convoluted and less robust way of detecting the C++ standard using pre-defined compiler macros such as __cplusplus and __GXX_EXPERIMENTAL_CXX0X__. The other limitation of this decision is that all “extra” C++11 functions, such as move constructors, etc., have to be inline or templates. While these restrictions sound constraining, so far we didn’t have any serious issues maintaining C++11 support header-only. Things fitted quite naturally into this model but that, of course, may change in the future.

The other issue that we had to deal with is the different level of C++11 support provided by different compiler implementations. While GCC is more or less the gold standard in this regard, VC++ 10 lacked quite a few features that we needed, specifically, deleted functions, explicit conversion operators, and default function template arguments. As a result, we had to introduce additional macros that indicate which C++11 features are available. This felt like early C++98 days all over again. Interestingly, none of the above mentioned three features will be supported in the upcoming VC++ 11. In fact, if you look at the VC++ C++11 support table, it is quite clear that Microsoft is concentrating on the user-facing features, like the range-based for-loop. This means there will probably be some grief for some time for library writers.

I think everyone had enough of rvalue references for now so let’s look at another interesting C++11 technique: delayed function signature instantiation. It is made possible thanks to the default function template arguments.

To understand the motivation behind this technique, let’s first review the various stages of instantiation of a class template. At the first stage all we get is just the template-id. Here is an example:

template <typename T>
class foo;
 
class bar;
 
typedef foo<bar> foo_bar;

At this stage both the template and its type arguments only need to be forward-declared and the resulting template-id can be used in places where the size of a class nor its members need to be known. For example, to form a pointer or a reference:

foo_bar* p = 0;     // ok
void f (foo<bar>&); // ok
foo_bar x;          // error: need size
p->f ();            // error: foo<bar>::f is unknown

In other words, this is the same as forward-declaration for non-template classes.

The last two lines in the above example wouldn’t have been errors if we had defined the foo class template. Instead, it would have triggered the second instantiation stage during which the class definition (i.e., its body) is instantiated. In particular, this includes instantiation of all data members and member function signatures. However, this stage does not involve instantiation of member function bodies. This only happens at the third stage, when we actually use (e.g., call or take a pointer to) specific functions. Here is another example that illustrates all the stages together:

template <typename T>
class foo
{
public:
  void f (T* p)
  {
    delete p_;
    p_ = p;
  }
 
  T* p_;
};
 
class bar;
 
void f (foo<bar>&); // stage 1
foo<bar> x;         // stage 2
x.f ();             // stage 3

While the class template definition is required for the second stage and the function definition is required for the third stage, whether the type template arguments must be defined at any of these stages depends on the template implementation. For example, the foo class template above does not require the template argument to be defined during the second stage but does require it to be defined during the third stage when f()’s body is instantiated.

Probably the best known example of class templates that don’t require the template argument to be defined during the second stage are smart pointers. This is because, like with raw pointers, we often need to form smart pointers to forward-declared types:

class bar;
 
bar* create ();                 // ok
std::shared_ptr<bar> create (); // ok

It is fairly straightforward to implement normal smart pointers like std::shared_ptr in such a way as to not require the template argument to be defined. But here is a problem that I ran into when implementing a special kind of smart pointer in ODB, called a lazy pointer. If you read some of my previous posts you probably remember what a lazy pointer is (it turned out to be a very fertile ground for discovering interesting C++11 techniques). For those new to the idea, here is quick recap: when an object that contains lazy pointers to other objects is loaded from the database, these other objects are not loaded right away (which would be the case for normal, eager pointers such as std::shared_ptr). Instead, just the object ids are loaded and the objects themselves can be loaded later, when and if required.

A lazy pointer can be initialized with an actual pointer to a persistent object, in which case the pointer is said to be loaded. Or we can initialize it with an object id, in which case the pointer is unloaded.

When I first set out to implement a lazy pointer, I naturally added the following extra constructor to support creating unloaded pointers (in reality id_type is not defined by T but rather by odb::object_traits<T>; however this difference is not material to the discussion):

template <class T>
class lazy_shared_ptr
{
  lazy_shared_ptr (database&, const typename T::id_type&);
 
  ...
};

Do you see the problem? Remember that during the second stage function signatures get instantiated. And in order to instantiate the signature of the above constructor, the template argument must be defined, since we are looking for id_type inside this type. As a result, lazy_shared_ptr can no longer be used with forward-declared classes.

As it turns out, we can delay function signature instantiation until the third stage (i.e., when the function is actually used) by making the function itself a template. Here is how we can fix the above constructor so that we can continue using lazy_shared_ptr with forward-declared types. This method works even in C++98:

template <class T>
class lazy_shared_ptr
{
  ...
 
  template <typename ID>
  lazy_shared_ptr (database&, const ID&);
};

As a side note, some of you who read my previous posts about rvalue references were wondering why I used the constructor template here. Well, now you know.

The above C++98-compatible implementation has a number of drawbacks. The biggest is that we cannot use this technique for function return types. In ODB, lazy pointers also allow querying the object id of a stored object. In the C++98 mode, to keep the implementation usable on forward-declared types, I had to resort to this ugly interface:

template <class T>
class lazy_shared_ptr
{
  ...
 
  template <typename T1>
  typename T1::id_type object_id () const;
};
 
lazy_shared_ptr<object> lp = ...
cerr << lp->object_id<object> (); << endl;

That is, the user has to explicitly specify the object type when calling object_id().

The second problem has to do with the looseness of the resulting interface. Now we can pass any value as id when initializing lazy_shared_ptr. While an incompatible type will get caught, it will only happen in the implementation with the resulting diagnostics pointing to the wrong place and saying the wrong thing (we have to provide our own correct “diagnostics” in the comment):

template <class T>
class lazy_shared_ptr
{
  ...
 
  template <typename ID>
  lazy_shared_ptr (database&, const ID& id)
  {
    // Compiler error pointing here? Perhaps the id
    // argument is wrong?
    //
    const typename T::id_type& real_id (id);
    ...
  }
};

Support for default function template arguments in C++11 allows us to resolve both of these problems. Let’s start with the return type:

template <class T>
class lazy_shared_ptr
{
  ...
 
  template <typename T1 = T>
  typename T1::id_type object_id () const;
};
 
lazy_shared_ptr<object> lp = ...
cerr << lp->object_id (); << endl;

The solution to the second problem is equally simple:

template <class T>
class lazy_shared_ptr
{
  ...
 
  template <typename T1 = T>
  lazy_shared_ptr (database&, const typename T1::id_type&);
};

The idea here is to inhibit template argument deduction in order to force the default type to always be used. This is similar to the trick used in std::forward().

My original post about rvalue reference pitfalls from last week was followed by quite a few comments, including some interesting suggestions that are worth discussing.

While most of the discussion centered around the second problem, Jonathan Rogers pointed out the following interesting observation. Consider again the lazy_shared_ptr constructor from the original article that takes shared_ptr:

template <class T>
class lazy_shared_ptr
{
  ...
 
  template <class T1>
  lazy_shared_ptr (database&, const std::shared_ptr<T1>& p)
    : p_ (p)
  {
  }
};

If we want to support efficient initialization (shall we call it move initialization?), then it seems natural to add an rvalue reference overload:

template <class T>
class lazy_shared_ptr
{
  ...
 
  template <class T1>
  lazy_shared_ptr (database&, const std::shared_ptr<T1>& p)
    : p_ (p)
  {
  }
 
  template <class T1>
  lazy_shared_ptr (database&, std::shared_ptr<T1>&& p)
    : p_ (std::move (p))
  {
  }
};

While this works, as Jonathan pointed out, another alternative to provide the same functionality is to just have a single constructor that takes its argument by value:

template <class T>
class lazy_shared_ptr
{
  ...
 
  template <class T1>
  lazy_shared_ptr (database&, std::shared_ptr<T1> p)
    : p_ (std::move (p))
  {
  }
};

Let’s consider what happens when we use both versions to initialize lazy_shared_ptr with lvalues and rvalues. When we use the original implementation with an lvalue, the first constructor (the one taking const lvalue reference) is selected. The value is then copied to p_ using shared_ptr’s copy constructor. Using the second implementation with an lvalue causes that copy constructor to be called right away to create the temporary. This temporary is then passed to the lazy_shared_ptr constructor where it is moved to p_. So in this case the second implementation requires an extra move constructor call.

Let’s now pass an rvalue. In the first implementation the second constructor is selected and the value is passed as an rvalue reference. It is then moved to p_. When the second implementation is used, a temporary is again created but this time using a move instead of a copy constructor. The temporary is then moved to p_. In this case, again, the second implementation requires an extra move constructor call.

Considering that move constructors are normally very cheap, this makes for a good way to keep your code short and concise. But the real advantage of this approach becomes apparent when we have multiple arguments that we want to pass efficiently (this was also a topic of Sumant’s post from a few days ago). If we use the rvalue reference approach, then for n arguments we will need 2^n constructor versions.

Note, however, that the pass by value approach is only a good idea if you know for sure that the argument type provides a move constructor. If that’s not the case, then this approach will perform significantly worse compared to the rvalue reference version. This is the reason, for example, why it is not a good idea to use this technique in std::vector’s push_back().

Ok, let’s now turn to the problem that triggered a lot of comments and suggestions. Here is a quick recap. We have two constructors like these:

template <class T>
class lazy_shared_ptr
{
  ...
 
  template <class ID>
  lazy_shared_ptr (database&, const ID&);
 
  template <class T1>
  lazy_shared_ptr (database&, const std::shared_ptr<T1>&);
};

One initializes a lazy pointer using an object id, creating an unloaded pointer. The other initializes it with the pointer to the actual object, creating a loaded pointer.

Now we want to add move initialization overloads for these two constructors. As it turns out, the straightforward approach doesn’t quite work:

template <class T>
class lazy_shared_ptr
{
  ...
 
  template <class ID>
  lazy_shared_ptr (database&, const ID&);
 
  template <class ID>
  lazy_shared_ptr (database&, ID&&);
 
  template <class T1>
  lazy_shared_ptr (database&, const std::shared_ptr<T1>&);
 
  template <class T1>
  lazy_shared_ptr (database&, std::shared_ptr<T1>&&);
};

As you may recall, we have two problems here. The first manifests itself when we try to initialize lazy_shared_ptr with an lvalue of the shared_ptr type:

shared_ptr<object> p = ...;
lazy_shared_ptr<object> lp (db, p);

Instead of selecting the third constructor, the overload resolution rules select the second because the rvalue reference in its second argument becomes an lvalue reference (see the original post for details on why this happens).

The second problem occurs when we try to initialize a lazy pointer with an object id that is again an lvalue. For example:

string id = ...
lazy_shared_ptr<object> lp (db, id);

In this case, instead of selecting the first constructor, the overload resolution again selects the second constructor which is again transformed to a version that has an lvalue instead of an rvalue reference for its second argument. If you are wondering why this is a problem (after all, the first two constructors accomplish essentially the same), consider that while the first constructor’s hypothetical implementation will use a copy constructor to initialize the id, the second constructor will most likely use the move constructor to accomplish the same. Which means that the state of our lvalue will be transferred without us explicitly asking for it, as we normally do with the std::move() call.

As Thomas noted in the comments and as I should have mentioned explicitly in the original post, the C++ mechanisms that are causing problems here are exactly the same ones that allow for perfect argument forwarding. In fact, rvalue references are primarily used to implement two related but also quite distinct things: the move semantics and perfect forwarding. What happens here is that we are trying to implement the move semantics but are getting perfect forwarding instead. To paraphrase the conclusion of my original post, any time you write a function like this:

template <typename T>
void f (T&&);

You always get perfect forwarding and never move semantics.

If we look closely at the two problematic cases above, we will notice that they both happen when the template argument is an lvalue reference which results in our rvalue reference becoming lvalue. When we pass an rvalue, everything works great. In fact, we never want our move initialization constructor to be called for lvalues since we have other overloads (const lvalue reference) taking care of these cases. So what we want is to disable the move initialization constructor when the template argument is an lvalue reference. As it turns out, this is not that difficult in C++11:

  template <class ID,
            typename std::enable_if<
              !std::is_lvalue_reference<ID>::value,
              int>::type = 0>
  lazy_shared_ptr (database&, ID&&)

To put this in more general terms, it is possible to reduce perfect forwarding back to just move semantics by disabling a function for template arguments that are lvalue references.