[odb-users] Performance issues for large sets

[Quentin Deldycke]

Hi Boris,

--
Deldycke Quentin


On 20 September 2014 10:30, Boris Kolpackov <boris at codesynthesis.com> wrote:

> Hi Quentin,
>
> Quentin Deldycke <quentindeldycke at gmail.com> writes:
>
> > > > Our 'old' system used C-ISAM. It can perform a "while" of 1000
> elements
> > > > in 0.003s - 0.007s.
> > > > The same request with odb iterate this 1000 elements in 1.3s - 2.6s.
> > > >
> > > > The use of odb views allows me to perform this iteration in 0.01s -
> > > > 0.03s.
> > >
> > > Was the 'old', C-ISAM system using the same MySQL database that you
> > > are using with ODB?
> > >
> >
> > No actually it is a different system. Really old school one, but the data
> > we are treating are exactly the same beetween both.
>
> Well, if it was a different database, then the above comparison is apples
> to oranges. The difference you observe between ODB views and C-ISAM is
> because of the different underlying databases, not APIs. I am sure if
> you used, say, SQLite instead of MySQL for ODB tests you would get at
> least an order of magnitude better performance than C-ISAM.
>
> You are right. Sadly it is not of my own choice. It needs to be
a MySQL server as
we want to unify all our database system. Sadly ODB is used, for the moment,
as a layer in a more complex (old and ugly) ORM from user point of view...
Anyway, where performance is really bad, I will use real ODB code, and not
the
"compatibility" API i try to provide for existing code. And i know that at
this moment,
I will achieve better performance and functionallity.

By the way for my point of view, it is still interesting
to see the difference of performance ODB could acquire with different
databases systems.
If i have time, i will try to make some testing to see the difference of
performance depending
of the underlying database. and give you some results. It can always be
interesting....


> > I think that lazy pointers are a good option for me. I will just modify
> > the way of working so it lazy load when the '->' operator gets used. So
> > my users won't need to change their code.
>
> Just keep in mind that when you load a lazy pointer, it has to happen
> within a transaction. That's why the ODB lazy pointers don't provide
> such implicit loading.
>

Yes, i made some tries, and it seems to work nearly without problems. I
subclassed your
lazy_shared_ptr class to provide the functionnality i wanted. I only had a
weird compilation
problem. (I will verify if it is not my side and, if not, i will send a
mail about).

For transactions, my first tries show me that it is ok.

FYI: We have improved from 1.3-2.6s to 0.3-0.5s by 1000 iteration.

I still have 1 request going each iteration because i have a std::map
inside my object.
I will try to lazy-load it also. I am sure i can still reduce iteration
time!
If the user want to access more deep objects, It is not a problem for me
that it
perform a "select" at this moment.

Anyway, thanks for your help! The performance issue I have
for large sets going to be "a past story"!


>
> Boris
>