[odb-users] Question regarding odb::timeout
cezanmeyen at gmail.com
Thu Apr 2 08:52:42 EDT 2020
I’m using ODB with SQLite(in tmpfs) for multi-process data storage and
exchange. It is working fine but I’ve encountered a lot of odb::timeout
exceptions when multiple processes are writing at the same time. My initial
solution was simply retrying every time it threw the exception(as the ODB
doc states). This resulted in a lot of retries per persist and a
significant performance impact. My second and current solution is using an
inter-process mutex on the persist actions. This will ensure that there is
only a single process is writing data at a given time, so no timeout
exceptions are thrown. This performed better when compared to the first
solution and seemed a good solution at the time. It now however poses a big
problem when scaling up with more and more concurrent writers. Since the
file is locked the wait times increase with the amount of persist processes.
I’ve run tests with MySQL and PostgreSQL as they are made to handle
multi-process data exchange better. But they both performed worse than my
SQLite + mutex solution. In these tests they were both mounted in tmpfs
since their in-memory tables weren’t usable for out use-case.
My questions are:
- When exactly does the odb::timeout exception get thrown?
- Is there a way to setup odb and sqlite to handle multi-process
writers more consistent?
o I am already working with WAL
- Does ODB use the locks provided by the sqlite3 API?
o If not is it possible to interact with these locks through the ODB API
or should this be done natively?
A more general question:
- Is there a different way to deal with this congestion problem,
that you know of?
More information about the odb-users