[odb-users] persisting objects increases long-term memory usage
(with sqlite)
jose miguel mut
jmmut at ebi.ac.uk
Fri Oct 28 04:42:42 EDT 2016
Hello,
We are using ODB 2.4.0 with sqlite (with great delight) but have noticed
that the memory used increases with time. Is this expected? Or maybe we
are not using the ODB API properly? For long executions (up to 10^10
instances), a constant memory usage would be ideal. The increase seems
linear and is noticeable even in short executions.
To clear any doubt, I am not talking about memory leaks, those are well
handled as valgrind shows (those 72,704 bytes seem to be about an
unrelated issue with c++ stdlib):
==892== LEAK SUMMARY:
==892== definitely lost: 0 bytes in 0 blocks
==892== indirectly lost: 0 bytes in 0 blocks
==892== possibly lost: 0 bytes in 0 blocks
==892== still reachable: 72,704 bytes in 1 blocks
==892== suppressed: 0 bytes in 0 blocks
I am talking about memory usage, I attached the most important part of
the output of a `valgrind --tool=massif` analysis. You can see there how
the 80% percent of the final usage is allocated by `sqlite_step`, called
by `odb::sqlite::insert_statement::execute() (statement.cxx:649)`. In
this execution, no reading was done, only writing.
I think that ODB might be caching some statements, so I would like to
know if it's possible to not cache anything.
The manual says that sessions may cache some things, but we are not
using sessions.
We are using transactions, and flushing every 1000 elements (Error is
the class we are persisting):
```
void OdbReportRW::write(Error &error)
{
if (current_transaction_size == 0) {
// start transaction
transaction.reset(db->begin());
}
db->persist(error);
++current_transaction_size;
if (current_transaction_size == transaction_size) {
// commit transaction
flush();
current_transaction_size = 0;
}
}
void OdbReportRW::flush()
{
if (transaction.has_current()) {
transaction.commit();
}
}
```
So, wrapping up, is it possible to have a constant memory usage when
persisting objects?
Regards,
Jose
-------------- next part --------------
--------------------------------------------------------------------------------
Command: ./builds/debug/bin/vcf_validator -v v4.1 -i ../../Documents/biofiles/European_Holstein.NewZealand_population.10000.vcf -r stdout,database -o /tmp
Massif arguments: --time-unit=B
ms_print arguments: massif.out.623
--------------------------------------------------------------------------------
MB
2.319^ #
| ::@@#
| @@@@@: @@#
| ::@@@ @@@: @@#
| @@@::: @ @ @@@: @@#
| ::::@ @: : @ @ @@@: @@#
| @@:@@:: :@ @: : @ @ @@@: @@#
| @@@@@@ :@ :: :@ @: : @ @ @@@: @@#
| ::@@@@@@ @ :@ :: :@ @: : @ @ @@@: @@#
| @@@: @@@@@@ @ :@ :: :@ @: : @ @ @@@: @@#
| ::@@@:@ @: @@@@@@ @ :@ :: :@ @: : @ @ @@@: @@#
| :@@@: @ @:@ @: @@@@@@ @ :@ :: :@ @: : @ @ @@@: @@#
| ::::::@@ : @ @:@ @: @@@@@@ @ :@ :: :@ @: : @ @ @@@: @@#
| :::::@@::: ::@@ : @ @:@ @: @@@@@@ @ :@ :: :@ @: : @ @ @@@: @@#
| @::: :@ ::: ::@@ : @ @:@ @: @@@@@@ @ :@ :: :@ @: : @ @ @@@: @@#
| @::@::: :@ ::: ::@@ : @ @:@ @: @@@@@@ @ :@ :: :@ @: : @ @ @@@: @@#
| :@@@: @::: :@ ::: ::@@ : @ @:@ @: @@@@@@ @ :@ :: :@ @: : @ @ @@@: @@#
| @:@ @: @::: :@ ::: ::@@ : @ @:@ @: @@@@@@ @ :@ :: :@ @: : @ @ @@@: @@#
| @:@ @: @::: :@ ::: ::@@ : @ @:@ @: @@@@@@ @ :@ :: :@ @: : @ @ @@@: @@#
| @:@ @: @::: :@ ::: ::@@ : @ @:@ @: @@@@@@ @ :@ :: :@ @: : @ @ @@@: @@#
0 +----------------------------------------------------------------------->GB
0 9.917
Number of snapshots: 53
Detailed snapshots: [1, 3, 4, 6, 11, 17, 18, 20, 21, 23, 24, 26, 27, 28, 29, 30, 31, 32, 34, 38, 39, 42, 43, 44, 45, 46, 48, 50, 52 (peak)]
--------------------------------------------------------------------------------
n time(b) total(b) useful-heap(b) extra-heap(b) stacks(b)
--------------------------------------------------------------------------------
51 10,613,896,096 2,419,152 2,320,601 98,551 0
52 10,648,454,520 2,431,464 2,332,669 98,795 0
95.94% (2,332,669b) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
->80.96% (1,968,584b) 0x71d1dc5: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6)
| ->80.96% (1,968,584b) 0x71ac568: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6)
| ->80.96% (1,968,584b) 0x71b4c46: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6)
| | ->76.21% (1,852,944b) 0x71b8899: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6)
| | | ->76.12% (1,850,880b) 0x71b8af0: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6)
| | | | ->76.12% (1,850,880b) 0x71b2367: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6)
| | | | ->76.12% (1,850,880b) 0x71de5d4: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6)
| | | | ->76.12% (1,850,880b) 0x71de88b: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6)
| | | | ->76.07% (1,849,600b) 0x71e18ea: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6)
| | | | | ->73.96% (1,798,400b) 0x71e55d1: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6)
| | | | | | ->73.96% (1,798,400b) 0x71e66f7: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6)
| | | | | | ->73.96% (1,798,400b) 0x720f460: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6)
| | | | | | ->73.96% (1,798,400b) 0x7212f85: sqlite3_step (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6)
| | | | | | ->73.96% (1,798,400b) 0x59f8a88: odb::sqlite::insert_statement::execute() (statement.cxx:649)
| | | | | | ->65.54% (1,593,600b) 0x6fe3db: odb::access::object_traits_impl<ebi::vcf::error, (odb::database_id)1>::persist(odb::database&, ebi::vcf::error&, bool, bool) (error-odb.cpp:538)
| | | | | | | ->43.69% (1,062,400b) 0x705c1f: odb::access::object_traits_impl<ebi::vcf::nometadefinitionerror, (odb::database_id)1>::persist(odb::database&, ebi::vcf::nometadefinitionerror&, bool, bool) (error-odb.cpp:3558)
| | | | | | | | ->43.69% (1,062,400b) 0x73542a: odb::dispatch_persist<ebi::vcf::nometadefinitionerror, (odb::database_id)1, true>::call(odb::database&, ebi::vcf::nometadefinitionerror const&) (polymorphic-map.hxx:146)
| | | | | | | | ->43.69% (1,062,400b) 0x726c7c: bool odb::dispatch_impl<ebi::vcf::nometadefinitionerror, (odb::database_id)1>(odb::polymorphic_concrete_info<odb::object_traits<ebi::vcf::nometadefinitionerror>::root_type>::call_type, odb::database&, odb::object_traits<ebi::vcf::nometadefinitionerror>::root_type const*, void const*) (polymorphic-map.hxx:180)
| | | | | | | | ->43.69% (1,062,400b) 0x6fe1f6: odb::access::object_traits_impl<ebi::vcf::error, (odb::database_id)1>::persist(odb::database&, ebi::vcf::error&, bool, bool) (error-odb.cpp:495)
| | | | | | | | ->43.69% (1,062,400b) 0x5358a2: odb::object_traits<ebi::vcf::error>::id_type odb::database::persist_<ebi::vcf::error, (odb::database_id)5>(ebi::vcf::error&) (database.txx:38)
| | | | | | | | ->43.69% (1,062,400b) 0x534d03: odb::object_traits<ebi::vcf::error>::id_type odb::database::persist<ebi::vcf::error>(ebi::vcf::error&) (database.ixx:160)
| | | | | | | | ->43.69% (1,062,400b) 0x5335ce: ebi::vcf::odbreportrw::write(ebi::vcf::error&) (odb_report.cpp:98)
| | | | | | | | ->43.69% (1,062,400b) 0x533556: ebi::vcf::odbreportrw::write_warning(ebi::vcf::error&) (odb_report.cpp:88)
| | | | | | | | ->43.69% (1,062,400b) 0x5380e4: ebi::vcf::is_valid_vcf_file(std::istream&, ebi::vcf::parser&, std::vector<std::unique_ptr<ebi::vcf::reportwriter, std::default_delete<ebi::vcf::reportwriter> >, std::allocator<std::unique_ptr<ebi::vcf::reportwriter, std::default_delete<ebi::vcf::reportwriter> > > >&) (validator.cpp:142)
| | | | | | | | ->43.69% (1,062,400b) 0x5203d9: main (main.cpp:242)
| | | | | | | |
| | | | | | | ->21.85% (531,200b) 0x703e0d: odb::access::object_traits_impl<ebi::vcf::bodysectionerror, (odb::database_id)1>::persist(odb::database&, ebi::vcf::bodysectionerror&, bool, bool) (error-odb.cpp:2764)
| | | | | | | ->21.85% (531,200b) 0x71a3fb: odb::access::object_traits_impl<ebi::vcf::samplesfieldbodyerror, (odb::database_id)1>::persist(odb::database&, ebi::vcf::samplesfieldbodyerror&, bool, bool) (error-odb.cpp:11883)
| | | | | | | ->21.85% (531,200b) 0x73a330: odb::dispatch_persist<ebi::vcf::samplesfieldbodyerror, (odb::database_id)1, true>::call(odb::database&, ebi::vcf::samplesfieldbodyerror const&) (polymorphic-map.hxx:146)
| | | | | | | ->21.85% (531,200b) 0x730886: bool odb::dispatch_impl<ebi::vcf::samplesfieldbodyerror, (odb::database_id)1>(odb::polymorphic_concrete_info<odb::object_traits<ebi::vcf::samplesfieldbodyerror>::root_type>::call_type, odb::database&, odb::object_traits<ebi::vcf::samplesfieldbodyerror>::root_type const*, void const*) (polymorphic-map.hxx:180)
| | | | | | | ->21.85% (531,200b) 0x6fe1f6: odb::access::object_traits_impl<ebi::vcf::error, (odb::database_id)1>::persist(odb::database&, ebi::vcf::error&, bool, bool) (error-odb.cpp:495)
| | | | | | | ->21.85% (531,200b) 0x5358a2: odb::object_traits<ebi::vcf::error>::id_type odb::database::persist_<ebi::vcf::error, (odb::database_id)5>(ebi::vcf::error&) (database.txx:38)
| | | | | | | ->21.85% (531,200b) 0x534d03: odb::object_traits<ebi::vcf::error>::id_type odb::database::persist<ebi::vcf::error>(ebi::vcf::error&) (database.ixx:160)
| | | | | | | ->21.85% (531,200b) 0x5335ce: ebi::vcf::odbreportrw::write(ebi::vcf::error&) (odb_report.cpp:98)
| | | | | | | ->21.85% (531,200b) 0x533526: ebi::vcf::odbreportrw::write_error(ebi::vcf::error&) (odb_report.cpp:83)
| | | | | | | ->21.85% (531,200b) 0x537fbb: ebi::vcf::is_valid_vcf_file(std::istream&, ebi::vcf::parser&, std::vector<std::unique_ptr<ebi::vcf::reportwriter, std::default_delete<ebi::vcf::reportwriter> >, std::allocator<std::unique_ptr<ebi::vcf::reportwriter, std::default_delete<ebi::vcf::reportwriter> > > >&) (validator.cpp:137)
| | | | | | | ->21.85% (531,200b) 0x5203d9: main (main.cpp:242)
| | | | | | |
| | | | | | ->05.26% (128,000b) 0x705d54: odb::access::object_traits_impl<ebi::vcf::nometadefinitionerror, (odb::database_id)1>::persist(odb::database&, ebi::vcf::nometadefinitionerror&, bool, bool) (error-odb.cpp:3578)
| | | | | | | ->05.26% (128,000b) 0x73542a: odb::dispatch_persist<ebi::vcf::nometadefinitionerror, (odb::database_id)1, true>::call(odb::database&, ebi::vcf::nometadefinitionerror const&) (polymorphic-map.hxx:146)
| | | | | | | ->05.26% (128,000b) 0x726c7c: bool odb::dispatch_impl<ebi::vcf::nometadefinitionerror, (odb::database_id)1>(odb::polymorphic_concrete_info<odb::object_traits<ebi::vcf::nometadefinitionerror>::root_type>::call_type, odb::database&, odb::object_traits<ebi::vcf::nometadefinitionerror>::root_type const*, void const*) (polymorphic-map.hxx:180)
| | | | | | | ->05.26% (128,000b) 0x6fe1f6: odb::access::object_traits_impl<ebi::vcf::error, (odb::database_id)1>::persist(odb::database&, ebi::vcf::error&, bool, bool) (error-odb.cpp:495)
| | | | | | | ->05.26% (128,000b) 0x5358a2: odb::object_traits<ebi::vcf::error>::id_type odb::database::persist_<ebi::vcf::error, (odb::database_id)5>(ebi::vcf::error&) (database.txx:38)
| | | | | | | ->05.26% (128,000b) 0x534d03: odb::object_traits<ebi::vcf::error>::id_type odb::database::persist<ebi::vcf::error>(ebi::vcf::error&) (database.ixx:160)
| | | | | | | ->05.26% (128,000b) 0x5335ce: ebi::vcf::odbreportrw::write(ebi::vcf::error&) (odb_report.cpp:98)
| | | | | | | ->05.26% (128,000b) 0x533556: ebi::vcf::odbreportrw::write_warning(ebi::vcf::error&) (odb_report.cpp:88)
| | | | | | | ->05.26% (128,000b) 0x5380e4: ebi::vcf::is_valid_vcf_file(std::istream&, ebi::vcf::parser&, std::vector<std::unique_ptr<ebi::vcf::reportwriter, std::default_delete<ebi::vcf::reportwriter> >, std::allocator<std::unique_ptr<ebi::vcf::reportwriter, std::default_delete<ebi::vcf::reportwriter> > > >&) (validator.cpp:142)
| | | | | | | ->05.26% (128,000b) 0x5203d9: main (main.cpp:242)
| | | | | | |
| | | | | | ->02.00% (48,640b) 0x71a530: odb::access::object_traits_impl<ebi::vcf::samplesfieldbodyerror, (odb::database_id)1>::persist(odb::database&, ebi::vcf::samplesfieldbodyerror&, bool, bool) (error-odb.cpp:11903)
| | | | | | | ->02.00% (48,640b) 0x73a330: odb::dispatch_persist<ebi::vcf::samplesfieldbodyerror, (odb::database_id)1, true>::call(odb::database&, ebi::vcf::samplesfieldbodyerror const&) (polymorphic-map.hxx:146)
| | | | | | | ->02.00% (48,640b) 0x730886: bool odb::dispatch_impl<ebi::vcf::samplesfieldbodyerror, (odb::database_id)1>(odb::polymorphic_concrete_info<odb::object_traits<ebi::vcf::samplesfieldbodyerror>::root_type>::call_type, odb::database&, odb::object_traits<ebi::vcf::samplesfieldbodyerror>::root_type const*, void const*) (polymorphic-map.hxx:180)
| | | | | | | ->02.00% (48,640b) 0x6fe1f6: odb::access::object_traits_impl<ebi::vcf::error, (odb::database_id)1>::persist(odb::database&, ebi::vcf::error&, bool, bool) (error-odb.cpp:495)
| | | | | | | ->02.00% (48,640b) 0x5358a2: odb::object_traits<ebi::vcf::error>::id_type odb::database::persist_<ebi::vcf::error, (odb::database_id)5>(ebi::vcf::error&) (database.txx:38)
| | | | | | | ->02.00% (48,640b) 0x534d03: odb::object_traits<ebi::vcf::error>::id_type odb::database::persist<ebi::vcf::error>(ebi::vcf::error&) (database.ixx:160)
| | | | | | | ->02.00% (48,640b) 0x5335ce: ebi::vcf::odbreportrw::write(ebi::vcf::error&) (odb_report.cpp:98)
| | | | | | | ->02.00% (48,640b) 0x533526: ebi::vcf::odbreportrw::write_error(ebi::vcf::error&) (odb_report.cpp:83)
| | | | | | | ->02.00% (48,640b) 0x537fbb: ebi::vcf::is_valid_vcf_file(std::istream&, ebi::vcf::parser&, std::vector<std::unique_ptr<ebi::vcf::reportwriter, std::default_delete<ebi::vcf::reportwriter> >, std::allocator<std::unique_ptr<ebi::vcf::reportwriter, std::default_delete<ebi::vcf::reportwriter> > > >&) (validator.cpp:137)
| | | | | | | ->02.00% (48,640b) 0x5203d9: main (main.cpp:242)
| | | | | | |
| | | | | | ->01.16% (28,160b) 0x703f42: odb::access::object_traits_impl<ebi::vcf::bodysectionerror, (odb::database_id)1>::persist(odb::database&, ebi::vcf::bodysectionerror&, bool, bool) (error-odb.cpp:2784)
| | | | | | ->01.16% (28,160b) 0x71a3fb: odb::access::object_traits_impl<ebi::vcf::samplesfieldbodyerror, (odb::database_id)1>::persist(odb::database&, ebi::vcf::samplesfieldbodyerror&, bool, bool) (error-odb.cpp:11883)
| | | | | | ->01.16% (28,160b) 0x73a330: odb::dispatch_persist<ebi::vcf::samplesfieldbodyerror, (odb::database_id)1, true>::call(odb::database&, ebi::vcf::samplesfieldbodyerror const&) (polymorphic-map.hxx:146)
| | | | | | ->01.16% (28,160b) 0x730886: bool odb::dispatch_impl<ebi::vcf::samplesfieldbodyerror, (odb::database_id)1>(odb::polymorphic_concrete_info<odb::object_traits<ebi::vcf::samplesfieldbodyerror>::root_type>::call_type, odb::database&, odb::object_traits<ebi::vcf::samplesfieldbodyerror>::root_type const*, void const*) (polymorphic-map.hxx:180)
| | | | | | ->01.16% (28,160b) 0x6fe1f6: odb::access::object_traits_impl<ebi::vcf::error, (odb::database_id)1>::persist(odb::database&, ebi::vcf::error&, bool, bool) (error-odb.cpp:495)
| | | | | | ->01.16% (28,160b) 0x5358a2: odb::object_traits<ebi::vcf::error>::id_type odb::database::persist_<ebi::vcf::error, (odb::database_id)5>(ebi::vcf::error&) (database.txx:38)
| | | | | | ->01.16% (28,160b) 0x534d03: odb::object_traits<ebi::vcf::error>::id_type odb::database::persist<ebi::vcf::error>(ebi::vcf::error&) (database.ixx:160)
| | | | | | ->01.16% (28,160b) 0x5335ce: ebi::vcf::odbreportrw::write(ebi::vcf::error&) (odb_report.cpp:98)
| | | | | | ->01.16% (28,160b) 0x533526: ebi::vcf::odbreportrw::write_error(ebi::vcf::error&) (odb_report.cpp:83)
| | | | | | ->01.16% (28,160b) 0x537fbb: ebi::vcf::is_valid_vcf_file(std::istream&, ebi::vcf::parser&, std::vector<std::unique_ptr<ebi::vcf::reportwriter, std::default_delete<ebi::vcf::reportwriter> >, std::allocator<std::unique_ptr<ebi::vcf::reportwriter, std::default_delete<ebi::vcf::reportwriter> > > >&) (validator.cpp:137)
| | | | | | ->01.16% (28,160b) 0x5203d9: main (main.cpp:242)
| | | | | |
| | | | | ->01.05% (25,600b) 0x71e6d50: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6)
| | | | | | ->01.05% (25,600b) 0x720d668: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6)
| | | | | | ->01.05% (25,600b) 0x7212f85: sqlite3_step (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6)
| | | | | | ->01.05% (25,600b) 0x59f82a0: odb::sqlite::generic_statement::execute() (statement.cxx:390)
| | | | | | ->01.05% (25,600b) 0x59ec832: odb::sqlite::connection::execute(char const*, unsigned long) (connection.cxx:146)
| | | | | | ->01.05% (25,600b) 0x72291c: odb::database::execute(char const*) (database.ixx:869)
| | | | | | ->01.05% (25,600b) in 19 places, all below massif's threshold (01.00%)
| | | | | |
| | | | | ->01.05% (25,600b) in 2 places, all below massif's threshold (01.00%)
| | | | |
| | | | ->00.05% (1,280b) in 1+ places, all below ms_print's threshold (01.00%)
| | | |
| | | ->00.08% (2,064b) in 1+ places, all below ms_print's threshold (01.00%)
| | |
| | ->02.63% (64,008b) 0x71be910: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6)
| | | ->02.63% (64,008b) 0x7232617: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6)
| | | ->02.63% (64,008b) 0x59ed1ae: odb::sqlite::connection::connection(odb::sqlite::database&, int) (connection.cxx:62)
| | | ->02.63% (64,008b) 0x59ee437: odb::sqlite::connection_pool_factory::pooled_connection::pooled_connection(odb::sqlite::database&, int) (connection-factory.cxx:245)
| | | ->02.63% (64,008b) 0x59ee4a7: odb::sqlite::connection_pool_factory::create() (connection-factory.cxx:137)
| | | ->02.63% (64,008b) 0x59ee0ca: odb::sqlite::connection_pool_factory::connect() (connection-factory.cxx:177)
| | | ->02.63% (64,008b) 0x59ef1c1: odb::sqlite::database::connection_() (database.cxx:182)
| | | ->02.63% (64,008b) 0x534243: odb::database::connection() (database.ixx:127)
| | | ->02.63% (64,008b) 0x532f99: ebi::vcf::odbreportrw::odbreportrw(std::string const&) (odb_report.cpp:44)
| | | ->02.63% (64,008b) 0x51fb97: (anonymous namespace)::get_outputs(std::string const&, std::string const&) (main.cpp:198)
| | | ->02.63% (64,008b) 0x520280: main (main.cpp:231)
| | |
| | ->01.22% (29,640b) 0x71b4d64: ??? (in /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6)
| | | ->01.22% (29,640b) in 17 places, all below massif's threshold (01.00%)
| | |
| | ->00.90% (21,992b) in 1+ places, all below ms_print's threshold (01.00%)
| |
| ->00.00% (0b) in 1+ places, all below ms_print's threshold (01.00%)
More information about the odb-users
mailing list