Clusterer module and memory leak!

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Clusterer module and memory leak!

SamyGo
Hi,

I'm experimenting with the clusterer module and in-config cluster_send/receive requests. On my test servers I start seeing the following errors. 

ERROR:core:qm_malloc: not enough free pkg memory (784 bytes left, need 65536), please increase the "-M" command line parameter!
ERROR:core:bin_init: No more pkg memory!
ERROR:clusterer:prep_gen_msg: Failed to init bin send buffer
ERROR:clusterer:bcast_gen_msg: Failed to build generic clusterer message

My opensips process has params -m 256 and -M 64 

version: opensips 2.4.0-dev (x86_64/linux)
flags: STATS: On, SHM_EXTRA_STATS, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, QM_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
git revision: d1a4419
main.c compiled on 20:42:11 Nov 28 2017 with gcc 4.8


No load on server, no registrations or calls except some scanners trying the code and within couple of days the above errors shows up. There is some memory leak. 

Kindly help on understanding why this is happening.

Regards,
Sammy


_______________________________________________
Devel mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel
Reply | Threaded
Open this post in threaded view
|

Re: Clusterer module and memory leak!

Liviu Chircu

Hi Samy,

If you happen to have the code compiled with the "DBG_MALLOC" flag and catch your OpenSIPS in this state again:

* opensipsctl fifo log_level 14 <pid which went oom>
* kill -SIGUSR1 <pid which went oom>
* opensipsctl fifo log_level 3 <pid which went oom> (restore it to whatever it had before)

This should make that process dump its memory map in the logs, so we can tell where the leak is at.

Regards,

Liviu Chircu
OpenSIPS Developer
http://www.opensips-solutions.com
On 04.12.2017 22:20, SamyGo wrote:
Hi,

I'm experimenting with the clusterer module and in-config cluster_send/receive requests. On my test servers I start seeing the following errors. 

ERROR:core:qm_malloc: not enough free pkg memory (784 bytes left, need 65536), please increase the "-M" command line parameter!
ERROR:core:bin_init: No more pkg memory!
ERROR:clusterer:prep_gen_msg: Failed to init bin send buffer
ERROR:clusterer:bcast_gen_msg: Failed to build generic clusterer message

My opensips process has params -m 256 and -M 64 

version: opensips 2.4.0-dev (x86_64/linux)
flags: STATS: On, SHM_EXTRA_STATS, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, QM_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
git revision: d1a4419
main.c compiled on 20:42:11 Nov 28 2017 with gcc 4.8


No load on server, no registrations or calls except some scanners trying the code and within couple of days the above errors shows up. There is some memory leak. 

Kindly help on understanding why this is happening.

Regards,
Sammy



_______________________________________________
Devel mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel


_______________________________________________
Devel mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel
Reply | Threaded
Open this post in threaded view
|

Re: Clusterer module and memory leak!

Edwin
In reply to this post by SamyGo
Hi Sammy,

I think I am experiencing the same problem, but with load on the server. See
issue 1214 ( https://github.com/OpenSIPS/opensips/issues/1214 )

Regards,

Edwin



--
Sent from: http://opensips-open-sip-server.1449251.n2.nabble.com/OpenSIPS-Devel-f1449246.html

_______________________________________________
Devel mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel
Reply | Threaded
Open this post in threaded view
|

Re: Clusterer module and memory leak!

Razvan Crainea-3
Hi, Edwin!

As Bogdan already asked on the ticket, did you manage to get a memory
dump to figure out if there is indeed a leak and what was causing it?

Best regards,

Răzvan Crainea
OpenSIPS Developer
www.opensips-solutions.com

On 12/07/2017 04:52 PM, Edwin wrote:

> Hi Sammy,
>
> I think I am experiencing the same problem, but with load on the server. See
> issue 1214 ( https://github.com/OpenSIPS/opensips/issues/1214 )
>
> Regards,
>
> Edwin
>
>
>
> --
> Sent from: http://opensips-open-sip-server.1449251.n2.nabble.com/OpenSIPS-Devel-f1449246.html
>
> _______________________________________________
> Devel mailing list
> [hidden email]
> http://lists.opensips.org/cgi-bin/mailman/listinfo/devel


_______________________________________________
Devel mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel