Not enough memory to sync cluster data

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Not enough memory to sync cluster data

vasilevalex
Hi all,

I have simple cluster with full-sharing usrloc. Everything is in memory, no
DB for usrloc.
When starting backup server it syncing usrloc data. So I got errors:

Dec  1 20:15:12 test02 /usr/sbin/opensips[30896]: ERROR:core:fm_malloc: not
enough free pkg memory (30400 bytes left, need 35904), please increase the
"-M" command line parameter!
Dec  1 20:15:12 test02 /usr/sbin/opensips[30896]: INFO:core:fm_malloc:
attempting defragmentation...
Dec  1 20:15:12 test02 /usr/sbin/opensips[30896]: INFO:core:fm_malloc:
unable to alloc a big enough fragment!
Dec  1 20:15:12 test02 /usr/sbin/opensips[30896]: INFO:core:fm_malloc:
unable to alloc a big enough fragment!
Dec  1 20:15:12 test02 /usr/sbin/opensips[30896]: INFO:core:fm_malloc:
attempting defragmentation...
Dec  1 20:15:12 test02 /usr/sbin/opensips[30896]: INFO:core:fm_malloc:
unable to alloc a big enough fragment!
Dec  1 20:15:12 test02 /usr/sbin/opensips[30896]: INFO:core:fm_malloc:
attempting defragmentation...

Start parameters:
/usr/sbin/opensips -m 512 -M 32

And this errors starts with only little bit more than 500 phones.
When cluster in sync, on Backup server I has real_used_size about 11 Mb.

Of course I can increase package memory for example to 64 Mb. But what if I
want not 1000 phones, but 10000 ? May be syncing must use shared memory?
Unfortunately I don't know this process so deep.
Thanks.




--
Sent from: http://opensips-open-sip-server.1449251.n2.nabble.com/OpenSIPS-Users-f1449235.html

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Not enough memory to sync cluster data

Mohit Sachan
plz tell me how to configure opensips for cluster replication (usrloc or dialog)

On Mon, Dec 3, 2018 at 1:00 PM vasilevalex <[hidden email]> wrote:
Hi all,

I have simple cluster with full-sharing usrloc. Everything is in memory, no
DB for usrloc.
When starting backup server it syncing usrloc data. So I got errors:

Dec  1 20:15:12 test02 /usr/sbin/opensips[30896]: ERROR:core:fm_malloc: not
enough free pkg memory (30400 bytes left, need 35904), please increase the
"-M" command line parameter!
Dec  1 20:15:12 test02 /usr/sbin/opensips[30896]: INFO:core:fm_malloc:
attempting defragmentation...
Dec  1 20:15:12 test02 /usr/sbin/opensips[30896]: INFO:core:fm_malloc:
unable to alloc a big enough fragment!
Dec  1 20:15:12 test02 /usr/sbin/opensips[30896]: INFO:core:fm_malloc:
unable to alloc a big enough fragment!
Dec  1 20:15:12 test02 /usr/sbin/opensips[30896]: INFO:core:fm_malloc:
attempting defragmentation...
Dec  1 20:15:12 test02 /usr/sbin/opensips[30896]: INFO:core:fm_malloc:
unable to alloc a big enough fragment!
Dec  1 20:15:12 test02 /usr/sbin/opensips[30896]: INFO:core:fm_malloc:
attempting defragmentation...

Start parameters:
/usr/sbin/opensips -m 512 -M 32

And this errors starts with only little bit more than 500 phones.
When cluster in sync, on Backup server I has real_used_size about 11 Mb.

Of course I can increase package memory for example to 64 Mb. But what if I
want not 1000 phones, but 10000 ? May be syncing must use shared memory?
Unfortunately I don't know this process so deep.
Thanks.




--
Sent from: http://opensips-open-sip-server.1449251.n2.nabble.com/OpenSIPS-Users-f1449235.html

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Not enough memory to sync cluster data

Răzvan Crainea-2
@Alexei: unfortunately there is no way in OpenSIPS to auto-scale the
private or public memory - you'll have to decide from the beginning how
much traffic you are going to support and scale the memory usage
accordingly. Syncing cannot be done using shared memory, so the only
solution I can see is to increase the pkg (-M) parameter to a
comfortable value.

@Mohit: In order to use cluster replication for dialogs, you can follow
this[1] tutorial, or this[2] one for usrloc replication.
Both methods require a full working cluster. For details about cluster
configuration, please read the module's manual[3].

[1]
https://blog.opensips.org/2018/03/23/clustering-ongoing-calls-with-opensips-2-4/
[2]
https://blog.opensips.org/2018/09/13/clustered-sip-user-location-the-full-sharing-topology/
[3] https://opensips.org/html/docs/modules/2.4.x/clusterer#idp6120384

Best regards,
Răzvan

On 12/4/18 7:30 AM, Mohit Sachan wrote:

> plz tell me how to configure opensips for cluster replication (usrloc or
> dialog)
>
> On Mon, Dec 3, 2018 at 1:00 PM vasilevalex <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     Hi all,
>
>     I have simple cluster with full-sharing usrloc. Everything is in
>     memory, no
>     DB for usrloc.
>     When starting backup server it syncing usrloc data. So I got errors:
>
>     Dec  1 20:15:12 test02 /usr/sbin/opensips[30896]:
>     ERROR:core:fm_malloc: not
>     enough free pkg memory (30400 bytes left, need 35904), please
>     increase the
>     "-M" command line parameter!
>     Dec  1 20:15:12 test02 /usr/sbin/opensips[30896]: INFO:core:fm_malloc:
>     attempting defragmentation...
>     Dec  1 20:15:12 test02 /usr/sbin/opensips[30896]: INFO:core:fm_malloc:
>     unable to alloc a big enough fragment!
>     Dec  1 20:15:12 test02 /usr/sbin/opensips[30896]: INFO:core:fm_malloc:
>     unable to alloc a big enough fragment!
>     Dec  1 20:15:12 test02 /usr/sbin/opensips[30896]: INFO:core:fm_malloc:
>     attempting defragmentation...
>     Dec  1 20:15:12 test02 /usr/sbin/opensips[30896]: INFO:core:fm_malloc:
>     unable to alloc a big enough fragment!
>     Dec  1 20:15:12 test02 /usr/sbin/opensips[30896]: INFO:core:fm_malloc:
>     attempting defragmentation...
>
>     Start parameters:
>     /usr/sbin/opensips -m 512 -M 32
>
>     And this errors starts with only little bit more than 500 phones.
>     When cluster in sync, on Backup server I has real_used_size about 11 Mb.
>
>     Of course I can increase package memory for example to 64 Mb. But
>     what if I
>     want not 1000 phones, but 10000 ? May be syncing must use shared memory?
>     Unfortunately I don't know this process so deep.
>     Thanks.
>
>
>
>
>     --
>     Sent from:
>     http://opensips-open-sip-server.1449251.n2.nabble.com/OpenSIPS-Users-f1449235.html
>
>     _______________________________________________
>     Users mailing list
>     [hidden email] <mailto:[hidden email]>
>     http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
>
> _______________________________________________
> Users mailing list
> [hidden email]
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>

--
Răzvan Crainea
OpenSIPS Core Developer
   http://www.opensips-solutions.com
Meet the OpenSIPS team at the next OpenSIPS Summit:
   https://www.opensips.org/events

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Not enough memory to sync cluster data

Liviu Chircu
Hi guys,

I think  Alexey's point is different:  the idea is that PKG memory is
used to store data which is proportional to SHM memory.  The same type
of problem exists with the dialog/usrloc MI "dump" commands: "since
dialogs and contacts are stored in SHM, can we really expect to be able
to build a big PKG buffer where to print them out?"

IMO, both of these operations (cluster sync and MI dumping of large
amounts of SHM data) should use SHM buffers.  We're only talking about a
single alloc operation that occurs seldom, and it's not the end of the
world if you grab the SHM lock for a couple of microseconds when that
happens.  The payoff for having gotten rid of the original problem is
much higher.

Cheers,

Liviu Chircu
OpenSIPS Developer
http://www.opensips-solutions.com

On 05.12.2018 10:22, Răzvan Crainea wrote:

> @Alexei: unfortunately there is no way in OpenSIPS to auto-scale the
> private or public memory - you'll have to decide from the beginning
> how much traffic you are going to support and scale the memory usage
> accordingly. Syncing cannot be done using shared memory, so the only
> solution I can see is to increase the pkg (-M) parameter to a
> comfortable value.
>
> @Mohit: In order to use cluster replication for dialogs, you can
> follow this[1] tutorial, or this[2] one for usrloc replication.
> Both methods require a full working cluster. For details about cluster
> configuration, please read the module's manual[3].
>
> [1]
> https://blog.opensips.org/2018/03/23/clustering-ongoing-calls-with-opensips-2-4/
> [2]
> https://blog.opensips.org/2018/09/13/clustered-sip-user-location-the-full-sharing-topology/
> [3] https://opensips.org/html/docs/modules/2.4.x/clusterer#idp6120384
>
> Best regards,
> Răzvan
>
> On 12/4/18 7:30 AM, Mohit Sachan wrote:
>> plz tell me how to configure opensips for cluster replication
>> (usrloc or dialog)
>>
>> On Mon, Dec 3, 2018 at 1:00 PM vasilevalex <[hidden email]
>> <mailto:[hidden email]>> wrote:
>>
>>     Hi all,
>>
>>     I have simple cluster with full-sharing usrloc. Everything is in
>>     memory, no
>>     DB for usrloc.
>>     When starting backup server it syncing usrloc data. So I got errors:
>>
>>     Dec  1 20:15:12 test02 /usr/sbin/opensips[30896]:
>>     ERROR:core:fm_malloc: not
>>     enough free pkg memory (30400 bytes left, need 35904), please
>>     increase the
>>     "-M" command line parameter!
>>     Dec  1 20:15:12 test02 /usr/sbin/opensips[30896]:
>> INFO:core:fm_malloc:
>>     attempting defragmentation...
>>     Dec  1 20:15:12 test02 /usr/sbin/opensips[30896]:
>> INFO:core:fm_malloc:
>>     unable to alloc a big enough fragment!
>>     Dec  1 20:15:12 test02 /usr/sbin/opensips[30896]:
>> INFO:core:fm_malloc:
>>     unable to alloc a big enough fragment!
>>     Dec  1 20:15:12 test02 /usr/sbin/opensips[30896]:
>> INFO:core:fm_malloc:
>>     attempting defragmentation...
>>     Dec  1 20:15:12 test02 /usr/sbin/opensips[30896]:
>> INFO:core:fm_malloc:
>>     unable to alloc a big enough fragment!
>>     Dec  1 20:15:12 test02 /usr/sbin/opensips[30896]:
>> INFO:core:fm_malloc:
>>     attempting defragmentation...
>>
>>     Start parameters:
>>     /usr/sbin/opensips -m 512 -M 32
>>
>>     And this errors starts with only little bit more than 500 phones.
>>     When cluster in sync, on Backup server I has real_used_size about
>> 11 Mb.
>>
>>     Of course I can increase package memory for example to 64 Mb. But
>>     what if I
>>     want not 1000 phones, but 10000 ? May be syncing must use shared
>> memory?
>>     Unfortunately I don't know this process so deep.
>>     Thanks.
>>
>>
>>
>>
>>     --
>>     Sent from:
>> http://opensips-open-sip-server.1449251.n2.nabble.com/OpenSIPS-Users-f1449235.html
>>
>>     _______________________________________________
>>     Users mailing list
>>     [hidden email] <mailto:[hidden email]>
>>     http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>
>>
>> _______________________________________________
>> Users mailing list
>> [hidden email]
>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>
>

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Not enough memory to sync cluster data

vasilevalex
Hi all,

Thank you @Liviu, this is exactly what I mean. Either use SHM or may be
split all data into smaller parts to fit PKG memory. This is just thoughts.
Because if people start using real data in cluster they will reach PKG limit
very fast.



--
Sent from: http://opensips-open-sip-server.1449251.n2.nabble.com/OpenSIPS-Users-f1449235.html

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Not enough memory to sync cluster data

Răzvan Crainea-2
I don't agree with this :). SHM has a completely different purpose (that
sharing data between processes), not just the virtue of being large. And
I'm not arguing about performance here (SHM lock, write/read
operations), but rather about things that this change will influence:
1. fragmentation issues
2. you will need to increase the SHM memory anyway to support this: if
you need let's say 1GB RAM to hold 1M dialogs, you'll need to double SHM
memory to 2GB for syncing. And if you consider that in the same time a
dlg list might come, you'll probably need 1GB more for that, reaching to
a triple amount of SHM memory. I don't find this practical.

IMO, you know better how you're going to use the system, and if you do
want to do usrloc/dialog sync for large amount of data, just set the PKG
value to the same of SHM. This seems to be more clean, efficient, and if
you don't need it, the OS will not even allocate it (due to the
demand-paging mechanism). So I don't see where you reservations for
setting a higher value of the -M parameter come from.

Best regards,
Răzvan


On 12/5/18 11:11 PM, vasilevalex wrote:

> Hi all,
>
> Thank you @Liviu, this is exactly what I mean. Either use SHM or may be
> split all data into smaller parts to fit PKG memory. This is just thoughts.
> Because if people start using real data in cluster they will reach PKG limit
> very fast.
>
>
>
> --
> Sent from: http://opensips-open-sip-server.1449251.n2.nabble.com/OpenSIPS-Users-f1449235.html
>
> _______________________________________________
> Users mailing list
> [hidden email]
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>

--
Răzvan Crainea
OpenSIPS Core Developer
   http://www.opensips-solutions.com
Meet the OpenSIPS team at the next OpenSIPS Summit:
   https://www.opensips.org/events

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Not enough memory to sync cluster data

vasilevalex
Thanks @Răzvan. As I said, I don't know opensips memory management so deep.
And I'm usually trying to use as many resources as actually required for the
task.
I just didn't want "overselling", like I have 80 opensips processes with 32
Mb each. That's ok if they all decide to use all the memory. But if I allow
128Mb for each one and they suddenly decide to allocate it :) I understand
that this is practically will never happen.

For now I'll just increase PKG memory.
Everybody thanks for help and explanation.



--
Sent from: http://opensips-open-sip-server.1449251.n2.nabble.com/OpenSIPS-Users-f1449235.html

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Not enough memory to sync cluster data

Vitalii Aleksandrov
In reply to this post by Răzvan Crainea-2

> This seems to be more clean, efficient, and if you don't need it, the
> OS will not even allocate it (due to the demand-paging mechanism). So
> I don't see where you reservations for setting a higher value of the
> -M parameter come from.
>
> Best regards,
> Răzvan
>
Just my 2 cents about PKG mem. Indeed OS will not allocate all requested
PKG mem at the strartup. After a while if sync mechanism continues to
work continuously all worker processes will be involved at least once in
a syncing process (send or receive big amount of data). That means that
all processes will try to init that memory and OS will finally map
requested virtual memory to psychical one. When the sync is finished and
after pkg_free() called that memory becomes free and reusable from the
process point of view, but if I recall correctly how the memory manager
works (at least worked in kamailio) those chunks will be just marked as
free in internal data structure but never returned back to the OS. To
make long story short in the end after a while OS will have to really
allocate RAM for all opensips worker processes if all of them are
involved in proto_bin processing (not sure about it).
Please correct me if I'm wrong.

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Not enough memory to sync cluster data

Răzvan Crainea-2


On 12/6/18 1:16 PM, Vitalii Aleksandrov wrote:

>
>> This seems to be more clean, efficient, and if you don't need it, the
>> OS will not even allocate it (due to the demand-paging mechanism). So
>> I don't see where you reservations for setting a higher value of the
>> -M parameter come from.
>>
>> Best regards,
>> Răzvan
>>
> Just my 2 cents about PKG mem. Indeed OS will not allocate all requested
> PKG mem at the strartup. After a while if sync mechanism continues to
> work continuously all worker processes will be involved at least once in
> a syncing process (send or receive big amount of data). That means that
> all processes will try to init that memory and OS will finally map
> requested virtual memory to psychical one. When the sync is finished and
> after pkg_free() called that memory becomes free and reusable from the
> process point of view, but if I recall correctly how the memory manager
> works (at least worked in kamailio) those chunks will be just marked as
> free in internal data structure but never returned back to the OS. To
> make long story short in the end after a while OS will have to really
> allocate RAM for all opensips worker processes if all of them are
> involved in proto_bin processing (not sure about it).
> Please correct me if I'm wrong.
>

Hi, Vitalii!

You are absolutely right! However, syncing is not such a frequent
process, it is only done either during a restart of the backup server,
or if an MI command triggers it, not sure why, but most likely also due
to a restart of the backup server. So I'd argue only a bunch of sync
commands will be used per run, but indeed, these might occur in any process.

@Alexei, getting back to the original error, can point us the exact
error that made you think about usrloc sync-ing? I am asking because
during sync, usrloc data is sent in smaller chunks, not the entire table
at once. Therefore your assumption might be a bit misleading.

Best regards,

--
Răzvan Crainea
OpenSIPS Core Developer
   http://www.opensips-solutions.com
Meet the OpenSIPS team at the next OpenSIPS Summit:
   https://www.opensips.org/events

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Not enough memory to sync cluster data

vasilevalex
Hi all,

Yes, may be my assumption was wrong. @Răzvan please look at logs and routing
script parts:

Process of starting OpenSIPS (I skip repeated messages and add some
comments):
Dec  1 20:13:01 test02 /usr/sbin/opensips[30853]: NOTICE:core:main: version:
opensips 2.4.3 (x86_64/linux)
Dec  1 20:13:01 test02 /usr/sbin/opensips[30853]: INFO:core:main: using 512
Mb of shared memory
Dec  1 20:13:01 test02 /usr/sbin/opensips[30853]: INFO:core:main: using 32
Mb of private process memory
<... skipped ...>
Dec  1 20:13:01 test02 /usr/sbin/opensips[30853]:
INFO:core:evi_publish_event: Registered event <E_CORE_THRESHOLD(0)>
Dec  1 20:13:01 test02 /usr/sbin/opensips[30853]:
INFO:core:evi_publish_event: Registered event <E_CORE_SHM_THRESHOLD(1)>
Dec  1 20:13:01 test02 /usr/sbin/opensips[30853]:
INFO:core:evi_publish_event: Registered event <E_CORE_PKG_THRESHOLD(2)>
Dec  1 20:13:01 test02 /usr/sbin/opensips[30853]:
INFO:core:evi_publish_event: Registered event <E_MYSQL_CONNECTION(3)>
Dec  1 20:13:01 test02 /usr/sbin/opensips[30853]: INFO:proto_bin:mod_init:
initializing BIN protocol
Dec  1 20:13:01 test02 /usr/sbin/opensips[30853]: INFO:clusterer:mod_init:
Clusterer module - initializing
Dec  1 20:13:01 test02 /usr/sbin/opensips[30853]:
INFO:core:evi_publish_event: Registered event <E_CLUSTERER_REQ_RECEIVED(4)>
Dec  1 20:13:01 test02 /usr/sbin/opensips[30853]:
INFO:core:evi_publish_event: Registered event <E_CLUSTERER_RPL_RECEIVED(5)>
Dec  1 20:13:01 test02 /usr/sbin/opensips[30853]: INFO:usrloc:ul_init_locks:
locks array size 512
Dec  1 20:13:01 test02 /usr/sbin/opensips[30853]:
INFO:core:evi_publish_event: Registered event <E_UL_AOR_INSERT(6)>
Dec  1 20:13:01 test02 /usr/sbin/opensips[30853]:
INFO:core:evi_publish_event: Registered event <E_UL_AOR_DELETE(7)>
Dec  1 20:13:01 test02 /usr/sbin/opensips[30853]:
INFO:core:evi_publish_event: Registered event <E_UL_CONTACT_INSERT(8)>
Dec  1 20:13:01 test02 /usr/sbin/opensips[30853]:
INFO:core:evi_publish_event: Registered event <E_UL_CONTACT_DELETE(9)>
Dec  1 20:13:01 test02 /usr/sbin/opensips[30853]:
INFO:core:evi_publish_event: Registered event <E_UL_CONTACT_UPDATE(10)>
Dec  1 20:13:01 test02 /usr/sbin/opensips[30853]:
INFO:core:evi_publish_event: Registered event <E_UL_LATENCY_UPDATE(11)>
<... skipped normal start messages ...>
### Than there are a lot of such messages (300-400):
Dec  1 20:13:04 test02 /usr/sbin/opensips[30896]:
INFO:usrloc:receive_ucontact_delete: failed to fetch local urecord -
ignoring request (ci: '313534303437393137333433343135-n7utfvs2rllm')
Dec  1 20:13:04 test02 /usr/sbin/opensips[30896]:
INFO:usrloc:receive_ucontact_delete: failed to fetch local urecord -
ignoring request (ci: '313534333537353638353630323134-gbdnp67daddu')
Dec  1 20:13:04 test02 /usr/sbin/opensips[30896]:
INFO:usrloc:receive_ucontact_delete: failed to fetch local urecord -
ignoring request (ci: '313534333233353433383232373838-3a3w8ljwjq7f')
<... skipped ...>
### I think this delete messages came from Active server, when it starts
deleting dead contacts, as TLS connections were of course closed when
switching.
### Than there are a lot of messages generated by OpenSIPS event route
Dec  1 20:15:11 test02 /usr/sbin/opensips[30896]: ATTENTION! Used 26843600
of 33554432 private memory by PID 30896
Dec  1 20:15:11 test02 /usr/sbin/opensips[30896]: ATTENTION! Used 27194504
of 33554432 private memory by PID 30896
Dec  1 20:15:11 test02 /usr/sbin/opensips[30896]: ATTENTION! Used 27523656
of 33554432 private memory by PID 30896
Dec  1 20:15:11 test02 /usr/sbin/opensips[30896]: ATTENTION! Used 27850224
of 33554432 private memory by PID 30896
<... skipped ...>
### Than start error messages
Dec  1 20:15:12 test02 /usr/sbin/opensips[30896]: ERROR:core:fm_malloc: not
enough free pkg memory (30400 bytes left, need 35904), please increase the
"-M" command line parameter!
Dec  1 20:15:12 test02 /usr/sbin/opensips[30896]: INFO:core:fm_malloc:
attempting defragmentation...
Dec  1 20:15:12 test02 /usr/sbin/opensips[30896]: INFO:core:fm_malloc:
unable to alloc a big enough fragment!
Dec  1 20:15:12 test02 /usr/sbin/opensips[30896]:
ERROR:rest_client:start_async_http_req: Init curl handle failed!
Dec  1 20:15:12 test02 /usr/sbin/opensips[30896]: ERROR:core:fm_malloc: not
enough free pkg memory (30400 bytes left, need 35904), please increase the
"-M" command line parameter!
Dec  1 20:15:12 test02 /usr/sbin/opensips[30896]: INFO:core:fm_malloc:
attempting defragmentation...
Dec  1 20:15:12 test02 /usr/sbin/opensips[30896]: INFO:core:fm_malloc:
unable to alloc a big enough fragment!
Dec  1 20:15:12 test02 /usr/sbin/opensips[30896]:
ERROR:rest_client:start_async_http_req: Init curl handle failed!
### rest_client called async from event route
<... skipped ...>
Dec  1 20:15:12 test02 /usr/sbin/opensips[30896]: INFO:core:fm_malloc:
attempting defragmentation...
Dec  1 20:15:12 test02 /usr/sbin/opensips[30896]: INFO:core:fm_malloc:
unable to alloc a big enough fragment!
Dec  1 20:15:12 test02 /usr/sbin/opensips[30896]: INFO:core:fm_malloc:
attempting defragmentation...
Dec  1 20:15:12 test02 /usr/sbin/opensips[30896]: INFO:core:fm_malloc:
unable to alloc a big enough fragment!
<... skipped ...>
### Than server start working like usual, without any unexpected errors.
Dec  1 20:27:52 test02 /usr/sbin/opensips[30896]:
INFO:usrloc:receive_ucontact_update: failed to fetch local urecord - create
new record and contact (ci: '313534333639353234383631313333-k2uymmqblqnx')
Dec  1 20:35:46 test02 /usr/sbin/opensips[30896]:
INFO:usrloc:receive_ucontact_update: failed to fetch local urecord - create
new record and contact (ci: '313534333639353235323136363838-5p34q9svc9v1')
Dec  1 20:45:09 test02 /usr/sbin/opensips[30896]:
INFO:usrloc:receive_ucontact_update: failed to fetch local urecord - create
new record and contact (ci: '313534333639353330363630373039-mv4k1u64jm0s')


Now some routes from OpenSIPS script:

event_route[E_UL_CONTACT_INSERT] {
        fetch_event_params("aor=$avp(aor);uri=$avp(uri);received=$avp(received)");
        $avp(ul_method) = "insert";
        $var(body) = "{ ....}\r\n"
        launch(rest_post("API_URL", "$var(body)", "application/json", "$var(ret)"),
resume_post);
}

event_route[E_UL_CONTACT_UPDATE] {
        fetch_event_params("aor=$avp(aor);uri=$avp(uri);received=$avp(received)");
        $avp(ul_method) = "update";
        $var(body) = "{ ....}\r\n"
        launch(rest_post("API_URL", "$var(body)", "application/json", "$var(ret)"),
resume_post);
}

route[resume_post] {
        if ($rc < 0) {
                xlog("Error $rc in HTTP POST\n");
        }
        exit;
}

event_route[E_CORE_PKG_THRESHOLD] {
        fetch_event_params("used=$avp(used);size=$avp(size);pid=$avp(pid)");
        xlog("ATTENTION! Used $avp(used) of $avp(size) private memory by PID
$avp(pid)\n");
}

So may be every contact added by syncing data also calls
E_UL_CONTACT_INSERT? And libcurl call consumes a lot of memory?
Thanks.



--
Sent from: http://opensips-open-sip-server.1449251.n2.nabble.com/OpenSIPS-Users-f1449235.html

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Not enough memory to sync cluster data

Răzvan Crainea-2
Actually this looks like a memory leak.
Can you tell us what kind of process is 30896? You can find out running
`opensipsctl fifo ps | grep 30896`
Also, what version of opensips are you using (output of `opensips -V`).

Best regards,
Răzvan

On 12/7/18 10:52 AM, vasilevalex wrote:

> Hi all,
>
> Yes, may be my assumption was wrong. @Răzvan please look at logs and routing
> script parts:
>
> Process of starting OpenSIPS (I skip repeated messages and add some
> comments):
> Dec  1 20:13:01 test02 /usr/sbin/opensips[30853]: NOTICE:core:main: version:
> opensips 2.4.3 (x86_64/linux)
> Dec  1 20:13:01 test02 /usr/sbin/opensips[30853]: INFO:core:main: using 512
> Mb of shared memory
> Dec  1 20:13:01 test02 /usr/sbin/opensips[30853]: INFO:core:main: using 32
> Mb of private process memory
> <... skipped ...>
> Dec  1 20:13:01 test02 /usr/sbin/opensips[30853]:
> INFO:core:evi_publish_event: Registered event <E_CORE_THRESHOLD(0)>
> Dec  1 20:13:01 test02 /usr/sbin/opensips[30853]:
> INFO:core:evi_publish_event: Registered event <E_CORE_SHM_THRESHOLD(1)>
> Dec  1 20:13:01 test02 /usr/sbin/opensips[30853]:
> INFO:core:evi_publish_event: Registered event <E_CORE_PKG_THRESHOLD(2)>
> Dec  1 20:13:01 test02 /usr/sbin/opensips[30853]:
> INFO:core:evi_publish_event: Registered event <E_MYSQL_CONNECTION(3)>
> Dec  1 20:13:01 test02 /usr/sbin/opensips[30853]: INFO:proto_bin:mod_init:
> initializing BIN protocol
> Dec  1 20:13:01 test02 /usr/sbin/opensips[30853]: INFO:clusterer:mod_init:
> Clusterer module - initializing
> Dec  1 20:13:01 test02 /usr/sbin/opensips[30853]:
> INFO:core:evi_publish_event: Registered event <E_CLUSTERER_REQ_RECEIVED(4)>
> Dec  1 20:13:01 test02 /usr/sbin/opensips[30853]:
> INFO:core:evi_publish_event: Registered event <E_CLUSTERER_RPL_RECEIVED(5)>
> Dec  1 20:13:01 test02 /usr/sbin/opensips[30853]: INFO:usrloc:ul_init_locks:
> locks array size 512
> Dec  1 20:13:01 test02 /usr/sbin/opensips[30853]:
> INFO:core:evi_publish_event: Registered event <E_UL_AOR_INSERT(6)>
> Dec  1 20:13:01 test02 /usr/sbin/opensips[30853]:
> INFO:core:evi_publish_event: Registered event <E_UL_AOR_DELETE(7)>
> Dec  1 20:13:01 test02 /usr/sbin/opensips[30853]:
> INFO:core:evi_publish_event: Registered event <E_UL_CONTACT_INSERT(8)>
> Dec  1 20:13:01 test02 /usr/sbin/opensips[30853]:
> INFO:core:evi_publish_event: Registered event <E_UL_CONTACT_DELETE(9)>
> Dec  1 20:13:01 test02 /usr/sbin/opensips[30853]:
> INFO:core:evi_publish_event: Registered event <E_UL_CONTACT_UPDATE(10)>
> Dec  1 20:13:01 test02 /usr/sbin/opensips[30853]:
> INFO:core:evi_publish_event: Registered event <E_UL_LATENCY_UPDATE(11)>
> <... skipped normal start messages ...>
> ### Than there are a lot of such messages (300-400):
> Dec  1 20:13:04 test02 /usr/sbin/opensips[30896]:
> INFO:usrloc:receive_ucontact_delete: failed to fetch local urecord -
> ignoring request (ci: '313534303437393137333433343135-n7utfvs2rllm')
> Dec  1 20:13:04 test02 /usr/sbin/opensips[30896]:
> INFO:usrloc:receive_ucontact_delete: failed to fetch local urecord -
> ignoring request (ci: '313534333537353638353630323134-gbdnp67daddu')
> Dec  1 20:13:04 test02 /usr/sbin/opensips[30896]:
> INFO:usrloc:receive_ucontact_delete: failed to fetch local urecord -
> ignoring request (ci: '313534333233353433383232373838-3a3w8ljwjq7f')
> <... skipped ...>
> ### I think this delete messages came from Active server, when it starts
> deleting dead contacts, as TLS connections were of course closed when
> switching.
> ### Than there are a lot of messages generated by OpenSIPS event route
> Dec  1 20:15:11 test02 /usr/sbin/opensips[30896]: ATTENTION! Used 26843600
> of 33554432 private memory by PID 30896
> Dec  1 20:15:11 test02 /usr/sbin/opensips[30896]: ATTENTION! Used 27194504
> of 33554432 private memory by PID 30896
> Dec  1 20:15:11 test02 /usr/sbin/opensips[30896]: ATTENTION! Used 27523656
> of 33554432 private memory by PID 30896
> Dec  1 20:15:11 test02 /usr/sbin/opensips[30896]: ATTENTION! Used 27850224
> of 33554432 private memory by PID 30896
> <... skipped ...>
> ### Than start error messages
> Dec  1 20:15:12 test02 /usr/sbin/opensips[30896]: ERROR:core:fm_malloc: not
> enough free pkg memory (30400 bytes left, need 35904), please increase the
> "-M" command line parameter!
> Dec  1 20:15:12 test02 /usr/sbin/opensips[30896]: INFO:core:fm_malloc:
> attempting defragmentation...
> Dec  1 20:15:12 test02 /usr/sbin/opensips[30896]: INFO:core:fm_malloc:
> unable to alloc a big enough fragment!
> Dec  1 20:15:12 test02 /usr/sbin/opensips[30896]:
> ERROR:rest_client:start_async_http_req: Init curl handle failed!
> Dec  1 20:15:12 test02 /usr/sbin/opensips[30896]: ERROR:core:fm_malloc: not
> enough free pkg memory (30400 bytes left, need 35904), please increase the
> "-M" command line parameter!
> Dec  1 20:15:12 test02 /usr/sbin/opensips[30896]: INFO:core:fm_malloc:
> attempting defragmentation...
> Dec  1 20:15:12 test02 /usr/sbin/opensips[30896]: INFO:core:fm_malloc:
> unable to alloc a big enough fragment!
> Dec  1 20:15:12 test02 /usr/sbin/opensips[30896]:
> ERROR:rest_client:start_async_http_req: Init curl handle failed!
> ### rest_client called async from event route
> <... skipped ...>
> Dec  1 20:15:12 test02 /usr/sbin/opensips[30896]: INFO:core:fm_malloc:
> attempting defragmentation...
> Dec  1 20:15:12 test02 /usr/sbin/opensips[30896]: INFO:core:fm_malloc:
> unable to alloc a big enough fragment!
> Dec  1 20:15:12 test02 /usr/sbin/opensips[30896]: INFO:core:fm_malloc:
> attempting defragmentation...
> Dec  1 20:15:12 test02 /usr/sbin/opensips[30896]: INFO:core:fm_malloc:
> unable to alloc a big enough fragment!
> <... skipped ...>
> ### Than server start working like usual, without any unexpected errors.
> Dec  1 20:27:52 test02 /usr/sbin/opensips[30896]:
> INFO:usrloc:receive_ucontact_update: failed to fetch local urecord - create
> new record and contact (ci: '313534333639353234383631313333-k2uymmqblqnx')
> Dec  1 20:35:46 test02 /usr/sbin/opensips[30896]:
> INFO:usrloc:receive_ucontact_update: failed to fetch local urecord - create
> new record and contact (ci: '313534333639353235323136363838-5p34q9svc9v1')
> Dec  1 20:45:09 test02 /usr/sbin/opensips[30896]:
> INFO:usrloc:receive_ucontact_update: failed to fetch local urecord - create
> new record and contact (ci: '313534333639353330363630373039-mv4k1u64jm0s')
>
>
> Now some routes from OpenSIPS script:
>
> event_route[E_UL_CONTACT_INSERT] {
> fetch_event_params("aor=$avp(aor);uri=$avp(uri);received=$avp(received)");
> $avp(ul_method) = "insert";
> $var(body) = "{ ....}\r\n"
> launch(rest_post("API_URL", "$var(body)", "application/json", "$var(ret)"),
> resume_post);
> }
>
> event_route[E_UL_CONTACT_UPDATE] {
> fetch_event_params("aor=$avp(aor);uri=$avp(uri);received=$avp(received)");
> $avp(ul_method) = "update";
> $var(body) = "{ ....}\r\n"
> launch(rest_post("API_URL", "$var(body)", "application/json", "$var(ret)"),
> resume_post);
> }
>
> route[resume_post] {
> if ($rc < 0) {
> xlog("Error $rc in HTTP POST\n");
> }
> exit;
> }
>
> event_route[E_CORE_PKG_THRESHOLD] {
> fetch_event_params("used=$avp(used);size=$avp(size);pid=$avp(pid)");
> xlog("ATTENTION! Used $avp(used) of $avp(size) private memory by PID
> $avp(pid)\n");
> }
>
> So may be every contact added by syncing data also calls
> E_UL_CONTACT_INSERT? And libcurl call consumes a lot of memory?
> Thanks.
>
>
>
> --
> Sent from: http://opensips-open-sip-server.1449251.n2.nabble.com/OpenSIPS-Users-f1449235.html
>
> _______________________________________________
> Users mailing list
> [hidden email]
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>

--
Răzvan Crainea
OpenSIPS Core Developer
   http://www.opensips-solutions.com
Meet the OpenSIPS team at the next OpenSIPS Summit:
   https://www.opensips.org/events

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Not enough memory to sync cluster data

vasilevalex
Yes, shure.

Process::  ID=42 PID=30896 Type=TCP receiver

# opensips -V
version: opensips 2.4.3 (x86_64/linux)
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, F_MALLOC,
FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
git revision: unknown
main.c compiled on 13:27:19 Nov 28 2018 with gcc 4.8.5

This is actually git 2.4 branch with couple patches:

git cherry-pick 94a3ede1e276984a91f93f6ece832d174b071ab8
git cherry-pick 05ca54a37d82c605e2cd6d10e5a62fb4f7c35b78





--
Sent from: http://opensips-open-sip-server.1449251.n2.nabble.com/OpenSIPS-Users-f1449235.html

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users