Quest to find memory leak

classic Classic list List threaded Threaded
33 messages Options
12
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Quest to find memory leak

John Nash
I am using OpenSIPS (2.1.5 (x86_64/linux)) in production. I observed private memory is decreasing constantly for one process mainly and ultimately leading to memory errors and crash.

To debug this issue I prepared a test server and compiled opensips as per https://www.opensips.org/Documentation/TroubleShooting-OutOfMem

I made only one single call (which was rejected by opensips as it was not authorized user) and I saw private free memory decreased. I was hoping since transaction is done ideally it should release memory and should show me same memory as startup but it did not. I verified this with many call attempts and i see free memory is always decreasing slowly.

I used kill -SIGUSR1 <lowest pid> to create memory dump. But i am unable to make sense of it. It shows log like ...

r  6 07:29:19 Server3021 opensips[13276]: Memory status (pkg):
Mar  6 07:29:19 Server3021 opensips[13276]: qm_status (0x7f5b8ebba010):
Mar  6 07:29:19 Server3021 opensips[13276]:  heap size= 4194304
Mar  6 07:29:19 Server3021 opensips[13276]:  used= 346768, used+overhead=848792, free=3345512
Mar  6 07:29:19 Server3021 opensips[13276]:  max used (+overhead)= 931920
Mar  6 07:29:19 Server3021 opensips[13276]: dumping all alloc'ed. fragments:
Mar  6 07:29:19 Server3021 opensips[13276]:       0. N  address=0x7f5b8ebef528 frag=0x7f5b8ebef4f8 size=40 used=1
Mar  6 07:29:19 Server3021 opensips[13276]:             alloc'd from script_cb.c: add_callback(60)
Mar  6 07:29:19 Server3021 opensips[13276]:         start check=f0f0f0f0f0f0f0f0, end check= c0c0c0c0c0c0c0c0, abcdefedabcdefed
Mar  6 07:29:19 Server3021 opensips[13276]:       1. N  address=0x7f5b8ebef5b0 

I pasted only few lines in this mail. What should be my next step?...How can i really trace what is wrong in my script or any other memory leak?


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Quest to find memory leak

Razvan Crainea-3
Hi, John!

Transactions are stored in shared memory, not in the private one. So the possible leak you are facing its not related to transactions.
During runtime, OpenSIPS might resize some internal structures, which may lead to increase memory usage. However, after a while, these allocations should stabilize
.
Can you post the output of the kill -SIGUSR1 on pastebin so we can take a look? Also, what type of process is the one you are seeing the leak into? You can find out using the 'opensipsctl ps' command.

Best regards,
Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com
On 03/06/2017 09:55 AM, John Nash wrote:
I am using OpenSIPS (2.1.5 (x86_64/linux)) in production. I observed private memory is decreasing constantly for one process mainly and ultimately leading to memory errors and crash.

To debug this issue I prepared a test server and compiled opensips as per https://www.opensips.org/Documentation/TroubleShooting-OutOfMem

I made only one single call (which was rejected by opensips as it was not authorized user) and I saw private free memory decreased. I was hoping since transaction is done ideally it should release memory and should show me same memory as startup but it did not. I verified this with many call attempts and i see free memory is always decreasing slowly.

I used kill -SIGUSR1 <lowest pid> to create memory dump. But i am unable to make sense of it. It shows log like ...

r  6 07:29:19 Server3021 opensips[13276]: Memory status (pkg):
Mar  6 07:29:19 Server3021 opensips[13276]: qm_status (0x7f5b8ebba010):
Mar  6 07:29:19 Server3021 opensips[13276]:  heap size= 4194304
Mar  6 07:29:19 Server3021 opensips[13276]:  used= 346768, used+overhead=848792, free=3345512
Mar  6 07:29:19 Server3021 opensips[13276]:  max used (+overhead)= 931920
Mar  6 07:29:19 Server3021 opensips[13276]: dumping all alloc'ed. fragments:
Mar  6 07:29:19 Server3021 opensips[13276]:       0. N  address=0x7f5b8ebef528 frag=0x7f5b8ebef4f8 size=40 used=1
Mar  6 07:29:19 Server3021 opensips[13276]:             alloc'd from script_cb.c: add_callback(60)
Mar  6 07:29:19 Server3021 opensips[13276]:         start check=f0f0f0f0f0f0f0f0, end check= c0c0c0c0c0c0c0c0, abcdefedabcdefed
Mar  6 07:29:19 Server3021 opensips[13276]:       1. N  address=0x7f5b8ebef5b0 

I pasted only few lines in this mail. What should be my next step?...How can i really trace what is wrong in my script or any other memory leak?



_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Quest to find memory leak

John Nash
Dear Razvan,

Below is the info on my processes
Process::  ID=0 PID=17351 Type=attendant
Process::  ID=1 PID=17352 Type=MI FIFO
Process::  ID=2 PID=17353 Type=MI Datagram
Process::  ID=3 PID=17354 Type=time_keeper
Process::  ID=4 PID=17355 Type=timer
Process::  ID=5 PID=17356 Type=SIP receiver udp:1.1.1.1:9094 
Process::  ID=6 PID=17357 Type=SIP receiver udp:1.1.1.1:5060 
Process::  ID=7 PID=17358 Type=SIP receiver udp:192.168.45.5:5064 
Process::  ID=8 PID=17359 Type=Timer handler

1.1.1.1 is public IP (I changed). The decrease in memory I see is for Process::  ID=7 PID=17358 mainly. My call flow is as following 

- New Invite hits the opensips on 1.1.1.1:9094
- Apart from message validity checks I query DB to check if its a valid user (Using local cache also there)
- Create dialog, Topology_hiding functions are called along with some avp population
- Using dispatcher ds_select_domain Call sent to udp:192.168.45.2:7060 (using force socket). This 192.168.45.2:7060 is actually freeswitch 
- Call again comes back to opensips on udp:192.168.45.5:5064
- New dialog is created and topology_hiding is called
- Drouting function do_routing is called which tries one gateway and fails


Dump i need to create with memlog=4 memdump=1 right?









On Mon, Mar 6, 2017 at 2:05 PM, Răzvan Crainea <[hidden email]> wrote:
Hi, John!

Transactions are stored in shared memory, not in the private one. So the possible leak you are facing its not related to transactions.
During runtime, OpenSIPS might resize some internal structures, which may lead to increase memory usage. However, after a while, these allocations should stabilize
.
Can you post the output of the kill -SIGUSR1 on pastebin so we can take a look? Also, what type of process is the one you are seeing the leak into? You can find out using the 'opensipsctl ps' command.

Best regards,
Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com
On 03/06/2017 09:55 AM, John Nash wrote:
I am using OpenSIPS (2.1.5 (x86_64/linux)) in production. I observed private memory is decreasing constantly for one process mainly and ultimately leading to memory errors and crash.

To debug this issue I prepared a test server and compiled opensips as per https://www.opensips.org/Documentation/TroubleShooting-OutOfMem

I made only one single call (which was rejected by opensips as it was not authorized user) and I saw private free memory decreased. I was hoping since transaction is done ideally it should release memory and should show me same memory as startup but it did not. I verified this with many call attempts and i see free memory is always decreasing slowly.

I used kill -SIGUSR1 <lowest pid> to create memory dump. But i am unable to make sense of it. It shows log like ...

r  6 07:29:19 Server3021 opensips[13276]: Memory status (pkg):
Mar  6 07:29:19 Server3021 opensips[13276]: qm_status (0x7f5b8ebba010):
Mar  6 07:29:19 Server3021 opensips[13276]:  heap size= 4194304
Mar  6 07:29:19 Server3021 opensips[13276]:  used= 346768, used+overhead=848792, free=3345512
Mar  6 07:29:19 Server3021 opensips[13276]:  max used (+overhead)= 931920
Mar  6 07:29:19 Server3021 opensips[13276]: dumping all alloc'ed. fragments:
Mar  6 07:29:19 Server3021 opensips[13276]:       0. N  address=0x7f5b8ebef528 frag=0x7f5b8ebef4f8 size=40 used=1
Mar  6 07:29:19 Server3021 opensips[13276]:             alloc'd from script_cb.c: add_callback(60)
Mar  6 07:29:19 Server3021 opensips[13276]:         start check=f0f0f0f0f0f0f0f0, end check= c0c0c0c0c0c0c0c0, abcdefedabcdefed
Mar  6 07:29:19 Server3021 opensips[13276]:       1. N  address=0x7f5b8ebef5b0 

I pasted only few lines in this mail. What should be my next step?...How can i really trace what is wrong in my script or any other memory leak?



_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users



_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Quest to find memory leak

Razvan Crainea-3
Ok, so it is the first listener for the private IP that leaks. Next, is the memory stabilizing in time? Or it is continously decreasing?
Yes, that's how you should make the dump.

Best regards,
Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com
On 03/06/2017 10:57 AM, John Nash wrote:
Dear Razvan,

Below is the info on my processes
Process::  ID=0 PID=17351 Type=attendant
Process::  ID=1 PID=17352 Type=MI FIFO
Process::  ID=2 PID=17353 Type=MI Datagram
Process::  ID=3 PID=17354 Type=time_keeper
Process::  ID=4 PID=17355 Type=timer
Process::  ID=5 PID=17356 Type=SIP receiver udp:1.1.1.1:9094 
Process::  ID=6 PID=17357 Type=SIP receiver udp:1.1.1.1:5060 
Process::  ID=7 PID=17358 Type=SIP receiver udp:192.168.45.5:5064 
Process::  ID=8 PID=17359 Type=Timer handler

1.1.1.1 is public IP (I changed). The decrease in memory I see is for Process::  ID=7 PID=17358 mainly. My call flow is as following 

- New Invite hits the opensips on 1.1.1.1:9094
- Apart from message validity checks I query DB to check if its a valid user (Using local cache also there)
- Create dialog, Topology_hiding functions are called along with some avp population
- Using dispatcher ds_select_domain Call sent to udp:192.168.45.2:7060 (using force socket). This 192.168.45.2:7060 is actually freeswitch 
- Call again comes back to opensips on udp:192.168.45.5:5064
- New dialog is created and topology_hiding is called
- Drouting function do_routing is called which tries one gateway and fails


Dump i need to create with memlog=4 memdump=1 right?









On Mon, Mar 6, 2017 at 2:05 PM, Răzvan Crainea <[hidden email]> wrote:
Hi, John!

Transactions are stored in shared memory, not in the private one. So the possible leak you are facing its not related to transactions.
During runtime, OpenSIPS might resize some internal structures, which may lead to increase memory usage. However, after a while, these allocations should stabilize
.
Can you post the output of the kill -SIGUSR1 on pastebin so we can take a look? Also, what type of process is the one you are seeing the leak into? You can find out using the 'opensipsctl ps' command.

Best regards,
Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com
On 03/06/2017 09:55 AM, John Nash wrote:
I am using OpenSIPS (2.1.5 (x86_64/linux)) in production. I observed private memory is decreasing constantly for one process mainly and ultimately leading to memory errors and crash.

To debug this issue I prepared a test server and compiled opensips as per https://www.opensips.org/Documentation/TroubleShooting-OutOfMem

I made only one single call (which was rejected by opensips as it was not authorized user) and I saw private free memory decreased. I was hoping since transaction is done ideally it should release memory and should show me same memory as startup but it did not. I verified this with many call attempts and i see free memory is always decreasing slowly.

I used kill -SIGUSR1 <lowest pid> to create memory dump. But i am unable to make sense of it. It shows log like ...

r  6 07:29:19 Server3021 opensips[13276]: Memory status (pkg):
Mar  6 07:29:19 Server3021 opensips[13276]: qm_status (0x7f5b8ebba010):
Mar  6 07:29:19 Server3021 opensips[13276]:  heap size= 4194304
Mar  6 07:29:19 Server3021 opensips[13276]:  used= 346768, used+overhead=848792, free=3345512
Mar  6 07:29:19 Server3021 opensips[13276]:  max used (+overhead)= 931920
Mar  6 07:29:19 Server3021 opensips[13276]: dumping all alloc'ed. fragments:
Mar  6 07:29:19 Server3021 opensips[13276]:       0. N  address=0x7f5b8ebef528 frag=0x7f5b8ebef4f8 size=40 used=1
Mar  6 07:29:19 Server3021 opensips[13276]:             alloc'd from script_cb.c: add_callback(60)
Mar  6 07:29:19 Server3021 opensips[13276]:         start check=f0f0f0f0f0f0f0f0, end check= c0c0c0c0c0c0c0c0, abcdefedabcdefed
Mar  6 07:29:19 Server3021 opensips[13276]:       1. N  address=0x7f5b8ebef5b0 

I pasted only few lines in this mail. What should be my next step?...How can i really trace what is wrong in my script or any other memory leak?



_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
_______________________________________________ Users mailing list [hidden email] http://lists.opensips.org/cgi-bin/mailman/listinfo/users
_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Quest to find memory leak

John Nash
with every call attempt it decreases. I tried some changes by rejecting invite before drouting call (That means after auth , dispatcher) and found memory is stable but when drouting sends Invite to external gateway and external gateway rejects it. Then this issue happens.

Inuse transactions and active dialogs also 0. Somthing wrong happening in handling of failure replies. But apart from use_next_gw and setting some avps for CDR not much going on there.

On Mon, Mar 6, 2017 at 5:54 PM, Răzvan Crainea <[hidden email]> wrote:
Ok, so it is the first listener for the private IP that leaks. Next, is the memory stabilizing in time? Or it is continously decreasing?
Yes, that's how you should make the dump.

Best regards,
Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com
On 03/06/2017 10:57 AM, John Nash wrote:
Dear Razvan,

Below is the info on my processes
Process::  ID=0 PID=17351 Type=attendant
Process::  ID=1 PID=17352 Type=MI FIFO
Process::  ID=2 PID=17353 Type=MI Datagram
Process::  ID=3 PID=17354 Type=time_keeper
Process::  ID=4 PID=17355 Type=timer
Process::  ID=5 PID=17356 Type=SIP receiver udp:1.1.1.1:9094 
Process::  ID=6 PID=17357 Type=SIP receiver udp:1.1.1.1:5060 
Process::  ID=7 PID=17358 Type=SIP receiver udp:192.168.45.5:5064 
Process::  ID=8 PID=17359 Type=Timer handler

1.1.1.1 is public IP (I changed). The decrease in memory I see is for Process::  ID=7 PID=17358 mainly. My call flow is as following 

- New Invite hits the opensips on 1.1.1.1:9094
- Apart from message validity checks I query DB to check if its a valid user (Using local cache also there)
- Create dialog, Topology_hiding functions are called along with some avp population
- Using dispatcher ds_select_domain Call sent to udp:192.168.45.2:7060 (using force socket). This 192.168.45.2:7060 is actually freeswitch 
- Call again comes back to opensips on udp:192.168.45.5:5064
- New dialog is created and topology_hiding is called
- Drouting function do_routing is called which tries one gateway and fails


Dump i need to create with memlog=4 memdump=1 right?









On Mon, Mar 6, 2017 at 2:05 PM, Răzvan Crainea <[hidden email]> wrote:
Hi, John!

Transactions are stored in shared memory, not in the private one. So the possible leak you are facing its not related to transactions.
During runtime, OpenSIPS might resize some internal structures, which may lead to increase memory usage. However, after a while, these allocations should stabilize
.
Can you post the output of the kill -SIGUSR1 on pastebin so we can take a look? Also, what type of process is the one you are seeing the leak into? You can find out using the 'opensipsctl ps' command.

Best regards,
Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com
On 03/06/2017 09:55 AM, John Nash wrote:
I am using OpenSIPS (2.1.5 (x86_64/linux)) in production. I observed private memory is decreasing constantly for one process mainly and ultimately leading to memory errors and crash.

To debug this issue I prepared a test server and compiled opensips as per https://www.opensips.org/Documentation/TroubleShooting-OutOfMem

I made only one single call (which was rejected by opensips as it was not authorized user) and I saw private free memory decreased. I was hoping since transaction is done ideally it should release memory and should show me same memory as startup but it did not. I verified this with many call attempts and i see free memory is always decreasing slowly.

I used kill -SIGUSR1 <lowest pid> to create memory dump. But i am unable to make sense of it. It shows log like ...

r  6 07:29:19 Server3021 opensips[13276]: Memory status (pkg):
Mar  6 07:29:19 Server3021 opensips[13276]: qm_status (0x7f5b8ebba010):
Mar  6 07:29:19 Server3021 opensips[13276]:  heap size= 4194304
Mar  6 07:29:19 Server3021 opensips[13276]:  used= 346768, used+overhead=848792, free=3345512
Mar  6 07:29:19 Server3021 opensips[13276]:  max used (+overhead)= 931920
Mar  6 07:29:19 Server3021 opensips[13276]: dumping all alloc'ed. fragments:
Mar  6 07:29:19 Server3021 opensips[13276]:       0. N  address=0x7f5b8ebef528 frag=0x7f5b8ebef4f8 size=40 used=1
Mar  6 07:29:19 Server3021 opensips[13276]:             alloc'd from script_cb.c: add_callback(60)
Mar  6 07:29:19 Server3021 opensips[13276]:         start check=f0f0f0f0f0f0f0f0, end check= c0c0c0c0c0c0c0c0, abcdefedabcdefed
Mar  6 07:29:19 Server3021 opensips[13276]:       1. N  address=0x7f5b8ebef5b0 

I pasted only few lines in this mail. What should be my next step?...How can i really trace what is wrong in my script or any other memory leak?



_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
_______________________________________________ Users mailing list [hidden email] http://lists.opensips.org/cgi-bin/mailman/listinfo/users
_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users



_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Quest to find memory leak

Razvan Crainea-3
None of the "actions" you are talking about have big impact on private memory, but the shared one. Better do the dump and send it over to point out what is "eating" memory.

Best regards,
Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com
On 03/06/2017 02:39 PM, John Nash wrote:
with every call attempt it decreases. I tried some changes by rejecting invite before drouting call (That means after auth , dispatcher) and found memory is stable but when drouting sends Invite to external gateway and external gateway rejects it. Then this issue happens.

Inuse transactions and active dialogs also 0. Somthing wrong happening in handling of failure replies. But apart from use_next_gw and setting some avps for CDR not much going on there.

On Mon, Mar 6, 2017 at 5:54 PM, Răzvan Crainea <[hidden email]> wrote:
Ok, so it is the first listener for the private IP that leaks. Next, is the memory stabilizing in time? Or it is continously decreasing?
Yes, that's how you should make the dump.

Best regards,
Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com
On 03/06/2017 10:57 AM, John Nash wrote:
Dear Razvan,

Below is the info on my processes
Process::  ID=0 PID=17351 Type=attendant
Process::  ID=1 PID=17352 Type=MI FIFO
Process::  ID=2 PID=17353 Type=MI Datagram
Process::  ID=3 PID=17354 Type=time_keeper
Process::  ID=4 PID=17355 Type=timer
Process::  ID=5 PID=17356 Type=SIP receiver udp:1.1.1.1:9094 
Process::  ID=6 PID=17357 Type=SIP receiver udp:1.1.1.1:5060 
Process::  ID=7 PID=17358 Type=SIP receiver udp:192.168.45.5:5064 
Process::  ID=8 PID=17359 Type=Timer handler

1.1.1.1 is public IP (I changed). The decrease in memory I see is for Process::  ID=7 PID=17358 mainly. My call flow is as following 

- New Invite hits the opensips on 1.1.1.1:9094
- Apart from message validity checks I query DB to check if its a valid user (Using local cache also there)
- Create dialog, Topology_hiding functions are called along with some avp population
- Using dispatcher ds_select_domain Call sent to udp:192.168.45.2:7060 (using force socket). This 192.168.45.2:7060 is actually freeswitch 
- Call again comes back to opensips on udp:192.168.45.5:5064
- New dialog is created and topology_hiding is called
- Drouting function do_routing is called which tries one gateway and fails


Dump i need to create with memlog=4 memdump=1 right?









On Mon, Mar 6, 2017 at 2:05 PM, Răzvan Crainea <[hidden email]> wrote:
Hi, John!

Transactions are stored in shared memory, not in the private one. So the possible leak you are facing its not related to transactions.
During runtime, OpenSIPS might resize some internal structures, which may lead to increase memory usage. However, after a while, these allocations should stabilize
.
Can you post the output of the kill -SIGUSR1 on pastebin so we can take a look? Also, what type of process is the one you are seeing the leak into? You can find out using the 'opensipsctl ps' command.

Best regards,
Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com
On 03/06/2017 09:55 AM, John Nash wrote:
I am using OpenSIPS (2.1.5 (x86_64/linux)) in production. I observed private memory is decreasing constantly for one process mainly and ultimately leading to memory errors and crash.

To debug this issue I prepared a test server and compiled opensips as per https://www.opensips.org/Documentation/TroubleShooting-OutOfMem

I made only one single call (which was rejected by opensips as it was not authorized user) and I saw private free memory decreased. I was hoping since transaction is done ideally it should release memory and should show me same memory as startup but it did not. I verified this with many call attempts and i see free memory is always decreasing slowly.

I used kill -SIGUSR1 <lowest pid> to create memory dump. But i am unable to make sense of it. It shows log like ...

r  6 07:29:19 Server3021 opensips[13276]: Memory status (pkg):
Mar  6 07:29:19 Server3021 opensips[13276]: qm_status (0x7f5b8ebba010):
Mar  6 07:29:19 Server3021 opensips[13276]:  heap size= 4194304
Mar  6 07:29:19 Server3021 opensips[13276]:  used= 346768, used+overhead=848792, free=3345512
Mar  6 07:29:19 Server3021 opensips[13276]:  max used (+overhead)= 931920
Mar  6 07:29:19 Server3021 opensips[13276]: dumping all alloc'ed. fragments:
Mar  6 07:29:19 Server3021 opensips[13276]:       0. N  address=0x7f5b8ebef528 frag=0x7f5b8ebef4f8 size=40 used=1
Mar  6 07:29:19 Server3021 opensips[13276]:             alloc'd from script_cb.c: add_callback(60)
Mar  6 07:29:19 Server3021 opensips[13276]:         start check=f0f0f0f0f0f0f0f0, end check= c0c0c0c0c0c0c0c0, abcdefedabcdefed
Mar  6 07:29:19 Server3021 opensips[13276]:       1. N  address=0x7f5b8ebef5b0 

I pasted only few lines in this mail. What should be my next step?...How can i really trace what is wrong in my script or any other memory leak?



_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
_______________________________________________ Users mailing list [hidden email] http://lists.opensips.org/cgi-bin/mailman/listinfo/users
_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
_______________________________________________ Users mailing list [hidden email] http://lists.opensips.org/cgi-bin/mailman/listinfo/users
_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Quest to find memory leak

John Nash

On Mon, Mar 6, 2017 at 6:20 PM, Răzvan Crainea <[hidden email]> wrote:
None of the "actions" you are talking about have big impact on private memory, but the shared one. Better do the dump and send it over to point out what is "eating" memory.

Best regards,
Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com
On 03/06/2017 02:39 PM, John Nash wrote:
with every call attempt it decreases. I tried some changes by rejecting invite before drouting call (That means after auth , dispatcher) and found memory is stable but when drouting sends Invite to external gateway and external gateway rejects it. Then this issue happens.

Inuse transactions and active dialogs also 0. Somthing wrong happening in handling of failure replies. But apart from use_next_gw and setting some avps for CDR not much going on there.

On Mon, Mar 6, 2017 at 5:54 PM, Răzvan Crainea <[hidden email]> wrote:
Ok, so it is the first listener for the private IP that leaks. Next, is the memory stabilizing in time? Or it is continously decreasing?
Yes, that's how you should make the dump.

Best regards,
Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com
On 03/06/2017 10:57 AM, John Nash wrote:
Dear Razvan,

Below is the info on my processes
Process::  ID=0 PID=17351 Type=attendant
Process::  ID=1 PID=17352 Type=MI FIFO
Process::  ID=2 PID=17353 Type=MI Datagram
Process::  ID=3 PID=17354 Type=time_keeper
Process::  ID=4 PID=17355 Type=timer
Process::  ID=5 PID=17356 Type=SIP receiver udp:1.1.1.1:9094 
Process::  ID=6 PID=17357 Type=SIP receiver udp:1.1.1.1:5060 
Process::  ID=7 PID=17358 Type=SIP receiver udp:192.168.45.5:5064 
Process::  ID=8 PID=17359 Type=Timer handler

1.1.1.1 is public IP (I changed). The decrease in memory I see is for Process::  ID=7 PID=17358 mainly. My call flow is as following 

- New Invite hits the opensips on 1.1.1.1:9094
- Apart from message validity checks I query DB to check if its a valid user (Using local cache also there)
- Create dialog, Topology_hiding functions are called along with some avp population
- Using dispatcher ds_select_domain Call sent to udp:192.168.45.2:7060 (using force socket). This 192.168.45.2:7060 is actually freeswitch 
- Call again comes back to opensips on udp:192.168.45.5:5064
- New dialog is created and topology_hiding is called
- Drouting function do_routing is called which tries one gateway and fails


Dump i need to create with memlog=4 memdump=1 right?









On Mon, Mar 6, 2017 at 2:05 PM, Răzvan Crainea <[hidden email]> wrote:
Hi, John!

Transactions are stored in shared memory, not in the private one. So the possible leak you are facing its not related to transactions.
During runtime, OpenSIPS might resize some internal structures, which may lead to increase memory usage. However, after a while, these allocations should stabilize
.
Can you post the output of the kill -SIGUSR1 on pastebin so we can take a look? Also, what type of process is the one you are seeing the leak into? You can find out using the 'opensipsctl ps' command.

Best regards,
Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com
On 03/06/2017 09:55 AM, John Nash wrote:
I am using OpenSIPS (2.1.5 (x86_64/linux)) in production. I observed private memory is decreasing constantly for one process mainly and ultimately leading to memory errors and crash.

To debug this issue I prepared a test server and compiled opensips as per https://www.opensips.org/Documentation/TroubleShooting-OutOfMem

I made only one single call (which was rejected by opensips as it was not authorized user) and I saw private free memory decreased. I was hoping since transaction is done ideally it should release memory and should show me same memory as startup but it did not. I verified this with many call attempts and i see free memory is always decreasing slowly.

I used kill -SIGUSR1 <lowest pid> to create memory dump. But i am unable to make sense of it. It shows log like ...

r  6 07:29:19 Server3021 opensips[13276]: Memory status (pkg):
Mar  6 07:29:19 Server3021 opensips[13276]: qm_status (0x7f5b8ebba010):
Mar  6 07:29:19 Server3021 opensips[13276]:  heap size= 4194304
Mar  6 07:29:19 Server3021 opensips[13276]:  used= 346768, used+overhead=848792, free=3345512
Mar  6 07:29:19 Server3021 opensips[13276]:  max used (+overhead)= 931920
Mar  6 07:29:19 Server3021 opensips[13276]: dumping all alloc'ed. fragments:
Mar  6 07:29:19 Server3021 opensips[13276]:       0. N  address=0x7f5b8ebef528 frag=0x7f5b8ebef4f8 size=40 used=1
Mar  6 07:29:19 Server3021 opensips[13276]:             alloc'd from script_cb.c: add_callback(60)
Mar  6 07:29:19 Server3021 opensips[13276]:         start check=f0f0f0f0f0f0f0f0, end check= c0c0c0c0c0c0c0c0, abcdefedabcdefed
Mar  6 07:29:19 Server3021 opensips[13276]:       1. N  address=0x7f5b8ebef5b0 

I pasted only few lines in this mail. What should be my next step?...How can i really trace what is wrong in my script or any other memory leak?



_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
_______________________________________________ Users mailing list [hidden email] http://lists.opensips.org/cgi-bin/mailman/listinfo/users
_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
_______________________________________________ Users mailing list [hidden email] http://lists.opensips.org/cgi-bin/mailman/listinfo/users
_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users



_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Quest to find memory leak

Razvan Crainea-3
Hi, John!

From the dump you sent, I don't see any leaks. Perhaps some of those fragments increase over time. Can you make a memory dump after the server runs some time, like after it gets 100 messages?

Best regards,
Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com
On 03/06/2017 03:02 PM, John Nash wrote:

On Mon, Mar 6, 2017 at 6:20 PM, Răzvan Crainea <[hidden email]> wrote:
None of the "actions" you are talking about have big impact on private memory, but the shared one. Better do the dump and send it over to point out what is "eating" memory.

Best regards,
Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com
On 03/06/2017 02:39 PM, John Nash wrote:
with every call attempt it decreases. I tried some changes by rejecting invite before drouting call (That means after auth , dispatcher) and found memory is stable but when drouting sends Invite to external gateway and external gateway rejects it. Then this issue happens.

Inuse transactions and active dialogs also 0. Somthing wrong happening in handling of failure replies. But apart from use_next_gw and setting some avps for CDR not much going on there.

On Mon, Mar 6, 2017 at 5:54 PM, Răzvan Crainea <[hidden email]> wrote:
Ok, so it is the first listener for the private IP that leaks. Next, is the memory stabilizing in time? Or it is continously decreasing?
Yes, that's how you should make the dump.

Best regards,
Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com
On 03/06/2017 10:57 AM, John Nash wrote:
Dear Razvan,

Below is the info on my processes
Process::  ID=0 PID=17351 Type=attendant
Process::  ID=1 PID=17352 Type=MI FIFO
Process::  ID=2 PID=17353 Type=MI Datagram
Process::  ID=3 PID=17354 Type=time_keeper
Process::  ID=4 PID=17355 Type=timer
Process::  ID=5 PID=17356 Type=SIP receiver udp:1.1.1.1:9094 
Process::  ID=6 PID=17357 Type=SIP receiver udp:1.1.1.1:5060 
Process::  ID=7 PID=17358 Type=SIP receiver udp:192.168.45.5:5064 
Process::  ID=8 PID=17359 Type=Timer handler

1.1.1.1 is public IP (I changed). The decrease in memory I see is for Process::  ID=7 PID=17358 mainly. My call flow is as following 

- New Invite hits the opensips on 1.1.1.1:9094
- Apart from message validity checks I query DB to check if its a valid user (Using local cache also there)
- Create dialog, Topology_hiding functions are called along with some avp population
- Using dispatcher ds_select_domain Call sent to udp:192.168.45.2:7060 (using force socket). This 192.168.45.2:7060 is actually freeswitch 
- Call again comes back to opensips on udp:192.168.45.5:5064
- New dialog is created and topology_hiding is called
- Drouting function do_routing is called which tries one gateway and fails


Dump i need to create with memlog=4 memdump=1 right?









On Mon, Mar 6, 2017 at 2:05 PM, Răzvan Crainea <[hidden email]> wrote:
Hi, John!

Transactions are stored in shared memory, not in the private one. So the possible leak you are facing its not related to transactions.
During runtime, OpenSIPS might resize some internal structures, which may lead to increase memory usage. However, after a while, these allocations should stabilize
.
Can you post the output of the kill -SIGUSR1 on pastebin so we can take a look? Also, what type of process is the one you are seeing the leak into? You can find out using the 'opensipsctl ps' command.

Best regards,
Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com
On 03/06/2017 09:55 AM, John Nash wrote:
I am using OpenSIPS (2.1.5 (x86_64/linux)) in production. I observed private memory is decreasing constantly for one process mainly and ultimately leading to memory errors and crash.

To debug this issue I prepared a test server and compiled opensips as per https://www.opensips.org/Documentation/TroubleShooting-OutOfMem

I made only one single call (which was rejected by opensips as it was not authorized user) and I saw private free memory decreased. I was hoping since transaction is done ideally it should release memory and should show me same memory as startup but it did not. I verified this with many call attempts and i see free memory is always decreasing slowly.

I used kill -SIGUSR1 <lowest pid> to create memory dump. But i am unable to make sense of it. It shows log like ...

r  6 07:29:19 Server3021 opensips[13276]: Memory status (pkg):
Mar  6 07:29:19 Server3021 opensips[13276]: qm_status (0x7f5b8ebba010):
Mar  6 07:29:19 Server3021 opensips[13276]:  heap size= 4194304
Mar  6 07:29:19 Server3021 opensips[13276]:  used= 346768, used+overhead=848792, free=3345512
Mar  6 07:29:19 Server3021 opensips[13276]:  max used (+overhead)= 931920
Mar  6 07:29:19 Server3021 opensips[13276]: dumping all alloc'ed. fragments:
Mar  6 07:29:19 Server3021 opensips[13276]:       0. N  address=0x7f5b8ebef528 frag=0x7f5b8ebef4f8 size=40 used=1
Mar  6 07:29:19 Server3021 opensips[13276]:             alloc'd from script_cb.c: add_callback(60)
Mar  6 07:29:19 Server3021 opensips[13276]:         start check=f0f0f0f0f0f0f0f0, end check= c0c0c0c0c0c0c0c0, abcdefedabcdefed
Mar  6 07:29:19 Server3021 opensips[13276]:       1. N  address=0x7f5b8ebef5b0 

I pasted only few lines in this mail. What should be my next step?...How can i really trace what is wrong in my script or any other memory leak?



_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
_______________________________________________ Users mailing list [hidden email] http://lists.opensips.org/cgi-bin/mailman/listinfo/users
_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
_______________________________________________ Users mailing list [hidden email] http://lists.opensips.org/cgi-bin/mailman/listinfo/users
_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
_______________________________________________ Users mailing list [hidden email] http://lists.opensips.org/cgi-bin/mailman/listinfo/users
_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Quest to find memory leak

John Nash
Ok will try that. Is it possible that wrong usage of drouting may cause this to happen instead of actual leak?... What are the things private memory is used for?

On Mon, Mar 6, 2017 at 6:48 PM, Răzvan Crainea <[hidden email]> wrote:
Hi, John!

From the dump you sent, I don't see any leaks. Perhaps some of those fragments increase over time. Can you make a memory dump after the server runs some time, like after it gets 100 messages?

Best regards,
Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com
On 03/06/2017 03:02 PM, John Nash wrote:

On Mon, Mar 6, 2017 at 6:20 PM, Răzvan Crainea <[hidden email]> wrote:
None of the "actions" you are talking about have big impact on private memory, but the shared one. Better do the dump and send it over to point out what is "eating" memory.

Best regards,
Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com
On 03/06/2017 02:39 PM, John Nash wrote:
with every call attempt it decreases. I tried some changes by rejecting invite before drouting call (That means after auth , dispatcher) and found memory is stable but when drouting sends Invite to external gateway and external gateway rejects it. Then this issue happens.

Inuse transactions and active dialogs also 0. Somthing wrong happening in handling of failure replies. But apart from use_next_gw and setting some avps for CDR not much going on there.

On Mon, Mar 6, 2017 at 5:54 PM, Răzvan Crainea <[hidden email]> wrote:
Ok, so it is the first listener for the private IP that leaks. Next, is the memory stabilizing in time? Or it is continously decreasing?
Yes, that's how you should make the dump.

Best regards,
Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com
On 03/06/2017 10:57 AM, John Nash wrote:
Dear Razvan,

Below is the info on my processes
Process::  ID=0 PID=17351 Type=attendant
Process::  ID=1 PID=17352 Type=MI FIFO
Process::  ID=2 PID=17353 Type=MI Datagram
Process::  ID=3 PID=17354 Type=time_keeper
Process::  ID=4 PID=17355 Type=timer
Process::  ID=5 PID=17356 Type=SIP receiver udp:1.1.1.1:9094 
Process::  ID=6 PID=17357 Type=SIP receiver udp:1.1.1.1:5060 
Process::  ID=7 PID=17358 Type=SIP receiver udp:192.168.45.5:5064 
Process::  ID=8 PID=17359 Type=Timer handler

1.1.1.1 is public IP (I changed). The decrease in memory I see is for Process::  ID=7 PID=17358 mainly. My call flow is as following 

- New Invite hits the opensips on 1.1.1.1:9094
- Apart from message validity checks I query DB to check if its a valid user (Using local cache also there)
- Create dialog, Topology_hiding functions are called along with some avp population
- Using dispatcher ds_select_domain Call sent to udp:192.168.45.2:7060 (using force socket). This 192.168.45.2:7060 is actually freeswitch 
- Call again comes back to opensips on udp:192.168.45.5:5064
- New dialog is created and topology_hiding is called
- Drouting function do_routing is called which tries one gateway and fails


Dump i need to create with memlog=4 memdump=1 right?









On Mon, Mar 6, 2017 at 2:05 PM, Răzvan Crainea <[hidden email]> wrote:
Hi, John!

Transactions are stored in shared memory, not in the private one. So the possible leak you are facing its not related to transactions.
During runtime, OpenSIPS might resize some internal structures, which may lead to increase memory usage. However, after a while, these allocations should stabilize
.
Can you post the output of the kill -SIGUSR1 on pastebin so we can take a look? Also, what type of process is the one you are seeing the leak into? You can find out using the 'opensipsctl ps' command.

Best regards,
Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com
On 03/06/2017 09:55 AM, John Nash wrote:
I am using OpenSIPS (2.1.5 (x86_64/linux)) in production. I observed private memory is decreasing constantly for one process mainly and ultimately leading to memory errors and crash.

To debug this issue I prepared a test server and compiled opensips as per https://www.opensips.org/Documentation/TroubleShooting-OutOfMem

I made only one single call (which was rejected by opensips as it was not authorized user) and I saw private free memory decreased. I was hoping since transaction is done ideally it should release memory and should show me same memory as startup but it did not. I verified this with many call attempts and i see free memory is always decreasing slowly.

I used kill -SIGUSR1 <lowest pid> to create memory dump. But i am unable to make sense of it. It shows log like ...

r  6 07:29:19 Server3021 opensips[13276]: Memory status (pkg):
Mar  6 07:29:19 Server3021 opensips[13276]: qm_status (0x7f5b8ebba010):
Mar  6 07:29:19 Server3021 opensips[13276]:  heap size= 4194304
Mar  6 07:29:19 Server3021 opensips[13276]:  used= 346768, used+overhead=848792, free=3345512
Mar  6 07:29:19 Server3021 opensips[13276]:  max used (+overhead)= 931920
Mar  6 07:29:19 Server3021 opensips[13276]: dumping all alloc'ed. fragments:
Mar  6 07:29:19 Server3021 opensips[13276]:       0. N  address=0x7f5b8ebef528 frag=0x7f5b8ebef4f8 size=40 used=1
Mar  6 07:29:19 Server3021 opensips[13276]:             alloc'd from script_cb.c: add_callback(60)
Mar  6 07:29:19 Server3021 opensips[13276]:         start check=f0f0f0f0f0f0f0f0, end check= c0c0c0c0c0c0c0c0, abcdefedabcdefed
Mar  6 07:29:19 Server3021 opensips[13276]:       1. N  address=0x7f5b8ebef5b0 

I pasted only few lines in this mail. What should be my next step?...How can i really trace what is wrong in my script or any other memory leak?



_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
_______________________________________________ Users mailing list [hidden email] http://lists.opensips.org/cgi-bin/mailman/listinfo/users
_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
_______________________________________________ Users mailing list [hidden email] http://lists.opensips.org/cgi-bin/mailman/listinfo/users
_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
_______________________________________________ Users mailing list [hidden email] http://lists.opensips.org/cgi-bin/mailman/listinfo/users
_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users



_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Quest to find memory leak

John Nash
here is another trace 

I see lot of alloc request but no free.

On Mon, Mar 6, 2017 at 6:57 PM, John Nash <[hidden email]> wrote:
Ok will try that. Is it possible that wrong usage of drouting may cause this to happen instead of actual leak?... What are the things private memory is used for?

On Mon, Mar 6, 2017 at 6:48 PM, Răzvan Crainea <[hidden email]> wrote:
Hi, John!

From the dump you sent, I don't see any leaks. Perhaps some of those fragments increase over time. Can you make a memory dump after the server runs some time, like after it gets 100 messages?

Best regards,
Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com
On 03/06/2017 03:02 PM, John Nash wrote:

On Mon, Mar 6, 2017 at 6:20 PM, Răzvan Crainea <[hidden email]> wrote:
None of the "actions" you are talking about have big impact on private memory, but the shared one. Better do the dump and send it over to point out what is "eating" memory.

Best regards,
Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com
On 03/06/2017 02:39 PM, John Nash wrote:
with every call attempt it decreases. I tried some changes by rejecting invite before drouting call (That means after auth , dispatcher) and found memory is stable but when drouting sends Invite to external gateway and external gateway rejects it. Then this issue happens.

Inuse transactions and active dialogs also 0. Somthing wrong happening in handling of failure replies. But apart from use_next_gw and setting some avps for CDR not much going on there.

On Mon, Mar 6, 2017 at 5:54 PM, Răzvan Crainea <[hidden email]> wrote:
Ok, so it is the first listener for the private IP that leaks. Next, is the memory stabilizing in time? Or it is continously decreasing?
Yes, that's how you should make the dump.

Best regards,
Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com
On 03/06/2017 10:57 AM, John Nash wrote:
Dear Razvan,

Below is the info on my processes
Process::  ID=0 PID=17351 Type=attendant
Process::  ID=1 PID=17352 Type=MI FIFO
Process::  ID=2 PID=17353 Type=MI Datagram
Process::  ID=3 PID=17354 Type=time_keeper
Process::  ID=4 PID=17355 Type=timer
Process::  ID=5 PID=17356 Type=SIP receiver udp:1.1.1.1:9094 
Process::  ID=6 PID=17357 Type=SIP receiver udp:1.1.1.1:5060 
Process::  ID=7 PID=17358 Type=SIP receiver udp:192.168.45.5:5064 
Process::  ID=8 PID=17359 Type=Timer handler

1.1.1.1 is public IP (I changed). The decrease in memory I see is for Process::  ID=7 PID=17358 mainly. My call flow is as following 

- New Invite hits the opensips on 1.1.1.1:9094
- Apart from message validity checks I query DB to check if its a valid user (Using local cache also there)
- Create dialog, Topology_hiding functions are called along with some avp population
- Using dispatcher ds_select_domain Call sent to udp:192.168.45.2:7060 (using force socket). This 192.168.45.2:7060 is actually freeswitch 
- Call again comes back to opensips on udp:192.168.45.5:5064
- New dialog is created and topology_hiding is called
- Drouting function do_routing is called which tries one gateway and fails


Dump i need to create with memlog=4 memdump=1 right?









On Mon, Mar 6, 2017 at 2:05 PM, Răzvan Crainea <[hidden email]> wrote:
Hi, John!

Transactions are stored in shared memory, not in the private one. So the possible leak you are facing its not related to transactions.
During runtime, OpenSIPS might resize some internal structures, which may lead to increase memory usage. However, after a while, these allocations should stabilize
.
Can you post the output of the kill -SIGUSR1 on pastebin so we can take a look? Also, what type of process is the one you are seeing the leak into? You can find out using the 'opensipsctl ps' command.

Best regards,
Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com
On 03/06/2017 09:55 AM, John Nash wrote:
I am using OpenSIPS (2.1.5 (x86_64/linux)) in production. I observed private memory is decreasing constantly for one process mainly and ultimately leading to memory errors and crash.

To debug this issue I prepared a test server and compiled opensips as per https://www.opensips.org/Documentation/TroubleShooting-OutOfMem

I made only one single call (which was rejected by opensips as it was not authorized user) and I saw private free memory decreased. I was hoping since transaction is done ideally it should release memory and should show me same memory as startup but it did not. I verified this with many call attempts and i see free memory is always decreasing slowly.

I used kill -SIGUSR1 <lowest pid> to create memory dump. But i am unable to make sense of it. It shows log like ...

r  6 07:29:19 Server3021 opensips[13276]: Memory status (pkg):
Mar  6 07:29:19 Server3021 opensips[13276]: qm_status (0x7f5b8ebba010):
Mar  6 07:29:19 Server3021 opensips[13276]:  heap size= 4194304
Mar  6 07:29:19 Server3021 opensips[13276]:  used= 346768, used+overhead=848792, free=3345512
Mar  6 07:29:19 Server3021 opensips[13276]:  max used (+overhead)= 931920
Mar  6 07:29:19 Server3021 opensips[13276]: dumping all alloc'ed. fragments:
Mar  6 07:29:19 Server3021 opensips[13276]:       0. N  address=0x7f5b8ebef528 frag=0x7f5b8ebef4f8 size=40 used=1
Mar  6 07:29:19 Server3021 opensips[13276]:             alloc'd from script_cb.c: add_callback(60)
Mar  6 07:29:19 Server3021 opensips[13276]:         start check=f0f0f0f0f0f0f0f0, end check= c0c0c0c0c0c0c0c0, abcdefedabcdefed
Mar  6 07:29:19 Server3021 opensips[13276]:       1. N  address=0x7f5b8ebef5b0 

I pasted only few lines in this mail. What should be my next step?...How can i really trace what is wrong in my script or any other memory leak?



_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
_______________________________________________ Users mailing list [hidden email] http://lists.opensips.org/cgi-bin/mailman/listinfo/users
_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
_______________________________________________ Users mailing list [hidden email] http://lists.opensips.org/cgi-bin/mailman/listinfo/users
_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
_______________________________________________ Users mailing list [hidden email] http://lists.opensips.org/cgi-bin/mailman/listinfo/users
_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users




_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Quest to find memory leak

Razvan Crainea-3
Hi, John!

Again, this trace doesn't show any leak.
Are you sure you are having a private memory leak and not a shared
memory leak?

Best regards,

Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com
On 03/06/2017 08:09 PM, John Nash wrote:
here is another trace 

I see lot of alloc request but no free.

On Mon, Mar 6, 2017 at 6:57 PM, John Nash <[hidden email]> wrote:
Ok will try that. Is it possible that wrong usage of drouting may cause this to happen instead of actual leak?... What are the things private memory is used for?

On Mon, Mar 6, 2017 at 6:48 PM, Răzvan Crainea <[hidden email]> wrote:
Hi, John!

From the dump you sent, I don't see any leaks. Perhaps some of those fragments increase over time. Can you make a memory dump after the server runs some time, like after it gets 100 messages?

Best regards,
Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com
On 03/06/2017 03:02 PM, John Nash wrote:

On Mon, Mar 6, 2017 at 6:20 PM, Răzvan Crainea <[hidden email]> wrote:
None of the "actions" you are talking about have big impact on private memory, but the shared one. Better do the dump and send it over to point out what is "eating" memory.

Best regards,
Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com
On 03/06/2017 02:39 PM, John Nash wrote:
with every call attempt it decreases. I tried some changes by rejecting invite before drouting call (That means after auth , dispatcher) and found memory is stable but when drouting sends Invite to external gateway and external gateway rejects it. Then this issue happens.

Inuse transactions and active dialogs also 0. Somthing wrong happening in handling of failure replies. But apart from use_next_gw and setting some avps for CDR not much going on there.

On Mon, Mar 6, 2017 at 5:54 PM, Răzvan Crainea <[hidden email]> wrote:
Ok, so it is the first listener for the private IP that leaks. Next, is the memory stabilizing in time? Or it is continously decreasing?
Yes, that's how you should make the dump.

Best regards,
Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Quest to find memory leak

John Nash
when I check stats after a call attempt pkmem:7-free_size:: 3304280

In this entry with every call I see a drop of 1000 bytes around and this never restores.

On Tue, Mar 7, 2017 at 2:16 PM, Răzvan Crainea <[hidden email]> wrote:
Hi, John!

Again, this trace doesn't show any leak.
Are you sure you are having a private memory leak and not a shared
memory leak?

Best regards,

Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com
On 03/06/2017 08:09 PM, John Nash wrote:
here is another trace 

I see lot of alloc request but no free.

On Mon, Mar 6, 2017 at 6:57 PM, John Nash <[hidden email]> wrote:
Ok will try that. Is it possible that wrong usage of drouting may cause this to happen instead of actual leak?... What are the things private memory is used for?

On Mon, Mar 6, 2017 at 6:48 PM, Răzvan Crainea <[hidden email]> wrote:
Hi, John!

From the dump you sent, I don't see any leaks. Perhaps some of those fragments increase over time. Can you make a memory dump after the server runs some time, like after it gets 100 messages?

Best regards,
Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com
On 03/06/2017 03:02 PM, John Nash wrote:

On Mon, Mar 6, 2017 at 6:20 PM, Răzvan Crainea <[hidden email]> wrote:
None of the "actions" you are talking about have big impact on private memory, but the shared one. Better do the dump and send it over to point out what is "eating" memory.

Best regards,
Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com
On 03/06/2017 02:39 PM, John Nash wrote:
with every call attempt it decreases. I tried some changes by rejecting invite before drouting call (That means after auth , dispatcher) and found memory is stable but when drouting sends Invite to external gateway and external gateway rejects it. Then this issue happens.

Inuse transactions and active dialogs also 0. Somthing wrong happening in handling of failure replies. But apart from use_next_gw and setting some avps for CDR not much going on there.

On Mon, Mar 6, 2017 at 5:54 PM, Răzvan Crainea <[hidden email]> wrote:
Ok, so it is the first listener for the private IP that leaks. Next, is the memory stabilizing in time? Or it is continously decreasing?
Yes, that's how you should make the dump.

Best regards,
Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users



_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Quest to find memory leak

Razvan Crainea-3
So I understand that after ~3K calls, that process completely runs out of memory?
How many calls have you done before this trace: http://pastebin.com/9Ge2NEVQ

Best regards,
Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com
On 03/07/2017 11:32 AM, John Nash wrote:
when I check stats after a call attempt pkmem:7-free_size:: 3304280

In this entry with every call I see a drop of 1000 bytes around and this never restores.

On Tue, Mar 7, 2017 at 2:16 PM, Răzvan Crainea <[hidden email]> wrote:
Hi, John!

Again, this trace doesn't show any leak.
Are you sure you are having a private memory leak and not a shared
memory leak?

Best regards,

Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com
On 03/06/2017 08:09 PM, John Nash wrote:
here is another trace 

I see lot of alloc request but no free.

On Mon, Mar 6, 2017 at 6:57 PM, John Nash <[hidden email]> wrote:
Ok will try that. Is it possible that wrong usage of drouting may cause this to happen instead of actual leak?... What are the things private memory is used for?

On Mon, Mar 6, 2017 at 6:48 PM, Răzvan Crainea <[hidden email]> wrote:
Hi, John!

From the dump you sent, I don't see any leaks. Perhaps some of those fragments increase over time. Can you make a memory dump after the server runs some time, like after it gets 100 messages?

Best regards,
Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com
On 03/06/2017 03:02 PM, John Nash wrote:

On Mon, Mar 6, 2017 at 6:20 PM, Răzvan Crainea <[hidden email]> wrote:
None of the "actions" you are talking about have big impact on private memory, but the shared one. Better do the dump and send it over to point out what is "eating" memory.

Best regards,
Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com
On 03/06/2017 02:39 PM, John Nash wrote:
with every call attempt it decreases. I tried some changes by rejecting invite before drouting call (That means after auth , dispatcher) and found memory is stable but when drouting sends Invite to external gateway and external gateway rejects it. Then this issue happens.

Inuse transactions and active dialogs also 0. Somthing wrong happening in handling of failure replies. But apart from use_next_gw and setting some avps for CDR not much going on there.

On Mon, Mar 6, 2017 at 5:54 PM, Răzvan Crainea <[hidden email]> wrote:
Ok, so it is the first listener for the private IP that leaks. Next, is the memory stabilizing in time? Or it is continously decreasing?
Yes, that's how you should make the dump.

Best regards,
Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users




_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Quest to find memory leak

John Nash
only 6 or 7 calls 

On Tue, Mar 7, 2017 at 3:09 PM, Răzvan Crainea <[hidden email]> wrote:
So I understand that after ~3K calls, that process completely runs out of memory?
How many calls have you done before this trace: http://pastebin.com/9Ge2NEVQ

Best regards,
Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com
On 03/07/2017 11:32 AM, John Nash wrote:
when I check stats after a call attempt pkmem:7-free_size:: 3304280

In this entry with every call I see a drop of 1000 bytes around and this never restores.

On Tue, Mar 7, 2017 at 2:16 PM, Răzvan Crainea <[hidden email]> wrote:
Hi, John!

Again, this trace doesn't show any leak.
Are you sure you are having a private memory leak and not a shared
memory leak?

Best regards,

Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com
On 03/06/2017 08:09 PM, John Nash wrote:
here is another trace 

I see lot of alloc request but no free.

On Mon, Mar 6, 2017 at 6:57 PM, John Nash <[hidden email]> wrote:
Ok will try that. Is it possible that wrong usage of drouting may cause this to happen instead of actual leak?... What are the things private memory is used for?

On Mon, Mar 6, 2017 at 6:48 PM, Răzvan Crainea <[hidden email]> wrote:
Hi, John!

From the dump you sent, I don't see any leaks. Perhaps some of those fragments increase over time. Can you make a memory dump after the server runs some time, like after it gets 100 messages?

Best regards,
Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com
On 03/06/2017 03:02 PM, John Nash wrote:

On Mon, Mar 6, 2017 at 6:20 PM, Răzvan Crainea <[hidden email]> wrote:
None of the "actions" you are talking about have big impact on private memory, but the shared one. Better do the dump and send it over to point out what is "eating" memory.

Best regards,
Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com
On 03/06/2017 02:39 PM, John Nash wrote:
with every call attempt it decreases. I tried some changes by rejecting invite before drouting call (That means after auth , dispatcher) and found memory is stable but when drouting sends Invite to external gateway and external gateway rejects it. Then this issue happens.

Inuse transactions and active dialogs also 0. Somthing wrong happening in handling of failure replies. But apart from use_next_gw and setting some avps for CDR not much going on there.

On Mon, Mar 6, 2017 at 5:54 PM, Răzvan Crainea <[hidden email]> wrote:
Ok, so it is the first listener for the private IP that leaks. Next, is the memory stabilizing in time? Or it is continously decreasing?
Yes, that's how you should make the dump.

Best regards,
Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users




_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users



_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Quest to find memory leak

John Nash
Please note when i call do_routing in such a way that its unable to find any rules matching and reject call i do not see free memory drop. But if it finds a route, sends call to that gateway memory drops with each attempt.

On Tue, Mar 7, 2017 at 3:17 PM, John Nash <[hidden email]> wrote:
only 6 or 7 calls 

On Tue, Mar 7, 2017 at 3:09 PM, Răzvan Crainea <[hidden email]> wrote:
So I understand that after ~3K calls, that process completely runs out of memory?
How many calls have you done before this trace: http://pastebin.com/9Ge2NEVQ

Best regards,
Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com
On 03/07/2017 11:32 AM, John Nash wrote:
when I check stats after a call attempt pkmem:7-free_size:: 3304280

In this entry with every call I see a drop of 1000 bytes around and this never restores.

On Tue, Mar 7, 2017 at 2:16 PM, Răzvan Crainea <[hidden email]> wrote:
Hi, John!

Again, this trace doesn't show any leak.
Are you sure you are having a private memory leak and not a shared
memory leak?

Best regards,

Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com
On 03/06/2017 08:09 PM, John Nash wrote:
here is another trace 

I see lot of alloc request but no free.

On Mon, Mar 6, 2017 at 6:57 PM, John Nash <[hidden email]> wrote:
Ok will try that. Is it possible that wrong usage of drouting may cause this to happen instead of actual leak?... What are the things private memory is used for?

On Mon, Mar 6, 2017 at 6:48 PM, Răzvan Crainea <[hidden email]> wrote:
Hi, John!

From the dump you sent, I don't see any leaks. Perhaps some of those fragments increase over time. Can you make a memory dump after the server runs some time, like after it gets 100 messages?

Best regards,
Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com
On 03/06/2017 03:02 PM, John Nash wrote:

On Mon, Mar 6, 2017 at 6:20 PM, Răzvan Crainea <[hidden email]> wrote:
None of the "actions" you are talking about have big impact on private memory, but the shared one. Better do the dump and send it over to point out what is "eating" memory.

Best regards,
Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com
On 03/06/2017 02:39 PM, John Nash wrote:
with every call attempt it decreases. I tried some changes by rejecting invite before drouting call (That means after auth , dispatcher) and found memory is stable but when drouting sends Invite to external gateway and external gateway rejects it. Then this issue happens.

Inuse transactions and active dialogs also 0. Somthing wrong happening in handling of failure replies. But apart from use_next_gw and setting some avps for CDR not much going on there.

On Mon, Mar 6, 2017 at 5:54 PM, Răzvan Crainea <[hidden email]> wrote:
Ok, so it is the first listener for the private IP that leaks. Next, is the memory stabilizing in time? Or it is continously decreasing?
Yes, that's how you should make the dump.

Best regards,
Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users




_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users




_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Quest to find memory leak

Razvan Crainea-3
What allocator are you using? Can you post the output of 'opensips -V'?

Best regards,
Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com
On 03/07/2017 12:23 PM, John Nash wrote:
Please note when i call do_routing in such a way that its unable to find any rules matching and reject call i do not see free memory drop. But if it finds a route, sends call to that gateway memory drops with each attempt.

On Tue, Mar 7, 2017 at 3:17 PM, John Nash <[hidden email]> wrote:
only 6 or 7 calls 

On Tue, Mar 7, 2017 at 3:09 PM, Răzvan Crainea <[hidden email]> wrote:
So I understand that after ~3K calls, that process completely runs out of memory?
How many calls have you done before this trace: http://pastebin.com/9Ge2NEVQ

Best regards,
Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com
On 03/07/2017 11:32 AM, John Nash wrote:
when I check stats after a call attempt pkmem:7-free_size:: 3304280

In this entry with every call I see a drop of 1000 bytes around and this never restores.

On Tue, Mar 7, 2017 at 2:16 PM, Răzvan Crainea <[hidden email]> wrote:
Hi, John!

Again, this trace doesn't show any leak.
Are you sure you are having a private memory leak and not a shared
memory leak?

Best regards,

Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com
On 03/06/2017 08:09 PM, John Nash wrote:
here is another trace 

I see lot of alloc request but no free.

On Mon, Mar 6, 2017 at 6:57 PM, John Nash <[hidden email]> wrote:
Ok will try that. Is it possible that wrong usage of drouting may cause this to happen instead of actual leak?... What are the things private memory is used for?

On Mon, Mar 6, 2017 at 6:48 PM, Răzvan Crainea <[hidden email]> wrote:
Hi, John!

From the dump you sent, I don't see any leaks. Perhaps some of those fragments increase over time. Can you make a memory dump after the server runs some time, like after it gets 100 messages?

Best regards,
Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com
On 03/06/2017 03:02 PM, John Nash wrote:

On Mon, Mar 6, 2017 at 6:20 PM, Răzvan Crainea <[hidden email]> wrote:
None of the "actions" you are talking about have big impact on private memory, but the shared one. Better do the dump and send it over to point out what is "eating" memory.

Best regards,
Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com
On 03/06/2017 02:39 PM, John Nash wrote:
with every call attempt it decreases. I tried some changes by rejecting invite before drouting call (That means after auth , dispatcher) and found memory is stable but when drouting sends Invite to external gateway and external gateway rejects it. Then this issue happens.

Inuse transactions and active dialogs also 0. Somthing wrong happening in handling of failure replies. But apart from use_next_gw and setting some avps for CDR not much going on there.

On Mon, Mar 6, 2017 at 5:54 PM, Răzvan Crainea <[hidden email]> wrote:
Ok, so it is the first listener for the private IP that leaks. Next, is the memory stabilizing in time? Or it is continously decreasing?
Yes, that's how you should make the dump.

Best regards,
Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users




_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
_______________________________________________ Users mailing list [hidden email] http://lists.opensips.org/cgi-bin/mailman/listinfo/users
_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Quest to find memory leak

John Nash
version: opensips 2.1.5 (x86_64/linux)
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
git revision: 39b19dd
main.c compiled on 19:27:59 Mar  5 2017 with gcc 4.4.7


On Tue, Mar 7, 2017 at 4:25 PM, Răzvan Crainea <[hidden email]> wrote:
What allocator are you using? Can you post the output of 'opensips -V'?

Best regards,
Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com
On 03/07/2017 12:23 PM, John Nash wrote:
Please note when i call do_routing in such a way that its unable to find any rules matching and reject call i do not see free memory drop. But if it finds a route, sends call to that gateway memory drops with each attempt.

On Tue, Mar 7, 2017 at 3:17 PM, John Nash <[hidden email]> wrote:
only 6 or 7 calls 

On Tue, Mar 7, 2017 at 3:09 PM, Răzvan Crainea <[hidden email]> wrote:
So I understand that after ~3K calls, that process completely runs out of memory?
How many calls have you done before this trace: http://pastebin.com/9Ge2NEVQ

Best regards,
Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com
On 03/07/2017 11:32 AM, John Nash wrote:
when I check stats after a call attempt pkmem:7-free_size:: 3304280

In this entry with every call I see a drop of 1000 bytes around and this never restores.

On Tue, Mar 7, 2017 at 2:16 PM, Răzvan Crainea <[hidden email]> wrote:
Hi, John!

Again, this trace doesn't show any leak.
Are you sure you are having a private memory leak and not a shared
memory leak?

Best regards,

Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com
On 03/06/2017 08:09 PM, John Nash wrote:
here is another trace 

I see lot of alloc request but no free.

On Mon, Mar 6, 2017 at 6:57 PM, John Nash <[hidden email]> wrote:
Ok will try that. Is it possible that wrong usage of drouting may cause this to happen instead of actual leak?... What are the things private memory is used for?

On Mon, Mar 6, 2017 at 6:48 PM, Răzvan Crainea <[hidden email]> wrote:
Hi, John!

From the dump you sent, I don't see any leaks. Perhaps some of those fragments increase over time. Can you make a memory dump after the server runs some time, like after it gets 100 messages?

Best regards,
Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com
On 03/06/2017 03:02 PM, John Nash wrote:

On Mon, Mar 6, 2017 at 6:20 PM, Răzvan Crainea <[hidden email]> wrote:
None of the "actions" you are talking about have big impact on private memory, but the shared one. Better do the dump and send it over to point out what is "eating" memory.

Best regards,
Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com
On 03/06/2017 02:39 PM, John Nash wrote:
with every call attempt it decreases. I tried some changes by rejecting invite before drouting call (That means after auth , dispatcher) and found memory is stable but when drouting sends Invite to external gateway and external gateway rejects it. Then this issue happens.

Inuse transactions and active dialogs also 0. Somthing wrong happening in handling of failure replies. But apart from use_next_gw and setting some avps for CDR not much going on there.

On Mon, Mar 6, 2017 at 5:54 PM, Răzvan Crainea <[hidden email]> wrote:
Ok, so it is the first listener for the private IP that leaks. Next, is the memory stabilizing in time? Or it is continously decreasing?
Yes, that's how you should make the dump.

Best regards,
Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users




_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
_______________________________________________ Users mailing list [hidden email] http://lists.opensips.org/cgi-bin/mailman/listinfo/users
_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users



_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Quest to find memory leak

John Nash
any suggestion for me?..should i try to crash opensips by sending many calls?

On Tue, Mar 7, 2017 at 4:54 PM, John Nash <[hidden email]> wrote:
version: opensips 2.1.5 (x86_64/linux)
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
git revision: 39b19dd
main.c compiled on 19:27:59 Mar  5 2017 with gcc 4.4.7


On Tue, Mar 7, 2017 at 4:25 PM, Răzvan Crainea <[hidden email]> wrote:
What allocator are you using? Can you post the output of 'opensips -V'?

Best regards,
Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com
On 03/07/2017 12:23 PM, John Nash wrote:
Please note when i call do_routing in such a way that its unable to find any rules matching and reject call i do not see free memory drop. But if it finds a route, sends call to that gateway memory drops with each attempt.

On Tue, Mar 7, 2017 at 3:17 PM, John Nash <[hidden email]> wrote:
only 6 or 7 calls 

On Tue, Mar 7, 2017 at 3:09 PM, Răzvan Crainea <[hidden email]> wrote:
So I understand that after ~3K calls, that process completely runs out of memory?
How many calls have you done before this trace: http://pastebin.com/9Ge2NEVQ

Best regards,
Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com
On 03/07/2017 11:32 AM, John Nash wrote:
when I check stats after a call attempt pkmem:7-free_size:: 3304280

In this entry with every call I see a drop of 1000 bytes around and this never restores.

On Tue, Mar 7, 2017 at 2:16 PM, Răzvan Crainea <[hidden email]> wrote:
Hi, John!

Again, this trace doesn't show any leak.
Are you sure you are having a private memory leak and not a shared
memory leak?

Best regards,

Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com
On 03/06/2017 08:09 PM, John Nash wrote:
here is another trace 

I see lot of alloc request but no free.

On Mon, Mar 6, 2017 at 6:57 PM, John Nash <[hidden email]> wrote:
Ok will try that. Is it possible that wrong usage of drouting may cause this to happen instead of actual leak?... What are the things private memory is used for?

On Mon, Mar 6, 2017 at 6:48 PM, Răzvan Crainea <[hidden email]> wrote:
Hi, John!

From the dump you sent, I don't see any leaks. Perhaps some of those fragments increase over time. Can you make a memory dump after the server runs some time, like after it gets 100 messages?

Best regards,
Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com
On 03/06/2017 03:02 PM, John Nash wrote:

On Mon, Mar 6, 2017 at 6:20 PM, Răzvan Crainea <[hidden email]> wrote:
None of the "actions" you are talking about have big impact on private memory, but the shared one. Better do the dump and send it over to point out what is "eating" memory.

Best regards,
Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com
On 03/06/2017 02:39 PM, John Nash wrote:
with every call attempt it decreases. I tried some changes by rejecting invite before drouting call (That means after auth , dispatcher) and found memory is stable but when drouting sends Invite to external gateway and external gateway rejects it. Then this issue happens.

Inuse transactions and active dialogs also 0. Somthing wrong happening in handling of failure replies. But apart from use_next_gw and setting some avps for CDR not much going on there.

On Mon, Mar 6, 2017 at 5:54 PM, Răzvan Crainea <[hidden email]> wrote:
Ok, so it is the first listener for the private IP that leaks. Next, is the memory stabilizing in time? Or it is continously decreasing?
Yes, that's how you should make the dump.

Best regards,
Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users




_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
_______________________________________________ Users mailing list [hidden email] http://lists.opensips.org/cgi-bin/mailman/listinfo/users
_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users




_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Quest to find memory leak

Razvan Crainea-3
Hi, John!

The traces you showed me are incomplete: they do not have all the memory chunks allocated, thus I can't say wether something is wrong or not.
As I said earlier, it is normal for opensips to use extra memory every call. But after a while, this should stabilize. After a while might mean more than 1000k calls. As long as you never reach the upper limit of the memory, you can't conclude that there is a memory leak. Even then, you're limit might be too low for the kind of traffic you are doing, so it still might not be a memory leak. But only then it is worth to investigate.
When we investigate, we need all the data (i.e. the entire trace of the memory dump).
So please try to send as many calls as possilble, and if this issue still persists, make a pkg memory dump when the server is in idle mode and send it over.

Best regards,
Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com
On 03/08/2017 11:26 AM, John Nash wrote:
any suggestion for me?..should i try to crash opensips by sending many calls?

On Tue, Mar 7, 2017 at 4:54 PM, John Nash <[hidden email]> wrote:
version: opensips 2.1.5 (x86_64/linux)
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
git revision: 39b19dd
main.c compiled on 19:27:59 Mar  5 2017 with gcc 4.4.7

memory stabilizing in time? Or it is continously decreasing?
Yes, that's how you should make the dump.

Best regards,
Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Quest to find memory leak

John Nash
Ok i will give another try what should be the values of memdump and memlog

On Wed, Mar 8, 2017 at 3:13 PM, Răzvan Crainea <[hidden email]> wrote:
Hi, John!

The traces you showed me are incomplete: they do not have all the memory chunks allocated, thus I can't say wether something is wrong or not.
As I said earlier, it is normal for opensips to use extra memory every call. But after a while, this should stabilize. After a while might mean more than 1000k calls. As long as you never reach the upper limit of the memory, you can't conclude that there is a memory leak. Even then, you're limit might be too low for the kind of traffic you are doing, so it still might not be a memory leak. But only then it is worth to investigate.
When we investigate, we need all the data (i.e. the entire trace of the memory dump).
So please try to send as many calls as possilble, and if this issue still persists, make a pkg memory dump when the server is in idle mode and send it over.

Best regards,
Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com
On 03/08/2017 11:26 AM, John Nash wrote:
any suggestion for me?..should i try to crash opensips by sending many calls?

On Tue, Mar 7, 2017 at 4:54 PM, John Nash <[hidden email]> wrote:
version: opensips 2.1.5 (x86_64/linux)
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
git revision: 39b19dd
main.c compiled on 19:27:59 Mar  5 2017 with gcc 4.4.7

memory stabilizing in time? Or it is continously decreasing?
Yes, that's how you should make the dump.

Best regards,
Răzvan Crainea
OpenSIPS Solutions
www.opensips-solutions.com


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users



_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
12
Loading...