out of pkg mem

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

out of pkg mem

Brett Nemeroff
Hey All,
I'm using 1.4.4-notls with the carrierroute module. Everything was working fine with about 6 routes loaded. Then I loaded an addition 120 routes and now when I execute:
opensipsctl fifo dlg_list

I get:
[root@sip1 opensips-1.4.4-notls]# opensipsctl fifo dlg_list
500 command 'dlg_list' failed
[root@sip1 opensips-1.4.4-notls]# 
Mar 19 21:23:44 sip1 /usr/local/sbin/opensips[6115]: ERROR:core:create_mi_node: no more pkg mem 
Mar 19 21:23:44 sip1 /usr/local/sbin/opensips[6115]: ERROR:dialog:internal_mi_print_dlg: failed to add node 
Mar 19 21:23:44 sip1 /usr/local/sbin/opensips[6115]: ERROR:dialog:internal_mi_print_dlgs: failed to print dialog 
Mar 19 21:23:44 sip1 /usr/local/sbin/opensips[6115]: ERROR:mi_fifo:mi_fifo_server: command (dlg_list) processing failed 

I've tried increasing the pkg mem in config.h:
#define PKG_MEM_POOL_SIZE 4*1024*1024           /*!< Used only if PKG_MALLOC is defined*/
#define SHM_MEM_SIZE 32                         /*!< Used if SH_MEM is defined*/

And followed it up with a make / make install / restart

And got the same result. I sent a SIGUSR1 to process 6115 and got the following:


[root@sip1 opensips-1.4.4-notls]# Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: Memory status (pkg): 
Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: fm_status (0x819ac00): 
Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]:  heap size= 4194304 
Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]:  used= 72488, used+overhead=681312, free=3512992 
Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]:  max used (+overhead)= 4194448 
Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: dumping free list: 
Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =   1 fragments no.:     3, unused:     0              bucket size:         8 -         8 (first         8) 
Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =   4 fragments no.: 20455, unused:     0              bucket size:        32 -        32 (first        32) 
Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =   5 fragments no.: 20315, unused:     0              bucket size:        40 -        40 (first        40) 
Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =   6 fragments no.:  9127, unused:     0              bucket size:        48 -        48 (first        48) 
Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =   7 fragments no.:    96, unused:     0              bucket size:        56 -        56 (first        56) 
Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =   8 fragments no.: 15195, unused:     0              bucket size:        64 -        64 (first        64) 
Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =   9 fragments no.:  3881, unused:     0              bucket size:        72 -        72 (first        72) 
Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =  10 fragments no.:  4311, unused:     0              bucket size:        80 -        80 (first        80) 
Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =  11 fragments no.:    10, unused:     0              bucket size:        88 -        88 (first        88) 
Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =  12 fragments no.:    16, unused:     0              bucket size:        96 -        96 (first        96) 
Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =  13 fragments no.:    25, unused:     0              bucket size:       104 -       104 (first       104) 
Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =  14 fragments no.:     4, unused:     0              bucket size:       112 -       112 (first       112) 
Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =  15 fragments no.:     1, unused:     0              bucket size:       120 -       120 (first       120) 
Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =  16 fragments no.:     1, unused:     0              bucket size:       128 -       128 (first       128) 
Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: TOTAL:  73440 free fragments = 3513160 free bytes 


Not sure what to do here... any ideas?

Thanks,
Brett


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: out of pkg mem

Bogdan-Andrei Iancu
Hi Brett,

but there should be no relation about how many routes you have in CR (in
shm) and printing the active dlgs (via pkg)....

Do you have an estimation about the number of ongoing dialogs (see via
stats) ? I mean you run some large number?

Regards,
Bogdan

Brett Nemeroff wrote:

> Hey All,
> I'm using 1.4.4-notls with the carrierroute module. Everything was
> working fine with about 6 routes loaded. Then I loaded an addition 120
> routes and now when I execute:
> opensipsctl fifo dlg_list
>
> I get:
> [root@sip1 opensips-1.4.4-notls]# opensipsctl fifo dlg_list
> 500 command 'dlg_list' failed
> [root@sip1 opensips-1.4.4-notls]#
> Mar 19 21:23:44 sip1 /usr/local/sbin/opensips[6115]:
> ERROR:core:create_mi_node: no more pkg mem
> Mar 19 21:23:44 sip1 /usr/local/sbin/opensips[6115]:
> ERROR:dialog:internal_mi_print_dlg: failed to add node
> Mar 19 21:23:44 sip1 /usr/local/sbin/opensips[6115]:
> ERROR:dialog:internal_mi_print_dlgs: failed to print dialog
> Mar 19 21:23:44 sip1 /usr/local/sbin/opensips[6115]:
> ERROR:mi_fifo:mi_fifo_server: command (dlg_list) processing failed
>
> I've tried increasing the pkg mem in config.h:
> #define PKG_MEM_POOL_SIZE 4*1024*1024           /*!< Used only if
> PKG_MALLOC is defined*/
> #define SHM_MEM_SIZE 32                         /*!< Used if SH_MEM is
> defined*/
>
> And followed it up with a make / make install / restart
>
> And got the same result. I sent a SIGUSR1 to process 6115 and got the
> following:
>
>
> [root@sip1 opensips-1.4.4-notls]# Mar 19 21:21:25 sip1
> /usr/local/sbin/opensips[6115]: Memory status (pkg):
> Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: fm_status
> (0x819ac00):
> Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]:  heap size= 4194304
> Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]:  used= 72488,
> used+overhead=681312, free=3512992
> Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]:  max used
> (+overhead)= 4194448
> Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: dumping free list:
> Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =   1
> fragments no.:     3, unused:     0              bucket size:        
> 8 -         8 (first         8)
> Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =   4
> fragments no.: 20455, unused:     0              bucket size:      
>  32 -        32 (first        32)
> Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =   5
> fragments no.: 20315, unused:     0              bucket size:      
>  40 -        40 (first        40)
> Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =   6
> fragments no.:  9127, unused:     0              bucket size:      
>  48 -        48 (first        48)
> Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =   7
> fragments no.:    96, unused:     0              bucket size:      
>  56 -        56 (first        56)
> Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =   8
> fragments no.: 15195, unused:     0              bucket size:      
>  64 -        64 (first        64)
> Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =   9
> fragments no.:  3881, unused:     0              bucket size:      
>  72 -        72 (first        72)
> Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =  10
> fragments no.:  4311, unused:     0              bucket size:      
>  80 -        80 (first        80)
> Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =  11
> fragments no.:    10, unused:     0              bucket size:      
>  88 -        88 (first        88)
> Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =  12
> fragments no.:    16, unused:     0              bucket size:      
>  96 -        96 (first        96)
> Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =  13
> fragments no.:    25, unused:     0              bucket size:      
> 104 -       104 (first       104)
> Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =  14
> fragments no.:     4, unused:     0              bucket size:      
> 112 -       112 (first       112)
> Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =  15
> fragments no.:     1, unused:     0              bucket size:      
> 120 -       120 (first       120)
> Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =  16
> fragments no.:     1, unused:     0              bucket size:      
> 128 -       128 (first       128)
> Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: TOTAL:  73440
> free fragments = 3513160 free bytes
>
>
> Not sure what to do here... any ideas?
>
> Thanks,
> Brett
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Users mailing list
> [hidden email]
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>  


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: out of pkg mem

Brett Nemeroff
Hi Bogdan,
Yeah, I didn't think it would be related, but it's all that's really changed. 

The dialogs don't seem to be an issue. there are like 3-4 calls up (really). It happens even if I kill the proc and start it back up. 

Any ideas? 


On Fri, Mar 20, 2009 at 7:21 AM, Bogdan-Andrei Iancu <[hidden email]> wrote:
Hi Brett,

but there should be no relation about how many routes you have in CR (in shm) and printing the active dlgs (via pkg)....

Do you have an estimation about the number of ongoing dialogs (see via stats) ? I mean you run some large number?

Regards,
Bogdan

Brett Nemeroff wrote:
Hey All,
I'm using 1.4.4-notls with the carrierroute module. Everything was working fine with about 6 routes loaded. Then I loaded an addition 120 routes and now when I execute:
opensipsctl fifo dlg_list

I get:
[root@sip1 opensips-1.4.4-notls]# opensipsctl fifo dlg_list
500 command 'dlg_list' failed
[root@sip1 opensips-1.4.4-notls]# Mar 19 21:23:44 sip1 /usr/local/sbin/opensips[6115]: ERROR:core:create_mi_node: no more pkg mem Mar 19 21:23:44 sip1 /usr/local/sbin/opensips[6115]: ERROR:dialog:internal_mi_print_dlg: failed to add node Mar 19 21:23:44 sip1 /usr/local/sbin/opensips[6115]: ERROR:dialog:internal_mi_print_dlgs: failed to print dialog Mar 19 21:23:44 sip1 /usr/local/sbin/opensips[6115]: ERROR:mi_fifo:mi_fifo_server: command (dlg_list) processing failed
I've tried increasing the pkg mem in config.h:
#define PKG_MEM_POOL_SIZE 4*1024*1024           /*!< Used only if PKG_MALLOC is defined*/
#define SHM_MEM_SIZE 32                         /*!< Used if SH_MEM is defined*/

And followed it up with a make / make install / restart

And got the same result. I sent a SIGUSR1 to process 6115 and got the following:


[root@sip1 opensips-1.4.4-notls]# Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: Memory status (pkg): Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: fm_status (0x819ac00): Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]:  heap size= 4194304 Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]:  used= 72488, used+overhead=681312, free=3512992 Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]:  max used (+overhead)= 4194448 Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: dumping free list: Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =   1 fragments no.:     3, unused:     0              bucket size:         8 -         8 (first         8) Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =   4 fragments no.: 20455, unused:     0              bucket size:        32 -        32 (first        32) Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =   5 fragments no.: 20315, unused:     0              bucket size:        40 -        40 (first        40) Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =   6 fragments no.:  9127, unused:     0              bucket size:        48 -        48 (first        48) Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =   7 fragments no.:    96, unused:     0              bucket size:        56 -        56 (first        56) Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =   8 fragments no.: 15195, unused:     0              bucket size:        64 -        64 (first        64) Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =   9 fragments no.:  3881, unused:     0              bucket size:        72 -        72 (first        72) Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =  10 fragments no.:  4311, unused:     0              bucket size:        80 -        80 (first        80) Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =  11 fragments no.:    10, unused:     0              bucket size:        88 -        88 (first        88) Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =  12 fragments no.:    16, unused:     0              bucket size:        96 -        96 (first        96) Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =  13 fragments no.:    25, unused:     0              bucket size:       104 -       104 (first       104) Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =  14 fragments no.:     4, unused:     0              bucket size:       112 -       112 (first       112) Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =  15 fragments no.:     1, unused:     0              bucket size:       120 -       120 (first       120) Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =  16 fragments no.:     1, unused:     0              bucket size:       128 -       128 (first       128) Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: TOTAL:  73440 free fragments = 3513160 free bytes

Not sure what to do here... any ideas?

Thanks,
Brett

------------------------------------------------------------------------

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
 



_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: out of pkg mem

Brett Nemeroff
In reply to this post by Brett Nemeroff
Bogdan,
Now I get the following even on a:
opensipsctl cr reload

Mar 20 12:47:35 sip1 /usr/local/sbin/opensips[6115]: ERROR:db_mysql:db_mysql_convert_rows: no private memory left 
Mar 20 12:47:35 sip1 /usr/local/sbin/opensips[6115]: ERROR:db_mysql:db_mysql_convert_result: error while converting rows 
Mar 20 12:47:35 sip1 /usr/local/sbin/opensips[6115]: ERROR:db_mysql:db_mysql_store_result: error while converting result 
Mar 20 12:47:35 sip1 /usr/local/sbin/opensips[6115]: ERROR:core:db_do_query: error while storing result
Mar 20 12:47:35 sip1 /usr/local/sbin/opensips[6115]: ERROR:carrierroute:load_route_data: Failed to query database. 
Mar 20 12:47:35 sip1 /usr/local/sbin/opensips[6115]: ERROR:carrierroute:prepare_route_tree: could not load routing data 

I'm pretty concerned about this.. not sure why it all of a sudden began to happen. Server otherwise looks healthy?

Any ideas?

-Brett


On Thu, Mar 19, 2009 at 8:27 PM, Brett Nemeroff <[hidden email]> wrote:
Hey All,
I'm using 1.4.4-notls with the carrierroute module. Everything was working fine with about 6 routes loaded. Then I loaded an addition 120 routes and now when I execute:
opensipsctl fifo dlg_list

I get:
[root@sip1 opensips-1.4.4-notls]# opensipsctl fifo dlg_list
500 command 'dlg_list' failed
[root@sip1 opensips-1.4.4-notls]# 
Mar 19 21:23:44 sip1 /usr/local/sbin/opensips[6115]: ERROR:core:create_mi_node: no more pkg mem 
Mar 19 21:23:44 sip1 /usr/local/sbin/opensips[6115]: ERROR:dialog:internal_mi_print_dlg: failed to add node 
Mar 19 21:23:44 sip1 /usr/local/sbin/opensips[6115]: ERROR:dialog:internal_mi_print_dlgs: failed to print dialog 
Mar 19 21:23:44 sip1 /usr/local/sbin/opensips[6115]: ERROR:mi_fifo:mi_fifo_server: command (dlg_list) processing failed 

I've tried increasing the pkg mem in config.h:
#define PKG_MEM_POOL_SIZE 4*1024*1024           /*!< Used only if PKG_MALLOC is defined*/
#define SHM_MEM_SIZE 32                         /*!< Used if SH_MEM is defined*/

And followed it up with a make / make install / restart

And got the same result. I sent a SIGUSR1 to process 6115 and got the following:


[root@sip1 opensips-1.4.4-notls]# Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: Memory status (pkg): 
Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: fm_status (0x819ac00): 
Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]:  heap size= 4194304 
Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]:  used= 72488, used+overhead=681312, free=3512992 
Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]:  max used (+overhead)= 4194448 
Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: dumping free list: 
Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =   1 fragments no.:     3, unused:     0              bucket size:         8 -         8 (first         8) 
Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =   4 fragments no.: 20455, unused:     0              bucket size:        32 -        32 (first        32) 
Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =   5 fragments no.: 20315, unused:     0              bucket size:        40 -        40 (first        40) 
Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =   6 fragments no.:  9127, unused:     0              bucket size:        48 -        48 (first        48) 
Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =   7 fragments no.:    96, unused:     0              bucket size:        56 -        56 (first        56) 
Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =   8 fragments no.: 15195, unused:     0              bucket size:        64 -        64 (first        64) 
Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =   9 fragments no.:  3881, unused:     0              bucket size:        72 -        72 (first        72) 
Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =  10 fragments no.:  4311, unused:     0              bucket size:        80 -        80 (first        80) 
Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =  11 fragments no.:    10, unused:     0              bucket size:        88 -        88 (first        88) 
Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =  12 fragments no.:    16, unused:     0              bucket size:        96 -        96 (first        96) 
Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =  13 fragments no.:    25, unused:     0              bucket size:       104 -       104 (first       104) 
Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =  14 fragments no.:     4, unused:     0              bucket size:       112 -       112 (first       112) 
Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =  15 fragments no.:     1, unused:     0              bucket size:       120 -       120 (first       120) 
Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =  16 fragments no.:     1, unused:     0              bucket size:       128 -       128 (first       128) 
Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: TOTAL:  73440 free fragments = 3513160 free bytes 


Not sure what to do here... any ideas?

Thanks,
Brett



_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: out of pkg mem

Bogdan-Andrei Iancu
In reply to this post by Brett Nemeroff
Try enabling the memory debugger, to get more data about the available
pkg mem at runtime (do a SIGUSR1 to the fifo proc) and also about how
much the dlg_list wants to allocate.

Regards,
Bogdan

Brett Nemeroff wrote:

> Hi Bogdan,
> Yeah, I didn't think it would be related, but it's all that's really
> changed.
>
> The dialogs don't seem to be an issue. there are like 3-4 calls up
> (really). It happens even if I kill the proc and start it back up.
>
> Any ideas?
>
>
> On Fri, Mar 20, 2009 at 7:21 AM, Bogdan-Andrei Iancu
> <[hidden email] <mailto:[hidden email]>> wrote:
>
>     Hi Brett,
>
>     but there should be no relation about how many routes you have in
>     CR (in shm) and printing the active dlgs (via pkg)....
>
>     Do you have an estimation about the number of ongoing dialogs (see
>     via stats) ? I mean you run some large number?
>
>     Regards,
>     Bogdan
>
>     Brett Nemeroff wrote:
>
>         Hey All,
>         I'm using 1.4.4-notls with the carrierroute module. Everything
>         was working fine with about 6 routes loaded. Then I loaded an
>         addition 120 routes and now when I execute:
>         opensipsctl fifo dlg_list
>
>         I get:
>         [root@sip1 opensips-1.4.4-notls]# opensipsctl fifo dlg_list
>         500 command 'dlg_list' failed
>         [root@sip1 opensips-1.4.4-notls]# Mar 19 21:23:44 sip1
>         /usr/local/sbin/opensips[6115]: ERROR:core:create_mi_node: no
>         more pkg mem Mar 19 21:23:44 sip1
>         /usr/local/sbin/opensips[6115]:
>         ERROR:dialog:internal_mi_print_dlg: failed to add node Mar 19
>         21:23:44 sip1 /usr/local/sbin/opensips[6115]:
>         ERROR:dialog:internal_mi_print_dlgs: failed to print dialog
>         Mar 19 21:23:44 sip1 /usr/local/sbin/opensips[6115]:
>         ERROR:mi_fifo:mi_fifo_server: command (dlg_list) processing
>         failed
>         I've tried increasing the pkg mem in config.h:
>         #define PKG_MEM_POOL_SIZE 4*1024*1024           /*!< Used only
>         if PKG_MALLOC is defined*/
>         #define SHM_MEM_SIZE 32                         /*!< Used if
>         SH_MEM is defined*/
>
>         And followed it up with a make / make install / restart
>
>         And got the same result. I sent a SIGUSR1 to process 6115 and
>         got the following:
>
>
>         [root@sip1 opensips-1.4.4-notls]# Mar 19 21:21:25 sip1
>         /usr/local/sbin/opensips[6115]: Memory status (pkg): Mar 19
>         21:21:25 sip1 /usr/local/sbin/opensips[6115]: fm_status
>         (0x819ac00): Mar 19 21:21:25 sip1
>         /usr/local/sbin/opensips[6115]:  heap size= 4194304 Mar 19
>         21:21:25 sip1 /usr/local/sbin/opensips[6115]:  used= 72488,
>         used+overhead=681312, free=3512992 Mar 19 21:21:25 sip1
>         /usr/local/sbin/opensips[6115]:  max used (+overhead)= 4194448
>         Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: dumping
>         free list: Mar 19 21:21:25 sip1
>         /usr/local/sbin/opensips[6115]: hash =   1 fragments no.:    
>         3, unused:     0              bucket size:         8 -        
>         8 (first         8) Mar 19 21:21:25 sip1
>         /usr/local/sbin/opensips[6115]: hash =   4 fragments no.:
>         20455, unused:     0              bucket size:        32 -    
>            32 (first        32) Mar 19 21:21:25 sip1
>         /usr/local/sbin/opensips[6115]: hash =   5 fragments no.:
>         20315, unused:     0              bucket size:        40 -    
>            40 (first        40) Mar 19 21:21:25 sip1
>         /usr/local/sbin/opensips[6115]: hash =   6 fragments no.:
>          9127, unused:     0              bucket size:        48 -    
>            48 (first        48) Mar 19 21:21:25 sip1
>         /usr/local/sbin/opensips[6115]: hash =   7 fragments no.:  
>          96, unused:     0              bucket size:        56 -      
>          56 (first        56) Mar 19 21:21:25 sip1
>         /usr/local/sbin/opensips[6115]: hash =   8 fragments no.:
>         15195, unused:     0              bucket size:        64 -    
>            64 (first        64) Mar 19 21:21:25 sip1
>         /usr/local/sbin/opensips[6115]: hash =   9 fragments no.:
>          3881, unused:     0              bucket size:        72 -    
>            72 (first        72) Mar 19 21:21:25 sip1
>         /usr/local/sbin/opensips[6115]: hash =  10 fragments no.:
>          4311, unused:     0              bucket size:        80 -    
>            80 (first        80) Mar 19 21:21:25 sip1
>         /usr/local/sbin/opensips[6115]: hash =  11 fragments no.:  
>          10, unused:     0              bucket size:        88 -      
>          88 (first        88) Mar 19 21:21:25 sip1
>         /usr/local/sbin/opensips[6115]: hash =  12 fragments no.:  
>          16, unused:     0              bucket size:        96 -      
>          96 (first        96) Mar 19 21:21:25 sip1
>         /usr/local/sbin/opensips[6115]: hash =  13 fragments no.:  
>          25, unused:     0              bucket size:       104 -      
>         104 (first       104) Mar 19 21:21:25 sip1
>         /usr/local/sbin/opensips[6115]: hash =  14 fragments no.:    
>         4, unused:     0              bucket size:       112 -      
>         112 (first       112) Mar 19 21:21:25 sip1
>         /usr/local/sbin/opensips[6115]: hash =  15 fragments no.:    
>         1, unused:     0              bucket size:       120 -      
>         120 (first       120) Mar 19 21:21:25 sip1
>         /usr/local/sbin/opensips[6115]: hash =  16 fragments no.:    
>         1, unused:     0              bucket size:       128 -      
>         128 (first       128) Mar 19 21:21:25 sip1
>         /usr/local/sbin/opensips[6115]: TOTAL:  73440 free fragments =
>         3513160 free bytes
>
>         Not sure what to do here... any ideas?
>
>         Thanks,
>         Brett
>
>         ------------------------------------------------------------------------
>
>         _______________________________________________
>         Users mailing list
>         [hidden email] <mailto:[hidden email]>
>         http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>          
>
>
>


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: out of pkg mem

Bogdan-Andrei Iancu
In reply to this post by Brett Nemeroff
Brett,

maybe the CR module is somehow eating too much priv mem - See my
previous email.

Regards,
Bogdan

Brett Nemeroff wrote:

> Bogdan,
> Now I get the following even on a:
> opensipsctl cr reload
>
> Mar 20 12:47:35 sip1 /usr/local/sbin/opensips[6115]:
> ERROR:db_mysql:db_mysql_convert_rows: no private memory left
> Mar 20 12:47:35 sip1 /usr/local/sbin/opensips[6115]:
> ERROR:db_mysql:db_mysql_convert_result: error while converting rows
> Mar 20 12:47:35 sip1 /usr/local/sbin/opensips[6115]:
> ERROR:db_mysql:db_mysql_store_result: error while converting result
> Mar 20 12:47:35 sip1 /usr/local/sbin/opensips[6115]:
> ERROR:core:db_do_query: error while storing result
> Mar 20 12:47:35 sip1 /usr/local/sbin/opensips[6115]:
> ERROR:carrierroute:load_route_data: Failed to query database.
> Mar 20 12:47:35 sip1 /usr/local/sbin/opensips[6115]:
> ERROR:carrierroute:prepare_route_tree: could not load routing data
>
> I'm pretty concerned about this.. not sure why it all of a sudden
> began to happen. Server otherwise looks healthy?
>
> Any ideas?
>
> -Brett
>
>
> On Thu, Mar 19, 2009 at 8:27 PM, Brett Nemeroff <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     Hey All,
>     I'm using 1.4.4-notls with the carrierroute module. Everything was
>     working fine with about 6 routes loaded. Then I loaded an addition
>     120 routes and now when I execute:
>     opensipsctl fifo dlg_list
>
>     I get:
>     [root@sip1 opensips-1.4.4-notls]# opensipsctl fifo dlg_list
>     500 command 'dlg_list' failed
>     [root@sip1 opensips-1.4.4-notls]#
>     Mar 19 21:23:44 sip1 /usr/local/sbin/opensips[6115]:
>     ERROR:core:create_mi_node: no more pkg mem
>     Mar 19 21:23:44 sip1 /usr/local/sbin/opensips[6115]:
>     ERROR:dialog:internal_mi_print_dlg: failed to add node
>     Mar 19 21:23:44 sip1 /usr/local/sbin/opensips[6115]:
>     ERROR:dialog:internal_mi_print_dlgs: failed to print dialog
>     Mar 19 21:23:44 sip1 /usr/local/sbin/opensips[6115]:
>     ERROR:mi_fifo:mi_fifo_server: command (dlg_list) processing failed
>
>     I've tried increasing the pkg mem in config.h:
>     #define PKG_MEM_POOL_SIZE 4*1024*1024           /*!< Used only if
>     PKG_MALLOC is defined*/
>     #define SHM_MEM_SIZE 32                         /*!< Used if
>     SH_MEM is defined*/
>
>     And followed it up with a make / make install / restart
>
>     And got the same result. I sent a SIGUSR1 to process 6115 and got
>     the following:
>
>
>     [root@sip1 opensips-1.4.4-notls]# Mar 19 21:21:25 sip1
>     /usr/local/sbin/opensips[6115]: Memory status (pkg):
>     Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: fm_status
>     (0x819ac00):
>     Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]:  heap size=
>     4194304
>     Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]:  used= 72488,
>     used+overhead=681312, free=3512992
>     Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]:  max used
>     (+overhead)= 4194448
>     Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: dumping free
>     list:
>     Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =   1
>     fragments no.:     3, unused:     0              bucket size:    
>         8 -         8 (first         8)
>     Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =   4
>     fragments no.: 20455, unused:     0              bucket size:    
>        32 -        32 (first        32)
>     Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =   5
>     fragments no.: 20315, unused:     0              bucket size:    
>        40 -        40 (first        40)
>     Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =   6
>     fragments no.:  9127, unused:     0              bucket size:    
>        48 -        48 (first        48)
>     Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =   7
>     fragments no.:    96, unused:     0              bucket size:    
>        56 -        56 (first        56)
>     Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =   8
>     fragments no.: 15195, unused:     0              bucket size:    
>        64 -        64 (first        64)
>     Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =   9
>     fragments no.:  3881, unused:     0              bucket size:    
>        72 -        72 (first        72)
>     Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =  10
>     fragments no.:  4311, unused:     0              bucket size:    
>        80 -        80 (first        80)
>     Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =  11
>     fragments no.:    10, unused:     0              bucket size:    
>        88 -        88 (first        88)
>     Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =  12
>     fragments no.:    16, unused:     0              bucket size:    
>        96 -        96 (first        96)
>     Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =  13
>     fragments no.:    25, unused:     0              bucket size:    
>       104 -       104 (first       104)
>     Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =  14
>     fragments no.:     4, unused:     0              bucket size:    
>       112 -       112 (first       112)
>     Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =  15
>     fragments no.:     1, unused:     0              bucket size:    
>       120 -       120 (first       120)
>     Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =  16
>     fragments no.:     1, unused:     0              bucket size:    
>       128 -       128 (first       128)
>     Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: TOTAL:  73440
>     free fragments = 3513160 free bytes
>
>
>     Not sure what to do here... any ideas?
>
>     Thanks,
>     Brett
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Users mailing list
> [hidden email]
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>  


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: out of pkg mem

Brett Nemeroff
In reply to this post by Brett Nemeroff
Here's a piece of information that may be relevant....

So I restart opensips completely..

Then I do a:
opensipsctl cr reload

works fine

then I do:
opensipsctl fifo dlg_list

and I get:
500 command 'dlg_list' failed

Next I try:
opensipsctl cr reload

and I get:
500 failed to re-built tree, see log

And then opensipsctl cr reload never works again unless I restart the process..

The log message, as I mentioned in my last email shows:
Mar 20 12:52:21 sip1 /usr/local/sbin/opensips[23232]: ERROR:db_mysql:db_mysql_convert_rows: no private memory left 
Mar 20 12:52:21 sip1 /usr/local/sbin/opensips[23232]: ERROR:db_mysql:db_mysql_convert_result: error while converting rows 
Mar 20 12:52:21 sip1 /usr/local/sbin/opensips[23232]: ERROR:db_mysql:db_mysql_store_result: error while converting result 
Mar 20 12:52:21 sip1 /usr/local/sbin/opensips[23232]: ERROR:core:db_do_query: error while storing result
Mar 20 12:52:21 sip1 /usr/local/sbin/opensips[23232]: ERROR:carrierroute:load_route_data: Failed to query database. 
Mar 20 12:52:21 sip1 /usr/local/sbin/opensips[23232]: ERROR:carrierroute:prepare_route_tree: could not load routing data 

Help?
Thanks,
Brett


On Fri, Mar 20, 2009 at 11:49 AM, Brett Nemeroff <[hidden email]> wrote:
Bogdan,
Now I get the following even on a:
opensipsctl cr reload

Mar 20 12:47:35 sip1 /usr/local/sbin/opensips[6115]: ERROR:db_mysql:db_mysql_convert_rows: no private memory left 
Mar 20 12:47:35 sip1 /usr/local/sbin/opensips[6115]: ERROR:db_mysql:db_mysql_convert_result: error while converting rows 
Mar 20 12:47:35 sip1 /usr/local/sbin/opensips[6115]: ERROR:db_mysql:db_mysql_store_result: error while converting result 
Mar 20 12:47:35 sip1 /usr/local/sbin/opensips[6115]: ERROR:core:db_do_query: error while storing result
Mar 20 12:47:35 sip1 /usr/local/sbin/opensips[6115]: ERROR:carrierroute:load_route_data: Failed to query database. 
Mar 20 12:47:35 sip1 /usr/local/sbin/opensips[6115]: ERROR:carrierroute:prepare_route_tree: could not load routing data 

I'm pretty concerned about this.. not sure why it all of a sudden began to happen. Server otherwise looks healthy?

Any ideas?

-Brett


On Thu, Mar 19, 2009 at 8:27 PM, Brett Nemeroff <[hidden email]> wrote:
Hey All,
I'm using 1.4.4-notls with the carrierroute module. Everything was working fine with about 6 routes loaded. Then I loaded an addition 120 routes and now when I execute:
opensipsctl fifo dlg_list

I get:
[root@sip1 opensips-1.4.4-notls]# opensipsctl fifo dlg_list
500 command 'dlg_list' failed
[root@sip1 opensips-1.4.4-notls]# 
Mar 19 21:23:44 sip1 /usr/local/sbin/opensips[6115]: ERROR:core:create_mi_node: no more pkg mem 
Mar 19 21:23:44 sip1 /usr/local/sbin/opensips[6115]: ERROR:dialog:internal_mi_print_dlg: failed to add node 
Mar 19 21:23:44 sip1 /usr/local/sbin/opensips[6115]: ERROR:dialog:internal_mi_print_dlgs: failed to print dialog 
Mar 19 21:23:44 sip1 /usr/local/sbin/opensips[6115]: ERROR:mi_fifo:mi_fifo_server: command (dlg_list) processing failed 

I've tried increasing the pkg mem in config.h:
#define PKG_MEM_POOL_SIZE 4*1024*1024           /*!< Used only if PKG_MALLOC is defined*/
#define SHM_MEM_SIZE 32                         /*!< Used if SH_MEM is defined*/

And followed it up with a make / make install / restart

And got the same result. I sent a SIGUSR1 to process 6115 and got the following:


[root@sip1 opensips-1.4.4-notls]# Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: Memory status (pkg): 
Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: fm_status (0x819ac00): 
Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]:  heap size= 4194304 
Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]:  used= 72488, used+overhead=681312, free=3512992 
Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]:  max used (+overhead)= 4194448 
Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: dumping free list: 
Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =   1 fragments no.:     3, unused:     0              bucket size:         8 -         8 (first         8) 
Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =   4 fragments no.: 20455, unused:     0              bucket size:        32 -        32 (first        32) 
Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =   5 fragments no.: 20315, unused:     0              bucket size:        40 -        40 (first        40) 
Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =   6 fragments no.:  9127, unused:     0              bucket size:        48 -        48 (first        48) 
Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =   7 fragments no.:    96, unused:     0              bucket size:        56 -        56 (first        56) 
Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =   8 fragments no.: 15195, unused:     0              bucket size:        64 -        64 (first        64) 
Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =   9 fragments no.:  3881, unused:     0              bucket size:        72 -        72 (first        72) 
Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =  10 fragments no.:  4311, unused:     0              bucket size:        80 -        80 (first        80) 
Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =  11 fragments no.:    10, unused:     0              bucket size:        88 -        88 (first        88) 
Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =  12 fragments no.:    16, unused:     0              bucket size:        96 -        96 (first        96) 
Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =  13 fragments no.:    25, unused:     0              bucket size:       104 -       104 (first       104) 
Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =  14 fragments no.:     4, unused:     0              bucket size:       112 -       112 (first       112) 
Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =  15 fragments no.:     1, unused:     0              bucket size:       120 -       120 (first       120) 
Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =  16 fragments no.:     1, unused:     0              bucket size:       128 -       128 (first       128) 
Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: TOTAL:  73440 free fragments = 3513160 free bytes 


Not sure what to do here... any ideas?

Thanks,
Brett




_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: out of pkg mem

Bogdan-Andrei Iancu
Hi Brett,

tried to run the 50 calls through the proxy and do "dlg_list" , but
works ok - I do not have the CR module....Can you try all this but
without the CR module?

Regards,
Bogdan

Brett Nemeroff wrote:

> Here's a piece of information that may be relevant....
>
> So I restart opensips completely..
>
> Then I do a:
> opensipsctl cr reload
>
> works fine
>
> then I do:
> opensipsctl fifo dlg_list
>
> and I get:
> 500 command 'dlg_list' failed
>
> Next I try:
> opensipsctl cr reload
>
> and I get:
> 500 failed to re-built tree, see log
>
> And then opensipsctl cr reload never works again unless I restart the
> process..
>
> The log message, as I mentioned in my last email shows:
> Mar 20 12:52:21 sip1 /usr/local/sbin/opensips[23232]:
> ERROR:db_mysql:db_mysql_convert_rows: no private memory left
> Mar 20 12:52:21 sip1 /usr/local/sbin/opensips[23232]:
> ERROR:db_mysql:db_mysql_convert_result: error while converting rows
> Mar 20 12:52:21 sip1 /usr/local/sbin/opensips[23232]:
> ERROR:db_mysql:db_mysql_store_result: error while converting result
> Mar 20 12:52:21 sip1 /usr/local/sbin/opensips[23232]:
> ERROR:core:db_do_query: error while storing result
> Mar 20 12:52:21 sip1 /usr/local/sbin/opensips[23232]:
> ERROR:carrierroute:load_route_data: Failed to query database.
> Mar 20 12:52:21 sip1 /usr/local/sbin/opensips[23232]:
> ERROR:carrierroute:prepare_route_tree: could not load routing data
>
> Help?
> Thanks,
> Brett
>
>
> On Fri, Mar 20, 2009 at 11:49 AM, Brett Nemeroff <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     Bogdan,
>     Now I get the following even on a:
>     opensipsctl cr reload
>
>     Mar 20 12:47:35 sip1 /usr/local/sbin/opensips[6115]:
>     ERROR:db_mysql:db_mysql_convert_rows: no private memory left
>     Mar 20 12:47:35 sip1 /usr/local/sbin/opensips[6115]:
>     ERROR:db_mysql:db_mysql_convert_result: error while converting rows
>     Mar 20 12:47:35 sip1 /usr/local/sbin/opensips[6115]:
>     ERROR:db_mysql:db_mysql_store_result: error while converting result
>     Mar 20 12:47:35 sip1 /usr/local/sbin/opensips[6115]:
>     ERROR:core:db_do_query: error while storing result
>     Mar 20 12:47:35 sip1 /usr/local/sbin/opensips[6115]:
>     ERROR:carrierroute:load_route_data: Failed to query database.
>     Mar 20 12:47:35 sip1 /usr/local/sbin/opensips[6115]:
>     ERROR:carrierroute:prepare_route_tree: could not load routing data
>
>     I'm pretty concerned about this.. not sure why it all of a sudden
>     began to happen. Server otherwise looks healthy?
>
>     Any ideas?
>
>     -Brett
>
>
>     On Thu, Mar 19, 2009 at 8:27 PM, Brett Nemeroff
>     <[hidden email] <mailto:[hidden email]>> wrote:
>
>         Hey All,
>         I'm using 1.4.4-notls with the carrierroute module. Everything
>         was working fine with about 6 routes loaded. Then I loaded an
>         addition 120 routes and now when I execute:
>         opensipsctl fifo dlg_list
>
>         I get:
>         [root@sip1 opensips-1.4.4-notls]# opensipsctl fifo dlg_list
>         500 command 'dlg_list' failed
>         [root@sip1 opensips-1.4.4-notls]#
>         Mar 19 21:23:44 sip1 /usr/local/sbin/opensips[6115]:
>         ERROR:core:create_mi_node: no more pkg mem
>         Mar 19 21:23:44 sip1 /usr/local/sbin/opensips[6115]:
>         ERROR:dialog:internal_mi_print_dlg: failed to add node
>         Mar 19 21:23:44 sip1 /usr/local/sbin/opensips[6115]:
>         ERROR:dialog:internal_mi_print_dlgs: failed to print dialog
>         Mar 19 21:23:44 sip1 /usr/local/sbin/opensips[6115]:
>         ERROR:mi_fifo:mi_fifo_server: command (dlg_list) processing
>         failed
>
>         I've tried increasing the pkg mem in config.h:
>         #define PKG_MEM_POOL_SIZE 4*1024*1024           /*!< Used only
>         if PKG_MALLOC is defined*/
>         #define SHM_MEM_SIZE 32                         /*!< Used if
>         SH_MEM is defined*/
>
>         And followed it up with a make / make install / restart
>
>         And got the same result. I sent a SIGUSR1 to process 6115 and
>         got the following:
>
>
>         [root@sip1 opensips-1.4.4-notls]# Mar 19 21:21:25 sip1
>         /usr/local/sbin/opensips[6115]: Memory status (pkg):
>         Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: fm_status
>         (0x819ac00):
>         Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]:  heap
>         size= 4194304
>         Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]:  used=
>         72488, used+overhead=681312, free=3512992
>         Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]:  max used
>         (+overhead)= 4194448
>         Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: dumping
>         free list:
>         Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =  
>         1 fragments no.:     3, unused:     0              bucket
>         size:         8 -         8 (first         8)
>         Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =  
>         4 fragments no.: 20455, unused:     0              bucket
>         size:        32 -        32 (first        32)
>         Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =  
>         5 fragments no.: 20315, unused:     0              bucket
>         size:        40 -        40 (first        40)
>         Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =  
>         6 fragments no.:  9127, unused:     0              bucket
>         size:        48 -        48 (first        48)
>         Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =  
>         7 fragments no.:    96, unused:     0              bucket
>         size:        56 -        56 (first        56)
>         Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =  
>         8 fragments no.: 15195, unused:     0              bucket
>         size:        64 -        64 (first        64)
>         Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =  
>         9 fragments no.:  3881, unused:     0              bucket
>         size:        72 -        72 (first        72)
>         Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =
>          10 fragments no.:  4311, unused:     0              bucket
>         size:        80 -        80 (first        80)
>         Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =
>          11 fragments no.:    10, unused:     0              bucket
>         size:        88 -        88 (first        88)
>         Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =
>          12 fragments no.:    16, unused:     0              bucket
>         size:        96 -        96 (first        96)
>         Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =
>          13 fragments no.:    25, unused:     0              bucket
>         size:       104 -       104 (first       104)
>         Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =
>          14 fragments no.:     4, unused:     0              bucket
>         size:       112 -       112 (first       112)
>         Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =
>          15 fragments no.:     1, unused:     0              bucket
>         size:       120 -       120 (first       120)
>         Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =
>          16 fragments no.:     1, unused:     0              bucket
>         size:       128 -       128 (first       128)
>         Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: TOTAL:
>          73440 free fragments = 3513160 free bytes
>
>
>         Not sure what to do here... any ideas?
>
>         Thanks,
>         Brett
>
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Users mailing list
> [hidden email]
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>  


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: out of pkg mem

Brett Nemeroff
well it was working great for 2 months with low traffic.. reloading routes several times a day.. once I loaded these 120 routes into cr, now I'm having these problems.. 

I'll try to unload the routes and see if things change.. I'm not sure how it's relevant...
-Brett


On Fri, Mar 20, 2009 at 12:36 PM, Bogdan-Andrei Iancu <[hidden email]> wrote:
Hi Brett,

tried to run the 50 calls through the proxy and do "dlg_list" , but works ok - I do not have the CR module....Can you try all this but without the CR module?


Regards,
Bogdan

Brett Nemeroff wrote:
Here's a piece of information that may be relevant....

So I restart opensips completely..

Then I do a:
opensipsctl cr reload

works fine

then I do:
opensipsctl fifo dlg_list

and I get:
500 command 'dlg_list' failed

Next I try:
opensipsctl cr reload

and I get:
500 failed to re-built tree, see log

And then opensipsctl cr reload never works again unless I restart the process..

The log message, as I mentioned in my last email shows:
Mar 20 12:52:21 sip1 /usr/local/sbin/opensips[23232]: ERROR:db_mysql:db_mysql_convert_rows: no private memory left Mar 20 12:52:21 sip1 /usr/local/sbin/opensips[23232]: ERROR:db_mysql:db_mysql_convert_result: error while converting rows Mar 20 12:52:21 sip1 /usr/local/sbin/opensips[23232]: ERROR:db_mysql:db_mysql_store_result: error while converting result Mar 20 12:52:21 sip1 /usr/local/sbin/opensips[23232]: ERROR:core:db_do_query: error while storing result
Mar 20 12:52:21 sip1 /usr/local/sbin/opensips[23232]: ERROR:carrierroute:load_route_data: Failed to query database. Mar 20 12:52:21 sip1 /usr/local/sbin/opensips[23232]: ERROR:carrierroute:prepare_route_tree: could not load routing data
Help?
Thanks,
Brett


On Fri, Mar 20, 2009 at 11:49 AM, Brett Nemeroff <[hidden email] <mailto:[hidden email]>> wrote:

   Bogdan,
   Now I get the following even on a:
   opensipsctl cr reload

   Mar 20 12:47:35 sip1 /usr/local/sbin/opensips[6115]:
   ERROR:db_mysql:db_mysql_convert_rows: no private memory left    Mar 20 12:47:35 sip1 /usr/local/sbin/opensips[6115]:
   ERROR:db_mysql:db_mysql_convert_result: error while converting rows    Mar 20 12:47:35 sip1 /usr/local/sbin/opensips[6115]:
   ERROR:db_mysql:db_mysql_store_result: error while converting result    Mar 20 12:47:35 sip1 /usr/local/sbin/opensips[6115]:
   ERROR:core:db_do_query: error while storing result
   Mar 20 12:47:35 sip1 /usr/local/sbin/opensips[6115]:
   ERROR:carrierroute:load_route_data: Failed to query database.    Mar 20 12:47:35 sip1 /usr/local/sbin/opensips[6115]:
   ERROR:carrierroute:prepare_route_tree: could not load routing data
   I'm pretty concerned about this.. not sure why it all of a sudden
   began to happen. Server otherwise looks healthy?

   Any ideas?

   -Brett


   On Thu, Mar 19, 2009 at 8:27 PM, Brett Nemeroff
   <[hidden email] <mailto:[hidden email]>> wrote:

       Hey All,
       I'm using 1.4.4-notls with the carrierroute module. Everything
       was working fine with about 6 routes loaded. Then I loaded an
       addition 120 routes and now when I execute:
       opensipsctl fifo dlg_list

       I get:
       [root@sip1 opensips-1.4.4-notls]# opensipsctl fifo dlg_list
       500 command 'dlg_list' failed
       [root@sip1 opensips-1.4.4-notls]#        Mar 19 21:23:44 sip1 /usr/local/sbin/opensips[6115]:
       ERROR:core:create_mi_node: no more pkg mem        Mar 19 21:23:44 sip1 /usr/local/sbin/opensips[6115]:
       ERROR:dialog:internal_mi_print_dlg: failed to add node        Mar 19 21:23:44 sip1 /usr/local/sbin/opensips[6115]:
       ERROR:dialog:internal_mi_print_dlgs: failed to print dialog        Mar 19 21:23:44 sip1 /usr/local/sbin/opensips[6115]:
       ERROR:mi_fifo:mi_fifo_server: command (dlg_list) processing
       failed
       I've tried increasing the pkg mem in config.h:
       #define PKG_MEM_POOL_SIZE 4*1024*1024           /*!< Used only
       if PKG_MALLOC is defined*/
       #define SHM_MEM_SIZE 32                         /*!< Used if
       SH_MEM is defined*/

       And followed it up with a make / make install / restart

       And got the same result. I sent a SIGUSR1 to process 6115 and
       got the following:


       [root@sip1 opensips-1.4.4-notls]# Mar 19 21:21:25 sip1
       /usr/local/sbin/opensips[6115]: Memory status (pkg):        Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: fm_status
       (0x819ac00):        Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]:  heap
       size= 4194304        Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]:  used=
       72488, used+overhead=681312, free=3512992        Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]:  max used
       (+overhead)= 4194448        Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: dumping
       free list:        Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =          1 fragments no.:     3, unused:     0              bucket
       size:         8 -         8 (first         8)        Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =          4 fragments no.: 20455, unused:     0              bucket
       size:        32 -        32 (first        32)        Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =          5 fragments no.: 20315, unused:     0              bucket
       size:        40 -        40 (first        40)        Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =          6 fragments no.:  9127, unused:     0              bucket
       size:        48 -        48 (first        48)        Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =          7 fragments no.:    96, unused:     0              bucket
       size:        56 -        56 (first        56)        Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =          8 fragments no.: 15195, unused:     0              bucket
       size:        64 -        64 (first        64)        Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =          9 fragments no.:  3881, unused:     0              bucket
       size:        72 -        72 (first        72)        Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =
        10 fragments no.:  4311, unused:     0              bucket
       size:        80 -        80 (first        80)        Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =
        11 fragments no.:    10, unused:     0              bucket
       size:        88 -        88 (first        88)        Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =
        12 fragments no.:    16, unused:     0              bucket
       size:        96 -        96 (first        96)        Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =
        13 fragments no.:    25, unused:     0              bucket
       size:       104 -       104 (first       104)        Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =
        14 fragments no.:     4, unused:     0              bucket
       size:       112 -       112 (first       112)        Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =
        15 fragments no.:     1, unused:     0              bucket
       size:       120 -       120 (first       120)        Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: hash =
        16 fragments no.:     1, unused:     0              bucket
       size:       128 -       128 (first       128)        Mar 19 21:21:25 sip1 /usr/local/sbin/opensips[6115]: TOTAL:
        73440 free fragments = 3513160 free bytes

       Not sure what to do here... any ideas?

       Thanks,
       Brett



------------------------------------------------------------------------

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
 



_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users