Command "opensipsctl dialplan reload" randomly hangs

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Command "opensipsctl dialplan reload" randomly hangs

DanB-2
Hey Guys,

I have noticed the "opensipsctl dialplan reload" command randomly
hanging, sometimes even the server itself becoming non responsible,
other times reloading the dialplan into memory but not reporting
anything on console, the last one becoming unusable until server
restart. This happened in the past as well but with the traffic
increase, it becomes more and more annoying. I suspect the same bug
which was present in the past with fifo hanging.
I should mention that I got about 2000 records in the dialplan table,
so I would say not that much loaded.

The version I am running:
sip1:/home/employee/dan# opensips -V
version: opensips 1.6.1-notls (x86_64/linux)
flags: STATS: Off, USE_IPV6, USE_TCP, DISABLE_NAGLE, USE_MCAST,
SHM_MEM, SHM_MMAP, PKG_MALLOC, F_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
svnrevision: 2:6509M
@(#) $Id: main.c 6169 2009-09-22 12:48:37Z bogdan_iancu $
main.c compiled on 14:43:30 Jan 11 2010 with gcc 4.3.2


All I could find in the logs was database reconnection in the case of
hang but no error or something else reported (running debug 7).
Bellow some of the log:
Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
DBG:mi_fifo:mi_fifo_server: entered consume
Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
DBG:mi_fifo:mi_fifo_server: **** done consume
Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
DBG:mi_fifo:mi_fifo_server: done parsing the mi tree
Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]: DBG:dialplan:dp_load_db: init
Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
INFO:db_mysql:db_mysql_submit_query: disconect event for 0x77f060
Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
INFO:db_mysql:reset_all_statements: reseting all statements on
connection: (0x77fd18) 0x77f060
Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
DBG:db_mysql:db_mysql_connect: opening connection:
mysql://xxxx:xxxx@192.168.11.253/sipeandb
Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
DBG:db_mysql:db_mysql_connect: connection type is 192.168.11.253 via
TCP/IP
Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
DBG:db_mysql:db_mysql_connect: protocol version is 10
Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
DBG:db_mysql:db_mysql_connect: server version is 5.0.51a-24+lenny1-log
Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
INFO:db_mysql:db_mysql_submit_query: re-connected successful for
0x77f060
Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
DBG:core:db_new_result: allocate 48 bytes for result set at 0x784258
Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
DBG:db_mysql:db_mysql_get_columns: 8 columns returned from the query
Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
DBG:core:db_allocate_columns: allocate 224 bytes for result columns at
0x7842a0
Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
DBG:db_mysql:db_mysql_get_columns: RES_NAMES(0x7842e0)[0]=[dpid]
Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
DBG:db_mysql:db_mysql_get_columns: use DB_INT result type
Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
DBG:db_mysql:db_mysql_get_columns: RES_NAMES(0x7842f0)[1]=[pr]
Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
DBG:db_mysql:db_mysql_get_columns: use DB_INT result type
Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
DBG:db_mysql:db_mysql_get_columns: RES_NAMES(0x784300)[2]=[match_op]
Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
DBG:db_mysql:db_mysql_get_columns: use DB_INT result type
Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
DBG:db_mysql:db_mysql_get_columns: RES_NAMES(0x784310)[3]=[match_exp]
Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
DBG:db_mysql:db_mysql_get_columns: use DB_STRING result type
Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
DBG:db_mysql:db_mysql_get_columns: RES_NAMES(0x784320)[4]=[match_len]
Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
DBG:db_mysql:db_mysql_get_columns: use DB_INT result type
Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
DBG:db_mysql:db_mysql_get_columns: RES_NAMES(0x784330)[5]=[subst_exp]
Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
DBG:db_mysql:db_mysql_get_columns: use DB_STRING result type
Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
DBG:db_mysql:db_mysql_get_columns: RES_NAMES(0x784340)[6]=[repl_exp]
Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
DBG:db_mysql:db_mysql_get_columns: use DB_STRING result type
Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
DBG:db_mysql:db_mysql_get_columns: RES_NAMES(0x784350)[7]=[attrs]
Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
DBG:db_mysql:db_mysql_get_columns: use DB_STRING result type
Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
DBG:db_mysql:db_mysql_fetch_result: converting row 0 of 2042 count 500
Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
DBG:core:db_allocate_rows: allocate 136000 bytes for result rows and
values at 0x79c5d8

Ta,
DanB

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Command "opensipsctl dialplan reload" randomly hangs

Bogdan-Andrei Iancu
Hi Dan,

Did you notice any error from the fifo process during the reload ? it
may be something related to locking (during reload) of the table -> this
may affect all the other processes.

So, any errors? any dead processes (like fifo one) ?

Regards,
Bogdan

DanB wrote:

> Hey Guys,
>
> I have noticed the "opensipsctl dialplan reload" command randomly
> hanging, sometimes even the server itself becoming non responsible,
> other times reloading the dialplan into memory but not reporting
> anything on console, the last one becoming unusable until server
> restart. This happened in the past as well but with the traffic
> increase, it becomes more and more annoying. I suspect the same bug
> which was present in the past with fifo hanging.
> I should mention that I got about 2000 records in the dialplan table,
> so I would say not that much loaded.
>
> The version I am running:
> sip1:/home/employee/dan# opensips -V
> version: opensips 1.6.1-notls (x86_64/linux)
> flags: STATS: Off, USE_IPV6, USE_TCP, DISABLE_NAGLE, USE_MCAST,
> SHM_MEM, SHM_MMAP, PKG_MALLOC, F_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
> MAX_URI_SIZE 1024, BUF_SIZE 65535
> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
> svnrevision: 2:6509M
> @(#) $Id: main.c 6169 2009-09-22 12:48:37Z bogdan_iancu $
> main.c compiled on 14:43:30 Jan 11 2010 with gcc 4.3.2
>
>
> All I could find in the logs was database reconnection in the case of
> hang but no error or something else reported (running debug 7).
> Bellow some of the log:
> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
> DBG:mi_fifo:mi_fifo_server: entered consume
> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
> DBG:mi_fifo:mi_fifo_server: **** done consume
> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
> DBG:mi_fifo:mi_fifo_server: done parsing the mi tree
> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]: DBG:dialplan:dp_load_db: init
> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
> INFO:db_mysql:db_mysql_submit_query: disconect event for 0x77f060
> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
> INFO:db_mysql:reset_all_statements: reseting all statements on
> connection: (0x77fd18) 0x77f060
> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
> DBG:db_mysql:db_mysql_connect: opening connection:
> mysql://xxxx:xxxx@192.168.11.253/sipeandb
> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
> DBG:db_mysql:db_mysql_connect: connection type is 192.168.11.253 via
> TCP/IP
> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
> DBG:db_mysql:db_mysql_connect: protocol version is 10
> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
> DBG:db_mysql:db_mysql_connect: server version is 5.0.51a-24+lenny1-log
> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
> INFO:db_mysql:db_mysql_submit_query: re-connected successful for
> 0x77f060
> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
> DBG:core:db_new_result: allocate 48 bytes for result set at 0x784258
> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
> DBG:db_mysql:db_mysql_get_columns: 8 columns returned from the query
> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
> DBG:core:db_allocate_columns: allocate 224 bytes for result columns at
> 0x7842a0
> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
> DBG:db_mysql:db_mysql_get_columns: RES_NAMES(0x7842e0)[0]=[dpid]
> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
> DBG:db_mysql:db_mysql_get_columns: use DB_INT result type
> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
> DBG:db_mysql:db_mysql_get_columns: RES_NAMES(0x7842f0)[1]=[pr]
> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
> DBG:db_mysql:db_mysql_get_columns: use DB_INT result type
> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
> DBG:db_mysql:db_mysql_get_columns: RES_NAMES(0x784300)[2]=[match_op]
> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
> DBG:db_mysql:db_mysql_get_columns: use DB_INT result type
> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
> DBG:db_mysql:db_mysql_get_columns: RES_NAMES(0x784310)[3]=[match_exp]
> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
> DBG:db_mysql:db_mysql_get_columns: use DB_STRING result type
> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
> DBG:db_mysql:db_mysql_get_columns: RES_NAMES(0x784320)[4]=[match_len]
> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
> DBG:db_mysql:db_mysql_get_columns: use DB_INT result type
> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
> DBG:db_mysql:db_mysql_get_columns: RES_NAMES(0x784330)[5]=[subst_exp]
> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
> DBG:db_mysql:db_mysql_get_columns: use DB_STRING result type
> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
> DBG:db_mysql:db_mysql_get_columns: RES_NAMES(0x784340)[6]=[repl_exp]
> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
> DBG:db_mysql:db_mysql_get_columns: use DB_STRING result type
> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
> DBG:db_mysql:db_mysql_get_columns: RES_NAMES(0x784350)[7]=[attrs]
> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
> DBG:db_mysql:db_mysql_get_columns: use DB_STRING result type
> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
> DBG:db_mysql:db_mysql_fetch_result: converting row 0 of 2042 count 500
> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
> DBG:core:db_allocate_rows: allocate 136000 bytes for result rows and
> values at 0x79c5d8
>
> Ta,
> DanB
>
> _______________________________________________
> Users mailing list
> [hidden email]
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
>  


--
Bogdan-Andrei Iancu
www.voice-system.ro


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Command "opensipsctl dialplan reload" randomly hangs

DanB-2
Hey Bogdan,

Thanks for coming back so fast.

There was no error reported neither on console nor in the syslog
(debug 7). I will need to check for dead process since all I could
spot was no reply back and console hanging, and be able to stop it
only with Ctrl+C. After Ctrl+C could not get the any other opensipsctl
commands to work.
Will need to wait few days more to know about dead process.

DanB

On Thu, Apr 22, 2010 at 6:55 PM, Bogdan-Andrei Iancu
<[hidden email]> wrote:

> Hi Dan,
>
> Did you notice any error from the fifo process during the reload ? it
> may be something related to locking (during reload) of the table -> this
> may affect all the other processes.
>
> So, any errors? any dead processes (like fifo one) ?
>
> Regards,
> Bogdan
>
> DanB wrote:
>> Hey Guys,
>>
>> I have noticed the "opensipsctl dialplan reload" command randomly
>> hanging, sometimes even the server itself becoming non responsible,
>> other times reloading the dialplan into memory but not reporting
>> anything on console, the last one becoming unusable until server
>> restart. This happened in the past as well but with the traffic
>> increase, it becomes more and more annoying. I suspect the same bug
>> which was present in the past with fifo hanging.
>> I should mention that I got about 2000 records in the dialplan table,
>> so I would say not that much loaded.
>>
>> The version I am running:
>> sip1:/home/employee/dan# opensips -V
>> version: opensips 1.6.1-notls (x86_64/linux)
>> flags: STATS: Off, USE_IPV6, USE_TCP, DISABLE_NAGLE, USE_MCAST,
>> SHM_MEM, SHM_MMAP, PKG_MALLOC, F_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
>> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
>> MAX_URI_SIZE 1024, BUF_SIZE 65535
>> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
>> svnrevision: 2:6509M
>> @(#) $Id: main.c 6169 2009-09-22 12:48:37Z bogdan_iancu $
>> main.c compiled on 14:43:30 Jan 11 2010 with gcc 4.3.2
>>
>>
>> All I could find in the logs was database reconnection in the case of
>> hang but no error or something else reported (running debug 7).
>> Bellow some of the log:
>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>> DBG:mi_fifo:mi_fifo_server: entered consume
>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>> DBG:mi_fifo:mi_fifo_server: **** done consume
>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>> DBG:mi_fifo:mi_fifo_server: done parsing the mi tree
>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]: DBG:dialplan:dp_load_db: init
>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>> INFO:db_mysql:db_mysql_submit_query: disconect event for 0x77f060
>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>> INFO:db_mysql:reset_all_statements: reseting all statements on
>> connection: (0x77fd18) 0x77f060
>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>> DBG:db_mysql:db_mysql_connect: opening connection:
>> mysql://xxxx:xxxx@192.168.11.253/sipeandb
>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>> DBG:db_mysql:db_mysql_connect: connection type is 192.168.11.253 via
>> TCP/IP
>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>> DBG:db_mysql:db_mysql_connect: protocol version is 10
>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>> DBG:db_mysql:db_mysql_connect: server version is 5.0.51a-24+lenny1-log
>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>> INFO:db_mysql:db_mysql_submit_query: re-connected successful for
>> 0x77f060
>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>> DBG:core:db_new_result: allocate 48 bytes for result set at 0x784258
>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>> DBG:db_mysql:db_mysql_get_columns: 8 columns returned from the query
>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>> DBG:core:db_allocate_columns: allocate 224 bytes for result columns at
>> 0x7842a0
>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>> DBG:db_mysql:db_mysql_get_columns: RES_NAMES(0x7842e0)[0]=[dpid]
>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>> DBG:db_mysql:db_mysql_get_columns: use DB_INT result type
>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>> DBG:db_mysql:db_mysql_get_columns: RES_NAMES(0x7842f0)[1]=[pr]
>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>> DBG:db_mysql:db_mysql_get_columns: use DB_INT result type
>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>> DBG:db_mysql:db_mysql_get_columns: RES_NAMES(0x784300)[2]=[match_op]
>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>> DBG:db_mysql:db_mysql_get_columns: use DB_INT result type
>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>> DBG:db_mysql:db_mysql_get_columns: RES_NAMES(0x784310)[3]=[match_exp]
>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>> DBG:db_mysql:db_mysql_get_columns: use DB_STRING result type
>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>> DBG:db_mysql:db_mysql_get_columns: RES_NAMES(0x784320)[4]=[match_len]
>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>> DBG:db_mysql:db_mysql_get_columns: use DB_INT result type
>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>> DBG:db_mysql:db_mysql_get_columns: RES_NAMES(0x784330)[5]=[subst_exp]
>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>> DBG:db_mysql:db_mysql_get_columns: use DB_STRING result type
>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>> DBG:db_mysql:db_mysql_get_columns: RES_NAMES(0x784340)[6]=[repl_exp]
>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>> DBG:db_mysql:db_mysql_get_columns: use DB_STRING result type
>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>> DBG:db_mysql:db_mysql_get_columns: RES_NAMES(0x784350)[7]=[attrs]
>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>> DBG:db_mysql:db_mysql_get_columns: use DB_STRING result type
>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>> DBG:db_mysql:db_mysql_fetch_result: converting row 0 of 2042 count 500
>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>> DBG:core:db_allocate_rows: allocate 136000 bytes for result rows and
>> values at 0x79c5d8
>>
>> Ta,
>> DanB
>>
>> _______________________________________________
>> Users mailing list
>> [hidden email]
>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>
>>
>
>
> --
> Bogdan-Andrei Iancu
> www.voice-system.ro
>
>
> _______________________________________________
> Users mailing list
> [hidden email]
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Command "opensipsctl dialplan reload" randomly hangs

Bogdan-Andrei Iancu
Hi Dan,

Your descriptions point to a blocked fifo process. Blocking maybe
because of some internal locking (you see 99% cpu usage) or some I/O
(normal cpu usage).

So, do the followings:

1) do "opensipsctl fifo ps" to see the PID of the fifo process
2) make fifo to block
3) check if the fifo process (by pid) is there - if yes, see how much
cpu it uses and try to attache with gdb to it to get a backtrace.

Regards,
Bogdan

DanB wrote:

> Hey Bogdan,
>
> Thanks for coming back so fast.
>
> There was no error reported neither on console nor in the syslog
> (debug 7). I will need to check for dead process since all I could
> spot was no reply back and console hanging, and be able to stop it
> only with Ctrl+C. After Ctrl+C could not get the any other opensipsctl
> commands to work.
> Will need to wait few days more to know about dead process.
>
> DanB
>
> On Thu, Apr 22, 2010 at 6:55 PM, Bogdan-Andrei Iancu
> <[hidden email]> wrote:
>  
>> Hi Dan,
>>
>> Did you notice any error from the fifo process during the reload ? it
>> may be something related to locking (during reload) of the table -> this
>> may affect all the other processes.
>>
>> So, any errors? any dead processes (like fifo one) ?
>>
>> Regards,
>> Bogdan
>>
>> DanB wrote:
>>    
>>> Hey Guys,
>>>
>>> I have noticed the "opensipsctl dialplan reload" command randomly
>>> hanging, sometimes even the server itself becoming non responsible,
>>> other times reloading the dialplan into memory but not reporting
>>> anything on console, the last one becoming unusable until server
>>> restart. This happened in the past as well but with the traffic
>>> increase, it becomes more and more annoying. I suspect the same bug
>>> which was present in the past with fifo hanging.
>>> I should mention that I got about 2000 records in the dialplan table,
>>> so I would say not that much loaded.
>>>
>>> The version I am running:
>>> sip1:/home/employee/dan# opensips -V
>>> version: opensips 1.6.1-notls (x86_64/linux)
>>> flags: STATS: Off, USE_IPV6, USE_TCP, DISABLE_NAGLE, USE_MCAST,
>>> SHM_MEM, SHM_MMAP, PKG_MALLOC, F_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
>>> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
>>> MAX_URI_SIZE 1024, BUF_SIZE 65535
>>> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
>>> svnrevision: 2:6509M
>>> @(#) $Id: main.c 6169 2009-09-22 12:48:37Z bogdan_iancu $
>>> main.c compiled on 14:43:30 Jan 11 2010 with gcc 4.3.2
>>>
>>>
>>> All I could find in the logs was database reconnection in the case of
>>> hang but no error or something else reported (running debug 7).
>>> Bellow some of the log:
>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>> DBG:mi_fifo:mi_fifo_server: entered consume
>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>> DBG:mi_fifo:mi_fifo_server: **** done consume
>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>> DBG:mi_fifo:mi_fifo_server: done parsing the mi tree
>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]: DBG:dialplan:dp_load_db: init
>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>> INFO:db_mysql:db_mysql_submit_query: disconect event for 0x77f060
>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>> INFO:db_mysql:reset_all_statements: reseting all statements on
>>> connection: (0x77fd18) 0x77f060
>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>> DBG:db_mysql:db_mysql_connect: opening connection:
>>> mysql://xxxx:xxxx@192.168.11.253/sipeandb
>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>> DBG:db_mysql:db_mysql_connect: connection type is 192.168.11.253 via
>>> TCP/IP
>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>> DBG:db_mysql:db_mysql_connect: protocol version is 10
>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>> DBG:db_mysql:db_mysql_connect: server version is 5.0.51a-24+lenny1-log
>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>> INFO:db_mysql:db_mysql_submit_query: re-connected successful for
>>> 0x77f060
>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>> DBG:core:db_new_result: allocate 48 bytes for result set at 0x784258
>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>> DBG:db_mysql:db_mysql_get_columns: 8 columns returned from the query
>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>> DBG:core:db_allocate_columns: allocate 224 bytes for result columns at
>>> 0x7842a0
>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>> DBG:db_mysql:db_mysql_get_columns: RES_NAMES(0x7842e0)[0]=[dpid]
>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>> DBG:db_mysql:db_mysql_get_columns: use DB_INT result type
>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>> DBG:db_mysql:db_mysql_get_columns: RES_NAMES(0x7842f0)[1]=[pr]
>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>> DBG:db_mysql:db_mysql_get_columns: use DB_INT result type
>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>> DBG:db_mysql:db_mysql_get_columns: RES_NAMES(0x784300)[2]=[match_op]
>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>> DBG:db_mysql:db_mysql_get_columns: use DB_INT result type
>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>> DBG:db_mysql:db_mysql_get_columns: RES_NAMES(0x784310)[3]=[match_exp]
>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>> DBG:db_mysql:db_mysql_get_columns: use DB_STRING result type
>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>> DBG:db_mysql:db_mysql_get_columns: RES_NAMES(0x784320)[4]=[match_len]
>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>> DBG:db_mysql:db_mysql_get_columns: use DB_INT result type
>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>> DBG:db_mysql:db_mysql_get_columns: RES_NAMES(0x784330)[5]=[subst_exp]
>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>> DBG:db_mysql:db_mysql_get_columns: use DB_STRING result type
>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>> DBG:db_mysql:db_mysql_get_columns: RES_NAMES(0x784340)[6]=[repl_exp]
>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>> DBG:db_mysql:db_mysql_get_columns: use DB_STRING result type
>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>> DBG:db_mysql:db_mysql_get_columns: RES_NAMES(0x784350)[7]=[attrs]
>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>> DBG:db_mysql:db_mysql_get_columns: use DB_STRING result type
>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>> DBG:db_mysql:db_mysql_fetch_result: converting row 0 of 2042 count 500
>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>> DBG:core:db_allocate_rows: allocate 136000 bytes for result rows and
>>> values at 0x79c5d8
>>>
>>> Ta,
>>> DanB
>>>
>>> _______________________________________________
>>> Users mailing list
>>> [hidden email]
>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>
>>>
>>>      
>> --
>> Bogdan-Andrei Iancu
>> www.voice-system.ro
>>
>>
>> _______________________________________________
>> Users mailing list
>> [hidden email]
>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>
>>    
>
> _______________________________________________
> Users mailing list
> [hidden email]
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
>  


--
Bogdan-Andrei Iancu
www.voice-system.ro


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Command "opensipsctl dialplan reload" randomly hangs

DanB-2
Hey Bogdan,

Hereby some more tests (I should mention that I cannot block on demand
the fifo, it simply randomly blocks, so I must be lucky when I am
trying to reproduce it).

Based on ps the cpu and memory consuption do not increase during the
hang (CPU:0.0  Memory:1.0).

Ta,
DanB

Log of the actions:
-------------------------------------------------------------------

Before reload:

sip1:~# opensipsctl ps|grep FIFO

Process::  ID=13 PID=26368 Type=MI FIFO

sip1:~# ps uww -p 26368

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND

opensips 26368  0.0  1.0 1147236 10788 ?       S    07:30   0:00
/usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 1024 -u opensips -g
opensips

sip1:~#



During hanging reload:

sip1:~# ps uww -p 26368

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND

opensips 26368  0.0  1.0 1147644 11040 ?       R    07:30   0:00
/usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 1024 -u opensips -g
opensips


After reload interrupted with crtl+c and /etc/init.d/opensips restart


sip1:~# ps uww -p 26368

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND

sip1:~# opensipsctl ps|grep FIFO

Process::  ID=13 PID=26716 Type=MI FIFO

sip1:~# ps uww -p 26716

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND

opensips 26716  0.0  0.1 1147164 1988 ?        S    14:59   0:00
/usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 1024 -u opensips -g
opensips

On Mon, Apr 26, 2010 at 12:25 PM, Bogdan-Andrei Iancu
<[hidden email]> wrote:

> Hi Dan,
>
> Your descriptions point to a blocked fifo process. Blocking maybe
> because of some internal locking (you see 99% cpu usage) or some I/O
> (normal cpu usage).
>
> So, do the followings:
>
> 1) do "opensipsctl fifo ps" to see the PID of the fifo process
> 2) make fifo to block
> 3) check if the fifo process (by pid) is there - if yes, see how much
> cpu it uses and try to attache with gdb to it to get a backtrace.
>
> Regards,
> Bogdan
>
> DanB wrote:
>> Hey Bogdan,
>>
>> Thanks for coming back so fast.
>>
>> There was no error reported neither on console nor in the syslog
>> (debug 7). I will need to check for dead process since all I could
>> spot was no reply back and console hanging, and be able to stop it
>> only with Ctrl+C. After Ctrl+C could not get the any other opensipsctl
>> commands to work.
>> Will need to wait few days more to know about dead process.
>>
>> DanB
>>
>> On Thu, Apr 22, 2010 at 6:55 PM, Bogdan-Andrei Iancu
>> <[hidden email]> wrote:
>>
>>> Hi Dan,
>>>
>>> Did you notice any error from the fifo process during the reload ? it
>>> may be something related to locking (during reload) of the table -> this
>>> may affect all the other processes.
>>>
>>> So, any errors? any dead processes (like fifo one) ?
>>>
>>> Regards,
>>> Bogdan
>>>
>>> DanB wrote:
>>>
>>>> Hey Guys,
>>>>
>>>> I have noticed the "opensipsctl dialplan reload" command randomly
>>>> hanging, sometimes even the server itself becoming non responsible,
>>>> other times reloading the dialplan into memory but not reporting
>>>> anything on console, the last one becoming unusable until server
>>>> restart. This happened in the past as well but with the traffic
>>>> increase, it becomes more and more annoying. I suspect the same bug
>>>> which was present in the past with fifo hanging.
>>>> I should mention that I got about 2000 records in the dialplan table,
>>>> so I would say not that much loaded.
>>>>
>>>> The version I am running:
>>>> sip1:/home/employee/dan# opensips -V
>>>> version: opensips 1.6.1-notls (x86_64/linux)
>>>> flags: STATS: Off, USE_IPV6, USE_TCP, DISABLE_NAGLE, USE_MCAST,
>>>> SHM_MEM, SHM_MMAP, PKG_MALLOC, F_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
>>>> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
>>>> MAX_URI_SIZE 1024, BUF_SIZE 65535
>>>> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
>>>> svnrevision: 2:6509M
>>>> @(#) $Id: main.c 6169 2009-09-22 12:48:37Z bogdan_iancu $
>>>> main.c compiled on 14:43:30 Jan 11 2010 with gcc 4.3.2
>>>>
>>>>
>>>> All I could find in the logs was database reconnection in the case of
>>>> hang but no error or something else reported (running debug 7).
>>>> Bellow some of the log:
>>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>>> DBG:mi_fifo:mi_fifo_server: entered consume
>>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>>> DBG:mi_fifo:mi_fifo_server: **** done consume
>>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>>> DBG:mi_fifo:mi_fifo_server: done parsing the mi tree
>>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]: DBG:dialplan:dp_load_db: init
>>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>>> INFO:db_mysql:db_mysql_submit_query: disconect event for 0x77f060
>>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>>> INFO:db_mysql:reset_all_statements: reseting all statements on
>>>> connection: (0x77fd18) 0x77f060
>>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>>> DBG:db_mysql:db_mysql_connect: opening connection:
>>>> mysql://xxxx:xxxx@192.168.11.253/sipeandb
>>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>>> DBG:db_mysql:db_mysql_connect: connection type is 192.168.11.253 via
>>>> TCP/IP
>>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>>> DBG:db_mysql:db_mysql_connect: protocol version is 10
>>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>>> DBG:db_mysql:db_mysql_connect: server version is 5.0.51a-24+lenny1-log
>>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>>> INFO:db_mysql:db_mysql_submit_query: re-connected successful for
>>>> 0x77f060
>>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>>> DBG:core:db_new_result: allocate 48 bytes for result set at 0x784258
>>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>>> DBG:db_mysql:db_mysql_get_columns: 8 columns returned from the query
>>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>>> DBG:core:db_allocate_columns: allocate 224 bytes for result columns at
>>>> 0x7842a0
>>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>>> DBG:db_mysql:db_mysql_get_columns: RES_NAMES(0x7842e0)[0]=[dpid]
>>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>>> DBG:db_mysql:db_mysql_get_columns: use DB_INT result type
>>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>>> DBG:db_mysql:db_mysql_get_columns: RES_NAMES(0x7842f0)[1]=[pr]
>>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>>> DBG:db_mysql:db_mysql_get_columns: use DB_INT result type
>>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>>> DBG:db_mysql:db_mysql_get_columns: RES_NAMES(0x784300)[2]=[match_op]
>>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>>> DBG:db_mysql:db_mysql_get_columns: use DB_INT result type
>>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>>> DBG:db_mysql:db_mysql_get_columns: RES_NAMES(0x784310)[3]=[match_exp]
>>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>>> DBG:db_mysql:db_mysql_get_columns: use DB_STRING result type
>>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>>> DBG:db_mysql:db_mysql_get_columns: RES_NAMES(0x784320)[4]=[match_len]
>>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>>> DBG:db_mysql:db_mysql_get_columns: use DB_INT result type
>>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>>> DBG:db_mysql:db_mysql_get_columns: RES_NAMES(0x784330)[5]=[subst_exp]
>>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>>> DBG:db_mysql:db_mysql_get_columns: use DB_STRING result type
>>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>>> DBG:db_mysql:db_mysql_get_columns: RES_NAMES(0x784340)[6]=[repl_exp]
>>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>>> DBG:db_mysql:db_mysql_get_columns: use DB_STRING result type
>>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>>> DBG:db_mysql:db_mysql_get_columns: RES_NAMES(0x784350)[7]=[attrs]
>>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>>> DBG:db_mysql:db_mysql_get_columns: use DB_STRING result type
>>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>>> DBG:db_mysql:db_mysql_fetch_result: converting row 0 of 2042 count 500
>>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>>> DBG:core:db_allocate_rows: allocate 136000 bytes for result rows and
>>>> values at 0x79c5d8
>>>>
>>>> Ta,
>>>> DanB
>>>>
>>>> _______________________________________________
>>>> Users mailing list
>>>> [hidden email]
>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>
>>>>
>>>>
>>> --
>>> Bogdan-Andrei Iancu
>>> www.voice-system.ro
>>>
>>>
>>> _______________________________________________
>>> Users mailing list
>>> [hidden email]
>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>
>>>
>>
>> _______________________________________________
>> Users mailing list
>> [hidden email]
>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>
>>
>
>
> --
> Bogdan-Andrei Iancu
> www.voice-system.ro
>
>
> _______________________________________________
> Users mailing list
> [hidden email]
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Command "opensipsctl dialplan reload" randomly hangs

Bogdan-Andrei Iancu
Hi Dan,

The vital information is the backtrace (with gdb) of the FIFO process
while blocked = as there is not CPU usage, I say the procs stuck  in a
IO op. BTW, try to run the test with debug=6 (if possible).

If you need help with the gdb syntax, let me know.

Regards,
Bogdan

DanB wrote:

> Hey Bogdan,
>
> Hereby some more tests (I should mention that I cannot block on demand
> the fifo, it simply randomly blocks, so I must be lucky when I am
> trying to reproduce it).
>
> Based on ps the cpu and memory consuption do not increase during the
> hang (CPU:0.0  Memory:1.0).
>
> Ta,
> DanB
>
> Log of the actions:
> -------------------------------------------------------------------
>
> Before reload:
>
> sip1:~# opensipsctl ps|grep FIFO
>
> Process::  ID=13 PID=26368 Type=MI FIFO
>
> sip1:~# ps uww -p 26368
>
> USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
>
> opensips 26368  0.0  1.0 1147236 10788 ?       S    07:30   0:00
> /usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 1024 -u opensips -g
> opensips
>
> sip1:~#
>
>
>
> During hanging reload:
>
> sip1:~# ps uww -p 26368
>
> USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
>
> opensips 26368  0.0  1.0 1147644 11040 ?       R    07:30   0:00
> /usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 1024 -u opensips -g
> opensips
>
>
> After reload interrupted with crtl+c and /etc/init.d/opensips restart
>
>
> sip1:~# ps uww -p 26368
>
> USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
>
> sip1:~# opensipsctl ps|grep FIFO
>
> Process::  ID=13 PID=26716 Type=MI FIFO
>
> sip1:~# ps uww -p 26716
>
> USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
>
> opensips 26716  0.0  0.1 1147164 1988 ?        S    14:59   0:00
> /usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 1024 -u opensips -g
> opensips
>
> On Mon, Apr 26, 2010 at 12:25 PM, Bogdan-Andrei Iancu
> <[hidden email]> wrote:
>  
>> Hi Dan,
>>
>> Your descriptions point to a blocked fifo process. Blocking maybe
>> because of some internal locking (you see 99% cpu usage) or some I/O
>> (normal cpu usage).
>>
>> So, do the followings:
>>
>> 1) do "opensipsctl fifo ps" to see the PID of the fifo process
>> 2) make fifo to block
>> 3) check if the fifo process (by pid) is there - if yes, see how much
>> cpu it uses and try to attache with gdb to it to get a backtrace.
>>
>> Regards,
>> Bogdan
>>
>> DanB wrote:
>>    
>>> Hey Bogdan,
>>>
>>> Thanks for coming back so fast.
>>>
>>> There was no error reported neither on console nor in the syslog
>>> (debug 7). I will need to check for dead process since all I could
>>> spot was no reply back and console hanging, and be able to stop it
>>> only with Ctrl+C. After Ctrl+C could not get the any other opensipsctl
>>> commands to work.
>>> Will need to wait few days more to know about dead process.
>>>
>>> DanB
>>>
>>> On Thu, Apr 22, 2010 at 6:55 PM, Bogdan-Andrei Iancu
>>> <[hidden email]> wrote:
>>>
>>>      
>>>> Hi Dan,
>>>>
>>>> Did you notice any error from the fifo process during the reload ? it
>>>> may be something related to locking (during reload) of the table -> this
>>>> may affect all the other processes.
>>>>
>>>> So, any errors? any dead processes (like fifo one) ?
>>>>
>>>> Regards,
>>>> Bogdan
>>>>
>>>> DanB wrote:
>>>>
>>>>        
>>>>> Hey Guys,
>>>>>
>>>>> I have noticed the "opensipsctl dialplan reload" command randomly
>>>>> hanging, sometimes even the server itself becoming non responsible,
>>>>> other times reloading the dialplan into memory but not reporting
>>>>> anything on console, the last one becoming unusable until server
>>>>> restart. This happened in the past as well but with the traffic
>>>>> increase, it becomes more and more annoying. I suspect the same bug
>>>>> which was present in the past with fifo hanging.
>>>>> I should mention that I got about 2000 records in the dialplan table,
>>>>> so I would say not that much loaded.
>>>>>
>>>>> The version I am running:
>>>>> sip1:/home/employee/dan# opensips -V
>>>>> version: opensips 1.6.1-notls (x86_64/linux)
>>>>> flags: STATS: Off, USE_IPV6, USE_TCP, DISABLE_NAGLE, USE_MCAST,
>>>>> SHM_MEM, SHM_MMAP, PKG_MALLOC, F_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
>>>>> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
>>>>> MAX_URI_SIZE 1024, BUF_SIZE 65535
>>>>> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
>>>>> svnrevision: 2:6509M
>>>>> @(#) $Id: main.c 6169 2009-09-22 12:48:37Z bogdan_iancu $
>>>>> main.c compiled on 14:43:30 Jan 11 2010 with gcc 4.3.2
>>>>>
>>>>>
>>>>> All I could find in the logs was database reconnection in the case of
>>>>> hang but no error or something else reported (running debug 7).
>>>>> Bellow some of the log:
>>>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>>>> DBG:mi_fifo:mi_fifo_server: entered consume
>>>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>>>> DBG:mi_fifo:mi_fifo_server: **** done consume
>>>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>>>> DBG:mi_fifo:mi_fifo_server: done parsing the mi tree
>>>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]: DBG:dialplan:dp_load_db: init
>>>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>>>> INFO:db_mysql:db_mysql_submit_query: disconect event for 0x77f060
>>>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>>>> INFO:db_mysql:reset_all_statements: reseting all statements on
>>>>> connection: (0x77fd18) 0x77f060
>>>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>>>> DBG:db_mysql:db_mysql_connect: opening connection:
>>>>> mysql://xxxx:xxxx@192.168.11.253/sipeandb
>>>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>>>> DBG:db_mysql:db_mysql_connect: connection type is 192.168.11.253 via
>>>>> TCP/IP
>>>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>>>> DBG:db_mysql:db_mysql_connect: protocol version is 10
>>>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>>>> DBG:db_mysql:db_mysql_connect: server version is 5.0.51a-24+lenny1-log
>>>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>>>> INFO:db_mysql:db_mysql_submit_query: re-connected successful for
>>>>> 0x77f060
>>>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>>>> DBG:core:db_new_result: allocate 48 bytes for result set at 0x784258
>>>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>>>> DBG:db_mysql:db_mysql_get_columns: 8 columns returned from the query
>>>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>>>> DBG:core:db_allocate_columns: allocate 224 bytes for result columns at
>>>>> 0x7842a0
>>>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>>>> DBG:db_mysql:db_mysql_get_columns: RES_NAMES(0x7842e0)[0]=[dpid]
>>>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>>>> DBG:db_mysql:db_mysql_get_columns: use DB_INT result type
>>>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>>>> DBG:db_mysql:db_mysql_get_columns: RES_NAMES(0x7842f0)[1]=[pr]
>>>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>>>> DBG:db_mysql:db_mysql_get_columns: use DB_INT result type
>>>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>>>> DBG:db_mysql:db_mysql_get_columns: RES_NAMES(0x784300)[2]=[match_op]
>>>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>>>> DBG:db_mysql:db_mysql_get_columns: use DB_INT result type
>>>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>>>> DBG:db_mysql:db_mysql_get_columns: RES_NAMES(0x784310)[3]=[match_exp]
>>>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>>>> DBG:db_mysql:db_mysql_get_columns: use DB_STRING result type
>>>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>>>> DBG:db_mysql:db_mysql_get_columns: RES_NAMES(0x784320)[4]=[match_len]
>>>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>>>> DBG:db_mysql:db_mysql_get_columns: use DB_INT result type
>>>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>>>> DBG:db_mysql:db_mysql_get_columns: RES_NAMES(0x784330)[5]=[subst_exp]
>>>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>>>> DBG:db_mysql:db_mysql_get_columns: use DB_STRING result type
>>>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>>>> DBG:db_mysql:db_mysql_get_columns: RES_NAMES(0x784340)[6]=[repl_exp]
>>>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>>>> DBG:db_mysql:db_mysql_get_columns: use DB_STRING result type
>>>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>>>> DBG:db_mysql:db_mysql_get_columns: RES_NAMES(0x784350)[7]=[attrs]
>>>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>>>> DBG:db_mysql:db_mysql_get_columns: use DB_STRING result type
>>>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>>>> DBG:db_mysql:db_mysql_fetch_result: converting row 0 of 2042 count 500
>>>>> Apr 22 14:27:19 sip1 /usr/sbin/opensips[16299]:
>>>>> DBG:core:db_allocate_rows: allocate 136000 bytes for result rows and
>>>>> values at 0x79c5d8
>>>>>
>>>>> Ta,
>>>>> DanB
>>>>>
>>>>> _______________________________________________
>>>>> Users mailing list
>>>>> [hidden email]
>>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>>
>>>>>
>>>>>
>>>>>          
>>>> --
>>>> Bogdan-Andrei Iancu
>>>> www.voice-system.ro
>>>>
>>>>
>>>> _______________________________________________
>>>> Users mailing list
>>>> [hidden email]
>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>
>>>>
>>>>        
>>> _______________________________________________
>>> Users mailing list
>>> [hidden email]
>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>
>>>
>>>      
>> --
>> Bogdan-Andrei Iancu
>> www.voice-system.ro
>>
>>
>> _______________________________________________
>> Users mailing list
>> [hidden email]
>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>
>>    
>
> _______________________________________________
> Users mailing list
> [hidden email]
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
>  


--
Bogdan-Andrei Iancu
www.voice-system.ro


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Command "opensipsctl dialplan reload" randomly hangs

DanB-2
Hi Bogdan,

After so long time I finally got a debug for you (running in debug 5).
As higher the debug is as smaller the chances are to happen. Running
in debug 6 we could never reproduce it (even in debug 5 it took some
time). If we run it in debug 2 we get the hang every few days.
Let me know if you need additional traces.


Ta,
DanB



############################################### Start BT
#######################################

> sip1:~# opensipsctl ps|grep FIFO
> Process::  ID=13 PID=3838 Type=MI FIFO
> sip1:~# opensipsctl ps|grep FIFO
> Process::  ID=13 PID=3838 Type=MI FIFO
> sip1:~# opensipsctl dialplan reload
> Terminated
> sip1:~# /etc/init.d/
> sip1:~# gdb /usr/sbin/opensips 3838
> GNU gdb 6.8-debian
> Copyright (C) 2008 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-linux-gnu"...
> (no debugging symbols found)
> Attaching to program: /usr/sbin/opensips, process 3838
> Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done.
> Loaded symbols for /lib/libdl.so.2
> Reading symbols from /lib/libresolv.so.2...(no debugging symbols found)...done.
> Loaded symbols for /lib/libresolv.so.2
> Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done.
> Loaded symbols for /lib/libc.so.6
> Reading symbols from /lib/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
> Loaded symbols for /lib64/ld-linux-x86-64.so.2
> Reading symbols from /usr/lib/opensips/modules/options.so...(no debugging symbols found)...done.
> Loaded symbols for /usr/lib/opensips/modules/options.so
> Reading symbols from /usr/lib/opensips/modules/sl.so...
> (no debugging symbols found)...done.
> Loaded symbols for /usr/lib/opensips/modules/sl.so
> Reading symbols from /usr/lib/opensips/modules/tm.so...(no debugging symbols found)...done.
> Loaded symbols for /usr/lib/opensips/modules/tm.so
> Reading symbols from /usr/lib/opensips/modules/signaling.so...(no debugging symbols found)...done.
> Loaded symbols for /usr/lib/opensips/modules/signaling.so
> Reading symbols from /usr/lib/opensips/modules/db_mysql.so...(no debugging symbols found)...done.
> Loaded symbols for /usr/lib/opensips/modules/db_mysql.so
> Reading symbols from /usr/lib/libmysqlclient.so.15...(no debugging symbols found)...done.
> Loaded symbols for /usr/lib/libmysqlclient.so.15
> Reading symbols from /lib/libpthread.so.0...
> (no debugging symbols found)...done.
> [Thread debugging using libthread_db enabled]
> [New Thread 0x7ffac71226e0 (LWP 3838)]
> Loaded symbols for /lib/libpthread.so.0
> Reading symbols from /lib/libcrypt.so.1...(no debugging symbols found)...done.
> Loaded symbols for /lib/libcrypt.so.1
> Reading symbols from /lib/libnsl.so.1...(no debugging symbols found)...done.
> Loaded symbols for /lib/libnsl.so.1
> Reading symbols from /lib/libm.so.6...(no debugging symbols found)...done.
> Loaded symbols for /lib/libm.so.6
> Reading symbols from /usr/lib/libz.so.1...(no debugging symbols found)...done.
> Loaded symbols for /usr/lib/libz.so.1
> Reading symbols from /usr/lib/opensips/modules/rr.so...
> (no debugging symbols found)...done.
> Loaded symbols for /usr/lib/opensips/modules/rr.so
> Reading symbols from /usr/lib/opensips/modules/maxfwd.so...(no debugging symbols found)...done.
> Loaded symbols for /usr/lib/opensips/modules/maxfwd.so
> Reading symbols from /usr/lib/opensips/modules/usrloc.so...(no debugging symbols found)...done.
> Loaded symbols for /usr/lib/opensips/modules/usrloc.so
> Reading symbols from /usr/lib/opensips/modules/textops.so...(no debugging symbols found)...done.
> Loaded symbols for /usr/lib/opensips/modules/textops.so
> Reading symbols from /usr/lib/opensips/modules/mi_fifo.so...(no debugging symbols found)...done.
> Loaded symbols for /usr/lib/opensips/modules/mi_fifo.so
> Reading symbols from /usr/lib/opensips/modules/mi_datagram.so...
> (no debugging symbols found)...done.
> Loaded symbols for /usr/lib/opensips/modules/mi_datagram.so
> Reading symbols from /usr/lib/opensips/modules/uri.so...(no debugging symbols found)...done.
> Loaded symbols for /usr/lib/opensips/modules/uri.so
> Reading symbols from /usr/lib/opensips/modules/xlog.so...(no debugging symbols found)...done.
> Loaded symbols for /usr/lib/opensips/modules/xlog.so
> Reading symbols from /usr/lib/opensips/modules/acc.so...(no debugging symbols found)...done.
> Loaded symbols for /usr/lib/opensips/modules/acc.so
> Reading symbols from /usr/lib/opensips/modules/dialog.so...(no debugging symbols found)...done.
> Loaded symbols for /usr/lib/opensips/modules/dialog.so
> Reading symbols from /usr/lib/opensips/modules/mediaproxy.so...
> (no debugging symbols found)...done.
> Loaded symbols for /usr/lib/opensips/modules/mediaproxy.so
> Reading symbols from /usr/lib/opensips/modules/nathelper.so...(no debugging symbols found)...done.
> Loaded symbols for /usr/lib/opensips/modules/nathelper.so
> Reading symbols from /usr/lib/opensips/modules/dialplan.so...(no debugging symbols found)...done.
> Loaded symbols for /usr/lib/opensips/modules/dialplan.so
> Reading symbols from /usr/lib/opensips/modules/drouting.so...(no debugging symbols found)...done.
> Loaded symbols for /usr/lib/opensips/modules/drouting.so
> Reading symbols from /usr/lib/opensips/modules/permissions.so...(no debugging symbols found)...done.
> Loaded symbols for /usr/lib/opensips/modules/permissions.so
> Reading symbols from /usr/lib/opensips/modules/aaa_radius.so...
> (no debugging symbols found)...done.
> Loaded symbols for /usr/lib/opensips/modules/aaa_radius.so
> Reading symbols from /usr/lib/libradiusclient-ng.so.2...(no debugging symbols found)...done.
> Loaded symbols for /usr/lib/libradiusclient-ng.so.2
> Reading symbols from /usr/lib/opensips/modules/auth.so...(no debugging symbols found)...done.
> Loaded symbols for /usr/lib/opensips/modules/auth.so
> Reading symbols from /usr/lib/opensips/modules/avpops.so...(no debugging symbols found)...done.
> Loaded symbols for /usr/lib/opensips/modules/avpops.so
> Reading symbols from /usr/lib/opensips/modules/uac.so...(no debugging symbols found)...done.
> Loaded symbols for /usr/lib/opensips/modules/uac.so
> Reading symbols from /usr/lib/opensips/modules/mi_xmlrpc.so...
> (no debugging symbols found)...done.
> Loaded symbols for /usr/lib/opensips/modules/mi_xmlrpc.so
> Reading symbols from /usr/lib/libxmlrpc_server.so.3...(no debugging symbols found)...done.
> Loaded symbols for /usr/lib/libxmlrpc_server.so.3
> Reading symbols from /usr/lib/libxmlrpc.so.3...(no debugging symbols found)...done.
> Loaded symbols for /usr/lib/libxmlrpc.so.3
> Reading symbols from /usr/lib/libxmlrpc_util.so.3...(no debugging symbols found)...done.
> Loaded symbols for /usr/lib/libxmlrpc_util.so.3
> Reading symbols from /usr/lib/libxmlrpc_xmlparse.so.3...(no debugging symbols found)...done.
> Loaded symbols for /usr/lib/libxmlrpc_xmlparse.so.3
> Reading symbols from /usr/lib/libxmlrpc_xmltok.so.3...
> (no debugging symbols found)...done.
> Loaded symbols for /usr/lib/libxmlrpc_xmltok.so.3
> Reading symbols from /usr/lib/libnss_db.so.2...(no debugging symbols found)...done.
> Loaded symbols for /usr/lib/libnss_db.so.2
> Reading symbols from /lib/libnss_files.so.2...(no debugging symbols found)...done.
> Loaded symbols for /lib/libnss_files.so.2
> Reading symbols from /usr/lib/libdb-4.6.so...(no debugging symbols found)...done.
> Loaded symbols for /usr/lib/libdb-4.6.so
> (no debugging symbols found)
> 0x00007ffac6843fc0 in nanosleep () from /lib/libc.so.6
> (gdb) bt
> #0  0x00007ffac6843fc0 in nanosleep () from /lib/libc.so.6
> #1  0x00007ffac686f4c4 in usleep () from /lib/libc.so.6
> #2  0x00007ffac334528a in dp_load_db () from /usr/lib/opensips/modules/dialplan.so
> #3  0x00007ffac3340c0a in ?? () from /usr/lib/opensips/modules/dialplan.so
> #4  0x00007ffac43e6e19 in mi_fifo_server () from /usr/lib/opensips/modules/mi_fifo.so
> #5  0x00007ffac43e7d70 in ?? () from /usr/lib/opensips/modules/mi_fifo.so
> #6  0x000000000045de29 in start_module_procs ()
> #7  0x000000000042432e in main ()
> (gdb) quit
> The program is running.  Quit anyway (and detach it)? (y or n) /etc/init.d      op/y
> Please answer y or n.
> The program is running.  Quit anyway (and detach it)? (y or n) y
> Quitting: Quit

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Command "opensipsctl dialplan reload" randomly hangs

Bogdan-Andrei Iancu
Hi Dan,

Interesting....the BT shows that the reloader process is blocked waiting
for ongoing reader processes to finish....

Could you print in gdb the values:  *reload_flag and *data_refcnt  ?

Also, please check the logs to be sure that you do not have any child
proc died .....

Regards,
Bogdan

DanB wrote:

> Hi Bogdan,
>
> After so long time I finally got a debug for you (running in debug 5).
> As higher the debug is as smaller the chances are to happen. Running
> in debug 6 we could never reproduce it (even in debug 5 it took some
> time). If we run it in debug 2 we get the hang every few days.
> Let me know if you need additional traces.
>
>
> Ta,
> DanB
>
>
>
> ############################################### Start BT
> #######################################
>
>  
>> sip1:~# opensipsctl ps|grep FIFO
>> Process::  ID=13 PID=3838 Type=MI FIFO
>> sip1:~# opensipsctl ps|grep FIFO
>> Process::  ID=13 PID=3838 Type=MI FIFO
>> sip1:~# opensipsctl dialplan reload
>> Terminated
>> sip1:~# /etc/init.d/
>> sip1:~# gdb /usr/sbin/opensips 3838
>> GNU gdb 6.8-debian
>> Copyright (C) 2008 Free Software Foundation, Inc.
>> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
>> This is free software: you are free to change and redistribute it.
>> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
>> and "show warranty" for details.
>> This GDB was configured as "x86_64-linux-gnu"...
>> (no debugging symbols found)
>> Attaching to program: /usr/sbin/opensips, process 3838
>> Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done.
>> Loaded symbols for /lib/libdl.so.2
>> Reading symbols from /lib/libresolv.so.2...(no debugging symbols found)...done.
>> Loaded symbols for /lib/libresolv.so.2
>> Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done.
>> Loaded symbols for /lib/libc.so.6
>> Reading symbols from /lib/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
>> Loaded symbols for /lib64/ld-linux-x86-64.so.2
>> Reading symbols from /usr/lib/opensips/modules/options.so...(no debugging symbols found)...done.
>> Loaded symbols for /usr/lib/opensips/modules/options.so
>> Reading symbols from /usr/lib/opensips/modules/sl.so...
>> (no debugging symbols found)...done.
>> Loaded symbols for /usr/lib/opensips/modules/sl.so
>> Reading symbols from /usr/lib/opensips/modules/tm.so...(no debugging symbols found)...done.
>> Loaded symbols for /usr/lib/opensips/modules/tm.so
>> Reading symbols from /usr/lib/opensips/modules/signaling.so...(no debugging symbols found)...done.
>> Loaded symbols for /usr/lib/opensips/modules/signaling.so
>> Reading symbols from /usr/lib/opensips/modules/db_mysql.so...(no debugging symbols found)...done.
>> Loaded symbols for /usr/lib/opensips/modules/db_mysql.so
>> Reading symbols from /usr/lib/libmysqlclient.so.15...(no debugging symbols found)...done.
>> Loaded symbols for /usr/lib/libmysqlclient.so.15
>> Reading symbols from /lib/libpthread.so.0...
>> (no debugging symbols found)...done.
>> [Thread debugging using libthread_db enabled]
>> [New Thread 0x7ffac71226e0 (LWP 3838)]
>> Loaded symbols for /lib/libpthread.so.0
>> Reading symbols from /lib/libcrypt.so.1...(no debugging symbols found)...done.
>> Loaded symbols for /lib/libcrypt.so.1
>> Reading symbols from /lib/libnsl.so.1...(no debugging symbols found)...done.
>> Loaded symbols for /lib/libnsl.so.1
>> Reading symbols from /lib/libm.so.6...(no debugging symbols found)...done.
>> Loaded symbols for /lib/libm.so.6
>> Reading symbols from /usr/lib/libz.so.1...(no debugging symbols found)...done.
>> Loaded symbols for /usr/lib/libz.so.1
>> Reading symbols from /usr/lib/opensips/modules/rr.so...
>> (no debugging symbols found)...done.
>> Loaded symbols for /usr/lib/opensips/modules/rr.so
>> Reading symbols from /usr/lib/opensips/modules/maxfwd.so...(no debugging symbols found)...done.
>> Loaded symbols for /usr/lib/opensips/modules/maxfwd.so
>> Reading symbols from /usr/lib/opensips/modules/usrloc.so...(no debugging symbols found)...done.
>> Loaded symbols for /usr/lib/opensips/modules/usrloc.so
>> Reading symbols from /usr/lib/opensips/modules/textops.so...(no debugging symbols found)...done.
>> Loaded symbols for /usr/lib/opensips/modules/textops.so
>> Reading symbols from /usr/lib/opensips/modules/mi_fifo.so...(no debugging symbols found)...done.
>> Loaded symbols for /usr/lib/opensips/modules/mi_fifo.so
>> Reading symbols from /usr/lib/opensips/modules/mi_datagram.so...
>> (no debugging symbols found)...done.
>> Loaded symbols for /usr/lib/opensips/modules/mi_datagram.so
>> Reading symbols from /usr/lib/opensips/modules/uri.so...(no debugging symbols found)...done.
>> Loaded symbols for /usr/lib/opensips/modules/uri.so
>> Reading symbols from /usr/lib/opensips/modules/xlog.so...(no debugging symbols found)...done.
>> Loaded symbols for /usr/lib/opensips/modules/xlog.so
>> Reading symbols from /usr/lib/opensips/modules/acc.so...(no debugging symbols found)...done.
>> Loaded symbols for /usr/lib/opensips/modules/acc.so
>> Reading symbols from /usr/lib/opensips/modules/dialog.so...(no debugging symbols found)...done.
>> Loaded symbols for /usr/lib/opensips/modules/dialog.so
>> Reading symbols from /usr/lib/opensips/modules/mediaproxy.so...
>> (no debugging symbols found)...done.
>> Loaded symbols for /usr/lib/opensips/modules/mediaproxy.so
>> Reading symbols from /usr/lib/opensips/modules/nathelper.so...(no debugging symbols found)...done.
>> Loaded symbols for /usr/lib/opensips/modules/nathelper.so
>> Reading symbols from /usr/lib/opensips/modules/dialplan.so...(no debugging symbols found)...done.
>> Loaded symbols for /usr/lib/opensips/modules/dialplan.so
>> Reading symbols from /usr/lib/opensips/modules/drouting.so...(no debugging symbols found)...done.
>> Loaded symbols for /usr/lib/opensips/modules/drouting.so
>> Reading symbols from /usr/lib/opensips/modules/permissions.so...(no debugging symbols found)...done.
>> Loaded symbols for /usr/lib/opensips/modules/permissions.so
>> Reading symbols from /usr/lib/opensips/modules/aaa_radius.so...
>> (no debugging symbols found)...done.
>> Loaded symbols for /usr/lib/opensips/modules/aaa_radius.so
>> Reading symbols from /usr/lib/libradiusclient-ng.so.2...(no debugging symbols found)...done.
>> Loaded symbols for /usr/lib/libradiusclient-ng.so.2
>> Reading symbols from /usr/lib/opensips/modules/auth.so...(no debugging symbols found)...done.
>> Loaded symbols for /usr/lib/opensips/modules/auth.so
>> Reading symbols from /usr/lib/opensips/modules/avpops.so...(no debugging symbols found)...done.
>> Loaded symbols for /usr/lib/opensips/modules/avpops.so
>> Reading symbols from /usr/lib/opensips/modules/uac.so...(no debugging symbols found)...done.
>> Loaded symbols for /usr/lib/opensips/modules/uac.so
>> Reading symbols from /usr/lib/opensips/modules/mi_xmlrpc.so...
>> (no debugging symbols found)...done.
>> Loaded symbols for /usr/lib/opensips/modules/mi_xmlrpc.so
>> Reading symbols from /usr/lib/libxmlrpc_server.so.3...(no debugging symbols found)...done.
>> Loaded symbols for /usr/lib/libxmlrpc_server.so.3
>> Reading symbols from /usr/lib/libxmlrpc.so.3...(no debugging symbols found)...done.
>> Loaded symbols for /usr/lib/libxmlrpc.so.3
>> Reading symbols from /usr/lib/libxmlrpc_util.so.3...(no debugging symbols found)...done.
>> Loaded symbols for /usr/lib/libxmlrpc_util.so.3
>> Reading symbols from /usr/lib/libxmlrpc_xmlparse.so.3...(no debugging symbols found)...done.
>> Loaded symbols for /usr/lib/libxmlrpc_xmlparse.so.3
>> Reading symbols from /usr/lib/libxmlrpc_xmltok.so.3...
>> (no debugging symbols found)...done.
>> Loaded symbols for /usr/lib/libxmlrpc_xmltok.so.3
>> Reading symbols from /usr/lib/libnss_db.so.2...(no debugging symbols found)...done.
>> Loaded symbols for /usr/lib/libnss_db.so.2
>> Reading symbols from /lib/libnss_files.so.2...(no debugging symbols found)...done.
>> Loaded symbols for /lib/libnss_files.so.2
>> Reading symbols from /usr/lib/libdb-4.6.so...(no debugging symbols found)...done.
>> Loaded symbols for /usr/lib/libdb-4.6.so
>> (no debugging symbols found)
>> 0x00007ffac6843fc0 in nanosleep () from /lib/libc.so.6
>> (gdb) bt
>> #0  0x00007ffac6843fc0 in nanosleep () from /lib/libc.so.6
>> #1  0x00007ffac686f4c4 in usleep () from /lib/libc.so.6
>> #2  0x00007ffac334528a in dp_load_db () from /usr/lib/opensips/modules/dialplan.so
>> #3  0x00007ffac3340c0a in ?? () from /usr/lib/opensips/modules/dialplan.so
>> #4  0x00007ffac43e6e19 in mi_fifo_server () from /usr/lib/opensips/modules/mi_fifo.so
>> #5  0x00007ffac43e7d70 in ?? () from /usr/lib/opensips/modules/mi_fifo.so
>> #6  0x000000000045de29 in start_module_procs ()
>> #7  0x000000000042432e in main ()
>> (gdb) quit
>> The program is running.  Quit anyway (and detach it)? (y or n) /etc/init.d      op/y
>> Please answer y or n.
>> The program is running.  Quit anyway (and detach it)? (y or n) y
>> Quitting: Quit
>>    
>
> _______________________________________________
> Users mailing list
> [hidden email]
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
>  


--
Bogdan-Andrei Iancu
OpenSIPS Bootcamp
20 - 24 September 2010, Frankfurt, Germany
www.voice-system.ro


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Command "opensipsctl dialplan reload" randomly hangs

DanB-2
Hey Bogdan,

Do I need to wait for the crash to print the values or I can do it any time?
In the logs I could not spot anything.

Ta,
DanB

On Sun, Aug 1, 2010 at 7:30 PM, Bogdan-Andrei Iancu
<[hidden email]> wrote:

> Hi Dan,
>
> Interesting....the BT shows that the reloader process is blocked waiting
> for ongoing reader processes to finish....
>
> Could you print in gdb the values:  *reload_flag and *data_refcnt  ?
>
> Also, please check the logs to be sure that you do not have any child
> proc died .....
>
> Regards,
> Bogdan
>
> DanB wrote:
>> Hi Bogdan,
>>
>> After so long time I finally got a debug for you (running in debug 5).
>> As higher the debug is as smaller the chances are to happen. Running
>> in debug 6 we could never reproduce it (even in debug 5 it took some
>> time). If we run it in debug 2 we get the hang every few days.
>> Let me know if you need additional traces.
>>
>>
>> Ta,
>> DanB
>>
>>
>>
>> ############################################### Start BT
>> #######################################
>>
>>
>>> sip1:~# opensipsctl ps|grep FIFO
>>> Process::  ID=13 PID=3838 Type=MI FIFO
>>> sip1:~# opensipsctl ps|grep FIFO
>>> Process::  ID=13 PID=3838 Type=MI FIFO
>>> sip1:~# opensipsctl dialplan reload
>>> Terminated
>>> sip1:~# /etc/init.d/
>>> sip1:~# gdb /usr/sbin/opensips 3838
>>> GNU gdb 6.8-debian
>>> Copyright (C) 2008 Free Software Foundation, Inc.
>>> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
>>> This is free software: you are free to change and redistribute it.
>>> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
>>> and "show warranty" for details.
>>> This GDB was configured as "x86_64-linux-gnu"...
>>> (no debugging symbols found)
>>> Attaching to program: /usr/sbin/opensips, process 3838
>>> Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done.
>>> Loaded symbols for /lib/libdl.so.2
>>> Reading symbols from /lib/libresolv.so.2...(no debugging symbols found)...done.
>>> Loaded symbols for /lib/libresolv.so.2
>>> Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done.
>>> Loaded symbols for /lib/libc.so.6
>>> Reading symbols from /lib/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
>>> Loaded symbols for /lib64/ld-linux-x86-64.so.2
>>> Reading symbols from /usr/lib/opensips/modules/options.so...(no debugging symbols found)...done.
>>> Loaded symbols for /usr/lib/opensips/modules/options.so
>>> Reading symbols from /usr/lib/opensips/modules/sl.so...
>>> (no debugging symbols found)...done.
>>> Loaded symbols for /usr/lib/opensips/modules/sl.so
>>> Reading symbols from /usr/lib/opensips/modules/tm.so...(no debugging symbols found)...done.
>>> Loaded symbols for /usr/lib/opensips/modules/tm.so
>>> Reading symbols from /usr/lib/opensips/modules/signaling.so...(no debugging symbols found)...done.
>>> Loaded symbols for /usr/lib/opensips/modules/signaling.so
>>> Reading symbols from /usr/lib/opensips/modules/db_mysql.so...(no debugging symbols found)...done.
>>> Loaded symbols for /usr/lib/opensips/modules/db_mysql.so
>>> Reading symbols from /usr/lib/libmysqlclient.so.15...(no debugging symbols found)...done.
>>> Loaded symbols for /usr/lib/libmysqlclient.so.15
>>> Reading symbols from /lib/libpthread.so.0...
>>> (no debugging symbols found)...done.
>>> [Thread debugging using libthread_db enabled]
>>> [New Thread 0x7ffac71226e0 (LWP 3838)]
>>> Loaded symbols for /lib/libpthread.so.0
>>> Reading symbols from /lib/libcrypt.so.1...(no debugging symbols found)...done.
>>> Loaded symbols for /lib/libcrypt.so.1
>>> Reading symbols from /lib/libnsl.so.1...(no debugging symbols found)...done.
>>> Loaded symbols for /lib/libnsl.so.1
>>> Reading symbols from /lib/libm.so.6...(no debugging symbols found)...done.
>>> Loaded symbols for /lib/libm.so.6
>>> Reading symbols from /usr/lib/libz.so.1...(no debugging symbols found)...done.
>>> Loaded symbols for /usr/lib/libz.so.1
>>> Reading symbols from /usr/lib/opensips/modules/rr.so...
>>> (no debugging symbols found)...done.
>>> Loaded symbols for /usr/lib/opensips/modules/rr.so
>>> Reading symbols from /usr/lib/opensips/modules/maxfwd.so...(no debugging symbols found)...done.
>>> Loaded symbols for /usr/lib/opensips/modules/maxfwd.so
>>> Reading symbols from /usr/lib/opensips/modules/usrloc.so...(no debugging symbols found)...done.
>>> Loaded symbols for /usr/lib/opensips/modules/usrloc.so
>>> Reading symbols from /usr/lib/opensips/modules/textops.so...(no debugging symbols found)...done.
>>> Loaded symbols for /usr/lib/opensips/modules/textops.so
>>> Reading symbols from /usr/lib/opensips/modules/mi_fifo.so...(no debugging symbols found)...done.
>>> Loaded symbols for /usr/lib/opensips/modules/mi_fifo.so
>>> Reading symbols from /usr/lib/opensips/modules/mi_datagram.so...
>>> (no debugging symbols found)...done.
>>> Loaded symbols for /usr/lib/opensips/modules/mi_datagram.so
>>> Reading symbols from /usr/lib/opensips/modules/uri.so...(no debugging symbols found)...done.
>>> Loaded symbols for /usr/lib/opensips/modules/uri.so
>>> Reading symbols from /usr/lib/opensips/modules/xlog.so...(no debugging symbols found)...done.
>>> Loaded symbols for /usr/lib/opensips/modules/xlog.so
>>> Reading symbols from /usr/lib/opensips/modules/acc.so...(no debugging symbols found)...done.
>>> Loaded symbols for /usr/lib/opensips/modules/acc.so
>>> Reading symbols from /usr/lib/opensips/modules/dialog.so...(no debugging symbols found)...done.
>>> Loaded symbols for /usr/lib/opensips/modules/dialog.so
>>> Reading symbols from /usr/lib/opensips/modules/mediaproxy.so...
>>> (no debugging symbols found)...done.
>>> Loaded symbols for /usr/lib/opensips/modules/mediaproxy.so
>>> Reading symbols from /usr/lib/opensips/modules/nathelper.so...(no debugging symbols found)...done.
>>> Loaded symbols for /usr/lib/opensips/modules/nathelper.so
>>> Reading symbols from /usr/lib/opensips/modules/dialplan.so...(no debugging symbols found)...done.
>>> Loaded symbols for /usr/lib/opensips/modules/dialplan.so
>>> Reading symbols from /usr/lib/opensips/modules/drouting.so...(no debugging symbols found)...done.
>>> Loaded symbols for /usr/lib/opensips/modules/drouting.so
>>> Reading symbols from /usr/lib/opensips/modules/permissions.so...(no debugging symbols found)...done.
>>> Loaded symbols for /usr/lib/opensips/modules/permissions.so
>>> Reading symbols from /usr/lib/opensips/modules/aaa_radius.so...
>>> (no debugging symbols found)...done.
>>> Loaded symbols for /usr/lib/opensips/modules/aaa_radius.so
>>> Reading symbols from /usr/lib/libradiusclient-ng.so.2...(no debugging symbols found)...done.
>>> Loaded symbols for /usr/lib/libradiusclient-ng.so.2
>>> Reading symbols from /usr/lib/opensips/modules/auth.so...(no debugging symbols found)...done.
>>> Loaded symbols for /usr/lib/opensips/modules/auth.so
>>> Reading symbols from /usr/lib/opensips/modules/avpops.so...(no debugging symbols found)...done.
>>> Loaded symbols for /usr/lib/opensips/modules/avpops.so
>>> Reading symbols from /usr/lib/opensips/modules/uac.so...(no debugging symbols found)...done.
>>> Loaded symbols for /usr/lib/opensips/modules/uac.so
>>> Reading symbols from /usr/lib/opensips/modules/mi_xmlrpc.so...
>>> (no debugging symbols found)...done.
>>> Loaded symbols for /usr/lib/opensips/modules/mi_xmlrpc.so
>>> Reading symbols from /usr/lib/libxmlrpc_server.so.3...(no debugging symbols found)...done.
>>> Loaded symbols for /usr/lib/libxmlrpc_server.so.3
>>> Reading symbols from /usr/lib/libxmlrpc.so.3...(no debugging symbols found)...done.
>>> Loaded symbols for /usr/lib/libxmlrpc.so.3
>>> Reading symbols from /usr/lib/libxmlrpc_util.so.3...(no debugging symbols found)...done.
>>> Loaded symbols for /usr/lib/libxmlrpc_util.so.3
>>> Reading symbols from /usr/lib/libxmlrpc_xmlparse.so.3...(no debugging symbols found)...done.
>>> Loaded symbols for /usr/lib/libxmlrpc_xmlparse.so.3
>>> Reading symbols from /usr/lib/libxmlrpc_xmltok.so.3...
>>> (no debugging symbols found)...done.
>>> Loaded symbols for /usr/lib/libxmlrpc_xmltok.so.3
>>> Reading symbols from /usr/lib/libnss_db.so.2...(no debugging symbols found)...done.
>>> Loaded symbols for /usr/lib/libnss_db.so.2
>>> Reading symbols from /lib/libnss_files.so.2...(no debugging symbols found)...done.
>>> Loaded symbols for /lib/libnss_files.so.2
>>> Reading symbols from /usr/lib/libdb-4.6.so...(no debugging symbols found)...done.
>>> Loaded symbols for /usr/lib/libdb-4.6.so
>>> (no debugging symbols found)
>>> 0x00007ffac6843fc0 in nanosleep () from /lib/libc.so.6
>>> (gdb) bt
>>> #0  0x00007ffac6843fc0 in nanosleep () from /lib/libc.so.6
>>> #1  0x00007ffac686f4c4 in usleep () from /lib/libc.so.6
>>> #2  0x00007ffac334528a in dp_load_db () from /usr/lib/opensips/modules/dialplan.so
>>> #3  0x00007ffac3340c0a in ?? () from /usr/lib/opensips/modules/dialplan.so
>>> #4  0x00007ffac43e6e19 in mi_fifo_server () from /usr/lib/opensips/modules/mi_fifo.so
>>> #5  0x00007ffac43e7d70 in ?? () from /usr/lib/opensips/modules/mi_fifo.so
>>> #6  0x000000000045de29 in start_module_procs ()
>>> #7  0x000000000042432e in main ()
>>> (gdb) quit
>>> The program is running.  Quit anyway (and detach it)? (y or n) /etc/init.d      op/y
>>> Please answer y or n.
>>> The program is running.  Quit anyway (and detach it)? (y or n) y
>>> Quitting: Quit
>>>
>>
>> _______________________________________________
>> Users mailing list
>> [hidden email]
>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>
>>
>
>
> --
> Bogdan-Andrei Iancu
> OpenSIPS Bootcamp
> 20 - 24 September 2010, Frankfurt, Germany
> www.voice-system.ro
>
>
> _______________________________________________
> Users mailing list
> [hidden email]
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Command "opensipsctl dialplan reload" randomly hangs

Bogdan-Andrei Iancu
Hi Dan,

you need to print those values from GDB when attached to a blocked FIFO
process. So, wait again to block, attach with GDB, go to frame 2 an d
print the values.

BTW, are you using mi_xmlrpc in the same time also ?

Regards,
Bogdan

DanB wrote:

> Hey Bogdan,
>
> Do I need to wait for the crash to print the values or I can do it any time?
> In the logs I could not spot anything.
>
> Ta,
> DanB
>
> On Sun, Aug 1, 2010 at 7:30 PM, Bogdan-Andrei Iancu
> <[hidden email]> wrote:
>  
>> Hi Dan,
>>
>> Interesting....the BT shows that the reloader process is blocked waiting
>> for ongoing reader processes to finish....
>>
>> Could you print in gdb the values:  *reload_flag and *data_refcnt  ?
>>
>> Also, please check the logs to be sure that you do not have any child
>> proc died .....
>>
>> Regards,
>> Bogdan
>>
>> DanB wrote:
>>    
>>> Hi Bogdan,
>>>
>>> After so long time I finally got a debug for you (running in debug 5).
>>> As higher the debug is as smaller the chances are to happen. Running
>>> in debug 6 we could never reproduce it (even in debug 5 it took some
>>> time). If we run it in debug 2 we get the hang every few days.
>>> Let me know if you need additional traces.
>>>
>>>
>>> Ta,
>>> DanB
>>>
>>>
>>>
>>> ############################################### Start BT
>>> #######################################
>>>
>>>
>>>      
>>>> sip1:~# opensipsctl ps|grep FIFO
>>>> Process::  ID=13 PID=3838 Type=MI FIFO
>>>> sip1:~# opensipsctl ps|grep FIFO
>>>> Process::  ID=13 PID=3838 Type=MI FIFO
>>>> sip1:~# opensipsctl dialplan reload
>>>> Terminated
>>>> sip1:~# /etc/init.d/
>>>> sip1:~# gdb /usr/sbin/opensips 3838
>>>> GNU gdb 6.8-debian
>>>> Copyright (C) 2008 Free Software Foundation, Inc.
>>>> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
>>>> This is free software: you are free to change and redistribute it.
>>>> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
>>>> and "show warranty" for details.
>>>> This GDB was configured as "x86_64-linux-gnu"...
>>>> (no debugging symbols found)
>>>> Attaching to program: /usr/sbin/opensips, process 3838
>>>> Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done.
>>>> Loaded symbols for /lib/libdl.so.2
>>>> Reading symbols from /lib/libresolv.so.2...(no debugging symbols found)...done.
>>>> Loaded symbols for /lib/libresolv.so.2
>>>> Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done.
>>>> Loaded symbols for /lib/libc.so.6
>>>> Reading symbols from /lib/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
>>>> Loaded symbols for /lib64/ld-linux-x86-64.so.2
>>>> Reading symbols from /usr/lib/opensips/modules/options.so...(no debugging symbols found)...done.
>>>> Loaded symbols for /usr/lib/opensips/modules/options.so
>>>> Reading symbols from /usr/lib/opensips/modules/sl.so...
>>>> (no debugging symbols found)...done.
>>>> Loaded symbols for /usr/lib/opensips/modules/sl.so
>>>> Reading symbols from /usr/lib/opensips/modules/tm.so...(no debugging symbols found)...done.
>>>> Loaded symbols for /usr/lib/opensips/modules/tm.so
>>>> Reading symbols from /usr/lib/opensips/modules/signaling.so...(no debugging symbols found)...done.
>>>> Loaded symbols for /usr/lib/opensips/modules/signaling.so
>>>> Reading symbols from /usr/lib/opensips/modules/db_mysql.so...(no debugging symbols found)...done.
>>>> Loaded symbols for /usr/lib/opensips/modules/db_mysql.so
>>>> Reading symbols from /usr/lib/libmysqlclient.so.15...(no debugging symbols found)...done.
>>>> Loaded symbols for /usr/lib/libmysqlclient.so.15
>>>> Reading symbols from /lib/libpthread.so.0...
>>>> (no debugging symbols found)...done.
>>>> [Thread debugging using libthread_db enabled]
>>>> [New Thread 0x7ffac71226e0 (LWP 3838)]
>>>> Loaded symbols for /lib/libpthread.so.0
>>>> Reading symbols from /lib/libcrypt.so.1...(no debugging symbols found)...done.
>>>> Loaded symbols for /lib/libcrypt.so.1
>>>> Reading symbols from /lib/libnsl.so.1...(no debugging symbols found)...done.
>>>> Loaded symbols for /lib/libnsl.so.1
>>>> Reading symbols from /lib/libm.so.6...(no debugging symbols found)...done.
>>>> Loaded symbols for /lib/libm.so.6
>>>> Reading symbols from /usr/lib/libz.so.1...(no debugging symbols found)...done.
>>>> Loaded symbols for /usr/lib/libz.so.1
>>>> Reading symbols from /usr/lib/opensips/modules/rr.so...
>>>> (no debugging symbols found)...done.
>>>> Loaded symbols for /usr/lib/opensips/modules/rr.so
>>>> Reading symbols from /usr/lib/opensips/modules/maxfwd.so...(no debugging symbols found)...done.
>>>> Loaded symbols for /usr/lib/opensips/modules/maxfwd.so
>>>> Reading symbols from /usr/lib/opensips/modules/usrloc.so...(no debugging symbols found)...done.
>>>> Loaded symbols for /usr/lib/opensips/modules/usrloc.so
>>>> Reading symbols from /usr/lib/opensips/modules/textops.so...(no debugging symbols found)...done.
>>>> Loaded symbols for /usr/lib/opensips/modules/textops.so
>>>> Reading symbols from /usr/lib/opensips/modules/mi_fifo.so...(no debugging symbols found)...done.
>>>> Loaded symbols for /usr/lib/opensips/modules/mi_fifo.so
>>>> Reading symbols from /usr/lib/opensips/modules/mi_datagram.so...
>>>> (no debugging symbols found)...done.
>>>> Loaded symbols for /usr/lib/opensips/modules/mi_datagram.so
>>>> Reading symbols from /usr/lib/opensips/modules/uri.so...(no debugging symbols found)...done.
>>>> Loaded symbols for /usr/lib/opensips/modules/uri.so
>>>> Reading symbols from /usr/lib/opensips/modules/xlog.so...(no debugging symbols found)...done.
>>>> Loaded symbols for /usr/lib/opensips/modules/xlog.so
>>>> Reading symbols from /usr/lib/opensips/modules/acc.so...(no debugging symbols found)...done.
>>>> Loaded symbols for /usr/lib/opensips/modules/acc.so
>>>> Reading symbols from /usr/lib/opensips/modules/dialog.so...(no debugging symbols found)...done.
>>>> Loaded symbols for /usr/lib/opensips/modules/dialog.so
>>>> Reading symbols from /usr/lib/opensips/modules/mediaproxy.so...
>>>> (no debugging symbols found)...done.
>>>> Loaded symbols for /usr/lib/opensips/modules/mediaproxy.so
>>>> Reading symbols from /usr/lib/opensips/modules/nathelper.so...(no debugging symbols found)...done.
>>>> Loaded symbols for /usr/lib/opensips/modules/nathelper.so
>>>> Reading symbols from /usr/lib/opensips/modules/dialplan.so...(no debugging symbols found)...done.
>>>> Loaded symbols for /usr/lib/opensips/modules/dialplan.so
>>>> Reading symbols from /usr/lib/opensips/modules/drouting.so...(no debugging symbols found)...done.
>>>> Loaded symbols for /usr/lib/opensips/modules/drouting.so
>>>> Reading symbols from /usr/lib/opensips/modules/permissions.so...(no debugging symbols found)...done.
>>>> Loaded symbols for /usr/lib/opensips/modules/permissions.so
>>>> Reading symbols from /usr/lib/opensips/modules/aaa_radius.so...
>>>> (no debugging symbols found)...done.
>>>> Loaded symbols for /usr/lib/opensips/modules/aaa_radius.so
>>>> Reading symbols from /usr/lib/libradiusclient-ng.so.2...(no debugging symbols found)...done.
>>>> Loaded symbols for /usr/lib/libradiusclient-ng.so.2
>>>> Reading symbols from /usr/lib/opensips/modules/auth.so...(no debugging symbols found)...done.
>>>> Loaded symbols for /usr/lib/opensips/modules/auth.so
>>>> Reading symbols from /usr/lib/opensips/modules/avpops.so...(no debugging symbols found)...done.
>>>> Loaded symbols for /usr/lib/opensips/modules/avpops.so
>>>> Reading symbols from /usr/lib/opensips/modules/uac.so...(no debugging symbols found)...done.
>>>> Loaded symbols for /usr/lib/opensips/modules/uac.so
>>>> Reading symbols from /usr/lib/opensips/modules/mi_xmlrpc.so...
>>>> (no debugging symbols found)...done.
>>>> Loaded symbols for /usr/lib/opensips/modules/mi_xmlrpc.so
>>>> Reading symbols from /usr/lib/libxmlrpc_server.so.3...(no debugging symbols found)...done.
>>>> Loaded symbols for /usr/lib/libxmlrpc_server.so.3
>>>> Reading symbols from /usr/lib/libxmlrpc.so.3...(no debugging symbols found)...done.
>>>> Loaded symbols for /usr/lib/libxmlrpc.so.3
>>>> Reading symbols from /usr/lib/libxmlrpc_util.so.3...(no debugging symbols found)...done.
>>>> Loaded symbols for /usr/lib/libxmlrpc_util.so.3
>>>> Reading symbols from /usr/lib/libxmlrpc_xmlparse.so.3...(no debugging symbols found)...done.
>>>> Loaded symbols for /usr/lib/libxmlrpc_xmlparse.so.3
>>>> Reading symbols from /usr/lib/libxmlrpc_xmltok.so.3...
>>>> (no debugging symbols found)...done.
>>>> Loaded symbols for /usr/lib/libxmlrpc_xmltok.so.3
>>>> Reading symbols from /usr/lib/libnss_db.so.2...(no debugging symbols found)...done.
>>>> Loaded symbols for /usr/lib/libnss_db.so.2
>>>> Reading symbols from /lib/libnss_files.so.2...(no debugging symbols found)...done.
>>>> Loaded symbols for /lib/libnss_files.so.2
>>>> Reading symbols from /usr/lib/libdb-4.6.so...(no debugging symbols found)...done.
>>>> Loaded symbols for /usr/lib/libdb-4.6.so
>>>> (no debugging symbols found)
>>>> 0x00007ffac6843fc0 in nanosleep () from /lib/libc.so.6
>>>> (gdb) bt
>>>> #0  0x00007ffac6843fc0 in nanosleep () from /lib/libc.so.6
>>>> #1  0x00007ffac686f4c4 in usleep () from /lib/libc.so.6
>>>> #2  0x00007ffac334528a in dp_load_db () from /usr/lib/opensips/modules/dialplan.so
>>>> #3  0x00007ffac3340c0a in ?? () from /usr/lib/opensips/modules/dialplan.so
>>>> #4  0x00007ffac43e6e19 in mi_fifo_server () from /usr/lib/opensips/modules/mi_fifo.so
>>>> #5  0x00007ffac43e7d70 in ?? () from /usr/lib/opensips/modules/mi_fifo.so
>>>> #6  0x000000000045de29 in start_module_procs ()
>>>> #7  0x000000000042432e in main ()
>>>> (gdb) quit
>>>> The program is running.  Quit anyway (and detach it)? (y or n) /etc/init.d      op/y
>>>> Please answer y or n.
>>>> The program is running.  Quit anyway (and detach it)? (y or n) y
>>>> Quitting: Quit
>>>>
>>>>        
>>> _______________________________________________
>>> Users mailing list
>>> [hidden email]
>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>
>>>
>>>      
>> --
>> Bogdan-Andrei Iancu
>> OpenSIPS Bootcamp
>> 20 - 24 September 2010, Frankfurt, Germany
>> www.voice-system.ro
>>
>>
>> _______________________________________________
>> Users mailing list
>> [hidden email]
>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>
>>    
>
> _______________________________________________
> Users mailing list
> [hidden email]
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
>  


--
Bogdan-Andrei Iancu
OpenSIPS Bootcamp
20 - 24 September 2010, Frankfurt, Germany
www.voice-system.ro


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users