OpenSIPS libmysqlclient.so segfault after MySQL restart

classic Classic list List threaded Threaded
19 messages Options
Reply | Threaded
Open this post in threaded view
|

OpenSIPS libmysqlclient.so segfault after MySQL restart

Om Bikram Thapa
Hi,

OpenSIPS is dying with "libmysqlclient.so Segmentation Fault" after
MySQL server restart. The log shows
"opensips[16769]: segfault at 4c8 ip 7f36728bf283 sp 7fff7ba11a90
error 4 in libmysqlclient.so.15.0.0[7f3672862000+1bf000]"

and gdb bt shows:

-------------
#0  0x00007f36728bf283 in mysql_stmt_result_metadata () from
/usr/lib/libmysqlclient.so.15
#1  0x00007f3672c707b7 in db_mysql_do_prepared_query (conn=0x78b278,
query=<value optimized out>, v=0x7fff7ba11d20, n=1, uv=0x0, un=0) at
dbase.c:363
#2  0x00007f3672c72e1f in db_mysql_query (_h=0x78b278, _k=<value
optimized out>, _op=<value optimized out>, _v=0x7fff7ba11d20,
_c=<value optimized out>,
    _n=1, _nc=2, _o=0x0, _r=0x7fff7ba11dc0) at dbase.c:591
#3  0x00007f366f3f0398 in authorize (_m=0x78fe98, _realm=<value
optimized out>, _table=<value optimized out>, _hftype=<value optimized
out>)
    at authorize.c:107
#4  0x000000000040f869 in do_action (a=0x780a48, msg=0x78fe98) at action.c:961
#5  0x000000000040e7e5 in run_action_list (a=<value optimized out>,
msg=0x78fe98) at action.c:139
#6  0x000000000046f194 in eval_elem (e=0x780b18, msg=0x78fe98,
val=0x0) at route.c:1189
#7  0x00000000004708ed in eval_expr (e=0x120b1e0, msg=0x78fe98,
val=0x0) at route.c:1486
#8  0x000000000047089c in eval_expr (e=0x780b60, msg=0x78fe98,
val=0x0) at route.c:1502
#9  0x00000000004708c5 in eval_expr (e=0x780ba8, msg=0x78fe98,
val=0x0) at route.c:1507
#10 0x000000000040f91c in do_action (a=0x780f40, msg=0x78fe98) at action.c:688
#11 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
msg=0x78fe98) at action.c:139
#12 0x00000000004114d3 in do_action (a=0x781eb0, msg=0x78fe98) at action.c:705
#13 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
msg=0x78fe98) at action.c:139
#14 0x0000000000410e07 in do_action (a=0x77ee88, msg=0x78fe98) at action.c:119
#15 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
msg=0x78fe98) at action.c:139
#16 0x00000000004114d3 in do_action (a=0x77f028, msg=0x78fe98) at action.c:705
#17 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
msg=0x78fe98) at action.c:139
#18 0x00000000004125fe in run_top_route (a=0x777e78, msg=0x78fe98) at
action.c:119
#19 0x000000000045e898 in receive_msg (
    buf=0x749180 "REGISTER sip:xxxxx SIP/2.0\r\nCSeq: 2
REGISTER\r\nVia: SIP/2.0/UDP
x.x.x.x:5061;branch=z9hG4bK60f1b2dd-57fa-dd11-9401-0015c5404858;rport\r\nUser-Agent:
Ekiga/2.0.12\r\nAuthorization: Dige"..., len=749,
rcv_info=0x7fff7ba13530) at receive.c:165
#20 0x000000000049d3a6 in udp_rcv_loop () at udp_server.c:449
#21 0x00000000004291fb in main (argc=<value optimized out>,
argv=0x7fff7ba13718) at main.c:778
------------

Server runs fine after restart until MySQL is restarted again. In my
lab setup, OpenSIPS is getting killed triggered by registration
request every morning (probably due to MySQL being restarted daily
with logrotate).

The server is the latest trunk on Debian lenny/AMD64.

Thanks,
Om.

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: OpenSIPS libmysqlclient.so segfault after MySQL restart

Bogdan-Andrei Iancu
Hi Om,

Thanks for the report - I may have an idea what is the problem; I will
try to reproduce the crash in the following days.

Could you please open a bug report on the tracker - just not to forget
about it :D (see http://www.opensips.org/index.php?n=Development.Tracker)

Regards,
Bogdan

Om Bikram Thapa wrote:

> Hi,
>
> OpenSIPS is dying with "libmysqlclient.so Segmentation Fault" after
> MySQL server restart. The log shows
> "opensips[16769]: segfault at 4c8 ip 7f36728bf283 sp 7fff7ba11a90
> error 4 in libmysqlclient.so.15.0.0[7f3672862000+1bf000]"
>
> and gdb bt shows:
>
> -------------
> #0  0x00007f36728bf283 in mysql_stmt_result_metadata () from
> /usr/lib/libmysqlclient.so.15
> #1  0x00007f3672c707b7 in db_mysql_do_prepared_query (conn=0x78b278,
> query=<value optimized out>, v=0x7fff7ba11d20, n=1, uv=0x0, un=0) at
> dbase.c:363
> #2  0x00007f3672c72e1f in db_mysql_query (_h=0x78b278, _k=<value
> optimized out>, _op=<value optimized out>, _v=0x7fff7ba11d20,
> _c=<value optimized out>,
>     _n=1, _nc=2, _o=0x0, _r=0x7fff7ba11dc0) at dbase.c:591
> #3  0x00007f366f3f0398 in authorize (_m=0x78fe98, _realm=<value
> optimized out>, _table=<value optimized out>, _hftype=<value optimized
> out>)
>     at authorize.c:107
> #4  0x000000000040f869 in do_action (a=0x780a48, msg=0x78fe98) at action.c:961
> #5  0x000000000040e7e5 in run_action_list (a=<value optimized out>,
> msg=0x78fe98) at action.c:139
> #6  0x000000000046f194 in eval_elem (e=0x780b18, msg=0x78fe98,
> val=0x0) at route.c:1189
> #7  0x00000000004708ed in eval_expr (e=0x120b1e0, msg=0x78fe98,
> val=0x0) at route.c:1486
> #8  0x000000000047089c in eval_expr (e=0x780b60, msg=0x78fe98,
> val=0x0) at route.c:1502
> #9  0x00000000004708c5 in eval_expr (e=0x780ba8, msg=0x78fe98,
> val=0x0) at route.c:1507
> #10 0x000000000040f91c in do_action (a=0x780f40, msg=0x78fe98) at action.c:688
> #11 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
> msg=0x78fe98) at action.c:139
> #12 0x00000000004114d3 in do_action (a=0x781eb0, msg=0x78fe98) at action.c:705
> #13 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
> msg=0x78fe98) at action.c:139
> #14 0x0000000000410e07 in do_action (a=0x77ee88, msg=0x78fe98) at action.c:119
> #15 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
> msg=0x78fe98) at action.c:139
> #16 0x00000000004114d3 in do_action (a=0x77f028, msg=0x78fe98) at action.c:705
> #17 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
> msg=0x78fe98) at action.c:139
> #18 0x00000000004125fe in run_top_route (a=0x777e78, msg=0x78fe98) at
> action.c:119
> #19 0x000000000045e898 in receive_msg (
>     buf=0x749180 "REGISTER sip:xxxxx SIP/2.0\r\nCSeq: 2
> REGISTER\r\nVia: SIP/2.0/UDP
> x.x.x.x:5061;branch=z9hG4bK60f1b2dd-57fa-dd11-9401-0015c5404858;rport\r\nUser-Agent:
> Ekiga/2.0.12\r\nAuthorization: Dige"..., len=749,
> rcv_info=0x7fff7ba13530) at receive.c:165
> #20 0x000000000049d3a6 in udp_rcv_loop () at udp_server.c:449
> #21 0x00000000004291fb in main (argc=<value optimized out>,
> argv=0x7fff7ba13718) at main.c:778
> ------------
>
> Server runs fine after restart until MySQL is restarted again. In my
> lab setup, OpenSIPS is getting killed triggered by registration
> request every morning (probably due to MySQL being restarted daily
> with logrotate).
>
> The server is the latest trunk on Debian lenny/AMD64.
>
> Thanks,
> Om.
>
> _______________________________________________
> Users mailing list
> [hidden email]
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
>  


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: OpenSIPS libmysqlclient.so segfault after MySQL restart

Om Bikram Thapa
Hi Bogdan,

I have submitted the bug on the tracker.

Thanks,
Om.

On Tue, Feb 17, 2009 at 3:50 AM, Bogdan-Andrei Iancu
<[hidden email]> wrote:

> Hi Om,
>
> Thanks for the report - I may have an idea what is the problem; I will try
> to reproduce the crash in the following days.
>
> Could you please open a bug report on the tracker - just not to forget about
> it :D (see http://www.opensips.org/index.php?n=Development.Tracker)
>
> Regards,
> Bogdan
>
> Om Bikram Thapa wrote:
>>
>> Hi,
>>
>> OpenSIPS is dying with "libmysqlclient.so Segmentation Fault" after
>> MySQL server restart. The log shows
>> "opensips[16769]: segfault at 4c8 ip 7f36728bf283 sp 7fff7ba11a90
>> error 4 in libmysqlclient.so.15.0.0[7f3672862000+1bf000]"
>>
>> and gdb bt shows:
>>
>> -------------
>> #0  0x00007f36728bf283 in mysql_stmt_result_metadata () from
>> /usr/lib/libmysqlclient.so.15
>> #1  0x00007f3672c707b7 in db_mysql_do_prepared_query (conn=0x78b278,
>> query=<value optimized out>, v=0x7fff7ba11d20, n=1, uv=0x0, un=0) at
>> dbase.c:363
>> #2  0x00007f3672c72e1f in db_mysql_query (_h=0x78b278, _k=<value
>> optimized out>, _op=<value optimized out>, _v=0x7fff7ba11d20,
>> _c=<value optimized out>,
>>    _n=1, _nc=2, _o=0x0, _r=0x7fff7ba11dc0) at dbase.c:591
>> #3  0x00007f366f3f0398 in authorize (_m=0x78fe98, _realm=<value
>> optimized out>, _table=<value optimized out>, _hftype=<value optimized
>> out>)
>>    at authorize.c:107
>> #4  0x000000000040f869 in do_action (a=0x780a48, msg=0x78fe98) at
>> action.c:961
>> #5  0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>> msg=0x78fe98) at action.c:139
>> #6  0x000000000046f194 in eval_elem (e=0x780b18, msg=0x78fe98,
>> val=0x0) at route.c:1189
>> #7  0x00000000004708ed in eval_expr (e=0x120b1e0, msg=0x78fe98,
>> val=0x0) at route.c:1486
>> #8  0x000000000047089c in eval_expr (e=0x780b60, msg=0x78fe98,
>> val=0x0) at route.c:1502
>> #9  0x00000000004708c5 in eval_expr (e=0x780ba8, msg=0x78fe98,
>> val=0x0) at route.c:1507
>> #10 0x000000000040f91c in do_action (a=0x780f40, msg=0x78fe98) at
>> action.c:688
>> #11 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>> msg=0x78fe98) at action.c:139
>> #12 0x00000000004114d3 in do_action (a=0x781eb0, msg=0x78fe98) at
>> action.c:705
>> #13 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>> msg=0x78fe98) at action.c:139
>> #14 0x0000000000410e07 in do_action (a=0x77ee88, msg=0x78fe98) at
>> action.c:119
>> #15 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>> msg=0x78fe98) at action.c:139
>> #16 0x00000000004114d3 in do_action (a=0x77f028, msg=0x78fe98) at
>> action.c:705
>> #17 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>> msg=0x78fe98) at action.c:139
>> #18 0x00000000004125fe in run_top_route (a=0x777e78, msg=0x78fe98) at
>> action.c:119
>> #19 0x000000000045e898 in receive_msg (
>>    buf=0x749180 "REGISTER sip:xxxxx SIP/2.0\r\nCSeq: 2
>> REGISTER\r\nVia: SIP/2.0/UDP
>>
>> x.x.x.x:5061;branch=z9hG4bK60f1b2dd-57fa-dd11-9401-0015c5404858;rport\r\nUser-Agent:
>> Ekiga/2.0.12\r\nAuthorization: Dige"..., len=749,
>> rcv_info=0x7fff7ba13530) at receive.c:165
>> #20 0x000000000049d3a6 in udp_rcv_loop () at udp_server.c:449
>> #21 0x00000000004291fb in main (argc=<value optimized out>,
>> argv=0x7fff7ba13718) at main.c:778
>> ------------
>>
>> Server runs fine after restart until MySQL is restarted again. In my
>> lab setup, OpenSIPS is getting killed triggered by registration
>> request every morning (probably due to MySQL being restarted daily
>> with logrotate).
>>
>> The server is the latest trunk on Debian lenny/AMD64.
>>
>> Thanks,
>> Om.
>>
>> _______________________________________________
>> Users mailing list
>> [hidden email]
>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>
>>
>
>

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: OpenSIPS libmysqlclient.so segfault after MySQL restart

Bogdan-Andrei Iancu
Hi Om,

Ok - got it (2608188) - thank you again for your report.

Regards,
Bogdan


Om Bikram Thapa wrote:

> Hi Bogdan,
>
> I have submitted the bug on the tracker.
>
> Thanks,
> Om.
>
> On Tue, Feb 17, 2009 at 3:50 AM, Bogdan-Andrei Iancu
> <[hidden email]> wrote:
>  
>> Hi Om,
>>
>> Thanks for the report - I may have an idea what is the problem; I will try
>> to reproduce the crash in the following days.
>>
>> Could you please open a bug report on the tracker - just not to forget about
>> it :D (see http://www.opensips.org/index.php?n=Development.Tracker)
>>
>> Regards,
>> Bogdan
>>
>> Om Bikram Thapa wrote:
>>    
>>> Hi,
>>>
>>> OpenSIPS is dying with "libmysqlclient.so Segmentation Fault" after
>>> MySQL server restart. The log shows
>>> "opensips[16769]: segfault at 4c8 ip 7f36728bf283 sp 7fff7ba11a90
>>> error 4 in libmysqlclient.so.15.0.0[7f3672862000+1bf000]"
>>>
>>> and gdb bt shows:
>>>
>>> -------------
>>> #0  0x00007f36728bf283 in mysql_stmt_result_metadata () from
>>> /usr/lib/libmysqlclient.so.15
>>> #1  0x00007f3672c707b7 in db_mysql_do_prepared_query (conn=0x78b278,
>>> query=<value optimized out>, v=0x7fff7ba11d20, n=1, uv=0x0, un=0) at
>>> dbase.c:363
>>> #2  0x00007f3672c72e1f in db_mysql_query (_h=0x78b278, _k=<value
>>> optimized out>, _op=<value optimized out>, _v=0x7fff7ba11d20,
>>> _c=<value optimized out>,
>>>    _n=1, _nc=2, _o=0x0, _r=0x7fff7ba11dc0) at dbase.c:591
>>> #3  0x00007f366f3f0398 in authorize (_m=0x78fe98, _realm=<value
>>> optimized out>, _table=<value optimized out>, _hftype=<value optimized
>>> out>)
>>>    at authorize.c:107
>>> #4  0x000000000040f869 in do_action (a=0x780a48, msg=0x78fe98) at
>>> action.c:961
>>> #5  0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>> msg=0x78fe98) at action.c:139
>>> #6  0x000000000046f194 in eval_elem (e=0x780b18, msg=0x78fe98,
>>> val=0x0) at route.c:1189
>>> #7  0x00000000004708ed in eval_expr (e=0x120b1e0, msg=0x78fe98,
>>> val=0x0) at route.c:1486
>>> #8  0x000000000047089c in eval_expr (e=0x780b60, msg=0x78fe98,
>>> val=0x0) at route.c:1502
>>> #9  0x00000000004708c5 in eval_expr (e=0x780ba8, msg=0x78fe98,
>>> val=0x0) at route.c:1507
>>> #10 0x000000000040f91c in do_action (a=0x780f40, msg=0x78fe98) at
>>> action.c:688
>>> #11 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>> msg=0x78fe98) at action.c:139
>>> #12 0x00000000004114d3 in do_action (a=0x781eb0, msg=0x78fe98) at
>>> action.c:705
>>> #13 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>> msg=0x78fe98) at action.c:139
>>> #14 0x0000000000410e07 in do_action (a=0x77ee88, msg=0x78fe98) at
>>> action.c:119
>>> #15 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>> msg=0x78fe98) at action.c:139
>>> #16 0x00000000004114d3 in do_action (a=0x77f028, msg=0x78fe98) at
>>> action.c:705
>>> #17 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>> msg=0x78fe98) at action.c:139
>>> #18 0x00000000004125fe in run_top_route (a=0x777e78, msg=0x78fe98) at
>>> action.c:119
>>> #19 0x000000000045e898 in receive_msg (
>>>    buf=0x749180 "REGISTER sip:xxxxx SIP/2.0\r\nCSeq: 2
>>> REGISTER\r\nVia: SIP/2.0/UDP
>>>
>>> x.x.x.x:5061;branch=z9hG4bK60f1b2dd-57fa-dd11-9401-0015c5404858;rport\r\nUser-Agent:
>>> Ekiga/2.0.12\r\nAuthorization: Dige"..., len=749,
>>> rcv_info=0x7fff7ba13530) at receive.c:165
>>> #20 0x000000000049d3a6 in udp_rcv_loop () at udp_server.c:449
>>> #21 0x00000000004291fb in main (argc=<value optimized out>,
>>> argv=0x7fff7ba13718) at main.c:778
>>> ------------
>>>
>>> Server runs fine after restart until MySQL is restarted again. In my
>>> lab setup, OpenSIPS is getting killed triggered by registration
>>> request every morning (probably due to MySQL being restarted daily
>>> with logrotate).
>>>
>>> The server is the latest trunk on Debian lenny/AMD64.
>>>
>>> Thanks,
>>> Om.
>>>
>>> _______________________________________________
>>> Users mailing list
>>> [hidden email]
>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>
>>>
>>>      
>>    
>
>  


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: OpenSIPS libmysqlclient.so segfault after MySQL restart

Om Bikram Thapa
Hi Bogdan,

Just to follow up on this issue, the crash is happening even without
restarting MySQL server. It seems to show up every morning when
registration (or maybe any db operation) is performed. I think the
issue has to do with some kind of timeout related to OpenSIPS and
libmysqlclient after a long duration of inactivity.

Thanks,
Om.

On Tue, Feb 17, 2009 at 6:24 PM, Bogdan-Andrei Iancu
<[hidden email]> wrote:

> Hi Om,
>
> Ok - got it (2608188) - thank you again for your report.
>
> Regards,
> Bogdan
>
>
> Om Bikram Thapa wrote:
>>
>> Hi Bogdan,
>>
>> I have submitted the bug on the tracker.
>>
>> Thanks,
>> Om.
>>
>> On Tue, Feb 17, 2009 at 3:50 AM, Bogdan-Andrei Iancu
>> <[hidden email]> wrote:
>>
>>>
>>> Hi Om,
>>>
>>> Thanks for the report - I may have an idea what is the problem; I will
>>> try
>>> to reproduce the crash in the following days.
>>>
>>> Could you please open a bug report on the tracker - just not to forget
>>> about
>>> it :D (see http://www.opensips.org/index.php?n=Development.Tracker)
>>>
>>> Regards,
>>> Bogdan
>>>
>>> Om Bikram Thapa wrote:
>>>
>>>>
>>>> Hi,
>>>>
>>>> OpenSIPS is dying with "libmysqlclient.so Segmentation Fault" after
>>>> MySQL server restart. The log shows
>>>> "opensips[16769]: segfault at 4c8 ip 7f36728bf283 sp 7fff7ba11a90
>>>> error 4 in libmysqlclient.so.15.0.0[7f3672862000+1bf000]"
>>>>
>>>> and gdb bt shows:
>>>>
>>>> -------------
>>>> #0  0x00007f36728bf283 in mysql_stmt_result_metadata () from
>>>> /usr/lib/libmysqlclient.so.15
>>>> #1  0x00007f3672c707b7 in db_mysql_do_prepared_query (conn=0x78b278,
>>>> query=<value optimized out>, v=0x7fff7ba11d20, n=1, uv=0x0, un=0) at
>>>> dbase.c:363
>>>> #2  0x00007f3672c72e1f in db_mysql_query (_h=0x78b278, _k=<value
>>>> optimized out>, _op=<value optimized out>, _v=0x7fff7ba11d20,
>>>> _c=<value optimized out>,
>>>>   _n=1, _nc=2, _o=0x0, _r=0x7fff7ba11dc0) at dbase.c:591
>>>> #3  0x00007f366f3f0398 in authorize (_m=0x78fe98, _realm=<value
>>>> optimized out>, _table=<value optimized out>, _hftype=<value optimized
>>>> out>)
>>>>   at authorize.c:107
>>>> #4  0x000000000040f869 in do_action (a=0x780a48, msg=0x78fe98) at
>>>> action.c:961
>>>> #5  0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>> msg=0x78fe98) at action.c:139
>>>> #6  0x000000000046f194 in eval_elem (e=0x780b18, msg=0x78fe98,
>>>> val=0x0) at route.c:1189
>>>> #7  0x00000000004708ed in eval_expr (e=0x120b1e0, msg=0x78fe98,
>>>> val=0x0) at route.c:1486
>>>> #8  0x000000000047089c in eval_expr (e=0x780b60, msg=0x78fe98,
>>>> val=0x0) at route.c:1502
>>>> #9  0x00000000004708c5 in eval_expr (e=0x780ba8, msg=0x78fe98,
>>>> val=0x0) at route.c:1507
>>>> #10 0x000000000040f91c in do_action (a=0x780f40, msg=0x78fe98) at
>>>> action.c:688
>>>> #11 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>> msg=0x78fe98) at action.c:139
>>>> #12 0x00000000004114d3 in do_action (a=0x781eb0, msg=0x78fe98) at
>>>> action.c:705
>>>> #13 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>> msg=0x78fe98) at action.c:139
>>>> #14 0x0000000000410e07 in do_action (a=0x77ee88, msg=0x78fe98) at
>>>> action.c:119
>>>> #15 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>> msg=0x78fe98) at action.c:139
>>>> #16 0x00000000004114d3 in do_action (a=0x77f028, msg=0x78fe98) at
>>>> action.c:705
>>>> #17 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>> msg=0x78fe98) at action.c:139
>>>> #18 0x00000000004125fe in run_top_route (a=0x777e78, msg=0x78fe98) at
>>>> action.c:119
>>>> #19 0x000000000045e898 in receive_msg (
>>>>   buf=0x749180 "REGISTER sip:xxxxx SIP/2.0\r\nCSeq: 2
>>>> REGISTER\r\nVia: SIP/2.0/UDP
>>>>
>>>>
>>>> x.x.x.x:5061;branch=z9hG4bK60f1b2dd-57fa-dd11-9401-0015c5404858;rport\r\nUser-Agent:
>>>> Ekiga/2.0.12\r\nAuthorization: Dige"..., len=749,
>>>> rcv_info=0x7fff7ba13530) at receive.c:165
>>>> #20 0x000000000049d3a6 in udp_rcv_loop () at udp_server.c:449
>>>> #21 0x00000000004291fb in main (argc=<value optimized out>,
>>>> argv=0x7fff7ba13718) at main.c:778
>>>> ------------
>>>>
>>>> Server runs fine after restart until MySQL is restarted again. In my
>>>> lab setup, OpenSIPS is getting killed triggered by registration
>>>> request every morning (probably due to MySQL being restarted daily
>>>> with logrotate).
>>>>
>>>> The server is the latest trunk on Debian lenny/AMD64.
>>>>
>>>> Thanks,
>>>> Om.
>>>>
>>>> _______________________________________________
>>>> Users mailing list
>>>> [hidden email]
>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: OpenSIPS libmysqlclient.so segfault after MySQL restart

Bogdan-Andrei Iancu
I see....

Let me try to reproduce this....Thanks for the update!

Regards,
bogdan

Om Bikram Thapa wrote:

> Hi Bogdan,
>
> Just to follow up on this issue, the crash is happening even without
> restarting MySQL server. It seems to show up every morning when
> registration (or maybe any db operation) is performed. I think the
> issue has to do with some kind of timeout related to OpenSIPS and
> libmysqlclient after a long duration of inactivity.
>
> Thanks,
> Om.
>
> On Tue, Feb 17, 2009 at 6:24 PM, Bogdan-Andrei Iancu
> <[hidden email]> wrote:
>  
>> Hi Om,
>>
>> Ok - got it (2608188) - thank you again for your report.
>>
>> Regards,
>> Bogdan
>>
>>
>> Om Bikram Thapa wrote:
>>    
>>> Hi Bogdan,
>>>
>>> I have submitted the bug on the tracker.
>>>
>>> Thanks,
>>> Om.
>>>
>>> On Tue, Feb 17, 2009 at 3:50 AM, Bogdan-Andrei Iancu
>>> <[hidden email]> wrote:
>>>
>>>      
>>>> Hi Om,
>>>>
>>>> Thanks for the report - I may have an idea what is the problem; I will
>>>> try
>>>> to reproduce the crash in the following days.
>>>>
>>>> Could you please open a bug report on the tracker - just not to forget
>>>> about
>>>> it :D (see http://www.opensips.org/index.php?n=Development.Tracker)
>>>>
>>>> Regards,
>>>> Bogdan
>>>>
>>>> Om Bikram Thapa wrote:
>>>>
>>>>        
>>>>> Hi,
>>>>>
>>>>> OpenSIPS is dying with "libmysqlclient.so Segmentation Fault" after
>>>>> MySQL server restart. The log shows
>>>>> "opensips[16769]: segfault at 4c8 ip 7f36728bf283 sp 7fff7ba11a90
>>>>> error 4 in libmysqlclient.so.15.0.0[7f3672862000+1bf000]"
>>>>>
>>>>> and gdb bt shows:
>>>>>
>>>>> -------------
>>>>> #0  0x00007f36728bf283 in mysql_stmt_result_metadata () from
>>>>> /usr/lib/libmysqlclient.so.15
>>>>> #1  0x00007f3672c707b7 in db_mysql_do_prepared_query (conn=0x78b278,
>>>>> query=<value optimized out>, v=0x7fff7ba11d20, n=1, uv=0x0, un=0) at
>>>>> dbase.c:363
>>>>> #2  0x00007f3672c72e1f in db_mysql_query (_h=0x78b278, _k=<value
>>>>> optimized out>, _op=<value optimized out>, _v=0x7fff7ba11d20,
>>>>> _c=<value optimized out>,
>>>>>   _n=1, _nc=2, _o=0x0, _r=0x7fff7ba11dc0) at dbase.c:591
>>>>> #3  0x00007f366f3f0398 in authorize (_m=0x78fe98, _realm=<value
>>>>> optimized out>, _table=<value optimized out>, _hftype=<value optimized
>>>>> out>)
>>>>>   at authorize.c:107
>>>>> #4  0x000000000040f869 in do_action (a=0x780a48, msg=0x78fe98) at
>>>>> action.c:961
>>>>> #5  0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>>> msg=0x78fe98) at action.c:139
>>>>> #6  0x000000000046f194 in eval_elem (e=0x780b18, msg=0x78fe98,
>>>>> val=0x0) at route.c:1189
>>>>> #7  0x00000000004708ed in eval_expr (e=0x120b1e0, msg=0x78fe98,
>>>>> val=0x0) at route.c:1486
>>>>> #8  0x000000000047089c in eval_expr (e=0x780b60, msg=0x78fe98,
>>>>> val=0x0) at route.c:1502
>>>>> #9  0x00000000004708c5 in eval_expr (e=0x780ba8, msg=0x78fe98,
>>>>> val=0x0) at route.c:1507
>>>>> #10 0x000000000040f91c in do_action (a=0x780f40, msg=0x78fe98) at
>>>>> action.c:688
>>>>> #11 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>>> msg=0x78fe98) at action.c:139
>>>>> #12 0x00000000004114d3 in do_action (a=0x781eb0, msg=0x78fe98) at
>>>>> action.c:705
>>>>> #13 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>>> msg=0x78fe98) at action.c:139
>>>>> #14 0x0000000000410e07 in do_action (a=0x77ee88, msg=0x78fe98) at
>>>>> action.c:119
>>>>> #15 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>>> msg=0x78fe98) at action.c:139
>>>>> #16 0x00000000004114d3 in do_action (a=0x77f028, msg=0x78fe98) at
>>>>> action.c:705
>>>>> #17 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>>> msg=0x78fe98) at action.c:139
>>>>> #18 0x00000000004125fe in run_top_route (a=0x777e78, msg=0x78fe98) at
>>>>> action.c:119
>>>>> #19 0x000000000045e898 in receive_msg (
>>>>>   buf=0x749180 "REGISTER sip:xxxxx SIP/2.0\r\nCSeq: 2
>>>>> REGISTER\r\nVia: SIP/2.0/UDP
>>>>>
>>>>>
>>>>> x.x.x.x:5061;branch=z9hG4bK60f1b2dd-57fa-dd11-9401-0015c5404858;rport\r\nUser-Agent:
>>>>> Ekiga/2.0.12\r\nAuthorization: Dige"..., len=749,
>>>>> rcv_info=0x7fff7ba13530) at receive.c:165
>>>>> #20 0x000000000049d3a6 in udp_rcv_loop () at udp_server.c:449
>>>>> #21 0x00000000004291fb in main (argc=<value optimized out>,
>>>>> argv=0x7fff7ba13718) at main.c:778
>>>>> ------------
>>>>>
>>>>> Server runs fine after restart until MySQL is restarted again. In my
>>>>> lab setup, OpenSIPS is getting killed triggered by registration
>>>>> request every morning (probably due to MySQL being restarted daily
>>>>> with logrotate).
>>>>>
>>>>> The server is the latest trunk on Debian lenny/AMD64.
>>>>>
>>>>> Thanks,
>>>>> Om.
>>>>>
>>>>> _______________________________________________
>>>>> Users mailing list
>>>>> [hidden email]
>>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>>
>>>>>
>>>>>
>>>>>          
>>>>        
>>>      
>>    
>
>  


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: OpenSIPS libmysqlclient.so segfault after MySQL restart

Bogdan-Andrei Iancu
In reply to this post by Om Bikram Thapa
Hi Om,

There is a fix available on SVN - please update and see if it works now.

Thanks and regards,
Bogdan

Om Bikram Thapa wrote:

> Hi Bogdan,
>
> I have submitted the bug on the tracker.
>
> Thanks,
> Om.
>
> On Tue, Feb 17, 2009 at 3:50 AM, Bogdan-Andrei Iancu
> <[hidden email]> wrote:
>  
>> Hi Om,
>>
>> Thanks for the report - I may have an idea what is the problem; I will try
>> to reproduce the crash in the following days.
>>
>> Could you please open a bug report on the tracker - just not to forget about
>> it :D (see http://www.opensips.org/index.php?n=Development.Tracker)
>>
>> Regards,
>> Bogdan
>>
>> Om Bikram Thapa wrote:
>>    
>>> Hi,
>>>
>>> OpenSIPS is dying with "libmysqlclient.so Segmentation Fault" after
>>> MySQL server restart. The log shows
>>> "opensips[16769]: segfault at 4c8 ip 7f36728bf283 sp 7fff7ba11a90
>>> error 4 in libmysqlclient.so.15.0.0[7f3672862000+1bf000]"
>>>
>>> and gdb bt shows:
>>>
>>> -------------
>>> #0  0x00007f36728bf283 in mysql_stmt_result_metadata () from
>>> /usr/lib/libmysqlclient.so.15
>>> #1  0x00007f3672c707b7 in db_mysql_do_prepared_query (conn=0x78b278,
>>> query=<value optimized out>, v=0x7fff7ba11d20, n=1, uv=0x0, un=0) at
>>> dbase.c:363
>>> #2  0x00007f3672c72e1f in db_mysql_query (_h=0x78b278, _k=<value
>>> optimized out>, _op=<value optimized out>, _v=0x7fff7ba11d20,
>>> _c=<value optimized out>,
>>>    _n=1, _nc=2, _o=0x0, _r=0x7fff7ba11dc0) at dbase.c:591
>>> #3  0x00007f366f3f0398 in authorize (_m=0x78fe98, _realm=<value
>>> optimized out>, _table=<value optimized out>, _hftype=<value optimized
>>> out>)
>>>    at authorize.c:107
>>> #4  0x000000000040f869 in do_action (a=0x780a48, msg=0x78fe98) at
>>> action.c:961
>>> #5  0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>> msg=0x78fe98) at action.c:139
>>> #6  0x000000000046f194 in eval_elem (e=0x780b18, msg=0x78fe98,
>>> val=0x0) at route.c:1189
>>> #7  0x00000000004708ed in eval_expr (e=0x120b1e0, msg=0x78fe98,
>>> val=0x0) at route.c:1486
>>> #8  0x000000000047089c in eval_expr (e=0x780b60, msg=0x78fe98,
>>> val=0x0) at route.c:1502
>>> #9  0x00000000004708c5 in eval_expr (e=0x780ba8, msg=0x78fe98,
>>> val=0x0) at route.c:1507
>>> #10 0x000000000040f91c in do_action (a=0x780f40, msg=0x78fe98) at
>>> action.c:688
>>> #11 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>> msg=0x78fe98) at action.c:139
>>> #12 0x00000000004114d3 in do_action (a=0x781eb0, msg=0x78fe98) at
>>> action.c:705
>>> #13 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>> msg=0x78fe98) at action.c:139
>>> #14 0x0000000000410e07 in do_action (a=0x77ee88, msg=0x78fe98) at
>>> action.c:119
>>> #15 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>> msg=0x78fe98) at action.c:139
>>> #16 0x00000000004114d3 in do_action (a=0x77f028, msg=0x78fe98) at
>>> action.c:705
>>> #17 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>> msg=0x78fe98) at action.c:139
>>> #18 0x00000000004125fe in run_top_route (a=0x777e78, msg=0x78fe98) at
>>> action.c:119
>>> #19 0x000000000045e898 in receive_msg (
>>>    buf=0x749180 "REGISTER sip:xxxxx SIP/2.0\r\nCSeq: 2
>>> REGISTER\r\nVia: SIP/2.0/UDP
>>>
>>> x.x.x.x:5061;branch=z9hG4bK60f1b2dd-57fa-dd11-9401-0015c5404858;rport\r\nUser-Agent:
>>> Ekiga/2.0.12\r\nAuthorization: Dige"..., len=749,
>>> rcv_info=0x7fff7ba13530) at receive.c:165
>>> #20 0x000000000049d3a6 in udp_rcv_loop () at udp_server.c:449
>>> #21 0x00000000004291fb in main (argc=<value optimized out>,
>>> argv=0x7fff7ba13718) at main.c:778
>>> ------------
>>>
>>> Server runs fine after restart until MySQL is restarted again. In my
>>> lab setup, OpenSIPS is getting killed triggered by registration
>>> request every morning (probably due to MySQL being restarted daily
>>> with logrotate).
>>>
>>> The server is the latest trunk on Debian lenny/AMD64.
>>>
>>> Thanks,
>>> Om.
>>>
>>> _______________________________________________
>>> Users mailing list
>>> [hidden email]
>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>
>>>
>>>      
>>    
>
>  


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: OpenSIPS libmysqlclient.so segfault after MySQL restart

Om Bikram Thapa
Thanks Bogdan. I will update and let you know.

Regards,
Om.

On Thu, Feb 19, 2009 at 2:48 AM, Bogdan-Andrei Iancu
<[hidden email]> wrote:

> Hi Om,
>
> There is a fix available on SVN - please update and see if it works now.
>
> Thanks and regards,
> Bogdan
>
> Om Bikram Thapa wrote:
>>
>> Hi Bogdan,
>>
>> I have submitted the bug on the tracker.
>>
>> Thanks,
>> Om.
>>
>> On Tue, Feb 17, 2009 at 3:50 AM, Bogdan-Andrei Iancu
>> <[hidden email]> wrote:
>>
>>>
>>> Hi Om,
>>>
>>> Thanks for the report - I may have an idea what is the problem; I will
>>> try
>>> to reproduce the crash in the following days.
>>>
>>> Could you please open a bug report on the tracker - just not to forget
>>> about
>>> it :D (see http://www.opensips.org/index.php?n=Development.Tracker)
>>>
>>> Regards,
>>> Bogdan
>>>
>>> Om Bikram Thapa wrote:
>>>
>>>>
>>>> Hi,
>>>>
>>>> OpenSIPS is dying with "libmysqlclient.so Segmentation Fault" after
>>>> MySQL server restart. The log shows
>>>> "opensips[16769]: segfault at 4c8 ip 7f36728bf283 sp 7fff7ba11a90
>>>> error 4 in libmysqlclient.so.15.0.0[7f3672862000+1bf000]"
>>>>
>>>> and gdb bt shows:
>>>>
>>>> -------------
>>>> #0  0x00007f36728bf283 in mysql_stmt_result_metadata () from
>>>> /usr/lib/libmysqlclient.so.15
>>>> #1  0x00007f3672c707b7 in db_mysql_do_prepared_query (conn=0x78b278,
>>>> query=<value optimized out>, v=0x7fff7ba11d20, n=1, uv=0x0, un=0) at
>>>> dbase.c:363
>>>> #2  0x00007f3672c72e1f in db_mysql_query (_h=0x78b278, _k=<value
>>>> optimized out>, _op=<value optimized out>, _v=0x7fff7ba11d20,
>>>> _c=<value optimized out>,
>>>>   _n=1, _nc=2, _o=0x0, _r=0x7fff7ba11dc0) at dbase.c:591
>>>> #3  0x00007f366f3f0398 in authorize (_m=0x78fe98, _realm=<value
>>>> optimized out>, _table=<value optimized out>, _hftype=<value optimized
>>>> out>)
>>>>   at authorize.c:107
>>>> #4  0x000000000040f869 in do_action (a=0x780a48, msg=0x78fe98) at
>>>> action.c:961
>>>> #5  0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>> msg=0x78fe98) at action.c:139
>>>> #6  0x000000000046f194 in eval_elem (e=0x780b18, msg=0x78fe98,
>>>> val=0x0) at route.c:1189
>>>> #7  0x00000000004708ed in eval_expr (e=0x120b1e0, msg=0x78fe98,
>>>> val=0x0) at route.c:1486
>>>> #8  0x000000000047089c in eval_expr (e=0x780b60, msg=0x78fe98,
>>>> val=0x0) at route.c:1502
>>>> #9  0x00000000004708c5 in eval_expr (e=0x780ba8, msg=0x78fe98,
>>>> val=0x0) at route.c:1507
>>>> #10 0x000000000040f91c in do_action (a=0x780f40, msg=0x78fe98) at
>>>> action.c:688
>>>> #11 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>> msg=0x78fe98) at action.c:139
>>>> #12 0x00000000004114d3 in do_action (a=0x781eb0, msg=0x78fe98) at
>>>> action.c:705
>>>> #13 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>> msg=0x78fe98) at action.c:139
>>>> #14 0x0000000000410e07 in do_action (a=0x77ee88, msg=0x78fe98) at
>>>> action.c:119
>>>> #15 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>> msg=0x78fe98) at action.c:139
>>>> #16 0x00000000004114d3 in do_action (a=0x77f028, msg=0x78fe98) at
>>>> action.c:705
>>>> #17 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>> msg=0x78fe98) at action.c:139
>>>> #18 0x00000000004125fe in run_top_route (a=0x777e78, msg=0x78fe98) at
>>>> action.c:119
>>>> #19 0x000000000045e898 in receive_msg (
>>>>   buf=0x749180 "REGISTER sip:xxxxx SIP/2.0\r\nCSeq: 2
>>>> REGISTER\r\nVia: SIP/2.0/UDP
>>>>
>>>>
>>>> x.x.x.x:5061;branch=z9hG4bK60f1b2dd-57fa-dd11-9401-0015c5404858;rport\r\nUser-Agent:
>>>> Ekiga/2.0.12\r\nAuthorization: Dige"..., len=749,
>>>> rcv_info=0x7fff7ba13530) at receive.c:165
>>>> #20 0x000000000049d3a6 in udp_rcv_loop () at udp_server.c:449
>>>> #21 0x00000000004291fb in main (argc=<value optimized out>,
>>>> argv=0x7fff7ba13718) at main.c:778
>>>> ------------
>>>>
>>>> Server runs fine after restart until MySQL is restarted again. In my
>>>> lab setup, OpenSIPS is getting killed triggered by registration
>>>> request every morning (probably due to MySQL being restarted daily
>>>> with logrotate).
>>>>
>>>> The server is the latest trunk on Debian lenny/AMD64.
>>>>
>>>> Thanks,
>>>> Om.
>>>>
>>>> _______________________________________________
>>>> Users mailing list
>>>> [hidden email]
>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: OpenSIPS libmysqlclient.so segfault after MySQL restart

Om Bikram Thapa
Hi Bogdan,

Just updated to latest svn, but I still see crashes after MySQL
restart. The syslog shows following 3 lines after MySQL restart and
registration attempt:

--
Feb 19 12:17:43 proxy /usr/local/sbin/opensips[2333]:
ERROR:db_mysql:db_mysql_do_prepared_query: mysql_stmt_execute()
failed: (1) Unknown prepared statement handler (1) given to
mysql_stmt_execute
Feb 19 12:17:43 proxy /usr/local/sbin/opensips[2333]:
ERROR:auth_db:get_ha1: failed to query database
Feb 19 12:18:13 proxy kernel: [  105.116919] opensips[2333]: segfault
at 4c8 ip 7f167c109283 sp 7fff8525b270 error 4 in
libmysqlclient.so.15.0.0[7f167c0ac000+1bf000]
--

At first attempt, the registration fails with "SIP/2.0 500 Server
Internal Error" returned by the server, and subsequent registration
attempt crashes the server.

gdb backtrace gives following details:

---
#0  0x00007f0119426283 in mysql_stmt_result_metadata () from
/usr/lib/libmysqlclient.so.15
(gdb) bt
#0  0x00007f0119426283 in mysql_stmt_result_metadata () from
/usr/lib/libmysqlclient.so.15
#1  0x00007f01197d8f82 in db_mysql_do_prepared_query (conn=0x78b278,
query=<value optimized out>, v=0x7fff22576880, n=<value optimized
out>, uv=0x0, un=0)
    at dbase.c:368
#2  0x00007f01197daccd in db_mysql_query (_h=0x78b278, _k=<value
optimized out>, _op=<value optimized out>, _v=0x7fff22576880,
_c=<value optimized out>,
    _n=1, _nc=2, _o=0x0, _r=0x7fff22576920) at dbase.c:603
#3  0x00007f0115f56398 in authorize (_m=0x78fe98, _realm=<value
optimized out>, _table=<value optimized out>, _hftype=<value optimized
out>)
    at authorize.c:107
#4  0x000000000040f869 in do_action (a=0x780a48, msg=0x78fe98) at action.c:961
#5  0x000000000040e7e5 in run_action_list (a=<value optimized out>,
msg=0x78fe98) at action.c:139
#6  0x000000000046f194 in eval_elem (e=0x780b18, msg=0x78fe98,
val=0x0) at route.c:1189
#7  0x00000000004708ed in eval_expr (e=0x27648e0, msg=0x78fe98,
val=0x0) at route.c:1486
#8  0x000000000047089c in eval_expr (e=0x780b60, msg=0x78fe98,
val=0x0) at route.c:1502
#9  0x00000000004708c5 in eval_expr (e=0x780ba8, msg=0x78fe98,
val=0x0) at route.c:1507
#10 0x000000000040f91c in do_action (a=0x780f40, msg=0x78fe98) at action.c:688
#11 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
msg=0x78fe98) at action.c:139
#12 0x00000000004114d3 in do_action (a=0x781eb0, msg=0x78fe98) at action.c:705
#13 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
msg=0x78fe98) at action.c:139
#14 0x0000000000410e07 in do_action (a=0x77ee88, msg=0x78fe98) at action.c:119
#15 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
msg=0x78fe98) at action.c:139
#16 0x00000000004114d3 in do_action (a=0x77f028, msg=0x78fe98) at action.c:705
#17 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
msg=0x78fe98) at action.c:139
#18 0x00000000004125fe in run_top_route (a=0x777e78, msg=0x78fe98) at
action.c:119
#19 0x000000000045e898 in receive_msg (
    buf=0x749180 "REGISTER sip:x.x.x.x SIP/2.0\r\nCSeq: 58
REGISTER\r\nVia: SIP/2.0/UDP
x.x.x.x:5068;branch=z9hG4bK5ca4685b-befc-dd11-911f-0015c5404858;rport\r\nUser-Agent:
Ekiga/2.0.12\r\nAuthorization: Dig"..., len=750,
rcv_info=0x7fff22578090) at receive.c:165
#20 0x000000000049d3a6 in udp_rcv_loop () at udp_server.c:449
#21 0x00000000004291fb in main (argc=<value optimized out>,
argv=0x7fff22578278) at main.c:778
---

I have tried several times and each time I get the same sequence of results.

Thanks,
Om.

On Thu, Feb 19, 2009 at 11:22 AM, Om Bikram Thapa <[hidden email]> wrote:

> Thanks Bogdan. I will update and let you know.
>
> Regards,
> Om.
>
> On Thu, Feb 19, 2009 at 2:48 AM, Bogdan-Andrei Iancu
> <[hidden email]> wrote:
>> Hi Om,
>>
>> There is a fix available on SVN - please update and see if it works now.
>>
>> Thanks and regards,
>> Bogdan
>>
>> Om Bikram Thapa wrote:
>>>
>>> Hi Bogdan,
>>>
>>> I have submitted the bug on the tracker.
>>>
>>> Thanks,
>>> Om.
>>>
>>> On Tue, Feb 17, 2009 at 3:50 AM, Bogdan-Andrei Iancu
>>> <[hidden email]> wrote:
>>>
>>>>
>>>> Hi Om,
>>>>
>>>> Thanks for the report - I may have an idea what is the problem; I will
>>>> try
>>>> to reproduce the crash in the following days.
>>>>
>>>> Could you please open a bug report on the tracker - just not to forget
>>>> about
>>>> it :D (see http://www.opensips.org/index.php?n=Development.Tracker)
>>>>
>>>> Regards,
>>>> Bogdan
>>>>
>>>> Om Bikram Thapa wrote:
>>>>
>>>>>
>>>>> Hi,
>>>>>
>>>>> OpenSIPS is dying with "libmysqlclient.so Segmentation Fault" after
>>>>> MySQL server restart. The log shows
>>>>> "opensips[16769]: segfault at 4c8 ip 7f36728bf283 sp 7fff7ba11a90
>>>>> error 4 in libmysqlclient.so.15.0.0[7f3672862000+1bf000]"
>>>>>
>>>>> and gdb bt shows:
>>>>>
>>>>> -------------
>>>>> #0  0x00007f36728bf283 in mysql_stmt_result_metadata () from
>>>>> /usr/lib/libmysqlclient.so.15
>>>>> #1  0x00007f3672c707b7 in db_mysql_do_prepared_query (conn=0x78b278,
>>>>> query=<value optimized out>, v=0x7fff7ba11d20, n=1, uv=0x0, un=0) at
>>>>> dbase.c:363
>>>>> #2  0x00007f3672c72e1f in db_mysql_query (_h=0x78b278, _k=<value
>>>>> optimized out>, _op=<value optimized out>, _v=0x7fff7ba11d20,
>>>>> _c=<value optimized out>,
>>>>>   _n=1, _nc=2, _o=0x0, _r=0x7fff7ba11dc0) at dbase.c:591
>>>>> #3  0x00007f366f3f0398 in authorize (_m=0x78fe98, _realm=<value
>>>>> optimized out>, _table=<value optimized out>, _hftype=<value optimized
>>>>> out>)
>>>>>   at authorize.c:107
>>>>> #4  0x000000000040f869 in do_action (a=0x780a48, msg=0x78fe98) at
>>>>> action.c:961
>>>>> #5  0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>>> msg=0x78fe98) at action.c:139
>>>>> #6  0x000000000046f194 in eval_elem (e=0x780b18, msg=0x78fe98,
>>>>> val=0x0) at route.c:1189
>>>>> #7  0x00000000004708ed in eval_expr (e=0x120b1e0, msg=0x78fe98,
>>>>> val=0x0) at route.c:1486
>>>>> #8  0x000000000047089c in eval_expr (e=0x780b60, msg=0x78fe98,
>>>>> val=0x0) at route.c:1502
>>>>> #9  0x00000000004708c5 in eval_expr (e=0x780ba8, msg=0x78fe98,
>>>>> val=0x0) at route.c:1507
>>>>> #10 0x000000000040f91c in do_action (a=0x780f40, msg=0x78fe98) at
>>>>> action.c:688
>>>>> #11 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>>> msg=0x78fe98) at action.c:139
>>>>> #12 0x00000000004114d3 in do_action (a=0x781eb0, msg=0x78fe98) at
>>>>> action.c:705
>>>>> #13 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>>> msg=0x78fe98) at action.c:139
>>>>> #14 0x0000000000410e07 in do_action (a=0x77ee88, msg=0x78fe98) at
>>>>> action.c:119
>>>>> #15 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>>> msg=0x78fe98) at action.c:139
>>>>> #16 0x00000000004114d3 in do_action (a=0x77f028, msg=0x78fe98) at
>>>>> action.c:705
>>>>> #17 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>>> msg=0x78fe98) at action.c:139
>>>>> #18 0x00000000004125fe in run_top_route (a=0x777e78, msg=0x78fe98) at
>>>>> action.c:119
>>>>> #19 0x000000000045e898 in receive_msg (
>>>>>   buf=0x749180 "REGISTER sip:xxxxx SIP/2.0\r\nCSeq: 2
>>>>> REGISTER\r\nVia: SIP/2.0/UDP
>>>>>
>>>>>
>>>>> x.x.x.x:5061;branch=z9hG4bK60f1b2dd-57fa-dd11-9401-0015c5404858;rport\r\nUser-Agent:
>>>>> Ekiga/2.0.12\r\nAuthorization: Dige"..., len=749,
>>>>> rcv_info=0x7fff7ba13530) at receive.c:165
>>>>> #20 0x000000000049d3a6 in udp_rcv_loop () at udp_server.c:449
>>>>> #21 0x00000000004291fb in main (argc=<value optimized out>,
>>>>> argv=0x7fff7ba13718) at main.c:778
>>>>> ------------
>>>>>
>>>>> Server runs fine after restart until MySQL is restarted again. In my
>>>>> lab setup, OpenSIPS is getting killed triggered by registration
>>>>> request every morning (probably due to MySQL being restarted daily
>>>>> with logrotate).
>>>>>
>>>>> The server is the latest trunk on Debian lenny/AMD64.
>>>>>
>>>>> Thanks,
>>>>> Om.
>>>>>
>>>>> _______________________________________________
>>>>> Users mailing list
>>>>> [hidden email]
>>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: OpenSIPS libmysqlclient.so segfault after MySQL restart

Bogdan-Andrei Iancu
Hi Om,

I see....in my tests, I get a different error (Lost connection to Mysql
server)...let me see....

Regards,
Bogdan

Om Bikram Thapa wrote:

> Hi Bogdan,
>
> Just updated to latest svn, but I still see crashes after MySQL
> restart. The syslog shows following 3 lines after MySQL restart and
> registration attempt:
>
> --
> Feb 19 12:17:43 proxy /usr/local/sbin/opensips[2333]:
> ERROR:db_mysql:db_mysql_do_prepared_query: mysql_stmt_execute()
> failed: (1) Unknown prepared statement handler (1) given to
> mysql_stmt_execute
> Feb 19 12:17:43 proxy /usr/local/sbin/opensips[2333]:
> ERROR:auth_db:get_ha1: failed to query database
> Feb 19 12:18:13 proxy kernel: [  105.116919] opensips[2333]: segfault
> at 4c8 ip 7f167c109283 sp 7fff8525b270 error 4 in
> libmysqlclient.so.15.0.0[7f167c0ac000+1bf000]
> --
>
> At first attempt, the registration fails with "SIP/2.0 500 Server
> Internal Error" returned by the server, and subsequent registration
> attempt crashes the server.
>
> gdb backtrace gives following details:
>
> ---
> #0  0x00007f0119426283 in mysql_stmt_result_metadata () from
> /usr/lib/libmysqlclient.so.15
> (gdb) bt
> #0  0x00007f0119426283 in mysql_stmt_result_metadata () from
> /usr/lib/libmysqlclient.so.15
> #1  0x00007f01197d8f82 in db_mysql_do_prepared_query (conn=0x78b278,
> query=<value optimized out>, v=0x7fff22576880, n=<value optimized
> out>, uv=0x0, un=0)
>     at dbase.c:368
> #2  0x00007f01197daccd in db_mysql_query (_h=0x78b278, _k=<value
> optimized out>, _op=<value optimized out>, _v=0x7fff22576880,
> _c=<value optimized out>,
>     _n=1, _nc=2, _o=0x0, _r=0x7fff22576920) at dbase.c:603
> #3  0x00007f0115f56398 in authorize (_m=0x78fe98, _realm=<value
> optimized out>, _table=<value optimized out>, _hftype=<value optimized
> out>)
>     at authorize.c:107
> #4  0x000000000040f869 in do_action (a=0x780a48, msg=0x78fe98) at action.c:961
> #5  0x000000000040e7e5 in run_action_list (a=<value optimized out>,
> msg=0x78fe98) at action.c:139
> #6  0x000000000046f194 in eval_elem (e=0x780b18, msg=0x78fe98,
> val=0x0) at route.c:1189
> #7  0x00000000004708ed in eval_expr (e=0x27648e0, msg=0x78fe98,
> val=0x0) at route.c:1486
> #8  0x000000000047089c in eval_expr (e=0x780b60, msg=0x78fe98,
> val=0x0) at route.c:1502
> #9  0x00000000004708c5 in eval_expr (e=0x780ba8, msg=0x78fe98,
> val=0x0) at route.c:1507
> #10 0x000000000040f91c in do_action (a=0x780f40, msg=0x78fe98) at action.c:688
> #11 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
> msg=0x78fe98) at action.c:139
> #12 0x00000000004114d3 in do_action (a=0x781eb0, msg=0x78fe98) at action.c:705
> #13 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
> msg=0x78fe98) at action.c:139
> #14 0x0000000000410e07 in do_action (a=0x77ee88, msg=0x78fe98) at action.c:119
> #15 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
> msg=0x78fe98) at action.c:139
> #16 0x00000000004114d3 in do_action (a=0x77f028, msg=0x78fe98) at action.c:705
> #17 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
> msg=0x78fe98) at action.c:139
> #18 0x00000000004125fe in run_top_route (a=0x777e78, msg=0x78fe98) at
> action.c:119
> #19 0x000000000045e898 in receive_msg (
>     buf=0x749180 "REGISTER sip:x.x.x.x SIP/2.0\r\nCSeq: 58
> REGISTER\r\nVia: SIP/2.0/UDP
> x.x.x.x:5068;branch=z9hG4bK5ca4685b-befc-dd11-911f-0015c5404858;rport\r\nUser-Agent:
> Ekiga/2.0.12\r\nAuthorization: Dig"..., len=750,
> rcv_info=0x7fff22578090) at receive.c:165
> #20 0x000000000049d3a6 in udp_rcv_loop () at udp_server.c:449
> #21 0x00000000004291fb in main (argc=<value optimized out>,
> argv=0x7fff22578278) at main.c:778
> ---
>
> I have tried several times and each time I get the same sequence of results.
>
> Thanks,
> Om.
>
> On Thu, Feb 19, 2009 at 11:22 AM, Om Bikram Thapa <[hidden email]> wrote:
>  
>> Thanks Bogdan. I will update and let you know.
>>
>> Regards,
>> Om.
>>
>> On Thu, Feb 19, 2009 at 2:48 AM, Bogdan-Andrei Iancu
>> <[hidden email]> wrote:
>>    
>>> Hi Om,
>>>
>>> There is a fix available on SVN - please update and see if it works now.
>>>
>>> Thanks and regards,
>>> Bogdan
>>>
>>> Om Bikram Thapa wrote:
>>>      
>>>> Hi Bogdan,
>>>>
>>>> I have submitted the bug on the tracker.
>>>>
>>>> Thanks,
>>>> Om.
>>>>
>>>> On Tue, Feb 17, 2009 at 3:50 AM, Bogdan-Andrei Iancu
>>>> <[hidden email]> wrote:
>>>>
>>>>        
>>>>> Hi Om,
>>>>>
>>>>> Thanks for the report - I may have an idea what is the problem; I will
>>>>> try
>>>>> to reproduce the crash in the following days.
>>>>>
>>>>> Could you please open a bug report on the tracker - just not to forget
>>>>> about
>>>>> it :D (see http://www.opensips.org/index.php?n=Development.Tracker)
>>>>>
>>>>> Regards,
>>>>> Bogdan
>>>>>
>>>>> Om Bikram Thapa wrote:
>>>>>
>>>>>          
>>>>>> Hi,
>>>>>>
>>>>>> OpenSIPS is dying with "libmysqlclient.so Segmentation Fault" after
>>>>>> MySQL server restart. The log shows
>>>>>> "opensips[16769]: segfault at 4c8 ip 7f36728bf283 sp 7fff7ba11a90
>>>>>> error 4 in libmysqlclient.so.15.0.0[7f3672862000+1bf000]"
>>>>>>
>>>>>> and gdb bt shows:
>>>>>>
>>>>>> -------------
>>>>>> #0  0x00007f36728bf283 in mysql_stmt_result_metadata () from
>>>>>> /usr/lib/libmysqlclient.so.15
>>>>>> #1  0x00007f3672c707b7 in db_mysql_do_prepared_query (conn=0x78b278,
>>>>>> query=<value optimized out>, v=0x7fff7ba11d20, n=1, uv=0x0, un=0) at
>>>>>> dbase.c:363
>>>>>> #2  0x00007f3672c72e1f in db_mysql_query (_h=0x78b278, _k=<value
>>>>>> optimized out>, _op=<value optimized out>, _v=0x7fff7ba11d20,
>>>>>> _c=<value optimized out>,
>>>>>>   _n=1, _nc=2, _o=0x0, _r=0x7fff7ba11dc0) at dbase.c:591
>>>>>> #3  0x00007f366f3f0398 in authorize (_m=0x78fe98, _realm=<value
>>>>>> optimized out>, _table=<value optimized out>, _hftype=<value optimized
>>>>>> out>)
>>>>>>   at authorize.c:107
>>>>>> #4  0x000000000040f869 in do_action (a=0x780a48, msg=0x78fe98) at
>>>>>> action.c:961
>>>>>> #5  0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>>>> msg=0x78fe98) at action.c:139
>>>>>> #6  0x000000000046f194 in eval_elem (e=0x780b18, msg=0x78fe98,
>>>>>> val=0x0) at route.c:1189
>>>>>> #7  0x00000000004708ed in eval_expr (e=0x120b1e0, msg=0x78fe98,
>>>>>> val=0x0) at route.c:1486
>>>>>> #8  0x000000000047089c in eval_expr (e=0x780b60, msg=0x78fe98,
>>>>>> val=0x0) at route.c:1502
>>>>>> #9  0x00000000004708c5 in eval_expr (e=0x780ba8, msg=0x78fe98,
>>>>>> val=0x0) at route.c:1507
>>>>>> #10 0x000000000040f91c in do_action (a=0x780f40, msg=0x78fe98) at
>>>>>> action.c:688
>>>>>> #11 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>>>> msg=0x78fe98) at action.c:139
>>>>>> #12 0x00000000004114d3 in do_action (a=0x781eb0, msg=0x78fe98) at
>>>>>> action.c:705
>>>>>> #13 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>>>> msg=0x78fe98) at action.c:139
>>>>>> #14 0x0000000000410e07 in do_action (a=0x77ee88, msg=0x78fe98) at
>>>>>> action.c:119
>>>>>> #15 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>>>> msg=0x78fe98) at action.c:139
>>>>>> #16 0x00000000004114d3 in do_action (a=0x77f028, msg=0x78fe98) at
>>>>>> action.c:705
>>>>>> #17 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>>>> msg=0x78fe98) at action.c:139
>>>>>> #18 0x00000000004125fe in run_top_route (a=0x777e78, msg=0x78fe98) at
>>>>>> action.c:119
>>>>>> #19 0x000000000045e898 in receive_msg (
>>>>>>   buf=0x749180 "REGISTER sip:xxxxx SIP/2.0\r\nCSeq: 2
>>>>>> REGISTER\r\nVia: SIP/2.0/UDP
>>>>>>
>>>>>>
>>>>>> x.x.x.x:5061;branch=z9hG4bK60f1b2dd-57fa-dd11-9401-0015c5404858;rport\r\nUser-Agent:
>>>>>> Ekiga/2.0.12\r\nAuthorization: Dige"..., len=749,
>>>>>> rcv_info=0x7fff7ba13530) at receive.c:165
>>>>>> #20 0x000000000049d3a6 in udp_rcv_loop () at udp_server.c:449
>>>>>> #21 0x00000000004291fb in main (argc=<value optimized out>,
>>>>>> argv=0x7fff7ba13718) at main.c:778
>>>>>> ------------
>>>>>>
>>>>>> Server runs fine after restart until MySQL is restarted again. In my
>>>>>> lab setup, OpenSIPS is getting killed triggered by registration
>>>>>> request every morning (probably due to MySQL being restarted daily
>>>>>> with logrotate).
>>>>>>
>>>>>> The server is the latest trunk on Debian lenny/AMD64.
>>>>>>
>>>>>> Thanks,
>>>>>> Om.
>>>>>>
>>>>>> _______________________________________________
>>>>>> Users mailing list
>>>>>> [hidden email]
>>>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>>>
>>>>>>
>>>>>>
>>>>>>            
>>>>>          
>>>>        
>>>      
>
>  


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: OpenSIPS libmysqlclient.so segfault after MySQL restart

Bogdan-Andrei Iancu
In reply to this post by Om Bikram Thapa
Hi Om,

finally, after some struggle with the libmysqlclient, I hope the issue
is solve. Please give it another try and let me know the result.

Thanks and regards,
Bogdan

Om Bikram Thapa wrote:

> Hi Bogdan,
>
> Just updated to latest svn, but I still see crashes after MySQL
> restart. The syslog shows following 3 lines after MySQL restart and
> registration attempt:
>
> --
> Feb 19 12:17:43 proxy /usr/local/sbin/opensips[2333]:
> ERROR:db_mysql:db_mysql_do_prepared_query: mysql_stmt_execute()
> failed: (1) Unknown prepared statement handler (1) given to
> mysql_stmt_execute
> Feb 19 12:17:43 proxy /usr/local/sbin/opensips[2333]:
> ERROR:auth_db:get_ha1: failed to query database
> Feb 19 12:18:13 proxy kernel: [  105.116919] opensips[2333]: segfault
> at 4c8 ip 7f167c109283 sp 7fff8525b270 error 4 in
> libmysqlclient.so.15.0.0[7f167c0ac000+1bf000]
> --
>
> At first attempt, the registration fails with "SIP/2.0 500 Server
> Internal Error" returned by the server, and subsequent registration
> attempt crashes the server.
>
> gdb backtrace gives following details:
>
> ---
> #0  0x00007f0119426283 in mysql_stmt_result_metadata () from
> /usr/lib/libmysqlclient.so.15
> (gdb) bt
> #0  0x00007f0119426283 in mysql_stmt_result_metadata () from
> /usr/lib/libmysqlclient.so.15
> #1  0x00007f01197d8f82 in db_mysql_do_prepared_query (conn=0x78b278,
> query=<value optimized out>, v=0x7fff22576880, n=<value optimized
> out>, uv=0x0, un=0)
>     at dbase.c:368
> #2  0x00007f01197daccd in db_mysql_query (_h=0x78b278, _k=<value
> optimized out>, _op=<value optimized out>, _v=0x7fff22576880,
> _c=<value optimized out>,
>     _n=1, _nc=2, _o=0x0, _r=0x7fff22576920) at dbase.c:603
> #3  0x00007f0115f56398 in authorize (_m=0x78fe98, _realm=<value
> optimized out>, _table=<value optimized out>, _hftype=<value optimized
> out>)
>     at authorize.c:107
> #4  0x000000000040f869 in do_action (a=0x780a48, msg=0x78fe98) at action.c:961
> #5  0x000000000040e7e5 in run_action_list (a=<value optimized out>,
> msg=0x78fe98) at action.c:139
> #6  0x000000000046f194 in eval_elem (e=0x780b18, msg=0x78fe98,
> val=0x0) at route.c:1189
> #7  0x00000000004708ed in eval_expr (e=0x27648e0, msg=0x78fe98,
> val=0x0) at route.c:1486
> #8  0x000000000047089c in eval_expr (e=0x780b60, msg=0x78fe98,
> val=0x0) at route.c:1502
> #9  0x00000000004708c5 in eval_expr (e=0x780ba8, msg=0x78fe98,
> val=0x0) at route.c:1507
> #10 0x000000000040f91c in do_action (a=0x780f40, msg=0x78fe98) at action.c:688
> #11 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
> msg=0x78fe98) at action.c:139
> #12 0x00000000004114d3 in do_action (a=0x781eb0, msg=0x78fe98) at action.c:705
> #13 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
> msg=0x78fe98) at action.c:139
> #14 0x0000000000410e07 in do_action (a=0x77ee88, msg=0x78fe98) at action.c:119
> #15 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
> msg=0x78fe98) at action.c:139
> #16 0x00000000004114d3 in do_action (a=0x77f028, msg=0x78fe98) at action.c:705
> #17 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
> msg=0x78fe98) at action.c:139
> #18 0x00000000004125fe in run_top_route (a=0x777e78, msg=0x78fe98) at
> action.c:119
> #19 0x000000000045e898 in receive_msg (
>     buf=0x749180 "REGISTER sip:x.x.x.x SIP/2.0\r\nCSeq: 58
> REGISTER\r\nVia: SIP/2.0/UDP
> x.x.x.x:5068;branch=z9hG4bK5ca4685b-befc-dd11-911f-0015c5404858;rport\r\nUser-Agent:
> Ekiga/2.0.12\r\nAuthorization: Dig"..., len=750,
> rcv_info=0x7fff22578090) at receive.c:165
> #20 0x000000000049d3a6 in udp_rcv_loop () at udp_server.c:449
> #21 0x00000000004291fb in main (argc=<value optimized out>,
> argv=0x7fff22578278) at main.c:778
> ---
>
> I have tried several times and each time I get the same sequence of results.
>
> Thanks,
> Om.
>
> On Thu, Feb 19, 2009 at 11:22 AM, Om Bikram Thapa <[hidden email]> wrote:
>  
>> Thanks Bogdan. I will update and let you know.
>>
>> Regards,
>> Om.
>>
>> On Thu, Feb 19, 2009 at 2:48 AM, Bogdan-Andrei Iancu
>> <[hidden email]> wrote:
>>    
>>> Hi Om,
>>>
>>> There is a fix available on SVN - please update and see if it works now.
>>>
>>> Thanks and regards,
>>> Bogdan
>>>
>>> Om Bikram Thapa wrote:
>>>      
>>>> Hi Bogdan,
>>>>
>>>> I have submitted the bug on the tracker.
>>>>
>>>> Thanks,
>>>> Om.
>>>>
>>>> On Tue, Feb 17, 2009 at 3:50 AM, Bogdan-Andrei Iancu
>>>> <[hidden email]> wrote:
>>>>
>>>>        
>>>>> Hi Om,
>>>>>
>>>>> Thanks for the report - I may have an idea what is the problem; I will
>>>>> try
>>>>> to reproduce the crash in the following days.
>>>>>
>>>>> Could you please open a bug report on the tracker - just not to forget
>>>>> about
>>>>> it :D (see http://www.opensips.org/index.php?n=Development.Tracker)
>>>>>
>>>>> Regards,
>>>>> Bogdan
>>>>>
>>>>> Om Bikram Thapa wrote:
>>>>>
>>>>>          
>>>>>> Hi,
>>>>>>
>>>>>> OpenSIPS is dying with "libmysqlclient.so Segmentation Fault" after
>>>>>> MySQL server restart. The log shows
>>>>>> "opensips[16769]: segfault at 4c8 ip 7f36728bf283 sp 7fff7ba11a90
>>>>>> error 4 in libmysqlclient.so.15.0.0[7f3672862000+1bf000]"
>>>>>>
>>>>>> and gdb bt shows:
>>>>>>
>>>>>> -------------
>>>>>> #0  0x00007f36728bf283 in mysql_stmt_result_metadata () from
>>>>>> /usr/lib/libmysqlclient.so.15
>>>>>> #1  0x00007f3672c707b7 in db_mysql_do_prepared_query (conn=0x78b278,
>>>>>> query=<value optimized out>, v=0x7fff7ba11d20, n=1, uv=0x0, un=0) at
>>>>>> dbase.c:363
>>>>>> #2  0x00007f3672c72e1f in db_mysql_query (_h=0x78b278, _k=<value
>>>>>> optimized out>, _op=<value optimized out>, _v=0x7fff7ba11d20,
>>>>>> _c=<value optimized out>,
>>>>>>   _n=1, _nc=2, _o=0x0, _r=0x7fff7ba11dc0) at dbase.c:591
>>>>>> #3  0x00007f366f3f0398 in authorize (_m=0x78fe98, _realm=<value
>>>>>> optimized out>, _table=<value optimized out>, _hftype=<value optimized
>>>>>> out>)
>>>>>>   at authorize.c:107
>>>>>> #4  0x000000000040f869 in do_action (a=0x780a48, msg=0x78fe98) at
>>>>>> action.c:961
>>>>>> #5  0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>>>> msg=0x78fe98) at action.c:139
>>>>>> #6  0x000000000046f194 in eval_elem (e=0x780b18, msg=0x78fe98,
>>>>>> val=0x0) at route.c:1189
>>>>>> #7  0x00000000004708ed in eval_expr (e=0x120b1e0, msg=0x78fe98,
>>>>>> val=0x0) at route.c:1486
>>>>>> #8  0x000000000047089c in eval_expr (e=0x780b60, msg=0x78fe98,
>>>>>> val=0x0) at route.c:1502
>>>>>> #9  0x00000000004708c5 in eval_expr (e=0x780ba8, msg=0x78fe98,
>>>>>> val=0x0) at route.c:1507
>>>>>> #10 0x000000000040f91c in do_action (a=0x780f40, msg=0x78fe98) at
>>>>>> action.c:688
>>>>>> #11 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>>>> msg=0x78fe98) at action.c:139
>>>>>> #12 0x00000000004114d3 in do_action (a=0x781eb0, msg=0x78fe98) at
>>>>>> action.c:705
>>>>>> #13 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>>>> msg=0x78fe98) at action.c:139
>>>>>> #14 0x0000000000410e07 in do_action (a=0x77ee88, msg=0x78fe98) at
>>>>>> action.c:119
>>>>>> #15 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>>>> msg=0x78fe98) at action.c:139
>>>>>> #16 0x00000000004114d3 in do_action (a=0x77f028, msg=0x78fe98) at
>>>>>> action.c:705
>>>>>> #17 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>>>> msg=0x78fe98) at action.c:139
>>>>>> #18 0x00000000004125fe in run_top_route (a=0x777e78, msg=0x78fe98) at
>>>>>> action.c:119
>>>>>> #19 0x000000000045e898 in receive_msg (
>>>>>>   buf=0x749180 "REGISTER sip:xxxxx SIP/2.0\r\nCSeq: 2
>>>>>> REGISTER\r\nVia: SIP/2.0/UDP
>>>>>>
>>>>>>
>>>>>> x.x.x.x:5061;branch=z9hG4bK60f1b2dd-57fa-dd11-9401-0015c5404858;rport\r\nUser-Agent:
>>>>>> Ekiga/2.0.12\r\nAuthorization: Dige"..., len=749,
>>>>>> rcv_info=0x7fff7ba13530) at receive.c:165
>>>>>> #20 0x000000000049d3a6 in udp_rcv_loop () at udp_server.c:449
>>>>>> #21 0x00000000004291fb in main (argc=<value optimized out>,
>>>>>> argv=0x7fff7ba13718) at main.c:778
>>>>>> ------------
>>>>>>
>>>>>> Server runs fine after restart until MySQL is restarted again. In my
>>>>>> lab setup, OpenSIPS is getting killed triggered by registration
>>>>>> request every morning (probably due to MySQL being restarted daily
>>>>>> with logrotate).
>>>>>>
>>>>>> The server is the latest trunk on Debian lenny/AMD64.
>>>>>>
>>>>>> Thanks,
>>>>>> Om.
>>>>>>
>>>>>> _______________________________________________
>>>>>> Users mailing list
>>>>>> [hidden email]
>>>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>>>
>>>>>>
>>>>>>
>>>>>>            
>>>>>          
>>>>        
>>>      
>
>  


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: OpenSIPS libmysqlclient.so segfault after MySQL restart

Om Bikram Thapa
Hi Bogdan,

I will update and let you know the results.

Regards,
Om.

On Sat, Feb 28, 2009 at 3:08 AM, Bogdan-Andrei Iancu
<[hidden email]> wrote:

> Hi Om,
>
> finally, after some struggle with the libmysqlclient, I hope the issue is
> solve. Please give it another try and let me know the result.
>
> Thanks and regards,
> Bogdan
>
> Om Bikram Thapa wrote:
>>
>> Hi Bogdan,
>>
>> Just updated to latest svn, but I still see crashes after MySQL
>> restart. The syslog shows following 3 lines after MySQL restart and
>> registration attempt:
>>
>> --
>> Feb 19 12:17:43 proxy /usr/local/sbin/opensips[2333]:
>> ERROR:db_mysql:db_mysql_do_prepared_query: mysql_stmt_execute()
>> failed: (1) Unknown prepared statement handler (1) given to
>> mysql_stmt_execute
>> Feb 19 12:17:43 proxy /usr/local/sbin/opensips[2333]:
>> ERROR:auth_db:get_ha1: failed to query database
>> Feb 19 12:18:13 proxy kernel: [  105.116919] opensips[2333]: segfault
>> at 4c8 ip 7f167c109283 sp 7fff8525b270 error 4 in
>> libmysqlclient.so.15.0.0[7f167c0ac000+1bf000]
>> --
>>
>> At first attempt, the registration fails with "SIP/2.0 500 Server
>> Internal Error" returned by the server, and subsequent registration
>> attempt crashes the server.
>>
>> gdb backtrace gives following details:
>>
>> ---
>> #0  0x00007f0119426283 in mysql_stmt_result_metadata () from
>> /usr/lib/libmysqlclient.so.15
>> (gdb) bt
>> #0  0x00007f0119426283 in mysql_stmt_result_metadata () from
>> /usr/lib/libmysqlclient.so.15
>> #1  0x00007f01197d8f82 in db_mysql_do_prepared_query (conn=0x78b278,
>> query=<value optimized out>, v=0x7fff22576880, n=<value optimized
>> out>, uv=0x0, un=0)
>>    at dbase.c:368
>> #2  0x00007f01197daccd in db_mysql_query (_h=0x78b278, _k=<value
>> optimized out>, _op=<value optimized out>, _v=0x7fff22576880,
>> _c=<value optimized out>,
>>    _n=1, _nc=2, _o=0x0, _r=0x7fff22576920) at dbase.c:603
>> #3  0x00007f0115f56398 in authorize (_m=0x78fe98, _realm=<value
>> optimized out>, _table=<value optimized out>, _hftype=<value optimized
>> out>)
>>    at authorize.c:107
>> #4  0x000000000040f869 in do_action (a=0x780a48, msg=0x78fe98) at
>> action.c:961
>> #5  0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>> msg=0x78fe98) at action.c:139
>> #6  0x000000000046f194 in eval_elem (e=0x780b18, msg=0x78fe98,
>> val=0x0) at route.c:1189
>> #7  0x00000000004708ed in eval_expr (e=0x27648e0, msg=0x78fe98,
>> val=0x0) at route.c:1486
>> #8  0x000000000047089c in eval_expr (e=0x780b60, msg=0x78fe98,
>> val=0x0) at route.c:1502
>> #9  0x00000000004708c5 in eval_expr (e=0x780ba8, msg=0x78fe98,
>> val=0x0) at route.c:1507
>> #10 0x000000000040f91c in do_action (a=0x780f40, msg=0x78fe98) at
>> action.c:688
>> #11 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>> msg=0x78fe98) at action.c:139
>> #12 0x00000000004114d3 in do_action (a=0x781eb0, msg=0x78fe98) at
>> action.c:705
>> #13 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>> msg=0x78fe98) at action.c:139
>> #14 0x0000000000410e07 in do_action (a=0x77ee88, msg=0x78fe98) at
>> action.c:119
>> #15 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>> msg=0x78fe98) at action.c:139
>> #16 0x00000000004114d3 in do_action (a=0x77f028, msg=0x78fe98) at
>> action.c:705
>> #17 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>> msg=0x78fe98) at action.c:139
>> #18 0x00000000004125fe in run_top_route (a=0x777e78, msg=0x78fe98) at
>> action.c:119
>> #19 0x000000000045e898 in receive_msg (
>>    buf=0x749180 "REGISTER sip:x.x.x.x SIP/2.0\r\nCSeq: 58
>> REGISTER\r\nVia: SIP/2.0/UDP
>>
>> x.x.x.x:5068;branch=z9hG4bK5ca4685b-befc-dd11-911f-0015c5404858;rport\r\nUser-Agent:
>> Ekiga/2.0.12\r\nAuthorization: Dig"..., len=750,
>> rcv_info=0x7fff22578090) at receive.c:165
>> #20 0x000000000049d3a6 in udp_rcv_loop () at udp_server.c:449
>> #21 0x00000000004291fb in main (argc=<value optimized out>,
>> argv=0x7fff22578278) at main.c:778
>> ---
>>
>> I have tried several times and each time I get the same sequence of
>> results.
>>
>> Thanks,
>> Om.
>>
>> On Thu, Feb 19, 2009 at 11:22 AM, Om Bikram Thapa <[hidden email]>
>> wrote:
>>
>>>
>>> Thanks Bogdan. I will update and let you know.
>>>
>>> Regards,
>>> Om.
>>>
>>> On Thu, Feb 19, 2009 at 2:48 AM, Bogdan-Andrei Iancu
>>> <[hidden email]> wrote:
>>>
>>>>
>>>> Hi Om,
>>>>
>>>> There is a fix available on SVN - please update and see if it works now.
>>>>
>>>> Thanks and regards,
>>>> Bogdan
>>>>
>>>> Om Bikram Thapa wrote:
>>>>
>>>>>
>>>>> Hi Bogdan,
>>>>>
>>>>> I have submitted the bug on the tracker.
>>>>>
>>>>> Thanks,
>>>>> Om.
>>>>>
>>>>> On Tue, Feb 17, 2009 at 3:50 AM, Bogdan-Andrei Iancu
>>>>> <[hidden email]> wrote:
>>>>>
>>>>>
>>>>>>
>>>>>> Hi Om,
>>>>>>
>>>>>> Thanks for the report - I may have an idea what is the problem; I will
>>>>>> try
>>>>>> to reproduce the crash in the following days.
>>>>>>
>>>>>> Could you please open a bug report on the tracker - just not to forget
>>>>>> about
>>>>>> it :D (see http://www.opensips.org/index.php?n=Development.Tracker)
>>>>>>
>>>>>> Regards,
>>>>>> Bogdan
>>>>>>
>>>>>> Om Bikram Thapa wrote:
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> OpenSIPS is dying with "libmysqlclient.so Segmentation Fault" after
>>>>>>> MySQL server restart. The log shows
>>>>>>> "opensips[16769]: segfault at 4c8 ip 7f36728bf283 sp 7fff7ba11a90
>>>>>>> error 4 in libmysqlclient.so.15.0.0[7f3672862000+1bf000]"
>>>>>>>
>>>>>>> and gdb bt shows:
>>>>>>>
>>>>>>> -------------
>>>>>>> #0  0x00007f36728bf283 in mysql_stmt_result_metadata () from
>>>>>>> /usr/lib/libmysqlclient.so.15
>>>>>>> #1  0x00007f3672c707b7 in db_mysql_do_prepared_query (conn=0x78b278,
>>>>>>> query=<value optimized out>, v=0x7fff7ba11d20, n=1, uv=0x0, un=0) at
>>>>>>> dbase.c:363
>>>>>>> #2  0x00007f3672c72e1f in db_mysql_query (_h=0x78b278, _k=<value
>>>>>>> optimized out>, _op=<value optimized out>, _v=0x7fff7ba11d20,
>>>>>>> _c=<value optimized out>,
>>>>>>>  _n=1, _nc=2, _o=0x0, _r=0x7fff7ba11dc0) at dbase.c:591
>>>>>>> #3  0x00007f366f3f0398 in authorize (_m=0x78fe98, _realm=<value
>>>>>>> optimized out>, _table=<value optimized out>, _hftype=<value
>>>>>>> optimized
>>>>>>> out>)
>>>>>>>  at authorize.c:107
>>>>>>> #4  0x000000000040f869 in do_action (a=0x780a48, msg=0x78fe98) at
>>>>>>> action.c:961
>>>>>>> #5  0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>>>>> msg=0x78fe98) at action.c:139
>>>>>>> #6  0x000000000046f194 in eval_elem (e=0x780b18, msg=0x78fe98,
>>>>>>> val=0x0) at route.c:1189
>>>>>>> #7  0x00000000004708ed in eval_expr (e=0x120b1e0, msg=0x78fe98,
>>>>>>> val=0x0) at route.c:1486
>>>>>>> #8  0x000000000047089c in eval_expr (e=0x780b60, msg=0x78fe98,
>>>>>>> val=0x0) at route.c:1502
>>>>>>> #9  0x00000000004708c5 in eval_expr (e=0x780ba8, msg=0x78fe98,
>>>>>>> val=0x0) at route.c:1507
>>>>>>> #10 0x000000000040f91c in do_action (a=0x780f40, msg=0x78fe98) at
>>>>>>> action.c:688
>>>>>>> #11 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>>>>> msg=0x78fe98) at action.c:139
>>>>>>> #12 0x00000000004114d3 in do_action (a=0x781eb0, msg=0x78fe98) at
>>>>>>> action.c:705
>>>>>>> #13 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>>>>> msg=0x78fe98) at action.c:139
>>>>>>> #14 0x0000000000410e07 in do_action (a=0x77ee88, msg=0x78fe98) at
>>>>>>> action.c:119
>>>>>>> #15 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>>>>> msg=0x78fe98) at action.c:139
>>>>>>> #16 0x00000000004114d3 in do_action (a=0x77f028, msg=0x78fe98) at
>>>>>>> action.c:705
>>>>>>> #17 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>>>>> msg=0x78fe98) at action.c:139
>>>>>>> #18 0x00000000004125fe in run_top_route (a=0x777e78, msg=0x78fe98) at
>>>>>>> action.c:119
>>>>>>> #19 0x000000000045e898 in receive_msg (
>>>>>>>  buf=0x749180 "REGISTER sip:xxxxx SIP/2.0\r\nCSeq: 2
>>>>>>> REGISTER\r\nVia: SIP/2.0/UDP
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> x.x.x.x:5061;branch=z9hG4bK60f1b2dd-57fa-dd11-9401-0015c5404858;rport\r\nUser-Agent:
>>>>>>> Ekiga/2.0.12\r\nAuthorization: Dige"..., len=749,
>>>>>>> rcv_info=0x7fff7ba13530) at receive.c:165
>>>>>>> #20 0x000000000049d3a6 in udp_rcv_loop () at udp_server.c:449
>>>>>>> #21 0x00000000004291fb in main (argc=<value optimized out>,
>>>>>>> argv=0x7fff7ba13718) at main.c:778
>>>>>>> ------------
>>>>>>>
>>>>>>> Server runs fine after restart until MySQL is restarted again. In my
>>>>>>> lab setup, OpenSIPS is getting killed triggered by registration
>>>>>>> request every morning (probably due to MySQL being restarted daily
>>>>>>> with logrotate).
>>>>>>>
>>>>>>> The server is the latest trunk on Debian lenny/AMD64.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Om.
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Users mailing list
>>>>>>> [hidden email]
>>>>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>
>>
>
>

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: OpenSIPS libmysqlclient.so segfault after MySQL restart

Om Bikram Thapa
Hi Bogdan,

Solved! OpenSIPS (latest SVN) no longer crashes after MySQL restarts.

Thanks,
Om.

On Sun, Mar 1, 2009 at 11:33 AM, Om Bikram Thapa <[hidden email]> wrote:

> Hi Bogdan,
>
> I will update and let you know the results.
>
> Regards,
> Om.
>
> On Sat, Feb 28, 2009 at 3:08 AM, Bogdan-Andrei Iancu
> <[hidden email]> wrote:
>> Hi Om,
>>
>> finally, after some struggle with the libmysqlclient, I hope the issue is
>> solve. Please give it another try and let me know the result.
>>
>> Thanks and regards,
>> Bogdan
>>
>> Om Bikram Thapa wrote:
>>>
>>> Hi Bogdan,
>>>
>>> Just updated to latest svn, but I still see crashes after MySQL
>>> restart. The syslog shows following 3 lines after MySQL restart and
>>> registration attempt:
>>>
>>> --
>>> Feb 19 12:17:43 proxy /usr/local/sbin/opensips[2333]:
>>> ERROR:db_mysql:db_mysql_do_prepared_query: mysql_stmt_execute()
>>> failed: (1) Unknown prepared statement handler (1) given to
>>> mysql_stmt_execute
>>> Feb 19 12:17:43 proxy /usr/local/sbin/opensips[2333]:
>>> ERROR:auth_db:get_ha1: failed to query database
>>> Feb 19 12:18:13 proxy kernel: [  105.116919] opensips[2333]: segfault
>>> at 4c8 ip 7f167c109283 sp 7fff8525b270 error 4 in
>>> libmysqlclient.so.15.0.0[7f167c0ac000+1bf000]
>>> --
>>>
>>> At first attempt, the registration fails with "SIP/2.0 500 Server
>>> Internal Error" returned by the server, and subsequent registration
>>> attempt crashes the server.
>>>
>>> gdb backtrace gives following details:
>>>
>>> ---
>>> #0  0x00007f0119426283 in mysql_stmt_result_metadata () from
>>> /usr/lib/libmysqlclient.so.15
>>> (gdb) bt
>>> #0  0x00007f0119426283 in mysql_stmt_result_metadata () from
>>> /usr/lib/libmysqlclient.so.15
>>> #1  0x00007f01197d8f82 in db_mysql_do_prepared_query (conn=0x78b278,
>>> query=<value optimized out>, v=0x7fff22576880, n=<value optimized
>>> out>, uv=0x0, un=0)
>>>    at dbase.c:368
>>> #2  0x00007f01197daccd in db_mysql_query (_h=0x78b278, _k=<value
>>> optimized out>, _op=<value optimized out>, _v=0x7fff22576880,
>>> _c=<value optimized out>,
>>>    _n=1, _nc=2, _o=0x0, _r=0x7fff22576920) at dbase.c:603
>>> #3  0x00007f0115f56398 in authorize (_m=0x78fe98, _realm=<value
>>> optimized out>, _table=<value optimized out>, _hftype=<value optimized
>>> out>)
>>>    at authorize.c:107
>>> #4  0x000000000040f869 in do_action (a=0x780a48, msg=0x78fe98) at
>>> action.c:961
>>> #5  0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>> msg=0x78fe98) at action.c:139
>>> #6  0x000000000046f194 in eval_elem (e=0x780b18, msg=0x78fe98,
>>> val=0x0) at route.c:1189
>>> #7  0x00000000004708ed in eval_expr (e=0x27648e0, msg=0x78fe98,
>>> val=0x0) at route.c:1486
>>> #8  0x000000000047089c in eval_expr (e=0x780b60, msg=0x78fe98,
>>> val=0x0) at route.c:1502
>>> #9  0x00000000004708c5 in eval_expr (e=0x780ba8, msg=0x78fe98,
>>> val=0x0) at route.c:1507
>>> #10 0x000000000040f91c in do_action (a=0x780f40, msg=0x78fe98) at
>>> action.c:688
>>> #11 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>> msg=0x78fe98) at action.c:139
>>> #12 0x00000000004114d3 in do_action (a=0x781eb0, msg=0x78fe98) at
>>> action.c:705
>>> #13 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>> msg=0x78fe98) at action.c:139
>>> #14 0x0000000000410e07 in do_action (a=0x77ee88, msg=0x78fe98) at
>>> action.c:119
>>> #15 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>> msg=0x78fe98) at action.c:139
>>> #16 0x00000000004114d3 in do_action (a=0x77f028, msg=0x78fe98) at
>>> action.c:705
>>> #17 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>> msg=0x78fe98) at action.c:139
>>> #18 0x00000000004125fe in run_top_route (a=0x777e78, msg=0x78fe98) at
>>> action.c:119
>>> #19 0x000000000045e898 in receive_msg (
>>>    buf=0x749180 "REGISTER sip:x.x.x.x SIP/2.0\r\nCSeq: 58
>>> REGISTER\r\nVia: SIP/2.0/UDP
>>>
>>> x.x.x.x:5068;branch=z9hG4bK5ca4685b-befc-dd11-911f-0015c5404858;rport\r\nUser-Agent:
>>> Ekiga/2.0.12\r\nAuthorization: Dig"..., len=750,
>>> rcv_info=0x7fff22578090) at receive.c:165
>>> #20 0x000000000049d3a6 in udp_rcv_loop () at udp_server.c:449
>>> #21 0x00000000004291fb in main (argc=<value optimized out>,
>>> argv=0x7fff22578278) at main.c:778
>>> ---
>>>
>>> I have tried several times and each time I get the same sequence of
>>> results.
>>>
>>> Thanks,
>>> Om.
>>>
>>> On Thu, Feb 19, 2009 at 11:22 AM, Om Bikram Thapa <[hidden email]>
>>> wrote:
>>>
>>>>
>>>> Thanks Bogdan. I will update and let you know.
>>>>
>>>> Regards,
>>>> Om.
>>>>
>>>> On Thu, Feb 19, 2009 at 2:48 AM, Bogdan-Andrei Iancu
>>>> <[hidden email]> wrote:
>>>>
>>>>>
>>>>> Hi Om,
>>>>>
>>>>> There is a fix available on SVN - please update and see if it works now.
>>>>>
>>>>> Thanks and regards,
>>>>> Bogdan
>>>>>
>>>>> Om Bikram Thapa wrote:
>>>>>
>>>>>>
>>>>>> Hi Bogdan,
>>>>>>
>>>>>> I have submitted the bug on the tracker.
>>>>>>
>>>>>> Thanks,
>>>>>> Om.
>>>>>>
>>>>>> On Tue, Feb 17, 2009 at 3:50 AM, Bogdan-Andrei Iancu
>>>>>> <[hidden email]> wrote:
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Hi Om,
>>>>>>>
>>>>>>> Thanks for the report - I may have an idea what is the problem; I will
>>>>>>> try
>>>>>>> to reproduce the crash in the following days.
>>>>>>>
>>>>>>> Could you please open a bug report on the tracker - just not to forget
>>>>>>> about
>>>>>>> it :D (see http://www.opensips.org/index.php?n=Development.Tracker)
>>>>>>>
>>>>>>> Regards,
>>>>>>> Bogdan
>>>>>>>
>>>>>>> Om Bikram Thapa wrote:
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> OpenSIPS is dying with "libmysqlclient.so Segmentation Fault" after
>>>>>>>> MySQL server restart. The log shows
>>>>>>>> "opensips[16769]: segfault at 4c8 ip 7f36728bf283 sp 7fff7ba11a90
>>>>>>>> error 4 in libmysqlclient.so.15.0.0[7f3672862000+1bf000]"
>>>>>>>>
>>>>>>>> and gdb bt shows:
>>>>>>>>
>>>>>>>> -------------
>>>>>>>> #0  0x00007f36728bf283 in mysql_stmt_result_metadata () from
>>>>>>>> /usr/lib/libmysqlclient.so.15
>>>>>>>> #1  0x00007f3672c707b7 in db_mysql_do_prepared_query (conn=0x78b278,
>>>>>>>> query=<value optimized out>, v=0x7fff7ba11d20, n=1, uv=0x0, un=0) at
>>>>>>>> dbase.c:363
>>>>>>>> #2  0x00007f3672c72e1f in db_mysql_query (_h=0x78b278, _k=<value
>>>>>>>> optimized out>, _op=<value optimized out>, _v=0x7fff7ba11d20,
>>>>>>>> _c=<value optimized out>,
>>>>>>>>  _n=1, _nc=2, _o=0x0, _r=0x7fff7ba11dc0) at dbase.c:591
>>>>>>>> #3  0x00007f366f3f0398 in authorize (_m=0x78fe98, _realm=<value
>>>>>>>> optimized out>, _table=<value optimized out>, _hftype=<value
>>>>>>>> optimized
>>>>>>>> out>)
>>>>>>>>  at authorize.c:107
>>>>>>>> #4  0x000000000040f869 in do_action (a=0x780a48, msg=0x78fe98) at
>>>>>>>> action.c:961
>>>>>>>> #5  0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>>>>>> msg=0x78fe98) at action.c:139
>>>>>>>> #6  0x000000000046f194 in eval_elem (e=0x780b18, msg=0x78fe98,
>>>>>>>> val=0x0) at route.c:1189
>>>>>>>> #7  0x00000000004708ed in eval_expr (e=0x120b1e0, msg=0x78fe98,
>>>>>>>> val=0x0) at route.c:1486
>>>>>>>> #8  0x000000000047089c in eval_expr (e=0x780b60, msg=0x78fe98,
>>>>>>>> val=0x0) at route.c:1502
>>>>>>>> #9  0x00000000004708c5 in eval_expr (e=0x780ba8, msg=0x78fe98,
>>>>>>>> val=0x0) at route.c:1507
>>>>>>>> #10 0x000000000040f91c in do_action (a=0x780f40, msg=0x78fe98) at
>>>>>>>> action.c:688
>>>>>>>> #11 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>>>>>> msg=0x78fe98) at action.c:139
>>>>>>>> #12 0x00000000004114d3 in do_action (a=0x781eb0, msg=0x78fe98) at
>>>>>>>> action.c:705
>>>>>>>> #13 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>>>>>> msg=0x78fe98) at action.c:139
>>>>>>>> #14 0x0000000000410e07 in do_action (a=0x77ee88, msg=0x78fe98) at
>>>>>>>> action.c:119
>>>>>>>> #15 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>>>>>> msg=0x78fe98) at action.c:139
>>>>>>>> #16 0x00000000004114d3 in do_action (a=0x77f028, msg=0x78fe98) at
>>>>>>>> action.c:705
>>>>>>>> #17 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>>>>>> msg=0x78fe98) at action.c:139
>>>>>>>> #18 0x00000000004125fe in run_top_route (a=0x777e78, msg=0x78fe98) at
>>>>>>>> action.c:119
>>>>>>>> #19 0x000000000045e898 in receive_msg (
>>>>>>>>  buf=0x749180 "REGISTER sip:xxxxx SIP/2.0\r\nCSeq: 2
>>>>>>>> REGISTER\r\nVia: SIP/2.0/UDP
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> x.x.x.x:5061;branch=z9hG4bK60f1b2dd-57fa-dd11-9401-0015c5404858;rport\r\nUser-Agent:
>>>>>>>> Ekiga/2.0.12\r\nAuthorization: Dige"..., len=749,
>>>>>>>> rcv_info=0x7fff7ba13530) at receive.c:165
>>>>>>>> #20 0x000000000049d3a6 in udp_rcv_loop () at udp_server.c:449
>>>>>>>> #21 0x00000000004291fb in main (argc=<value optimized out>,
>>>>>>>> argv=0x7fff7ba13718) at main.c:778
>>>>>>>> ------------
>>>>>>>>
>>>>>>>> Server runs fine after restart until MySQL is restarted again. In my
>>>>>>>> lab setup, OpenSIPS is getting killed triggered by registration
>>>>>>>> request every morning (probably due to MySQL being restarted daily
>>>>>>>> with logrotate).
>>>>>>>>
>>>>>>>> The server is the latest trunk on Debian lenny/AMD64.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Om.
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Users mailing list
>>>>>>>> [hidden email]
>>>>>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>
>>>
>>
>>
>

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: OpenSIPS libmysqlclient.so segfault after MySQL restart

Om Bikram Thapa
Hi Bogdan,

Seems like I replied too soon. OpenSIPS again crashed on INVITE,
although REGISTER operation was fine after a couple of MySQL restarts.
I again restarted MySQL and got the server crashing on REGISTER as
well. gdb trace for both the event crashes point to the same code:

---
Core was generated by `/usr/local/sbin/opensips -P
/var/run/opensips/opensips.pid -m 64 -u root -g roo'.
Program terminated with signal 11, Segmentation fault.
[New process 7352]
#0  0x00007f13a0f38283 in mysql_stmt_result_metadata () from
/usr/lib/libmysqlclient.so.15
(gdb) bt
#0  0x00007f13a0f38283 in mysql_stmt_result_metadata () from
/usr/lib/libmysqlclient.so.15
#1  0x00007f13a12ea4ba in db_mysql_do_prepared_query (conn=0x7cf7c0,
query=<value optimized out>, v=0x7fffaa088fa0, n=<value optimized
out>, uv=0x0, un=0)
    at dbase.c:370
#2  0x00007f13a12ecb41 in db_mysql_query (_h=0x7cf7c0, _k=<value
optimized out>, _op=<value optimized out>, _v=0x7fffaa088fa0,
_c=<value optimized out>,
    _n=1, _nc=2, _o=0x0, _r=0x7fffaa089040) at dbase.c:621
#3  0x00007f139da6547c in authorize (_m=0x7d0490, _realm=<value
optimized out>, _table=<value optimized out>, _hftype=<value optimized
out>)
    at authorize.c:107
#4  0x000000000040f849 in do_action (a=0x7bd1f8, msg=0x7d0490) at action.c:961
#5  0x000000000040e7a5 in run_action_list (a=<value optimized out>,
msg=0x7d0490) at action.c:139
#6  0x0000000000470394 in eval_elem (e=0x7bd318, msg=0x7d0490,
val=0x0) at route.c:1189
#7  0x0000000000471b0d in eval_expr (e=0x14e0160, msg=0x7d0490,
val=0x0) at route.c:1486
#8  0x0000000000471abc in eval_expr (e=0x7bd3b0, msg=0x7d0490,
val=0x0) at route.c:1502
#9  0x0000000000471ae5 in eval_expr (e=0x7bd448, msg=0x7d0490,
val=0x0) at route.c:1507
#10 0x000000000040f8fc in do_action (a=0x7be468, msg=0x7d0490) at action.c:688
#11 0x000000000040e7a5 in run_action_list (a=<value optimized out>,
msg=0x7d0490) at action.c:139
#12 0x000000000041159b in do_action (a=0x7be788, msg=0x7d0490) at action.c:705
#13 0x000000000040e7a5 in run_action_list (a=<value optimized out>,
msg=0x7d0490) at action.c:139
#14 0x000000000041159b in do_action (a=0x7bfaa8, msg=0x7d0490) at action.c:705
#15 0x000000000040e7a5 in run_action_list (a=<value optimized out>,
msg=0x7d0490) at action.c:139
#16 0x0000000000410e9f in do_action (a=0x7b80a0, msg=0x7d0490) at action.c:119
#17 0x000000000040e7a5 in run_action_list (a=<value optimized out>,
msg=0x7d0490) at action.c:139
#18 0x0000000000411a1b in do_action (a=0x7b81c0, msg=0x7d0490) at action.c:711
#19 0x000000000040e7a5 in run_action_list (a=<value optimized out>,
msg=0x7d0490) at action.c:139
#20 0x00000000004126a5 in run_top_route (a=0x7ace70, msg=0x7d0490) at
action.c:119
#21 0x000000000045f7a8 in receive_msg (
    buf=0x74e140 "INVITE sip:[hidden email] SIP/2.0\r\nDate: Sun, 01 Mar
2009 09:06:10 GMT\r\nCSeq: 2 INVITE\r\nVia: SIP/2.0/UDP
x.x.x.x:5063;branch=z9hG4bK4e6660e0-ad04-de11-8875-001302b72316;rport\r\nUser-Agent:
E"..., len=1297, rcv_info=0x7fffaa08aba0) at receive.c:165
#22 0x000000000049eda6 in udp_rcv_loop () at udp_server.c:449
#23 0x00000000004298bb in main (argc=<value optimized out>,
argv=0x7fffaa08ad88) at main.c:778

---

Thanks,
Om.

On Sun, Mar 1, 2009 at 12:45 PM, Om Bikram Thapa <[hidden email]> wrote:

> Hi Bogdan,
>
> Solved! OpenSIPS (latest SVN) no longer crashes after MySQL restarts.
>
> Thanks,
> Om.
>
> On Sun, Mar 1, 2009 at 11:33 AM, Om Bikram Thapa <[hidden email]> wrote:
>> Hi Bogdan,
>>
>> I will update and let you know the results.
>>
>> Regards,
>> Om.
>>
>> On Sat, Feb 28, 2009 at 3:08 AM, Bogdan-Andrei Iancu
>> <[hidden email]> wrote:
>>> Hi Om,
>>>
>>> finally, after some struggle with the libmysqlclient, I hope the issue is
>>> solve. Please give it another try and let me know the result.
>>>
>>> Thanks and regards,
>>> Bogdan
>>>
>>> Om Bikram Thapa wrote:
>>>>
>>>> Hi Bogdan,
>>>>
>>>> Just updated to latest svn, but I still see crashes after MySQL
>>>> restart. The syslog shows following 3 lines after MySQL restart and
>>>> registration attempt:
>>>>
>>>> --
>>>> Feb 19 12:17:43 proxy /usr/local/sbin/opensips[2333]:
>>>> ERROR:db_mysql:db_mysql_do_prepared_query: mysql_stmt_execute()
>>>> failed: (1) Unknown prepared statement handler (1) given to
>>>> mysql_stmt_execute
>>>> Feb 19 12:17:43 proxy /usr/local/sbin/opensips[2333]:
>>>> ERROR:auth_db:get_ha1: failed to query database
>>>> Feb 19 12:18:13 proxy kernel: [  105.116919] opensips[2333]: segfault
>>>> at 4c8 ip 7f167c109283 sp 7fff8525b270 error 4 in
>>>> libmysqlclient.so.15.0.0[7f167c0ac000+1bf000]
>>>> --
>>>>
>>>> At first attempt, the registration fails with "SIP/2.0 500 Server
>>>> Internal Error" returned by the server, and subsequent registration
>>>> attempt crashes the server.
>>>>
>>>> gdb backtrace gives following details:
>>>>
>>>> ---
>>>> #0  0x00007f0119426283 in mysql_stmt_result_metadata () from
>>>> /usr/lib/libmysqlclient.so.15
>>>> (gdb) bt
>>>> #0  0x00007f0119426283 in mysql_stmt_result_metadata () from
>>>> /usr/lib/libmysqlclient.so.15
>>>> #1  0x00007f01197d8f82 in db_mysql_do_prepared_query (conn=0x78b278,
>>>> query=<value optimized out>, v=0x7fff22576880, n=<value optimized
>>>> out>, uv=0x0, un=0)
>>>>    at dbase.c:368
>>>> #2  0x00007f01197daccd in db_mysql_query (_h=0x78b278, _k=<value
>>>> optimized out>, _op=<value optimized out>, _v=0x7fff22576880,
>>>> _c=<value optimized out>,
>>>>    _n=1, _nc=2, _o=0x0, _r=0x7fff22576920) at dbase.c:603
>>>> #3  0x00007f0115f56398 in authorize (_m=0x78fe98, _realm=<value
>>>> optimized out>, _table=<value optimized out>, _hftype=<value optimized
>>>> out>)
>>>>    at authorize.c:107
>>>> #4  0x000000000040f869 in do_action (a=0x780a48, msg=0x78fe98) at
>>>> action.c:961
>>>> #5  0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>> msg=0x78fe98) at action.c:139
>>>> #6  0x000000000046f194 in eval_elem (e=0x780b18, msg=0x78fe98,
>>>> val=0x0) at route.c:1189
>>>> #7  0x00000000004708ed in eval_expr (e=0x27648e0, msg=0x78fe98,
>>>> val=0x0) at route.c:1486
>>>> #8  0x000000000047089c in eval_expr (e=0x780b60, msg=0x78fe98,
>>>> val=0x0) at route.c:1502
>>>> #9  0x00000000004708c5 in eval_expr (e=0x780ba8, msg=0x78fe98,
>>>> val=0x0) at route.c:1507
>>>> #10 0x000000000040f91c in do_action (a=0x780f40, msg=0x78fe98) at
>>>> action.c:688
>>>> #11 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>> msg=0x78fe98) at action.c:139
>>>> #12 0x00000000004114d3 in do_action (a=0x781eb0, msg=0x78fe98) at
>>>> action.c:705
>>>> #13 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>> msg=0x78fe98) at action.c:139
>>>> #14 0x0000000000410e07 in do_action (a=0x77ee88, msg=0x78fe98) at
>>>> action.c:119
>>>> #15 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>> msg=0x78fe98) at action.c:139
>>>> #16 0x00000000004114d3 in do_action (a=0x77f028, msg=0x78fe98) at
>>>> action.c:705
>>>> #17 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>> msg=0x78fe98) at action.c:139
>>>> #18 0x00000000004125fe in run_top_route (a=0x777e78, msg=0x78fe98) at
>>>> action.c:119
>>>> #19 0x000000000045e898 in receive_msg (
>>>>    buf=0x749180 "REGISTER sip:x.x.x.x SIP/2.0\r\nCSeq: 58
>>>> REGISTER\r\nVia: SIP/2.0/UDP
>>>>
>>>> x.x.x.x:5068;branch=z9hG4bK5ca4685b-befc-dd11-911f-0015c5404858;rport\r\nUser-Agent:
>>>> Ekiga/2.0.12\r\nAuthorization: Dig"..., len=750,
>>>> rcv_info=0x7fff22578090) at receive.c:165
>>>> #20 0x000000000049d3a6 in udp_rcv_loop () at udp_server.c:449
>>>> #21 0x00000000004291fb in main (argc=<value optimized out>,
>>>> argv=0x7fff22578278) at main.c:778
>>>> ---
>>>>
>>>> I have tried several times and each time I get the same sequence of
>>>> results.
>>>>
>>>> Thanks,
>>>> Om.
>>>>
>>>> On Thu, Feb 19, 2009 at 11:22 AM, Om Bikram Thapa <[hidden email]>
>>>> wrote:
>>>>
>>>>>
>>>>> Thanks Bogdan. I will update and let you know.
>>>>>
>>>>> Regards,
>>>>> Om.
>>>>>
>>>>> On Thu, Feb 19, 2009 at 2:48 AM, Bogdan-Andrei Iancu
>>>>> <[hidden email]> wrote:
>>>>>
>>>>>>
>>>>>> Hi Om,
>>>>>>
>>>>>> There is a fix available on SVN - please update and see if it works now.
>>>>>>
>>>>>> Thanks and regards,
>>>>>> Bogdan
>>>>>>
>>>>>> Om Bikram Thapa wrote:
>>>>>>
>>>>>>>
>>>>>>> Hi Bogdan,
>>>>>>>
>>>>>>> I have submitted the bug on the tracker.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Om.
>>>>>>>
>>>>>>> On Tue, Feb 17, 2009 at 3:50 AM, Bogdan-Andrei Iancu
>>>>>>> <[hidden email]> wrote:
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> Hi Om,
>>>>>>>>
>>>>>>>> Thanks for the report - I may have an idea what is the problem; I will
>>>>>>>> try
>>>>>>>> to reproduce the crash in the following days.
>>>>>>>>
>>>>>>>> Could you please open a bug report on the tracker - just not to forget
>>>>>>>> about
>>>>>>>> it :D (see http://www.opensips.org/index.php?n=Development.Tracker)
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Bogdan
>>>>>>>>
>>>>>>>> Om Bikram Thapa wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> OpenSIPS is dying with "libmysqlclient.so Segmentation Fault" after
>>>>>>>>> MySQL server restart. The log shows
>>>>>>>>> "opensips[16769]: segfault at 4c8 ip 7f36728bf283 sp 7fff7ba11a90
>>>>>>>>> error 4 in libmysqlclient.so.15.0.0[7f3672862000+1bf000]"
>>>>>>>>>
>>>>>>>>> and gdb bt shows:
>>>>>>>>>
>>>>>>>>> -------------
>>>>>>>>> #0  0x00007f36728bf283 in mysql_stmt_result_metadata () from
>>>>>>>>> /usr/lib/libmysqlclient.so.15
>>>>>>>>> #1  0x00007f3672c707b7 in db_mysql_do_prepared_query (conn=0x78b278,
>>>>>>>>> query=<value optimized out>, v=0x7fff7ba11d20, n=1, uv=0x0, un=0) at
>>>>>>>>> dbase.c:363
>>>>>>>>> #2  0x00007f3672c72e1f in db_mysql_query (_h=0x78b278, _k=<value
>>>>>>>>> optimized out>, _op=<value optimized out>, _v=0x7fff7ba11d20,
>>>>>>>>> _c=<value optimized out>,
>>>>>>>>>  _n=1, _nc=2, _o=0x0, _r=0x7fff7ba11dc0) at dbase.c:591
>>>>>>>>> #3  0x00007f366f3f0398 in authorize (_m=0x78fe98, _realm=<value
>>>>>>>>> optimized out>, _table=<value optimized out>, _hftype=<value
>>>>>>>>> optimized
>>>>>>>>> out>)
>>>>>>>>>  at authorize.c:107
>>>>>>>>> #4  0x000000000040f869 in do_action (a=0x780a48, msg=0x78fe98) at
>>>>>>>>> action.c:961
>>>>>>>>> #5  0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>>>>>>> msg=0x78fe98) at action.c:139
>>>>>>>>> #6  0x000000000046f194 in eval_elem (e=0x780b18, msg=0x78fe98,
>>>>>>>>> val=0x0) at route.c:1189
>>>>>>>>> #7  0x00000000004708ed in eval_expr (e=0x120b1e0, msg=0x78fe98,
>>>>>>>>> val=0x0) at route.c:1486
>>>>>>>>> #8  0x000000000047089c in eval_expr (e=0x780b60, msg=0x78fe98,
>>>>>>>>> val=0x0) at route.c:1502
>>>>>>>>> #9  0x00000000004708c5 in eval_expr (e=0x780ba8, msg=0x78fe98,
>>>>>>>>> val=0x0) at route.c:1507
>>>>>>>>> #10 0x000000000040f91c in do_action (a=0x780f40, msg=0x78fe98) at
>>>>>>>>> action.c:688
>>>>>>>>> #11 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>>>>>>> msg=0x78fe98) at action.c:139
>>>>>>>>> #12 0x00000000004114d3 in do_action (a=0x781eb0, msg=0x78fe98) at
>>>>>>>>> action.c:705
>>>>>>>>> #13 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>>>>>>> msg=0x78fe98) at action.c:139
>>>>>>>>> #14 0x0000000000410e07 in do_action (a=0x77ee88, msg=0x78fe98) at
>>>>>>>>> action.c:119
>>>>>>>>> #15 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>>>>>>> msg=0x78fe98) at action.c:139
>>>>>>>>> #16 0x00000000004114d3 in do_action (a=0x77f028, msg=0x78fe98) at
>>>>>>>>> action.c:705
>>>>>>>>> #17 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>>>>>>> msg=0x78fe98) at action.c:139
>>>>>>>>> #18 0x00000000004125fe in run_top_route (a=0x777e78, msg=0x78fe98) at
>>>>>>>>> action.c:119
>>>>>>>>> #19 0x000000000045e898 in receive_msg (
>>>>>>>>>  buf=0x749180 "REGISTER sip:xxxxx SIP/2.0\r\nCSeq: 2
>>>>>>>>> REGISTER\r\nVia: SIP/2.0/UDP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> x.x.x.x:5061;branch=z9hG4bK60f1b2dd-57fa-dd11-9401-0015c5404858;rport\r\nUser-Agent:
>>>>>>>>> Ekiga/2.0.12\r\nAuthorization: Dige"..., len=749,
>>>>>>>>> rcv_info=0x7fff7ba13530) at receive.c:165
>>>>>>>>> #20 0x000000000049d3a6 in udp_rcv_loop () at udp_server.c:449
>>>>>>>>> #21 0x00000000004291fb in main (argc=<value optimized out>,
>>>>>>>>> argv=0x7fff7ba13718) at main.c:778
>>>>>>>>> ------------
>>>>>>>>>
>>>>>>>>> Server runs fine after restart until MySQL is restarted again. In my
>>>>>>>>> lab setup, OpenSIPS is getting killed triggered by registration
>>>>>>>>> request every morning (probably due to MySQL being restarted daily
>>>>>>>>> with logrotate).
>>>>>>>>>
>>>>>>>>> The server is the latest trunk on Debian lenny/AMD64.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Om.
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Users mailing list
>>>>>>>>> [hidden email]
>>>>>>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>
>>>>
>>>
>>>
>>
>

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: OpenSIPS libmysqlclient.so segfault after MySQL restart

Bogdan-Andrei Iancu
Hi Om,

Can you send me the full debug log (debug=6) for this scenario (you can
send it privately, if too big).

I just want to check exactly what is going one.

Thanks for your help,
Bogdan

Om Bikram Thapa wrote:

> Hi Bogdan,
>
> Seems like I replied too soon. OpenSIPS again crashed on INVITE,
> although REGISTER operation was fine after a couple of MySQL restarts.
> I again restarted MySQL and got the server crashing on REGISTER as
> well. gdb trace for both the event crashes point to the same code:
>
> ---
> Core was generated by `/usr/local/sbin/opensips -P
> /var/run/opensips/opensips.pid -m 64 -u root -g roo'.
> Program terminated with signal 11, Segmentation fault.
> [New process 7352]
> #0  0x00007f13a0f38283 in mysql_stmt_result_metadata () from
> /usr/lib/libmysqlclient.so.15
> (gdb) bt
> #0  0x00007f13a0f38283 in mysql_stmt_result_metadata () from
> /usr/lib/libmysqlclient.so.15
> #1  0x00007f13a12ea4ba in db_mysql_do_prepared_query (conn=0x7cf7c0,
> query=<value optimized out>, v=0x7fffaa088fa0, n=<value optimized
> out>, uv=0x0, un=0)
>     at dbase.c:370
> #2  0x00007f13a12ecb41 in db_mysql_query (_h=0x7cf7c0, _k=<value
> optimized out>, _op=<value optimized out>, _v=0x7fffaa088fa0,
> _c=<value optimized out>,
>     _n=1, _nc=2, _o=0x0, _r=0x7fffaa089040) at dbase.c:621
> #3  0x00007f139da6547c in authorize (_m=0x7d0490, _realm=<value
> optimized out>, _table=<value optimized out>, _hftype=<value optimized
> out>)
>     at authorize.c:107
> #4  0x000000000040f849 in do_action (a=0x7bd1f8, msg=0x7d0490) at action.c:961
> #5  0x000000000040e7a5 in run_action_list (a=<value optimized out>,
> msg=0x7d0490) at action.c:139
> #6  0x0000000000470394 in eval_elem (e=0x7bd318, msg=0x7d0490,
> val=0x0) at route.c:1189
> #7  0x0000000000471b0d in eval_expr (e=0x14e0160, msg=0x7d0490,
> val=0x0) at route.c:1486
> #8  0x0000000000471abc in eval_expr (e=0x7bd3b0, msg=0x7d0490,
> val=0x0) at route.c:1502
> #9  0x0000000000471ae5 in eval_expr (e=0x7bd448, msg=0x7d0490,
> val=0x0) at route.c:1507
> #10 0x000000000040f8fc in do_action (a=0x7be468, msg=0x7d0490) at action.c:688
> #11 0x000000000040e7a5 in run_action_list (a=<value optimized out>,
> msg=0x7d0490) at action.c:139
> #12 0x000000000041159b in do_action (a=0x7be788, msg=0x7d0490) at action.c:705
> #13 0x000000000040e7a5 in run_action_list (a=<value optimized out>,
> msg=0x7d0490) at action.c:139
> #14 0x000000000041159b in do_action (a=0x7bfaa8, msg=0x7d0490) at action.c:705
> #15 0x000000000040e7a5 in run_action_list (a=<value optimized out>,
> msg=0x7d0490) at action.c:139
> #16 0x0000000000410e9f in do_action (a=0x7b80a0, msg=0x7d0490) at action.c:119
> #17 0x000000000040e7a5 in run_action_list (a=<value optimized out>,
> msg=0x7d0490) at action.c:139
> #18 0x0000000000411a1b in do_action (a=0x7b81c0, msg=0x7d0490) at action.c:711
> #19 0x000000000040e7a5 in run_action_list (a=<value optimized out>,
> msg=0x7d0490) at action.c:139
> #20 0x00000000004126a5 in run_top_route (a=0x7ace70, msg=0x7d0490) at
> action.c:119
> #21 0x000000000045f7a8 in receive_msg (
>     buf=0x74e140 "INVITE sip:[hidden email] SIP/2.0\r\nDate: Sun, 01 Mar
> 2009 09:06:10 GMT\r\nCSeq: 2 INVITE\r\nVia: SIP/2.0/UDP
> x.x.x.x:5063;branch=z9hG4bK4e6660e0-ad04-de11-8875-001302b72316;rport\r\nUser-Agent:
> E"..., len=1297, rcv_info=0x7fffaa08aba0) at receive.c:165
> #22 0x000000000049eda6 in udp_rcv_loop () at udp_server.c:449
> #23 0x00000000004298bb in main (argc=<value optimized out>,
> argv=0x7fffaa08ad88) at main.c:778
>
> ---
>
> Thanks,
> Om.
>
> On Sun, Mar 1, 2009 at 12:45 PM, Om Bikram Thapa <[hidden email]> wrote:
>  
>> Hi Bogdan,
>>
>> Solved! OpenSIPS (latest SVN) no longer crashes after MySQL restarts.
>>
>> Thanks,
>> Om.
>>
>> On Sun, Mar 1, 2009 at 11:33 AM, Om Bikram Thapa <[hidden email]> wrote:
>>    
>>> Hi Bogdan,
>>>
>>> I will update and let you know the results.
>>>
>>> Regards,
>>> Om.
>>>
>>> On Sat, Feb 28, 2009 at 3:08 AM, Bogdan-Andrei Iancu
>>> <[hidden email]> wrote:
>>>      
>>>> Hi Om,
>>>>
>>>> finally, after some struggle with the libmysqlclient, I hope the issue is
>>>> solve. Please give it another try and let me know the result.
>>>>
>>>> Thanks and regards,
>>>> Bogdan
>>>>
>>>> Om Bikram Thapa wrote:
>>>>        
>>>>> Hi Bogdan,
>>>>>
>>>>> Just updated to latest svn, but I still see crashes after MySQL
>>>>> restart. The syslog shows following 3 lines after MySQL restart and
>>>>> registration attempt:
>>>>>
>>>>> --
>>>>> Feb 19 12:17:43 proxy /usr/local/sbin/opensips[2333]:
>>>>> ERROR:db_mysql:db_mysql_do_prepared_query: mysql_stmt_execute()
>>>>> failed: (1) Unknown prepared statement handler (1) given to
>>>>> mysql_stmt_execute
>>>>> Feb 19 12:17:43 proxy /usr/local/sbin/opensips[2333]:
>>>>> ERROR:auth_db:get_ha1: failed to query database
>>>>> Feb 19 12:18:13 proxy kernel: [  105.116919] opensips[2333]: segfault
>>>>> at 4c8 ip 7f167c109283 sp 7fff8525b270 error 4 in
>>>>> libmysqlclient.so.15.0.0[7f167c0ac000+1bf000]
>>>>> --
>>>>>
>>>>> At first attempt, the registration fails with "SIP/2.0 500 Server
>>>>> Internal Error" returned by the server, and subsequent registration
>>>>> attempt crashes the server.
>>>>>
>>>>> gdb backtrace gives following details:
>>>>>
>>>>> ---
>>>>> #0  0x00007f0119426283 in mysql_stmt_result_metadata () from
>>>>> /usr/lib/libmysqlclient.so.15
>>>>> (gdb) bt
>>>>> #0  0x00007f0119426283 in mysql_stmt_result_metadata () from
>>>>> /usr/lib/libmysqlclient.so.15
>>>>> #1  0x00007f01197d8f82 in db_mysql_do_prepared_query (conn=0x78b278,
>>>>> query=<value optimized out>, v=0x7fff22576880, n=<value optimized
>>>>> out>, uv=0x0, un=0)
>>>>>    at dbase.c:368
>>>>> #2  0x00007f01197daccd in db_mysql_query (_h=0x78b278, _k=<value
>>>>> optimized out>, _op=<value optimized out>, _v=0x7fff22576880,
>>>>> _c=<value optimized out>,
>>>>>    _n=1, _nc=2, _o=0x0, _r=0x7fff22576920) at dbase.c:603
>>>>> #3  0x00007f0115f56398 in authorize (_m=0x78fe98, _realm=<value
>>>>> optimized out>, _table=<value optimized out>, _hftype=<value optimized
>>>>> out>)
>>>>>    at authorize.c:107
>>>>> #4  0x000000000040f869 in do_action (a=0x780a48, msg=0x78fe98) at
>>>>> action.c:961
>>>>> #5  0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>>> msg=0x78fe98) at action.c:139
>>>>> #6  0x000000000046f194 in eval_elem (e=0x780b18, msg=0x78fe98,
>>>>> val=0x0) at route.c:1189
>>>>> #7  0x00000000004708ed in eval_expr (e=0x27648e0, msg=0x78fe98,
>>>>> val=0x0) at route.c:1486
>>>>> #8  0x000000000047089c in eval_expr (e=0x780b60, msg=0x78fe98,
>>>>> val=0x0) at route.c:1502
>>>>> #9  0x00000000004708c5 in eval_expr (e=0x780ba8, msg=0x78fe98,
>>>>> val=0x0) at route.c:1507
>>>>> #10 0x000000000040f91c in do_action (a=0x780f40, msg=0x78fe98) at
>>>>> action.c:688
>>>>> #11 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>>> msg=0x78fe98) at action.c:139
>>>>> #12 0x00000000004114d3 in do_action (a=0x781eb0, msg=0x78fe98) at
>>>>> action.c:705
>>>>> #13 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>>> msg=0x78fe98) at action.c:139
>>>>> #14 0x0000000000410e07 in do_action (a=0x77ee88, msg=0x78fe98) at
>>>>> action.c:119
>>>>> #15 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>>> msg=0x78fe98) at action.c:139
>>>>> #16 0x00000000004114d3 in do_action (a=0x77f028, msg=0x78fe98) at
>>>>> action.c:705
>>>>> #17 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>>> msg=0x78fe98) at action.c:139
>>>>> #18 0x00000000004125fe in run_top_route (a=0x777e78, msg=0x78fe98) at
>>>>> action.c:119
>>>>> #19 0x000000000045e898 in receive_msg (
>>>>>    buf=0x749180 "REGISTER sip:x.x.x.x SIP/2.0\r\nCSeq: 58
>>>>> REGISTER\r\nVia: SIP/2.0/UDP
>>>>>
>>>>> x.x.x.x:5068;branch=z9hG4bK5ca4685b-befc-dd11-911f-0015c5404858;rport\r\nUser-Agent:
>>>>> Ekiga/2.0.12\r\nAuthorization: Dig"..., len=750,
>>>>> rcv_info=0x7fff22578090) at receive.c:165
>>>>> #20 0x000000000049d3a6 in udp_rcv_loop () at udp_server.c:449
>>>>> #21 0x00000000004291fb in main (argc=<value optimized out>,
>>>>> argv=0x7fff22578278) at main.c:778
>>>>> ---
>>>>>
>>>>> I have tried several times and each time I get the same sequence of
>>>>> results.
>>>>>
>>>>> Thanks,
>>>>> Om.
>>>>>
>>>>> On Thu, Feb 19, 2009 at 11:22 AM, Om Bikram Thapa <[hidden email]>
>>>>> wrote:
>>>>>
>>>>>          
>>>>>> Thanks Bogdan. I will update and let you know.
>>>>>>
>>>>>> Regards,
>>>>>> Om.
>>>>>>
>>>>>> On Thu, Feb 19, 2009 at 2:48 AM, Bogdan-Andrei Iancu
>>>>>> <[hidden email]> wrote:
>>>>>>
>>>>>>            
>>>>>>> Hi Om,
>>>>>>>
>>>>>>> There is a fix available on SVN - please update and see if it works now.
>>>>>>>
>>>>>>> Thanks and regards,
>>>>>>> Bogdan
>>>>>>>
>>>>>>> Om Bikram Thapa wrote:
>>>>>>>
>>>>>>>              
>>>>>>>> Hi Bogdan,
>>>>>>>>
>>>>>>>> I have submitted the bug on the tracker.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Om.
>>>>>>>>
>>>>>>>> On Tue, Feb 17, 2009 at 3:50 AM, Bogdan-Andrei Iancu
>>>>>>>> <[hidden email]> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>                
>>>>>>>>> Hi Om,
>>>>>>>>>
>>>>>>>>> Thanks for the report - I may have an idea what is the problem; I will
>>>>>>>>> try
>>>>>>>>> to reproduce the crash in the following days.
>>>>>>>>>
>>>>>>>>> Could you please open a bug report on the tracker - just not to forget
>>>>>>>>> about
>>>>>>>>> it :D (see http://www.opensips.org/index.php?n=Development.Tracker)
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Bogdan
>>>>>>>>>
>>>>>>>>> Om Bikram Thapa wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> OpenSIPS is dying with "libmysqlclient.so Segmentation Fault" after
>>>>>>>>>> MySQL server restart. The log shows
>>>>>>>>>> "opensips[16769]: segfault at 4c8 ip 7f36728bf283 sp 7fff7ba11a90
>>>>>>>>>> error 4 in libmysqlclient.so.15.0.0[7f3672862000+1bf000]"
>>>>>>>>>>
>>>>>>>>>> and gdb bt shows:
>>>>>>>>>>
>>>>>>>>>> -------------
>>>>>>>>>> #0  0x00007f36728bf283 in mysql_stmt_result_metadata () from
>>>>>>>>>> /usr/lib/libmysqlclient.so.15
>>>>>>>>>> #1  0x00007f3672c707b7 in db_mysql_do_prepared_query (conn=0x78b278,
>>>>>>>>>> query=<value optimized out>, v=0x7fff7ba11d20, n=1, uv=0x0, un=0) at
>>>>>>>>>> dbase.c:363
>>>>>>>>>> #2  0x00007f3672c72e1f in db_mysql_query (_h=0x78b278, _k=<value
>>>>>>>>>> optimized out>, _op=<value optimized out>, _v=0x7fff7ba11d20,
>>>>>>>>>> _c=<value optimized out>,
>>>>>>>>>>  _n=1, _nc=2, _o=0x0, _r=0x7fff7ba11dc0) at dbase.c:591
>>>>>>>>>> #3  0x00007f366f3f0398 in authorize (_m=0x78fe98, _realm=<value
>>>>>>>>>> optimized out>, _table=<value optimized out>, _hftype=<value
>>>>>>>>>> optimized
>>>>>>>>>> out>)
>>>>>>>>>>  at authorize.c:107
>>>>>>>>>> #4  0x000000000040f869 in do_action (a=0x780a48, msg=0x78fe98) at
>>>>>>>>>> action.c:961
>>>>>>>>>> #5  0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>>>>>>>> msg=0x78fe98) at action.c:139
>>>>>>>>>> #6  0x000000000046f194 in eval_elem (e=0x780b18, msg=0x78fe98,
>>>>>>>>>> val=0x0) at route.c:1189
>>>>>>>>>> #7  0x00000000004708ed in eval_expr (e=0x120b1e0, msg=0x78fe98,
>>>>>>>>>> val=0x0) at route.c:1486
>>>>>>>>>> #8  0x000000000047089c in eval_expr (e=0x780b60, msg=0x78fe98,
>>>>>>>>>> val=0x0) at route.c:1502
>>>>>>>>>> #9  0x00000000004708c5 in eval_expr (e=0x780ba8, msg=0x78fe98,
>>>>>>>>>> val=0x0) at route.c:1507
>>>>>>>>>> #10 0x000000000040f91c in do_action (a=0x780f40, msg=0x78fe98) at
>>>>>>>>>> action.c:688
>>>>>>>>>> #11 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>>>>>>>> msg=0x78fe98) at action.c:139
>>>>>>>>>> #12 0x00000000004114d3 in do_action (a=0x781eb0, msg=0x78fe98) at
>>>>>>>>>> action.c:705
>>>>>>>>>> #13 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>>>>>>>> msg=0x78fe98) at action.c:139
>>>>>>>>>> #14 0x0000000000410e07 in do_action (a=0x77ee88, msg=0x78fe98) at
>>>>>>>>>> action.c:119
>>>>>>>>>> #15 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>>>>>>>> msg=0x78fe98) at action.c:139
>>>>>>>>>> #16 0x00000000004114d3 in do_action (a=0x77f028, msg=0x78fe98) at
>>>>>>>>>> action.c:705
>>>>>>>>>> #17 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>>>>>>>> msg=0x78fe98) at action.c:139
>>>>>>>>>> #18 0x00000000004125fe in run_top_route (a=0x777e78, msg=0x78fe98) at
>>>>>>>>>> action.c:119
>>>>>>>>>> #19 0x000000000045e898 in receive_msg (
>>>>>>>>>>  buf=0x749180 "REGISTER sip:xxxxx SIP/2.0\r\nCSeq: 2
>>>>>>>>>> REGISTER\r\nVia: SIP/2.0/UDP
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> x.x.x.x:5061;branch=z9hG4bK60f1b2dd-57fa-dd11-9401-0015c5404858;rport\r\nUser-Agent:
>>>>>>>>>> Ekiga/2.0.12\r\nAuthorization: Dige"..., len=749,
>>>>>>>>>> rcv_info=0x7fff7ba13530) at receive.c:165
>>>>>>>>>> #20 0x000000000049d3a6 in udp_rcv_loop () at udp_server.c:449
>>>>>>>>>> #21 0x00000000004291fb in main (argc=<value optimized out>,
>>>>>>>>>> argv=0x7fff7ba13718) at main.c:778
>>>>>>>>>> ------------
>>>>>>>>>>
>>>>>>>>>> Server runs fine after restart until MySQL is restarted again. In my
>>>>>>>>>> lab setup, OpenSIPS is getting killed triggered by registration
>>>>>>>>>> request every morning (probably due to MySQL being restarted daily
>>>>>>>>>> with logrotate).
>>>>>>>>>>
>>>>>>>>>> The server is the latest trunk on Debian lenny/AMD64.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Om.
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Users mailing list
>>>>>>>>>> [hidden email]
>>>>>>>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                    
>>>>>>>>>                  
>>>>>>>>                
>>>>>>>              
>>>>>          
>>>>        
>
>  


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: OpenSIPS libmysqlclient.so segfault after MySQL restart

Bogdan-Andrei Iancu
Hi Om,

Thanks again for your help in fixing this. With the log you sent me I
figured out what was wrong - when a reconnect even was detected,
previously, I was re-initializing only the current statements. But on a
mysql connection you can have multiple statements, so you need to reset
all of them.

Also the re-connect even may be detected when running a standard text
query....

I made the necessary fixes on SVN - hopefully it should be ok now. :)

Regards,
Bogdan

Om Bikram Thapa wrote:

> Hi Bogdan,
>
> I am again able to crash opensips by restarting MySQL two times with
> 2/3 register/de-register (it was okay till then) and then crashed with
> INVITE.
>
> Here is the gdb bt:
>
> ---
> Core was generated by `/usr/local/sbin/opensips -P
> /var/run/opensips/opensips.pid -m 64 -u root -g roo'.
> Program terminated with signal 11, Segmentation fault.
> [New process 8102]
> #0  0x00007f3f536b2283 in mysql_stmt_result_metadata () from
> /usr/lib/libmysqlclient.so.15
> (gdb) bt
> #0  0x00007f3f536b2283 in mysql_stmt_result_metadata () from
> /usr/lib/libmysqlclient.so.15
> #1  0x00007f3f53a644ba in db_mysql_do_prepared_query (conn=0x7cf7c0,
> query=<value optimized out>, v=0x7fff5c803720, n=<value optimized
> out>, uv=0x0, un=0)
>     at dbase.c:370
> #2  0x00007f3f53a66b41 in db_mysql_query (_h=0x7cf7c0, _k=<value
> optimized out>, _op=<value optimized out>, _v=0x7fff5c803720,
> _c=<value optimized out>,
>     _n=1, _nc=2, _o=0x0, _r=0x7fff5c8037c0) at dbase.c:621
> #3  0x00007f3f501df47c in authorize (_m=0x7d0490, _realm=<value
> optimized out>, _table=<value optimized out>, _hftype=<value optimized
> out>)
>     at authorize.c:107
> #4  0x000000000040f849 in do_action (a=0x7bd1f8, msg=0x7d0490) at action.c:961
> #5  0x000000000040e7a5 in run_action_list (a=<value optimized out>,
> msg=0x7d0490) at action.c:139
> #6  0x0000000000470394 in eval_elem (e=0x7bd318, msg=0x7d0490,
> val=0x0) at route.c:1189
> #7  0x0000000000471b0d in eval_expr (e=0x92eff0, msg=0x7d0490,
> val=0x0) at route.c:1486
> #8  0x0000000000471abc in eval_expr (e=0x7bd3b0, msg=0x7d0490,
> val=0x0) at route.c:1502
> #9  0x0000000000471ae5 in eval_expr (e=0x7bd448, msg=0x7d0490,
> val=0x0) at route.c:1507
> #10 0x000000000040f8fc in do_action (a=0x7be468, msg=0x7d0490) at action.c:688
> #11 0x000000000040e7a5 in run_action_list (a=<value optimized out>,
> msg=0x7d0490) at action.c:139
> #12 0x000000000041159b in do_action (a=0x7be788, msg=0x7d0490) at action.c:705
> #13 0x000000000040e7a5 in run_action_list (a=<value optimized out>,
> msg=0x7d0490) at action.c:139
> #14 0x000000000041159b in do_action (a=0x7bfaa8, msg=0x7d0490) at action.c:705
> #15 0x000000000040e7a5 in run_action_list (a=<value optimized out>,
> msg=0x7d0490) at action.c:139
> #16 0x0000000000410e9f in do_action (a=0x7b80a0, msg=0x7d0490) at action.c:119
> #17 0x000000000040e7a5 in run_action_list (a=<value optimized out>,
> msg=0x7d0490) at action.c:139
> #18 0x0000000000411a1b in do_action (a=0x7b81c0, msg=0x7d0490) at action.c:711
> #19 0x000000000040e7a5 in run_action_list (a=<value optimized out>,
> msg=0x7d0490) at action.c:139
> #20 0x00000000004126a5 in run_top_route (a=0x7ace70, msg=0x7d0490) at
> action.c:119
> #21 0x000000000045f7a8 in receive_msg (
>     buf=0x74e140 "INVITE
> sip:[hidden email];user=phone;transport=UDP SIP/2.0\r\nVia:
> SIP/2.0/UDP 202.51.83.195:5060;branch=z9hG4bKilgvm1fi6nm39fimrfo222j;rport\r\nFrom:
> <sip:[hidden email]>;tag=dlidns7ti5hc7e2j"..., len=1399,
> rcv_info=0x7fff5c805320) at receive.c:165
> #22 0x000000000049eda6 in udp_rcv_loop () at udp_server.c:449
> #23 0x00000000004298bb in main (argc=<value optimized out>,
> argv=0x7fff5c805508) at main.c:778
> ---
>
> And debug=6 log file is attached herewith. FYI, the crash occured around 18:33.
>
> Thank you very much for your time.
>
> Regards,
> Om.
>
> On Mon, Mar 2, 2009 at 3:38 PM, Bogdan-Andrei Iancu
> <[hidden email]> wrote:
>  
>> Hi Om,
>>
>> Can you send me the full debug log (debug=6) for this scenario (you can send
>> it privately, if too big).
>>
>> I just want to check exactly what is going one.
>>
>> Thanks for your help,
>> Bogdan
>>
>> Om Bikram Thapa wrote:
>>    
>>> Hi Bogdan,
>>>
>>> Seems like I replied too soon. OpenSIPS again crashed on INVITE,
>>> although REGISTER operation was fine after a couple of MySQL restarts.
>>> I again restarted MySQL and got the server crashing on REGISTER as
>>> well. gdb trace for both the event crashes point to the same code:
>>>
>>> ---
>>> Core was generated by `/usr/local/sbin/opensips -P
>>> /var/run/opensips/opensips.pid -m 64 -u root -g roo'.
>>> Program terminated with signal 11, Segmentation fault.
>>> [New process 7352]
>>> #0  0x00007f13a0f38283 in mysql_stmt_result_metadata () from
>>> /usr/lib/libmysqlclient.so.15
>>> (gdb) bt
>>> #0  0x00007f13a0f38283 in mysql_stmt_result_metadata () from
>>> /usr/lib/libmysqlclient.so.15
>>> #1  0x00007f13a12ea4ba in db_mysql_do_prepared_query (conn=0x7cf7c0,
>>> query=<value optimized out>, v=0x7fffaa088fa0, n=<value optimized
>>> out>, uv=0x0, un=0)
>>>    at dbase.c:370
>>> #2  0x00007f13a12ecb41 in db_mysql_query (_h=0x7cf7c0, _k=<value
>>> optimized out>, _op=<value optimized out>, _v=0x7fffaa088fa0,
>>> _c=<value optimized out>,
>>>    _n=1, _nc=2, _o=0x0, _r=0x7fffaa089040) at dbase.c:621
>>> #3  0x00007f139da6547c in authorize (_m=0x7d0490, _realm=<value
>>> optimized out>, _table=<value optimized out>, _hftype=<value optimized
>>> out>)
>>>    at authorize.c:107
>>> #4  0x000000000040f849 in do_action (a=0x7bd1f8, msg=0x7d0490) at
>>> action.c:961
>>> #5  0x000000000040e7a5 in run_action_list (a=<value optimized out>,
>>> msg=0x7d0490) at action.c:139
>>> #6  0x0000000000470394 in eval_elem (e=0x7bd318, msg=0x7d0490,
>>> val=0x0) at route.c:1189
>>> #7  0x0000000000471b0d in eval_expr (e=0x14e0160, msg=0x7d0490,
>>> val=0x0) at route.c:1486
>>> #8  0x0000000000471abc in eval_expr (e=0x7bd3b0, msg=0x7d0490,
>>> val=0x0) at route.c:1502
>>> #9  0x0000000000471ae5 in eval_expr (e=0x7bd448, msg=0x7d0490,
>>> val=0x0) at route.c:1507
>>> #10 0x000000000040f8fc in do_action (a=0x7be468, msg=0x7d0490) at
>>> action.c:688
>>> #11 0x000000000040e7a5 in run_action_list (a=<value optimized out>,
>>> msg=0x7d0490) at action.c:139
>>> #12 0x000000000041159b in do_action (a=0x7be788, msg=0x7d0490) at
>>> action.c:705
>>> #13 0x000000000040e7a5 in run_action_list (a=<value optimized out>,
>>> msg=0x7d0490) at action.c:139
>>> #14 0x000000000041159b in do_action (a=0x7bfaa8, msg=0x7d0490) at
>>> action.c:705
>>> #15 0x000000000040e7a5 in run_action_list (a=<value optimized out>,
>>> msg=0x7d0490) at action.c:139
>>> #16 0x0000000000410e9f in do_action (a=0x7b80a0, msg=0x7d0490) at
>>> action.c:119
>>> #17 0x000000000040e7a5 in run_action_list (a=<value optimized out>,
>>> msg=0x7d0490) at action.c:139
>>> #18 0x0000000000411a1b in do_action (a=0x7b81c0, msg=0x7d0490) at
>>> action.c:711
>>> #19 0x000000000040e7a5 in run_action_list (a=<value optimized out>,
>>> msg=0x7d0490) at action.c:139
>>> #20 0x00000000004126a5 in run_top_route (a=0x7ace70, msg=0x7d0490) at
>>> action.c:119
>>> #21 0x000000000045f7a8 in receive_msg (
>>>    buf=0x74e140 "INVITE sip:[hidden email] SIP/2.0\r\nDate: Sun, 01 Mar
>>> 2009 09:06:10 GMT\r\nCSeq: 2 INVITE\r\nVia: SIP/2.0/UDP
>>>
>>> x.x.x.x:5063;branch=z9hG4bK4e6660e0-ad04-de11-8875-001302b72316;rport\r\nUser-Agent:
>>> E"..., len=1297, rcv_info=0x7fffaa08aba0) at receive.c:165
>>> #22 0x000000000049eda6 in udp_rcv_loop () at udp_server.c:449
>>> #23 0x00000000004298bb in main (argc=<value optimized out>,
>>> argv=0x7fffaa08ad88) at main.c:778
>>>
>>> ---
>>>
>>> Thanks,
>>> Om.
>>>
>>> On Sun, Mar 1, 2009 at 12:45 PM, Om Bikram Thapa <[hidden email]>
>>> wrote:
>>>
>>>      
>>>> Hi Bogdan,
>>>>
>>>> Solved! OpenSIPS (latest SVN) no longer crashes after MySQL restarts.
>>>>
>>>> Thanks,
>>>> Om.
>>>>
>>>> On Sun, Mar 1, 2009 at 11:33 AM, Om Bikram Thapa <[hidden email]>
>>>> wrote:
>>>>
>>>>        
>>>>> Hi Bogdan,
>>>>>
>>>>> I will update and let you know the results.
>>>>>
>>>>> Regards,
>>>>> Om.
>>>>>
>>>>> On Sat, Feb 28, 2009 at 3:08 AM, Bogdan-Andrei Iancu
>>>>> <[hidden email]> wrote:
>>>>>
>>>>>          
>>>>>> Hi Om,
>>>>>>
>>>>>> finally, after some struggle with the libmysqlclient, I hope the issue
>>>>>> is
>>>>>> solve. Please give it another try and let me know the result.
>>>>>>
>>>>>> Thanks and regards,
>>>>>> Bogdan
>>>>>>
>>>>>> Om Bikram Thapa wrote:
>>>>>>
>>>>>>            
>>>>>>> Hi Bogdan,
>>>>>>>
>>>>>>> Just updated to latest svn, but I still see crashes after MySQL
>>>>>>> restart. The syslog shows following 3 lines after MySQL restart and
>>>>>>> registration attempt:
>>>>>>>
>>>>>>> --
>>>>>>> Feb 19 12:17:43 proxy /usr/local/sbin/opensips[2333]:
>>>>>>> ERROR:db_mysql:db_mysql_do_prepared_query: mysql_stmt_execute()
>>>>>>> failed: (1) Unknown prepared statement handler (1) given to
>>>>>>> mysql_stmt_execute
>>>>>>> Feb 19 12:17:43 proxy /usr/local/sbin/opensips[2333]:
>>>>>>> ERROR:auth_db:get_ha1: failed to query database
>>>>>>> Feb 19 12:18:13 proxy kernel: [  105.116919] opensips[2333]: segfault
>>>>>>> at 4c8 ip 7f167c109283 sp 7fff8525b270 error 4 in
>>>>>>> libmysqlclient.so.15.0.0[7f167c0ac000+1bf000]
>>>>>>> --
>>>>>>>
>>>>>>> At first attempt, the registration fails with "SIP/2.0 500 Server
>>>>>>> Internal Error" returned by the server, and subsequent registration
>>>>>>> attempt crashes the server.
>>>>>>>
>>>>>>> gdb backtrace gives following details:
>>>>>>>
>>>>>>> ---
>>>>>>> #0  0x00007f0119426283 in mysql_stmt_result_metadata () from
>>>>>>> /usr/lib/libmysqlclient.so.15
>>>>>>> (gdb) bt
>>>>>>> #0  0x00007f0119426283 in mysql_stmt_result_metadata () from
>>>>>>> /usr/lib/libmysqlclient.so.15
>>>>>>> #1  0x00007f01197d8f82 in db_mysql_do_prepared_query (conn=0x78b278,
>>>>>>> query=<value optimized out>, v=0x7fff22576880, n=<value optimized
>>>>>>> out>, uv=0x0, un=0)
>>>>>>>   at dbase.c:368
>>>>>>> #2  0x00007f01197daccd in db_mysql_query (_h=0x78b278, _k=<value
>>>>>>> optimized out>, _op=<value optimized out>, _v=0x7fff22576880,
>>>>>>> _c=<value optimized out>,
>>>>>>>   _n=1, _nc=2, _o=0x0, _r=0x7fff22576920) at dbase.c:603
>>>>>>> #3  0x00007f0115f56398 in authorize (_m=0x78fe98, _realm=<value
>>>>>>> optimized out>, _table=<value optimized out>, _hftype=<value optimized
>>>>>>> out>)
>>>>>>>   at authorize.c:107
>>>>>>> #4  0x000000000040f869 in do_action (a=0x780a48, msg=0x78fe98) at
>>>>>>> action.c:961
>>>>>>> #5  0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>>>>> msg=0x78fe98) at action.c:139
>>>>>>> #6  0x000000000046f194 in eval_elem (e=0x780b18, msg=0x78fe98,
>>>>>>> val=0x0) at route.c:1189
>>>>>>> #7  0x00000000004708ed in eval_expr (e=0x27648e0, msg=0x78fe98,
>>>>>>> val=0x0) at route.c:1486
>>>>>>> #8  0x000000000047089c in eval_expr (e=0x780b60, msg=0x78fe98,
>>>>>>> val=0x0) at route.c:1502
>>>>>>> #9  0x00000000004708c5 in eval_expr (e=0x780ba8, msg=0x78fe98,
>>>>>>> val=0x0) at route.c:1507
>>>>>>> #10 0x000000000040f91c in do_action (a=0x780f40, msg=0x78fe98) at
>>>>>>> action.c:688
>>>>>>> #11 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>>>>> msg=0x78fe98) at action.c:139
>>>>>>> #12 0x00000000004114d3 in do_action (a=0x781eb0, msg=0x78fe98) at
>>>>>>> action.c:705
>>>>>>> #13 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>>>>> msg=0x78fe98) at action.c:139
>>>>>>> #14 0x0000000000410e07 in do_action (a=0x77ee88, msg=0x78fe98) at
>>>>>>> action.c:119
>>>>>>> #15 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>>>>> msg=0x78fe98) at action.c:139
>>>>>>> #16 0x00000000004114d3 in do_action (a=0x77f028, msg=0x78fe98) at
>>>>>>> action.c:705
>>>>>>> #17 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>>>>> msg=0x78fe98) at action.c:139
>>>>>>> #18 0x00000000004125fe in run_top_route (a=0x777e78, msg=0x78fe98) at
>>>>>>> action.c:119
>>>>>>> #19 0x000000000045e898 in receive_msg (
>>>>>>>   buf=0x749180 "REGISTER sip:x.x.x.x SIP/2.0\r\nCSeq: 58
>>>>>>> REGISTER\r\nVia: SIP/2.0/UDP
>>>>>>>
>>>>>>>
>>>>>>> x.x.x.x:5068;branch=z9hG4bK5ca4685b-befc-dd11-911f-0015c5404858;rport\r\nUser-Agent:
>>>>>>> Ekiga/2.0.12\r\nAuthorization: Dig"..., len=750,
>>>>>>> rcv_info=0x7fff22578090) at receive.c:165
>>>>>>> #20 0x000000000049d3a6 in udp_rcv_loop () at udp_server.c:449
>>>>>>> #21 0x00000000004291fb in main (argc=<value optimized out>,
>>>>>>> argv=0x7fff22578278) at main.c:778
>>>>>>> ---
>>>>>>>
>>>>>>> I have tried several times and each time I get the same sequence of
>>>>>>> results.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Om.
>>>>>>>
>>>>>>> On Thu, Feb 19, 2009 at 11:22 AM, Om Bikram Thapa <[hidden email]>
>>>>>>> wrote:
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>>>> Thanks Bogdan. I will update and let you know.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Om.
>>>>>>>>
>>>>>>>> On Thu, Feb 19, 2009 at 2:48 AM, Bogdan-Andrei Iancu
>>>>>>>> <[hidden email]> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>                
>>>>>>>>> Hi Om,
>>>>>>>>>
>>>>>>>>> There is a fix available on SVN - please update and see if it works
>>>>>>>>> now.
>>>>>>>>>
>>>>>>>>> Thanks and regards,
>>>>>>>>> Bogdan
>>>>>>>>>
>>>>>>>>> Om Bikram Thapa wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>> Hi Bogdan,
>>>>>>>>>>
>>>>>>>>>> I have submitted the bug on the tracker.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Om.
>>>>>>>>>>
>>>>>>>>>> On Tue, Feb 17, 2009 at 3:50 AM, Bogdan-Andrei Iancu
>>>>>>>>>> <[hidden email]> wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                    
>>>>>>>>>>> Hi Om,
>>>>>>>>>>>
>>>>>>>>>>> Thanks for the report - I may have an idea what is the problem; I
>>>>>>>>>>> will
>>>>>>>>>>> try
>>>>>>>>>>> to reproduce the crash in the following days.
>>>>>>>>>>>
>>>>>>>>>>> Could you please open a bug report on the tracker - just not to
>>>>>>>>>>> forget
>>>>>>>>>>> about
>>>>>>>>>>> it :D (see
>>>>>>>>>>> http://www.opensips.org/index.php?n=Development.Tracker)
>>>>>>>>>>>
>>>>>>>>>>> Regards,
>>>>>>>>>>> Bogdan
>>>>>>>>>>>
>>>>>>>>>>> Om Bikram Thapa wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                      
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> OpenSIPS is dying with "libmysqlclient.so Segmentation Fault"
>>>>>>>>>>>> after
>>>>>>>>>>>> MySQL server restart. The log shows
>>>>>>>>>>>> "opensips[16769]: segfault at 4c8 ip 7f36728bf283 sp 7fff7ba11a90
>>>>>>>>>>>> error 4 in libmysqlclient.so.15.0.0[7f3672862000+1bf000]"
>>>>>>>>>>>>
>>>>>>>>>>>> and gdb bt shows:
>>>>>>>>>>>>
>>>>>>>>>>>> -------------
>>>>>>>>>>>> #0  0x00007f36728bf283 in mysql_stmt_result_metadata () from
>>>>>>>>>>>> /usr/lib/libmysqlclient.so.15
>>>>>>>>>>>> #1  0x00007f3672c707b7 in db_mysql_do_prepared_query
>>>>>>>>>>>> (conn=0x78b278,
>>>>>>>>>>>> query=<value optimized out>, v=0x7fff7ba11d20, n=1, uv=0x0, un=0)
>>>>>>>>>>>> at
>>>>>>>>>>>> dbase.c:363
>>>>>>>>>>>> #2  0x00007f3672c72e1f in db_mysql_query (_h=0x78b278, _k=<value
>>>>>>>>>>>> optimized out>, _op=<value optimized out>, _v=0x7fff7ba11d20,
>>>>>>>>>>>> _c=<value optimized out>,
>>>>>>>>>>>>  _n=1, _nc=2, _o=0x0, _r=0x7fff7ba11dc0) at dbase.c:591
>>>>>>>>>>>> #3  0x00007f366f3f0398 in authorize (_m=0x78fe98, _realm=<value
>>>>>>>>>>>> optimized out>, _table=<value optimized out>, _hftype=<value
>>>>>>>>>>>> optimized
>>>>>>>>>>>> out>)
>>>>>>>>>>>>  at authorize.c:107
>>>>>>>>>>>> #4  0x000000000040f869 in do_action (a=0x780a48, msg=0x78fe98) at
>>>>>>>>>>>> action.c:961
>>>>>>>>>>>> #5  0x000000000040e7e5 in run_action_list (a=<value optimized
>>>>>>>>>>>> out>,
>>>>>>>>>>>> msg=0x78fe98) at action.c:139
>>>>>>>>>>>> #6  0x000000000046f194 in eval_elem (e=0x780b18, msg=0x78fe98,
>>>>>>>>>>>> val=0x0) at route.c:1189
>>>>>>>>>>>> #7  0x00000000004708ed in eval_expr (e=0x120b1e0, msg=0x78fe98,
>>>>>>>>>>>> val=0x0) at route.c:1486
>>>>>>>>>>>> #8  0x000000000047089c in eval_expr (e=0x780b60, msg=0x78fe98,
>>>>>>>>>>>> val=0x0) at route.c:1502
>>>>>>>>>>>> #9  0x00000000004708c5 in eval_expr (e=0x780ba8, msg=0x78fe98,
>>>>>>>>>>>> val=0x0) at route.c:1507
>>>>>>>>>>>> #10 0x000000000040f91c in do_action (a=0x780f40, msg=0x78fe98) at
>>>>>>>>>>>> action.c:688
>>>>>>>>>>>> #11 0x000000000040e7e5 in run_action_list (a=<value optimized
>>>>>>>>>>>> out>,
>>>>>>>>>>>> msg=0x78fe98) at action.c:139
>>>>>>>>>>>> #12 0x00000000004114d3 in do_action (a=0x781eb0, msg=0x78fe98) at
>>>>>>>>>>>> action.c:705
>>>>>>>>>>>> #13 0x000000000040e7e5 in run_action_list (a=<value optimized
>>>>>>>>>>>> out>,
>>>>>>>>>>>> msg=0x78fe98) at action.c:139
>>>>>>>>>>>> #14 0x0000000000410e07 in do_action (a=0x77ee88, msg=0x78fe98) at
>>>>>>>>>>>> action.c:119
>>>>>>>>>>>> #15 0x000000000040e7e5 in run_action_list (a=<value optimized
>>>>>>>>>>>> out>,
>>>>>>>>>>>> msg=0x78fe98) at action.c:139
>>>>>>>>>>>> #16 0x00000000004114d3 in do_action (a=0x77f028, msg=0x78fe98) at
>>>>>>>>>>>> action.c:705
>>>>>>>>>>>> #17 0x000000000040e7e5 in run_action_list (a=<value optimized
>>>>>>>>>>>> out>,
>>>>>>>>>>>> msg=0x78fe98) at action.c:139
>>>>>>>>>>>> #18 0x00000000004125fe in run_top_route (a=0x777e78,
>>>>>>>>>>>> msg=0x78fe98) at
>>>>>>>>>>>> action.c:119
>>>>>>>>>>>> #19 0x000000000045e898 in receive_msg (
>>>>>>>>>>>>  buf=0x749180 "REGISTER sip:xxxxx SIP/2.0\r\nCSeq: 2
>>>>>>>>>>>> REGISTER\r\nVia: SIP/2.0/UDP
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> x.x.x.x:5061;branch=z9hG4bK60f1b2dd-57fa-dd11-9401-0015c5404858;rport\r\nUser-Agent:
>>>>>>>>>>>> Ekiga/2.0.12\r\nAuthorization: Dige"..., len=749,
>>>>>>>>>>>> rcv_info=0x7fff7ba13530) at receive.c:165
>>>>>>>>>>>> #20 0x000000000049d3a6 in udp_rcv_loop () at udp_server.c:449
>>>>>>>>>>>> #21 0x00000000004291fb in main (argc=<value optimized out>,
>>>>>>>>>>>> argv=0x7fff7ba13718) at main.c:778
>>>>>>>>>>>> ------------
>>>>>>>>>>>>
>>>>>>>>>>>> Server runs fine after restart until MySQL is restarted again. In
>>>>>>>>>>>> my
>>>>>>>>>>>> lab setup, OpenSIPS is getting killed triggered by registration
>>>>>>>>>>>> request every morning (probably due to MySQL being restarted
>>>>>>>>>>>> daily
>>>>>>>>>>>> with logrotate).
>>>>>>>>>>>>
>>>>>>>>>>>> The server is the latest trunk on Debian lenny/AMD64.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Om.
>>>>>>>>>>>>
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> Users mailing list
>>>>>>>>>>>> [hidden email]
>>>>>>>>>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                        
>>>>>>>>>>>                      
>>>>>>>>>>                    
>>>>>>>>>                  
>>>>>>>              
>>>>>>            
>>>      
>>    


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: OpenSIPS libmysqlclient.so segfault after MySQL restart

Om Bikram Thapa
In reply to this post by Om Bikram Thapa
Hi Bogdan,

This SVN has definitely fixed the bug!

Thank you very much.

Best regards,
Om.

On Mon, Feb 16, 2009 at 11:40 AM, Om Bikram Thapa <[hidden email]> wrote:

> Hi,
>
> OpenSIPS is dying with "libmysqlclient.so Segmentation Fault" after
> MySQL server restart. The log shows
> "opensips[16769]: segfault at 4c8 ip 7f36728bf283 sp 7fff7ba11a90
> error 4 in libmysqlclient.so.15.0.0[7f3672862000+1bf000]"
>
> and gdb bt shows:
>
> -------------
> #0  0x00007f36728bf283 in mysql_stmt_result_metadata () from
> /usr/lib/libmysqlclient.so.15
> #1  0x00007f3672c707b7 in db_mysql_do_prepared_query (conn=0x78b278,
> query=<value optimized out>, v=0x7fff7ba11d20, n=1, uv=0x0, un=0) at
> dbase.c:363
> #2  0x00007f3672c72e1f in db_mysql_query (_h=0x78b278, _k=<value
> optimized out>, _op=<value optimized out>, _v=0x7fff7ba11d20,
> _c=<value optimized out>,
>    _n=1, _nc=2, _o=0x0, _r=0x7fff7ba11dc0) at dbase.c:591
> #3  0x00007f366f3f0398 in authorize (_m=0x78fe98, _realm=<value
> optimized out>, _table=<value optimized out>, _hftype=<value optimized
> out>)
>    at authorize.c:107
> #4  0x000000000040f869 in do_action (a=0x780a48, msg=0x78fe98) at action.c:961
> #5  0x000000000040e7e5 in run_action_list (a=<value optimized out>,
> msg=0x78fe98) at action.c:139
> #6  0x000000000046f194 in eval_elem (e=0x780b18, msg=0x78fe98,
> val=0x0) at route.c:1189
> #7  0x00000000004708ed in eval_expr (e=0x120b1e0, msg=0x78fe98,
> val=0x0) at route.c:1486
> #8  0x000000000047089c in eval_expr (e=0x780b60, msg=0x78fe98,
> val=0x0) at route.c:1502
> #9  0x00000000004708c5 in eval_expr (e=0x780ba8, msg=0x78fe98,
> val=0x0) at route.c:1507
> #10 0x000000000040f91c in do_action (a=0x780f40, msg=0x78fe98) at action.c:688
> #11 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
> msg=0x78fe98) at action.c:139
> #12 0x00000000004114d3 in do_action (a=0x781eb0, msg=0x78fe98) at action.c:705
> #13 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
> msg=0x78fe98) at action.c:139
> #14 0x0000000000410e07 in do_action (a=0x77ee88, msg=0x78fe98) at action.c:119
> #15 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
> msg=0x78fe98) at action.c:139
> #16 0x00000000004114d3 in do_action (a=0x77f028, msg=0x78fe98) at action.c:705
> #17 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
> msg=0x78fe98) at action.c:139
> #18 0x00000000004125fe in run_top_route (a=0x777e78, msg=0x78fe98) at
> action.c:119
> #19 0x000000000045e898 in receive_msg (
>    buf=0x749180 "REGISTER sip:xxxxx SIP/2.0\r\nCSeq: 2
> REGISTER\r\nVia: SIP/2.0/UDP
> x.x.x.x:5061;branch=z9hG4bK60f1b2dd-57fa-dd11-9401-0015c5404858;rport\r\nUser-Agent:
> Ekiga/2.0.12\r\nAuthorization: Dige"..., len=749,
> rcv_info=0x7fff7ba13530) at receive.c:165
> #20 0x000000000049d3a6 in udp_rcv_loop () at udp_server.c:449
> #21 0x00000000004291fb in main (argc=<value optimized out>,
> argv=0x7fff7ba13718) at main.c:778
> ------------
>
> Server runs fine after restart until MySQL is restarted again. In my
> lab setup, OpenSIPS is getting killed triggered by registration
> request every morning (probably due to MySQL being restarted daily
> with logrotate).
>
> The server is the latest trunk on Debian lenny/AMD64.
>
> Thanks,
> Om.
>

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: OpenSIPS libmysqlclient.so segfault after MySQL restart

Om Bikram Thapa
In reply to this post by Bogdan-Andrei Iancu
Hi Bogdan,

The latest SVN has fixed the libmysqlclient segmentation fault issue.

Thank you.
Om.

On Tue, Mar 3, 2009 at 10:21 PM, Bogdan-Andrei Iancu
<[hidden email]> wrote:

> Hi Om,
>
> Thanks again for your help in fixing this. With the log you sent me I
> figured out what was wrong - when a reconnect even was detected, previously,
> I was re-initializing only the current statements. But on a mysql connection
> you can have multiple statements, so you need to reset all of them.
>
> Also the re-connect even may be detected when running a standard text
> query....
>
> I made the necessary fixes on SVN - hopefully it should be ok now. :)
>
> Regards,
> Bogdan
>
> Om Bikram Thapa wrote:
>>
>> Hi Bogdan,
>>
>> I am again able to crash opensips by restarting MySQL two times with
>> 2/3 register/de-register (it was okay till then) and then crashed with
>> INVITE.
>>
>> Here is the gdb bt:
>>
>> ---
>> Core was generated by `/usr/local/sbin/opensips -P
>> /var/run/opensips/opensips.pid -m 64 -u root -g roo'.
>> Program terminated with signal 11, Segmentation fault.
>> [New process 8102]
>> #0  0x00007f3f536b2283 in mysql_stmt_result_metadata () from
>> /usr/lib/libmysqlclient.so.15
>> (gdb) bt
>> #0  0x00007f3f536b2283 in mysql_stmt_result_metadata () from
>> /usr/lib/libmysqlclient.so.15
>> #1  0x00007f3f53a644ba in db_mysql_do_prepared_query (conn=0x7cf7c0,
>> query=<value optimized out>, v=0x7fff5c803720, n=<value optimized
>> out>, uv=0x0, un=0)
>>    at dbase.c:370
>> #2  0x00007f3f53a66b41 in db_mysql_query (_h=0x7cf7c0, _k=<value
>> optimized out>, _op=<value optimized out>, _v=0x7fff5c803720,
>> _c=<value optimized out>,
>>    _n=1, _nc=2, _o=0x0, _r=0x7fff5c8037c0) at dbase.c:621
>> #3  0x00007f3f501df47c in authorize (_m=0x7d0490, _realm=<value
>> optimized out>, _table=<value optimized out>, _hftype=<value optimized
>> out>)
>>    at authorize.c:107
>> #4  0x000000000040f849 in do_action (a=0x7bd1f8, msg=0x7d0490) at
>> action.c:961
>> #5  0x000000000040e7a5 in run_action_list (a=<value optimized out>,
>> msg=0x7d0490) at action.c:139
>> #6  0x0000000000470394 in eval_elem (e=0x7bd318, msg=0x7d0490,
>> val=0x0) at route.c:1189
>> #7  0x0000000000471b0d in eval_expr (e=0x92eff0, msg=0x7d0490,
>> val=0x0) at route.c:1486
>> #8  0x0000000000471abc in eval_expr (e=0x7bd3b0, msg=0x7d0490,
>> val=0x0) at route.c:1502
>> #9  0x0000000000471ae5 in eval_expr (e=0x7bd448, msg=0x7d0490,
>> val=0x0) at route.c:1507
>> #10 0x000000000040f8fc in do_action (a=0x7be468, msg=0x7d0490) at
>> action.c:688
>> #11 0x000000000040e7a5 in run_action_list (a=<value optimized out>,
>> msg=0x7d0490) at action.c:139
>> #12 0x000000000041159b in do_action (a=0x7be788, msg=0x7d0490) at
>> action.c:705
>> #13 0x000000000040e7a5 in run_action_list (a=<value optimized out>,
>> msg=0x7d0490) at action.c:139
>> #14 0x000000000041159b in do_action (a=0x7bfaa8, msg=0x7d0490) at
>> action.c:705
>> #15 0x000000000040e7a5 in run_action_list (a=<value optimized out>,
>> msg=0x7d0490) at action.c:139
>> #16 0x0000000000410e9f in do_action (a=0x7b80a0, msg=0x7d0490) at
>> action.c:119
>> #17 0x000000000040e7a5 in run_action_list (a=<value optimized out>,
>> msg=0x7d0490) at action.c:139
>> #18 0x0000000000411a1b in do_action (a=0x7b81c0, msg=0x7d0490) at
>> action.c:711
>> #19 0x000000000040e7a5 in run_action_list (a=<value optimized out>,
>> msg=0x7d0490) at action.c:139
>> #20 0x00000000004126a5 in run_top_route (a=0x7ace70, msg=0x7d0490) at
>> action.c:119
>> #21 0x000000000045f7a8 in receive_msg (
>>    buf=0x74e140 "INVITE
>> sip:[hidden email];user=phone;transport=UDP SIP/2.0\r\nVia:
>> SIP/2.0/UDP
>> 202.51.83.195:5060;branch=z9hG4bKilgvm1fi6nm39fimrfo222j;rport\r\nFrom:
>> <sip:[hidden email]>;tag=dlidns7ti5hc7e2j"..., len=1399,
>> rcv_info=0x7fff5c805320) at receive.c:165
>> #22 0x000000000049eda6 in udp_rcv_loop () at udp_server.c:449
>> #23 0x00000000004298bb in main (argc=<value optimized out>,
>> argv=0x7fff5c805508) at main.c:778
>> ---
>>
>> And debug=6 log file is attached herewith. FYI, the crash occured around
>> 18:33.
>>
>> Thank you very much for your time.
>>
>> Regards,
>> Om.
>>
>> On Mon, Mar 2, 2009 at 3:38 PM, Bogdan-Andrei Iancu
>> <[hidden email]> wrote:
>>
>>>
>>> Hi Om,
>>>
>>> Can you send me the full debug log (debug=6) for this scenario (you can
>>> send
>>> it privately, if too big).
>>>
>>> I just want to check exactly what is going one.
>>>
>>> Thanks for your help,
>>> Bogdan
>>>
>>> Om Bikram Thapa wrote:
>>>
>>>>
>>>> Hi Bogdan,
>>>>
>>>> Seems like I replied too soon. OpenSIPS again crashed on INVITE,
>>>> although REGISTER operation was fine after a couple of MySQL restarts.
>>>> I again restarted MySQL and got the server crashing on REGISTER as
>>>> well. gdb trace for both the event crashes point to the same code:
>>>>
>>>> ---
>>>> Core was generated by `/usr/local/sbin/opensips -P
>>>> /var/run/opensips/opensips.pid -m 64 -u root -g roo'.
>>>> Program terminated with signal 11, Segmentation fault.
>>>> [New process 7352]
>>>> #0  0x00007f13a0f38283 in mysql_stmt_result_metadata () from
>>>> /usr/lib/libmysqlclient.so.15
>>>> (gdb) bt
>>>> #0  0x00007f13a0f38283 in mysql_stmt_result_metadata () from
>>>> /usr/lib/libmysqlclient.so.15
>>>> #1  0x00007f13a12ea4ba in db_mysql_do_prepared_query (conn=0x7cf7c0,
>>>> query=<value optimized out>, v=0x7fffaa088fa0, n=<value optimized
>>>> out>, uv=0x0, un=0)
>>>>   at dbase.c:370
>>>> #2  0x00007f13a12ecb41 in db_mysql_query (_h=0x7cf7c0, _k=<value
>>>> optimized out>, _op=<value optimized out>, _v=0x7fffaa088fa0,
>>>> _c=<value optimized out>,
>>>>   _n=1, _nc=2, _o=0x0, _r=0x7fffaa089040) at dbase.c:621
>>>> #3  0x00007f139da6547c in authorize (_m=0x7d0490, _realm=<value
>>>> optimized out>, _table=<value optimized out>, _hftype=<value optimized
>>>> out>)
>>>>   at authorize.c:107
>>>> #4  0x000000000040f849 in do_action (a=0x7bd1f8, msg=0x7d0490) at
>>>> action.c:961
>>>> #5  0x000000000040e7a5 in run_action_list (a=<value optimized out>,
>>>> msg=0x7d0490) at action.c:139
>>>> #6  0x0000000000470394 in eval_elem (e=0x7bd318, msg=0x7d0490,
>>>> val=0x0) at route.c:1189
>>>> #7  0x0000000000471b0d in eval_expr (e=0x14e0160, msg=0x7d0490,
>>>> val=0x0) at route.c:1486
>>>> #8  0x0000000000471abc in eval_expr (e=0x7bd3b0, msg=0x7d0490,
>>>> val=0x0) at route.c:1502
>>>> #9  0x0000000000471ae5 in eval_expr (e=0x7bd448, msg=0x7d0490,
>>>> val=0x0) at route.c:1507
>>>> #10 0x000000000040f8fc in do_action (a=0x7be468, msg=0x7d0490) at
>>>> action.c:688
>>>> #11 0x000000000040e7a5 in run_action_list (a=<value optimized out>,
>>>> msg=0x7d0490) at action.c:139
>>>> #12 0x000000000041159b in do_action (a=0x7be788, msg=0x7d0490) at
>>>> action.c:705
>>>> #13 0x000000000040e7a5 in run_action_list (a=<value optimized out>,
>>>> msg=0x7d0490) at action.c:139
>>>> #14 0x000000000041159b in do_action (a=0x7bfaa8, msg=0x7d0490) at
>>>> action.c:705
>>>> #15 0x000000000040e7a5 in run_action_list (a=<value optimized out>,
>>>> msg=0x7d0490) at action.c:139
>>>> #16 0x0000000000410e9f in do_action (a=0x7b80a0, msg=0x7d0490) at
>>>> action.c:119
>>>> #17 0x000000000040e7a5 in run_action_list (a=<value optimized out>,
>>>> msg=0x7d0490) at action.c:139
>>>> #18 0x0000000000411a1b in do_action (a=0x7b81c0, msg=0x7d0490) at
>>>> action.c:711
>>>> #19 0x000000000040e7a5 in run_action_list (a=<value optimized out>,
>>>> msg=0x7d0490) at action.c:139
>>>> #20 0x00000000004126a5 in run_top_route (a=0x7ace70, msg=0x7d0490) at
>>>> action.c:119
>>>> #21 0x000000000045f7a8 in receive_msg (
>>>>   buf=0x74e140 "INVITE sip:[hidden email] SIP/2.0\r\nDate: Sun, 01 Mar
>>>> 2009 09:06:10 GMT\r\nCSeq: 2 INVITE\r\nVia: SIP/2.0/UDP
>>>>
>>>>
>>>> x.x.x.x:5063;branch=z9hG4bK4e6660e0-ad04-de11-8875-001302b72316;rport\r\nUser-Agent:
>>>> E"..., len=1297, rcv_info=0x7fffaa08aba0) at receive.c:165
>>>> #22 0x000000000049eda6 in udp_rcv_loop () at udp_server.c:449
>>>> #23 0x00000000004298bb in main (argc=<value optimized out>,
>>>> argv=0x7fffaa08ad88) at main.c:778
>>>>
>>>> ---
>>>>
>>>> Thanks,
>>>> Om.
>>>>
>>>> On Sun, Mar 1, 2009 at 12:45 PM, Om Bikram Thapa <[hidden email]>
>>>> wrote:
>>>>
>>>>
>>>>>
>>>>> Hi Bogdan,
>>>>>
>>>>> Solved! OpenSIPS (latest SVN) no longer crashes after MySQL restarts.
>>>>>
>>>>> Thanks,
>>>>> Om.
>>>>>
>>>>> On Sun, Mar 1, 2009 at 11:33 AM, Om Bikram Thapa <[hidden email]>
>>>>> wrote:
>>>>>
>>>>>
>>>>>>
>>>>>> Hi Bogdan,
>>>>>>
>>>>>> I will update and let you know the results.
>>>>>>
>>>>>> Regards,
>>>>>> Om.
>>>>>>
>>>>>> On Sat, Feb 28, 2009 at 3:08 AM, Bogdan-Andrei Iancu
>>>>>> <[hidden email]> wrote:
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Hi Om,
>>>>>>>
>>>>>>> finally, after some struggle with the libmysqlclient, I hope the
>>>>>>> issue
>>>>>>> is
>>>>>>> solve. Please give it another try and let me know the result.
>>>>>>>
>>>>>>> Thanks and regards,
>>>>>>> Bogdan
>>>>>>>
>>>>>>> Om Bikram Thapa wrote:
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> Hi Bogdan,
>>>>>>>>
>>>>>>>> Just updated to latest svn, but I still see crashes after MySQL
>>>>>>>> restart. The syslog shows following 3 lines after MySQL restart and
>>>>>>>> registration attempt:
>>>>>>>>
>>>>>>>> --
>>>>>>>> Feb 19 12:17:43 proxy /usr/local/sbin/opensips[2333]:
>>>>>>>> ERROR:db_mysql:db_mysql_do_prepared_query: mysql_stmt_execute()
>>>>>>>> failed: (1) Unknown prepared statement handler (1) given to
>>>>>>>> mysql_stmt_execute
>>>>>>>> Feb 19 12:17:43 proxy /usr/local/sbin/opensips[2333]:
>>>>>>>> ERROR:auth_db:get_ha1: failed to query database
>>>>>>>> Feb 19 12:18:13 proxy kernel: [  105.116919] opensips[2333]:
>>>>>>>> segfault
>>>>>>>> at 4c8 ip 7f167c109283 sp 7fff8525b270 error 4 in
>>>>>>>> libmysqlclient.so.15.0.0[7f167c0ac000+1bf000]
>>>>>>>> --
>>>>>>>>
>>>>>>>> At first attempt, the registration fails with "SIP/2.0 500 Server
>>>>>>>> Internal Error" returned by the server, and subsequent registration
>>>>>>>> attempt crashes the server.
>>>>>>>>
>>>>>>>> gdb backtrace gives following details:
>>>>>>>>
>>>>>>>> ---
>>>>>>>> #0  0x00007f0119426283 in mysql_stmt_result_metadata () from
>>>>>>>> /usr/lib/libmysqlclient.so.15
>>>>>>>> (gdb) bt
>>>>>>>> #0  0x00007f0119426283 in mysql_stmt_result_metadata () from
>>>>>>>> /usr/lib/libmysqlclient.so.15
>>>>>>>> #1  0x00007f01197d8f82 in db_mysql_do_prepared_query (conn=0x78b278,
>>>>>>>> query=<value optimized out>, v=0x7fff22576880, n=<value optimized
>>>>>>>> out>, uv=0x0, un=0)
>>>>>>>>  at dbase.c:368
>>>>>>>> #2  0x00007f01197daccd in db_mysql_query (_h=0x78b278, _k=<value
>>>>>>>> optimized out>, _op=<value optimized out>, _v=0x7fff22576880,
>>>>>>>> _c=<value optimized out>,
>>>>>>>>  _n=1, _nc=2, _o=0x0, _r=0x7fff22576920) at dbase.c:603
>>>>>>>> #3  0x00007f0115f56398 in authorize (_m=0x78fe98, _realm=<value
>>>>>>>> optimized out>, _table=<value optimized out>, _hftype=<value
>>>>>>>> optimized
>>>>>>>> out>)
>>>>>>>>  at authorize.c:107
>>>>>>>> #4  0x000000000040f869 in do_action (a=0x780a48, msg=0x78fe98) at
>>>>>>>> action.c:961
>>>>>>>> #5  0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>>>>>> msg=0x78fe98) at action.c:139
>>>>>>>> #6  0x000000000046f194 in eval_elem (e=0x780b18, msg=0x78fe98,
>>>>>>>> val=0x0) at route.c:1189
>>>>>>>> #7  0x00000000004708ed in eval_expr (e=0x27648e0, msg=0x78fe98,
>>>>>>>> val=0x0) at route.c:1486
>>>>>>>> #8  0x000000000047089c in eval_expr (e=0x780b60, msg=0x78fe98,
>>>>>>>> val=0x0) at route.c:1502
>>>>>>>> #9  0x00000000004708c5 in eval_expr (e=0x780ba8, msg=0x78fe98,
>>>>>>>> val=0x0) at route.c:1507
>>>>>>>> #10 0x000000000040f91c in do_action (a=0x780f40, msg=0x78fe98) at
>>>>>>>> action.c:688
>>>>>>>> #11 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>>>>>> msg=0x78fe98) at action.c:139
>>>>>>>> #12 0x00000000004114d3 in do_action (a=0x781eb0, msg=0x78fe98) at
>>>>>>>> action.c:705
>>>>>>>> #13 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>>>>>> msg=0x78fe98) at action.c:139
>>>>>>>> #14 0x0000000000410e07 in do_action (a=0x77ee88, msg=0x78fe98) at
>>>>>>>> action.c:119
>>>>>>>> #15 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>>>>>> msg=0x78fe98) at action.c:139
>>>>>>>> #16 0x00000000004114d3 in do_action (a=0x77f028, msg=0x78fe98) at
>>>>>>>> action.c:705
>>>>>>>> #17 0x000000000040e7e5 in run_action_list (a=<value optimized out>,
>>>>>>>> msg=0x78fe98) at action.c:139
>>>>>>>> #18 0x00000000004125fe in run_top_route (a=0x777e78, msg=0x78fe98)
>>>>>>>> at
>>>>>>>> action.c:119
>>>>>>>> #19 0x000000000045e898 in receive_msg (
>>>>>>>>  buf=0x749180 "REGISTER sip:x.x.x.x SIP/2.0\r\nCSeq: 58
>>>>>>>> REGISTER\r\nVia: SIP/2.0/UDP
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> x.x.x.x:5068;branch=z9hG4bK5ca4685b-befc-dd11-911f-0015c5404858;rport\r\nUser-Agent:
>>>>>>>> Ekiga/2.0.12\r\nAuthorization: Dig"..., len=750,
>>>>>>>> rcv_info=0x7fff22578090) at receive.c:165
>>>>>>>> #20 0x000000000049d3a6 in udp_rcv_loop () at udp_server.c:449
>>>>>>>> #21 0x00000000004291fb in main (argc=<value optimized out>,
>>>>>>>> argv=0x7fff22578278) at main.c:778
>>>>>>>> ---
>>>>>>>>
>>>>>>>> I have tried several times and each time I get the same sequence of
>>>>>>>> results.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Om.
>>>>>>>>
>>>>>>>> On Thu, Feb 19, 2009 at 11:22 AM, Om Bikram Thapa
>>>>>>>> <[hidden email]>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks Bogdan. I will update and let you know.
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Om.
>>>>>>>>>
>>>>>>>>> On Thu, Feb 19, 2009 at 2:48 AM, Bogdan-Andrei Iancu
>>>>>>>>> <[hidden email]> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hi Om,
>>>>>>>>>>
>>>>>>>>>> There is a fix available on SVN - please update and see if it
>>>>>>>>>> works
>>>>>>>>>> now.
>>>>>>>>>>
>>>>>>>>>> Thanks and regards,
>>>>>>>>>> Bogdan
>>>>>>>>>>
>>>>>>>>>> Om Bikram Thapa wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Hi Bogdan,
>>>>>>>>>>>
>>>>>>>>>>> I have submitted the bug on the tracker.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Om.
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Feb 17, 2009 at 3:50 AM, Bogdan-Andrei Iancu
>>>>>>>>>>> <[hidden email]> wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Hi Om,
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks for the report - I may have an idea what is the problem;
>>>>>>>>>>>> I
>>>>>>>>>>>> will
>>>>>>>>>>>> try
>>>>>>>>>>>> to reproduce the crash in the following days.
>>>>>>>>>>>>
>>>>>>>>>>>> Could you please open a bug report on the tracker - just not to
>>>>>>>>>>>> forget
>>>>>>>>>>>> about
>>>>>>>>>>>> it :D (see
>>>>>>>>>>>> http://www.opensips.org/index.php?n=Development.Tracker)
>>>>>>>>>>>>
>>>>>>>>>>>> Regards,
>>>>>>>>>>>> Bogdan
>>>>>>>>>>>>
>>>>>>>>>>>> Om Bikram Thapa wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>> OpenSIPS is dying with "libmysqlclient.so Segmentation Fault"
>>>>>>>>>>>>> after
>>>>>>>>>>>>> MySQL server restart. The log shows
>>>>>>>>>>>>> "opensips[16769]: segfault at 4c8 ip 7f36728bf283 sp
>>>>>>>>>>>>> 7fff7ba11a90
>>>>>>>>>>>>> error 4 in libmysqlclient.so.15.0.0[7f3672862000+1bf000]"
>>>>>>>>>>>>>
>>>>>>>>>>>>> and gdb bt shows:
>>>>>>>>>>>>>
>>>>>>>>>>>>> -------------
>>>>>>>>>>>>> #0  0x00007f36728bf283 in mysql_stmt_result_metadata () from
>>>>>>>>>>>>> /usr/lib/libmysqlclient.so.15
>>>>>>>>>>>>> #1  0x00007f3672c707b7 in db_mysql_do_prepared_query
>>>>>>>>>>>>> (conn=0x78b278,
>>>>>>>>>>>>> query=<value optimized out>, v=0x7fff7ba11d20, n=1, uv=0x0,
>>>>>>>>>>>>> un=0)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> dbase.c:363
>>>>>>>>>>>>> #2  0x00007f3672c72e1f in db_mysql_query (_h=0x78b278,
>>>>>>>>>>>>> _k=<value
>>>>>>>>>>>>> optimized out>, _op=<value optimized out>, _v=0x7fff7ba11d20,
>>>>>>>>>>>>> _c=<value optimized out>,
>>>>>>>>>>>>>  _n=1, _nc=2, _o=0x0, _r=0x7fff7ba11dc0) at dbase.c:591
>>>>>>>>>>>>> #3  0x00007f366f3f0398 in authorize (_m=0x78fe98, _realm=<value
>>>>>>>>>>>>> optimized out>, _table=<value optimized out>, _hftype=<value
>>>>>>>>>>>>> optimized
>>>>>>>>>>>>> out>)
>>>>>>>>>>>>>  at authorize.c:107
>>>>>>>>>>>>> #4  0x000000000040f869 in do_action (a=0x780a48, msg=0x78fe98)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> action.c:961
>>>>>>>>>>>>> #5  0x000000000040e7e5 in run_action_list (a=<value optimized
>>>>>>>>>>>>> out>,
>>>>>>>>>>>>> msg=0x78fe98) at action.c:139
>>>>>>>>>>>>> #6  0x000000000046f194 in eval_elem (e=0x780b18, msg=0x78fe98,
>>>>>>>>>>>>> val=0x0) at route.c:1189
>>>>>>>>>>>>> #7  0x00000000004708ed in eval_expr (e=0x120b1e0, msg=0x78fe98,
>>>>>>>>>>>>> val=0x0) at route.c:1486
>>>>>>>>>>>>> #8  0x000000000047089c in eval_expr (e=0x780b60, msg=0x78fe98,
>>>>>>>>>>>>> val=0x0) at route.c:1502
>>>>>>>>>>>>> #9  0x00000000004708c5 in eval_expr (e=0x780ba8, msg=0x78fe98,
>>>>>>>>>>>>> val=0x0) at route.c:1507
>>>>>>>>>>>>> #10 0x000000000040f91c in do_action (a=0x780f40, msg=0x78fe98)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> action.c:688
>>>>>>>>>>>>> #11 0x000000000040e7e5 in run_action_list (a=<value optimized
>>>>>>>>>>>>> out>,
>>>>>>>>>>>>> msg=0x78fe98) at action.c:139
>>>>>>>>>>>>> #12 0x00000000004114d3 in do_action (a=0x781eb0, msg=0x78fe98)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> action.c:705
>>>>>>>>>>>>> #13 0x000000000040e7e5 in run_action_list (a=<value optimized
>>>>>>>>>>>>> out>,
>>>>>>>>>>>>> msg=0x78fe98) at action.c:139
>>>>>>>>>>>>> #14 0x0000000000410e07 in do_action (a=0x77ee88, msg=0x78fe98)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> action.c:119
>>>>>>>>>>>>> #15 0x000000000040e7e5 in run_action_list (a=<value optimized
>>>>>>>>>>>>> out>,
>>>>>>>>>>>>> msg=0x78fe98) at action.c:139
>>>>>>>>>>>>> #16 0x00000000004114d3 in do_action (a=0x77f028, msg=0x78fe98)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> action.c:705
>>>>>>>>>>>>> #17 0x000000000040e7e5 in run_action_list (a=<value optimized
>>>>>>>>>>>>> out>,
>>>>>>>>>>>>> msg=0x78fe98) at action.c:139
>>>>>>>>>>>>> #18 0x00000000004125fe in run_top_route (a=0x777e78,
>>>>>>>>>>>>> msg=0x78fe98) at
>>>>>>>>>>>>> action.c:119
>>>>>>>>>>>>> #19 0x000000000045e898 in receive_msg (
>>>>>>>>>>>>>  buf=0x749180 "REGISTER sip:xxxxx SIP/2.0\r\nCSeq: 2
>>>>>>>>>>>>> REGISTER\r\nVia: SIP/2.0/UDP
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> x.x.x.x:5061;branch=z9hG4bK60f1b2dd-57fa-dd11-9401-0015c5404858;rport\r\nUser-Agent:
>>>>>>>>>>>>> Ekiga/2.0.12\r\nAuthorization: Dige"..., len=749,
>>>>>>>>>>>>> rcv_info=0x7fff7ba13530) at receive.c:165
>>>>>>>>>>>>> #20 0x000000000049d3a6 in udp_rcv_loop () at udp_server.c:449
>>>>>>>>>>>>> #21 0x00000000004291fb in main (argc=<value optimized out>,
>>>>>>>>>>>>> argv=0x7fff7ba13718) at main.c:778
>>>>>>>>>>>>> ------------
>>>>>>>>>>>>>
>>>>>>>>>>>>> Server runs fine after restart until MySQL is restarted again.
>>>>>>>>>>>>> In
>>>>>>>>>>>>> my
>>>>>>>>>>>>> lab setup, OpenSIPS is getting killed triggered by registration
>>>>>>>>>>>>> request every morning (probably due to MySQL being restarted
>>>>>>>>>>>>> daily
>>>>>>>>>>>>> with logrotate).
>>>>>>>>>>>>>
>>>>>>>>>>>>> The server is the latest trunk on Debian lenny/AMD64.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Om.
>>>>>>>>>>>>>
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> Users mailing list
>>>>>>>>>>>>> [hidden email]
>>>>>>>>>>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>
>>>>
>>>
>>>
>
>

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: OpenSIPS libmysqlclient.so segfault after MySQL restart

Bogdan-Andrei Iancu
Hi Om,

Thanks for the help and for the testing!

Regards,
Bogdan

Om Bikram Thapa wrote:

> Hi Bogdan,
>
> The latest SVN has fixed the libmysqlclient segmentation fault issue.
>
> Thank you.
> Om.
>
> On Tue, Mar 3, 2009 at 10:21 PM, Bogdan-Andrei Iancu
> <[hidden email]> wrote:
>  
>> Hi Om,
>>
>> Thanks again for your help in fixing this. With the log you sent me I
>> figured out what was wrong - when a reconnect even was detected, previously,
>> I was re-initializing only the current statements. But on a mysql connection
>> you can have multiple statements, so you need to reset all of them.
>>
>> Also the re-connect even may be detected when running a standard text
>> query....
>>
>> I made the necessary fixes on SVN - hopefully it should be ok now. :)
>>
>> Regards,
>> Bogdan
>>
>> Om Bikram Thapa wrote:
>>    
>>> Hi Bogdan,
>>>
>>> I am again able to crash opensips by restarting MySQL two times with
>>> 2/3 register/de-register (it was okay till then) and then crashed with
>>> INVITE.
>>>
>>>
>>>      


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users