segfault in db_mysql.so and others..

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

segfault in db_mysql.so and others..

Brett Nemeroff
Hey All,
I'm running opensips 1.8.2 svnrevision: 2:9628M

I'm getting some random crashes. Load isn't terribly high on these boxes and I can't figure out what specifically is causing the crash. 

I'm seeing this in the logs:
/usr/sbin/opensips[9875]: ERROR:db_mysql:db_mysql_val2str: destination buffer too short
/usr/sbin/opensips[9875]: ERROR:core:db_print_values: Error while converting value to string
/usr/sbin/opensips[9875]: ERROR:core:db_do_insert: error while preparing insert operation
/usr/sbin/opensips[9875]: ERROR:core:ql_timer_routine: failed to insert rows to DB
/usr/sbin/opensips[9848]: CRITICAL:core:sig_alarm_abort: BUG - shutdown timeout triggered, dying...

And I have a bunch of core files. I think this is the one associated to the above log files from the looks of the timestamp:
#0  0x00007f850eb4a8a5 in raise () from /lib64/libc.so.6
No symbol table info available.
#1  0x00007f850eb4c085 in abort () from /lib64/libc.so.6
No symbol table info available.
#2  0x000000000042c245 in sig_alarm_abort (signo=<value optimized out>) at main.c:439
        __FUNCTION__ = "sig_alarm_abort"
#3  <signal handler called>
No symbol table info available.
#4  0x00000000004d2bfe in fm_status (qm=<value optimized out>) at mem/f_malloc.c:612
        f = 0x7f7f2f789ec0
        i = 189540279
        j = <value optimized out>
        h = <value optimized out>
        unused = 0
        size = 31059075021264
        __FUNCTION__ = "fm_status"
#5  0x000000000042c42f in cleanup (show_status=<value optimized out>) at main.c:380
        __FUNCTION__ = "cleanup"
#6  0x000000000042cd8b in handle_sigs () at main.c:549
        chld = <value optimized out>
        chld_status = 139
        overall_status = 139
        i = <value optimized out>
        do_exit = 1
        __FUNCTION__ = "handle_sigs"
#7  0x0000000000430bd5 in main_loop (argc=<value optimized out>, argv=<value optimized out>) at main.c:1008
        i = <value optimized out>
        pid = <value optimized out>
        si = <value optimized out>
        startup_done = <value optimized out>
        chd_rank = 32
        rc = <value optimized out>
        load_p = 0x7f7f244bdb08
#8  main (argc=<value optimized out>, argv=<value optimized out>) at main.c:1541
        cfg_log_stderr = <value optimized out>
        cfg_stream = <value optimized out>
        c = <value optimized out>
        r = <value optimized out>
        tmp = 0x7fffbd27e8f0 ""



I'm also seeing these associated to db_mysql.so:
#0  0x00000000004d3e84 in fm_remove_free (qm=0x7f7f241d8000, size=1304) at mem/f_malloc.c:172
        pf = 0x0
        hash = 163
#1  fm_malloc (qm=0x7f7f241d8000, size=1304) at mem/f_malloc.c:383
        frag = 0x7f7f2f737180
        n = <value optimized out>
        hash = <value optimized out>
        __FUNCTION__ = "fm_malloc"
#2  0x0000000000517d86 in shm_malloc (entry=0x7f7f29b44a00, row=0x7f8501d1a860, ins_rows=0x7fffbd27bf98) at db/../evi/../mem/shm_mem.h:202
        p = <value optimized out>
#3  ql_row_add (entry=0x7f7f29b44a00, row=0x7f8501d1a860, ins_rows=0x7fffbd27bf98) at db/db_insertq.c:257
        val_size = 1298
        i = <value optimized out>
        len = <value optimized out>
        no_rows = 0
        pos = <value optimized out>
        shm_row = <value optimized out>
        __FUNCTION__ = "ql_row_add"
#4  0x00007f85066e5c8e in db_mysql_do_prepared_query (conn=0x7f8506c666c0, v=0x7f8501d1a860, n=33, uv=<value optimized out>, un=0, query=0x7f85069007b0) at dbase.c:554
        i = <value optimized out>
        j = <value optimized out>
        code = <value optimized out>
        cols = <value optimized out>



Please let me know if you have any ideas as to what may be causing these or if you need any further debug info?

Thanks,
Brett


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: segfault in db_mysql.so and others..

Brett Nemeroff
Any ideas here? :/


On Tue, Mar 5, 2013 at 10:10 AM, Brett Nemeroff <[hidden email]> wrote:
Hey All,
I'm running opensips 1.8.2 svnrevision: 2:9628M

I'm getting some random crashes. Load isn't terribly high on these boxes and I can't figure out what specifically is causing the crash. 


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: segfault in db_mysql.so and others..

shaheryarkh
I am not sure but it seems that the data in one or more column is larger then column width provided by MySQL, resulting in buffer overflow. Can you check which SQL query might have caused this problem? There are two possibilities,

1. Table encoding is multi-byte char and opensips is somehow considering it single-byte char. But in this case it should ALWAYS crash, not randomly as you said.
2. You imported larger amount of data in column with lesser width while size check constraints were disabled. You need to find such records and increase respective column width to match data size. You may also consider to shorten the data in respective columns of those records.

Thank you.


On Wed, Mar 6, 2013 at 1:54 PM, Brett Nemeroff <[hidden email]> wrote:
Any ideas here? :/


On Tue, Mar 5, 2013 at 10:10 AM, Brett Nemeroff <[hidden email]> wrote:
Hey All,
I'm running opensips 1.8.2 svnrevision: 2:9628M

I'm getting some random crashes. Load isn't terribly high on these boxes and I can't figure out what specifically is causing the crash. 


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users




--
Muhammad Shahzad
-----------------------------------
CISCO Rich Media Communication Specialist (CRMCS)
CISCO Certified Network Associate (CCNA)
Cell: +49 176 99 83 10 85
MSN: [hidden email]
Email: [hidden email]

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users