OpenSIPS crashes.

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

OpenSIPS crashes.

Jan D.
Hello,

Since today our OpenSIPS craches. It happend three times now, I get the following error:

Apr 10 09:07:04 sip2 /usr/sbin/opensips[28331]: CRITICAL:core:free_lump: called on a not free-able lump:0x7f6abd4f1270 flags=2
Apr 10 12:21:29 sip2 /usr/sbin/opensips[31706]: CRITICAL:core:free_lump: called on a not free-able lump:0x7f29f0fa0630 flags=2
Apr 10 12:51:22 sip2 /usr/sbin/opensips[32567]: CRITICAL:core:free_lump: called on a not free-able lump:0x7f59b243c7c8 flags=2

The last two craches, we used:

root@sip2:~# dpkg -l | grep opensips
ii  opensips                           1.7.2-1-89231             very fast and configurable SIP server
ii  opensips-mysql-module              1.7.2-1-89231             MySQL database connectivity module for OpenSIPS

After the first one (09:07 hour) we did a update, so that one was on a slighty older build.
I cannot find any other "weirdness" on this machine.

Opensips is running with 1024 Mbyte:

/usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 1024 -u opensips -g opensips

Does anyone know what is going on?

greetings,

Jan

Reply | Threaded
Open this post in threaded view
|

Re: OpenSIPS crashes.

Jan D.
Oh by the way....

With craches, i mean OpenSIP has completely stopped.
I takes a /etc/init.d/opensips start to start again.

greeting,

Jan
Reply | Threaded
Open this post in threaded view
|

Re: OpenSIPS crashes.

Vlad Paiu
Hello,

Can you please extract a backtrace from the core dumps ? Just do
        gdb opensips <path_to_coredump>
and then
        bt full
and paste here the output.

Regards,

Vlad Paiu
OpenSIPS Developer
http://www.opensips-solutions.com


On 04/10/2012 03:54 PM, Jan D. wrote:

> Oh by the way....
>
> With craches, i mean OpenSIP has completely stopped.
> I takes a /etc/init.d/opensips start to start again.
>
> greeting,
>
> Jan
>
> --
> View this message in context: http://opensips-open-sip-server.1449251.n2.nabble.com/OpenSIPS-crashes-tp7452782p7452795.html
> Sent from the OpenSIPS - Users mailing list archive at Nabble.com.
>
> _______________________________________________
> Users mailing list
> [hidden email]
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: OpenSIPS crashes.

Jan D.
Vlad,

Sorry, no core dumps. Wasn't enabeld in /etc/default/opensips.
I'll enable it a.s.a.p., but have to waitig until a quiet moment somewhere tonight....
or when is craches again ;-)

Greetings,

Jan

 
Reply | Threaded
Open this post in threaded view
|

Re: OpenSIPS crashes.

Bogdan-Andrei Iancu-2
Jan,

btw, as I noticed you install opensips from debs, in order to be able to
read the core file, you need to install the opensips-dbg package also
(the debug symbols).

Regards,
Bogdan

On 04/10/2012 04:38 PM, Jan D. wrote:

> Vlad,
>
> Sorry, no core dumps. Wasn't enabeld in /etc/default/opensips.
> I'll enable it a.s.a.p., but have to waitig until a quiet moment somewhere
> tonight....
> or when is craches again ;-)
>
> Greetings,
>
> Jan
>
>
>
> --
> View this message in context: http://opensips-open-sip-server.1449251.n2.nabble.com/OpenSIPS-crashes-tp7452782p7452903.html
> Sent from the OpenSIPS - Users mailing list archive at Nabble.com.
>
> _______________________________________________
> Users mailing list
> [hidden email]
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>


--
Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
http://www.opensips-solutions.com


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: OpenSIPS crashes.

Jan D.
Bogdan,

Tried to apt-get opensips-dbg, but there is also an newer OpenSIPS/OpenSIPS-mysql package. Since apt-get wants to install also these two, i have to wait until tonight for the update.

Untill now no new craches.

We get a lot of errors caused by a customer with X-lite/Bria softphones. I it a outgoing callcenter, when they end the call before answerd the client send a bye while the dialog is still in early state. See my other post about this.

CRITICAL:dialog:log_next_state_dlg: bogus event 7 in state 2 for dlg 0x7f285df05ea8 [414:2037778884] with clid 'NmRkMDJmODQ1NTMyMjY4MjA1YWU5MzM0YTU0MWE5MmQ.' and tags '1713ad38' 'C4FE3750-10A7'

Can a lot of these errors be resposible for the craches? Someone mentiond possible freeze of the database/connection/module under heavy load and there sould be a fix for this. Does anyone know if this fix is already applied to the latest Debian-packages?

greetings,

Jan
Reply | Threaded
Open this post in threaded view
|

Re: OpenSIPS crashes.

Jan D.
In reply to this post by Vlad Paiu
Vlad,

This morning our opensips crash again. The last time this happend was 4 days ago. This is the error in the opensips.log

Apr 14 11:05:27 /usr/sbin/opensips[6017]: CRITICAL:core:free_lump: called on a not free-able lump:0x7fefaefc61a8 flags=2

We activated coredump after your previous post, so now we have output. I pasted the output below. Can you take a look at it, I'm very curious what could be wrong:

gdb opensips /home/corefiles/core.opensips.sig6.6017
bt full


Fetched 3,749 kB in 0s (5,021 kB/s)
Selecting previously unselected package libpython2.7.
(Reading database ... 27860 files and directories currently installed.)
Unpacking libpython2.7 (from .../libpython2.7_2.7.2-8_amd64.deb) ...
Selecting previously unselected package gdbserver.
Unpacking gdbserver (from .../gdbserver_7.4really-1_amd64.deb) ...
Selecting previously unselected package gdb.
Unpacking gdb (from .../gdb_7.4really-1_amd64.deb) ...
Setting up libpython2.7 (2.7.2-8) ...
Setting up gdbserver (7.4really-1) ...
Setting up gdb (7.4really-1) ...
root@sip2:/home/corefiles# gdb opensips /home/corefiles/core.opensips.sig6.6017
GNU gdb (GDB) 7.4-debian
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/sbin/opensips...Reading symbols from /usr/lib/debug/usr/sbin/opensips...done.
done.
[New LWP 6017]

warning: Can't read pathname for load map: Input/output error.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 1024 -u opensips -g ope'.
Program terminated with signal 6, Aborted.
#0  0x00007feff2b57475 in raise () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt full
#0  0x00007feff2b57475 in raise () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#1  0x00007feff2b5a6f0 in abort () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#2  0x000000000041bf92 in free_lump (lmp=0x7fefaefc61a8) at data_lump.c:438
        __FUNCTION__ = "free_lump"
#3  0x000000000041c107 in free_lump_list (l=<optimized out>) at data_lump.c:467
        t = 0x7fefaefc6170
        r = 0x0
#4  0x000000000041c250 in del_notflaged_lumps (lump_list=<optimized out>, not_flags=3) at data_lump.c:674
        r = 0x1
        crt = 0x7fefaefc6210
        prev = 0x7fefaefc5360
        prev_r = <optimized out>
#5  0x00007feff24f0bc8 in relay_reply (t=0x7fefaef9b720, p_msg=0x7fefaefc4d08, branch=1, msg_status=480, cancel_bitmap=<optimized out>) at t_reply.c:1216
        relay = 0
        save_clone = 0
        buf = 0x7d0f88 "SIP/2.0 603 Decline\r\nVia: SIP/2.0/UDP 83.255.31.120:5060;rport=5060;received=83.255.31.120;branch=z9hG4bK-1b1-1334394326-2364-384\r\nRecord-Route: <sip:90.143.10.230;lr;ftag=95ffcd055e0f78f7d5d397020e89"...
        res_len = 650
        relayed_code = 603
        relayed_msg = 0x7fefaefc4d08
        bm = {to_tag_val = {s = 0x0, len = 8017128}}
        totag_retr = <optimized out>
        reply_status = RPS_COMPLETED
        cb_s = {
          s = 0x72f127 "From: <sip:0254852660@sip.server.nl>;tag=95ffcd055e0f78f7d5d397020e89288d752087f7\r\nTo: <sip:31254712460@90.143.10.230:5060>;tag=z9hG4bK36fb.ed56e941.1\r\nCSeq: 1 INVITE\r\nContent-Length:  0\r\n\r\n", len = 0}
        text = {s = 0x72f17b "To: <sip:31254712460@90.143.10.230:5060>;tag=z9hG4bK36fb.ed56e941.1\r\nCSeq: 1 INVITE\r\nContent-Length:  0\r\n\r\n", len = 69}
        __FUNCTION__ = "relay_reply"
#6  0x00007feff24f19fe in reply_received (p_msg=0x7c9848) at t_reply.c:1527
        last_uac_status = <optimized out>
        branch = 1
        reply_status = <optimized out>
        timer = <optimized out>
        cancel_bitmap = 0
        t = 0x7fefaef9b720
        backup_list = 0x0
        __FUNCTION__ = "reply_received"
#7  0x0000000000420bb0 in forward_reply (msg=0x7c9848) at forward.c:568
        new_buf = <optimized out>
        to = <optimized out>
        new_len = <optimized out>
        mod = 0x75ada8
        proto = <optimized out>
        id = <optimized out>
        send_sock = <optimized out>
        len = <optimized out>
        __FUNCTION__ = "forward_reply"
#8  0x000000000044a8f0 in receive_msg (
    buf=0x72efa0 "SIP/2.0 480 Temporarily Unavailable\r\nVia: SIP/2.0/UDP 90.143.10.230;received=90.143.10.230;branch=z9hG4bK36fb.ed56e941.1\r\nVia: SIP/2.0/UDP 83.255.31.120:5060;rport=5060;received=83.255.31.120;branch=z"..., len=582,

rcv_info=0x7fff94fd5b30) at receive.c:203
        msg = 0x7c9848
        start = {tv_sec = 16, tv_usec = 0}
        __FUNCTION__ = "receive_msg"
#9  0x0000000000483435 in udp_rcv_loop () at udp_server.c:418
        len = 582
        buf = "SIP/2.0 480 Temporarily Unavailable\r\nVia: SIP/2.0/UDP 90.143.10.230;received=90.143.10.230;branch=z9hG4bK36fb.ed56e941.1\r\nVia: SIP/2.0/UDP 83.255.31.120:5060;rport=5060;received=83.255.31.120;branch=z"...
        tmp = <optimized out>
        from = 0x7c9810
---Type <return> to continue, or q <return> to quit---
        fromlen = 16
        ri = {src_ip = {af = 2, len = 4, u = {addrl = {140665747658078, 0}, addr32 = {1273746782, 32751, 0, 0}, addr16 = {54622, 19435, 32751, 0, 0, 0, 0, 0},
              addr = "^\325\353K\357\177\000\000\000\000\000\000\000\000\000"}}, dst_ip = {af = 2, len = 4, u = {addrl = {3859452505, 0}, addr32 = {3859452505, 0, 0, 0}, addr16 = {37465, 58890, 0, 0, 0, 0,
                0, 0}, addr = "Y\222\n\346", '\000' <repeats 11 times>}}, src_port = 5060, dst_port = 5060, proto = 1, proto_reserved1 = 0, proto_reserved2 = 0, src_su = {s = {sa_family = 2,
              sa_data = "\023\304^\325\353K\000\000\000\000\000\000\000"}, sin = {sin_family = 2, sin_port = 50195, sin_addr = {s_addr = 1273746782}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {
              sin6_family = 2, sin6_port = 50195, sin6_flowinfo = 1273746782, sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0,
                    0}}}, sin6_scope_id = 0}}, bind_address = 0x75a758}
        p = 0x7fff94fd5b38
        __FUNCTION__ = "udp_rcv_loop"
#10 0x0000000000427697 in main_loop () at main.c:872
        i = 17
        pid = <optimized out>
        si = 0x75a758
        startup_done = 0x0
        chd_rank = 18
        load_p = 0x7fefaef21f80
#11 main (argc=<optimized out>, argv=<optimized out>) at main.c:1490
        cfg_log_stderr = <optimized out>
        cfg_stream = <optimized out>
        c = <optimized out>
        r = <optimized out>
        tmp = 0x7fff94fd6df7 ""
        tmp_len = <optimized out>
        port = <optimized out>
        proto = <optimized out>
        ret = <optimized out>
        seed = 3855936534
        rfd = <optimized out>
        __FUNCTION__ = "main"
Reply | Threaded
Open this post in threaded view
|

Re: OpenSIPS crashes.

Jan D.
In reply to this post by Bogdan-Andrei Iancu-2
Bogdan,

Today opensips stopped again:

Apr 16 12:41:25 /usr/sbin/opensips[9491]: CRITICAL:core:free_lump: called on a not free-able lump:0x7f529e78c628 flags=2

Here is another backtrace, can you take a look at it. I'm also curious if this is software or hardware related.

#0  0x00007f52e2388475 in raise () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#1  0x00007f52e238b6f0 in abort () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#2  0x000000000041bf92 in free_lump (lmp=0x7f529e78c628) at data_lump.c:438
        __FUNCTION__ = "free_lump"
#3  0x000000000041c107 in free_lump_list (l=<optimized out>) at data_lump.c:467
        t = 0x7f529e78c5f0
        r = 0x0
#4  0x000000000041c250 in del_notflaged_lumps (lump_list=<optimized out>, not_flags=3) at data_lump.c:674
        r = 0x1
        crt = 0x7f529e78c690
        prev = 0x7f529e78b800
        prev_r = <optimized out>
#5  0x00007f52e1d21bc8 in relay_reply (t=0x7f529e7daa00, p_msg=0x7f529e78b1a8, branch=1, msg_status=408, cancel_bitmap=<optimized out>) at t_reply.c:1216
        relay = 0
        save_clone = 0
        buf = 0x7cbcf0 "SIP/2.0 486 Busy Here\r\nVia: SIP/2.0/UDP 83.255.31.120:5060;received=83.255.31.120;rport=5060;branch=z9hG4bK-3884-1334572881-1293-461\r\nRecord-Route: <sip:90.143.10.230;lr;ftag=95ffcd055e0f78f7d5d397020"...
        res_len = 635
        relayed_code = 486
        relayed_msg = 0x7f529e78b1a8
        bm = {to_tag_val = {
            s = 0x7f529e7c14a8 "INVITE sip:mysipuser@84.14.10.22:1025;transport=udp SIP/2.0\r\nRecord-Route: <sip:90.143.10.230;lr;ftag=95ffcd055e0f78f7d5d397020e89288d45acada7;did=78b.e7ec2514>\r\nVia: SIP/2.0/UDP 90.143.10.230;bra"..., len = -1635930632}}
        totag_retr = <optimized out>
        reply_status = RPS_COMPLETED
        cb_s = {s = 0x3d0900 <Address 0x3d0900 out of bounds>, len = -506395470}
        text = {s = 0x7f529e7dae20 "\370Fq\236R\177", len = 7406984}
        __FUNCTION__ = "relay_reply"
#6  0x00007f52e1d116b9 in fake_reply (code=<optimized out>, branch=<optimized out>, t=<optimized out>) at timer.c:259
        cancel_bitmap = 0
        reply_status = <optimized out>
#7  final_response_handler (fr_tl=<optimized out>) at timer.c:370
        r_buf = 0x7f529e7dadd8
        t = 0x7f529e7daa00
#8  timer_routine (ticks=<optimized out>, attr=<optimized out>) at timer.c:1011
        tl = <optimized out>
        tmp_tl = 0x0
        id = <optimized out>
        __FUNCTION__ = "timer_routine"
#9  0x000000000047442a in timer_ticker (timer_list=<optimized out>) at timer.c:355
        t = 0x75a028
#10 run_timer_process (tpl=<optimized out>) at timer.c:425
        multiple = 10
        cnt = <optimized out>
        tv = {tv_sec = 0, tv_usec = 0}
#11 start_timer_processes () at timer.c:522
        tpl = 0x7b1eb0
        pid = <optimized out>
        __FUNCTION__ = "start_timer_processes"
#12 0x0000000000427956 in main_loop () at main.c:929
        i = 25
        pid = 0
        si = 0x0
        startup_done = 0x0
        chd_rank = 25
        load_p = 0x7f529e752f80
#13 main (argc=<optimized out>, argv=<optimized out>) at main.c:1490
        cfg_log_stderr = <optimized out>
        cfg_stream = <optimized out>
        c = <optimized out>
        r = <optimized out>
        tmp = 0x7fff6c4a7ea3 ""
---Type <return> to continue, or q <return> to quit---
        tmp_len = <optimized out>
        port = <optimized out>
        proto = <optimized out>
        ret = <optimized out>
        seed = 309563501
        rfd = <optimized out>
        __FUNCTION__ = "main"
Reply | Threaded
Open this post in threaded view
|

Re: OpenSIPS crashes.

Vlad Paiu
Hi,

I have just committed a fix for this in the 1.7 and 1.8 branches, and
trunk as well.
Please update your sources.

Regards,
Vlad

Pe 4/16/2012 2:32 PM, Jan D. a scris:

> Bogdan,
>
> Today opensips stopped again:
>
> Apr 16 12:41:25 /usr/sbin/opensips[9491]: CRITICAL:core:free_lump: called on
> a not free-able lump:0x7f529e78c628 flags=2
>
> Here is another backtrace, can you take a look at it. I'm also curious if
> this is software or hardware related.
>
> #0  0x00007f52e2388475 in raise () from /lib/x86_64-linux-gnu/libc.so.6
> No symbol table info available.
> #1  0x00007f52e238b6f0 in abort () from /lib/x86_64-linux-gnu/libc.so.6
> No symbol table info available.
> #2  0x000000000041bf92 in free_lump (lmp=0x7f529e78c628) at data_lump.c:438
>          __FUNCTION__ = "free_lump"
> #3  0x000000000041c107 in free_lump_list (l=<optimized out>) at
> data_lump.c:467
>


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: OpenSIPS crashes.

Jan D.
Vlad,

Thanks

Is this update also committed to deb http://apt.opensips.org/ stable17 main

(is there a changelog for stable17?)