OpenSIPS core dumps

classic Classic list List threaded Threaded
23 messages Options
12
Reply | Threaded
Open this post in threaded view
|

OpenSIPS core dumps

thrillerbee
When OpenSIPS crashes, three corefiles are generated that are 2.1GB in size.  How do I use these files to understand what's causing the crash?

Thanks.

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: OpenSIPS core dumps

thrillerbee
I have this info from dmesg:

[1985853.285221] opensips[30865]: segfault at 10 ip 7f43899ce21f sp 7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000]
[1985856.379671] opensips[30858]: segfault at 10 ip 7f43899ce21f sp 7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000]
[1985896.961279] opensips[30868]: segfault at 10 ip 7f43899ce21f sp 7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000]

[2000131.245512] opensips[17672]: segfault at 10 ip 7fd0f21fb21f sp 7fff3a3b4f00 error 4 in db_flatstore.so[7fd0f21f8000+5000]
[2000161.735962] opensips[17668]: segfault at 10 ip 7fd0f21fb21f sp 7fff3a3b4e40 error 4 in db_flatstore.so[7fd0f21f8000+5000]
[2000167.299402] opensips[17670]: segfault at 10 ip 7fd0f21fb21f sp 7fff3a3b4f00 error 4 in db_flatstore.so[7fd0f21f8000+5000]

On Thu, Oct 14, 2010 at 1:57 PM, thrillerbee <[hidden email]> wrote:
When OpenSIPS crashes, three corefiles are generated that are 2.1GB in size.  How do I use these files to understand what's causing the crash?

Thanks.


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: OpenSIPS core dumps

Anca Vamanu-2
Hi,

You need to inspect them with gdb, run: gdb path_to_opensips_executable path_to_corefile, and then run 'bt full' and send the output.

Regards,
-- 
Anca Vamanu
www.voice-system.ro


On 10/14/2010 10:12 PM, thrillerbee wrote:
I have this info from dmesg:

[1985853.285221] opensips[30865]: segfault at 10 ip 7f43899ce21f sp 7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000]
[1985856.379671] opensips[30858]: segfault at 10 ip 7f43899ce21f sp 7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000]
[1985896.961279] opensips[30868]: segfault at 10 ip 7f43899ce21f sp 7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000]

[2000131.245512] opensips[17672]: segfault at 10 ip 7fd0f21fb21f sp 7fff3a3b4f00 error 4 in db_flatstore.so[7fd0f21f8000+5000]
[2000161.735962] opensips[17668]: segfault at 10 ip 7fd0f21fb21f sp 7fff3a3b4e40 error 4 in db_flatstore.so[7fd0f21f8000+5000]
[2000167.299402] opensips[17670]: segfault at 10 ip 7fd0f21fb21f sp 7fff3a3b4f00 error 4 in db_flatstore.so[7fd0f21f8000+5000]

On Thu, Oct 14, 2010 at 1:57 PM, thrillerbee <[hidden email]> wrote:
When OpenSIPS crashes, three corefiles are generated that are 2.1GB in size.  How do I use these files to understand what's causing the crash?

Thanks.


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: OpenSIPS core dumps

Bogdan-Andrei Iancu
Any chance with the backtraces ?

Regards,
Bogdan

Anca Vamanu wrote:

> Hi,
>
> You need to inspect them with gdb, run: gdb
> path_to_opensips_executable path_to_corefile, and then run 'bt full'
> and send the output.
>
> Regards,
> --
> Anca Vamanu
> www.voice-system.ro
>
>
> On 10/14/2010 10:12 PM, thrillerbee wrote:
>> I have this info from dmesg:
>>
>> [1985853.285221] opensips[30865]: segfault at 10 ip 7f43899ce21f sp
>> 7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000]
>> [1985856.379671] opensips[30858]: segfault at 10 ip 7f43899ce21f sp
>> 7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000]
>> [1985896.961279] opensips[30868]: segfault at 10 ip 7f43899ce21f sp
>> 7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000]
>>
>> [2000131.245512] opensips[17672]: segfault at 10 ip 7fd0f21fb21f sp
>> 7fff3a3b4f00 error 4 in db_flatstore.so[7fd0f21f8000+5000]
>> [2000161.735962] opensips[17668]: segfault at 10 ip 7fd0f21fb21f sp
>> 7fff3a3b4e40 error 4 in db_flatstore.so[7fd0f21f8000+5000]
>> [2000167.299402] opensips[17670]: segfault at 10 ip 7fd0f21fb21f sp
>> 7fff3a3b4f00 error 4 in db_flatstore.so[7fd0f21f8000+5000]
>>
>> On Thu, Oct 14, 2010 at 1:57 PM, thrillerbee <[hidden email]
>> <mailto:[hidden email]>> wrote:
>>
>>     When OpenSIPS crashes, three corefiles are generated that are
>>     2.1GB in size.  How do I use these files to understand what's
>>     causing the crash?
>>
>>     Thanks.
>>
>>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Users mailing list
> [hidden email]
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>  


--
Bogdan-Andrei Iancu
OpenSIPS Bootcamp
15 - 19 November 2010, Edison, New Jersey, USA
www.voice-system.ro


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: OpenSIPS core dumps

thrillerbee
Bogdan,

Yes,  I've attached 2 to my response - one was a crash from 10/29.  The other occurred a few minutes ago.

Thanks.

On Fri, Oct 22, 2010 at 11:41 AM, Bogdan-Andrei Iancu <[hidden email]> wrote:
Any chance with the backtraces ?

Regards,
Bogdan

Anca Vamanu wrote:
> Hi,
>
> You need to inspect them with gdb, run: gdb
> path_to_opensips_executable path_to_corefile, and then run 'bt full'
> and send the output.
>
> Regards,
> --
> Anca Vamanu
> www.voice-system.ro
>
>
> On 10/14/2010 10:12 PM, thrillerbee wrote:
>> I have this info from dmesg:
>>
>> [1985853.285221] opensips[30865]: segfault at 10 ip 7f43899ce21f sp
>> 7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000]
>> [1985856.379671] opensips[30858]: segfault at 10 ip 7f43899ce21f sp
>> 7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000]
>> [1985896.961279] opensips[30868]: segfault at 10 ip 7f43899ce21f sp
>> 7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000]
>>
>> [2000131.245512] opensips[17672]: segfault at 10 ip 7fd0f21fb21f sp
>> 7fff3a3b4f00 error 4 in db_flatstore.so[7fd0f21f8000+5000]
>> [2000161.735962] opensips[17668]: segfault at 10 ip 7fd0f21fb21f sp
>> 7fff3a3b4e40 error 4 in db_flatstore.so[7fd0f21f8000+5000]
>> [2000167.299402] opensips[17670]: segfault at 10 ip 7fd0f21fb21f sp
>> 7fff3a3b4f00 error 4 in db_flatstore.so[7fd0f21f8000+5000]
>>
>> On Thu, Oct 14, 2010 at 1:57 PM, thrillerbee <[hidden email]
>> <mailto:[hidden email]>> wrote:
>>
>>     When OpenSIPS crashes, three corefiles are generated that are
>>     2.1GB in size.  How do I use these files to understand what's
>>     causing the crash?
>>
>>     Thanks.
>>
>>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Users mailing list
> [hidden email]
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>


--
Bogdan-Andrei Iancu
OpenSIPS Bootcamp
15 - 19 November 2010, Edison, New Jersey, USA
www.voice-system.ro


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users

core1_102910_e.txt (7K) Download Attachment
core1_110110_e.txt (8K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: OpenSIPS core dumps

thrillerbee
Bogdan,

It crashed again tonight.  I've attached the backtrace.

Thanks.

On Mon, Nov 1, 2010 at 9:32 AM, thrillerbee <[hidden email]> wrote:
Bogdan,

Yes,  I've attached 2 to my response - one was a crash from 10/29.  The other occurred a few minutes ago.

Thanks.

On Fri, Oct 22, 2010 at 11:41 AM, Bogdan-Andrei Iancu <[hidden email]> wrote:
Any chance with the backtraces ?

Regards,
Bogdan

Anca Vamanu wrote:
> Hi,
>
> You need to inspect them with gdb, run: gdb
> path_to_opensips_executable path_to_corefile, and then run 'bt full'
> and send the output.
>
> Regards,
> --
> Anca Vamanu
> www.voice-system.ro
>
>
> On 10/14/2010 10:12 PM, thrillerbee wrote:
>> I have this info from dmesg:
>>
>> [1985853.285221] opensips[30865]: segfault at 10 ip 7f43899ce21f sp
>> 7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000]
>> [1985856.379671] opensips[30858]: segfault at 10 ip 7f43899ce21f sp
>> 7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000]
>> [1985896.961279] opensips[30868]: segfault at 10 ip 7f43899ce21f sp
>> 7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000]
>>
>> [2000131.245512] opensips[17672]: segfault at 10 ip 7fd0f21fb21f sp
>> 7fff3a3b4f00 error 4 in db_flatstore.so[7fd0f21f8000+5000]
>> [2000161.735962] opensips[17668]: segfault at 10 ip 7fd0f21fb21f sp
>> 7fff3a3b4e40 error 4 in db_flatstore.so[7fd0f21f8000+5000]
>> [2000167.299402] opensips[17670]: segfault at 10 ip 7fd0f21fb21f sp
>> 7fff3a3b4f00 error 4 in db_flatstore.so[7fd0f21f8000+5000]
>>
>> On Thu, Oct 14, 2010 at 1:57 PM, thrillerbee <[hidden email]
>> <mailto:[hidden email]>> wrote:
>>
>>     When OpenSIPS crashes, three corefiles are generated that are
>>     2.1GB in size.  How do I use these files to understand what's
>>     causing the crash?
>>
>>     Thanks.
>>
>>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Users mailing list
> [hidden email]
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>


--
Bogdan-Andrei Iancu
OpenSIPS Bootcamp
15 - 19 November 2010, Edison, New Jersey, USA
www.voice-system.ro


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users



_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users

core1_110110_2_e.txt (8K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: OpenSIPS core dumps

Bogdan-Andrei Iancu
Hi,

in frame 0, could you print:
    h
    h->tail
    ((struct flat_con*)(h->tail))->file

Also, before crash, do you see in the logs any errors from the
db_flatstore module ?

Regards,
Bogdan

thrillerbee wrote:

> Bogdan,
>
> It crashed again tonight.  I've attached the backtrace.
>
> Thanks.
>
> On Mon, Nov 1, 2010 at 9:32 AM, thrillerbee <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     Bogdan,
>
>     Yes,  I've attached 2 to my response - one was a crash from 10/29.
>      The other occurred a few minutes ago.
>
>     Thanks.
>
>     On Fri, Oct 22, 2010 at 11:41 AM, Bogdan-Andrei Iancu
>     <[hidden email] <mailto:[hidden email]>> wrote:
>
>         Any chance with the backtraces ?
>
>         Regards,
>         Bogdan
>
>         Anca Vamanu wrote:
>         > Hi,
>         >
>         > You need to inspect them with gdb, run: gdb
>         > path_to_opensips_executable path_to_corefile, and then run
>         'bt full'
>         > and send the output.
>         >
>         > Regards,
>         > --
>         > Anca Vamanu
>         > www.voice-system.ro <http://www.voice-system.ro>
>         >
>         >
>         > On 10/14/2010 10:12 PM, thrillerbee wrote:
>         >> I have this info from dmesg:
>         >>
>         >> [1985853.285221] opensips[30865]: segfault at 10 ip
>         7f43899ce21f sp
>         >> 7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000]
>         >> [1985856.379671] opensips[30858]: segfault at 10 ip
>         7f43899ce21f sp
>         >> 7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000]
>         >> [1985896.961279] opensips[30868]: segfault at 10 ip
>         7f43899ce21f sp
>         >> 7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000]
>         >>
>         >> [2000131.245512] opensips[17672]: segfault at 10 ip
>         7fd0f21fb21f sp
>         >> 7fff3a3b4f00 error 4 in db_flatstore.so[7fd0f21f8000+5000]
>         >> [2000161.735962] opensips[17668]: segfault at 10 ip
>         7fd0f21fb21f sp
>         >> 7fff3a3b4e40 error 4 in db_flatstore.so[7fd0f21f8000+5000]
>         >> [2000167.299402] opensips[17670]: segfault at 10 ip
>         7fd0f21fb21f sp
>         >> 7fff3a3b4f00 error 4 in db_flatstore.so[7fd0f21f8000+5000]
>         >>
>         >> On Thu, Oct 14, 2010 at 1:57 PM, thrillerbee
>         <[hidden email] <mailto:[hidden email]>
>         >> <mailto:[hidden email]
>         <mailto:[hidden email]>>> wrote:
>         >>
>         >>     When OpenSIPS crashes, three corefiles are generated
>         that are
>         >>     2.1GB in size.  How do I use these files to understand
>         what's
>         >>     causing the crash?
>         >>
>         >>     Thanks.
>         >>
>         >>
>         >
>         ------------------------------------------------------------------------
>         >
>         > _______________________________________________
>         > Users mailing list
>         > [hidden email] <mailto:[hidden email]>
>         > http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>         >
>
>
>         --
>         Bogdan-Andrei Iancu
>         OpenSIPS Bootcamp
>         15 - 19 November 2010, Edison, New Jersey, USA
>         www.voice-system.ro <http://www.voice-system.ro>
>
>
>         _______________________________________________
>         Users mailing list
>         [hidden email] <mailto:[hidden email]>
>         http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Users mailing list
> [hidden email]
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users


--
Bogdan-Andrei Iancu
OpenSIPS Bootcamp
15 - 19 November 2010, Edison, New Jersey, USA
www.voice-system.ro


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: OpenSIPS core dumps

thrillerbee
Bogdan,

Below is the info requested:
(gdb) frame 0
#0  0x00007f51999f221f in flat_db_insert (h=0x7f0978, k=0x7f5198732a60, v=0x7f5198732d20, n=19) at flatstore.c:165
165             f = CON_FILE(h);
(gdb) print h
$1 = (const db_con_t *) 0x7f0978
(gdb) print h->tail
$2 = 0
(gdb) print ((struct flat_con*)(h->tail))->file
Cannot access memory at address 0x10

Before each of the crashes yesterday, I saw these in the logs:
Nov  1 14:17:40 core1 kernel: [4287745.452111] opensips[22141]: segfault at 10 ip 7f51999f221f sp 7fffbcd8d510 error 4 in db_flatstore.so[7f51999ef000+5000]
Nov  1 23:52:58 core1 kernel: [4348562.990735] opensips[26978]: segfault at 10 ip 7f726cb9b21f sp 7ffff083f6f0 error 4 in db_flatstore.so[7f726cb98000+5000]

Are there any compiler flags I should use for debugging?
(gdb) info locals
f = <value optimized out>
i = <value optimized out>
l = <value optimized out>
p = <value optimized out>
__FUNCTION__ = "flat_db_insert"

Thanks again.


On Tue, Nov 2, 2010 at 4:18 AM, Bogdan-Andrei Iancu <[hidden email]> wrote:
Hi,

in frame 0, could you print:
   h
   h->tail
   ((struct flat_con*)(h->tail))->file

Also, before crash, do you see in the logs any errors from the
db_flatstore module ?

Regards,
Bogdan

thrillerbee wrote:
> Bogdan,
>
> It crashed again tonight.  I've attached the backtrace.
>
> Thanks.
>
> On Mon, Nov 1, 2010 at 9:32 AM, thrillerbee <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     Bogdan,
>
>     Yes,  I've attached 2 to my response - one was a crash from 10/29.
>      The other occurred a few minutes ago.
>
>     Thanks.
>
>     On Fri, Oct 22, 2010 at 11:41 AM, Bogdan-Andrei Iancu
>     <[hidden email] <mailto:[hidden email]>> wrote:
>
>         Any chance with the backtraces ?
>
>         Regards,
>         Bogdan
>
>         Anca Vamanu wrote:
>         > Hi,
>         >
>         > You need to inspect them with gdb, run: gdb
>         > path_to_opensips_executable path_to_corefile, and then run
>         'bt full'
>         > and send the output.
>         >
>         > Regards,
>         > --
>         > Anca Vamanu
>         > www.voice-system.ro <http://www.voice-system.ro>
>         >
>         >
>         > On 10/14/2010 10:12 PM, thrillerbee wrote:
>         >> I have this info from dmesg:
>         >>
>         >> [1985853.285221] opensips[30865]: segfault at 10 ip
>         7f43899ce21f sp
>         >> 7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000]
>         >> [1985856.379671] opensips[30858]: segfault at 10 ip
>         7f43899ce21f sp
>         >> 7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000]
>         >> [1985896.961279] opensips[30868]: segfault at 10 ip
>         7f43899ce21f sp
>         >> 7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000]
>         >>
>         >> [2000131.245512] opensips[17672]: segfault at 10 ip
>         7fd0f21fb21f sp
>         >> 7fff3a3b4f00 error 4 in db_flatstore.so[7fd0f21f8000+5000]
>         >> [2000161.735962] opensips[17668]: segfault at 10 ip
>         7fd0f21fb21f sp
>         >> 7fff3a3b4e40 error 4 in db_flatstore.so[7fd0f21f8000+5000]
>         >> [2000167.299402] opensips[17670]: segfault at 10 ip
>         7fd0f21fb21f sp
>         >> 7fff3a3b4f00 error 4 in db_flatstore.so[7fd0f21f8000+5000]
>         >>
>         >> On Thu, Oct 14, 2010 at 1:57 PM, thrillerbee
>         <[hidden email] <mailto:[hidden email]>
>         >> <mailto:[hidden email]
>         <mailto:[hidden email]>>> wrote:
>         >>
>         >>     When OpenSIPS crashes, three corefiles are generated
>         that are
>         >>     2.1GB in size.  How do I use these files to understand
>         what's
>         >>     causing the crash?
>         >>
>         >>     Thanks.
>         >>
>         >>
>         >
>         ------------------------------------------------------------------------
>         >
>         > _______________________________________________
>         > Users mailing list
>         > [hidden email] <mailto:[hidden email]>
>         > http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>         >
>
>
>         --
>         Bogdan-Andrei Iancu
>         OpenSIPS Bootcamp
>         15 - 19 November 2010, Edison, New Jersey, USA
>         www.voice-system.ro <http://www.voice-system.ro>
>
>
>         _______________________________________________
>         Users mailing list
>         [hidden email] <mailto:[hidden email]>
>         http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Users mailing list
> [hidden email]
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users


--
Bogdan-Andrei Iancu
OpenSIPS Bootcamp
15 - 19 November 2010, Edison, New Jersey, USA
www.voice-system.ro


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: OpenSIPS core dumps

thrillerbee
Bogdan,

One more detail that may help - I added another OpenSIPS proxy in parallel with this one (& load balancing between the two) and now both OpenSIPS proxies crash at the same time (within a couple seconds).

I can provide more core dumps if it will help.

Thanks.


On Tue, Nov 2, 2010 at 9:02 AM, thrillerbee <[hidden email]> wrote:
Bogdan,

Below is the info requested:
(gdb) frame 0
#0  0x00007f51999f221f in flat_db_insert (h=0x7f0978, k=0x7f5198732a60, v=0x7f5198732d20, n=19) at flatstore.c:165
165             f = CON_FILE(h);
(gdb) print h
$1 = (const db_con_t *) 0x7f0978
(gdb) print h->tail
$2 = 0
(gdb) print ((struct flat_con*)(h->tail))->file
Cannot access memory at address 0x10

Before each of the crashes yesterday, I saw these in the logs:
Nov  1 14:17:40 core1 kernel: [4287745.452111] opensips[22141]: segfault at 10 ip 7f51999f221f sp 7fffbcd8d510 error 4 in db_flatstore.so[7f51999ef000+5000]
Nov  1 23:52:58 core1 kernel: [4348562.990735] opensips[26978]: segfault at 10 ip 7f726cb9b21f sp 7ffff083f6f0 error 4 in db_flatstore.so[7f726cb98000+5000]

Are there any compiler flags I should use for debugging?
(gdb) info locals
f = <value optimized out>
i = <value optimized out>
l = <value optimized out>
p = <value optimized out>
__FUNCTION__ = "flat_db_insert"

Thanks again.


On Tue, Nov 2, 2010 at 4:18 AM, Bogdan-Andrei Iancu <[hidden email]> wrote:
Hi,

in frame 0, could you print:
   h
   h->tail
   ((struct flat_con*)(h->tail))->file

Also, before crash, do you see in the logs any errors from the
db_flatstore module ?

Regards,
Bogdan

thrillerbee wrote:
> Bogdan,
>
> It crashed again tonight.  I've attached the backtrace.
>
> Thanks.
>
> On Mon, Nov 1, 2010 at 9:32 AM, thrillerbee <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     Bogdan,
>
>     Yes,  I've attached 2 to my response - one was a crash from 10/29.
>      The other occurred a few minutes ago.
>
>     Thanks.
>
>     On Fri, Oct 22, 2010 at 11:41 AM, Bogdan-Andrei Iancu
>     <[hidden email] <mailto:[hidden email]>> wrote:
>
>         Any chance with the backtraces ?
>
>         Regards,
>         Bogdan
>
>         Anca Vamanu wrote:
>         > Hi,
>         >
>         > You need to inspect them with gdb, run: gdb
>         > path_to_opensips_executable path_to_corefile, and then run
>         'bt full'
>         > and send the output.
>         >
>         > Regards,
>         > --
>         > Anca Vamanu
>         > www.voice-system.ro <http://www.voice-system.ro>
>         >
>         >
>         > On 10/14/2010 10:12 PM, thrillerbee wrote:
>         >> I have this info from dmesg:
>         >>
>         >> [1985853.285221] opensips[30865]: segfault at 10 ip
>         7f43899ce21f sp
>         >> 7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000]
>         >> [1985856.379671] opensips[30858]: segfault at 10 ip
>         7f43899ce21f sp
>         >> 7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000]
>         >> [1985896.961279] opensips[30868]: segfault at 10 ip
>         7f43899ce21f sp
>         >> 7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000]
>         >>
>         >> [2000131.245512] opensips[17672]: segfault at 10 ip
>         7fd0f21fb21f sp
>         >> 7fff3a3b4f00 error 4 in db_flatstore.so[7fd0f21f8000+5000]
>         >> [2000161.735962] opensips[17668]: segfault at 10 ip
>         7fd0f21fb21f sp
>         >> 7fff3a3b4e40 error 4 in db_flatstore.so[7fd0f21f8000+5000]
>         >> [2000167.299402] opensips[17670]: segfault at 10 ip
>         7fd0f21fb21f sp
>         >> 7fff3a3b4f00 error 4 in db_flatstore.so[7fd0f21f8000+5000]
>         >>
>         >> On Thu, Oct 14, 2010 at 1:57 PM, thrillerbee
>         <[hidden email] <mailto:[hidden email]>
>         >> <mailto:[hidden email]
>         <mailto:[hidden email]>>> wrote:
>         >>
>         >>     When OpenSIPS crashes, three corefiles are generated
>         that are
>         >>     2.1GB in size.  How do I use these files to understand
>         what's
>         >>     causing the crash?
>         >>
>         >>     Thanks.
>         >>
>         >>
>         >
>         ------------------------------------------------------------------------
>         >
>         > _______________________________________________
>         > Users mailing list
>         > [hidden email] <mailto:[hidden email]>
>         > http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>         >
>
>
>         --
>         Bogdan-Andrei Iancu
>         OpenSIPS Bootcamp
>         15 - 19 November 2010, Edison, New Jersey, USA
>         www.voice-system.ro <http://www.voice-system.ro>
>
>
>         _______________________________________________
>         Users mailing list
>         [hidden email] <mailto:[hidden email]>
>         http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Users mailing list
> [hidden email]
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users


--
Bogdan-Andrei Iancu
OpenSIPS Bootcamp
15 - 19 November 2010, Edison, New Jersey, USA
www.voice-system.ro


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users



_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: OpenSIPS core dumps

Bogdan-Andrei Iancu
I got some clue about what is happening - as you are using flatstore for
acc, the acc module does not check the success of the "use_table" DB
operation - in 90% of the cases (for mysql, postgres, etc) this function
cannot fail, but it seams that for flatstore can. And if it fails, the
h->tail is set to NULL, leading to crash.

Now, before the crash itself, do you see any other ERROR messages (even
long before the crash) related to flatstore module ? Try:
    grep "ERROR" opensips_log_file | grep "flatstore"

Regards,
Bogdan

thrillerbee wrote:

> Bogdan,
>
> One more detail that may help - I added another OpenSIPS proxy in
> parallel with this one (& load balancing between the two) and now both
> OpenSIPS proxies crash at the same time (within a couple seconds).
>
> I can provide more core dumps if it will help.
>
> Thanks.
>
>
> On Tue, Nov 2, 2010 at 9:02 AM, thrillerbee <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     Bogdan,
>
>     Below is the info requested:
>     (gdb) frame 0
>     #0  0x00007f51999f221f in flat_db_insert (h=0x7f0978,
>     k=0x7f5198732a60, v=0x7f5198732d20, n=19) at flatstore.c:165
>     165             f = CON_FILE(h);
>     (gdb) print h
>     $1 = (const db_con_t *) 0x7f0978
>     (gdb) print h->tail
>     $2 = 0
>     (gdb) print ((struct flat_con*)(h->tail))->file
>     Cannot access memory at address 0x10
>
>     Before each of the crashes yesterday, I saw these in the logs:
>     Nov  1 14:17:40 core1 kernel: [4287745.452111] opensips[22141]:
>     segfault at 10 ip 7f51999f221f sp 7fffbcd8d510 error 4 in
>     db_flatstore.so[7f51999ef000+5000]
>     Nov  1 23:52:58 core1 kernel: [4348562.990735] opensips[26978]:
>     segfault at 10 ip 7f726cb9b21f sp 7ffff083f6f0 error 4 in
>     db_flatstore.so[7f726cb98000+5000]
>
>     Are there any compiler flags I should use for debugging?
>     (gdb) info locals
>     f = <value optimized out>
>     i = <value optimized out>
>     l = <value optimized out>
>     p = <value optimized out>
>     __FUNCTION__ = "flat_db_insert"
>
>     Thanks again.
>
>
>     On Tue, Nov 2, 2010 at 4:18 AM, Bogdan-Andrei Iancu
>     <[hidden email] <mailto:[hidden email]>> wrote:
>
>         Hi,
>
>         in frame 0, could you print:
>            h
>            h->tail
>            ((struct flat_con*)(h->tail))->file
>
>         Also, before crash, do you see in the logs any errors from the
>         db_flatstore module ?
>
>         Regards,
>         Bogdan
>
>         thrillerbee wrote:
>         > Bogdan,
>         >
>         > It crashed again tonight.  I've attached the backtrace.
>         >
>         > Thanks.
>         >
>         > On Mon, Nov 1, 2010 at 9:32 AM, thrillerbee
>         <[hidden email] <mailto:[hidden email]>
>         > <mailto:[hidden email]
>         <mailto:[hidden email]>>> wrote:
>         >
>         >     Bogdan,
>         >
>         >     Yes,  I've attached 2 to my response - one was a crash
>         from 10/29.
>         >      The other occurred a few minutes ago.
>         >
>         >     Thanks.
>         >
>         >     On Fri, Oct 22, 2010 at 11:41 AM, Bogdan-Andrei Iancu
>         >     <[hidden email] <mailto:[hidden email]>
>         <mailto:[hidden email]
>         <mailto:[hidden email]>>> wrote:
>         >
>         >         Any chance with the backtraces ?
>         >
>         >         Regards,
>         >         Bogdan
>         >
>         >         Anca Vamanu wrote:
>         >         > Hi,
>         >         >
>         >         > You need to inspect them with gdb, run: gdb
>         >         > path_to_opensips_executable path_to_corefile, and
>         then run
>         >         'bt full'
>         >         > and send the output.
>         >         >
>         >         > Regards,
>         >         > --
>         >         > Anca Vamanu
>         >         > www.voice-system.ro <http://www.voice-system.ro>
>         <http://www.voice-system.ro>
>         >         >
>         >         >
>         >         > On 10/14/2010 10:12 PM, thrillerbee wrote:
>         >         >> I have this info from dmesg:
>         >         >>
>         >         >> [1985853.285221] opensips[30865]: segfault at 10 ip
>         >         7f43899ce21f sp
>         >         >> 7fff8de1cf40 error 4 in
>         db_flatstore.so[7f43899cb000+5000]
>         >         >> [1985856.379671] opensips[30858]: segfault at 10 ip
>         >         7f43899ce21f sp
>         >         >> 7fff8de1cf40 error 4 in
>         db_flatstore.so[7f43899cb000+5000]
>         >         >> [1985896.961279] opensips[30868]: segfault at 10 ip
>         >         7f43899ce21f sp
>         >         >> 7fff8de1cf40 error 4 in
>         db_flatstore.so[7f43899cb000+5000]
>         >         >>
>         >         >> [2000131.245512] opensips[17672]: segfault at 10 ip
>         >         7fd0f21fb21f sp
>         >         >> 7fff3a3b4f00 error 4 in
>         db_flatstore.so[7fd0f21f8000+5000]
>         >         >> [2000161.735962] opensips[17668]: segfault at 10 ip
>         >         7fd0f21fb21f sp
>         >         >> 7fff3a3b4e40 error 4 in
>         db_flatstore.so[7fd0f21f8000+5000]
>         >         >> [2000167.299402] opensips[17670]: segfault at 10 ip
>         >         7fd0f21fb21f sp
>         >         >> 7fff3a3b4f00 error 4 in
>         db_flatstore.so[7fd0f21f8000+5000]
>         >         >>
>         >         >> On Thu, Oct 14, 2010 at 1:57 PM, thrillerbee
>         >         <[hidden email]
>         <mailto:[hidden email]> <mailto:[hidden email]
>         <mailto:[hidden email]>>
>         >         >> <mailto:[hidden email]
>         <mailto:[hidden email]>
>         >         <mailto:[hidden email]
>         <mailto:[hidden email]>>>> wrote:
>         >         >>
>         >         >>     When OpenSIPS crashes, three corefiles are
>         generated
>         >         that are
>         >         >>     2.1GB in size.  How do I use these files to
>         understand
>         >         what's
>         >         >>     causing the crash?
>         >         >>
>         >         >>     Thanks.
>         >         >>
>         >         >>
>         >         >
>         >        
>         ------------------------------------------------------------------------
>         >         >
>         >         > _______________________________________________
>         >         > Users mailing list
>         >         > [hidden email]
>         <mailto:[hidden email]>
>         <mailto:[hidden email]
>         <mailto:[hidden email]>>
>         >         >
>         http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>         >         >
>         >
>         >
>         >         --
>         >         Bogdan-Andrei Iancu
>         >         OpenSIPS Bootcamp
>         >         15 - 19 November 2010, Edison, New Jersey, USA
>         >         www.voice-system.ro <http://www.voice-system.ro>
>         <http://www.voice-system.ro>
>         >
>         >
>         >         _______________________________________________
>         >         Users mailing list
>         >         [hidden email]
>         <mailto:[hidden email]>
>         <mailto:[hidden email]
>         <mailto:[hidden email]>>
>         >         http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>         >
>         >
>         >
>         >
>         ------------------------------------------------------------------------
>         >
>         > _______________________________________________
>         > Users mailing list
>         > [hidden email] <mailto:[hidden email]>
>         > http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
>
>         --
>         Bogdan-Andrei Iancu
>         OpenSIPS Bootcamp
>         15 - 19 November 2010, Edison, New Jersey, USA
>         www.voice-system.ro <http://www.voice-system.ro>
>
>
>         _______________________________________________
>         Users mailing list
>         [hidden email] <mailto:[hidden email]>
>         http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Users mailing list
> [hidden email]
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>  


--
Bogdan-Andrei Iancu
OpenSIPS Bootcamp
15 - 19 November 2010, Edison, New Jersey, USA
www.voice-system.ro


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: OpenSIPS core dumps

thrillerbee
Bogdan,

I am not seeing any other errors in the logs.  Is there anything else I can look at?  Both proxies are crashing everyday.

Thanks.

On Wed, Nov 3, 2010 at 4:21 AM, Bogdan-Andrei Iancu <[hidden email]> wrote:
I got some clue about what is happening - as you are using flatstore for
acc, the acc module does not check the success of the "use_table" DB
operation - in 90% of the cases (for mysql, postgres, etc) this function
cannot fail, but it seams that for flatstore can. And if it fails, the
h->tail is set to NULL, leading to crash.

Now, before the crash itself, do you see any other ERROR messages (even
long before the crash) related to flatstore module ? Try:
   grep "ERROR" opensips_log_file | grep "flatstore"

Regards,
Bogdan

thrillerbee wrote:
> Bogdan,
>
> One more detail that may help - I added another OpenSIPS proxy in
> parallel with this one (& load balancing between the two) and now both
> OpenSIPS proxies crash at the same time (within a couple seconds).
>
> I can provide more core dumps if it will help.
>
> Thanks.
>
>
> On Tue, Nov 2, 2010 at 9:02 AM, thrillerbee <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     Bogdan,
>
>     Below is the info requested:
>     (gdb) frame 0
>     #0  0x00007f51999f221f in flat_db_insert (h=0x7f0978,
>     k=0x7f5198732a60, v=0x7f5198732d20, n=19) at flatstore.c:165
>     165             f = CON_FILE(h);
>     (gdb) print h
>     $1 = (const db_con_t *) 0x7f0978
>     (gdb) print h->tail
>     $2 = 0
>     (gdb) print ((struct flat_con*)(h->tail))->file
>     Cannot access memory at address 0x10
>
>     Before each of the crashes yesterday, I saw these in the logs:
>     Nov  1 14:17:40 core1 kernel: [4287745.452111] opensips[22141]:
>     segfault at 10 ip 7f51999f221f sp 7fffbcd8d510 error 4 in
>     db_flatstore.so[7f51999ef000+5000]
>     Nov  1 23:52:58 core1 kernel: [4348562.990735] opensips[26978]:
>     segfault at 10 ip 7f726cb9b21f sp 7ffff083f6f0 error 4 in
>     db_flatstore.so[7f726cb98000+5000]
>
>     Are there any compiler flags I should use for debugging?
>     (gdb) info locals
>     f = <value optimized out>
>     i = <value optimized out>
>     l = <value optimized out>
>     p = <value optimized out>
>     __FUNCTION__ = "flat_db_insert"
>
>     Thanks again.
>
>
>     On Tue, Nov 2, 2010 at 4:18 AM, Bogdan-Andrei Iancu
>     <[hidden email] <mailto:[hidden email]>> wrote:
>
>         Hi,
>
>         in frame 0, could you print:
>            h
>            h->tail
>            ((struct flat_con*)(h->tail))->file
>
>         Also, before crash, do you see in the logs any errors from the
>         db_flatstore module ?
>
>         Regards,
>         Bogdan
>
>         thrillerbee wrote:
>         > Bogdan,
>         >
>         > It crashed again tonight.  I've attached the backtrace.
>         >
>         > Thanks.
>         >
>         > On Mon, Nov 1, 2010 at 9:32 AM, thrillerbee
>         <[hidden email] <mailto:[hidden email]>
>         > <mailto:[hidden email]
>         <mailto:[hidden email]>>> wrote:
>         >
>         >     Bogdan,
>         >
>         >     Yes,  I've attached 2 to my response - one was a crash
>         from 10/29.
>         >      The other occurred a few minutes ago.
>         >
>         >     Thanks.
>         >
>         >     On Fri, Oct 22, 2010 at 11:41 AM, Bogdan-Andrei Iancu
>         >     <[hidden email] <mailto:[hidden email]>
>         <mailto:[hidden email]
>         <mailto:[hidden email]>>> wrote:
>         >
>         >         Any chance with the backtraces ?
>         >
>         >         Regards,
>         >         Bogdan
>         >
>         >         Anca Vamanu wrote:
>         >         > Hi,
>         >         >
>         >         > You need to inspect them with gdb, run: gdb
>         >         > path_to_opensips_executable path_to_corefile, and
>         then run
>         >         'bt full'
>         >         > and send the output.
>         >         >
>         >         > Regards,
>         >         > --
>         >         > Anca Vamanu
>         >         > www.voice-system.ro <http://www.voice-system.ro>
>         <http://www.voice-system.ro>
>         >         >
>         >         >
>         >         > On 10/14/2010 10:12 PM, thrillerbee wrote:
>         >         >> I have this info from dmesg:
>         >         >>
>         >         >> [1985853.285221] opensips[30865]: segfault at 10 ip
>         >         7f43899ce21f sp
>         >         >> 7fff8de1cf40 error 4 in
>         db_flatstore.so[7f43899cb000+5000]
>         >         >> [1985856.379671] opensips[30858]: segfault at 10 ip
>         >         7f43899ce21f sp
>         >         >> 7fff8de1cf40 error 4 in
>         db_flatstore.so[7f43899cb000+5000]
>         >         >> [1985896.961279] opensips[30868]: segfault at 10 ip
>         >         7f43899ce21f sp
>         >         >> 7fff8de1cf40 error 4 in
>         db_flatstore.so[7f43899cb000+5000]
>         >         >>
>         >         >> [2000131.245512] opensips[17672]: segfault at 10 ip
>         >         7fd0f21fb21f sp
>         >         >> 7fff3a3b4f00 error 4 in
>         db_flatstore.so[7fd0f21f8000+5000]
>         >         >> [2000161.735962] opensips[17668]: segfault at 10 ip
>         >         7fd0f21fb21f sp
>         >         >> 7fff3a3b4e40 error 4 in
>         db_flatstore.so[7fd0f21f8000+5000]
>         >         >> [2000167.299402] opensips[17670]: segfault at 10 ip
>         >         7fd0f21fb21f sp
>         >         >> 7fff3a3b4f00 error 4 in
>         db_flatstore.so[7fd0f21f8000+5000]
>         >         >>
>         >         >> On Thu, Oct 14, 2010 at 1:57 PM, thrillerbee
>         >         <[hidden email]
>         <mailto:[hidden email]> <mailto:[hidden email]
>         <mailto:[hidden email]>>
>         >         >> <mailto:[hidden email]
>         <mailto:[hidden email]>
>         >         <mailto:[hidden email]
>         <mailto:[hidden email]>>>> wrote:
>         >         >>
>         >         >>     When OpenSIPS crashes, three corefiles are
>         generated
>         >         that are
>         >         >>     2.1GB in size.  How do I use these files to
>         understand
>         >         what's
>         >         >>     causing the crash?
>         >         >>
>         >         >>     Thanks.
>         >         >>
>         >         >>
>         >         >
>         >
>         ------------------------------------------------------------------------
>         >         >
>         >         > _______________________________________________
>         >         > Users mailing list
>         >         > [hidden email]
>         <mailto:[hidden email]>
>         <mailto:[hidden email]
>         <mailto:[hidden email]>>
>         >         >
>         http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>         >         >
>         >
>         >
>         >         --
>         >         Bogdan-Andrei Iancu
>         >         OpenSIPS Bootcamp
>         >         15 - 19 November 2010, Edison, New Jersey, USA
>         >         www.voice-system.ro <http://www.voice-system.ro>
>         <http://www.voice-system.ro>
>         >
>         >
>         >         _______________________________________________
>         >         Users mailing list
>         >         [hidden email]
>         <mailto:[hidden email]>
>         <mailto:[hidden email]
>         <mailto:[hidden email]>>
>         >         http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>         >
>         >
>         >
>         >
>         ------------------------------------------------------------------------
>         >
>         > _______________________________________________
>         > Users mailing list
>         > [hidden email] <mailto:[hidden email]>
>         > http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
>
>         --
>         Bogdan-Andrei Iancu
>         OpenSIPS Bootcamp
>         15 - 19 November 2010, Edison, New Jersey, USA
>         www.voice-system.ro <http://www.voice-system.ro>
>
>
>         _______________________________________________
>         Users mailing list
>         [hidden email] <mailto:[hidden email]>
>         http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Users mailing list
> [hidden email]
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>


--
Bogdan-Andrei Iancu
OpenSIPS Bootcamp
15 - 19 November 2010, Edison, New Jersey, USA
www.voice-system.ro


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: OpenSIPS core dumps

Bogdan-Andrei Iancu
Hi,

strange if you do not have any errors :(....

I just made a fix on both trunk and 1.6 to extend some checks in
flatstore and prevent crashing (even if the DB op will not be executed).

Could you update from SVN and see if stops crashing ?

Regards,
Bogdan


thrillerbee wrote:

> Bogdan,
>
> I am not seeing any other errors in the logs.  Is there anything else
> I can look at?  Both proxies are crashing everyday.
>
> Thanks.
>
> On Wed, Nov 3, 2010 at 4:21 AM, Bogdan-Andrei Iancu
> <[hidden email] <mailto:[hidden email]>> wrote:
>
>     I got some clue about what is happening - as you are using
>     flatstore for
>     acc, the acc module does not check the success of the "use_table" DB
>     operation - in 90% of the cases (for mysql, postgres, etc) this
>     function
>     cannot fail, but it seams that for flatstore can. And if it fails, the
>     h->tail is set to NULL, leading to crash.
>
>     Now, before the crash itself, do you see any other ERROR messages
>     (even
>     long before the crash) related to flatstore module ? Try:
>        grep "ERROR" opensips_log_file | grep "flatstore"
>
>     Regards,
>     Bogdan
>
>     thrillerbee wrote:
>     > Bogdan,
>     >
>     > One more detail that may help - I added another OpenSIPS proxy in
>     > parallel with this one (& load balancing between the two) and
>     now both
>     > OpenSIPS proxies crash at the same time (within a couple seconds).
>     >
>     > I can provide more core dumps if it will help.
>     >
>     > Thanks.
>     >
>     >
>     > On Tue, Nov 2, 2010 at 9:02 AM, thrillerbee
>     <[hidden email] <mailto:[hidden email]>
>     > <mailto:[hidden email] <mailto:[hidden email]>>>
>     wrote:
>     >
>     >     Bogdan,
>     >
>     >     Below is the info requested:
>     >     (gdb) frame 0
>     >     #0  0x00007f51999f221f in flat_db_insert (h=0x7f0978,
>     >     k=0x7f5198732a60, v=0x7f5198732d20, n=19) at flatstore.c:165
>     >     165             f = CON_FILE(h);
>     >     (gdb) print h
>     >     $1 = (const db_con_t *) 0x7f0978
>     >     (gdb) print h->tail
>     >     $2 = 0
>     >     (gdb) print ((struct flat_con*)(h->tail))->file
>     >     Cannot access memory at address 0x10
>     >
>     >     Before each of the crashes yesterday, I saw these in the logs:
>     >     Nov  1 14:17:40 core1 kernel: [4287745.452111] opensips[22141]:
>     >     segfault at 10 ip 7f51999f221f sp 7fffbcd8d510 error 4 in
>     >     db_flatstore.so[7f51999ef000+5000]
>     >     Nov  1 23:52:58 core1 kernel: [4348562.990735] opensips[26978]:
>     >     segfault at 10 ip 7f726cb9b21f sp 7ffff083f6f0 error 4 in
>     >     db_flatstore.so[7f726cb98000+5000]
>     >
>     >     Are there any compiler flags I should use for debugging?
>     >     (gdb) info locals
>     >     f = <value optimized out>
>     >     i = <value optimized out>
>     >     l = <value optimized out>
>     >     p = <value optimized out>
>     >     __FUNCTION__ = "flat_db_insert"
>     >
>     >     Thanks again.
>     >
>     >
>     >     On Tue, Nov 2, 2010 at 4:18 AM, Bogdan-Andrei Iancu
>     >     <[hidden email] <mailto:[hidden email]>
>     <mailto:[hidden email] <mailto:[hidden email]>>>
>     wrote:
>     >
>     >         Hi,
>     >
>     >         in frame 0, could you print:
>     >            h
>     >            h->tail
>     >            ((struct flat_con*)(h->tail))->file
>     >
>     >         Also, before crash, do you see in the logs any errors
>     from the
>     >         db_flatstore module ?
>     >
>     >         Regards,
>     >         Bogdan
>     >
>     >         thrillerbee wrote:
>     >         > Bogdan,
>     >         >
>     >         > It crashed again tonight.  I've attached the backtrace.
>     >         >
>     >         > Thanks.
>     >         >
>     >         > On Mon, Nov 1, 2010 at 9:32 AM, thrillerbee
>     >         <[hidden email] <mailto:[hidden email]>
>     <mailto:[hidden email] <mailto:[hidden email]>>
>     >         > <mailto:[hidden email]
>     <mailto:[hidden email]>
>     >         <mailto:[hidden email]
>     <mailto:[hidden email]>>>> wrote:
>     >         >
>     >         >     Bogdan,
>     >         >
>     >         >     Yes,  I've attached 2 to my response - one was a crash
>     >         from 10/29.
>     >         >      The other occurred a few minutes ago.
>     >         >
>     >         >     Thanks.
>     >         >
>     >         >     On Fri, Oct 22, 2010 at 11:41 AM, Bogdan-Andrei Iancu
>     >         >     <[hidden email]
>     <mailto:[hidden email]> <mailto:[hidden email]
>     <mailto:[hidden email]>>
>     >         <mailto:[hidden email]
>     <mailto:[hidden email]>
>     >         <mailto:[hidden email]
>     <mailto:[hidden email]>>>> wrote:
>     >         >
>     >         >         Any chance with the backtraces ?
>     >         >
>     >         >         Regards,
>     >         >         Bogdan
>     >         >
>     >         >         Anca Vamanu wrote:
>     >         >         > Hi,
>     >         >         >
>     >         >         > You need to inspect them with gdb, run: gdb
>     >         >         > path_to_opensips_executable
>     path_to_corefile, and
>     >         then run
>     >         >         'bt full'
>     >         >         > and send the output.
>     >         >         >
>     >         >         > Regards,
>     >         >         > --
>     >         >         > Anca Vamanu
>     >         >         > www.voice-system.ro
>     <http://www.voice-system.ro> <http://www.voice-system.ro>
>     >         <http://www.voice-system.ro>
>     >         >         >
>     >         >         >
>     >         >         > On 10/14/2010 10:12 PM, thrillerbee wrote:
>     >         >         >> I have this info from dmesg:
>     >         >         >>
>     >         >         >> [1985853.285221] opensips[30865]: segfault
>     at 10 ip
>     >         >         7f43899ce21f sp
>     >         >         >> 7fff8de1cf40 error 4 in
>     >         db_flatstore.so[7f43899cb000+5000]
>     >         >         >> [1985856.379671] opensips[30858]: segfault
>     at 10 ip
>     >         >         7f43899ce21f sp
>     >         >         >> 7fff8de1cf40 error 4 in
>     >         db_flatstore.so[7f43899cb000+5000]
>     >         >         >> [1985896.961279] opensips[30868]: segfault
>     at 10 ip
>     >         >         7f43899ce21f sp
>     >         >         >> 7fff8de1cf40 error 4 in
>     >         db_flatstore.so[7f43899cb000+5000]
>     >         >         >>
>     >         >         >> [2000131.245512] opensips[17672]: segfault
>     at 10 ip
>     >         >         7fd0f21fb21f sp
>     >         >         >> 7fff3a3b4f00 error 4 in
>     >         db_flatstore.so[7fd0f21f8000+5000]
>     >         >         >> [2000161.735962] opensips[17668]: segfault
>     at 10 ip
>     >         >         7fd0f21fb21f sp
>     >         >         >> 7fff3a3b4e40 error 4 in
>     >         db_flatstore.so[7fd0f21f8000+5000]
>     >         >         >> [2000167.299402] opensips[17670]: segfault
>     at 10 ip
>     >         >         7fd0f21fb21f sp
>     >         >         >> 7fff3a3b4f00 error 4 in
>     >         db_flatstore.so[7fd0f21f8000+5000]
>     >         >         >>
>     >         >         >> On Thu, Oct 14, 2010 at 1:57 PM, thrillerbee
>     >         >         <[hidden email]
>     <mailto:[hidden email]>
>     >         <mailto:[hidden email]
>     <mailto:[hidden email]>> <mailto:[hidden email]
>     <mailto:[hidden email]>
>     >         <mailto:[hidden email]
>     <mailto:[hidden email]>>>
>     >         >         >> <mailto:[hidden email]
>     <mailto:[hidden email]>
>     >         <mailto:[hidden email]
>     <mailto:[hidden email]>>
>     >         >         <mailto:[hidden email]
>     <mailto:[hidden email]>
>     >         <mailto:[hidden email]
>     <mailto:[hidden email]>>>>> wrote:
>     >         >         >>
>     >         >         >>     When OpenSIPS crashes, three corefiles are
>     >         generated
>     >         >         that are
>     >         >         >>     2.1GB in size.  How do I use these files to
>     >         understand
>     >         >         what's
>     >         >         >>     causing the crash?
>     >         >         >>
>     >         >         >>     Thanks.
>     >         >         >>
>     >  
>


--
Bogdan-Andrei Iancu
OpenSIPS Bootcamp
15 - 19 November 2010, Edison, New Jersey, USA
www.voice-system.ro


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: OpenSIPS core dumps

thrillerbee
Bogdan,

It seems the issue is with 'opensipsctl fifo' - it's very sensitive to simultaneous calls.  Basically, I've combined all my scripts to prevent 'opensipsctl fifo' from being called too frequently and that seems (so far) to have mitigated the issue.  Is there anything one should know about how (not) to use opensipsctl?

Thanks.

On Mon, Nov 8, 2010 at 6:07 AM, Bogdan-Andrei Iancu <[hidden email]> wrote:
Hi,

strange if you do not have any errors :(....

I just made a fix on both trunk and 1.6 to extend some checks in
flatstore and prevent crashing (even if the DB op will not be executed).

Could you update from SVN and see if stops crashing ?

Regards,
Bogdan


thrillerbee wrote:
> Bogdan,
>
> I am not seeing any other errors in the logs.  Is there anything else
> I can look at?  Both proxies are crashing everyday.
>
> Thanks.
>
> On Wed, Nov 3, 2010 at 4:21 AM, Bogdan-Andrei Iancu
> <[hidden email] <mailto:[hidden email]>> wrote:
>
>     I got some clue about what is happening - as you are using
>     flatstore for
>     acc, the acc module does not check the success of the "use_table" DB
>     operation - in 90% of the cases (for mysql, postgres, etc) this
>     function
>     cannot fail, but it seams that for flatstore can. And if it fails, the
>     h->tail is set to NULL, leading to crash.
>
>     Now, before the crash itself, do you see any other ERROR messages
>     (even
>     long before the crash) related to flatstore module ? Try:
>        grep "ERROR" opensips_log_file | grep "flatstore"
>
>     Regards,
>     Bogdan
>
>     thrillerbee wrote:
>     > Bogdan,
>     >
>     > One more detail that may help - I added another OpenSIPS proxy in
>     > parallel with this one (& load balancing between the two) and
>     now both
>     > OpenSIPS proxies crash at the same time (within a couple seconds).
>     >
>     > I can provide more core dumps if it will help.
>     >
>     > Thanks.
>     >
>     >
>     > On Tue, Nov 2, 2010 at 9:02 AM, thrillerbee
>     <[hidden email] <mailto:[hidden email]>
>     > <mailto:[hidden email] <mailto:[hidden email]>>>
>     wrote:
>     >
>     >     Bogdan,
>     >
>     >     Below is the info requested:
>     >     (gdb) frame 0
>     >     #0  0x00007f51999f221f in flat_db_insert (h=0x7f0978,
>     >     k=0x7f5198732a60, v=0x7f5198732d20, n=19) at flatstore.c:165
>     >     165             f = CON_FILE(h);
>     >     (gdb) print h
>     >     $1 = (const db_con_t *) 0x7f0978
>     >     (gdb) print h->tail
>     >     $2 = 0
>     >     (gdb) print ((struct flat_con*)(h->tail))->file
>     >     Cannot access memory at address 0x10
>     >
>     >     Before each of the crashes yesterday, I saw these in the logs:
>     >     Nov  1 14:17:40 core1 kernel: [4287745.452111] opensips[22141]:
>     >     segfault at 10 ip 7f51999f221f sp 7fffbcd8d510 error 4 in
>     >     db_flatstore.so[7f51999ef000+5000]
>     >     Nov  1 23:52:58 core1 kernel: [4348562.990735] opensips[26978]:
>     >     segfault at 10 ip 7f726cb9b21f sp 7ffff083f6f0 error 4 in
>     >     db_flatstore.so[7f726cb98000+5000]
>     >
>     >     Are there any compiler flags I should use for debugging?
>     >     (gdb) info locals
>     >     f = <value optimized out>
>     >     i = <value optimized out>
>     >     l = <value optimized out>
>     >     p = <value optimized out>
>     >     __FUNCTION__ = "flat_db_insert"
>     >
>     >     Thanks again.
>     >
>     >
>     >     On Tue, Nov 2, 2010 at 4:18 AM, Bogdan-Andrei Iancu
>     >     <[hidden email] <mailto:[hidden email]>
>     <mailto:[hidden email] <mailto:[hidden email]>>>
>     wrote:
>     >
>     >         Hi,
>     >
>     >         in frame 0, could you print:
>     >            h
>     >            h->tail
>     >            ((struct flat_con*)(h->tail))->file
>     >
>     >         Also, before crash, do you see in the logs any errors
>     from the
>     >         db_flatstore module ?
>     >
>     >         Regards,
>     >         Bogdan
>     >
>     >         thrillerbee wrote:
>     >         > Bogdan,
>     >         >
>     >         > It crashed again tonight.  I've attached the backtrace.
>     >         >
>     >         > Thanks.
>     >         >
>     >         > On Mon, Nov 1, 2010 at 9:32 AM, thrillerbee
>     >         <[hidden email] <mailto:[hidden email]>
>     <mailto:[hidden email] <mailto:[hidden email]>>
>     >         > <mailto:[hidden email]
>     <mailto:[hidden email]>
>     >         <mailto:[hidden email]
>     <mailto:[hidden email]>>>> wrote:
>     >         >
>     >         >     Bogdan,
>     >         >
>     >         >     Yes,  I've attached 2 to my response - one was a crash
>     >         from 10/29.
>     >         >      The other occurred a few minutes ago.
>     >         >
>     >         >     Thanks.
>     >         >
>     >         >     On Fri, Oct 22, 2010 at 11:41 AM, Bogdan-Andrei Iancu
>     >         >     <[hidden email]
>     <mailto:[hidden email]> <mailto:[hidden email]
>     <mailto:[hidden email]>>
>     >         <mailto:[hidden email]
>     <mailto:[hidden email]>
>     >         <mailto:[hidden email]
>     <mailto:[hidden email]>>>> wrote:
>     >         >
>     >         >         Any chance with the backtraces ?
>     >         >
>     >         >         Regards,
>     >         >         Bogdan
>     >         >
>     >         >         Anca Vamanu wrote:
>     >         >         > Hi,
>     >         >         >
>     >         >         > You need to inspect them with gdb, run: gdb
>     >         >         > path_to_opensips_executable
>     path_to_corefile, and
>     >         then run
>     >         >         'bt full'
>     >         >         > and send the output.
>     >         >         >
>     >         >         > Regards,
>     >         >         > --
>     >         >         > Anca Vamanu
>     >         >         > www.voice-system.ro
>     <http://www.voice-system.ro> <http://www.voice-system.ro>
>     >         <http://www.voice-system.ro>
>     >         >         >
>     >         >         >
>     >         >         > On 10/14/2010 10:12 PM, thrillerbee wrote:
>     >         >         >> I have this info from dmesg:
>     >         >         >>
>     >         >         >> [1985853.285221] opensips[30865]: segfault
>     at 10 ip
>     >         >         7f43899ce21f sp
>     >         >         >> 7fff8de1cf40 error 4 in
>     >         db_flatstore.so[7f43899cb000+5000]
>     >         >         >> [1985856.379671] opensips[30858]: segfault
>     at 10 ip
>     >         >         7f43899ce21f sp
>     >         >         >> 7fff8de1cf40 error 4 in
>     >         db_flatstore.so[7f43899cb000+5000]
>     >         >         >> [1985896.961279] opensips[30868]: segfault
>     at 10 ip
>     >         >         7f43899ce21f sp
>     >         >         >> 7fff8de1cf40 error 4 in
>     >         db_flatstore.so[7f43899cb000+5000]
>     >         >         >>
>     >         >         >> [2000131.245512] opensips[17672]: segfault
>     at 10 ip
>     >         >         7fd0f21fb21f sp
>     >         >         >> 7fff3a3b4f00 error 4 in
>     >         db_flatstore.so[7fd0f21f8000+5000]
>     >         >         >> [2000161.735962] opensips[17668]: segfault
>     at 10 ip
>     >         >         7fd0f21fb21f sp
>     >         >         >> 7fff3a3b4e40 error 4 in
>     >         db_flatstore.so[7fd0f21f8000+5000]
>     >         >         >> [2000167.299402] opensips[17670]: segfault
>     at 10 ip
>     >         >         7fd0f21fb21f sp
>     >         >         >> 7fff3a3b4f00 error 4 in
>     >         db_flatstore.so[7fd0f21f8000+5000]
>     >         >         >>
>     >         >         >> On Thu, Oct 14, 2010 at 1:57 PM, thrillerbee
>     >         >         <[hidden email]
>     <mailto:[hidden email]>
>     >         <mailto:[hidden email]
>     <mailto:[hidden email]>> <mailto:[hidden email]
>     <mailto:[hidden email]>
>     >         <mailto:[hidden email]
>     <mailto:[hidden email]>>>
>     >         >         >> <mailto:[hidden email]
>     <mailto:[hidden email]>
>     >         <mailto:[hidden email]
>     <mailto:[hidden email]>>
>     >         >         <mailto:[hidden email]
>     <mailto:[hidden email]>
>     >         <mailto:[hidden email]
>     <mailto:[hidden email]>>>>> wrote:
>     >         >         >>
>     >         >         >>     When OpenSIPS crashes, three corefiles are
>     >         generated
>     >         >         that are
>     >         >         >>     2.1GB in size.  How do I use these files to
>     >         understand
>     >         >         what's
>     >         >         >>     causing the crash?
>     >         >         >>
>     >         >         >>     Thanks.
>     >         >         >>
>     >
>


--
Bogdan-Andrei Iancu
OpenSIPS Bootcamp
15 - 19 November 2010, Edison, New Jersey, USA
www.voice-system.ro


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: OpenSIPS core dumps

Bogdan-Andrei Iancu
Hi,

opensipsctl takes care that each command takes a separate fifo reply, so
here it should be no problem. But the problem may be when comes with
sending multiple commands (via FIFO) in the same time - this translates
into parallel writes to the same file and depends on the atomicity of
the write op.

But in the worst case, a mixture at the FIFO level may lead to bogus
command and not in any kind of crash....Do you use the "flat_rotate" MI
command ?

Regards,
Bogdan

thrillerbee wrote:

> Bogdan,
>
> It seems the issue is with 'opensipsctl fifo' - it's very sensitive to
> simultaneous calls.  Basically, I've combined all my scripts to
> prevent 'opensipsctl fifo' from being called too frequently and that
> seems (so far) to have mitigated the issue.  Is there anything one
> should know about how (not) to use /opensipsctl/?
>
> Thanks.
>
> On Mon, Nov 8, 2010 at 6:07 AM, Bogdan-Andrei Iancu
> <[hidden email] <mailto:[hidden email]>> wrote:
>
>     Hi,
>
>     strange if you do not have any errors :(....
>
>     I just made a fix on both trunk and 1.6 to extend some checks in
>     flatstore and prevent crashing (even if the DB op will not be
>     executed).
>
>     Could you update from SVN and see if stops crashing ?
>
>     Regards,
>     Bogdan
>
>
>     thrillerbee wrote:
>     > Bogdan,
>     >
>     > I am not seeing any other errors in the logs.  Is there anything
>     else
>     > I can look at?  Both proxies are crashing everyday.
>     >
>     > Thanks.
>     >
>     > On Wed, Nov 3, 2010 at 4:21 AM, Bogdan-Andrei Iancu
>     > <[hidden email] <mailto:[hidden email]>
>     <mailto:[hidden email] <mailto:[hidden email]>>>
>     wrote:
>     >
>     >     I got some clue about what is happening - as you are using
>     >     flatstore for
>     >     acc, the acc module does not check the success of the
>     "use_table" DB
>     >     operation - in 90% of the cases (for mysql, postgres, etc) this
>     >     function
>     >     cannot fail, but it seams that for flatstore can. And if it
>     fails, the
>     >     h->tail is set to NULL, leading to crash.
>     >
>     >     Now, before the crash itself, do you see any other ERROR
>     messages
>     >     (even
>     >     long before the crash) related to flatstore module ? Try:
>     >        grep "ERROR" opensips_log_file | grep "flatstore"
>     >
>     >     Regards,
>     >     Bogdan
>     >
>     >     thrillerbee wrote:
>     >     > Bogdan,
>     >     >
>     >     > One more detail that may help - I added another OpenSIPS
>     proxy in
>     >     > parallel with this one (& load balancing between the two) and
>     >     now both
>     >     > OpenSIPS proxies crash at the same time (within a couple
>     seconds).
>     >     >
>     >     > I can provide more core dumps if it will help.
>     >     >
>     >     > Thanks.
>     >     >
>     >     >
>     >     > On Tue, Nov 2, 2010 at 9:02 AM, thrillerbee
>     >     <[hidden email] <mailto:[hidden email]>
>     <mailto:[hidden email] <mailto:[hidden email]>>
>     >     > <mailto:[hidden email]
>     <mailto:[hidden email]> <mailto:[hidden email]
>     <mailto:[hidden email]>>>>
>     >     wrote:
>     >     >
>     >     >     Bogdan,
>     >     >
>     >     >     Below is the info requested:
>     >     >     (gdb) frame 0
>     >     >     #0  0x00007f51999f221f in flat_db_insert (h=0x7f0978,
>     >     >     k=0x7f5198732a60, v=0x7f5198732d20, n=19) at
>     flatstore.c:165
>     >     >     165             f = CON_FILE(h);
>     >     >     (gdb) print h
>     >     >     $1 = (const db_con_t *) 0x7f0978
>     >     >     (gdb) print h->tail
>     >     >     $2 = 0
>     >     >     (gdb) print ((struct flat_con*)(h->tail))->file
>     >     >     Cannot access memory at address 0x10
>     >     >
>     >     >     Before each of the crashes yesterday, I saw these in
>     the logs:
>     >     >     Nov  1 14:17:40 core1 kernel: [4287745.452111]
>     opensips[22141]:
>     >     >     segfault at 10 ip 7f51999f221f sp 7fffbcd8d510 error 4 in
>     >     >     db_flatstore.so[7f51999ef000+5000]
>     >     >     Nov  1 23:52:58 core1 kernel: [4348562.990735]
>     opensips[26978]:
>     >     >     segfault at 10 ip 7f726cb9b21f sp 7ffff083f6f0 error 4 in
>     >     >     db_flatstore.so[7f726cb98000+5000]
>     >     >
>     >     >     Are there any compiler flags I should use for debugging?
>     >     >     (gdb) info locals
>     >     >     f = <value optimized out>
>     >     >     i = <value optimized out>
>     >     >     l = <value optimized out>
>     >     >     p = <value optimized out>
>     >     >     __FUNCTION__ = "flat_db_insert"
>     >     >
>     >     >     Thanks again.
>     >     >
>     >     >
>     >     >     On Tue, Nov 2, 2010 at 4:18 AM, Bogdan-Andrei Iancu
>     >     >     <[hidden email]
>     <mailto:[hidden email]> <mailto:[hidden email]
>     <mailto:[hidden email]>>
>     >     <mailto:[hidden email]
>     <mailto:[hidden email]> <mailto:[hidden email]
>     <mailto:[hidden email]>>>>
>     >     wrote:
>     >     >
>     >     >         Hi,
>     >     >
>     >     >         in frame 0, could you print:
>     >     >            h
>     >     >            h->tail
>     >     >            ((struct flat_con*)(h->tail))->file
>     >     >
>     >     >         Also, before crash, do you see in the logs any errors
>     >     from the
>     >     >         db_flatstore module ?
>     >     >
>     >     >         Regards,
>     >     >         Bogdan
>     >     >
>     >     >         thrillerbee wrote:
>     >     >         > Bogdan,
>     >     >         >
>     >     >         > It crashed again tonight.  I've attached the
>     backtrace.
>     >     >         >
>     >     >         > Thanks.
>     >     >         >
>     >     >         > On Mon, Nov 1, 2010 at 9:32 AM, thrillerbee
>     >     >         <[hidden email]
>     <mailto:[hidden email]> <mailto:[hidden email]
>     <mailto:[hidden email]>>
>     >     <mailto:[hidden email] <mailto:[hidden email]>
>     <mailto:[hidden email] <mailto:[hidden email]>>>
>     >     >         > <mailto:[hidden email]
>     <mailto:[hidden email]>
>     >     <mailto:[hidden email] <mailto:[hidden email]>>
>     >     >         <mailto:[hidden email]
>     <mailto:[hidden email]>
>     >     <mailto:[hidden email]
>     <mailto:[hidden email]>>>>> wrote:
>     >     >         >
>     >     >         >     Bogdan,
>     >     >         >
>     >     >         >     Yes,  I've attached 2 to my response - one
>     was a crash
>     >     >         from 10/29.
>     >     >         >      The other occurred a few minutes ago.
>     >     >         >
>     >     >         >     Thanks.
>     >     >         >
>     >     >         >     On Fri, Oct 22, 2010 at 11:41 AM,
>     Bogdan-Andrei Iancu
>     >     >         >     <[hidden email]
>     <mailto:[hidden email]>
>     >     <mailto:[hidden email]
>     <mailto:[hidden email]>> <mailto:[hidden email]
>     <mailto:[hidden email]>
>     >     <mailto:[hidden email] <mailto:[hidden email]>>>
>     >     >         <mailto:[hidden email]
>     <mailto:[hidden email]>
>     >     <mailto:[hidden email] <mailto:[hidden email]>>
>     >     >         <mailto:[hidden email]
>     <mailto:[hidden email]>
>     >     <mailto:[hidden email]
>     <mailto:[hidden email]>>>>> wrote:
>     >     >         >
>     >     >         >         Any chance with the backtraces ?
>     >     >         >
>     >     >         >         Regards,
>     >     >         >         Bogdan
>     >     >         >
>     >     >         >         Anca Vamanu wrote:
>     >     >         >         > Hi,
>     >     >         >         >
>     >     >         >         > You need to inspect them with gdb,
>     run: gdb
>     >     >         >         > path_to_opensips_executable
>     >     path_to_corefile, and
>     >     >         then run
>     >     >         >         'bt full'
>     >     >         >         > and send the output.
>     >     >         >         >
>     >     >         >         > Regards,
>     >     >         >         > --
>     >     >         >         > Anca Vamanu
>     >     >         >         > www.voice-system.ro
>     <http://www.voice-system.ro>
>     >     <http://www.voice-system.ro> <http://www.voice-system.ro>
>     >     >         <http://www.voice-system.ro>
>     >     >         >         >
>     >     >         >         >
>     >     >         >         > On 10/14/2010 10:12 PM, thrillerbee wrote:
>     >     >         >         >> I have this info from dmesg:
>     >     >         >         >>
>     >     >         >         >> [1985853.285221] opensips[30865]:
>     segfault
>     >     at 10 ip
>     >     >         >         7f43899ce21f sp
>     >     >         >         >> 7fff8de1cf40 error 4 in
>     >     >         db_flatstore.so[7f43899cb000+5000]
>     >     >         >         >> [1985856.379671] opensips[30858]:
>     segfault
>     >     at 10 ip
>     >     >         >         7f43899ce21f sp
>     >     >         >         >> 7fff8de1cf40 error 4 in
>     >     >         db_flatstore.so[7f43899cb000+5000]
>     >     >         >         >> [1985896.961279] opensips[30868]:
>     segfault
>     >     at 10 ip
>     >     >         >         7f43899ce21f sp
>     >     >         >         >> 7fff8de1cf40 error 4 in
>     >     >         db_flatstore.so[7f43899cb000+5000]
>     >     >         >         >>
>     >     >         >         >> [2000131.245512] opensips[17672]:
>     segfault
>     >     at 10 ip
>     >     >         >         7fd0f21fb21f sp
>     >     >         >         >> 7fff3a3b4f00 error 4 in
>     >     >         db_flatstore.so[7fd0f21f8000+5000]
>     >     >         >         >> [2000161.735962] opensips[17668]:
>     segfault
>     >     at 10 ip
>     >     >         >         7fd0f21fb21f sp
>     >     >         >         >> 7fff3a3b4e40 error 4 in
>     >     >         db_flatstore.so[7fd0f21f8000+5000]
>     >     >         >         >> [2000167.299402] opensips[17670]:
>     segfault
>     >     at 10 ip
>     >     >         >         7fd0f21fb21f sp
>     >     >         >         >> 7fff3a3b4f00 error 4 in
>     >     >         db_flatstore.so[7fd0f21f8000+5000]
>     >     >         >         >>
>     >     >         >         >> On Thu, Oct 14, 2010 at 1:57 PM,
>     thrillerbee
>     >     >         >         <[hidden email]
>     <mailto:[hidden email]>
>     >     <mailto:[hidden email] <mailto:[hidden email]>>
>     >     >         <mailto:[hidden email]
>     <mailto:[hidden email]>
>     >     <mailto:[hidden email]
>     <mailto:[hidden email]>>> <mailto:[hidden email]
>     <mailto:[hidden email]>
>     >     <mailto:[hidden email] <mailto:[hidden email]>>
>     >     >         <mailto:[hidden email]
>     <mailto:[hidden email]>
>     >     <mailto:[hidden email] <mailto:[hidden email]>>>>
>     >     >         >         >> <mailto:[hidden email]
>     <mailto:[hidden email]>
>     >     <mailto:[hidden email] <mailto:[hidden email]>>
>     >     >         <mailto:[hidden email]
>     <mailto:[hidden email]>
>     >     <mailto:[hidden email] <mailto:[hidden email]>>>
>     >     >         >         <mailto:[hidden email]
>     <mailto:[hidden email]>
>     >     <mailto:[hidden email] <mailto:[hidden email]>>
>     >     >         <mailto:[hidden email]
>     <mailto:[hidden email]>
>     >     <mailto:[hidden email]
>     <mailto:[hidden email]>>>>>> wrote:
>     >     >         >         >>
>     >     >         >         >>     When OpenSIPS crashes, three
>     corefiles are
>     >     >         generated
>     >     >         >         that are
>     >     >         >         >>     2.1GB in size.  How do I use
>     these files to
>     >     >         understand
>     >     >         >         what's
>     >     >         >         >>     causing the crash?
>     >     >         >         >>
>     >     >         >         >>     Thanks.
>     >     >         >         >>
>     >     >
>     >
>
>
>     --
>     Bogdan-Andrei Iancu
>     OpenSIPS Bootcamp
>     15 - 19 November 2010, Edison, New Jersey, USA
>     www.voice-system.ro <http://www.voice-system.ro>
>
>
>     _______________________________________________
>     Users mailing list
>     [hidden email] <mailto:[hidden email]>
>     http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Users mailing list
> [hidden email]
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>  


--
Bogdan-Andrei Iancu
OpenSIPS Bootcamp
15 - 19 November 2010, Edison, New Jersey, USA
www.voice-system.ro


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: OpenSIPS core dumps

thrillerbee
Bogdan,

Well, I spoke too soon - it's not just an issue with the opensipsctl fifo calls - looks more like a memory leak.  It crashed again today, but I did get some errors in the syslog this time right before the crash:
Nov 10 15:42:32 core1 /usr/local/sbin/opensips[27044]: ERROR:db_flatstore:new_flat_id: no pkg memory left
Nov 10 15:42:32 core1 kernel: [5508366.582447] opensips[27044]: segfault at 10 ip 7fa7ff74c21f sp 7fffdc101700 error 4 in db_flatstore.so[7fa7ff749000+5000]
To be thorough, I've attached the backtrace & output from print commands (although they're the same as before).

To answer your question, yes - I do use the flat_rotate MI command.

Thanks again.

On Wed, Nov 10, 2010 at 4:04 AM, Bogdan-Andrei Iancu <[hidden email]> wrote:
Hi,

opensipsctl takes care that each command takes a separate fifo reply, so here it should be no problem. But the problem may be when comes with sending multiple commands (via FIFO) in the same time - this translates into parallel writes to the same file and depends on the atomicity of the write op.

But in the worst case, a mixture at the FIFO level may lead to bogus command and not in any kind of crash....Do you use the "flat_rotate" MI command ?

Regards,
Bogdan

thrillerbee wrote:
Bogdan,

It seems the issue is with 'opensipsctl fifo' - it's very sensitive to simultaneous calls.  Basically, I've combined all my scripts to prevent 'opensipsctl fifo' from being called too frequently and that seems (so far) to have mitigated the issue.  Is there anything one should know about how (not) to use /opensipsctl/?

Thanks.

On Mon, Nov 8, 2010 at 6:07 AM, Bogdan-Andrei Iancu <[hidden email] <mailto:[hidden email]>> wrote:

   Hi,

   strange if you do not have any errors :(....

   I just made a fix on both trunk and 1.6 to extend some checks in
   flatstore and prevent crashing (even if the DB op will not be
   executed).

   Could you update from SVN and see if stops crashing ?

   Regards,
   Bogdan

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users

core1_111010_e.txt (7K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: OpenSIPS core dumps

thrillerbee
My other proxy crashed as well with these ERRORs in the syslog:

Nov 10 22:01:02 core2 /usr/local/sbin/opensips[22959]: ERROR:db_flatstore:get_name: pkg memory allocation failure
Nov 10 22:01:02 core2 /usr/local/sbin/opensips[22959]: ERROR:db_flatstore:flat_reopen_connection: failed to get_name
Nov 10 22:01:02 core2 /usr/local/sbin/opensips[22959]: ERROR:db_flatstore:flat_db_insert: uninitialized connection
Nov 10 22:01:02 core2 /usr/local/sbin/opensips[22959]: ERROR:db_flatstore:flat_db_insert: uninitialized connection
...
Nov 10 22:01:21 core2 /usr/local/sbin/opensips[22959]: ERROR:db_flatstore:flat_db_insert: uninitialized connection
Nov 10 22:01:22 core2 /usr/local/sbin/opensips[22959]: ERROR:db_flatstore:flat_db_insert: uninitialized connection
Nov 10 22:01:22 core2 /usr/local/sbin/opensips[22959]: ERROR:db_flatstore:new_flat_id: no pkg memory left
Nov 10 22:01:22 core2 kernel: [4297088.404734] opensips[22959]: segfault at 10 ip 7f3db577e21f sp 7fffa260d640 error 4 in db_flatstore.so[7f3db577b000+5000]

On Wed, Nov 10, 2010 at 10:19 AM, thrillerbee <[hidden email]> wrote:
Bogdan,

Well, I spoke too soon - it's not just an issue with the opensipsctl fifo calls - looks more like a memory leak.  It crashed again today, but I did get some errors in the syslog this time right before the crash:
Nov 10 15:42:32 core1 /usr/local/sbin/opensips[27044]: ERROR:db_flatstore:new_flat_id: no pkg memory left
Nov 10 15:42:32 core1 kernel: [5508366.582447] opensips[27044]: segfault at 10 ip 7fa7ff74c21f sp 7fffdc101700 error 4 in db_flatstore.so[7fa7ff749000+5000]
To be thorough, I've attached the backtrace & output from print commands (although they're the same as before).

To answer your question, yes - I do use the flat_rotate MI command.

Thanks again.

On Wed, Nov 10, 2010 at 4:04 AM, Bogdan-Andrei Iancu <[hidden email]> wrote:
Hi,

opensipsctl takes care that each command takes a separate fifo reply, so here it should be no problem. But the problem may be when comes with sending multiple commands (via FIFO) in the same time - this translates into parallel writes to the same file and depends on the atomicity of the write op.

But in the worst case, a mixture at the FIFO level may lead to bogus command and not in any kind of crash....Do you use the "flat_rotate" MI command ?

Regards,
Bogdan

thrillerbee wrote:
Bogdan,

It seems the issue is with 'opensipsctl fifo' - it's very sensitive to simultaneous calls.  Basically, I've combined all my scripts to prevent 'opensipsctl fifo' from being called too frequently and that seems (so far) to have mitigated the issue.  Is there anything one should know about how (not) to use /opensipsctl/?

Thanks.

On Mon, Nov 8, 2010 at 6:07 AM, Bogdan-Andrei Iancu <[hidden email] <mailto:[hidden email]>> wrote:

   Hi,

   strange if you do not have any errors :(....

   I just made a fix on both trunk and 1.6 to extend some checks in
   flatstore and prevent crashing (even if the DB op will not be
   executed).

   Could you update from SVN and see if stops crashing ?

   Regards,
   Bogdan


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: OpenSIPS core dumps

Bogdan-Andrei Iancu
This last crash had the same bt as the one from previous email ? the
prev fix I made reports this:
          ERROR:db_flatstore:flat_db_insert: uninitialized connection
and does not crash, so this time the crash may be in a different place.

Now, it seams after all that the root of your problem is the mem
exhaustion. To see what  causes this  (if a mem leak or simply not
enough mem), there is a doc -
http://www.opensips.org/Resources/DocsTsMem (it is for shm, but the same
for pkg).

Maybe, first you should simply try to increase the pkg mem ( in config.h
you have PKG_MEM_POOL_SIZE that you can increase) - you NEED to
recompile and reinstall after that.

If more pkg mem does not solve the problem, I can help you with using
the memory debugger to see what is going on there.

Regards,
Bogdan


thrillerbee wrote:

> My other proxy crashed as well with these ERRORs in the syslog:
>
> Nov 10 22:01:02 core2 /usr/local/sbin/opensips[22959]:
> ERROR:db_flatstore:get_name: pkg memory allocation failure
> Nov 10 22:01:02 core2 /usr/local/sbin/opensips[22959]:
> ERROR:db_flatstore:flat_reopen_connection: failed to get_name
> Nov 10 22:01:02 core2 /usr/local/sbin/opensips[22959]:
> ERROR:db_flatstore:flat_db_insert: uninitialized connection
> Nov 10 22:01:02 core2 /usr/local/sbin/opensips[22959]:
> ERROR:db_flatstore:flat_db_insert: uninitialized connection
> ...
> Nov 10 22:01:21 core2 /usr/local/sbin/opensips[22959]:
> ERROR:db_flatstore:flat_db_insert: uninitialized connection
> Nov 10 22:01:22 core2 /usr/local/sbin/opensips[22959]:
> ERROR:db_flatstore:flat_db_insert: uninitialized connection
> Nov 10 22:01:22 core2 /usr/local/sbin/opensips[22959]:
> ERROR:db_flatstore:new_flat_id: no pkg memory left
> Nov 10 22:01:22 core2 kernel: [4297088.404734] opensips[22959]:
> segfault at 10 ip 7f3db577e21f sp 7fffa260d640 error 4 in
> db_flatstore.so[7f3db577b000+5000]
>
> On Wed, Nov 10, 2010 at 10:19 AM, thrillerbee <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     Bogdan,
>
>     Well, I spoke too soon - it's not just an issue with the
>     opensipsctl fifo calls - looks more like a memory leak.  It
>     crashed again today, but I did get some errors in the syslog this
>     time right before the crash:
>     Nov 10 15:42:32 core1 /usr/local/sbin/opensips[27044]:
>     ERROR:db_flatstore:new_flat_id: no pkg memory left
>     Nov 10 15:42:32 core1 kernel: [5508366.582447] opensips[27044]:
>     segfault at 10 ip 7fa7ff74c21f sp 7fffdc101700 error 4 in
>     db_flatstore.so[7fa7ff749000+5000]
>     To be thorough, I've attached the backtrace & output from print
>     commands (although they're the same as before).
>
>     To answer your question, yes - I do use the flat_rotate MI command.
>
>     Thanks again.
>
>     On Wed, Nov 10, 2010 at 4:04 AM, Bogdan-Andrei Iancu
>     <[hidden email] <mailto:[hidden email]>> wrote:
>
>         Hi,
>
>         opensipsctl takes care that each command takes a separate fifo
>         reply, so here it should be no problem. But the problem may be
>         when comes with sending multiple commands (via FIFO) in the
>         same time - this translates into parallel writes to the same
>         file and depends on the atomicity of the write op.
>
>         But in the worst case, a mixture at the FIFO level may lead to
>         bogus command and not in any kind of crash....Do you use the
>         "flat_rotate" MI command ?
>
>         Regards,
>         Bogdan
>
>         thrillerbee wrote:
>
>             Bogdan,
>
>             It seems the issue is with 'opensipsctl fifo' - it's very
>             sensitive to simultaneous calls.  Basically, I've combined
>             all my scripts to prevent 'opensipsctl fifo' from being
>             called too frequently and that seems (so far) to have
>             mitigated the issue.  Is there anything one should know
>             about how (not) to use /opensipsctl/?
>
>             Thanks.
>
>             On Mon, Nov 8, 2010 at 6:07 AM, Bogdan-Andrei Iancu
>             <[hidden email] <mailto:[hidden email]>
>             <mailto:[hidden email]
>             <mailto:[hidden email]>>> wrote:
>
>                Hi,
>
>                strange if you do not have any errors :(....
>
>                I just made a fix on both trunk and 1.6 to extend some
>             checks in
>                flatstore and prevent crashing (even if the DB op will
>             not be
>                executed).
>
>                Could you update from SVN and see if stops crashing ?
>
>                Regards,
>                Bogdan
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Users mailing list
> [hidden email]
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>  


--
Bogdan-Andrei Iancu
OpenSIPS Bootcamp
15 - 19 November 2010, Edison, New Jersey, USA
www.voice-system.ro


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: OpenSIPS core dumps

thrillerbee
Bogdan,

I had already increased the PKG_MEM_POOL_SIZE by *10.  I'm making the changes suggested in the link you provided to try to narrow down the memory issue.  I'll send over as soon as I have another crash with mem info.

Thanks.

On Thu, Nov 11, 2010 at 5:10 AM, Bogdan-Andrei Iancu <[hidden email]> wrote:
This last crash had the same bt as the one from previous email ? the prev fix I made reports this:

        ERROR:db_flatstore:flat_db_insert: uninitialized connection
and does not crash, so this time the crash may be in a different place.

Now, it seams after all that the root of your problem is the mem exhaustion. To see what  causes this  (if a mem leak or simply not enough mem), there is a doc - http://www.opensips.org/Resources/DocsTsMem (it is for shm, but the same for pkg).

Maybe, first you should simply try to increase the pkg mem ( in config.h you have PKG_MEM_POOL_SIZE that you can increase) - you NEED to recompile and reinstall after that.

If more pkg mem does not solve the problem, I can help you with using the memory debugger to see what is going on there.

Regards,
Bogdan


thrillerbee wrote:
My other proxy crashed as well with these ERRORs in the syslog:

Nov 10 22:01:02 core2 /usr/local/sbin/opensips[22959]: ERROR:db_flatstore:get_name: pkg memory allocation failure
Nov 10 22:01:02 core2 /usr/local/sbin/opensips[22959]: ERROR:db_flatstore:flat_reopen_connection: failed to get_name
Nov 10 22:01:02 core2 /usr/local/sbin/opensips[22959]: ERROR:db_flatstore:flat_db_insert: uninitialized connection
Nov 10 22:01:02 core2 /usr/local/sbin/opensips[22959]: ERROR:db_flatstore:flat_db_insert: uninitialized connection
...
Nov 10 22:01:21 core2 /usr/local/sbin/opensips[22959]: ERROR:db_flatstore:flat_db_insert: uninitialized connection
Nov 10 22:01:22 core2 /usr/local/sbin/opensips[22959]: ERROR:db_flatstore:flat_db_insert: uninitialized connection
Nov 10 22:01:22 core2 /usr/local/sbin/opensips[22959]: ERROR:db_flatstore:new_flat_id: no pkg memory left
Nov 10 22:01:22 core2 kernel: [4297088.404734] opensips[22959]: segfault at 10 ip 7f3db577e21f sp 7fffa260d640 error 4 in db_flatstore.so[7f3db577b000+5000]

On Wed, Nov 10, 2010 at 10:19 AM, thrillerbee <[hidden email] <mailto:[hidden email]>> wrote:

   Bogdan,

   Well, I spoke too soon - it's not just an issue with the
   opensipsctl fifo calls - looks more like a memory leak.  It
   crashed again today, but I did get some errors in the syslog this
   time right before the crash:
   Nov 10 15:42:32 core1 /usr/local/sbin/opensips[27044]:
   ERROR:db_flatstore:new_flat_id: no pkg memory left
   Nov 10 15:42:32 core1 kernel: [5508366.582447] opensips[27044]:
   segfault at 10 ip 7fa7ff74c21f sp 7fffdc101700 error 4 in
   db_flatstore.so[7fa7ff749000+5000]
   To be thorough, I've attached the backtrace & output from print
   commands (although they're the same as before).

   To answer your question, yes - I do use the flat_rotate MI command.

   Thanks again.

   On Wed, Nov 10, 2010 at 4:04 AM, Bogdan-Andrei Iancu
   <[hidden email] <mailto:[hidden email]>> wrote:

       Hi,

       opensipsctl takes care that each command takes a separate fifo
       reply, so here it should be no problem. But the problem may be
       when comes with sending multiple commands (via FIFO) in the
       same time - this translates into parallel writes to the same
       file and depends on the atomicity of the write op.

       But in the worst case, a mixture at the FIFO level may lead to
       bogus command and not in any kind of crash....Do you use the
       "flat_rotate" MI command ?

       Regards,
       Bogdan

       thrillerbee wrote:

           Bogdan,

           It seems the issue is with 'opensipsctl fifo' - it's very
           sensitive to simultaneous calls.  Basically, I've combined
           all my scripts to prevent 'opensipsctl fifo' from being
           called too frequently and that seems (so far) to have
           mitigated the issue.  Is there anything one should know
           about how (not) to use /opensipsctl/?

           Thanks.

           On Mon, Nov 8, 2010 at 6:07 AM, Bogdan-Andrei Iancu
           <[hidden email] <mailto:[hidden email]>
           <mailto:[hidden email]
           <mailto:[hidden email]>>> wrote:

              Hi,

              strange if you do not have any errors :(....

              I just made a fix on both trunk and 1.6 to extend some
           checks in
              flatstore and prevent crashing (even if the DB op will
           not be
              executed).

              Could you update from SVN and see if stops crashing ?

              Regards,
              Bogdan


------------------------------------------------------------------------

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
 


--
Bogdan-Andrei Iancu
OpenSIPS Bootcamp
15 - 19 November 2010, Edison, New Jersey, USA
www.voice-system.ro


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: OpenSIPS core dumps

thrillerbee
Bogdan,

Since I made those changes, it doesn't core dump anymore - it just runs out of memory and stops processing packets.  It happened on both of my proxies - the mem dump can be downloaded here:

Thanks.

On Thu, Nov 11, 2010 at 12:43 PM, thrillerbee <[hidden email]> wrote:
Bogdan,

I had already increased the PKG_MEM_POOL_SIZE by *10.  I'm making the changes suggested in the link you provided to try to narrow down the memory issue.  I'll send over as soon as I have another crash with mem info.

Thanks.


On Thu, Nov 11, 2010 at 5:10 AM, Bogdan-Andrei Iancu <[hidden email]> wrote:
This last crash had the same bt as the one from previous email ? the prev fix I made reports this:

        ERROR:db_flatstore:flat_db_insert: uninitialized connection
and does not crash, so this time the crash may be in a different place.

Now, it seams after all that the root of your problem is the mem exhaustion. To see what  causes this  (if a mem leak or simply not enough mem), there is a doc - http://www.opensips.org/Resources/DocsTsMem (it is for shm, but the same for pkg).

Maybe, first you should simply try to increase the pkg mem ( in config.h you have PKG_MEM_POOL_SIZE that you can increase) - you NEED to recompile and reinstall after that.

If more pkg mem does not solve the problem, I can help you with using the memory debugger to see what is going on there.

Regards,
Bogdan


thrillerbee wrote:
My other proxy crashed as well with these ERRORs in the syslog:

Nov 10 22:01:02 core2 /usr/local/sbin/opensips[22959]: ERROR:db_flatstore:get_name: pkg memory allocation failure
Nov 10 22:01:02 core2 /usr/local/sbin/opensips[22959]: ERROR:db_flatstore:flat_reopen_connection: failed to get_name
Nov 10 22:01:02 core2 /usr/local/sbin/opensips[22959]: ERROR:db_flatstore:flat_db_insert: uninitialized connection
Nov 10 22:01:02 core2 /usr/local/sbin/opensips[22959]: ERROR:db_flatstore:flat_db_insert: uninitialized connection
...
Nov 10 22:01:21 core2 /usr/local/sbin/opensips[22959]: ERROR:db_flatstore:flat_db_insert: uninitialized connection
Nov 10 22:01:22 core2 /usr/local/sbin/opensips[22959]: ERROR:db_flatstore:flat_db_insert: uninitialized connection
Nov 10 22:01:22 core2 /usr/local/sbin/opensips[22959]: ERROR:db_flatstore:new_flat_id: no pkg memory left
Nov 10 22:01:22 core2 kernel: [4297088.404734] opensips[22959]: segfault at 10 ip 7f3db577e21f sp 7fffa260d640 error 4 in db_flatstore.so[7f3db577b000+5000]

On Wed, Nov 10, 2010 at 10:19 AM, thrillerbee <[hidden email] <mailto:[hidden email]>> wrote:

   Bogdan,

   Well, I spoke too soon - it's not just an issue with the
   opensipsctl fifo calls - looks more like a memory leak.  It
   crashed again today, but I did get some errors in the syslog this
   time right before the crash:
   Nov 10 15:42:32 core1 /usr/local/sbin/opensips[27044]:
   ERROR:db_flatstore:new_flat_id: no pkg memory left
   Nov 10 15:42:32 core1 kernel: [5508366.582447] opensips[27044]:
   segfault at 10 ip 7fa7ff74c21f sp 7fffdc101700 error 4 in
   db_flatstore.so[7fa7ff749000+5000]
   To be thorough, I've attached the backtrace & output from print
   commands (although they're the same as before).

   To answer your question, yes - I do use the flat_rotate MI command.

   Thanks again.

   On Wed, Nov 10, 2010 at 4:04 AM, Bogdan-Andrei Iancu
   <[hidden email] <mailto:[hidden email]>> wrote:

       Hi,

       opensipsctl takes care that each command takes a separate fifo
       reply, so here it should be no problem. But the problem may be
       when comes with sending multiple commands (via FIFO) in the
       same time - this translates into parallel writes to the same
       file and depends on the atomicity of the write op.

       But in the worst case, a mixture at the FIFO level may lead to
       bogus command and not in any kind of crash....Do you use the
       "flat_rotate" MI command ?

       Regards,
       Bogdan

       thrillerbee wrote:

           Bogdan,

           It seems the issue is with 'opensipsctl fifo' - it's very
           sensitive to simultaneous calls.  Basically, I've combined
           all my scripts to prevent 'opensipsctl fifo' from being
           called too frequently and that seems (so far) to have
           mitigated the issue.  Is there anything one should know
           about how (not) to use /opensipsctl/?

           Thanks.

           On Mon, Nov 8, 2010 at 6:07 AM, Bogdan-Andrei Iancu
           <[hidden email] <mailto:[hidden email]>
           <mailto:[hidden email]
           <mailto:[hidden email]>>> wrote:

              Hi,

              strange if you do not have any errors :(....

              I just made a fix on both trunk and 1.6 to extend some
           checks in
              flatstore and prevent crashing (even if the DB op will
           not be
              executed).

              Could you update from SVN and see if stops crashing ?

              Regards,
              Bogdan


------------------------------------------------------------------------

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
 


--
Bogdan-Andrei Iancu
OpenSIPS Bootcamp
15 - 19 November 2010, Edison, New Jersey, USA
www.voice-system.ro


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users



_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: OpenSIPS core dumps

John Khvatov-2
In reply to this post by thrillerbee
Hello.

On 10.11.2010, at 19:19, thrillerbee wrote:
Well, I spoke too soon - it's not just an issue with the opensipsctl fifo calls - looks more like a memory leak.  It crashed again today, but I did get some errors in the syslog this time right before the crash:
Nov 10 15:42:32 core1 /usr/local/sbin/opensips[27044]: ERROR:db_flatstore:new_flat_id: no pkg memory left
Nov 10 15:42:32 core1 kernel: [5508366.582447] opensips[27044]: segfault at 10 ip 7fa7ff74c21f sp 7fffdc101700 error 4 in db_flatstore.so[7fa7ff749000+5000]
To be thorough, I've attached the backtrace & output from print commands (although they're the same as before).

I have the same issue. OpenSIPS crashes at db_flatstore.so due to lack of memory (according resident memory usage graph). Recently I removed the use of the t_newtran() function and localstore module in opensips.cfg and crash did not happens for 11 days. Earlier crash happens every two-tree days.

Do you use t_newtran() in your config?

-- 
WBR, John Khvatov


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
12