Mysql Crashes and panics

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Mysql Crashes and panics

Brett Nemeroff
All,
Ok, so I fixed my kernel panic issue.. But I'm not real happy with the fix. I'm loading up like 157,000 routes.. each route has between 1-4 gateways in the gwlist. however, **I only have 1 actual route in the dr_gateways table**.. I hadn't added the other 3 gateways..

Now when dr_reload runs, it sucks up routes, but if there is a missing gateway for that route, the entire route is dropped. I didn't expect that, I thought it would have just skipped loading that gateway for that route. Seems that all of that loading / unloading of routes halted my system. I have no idea why.. But it's consistent and repeatable **on Ubuntu 8.04** I did the same setup, same routes in CentOS 5.2 and could not for the life of me reproduce the problem.

It's also worth mentioning that I hadn't increased the SHM in that config...


Ok, so different issue.. This is the same "opensips crashes under heavy db load" issue.. I load up Opensips with 100 CPS with SIPP. It's real busy, and processing real happily.

Then I run an external perl based rating script.. For about 15-20 second, all is good.. then I start getting retrans, then opensips dies. I reported this like a week ago, and I think bogdan pushed out some fixes that are in 1.5.1 (which I am running)... Well, my rating script was running, and I got a DateTime error.. it was telling me that a Date/Time written in the DB wasn't valid.. so I checked it out.. Look at this from my acc table:


+-------+--------+------------+-------------------------+----------------------------+----------+----------------------------+---------------------+-----------------------+------------+--------+------------------+------------+-----------+------+-------+
| id    | method | from_tag   | to_tag                  | callid                     | sip_code | sip_reason                 | time                | ua                    | from_did   | to_did | srcip            | gw         | harvested | pdd  | trace |
+-------+--------+------------+-------------------------+----------------------------+----------+----------------------------+---------------------+-----------------------+------------+--------+------------------+------------+-----------+------+-------+
| 69622 | 0      | 1510444289 | [hidden email] | sip:sipp@1.2.3.4:5061 | 341      | sip:9999@1.2.3.4:5060 | 0000-00-00 00:00:00 | udp:1.2.3.4:5060 | 1239854732 | 4      | udp:1.2.3.4 | 1239897933 |         0 |    1 |     1 |
+-------+--------+------------+-------------------------+----------------------------+----------+----------------------------+---------------------+-----------------------+------------+--------+------------------+------------+-----------+------+-------+
1 row in set (0.00 sec)


The data is all in the wrong fields!! This happened on exactly ONE record.

In addition, when I watch "SHOW PROCESSLIST" in mysql, i normally (even at 100CPS) just see like 4-6 "acc" inserts.. when my rating script runs, I see my screen fill up (completely) with "acc" inserts. So I'm about to test this with the max connections in mysql raised up to.. say... 1000. Maybe the cause is running out of mysql connections for acc transactions under high load..

-Brett



_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Mysql Crashes and panics

Bogdan-Andrei Iancu
Hi Brett,

Brett Nemeroff wrote:
> All,
> Ok, so I fixed my kernel panic issue
Cool, what was the cause ?


> .. But I'm not real happy with the fix. I'm loading up like 157,000
> routes.. each route has between 1-4 gateways in the gwlist. however,
> **I only have 1 actual route in the dr_gateways table**.. I hadn't
> added the other 3 gateways..
>
> Now when dr_reload runs, it sucks up routes, but if there is a missing
> gateway for that route, the entire route is dropped. I didn't expect
> that, I thought it would have just skipped loading that gateway for
> that route.
Make sense - please open a bug report on the tracker about this.
> Seems that all of that loading / unloading of routes halted my system.
> I have no idea why.. But it's consistent and repeatable **on Ubuntu
> 8.04** I did the same setup, same routes in CentOS 5.2 and could not
> for the life of me reproduce the problem.
So, do you think it is OS or setup-only specific ?
>
> It's also worth mentioning that I hadn't increased the SHM in that
> config...
but do you get any error about mem (like running out of mem).
>
>
> Ok, so different issue.. This is the same "opensips crashes under
> heavy db load" issue.. I load up Opensips with 100 CPS with SIPP. It's
> real busy, and processing real happily.
OK
>
> Then I run an external perl based rating script.. For about 15-20
> second, all is good.. then I start getting retrans, then opensips
> dies. I reported this like a week ago, and I think bogdan pushed out
> some fixes that are in 1.5.1 (which I am running)... Well, my rating
> script was running, and I got a DateTime error.. it was telling me
> that a Date/Time written in the DB wasn't valid..
Could you post the exact error from the log ?

> so I checked it out.. Look at this from my acc table:
>
>
> +-------+--------+------------+-------------------------+----------------------------+----------+----------------------------+---------------------+-----------------------+------------+--------+------------------+------------+-----------+------+-------+
> | id    | method | from_tag   | to_tag                  | callid      
>               | sip_code | sip_reason                 | time          
>      | ua                    | from_did   | to_did | srcip          
>  | gw         | harvested | pdd  | trace |
> +-------+--------+------------+-------------------------+----------------------------+----------+----------------------------+---------------------+-----------------------+------------+--------+------------------+------------+-----------+------+-------+
> | 69622 | 0      | 1510444289 | 22645-3410@1.2.3.4
> <mailto:22645-3410@1.2.3.4> | sip:sipp@1.2.3.4:5061
> <http://sip:sipp@1.2.3.4:5061> | 341      | sip:9999@1.2.3.4:5060
> <http://sip:9999@1.2.3.4:5060> | 0000-00-00 00:00:00 |
> udp:1.2.3.4:5060 <http://1.2.3.4:5060> | 1239854732 | 4      |
> udp:1.2.3.4 | 1239897933 |         0 |    1 |     1 |
> +-------+--------+------------+-------------------------+----------------------------+----------+----------------------------+---------------------+-----------------------+------------+--------+------------------+------------+-----------+------+-------+
> 1 row in set (0.00 sec)
>
> The data is all in the wrong fields!! This happened on exactly ONE record.
yeap....it looks messy...
>
> In addition, when I watch "SHOW PROCESSLIST" in mysql, i normally
> (even at 100CPS) just see like 4-6 "acc" inserts
do not fully follow you here - 100CPS should produce (INVITE+BYE)*100
acc records per sec -> 200 acc per sec.
> .. when my rating script runs, I see my screen fill up (completely)
> with "acc" inserts. So I'm about to test this with the max connections
> in mysql raised up to.. say... 1000. Maybe the cause is running out of
> mysql connections for acc transactions under high load..
opensips does not dynamically create more DB connection - whatever the
load is, the number of DB connection is the same - their number depends
only of the number of processes in opensips.

Regards,
Bogdan

>
> -Brett
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Users mailing list
> [hidden email]
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>  


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Mysql Crashes and panics

Geoffrey Mina
It sounds to me like your rating script is doing a full table lock and
thus not allowing opensips to insert/update rows into the table until
it completes.  Are you using InnoDB or MyISAM tables?  If you are
using InnoDB I would do a "SHOW INNODB STATUS" at the time of the
'lockup' to verify that you have, in fact, locked the entire table,
and namely the next index.

On Thu, Apr 16, 2009 at 4:27 AM, Bogdan-Andrei Iancu
<[hidden email]> wrote:

> Hi Brett,
>
> Brett Nemeroff wrote:
>> All,
>> Ok, so I fixed my kernel panic issue
> Cool, what was the cause ?
>
>
>> .. But I'm not real happy with the fix. I'm loading up like 157,000
>> routes.. each route has between 1-4 gateways in the gwlist. however,
>> **I only have 1 actual route in the dr_gateways table**.. I hadn't
>> added the other 3 gateways..
>>
>> Now when dr_reload runs, it sucks up routes, but if there is a missing
>> gateway for that route, the entire route is dropped. I didn't expect
>> that, I thought it would have just skipped loading that gateway for
>> that route.
> Make sense - please open a bug report on the tracker about this.
>> Seems that all of that loading / unloading of routes halted my system.
>> I have no idea why.. But it's consistent and repeatable **on Ubuntu
>> 8.04** I did the same setup, same routes in CentOS 5.2 and could not
>> for the life of me reproduce the problem.
> So, do you think it is OS or setup-only specific ?
>>
>> It's also worth mentioning that I hadn't increased the SHM in that
>> config...
> but do you get any error about mem (like running out of mem).
>>
>>
>> Ok, so different issue.. This is the same "opensips crashes under
>> heavy db load" issue.. I load up Opensips with 100 CPS with SIPP. It's
>> real busy, and processing real happily.
> OK
>>
>> Then I run an external perl based rating script.. For about 15-20
>> second, all is good.. then I start getting retrans, then opensips
>> dies. I reported this like a week ago, and I think bogdan pushed out
>> some fixes that are in 1.5.1 (which I am running)... Well, my rating
>> script was running, and I got a DateTime error.. it was telling me
>> that a Date/Time written in the DB wasn't valid..
> Could you post the exact error from the log ?
>> so I checked it out.. Look at this from my acc table:
>>
>>
>> +-------+--------+------------+-------------------------+----------------------------+----------+----------------------------+---------------------+-----------------------+------------+--------+------------------+------------+-----------+------+-------+
>> | id    | method | from_tag   | to_tag                  | callid
>>               | sip_code | sip_reason                 | time
>>      | ua                    | from_did   | to_did | srcip
>>  | gw         | harvested | pdd  | trace |
>> +-------+--------+------------+-------------------------+----------------------------+----------+----------------------------+---------------------+-----------------------+------------+--------+------------------+------------+-----------+------+-------+
>> | 69622 | 0      | 1510444289 | 22645-3410@1.2.3.4
>> <mailto:22645-3410@1.2.3.4> | sip:sipp@1.2.3.4:5061
>> <http://sip:sipp@1.2.3.4:5061> | 341      | sip:9999@1.2.3.4:5060
>> <http://sip:9999@1.2.3.4:5060> | 0000-00-00 00:00:00 |
>> udp:1.2.3.4:5060 <http://1.2.3.4:5060> | 1239854732 | 4      |
>> udp:1.2.3.4 | 1239897933 |         0 |    1 |     1 |
>> +-------+--------+------------+-------------------------+----------------------------+----------+----------------------------+---------------------+-----------------------+------------+--------+------------------+------------+-----------+------+-------+
>> 1 row in set (0.00 sec)
>>
>> The data is all in the wrong fields!! This happened on exactly ONE record.
> yeap....it looks messy...
>>
>> In addition, when I watch "SHOW PROCESSLIST" in mysql, i normally
>> (even at 100CPS) just see like 4-6 "acc" inserts
> do not fully follow you here - 100CPS should produce (INVITE+BYE)*100
> acc records per sec -> 200 acc per sec.
>> .. when my rating script runs, I see my screen fill up (completely)
>> with "acc" inserts. So I'm about to test this with the max connections
>> in mysql raised up to.. say... 1000. Maybe the cause is running out of
>> mysql connections for acc transactions under high load..
> opensips does not dynamically create more DB connection - whatever the
> load is, the number of DB connection is the same - their number depends
> only of the number of processes in opensips.
>
> Regards,
> Bogdan
>>
>> -Brett
>>
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> Users mailing list
>> [hidden email]
>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>
>
>
> _______________________________________________
> Users mailing list
> [hidden email]
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users