3.00 load avg

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

3.00 load avg

Jeff Pyle
3.00 load avg Hello,

Today I noticed Opensips is causing my machine to go to a load average of 3.00, yet it’s 100% idle.  I can’t figure out what’s causing this or how both are possible.  I’ve never encountered it before.

top - 12:35:45 up 16 days,  1:33,  3 users,  load average: 3.00, 3.00, 2.91
Tasks: 120 total,   2 running, 118 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st

When I stop Opensips, the load average drops to 0.00 or 0.01.  If I start Opensips again, even if it doesn’t process any calls, the load average rises to 3.00 again.

Does anyone have any suggestions on how I can diagnose the cause of this?


Thanks,
Jeff






_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: 3.00 load avg

Bogdan-Andrei Iancu
Hi Jeff,

A simple google on the topic "top command load average"

http://www.teamquest.com/resources/gunther/display/5/

See starting with section 2.2

Shortly, is about the number of processes in your case.

Regards,
Bogdan

Jeff Pyle wrote:

> Hello,
>
> Today I noticed Opensips is causing my machine to go to a load average
> of 3.00, yet it’s 100% idle. I can’t figure out what’s causing this or
> how both are possible. I’ve never encountered it before.
>
> top - 12:35:45 up 16 days, 1:33, 3 users, load average: 3.00, 3.00, 2.91
> Tasks: 120 total, 2 running, 118 sleeping, 0 stopped, 0 zombie
> Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
>
> When I stop Opensips, the load average drops to 0.00 or 0.01. If I
> start Opensips again, even if it doesn’t process any calls, the load
> average rises to 3.00 again.
>
> Does anyone have any suggestions on how I can diagnose the cause of this?
>
>
> Thanks,
> Jeff
>
>
>
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Users mailing list
> [hidden email]
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>  


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: 3.00 load avg

Dan Pascu
On Friday 13 March 2009, Bogdan-Andrei Iancu wrote:
> Hi Jeff,
>
> A simple google on the topic "top command load average"
>
> http://www.teamquest.com/resources/gunther/display/5/
>
> See starting with section 2.2
>
> Shortly, is about the number of processes in your case.

It's the number of _active_ processes. Or to be more precise it's the
number of processes waiting in the queue to get scheduled to execute on
the CPU. You cannot get a 3.0 load average with all processes being idle,
so something is definitely abnormal. You need 3 processes loaded at 100%
to reach a load average of 3.0 (with 1 CPU).

But I'm curious if opensips can really trigger this, or is just a glitch
in the OS reporting the load average?

An interesting piece of information would be to know if the opensips
processes have any incoming/outgoing network queues (seen with
netstat -pan | grep opensips).

>
> Regards,
> Bogdan
>
> Jeff Pyle wrote:
> > Hello,
> >
> > Today I noticed Opensips is causing my machine to go to a load
> > average of 3.00, yet it’s 100% idle. I can’t figure out what’s
> > causing this or how both are possible. I’ve never encountered it
> > before.
> >
> > top - 12:35:45 up 16 days, 1:33, 3 users, load average: 3.00, 3.00,
> > 2.91 Tasks: 120 total, 2 running, 118 sleeping, 0 stopped, 0 zombie
> > Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si,
> > 0.0%st
> >
> > When I stop Opensips, the load average drops to 0.00 or 0.01. If I
> > start Opensips again, even if it doesn’t process any calls, the load
> > average rises to 3.00 again.
> >
> > Does anyone have any suggestions on how I can diagnose the cause of
> > this?
> >
> >
> > Thanks,
> > Jeff
> >
> >
> >
> >
> >
> > ---------------------------------------------------------------------
> >---
> >
> > _______________________________________________
> > Users mailing list
> > [hidden email]
> > http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
> _______________________________________________
> Users mailing list
> [hidden email]
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users



--
Dan

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: 3.00 load avg

Jeff Pyle
Dan,

# netstat -pan | grep opensips
udp        0      0 ww.xx.yy.zz:5060            0.0.0.0:*
13788/opensips    
unix  2      [ ]         DGRAM                    31333  13821/opensips
/tmp/opensips.sock
unix  3      [ ]         STREAM     CONNECTED     31355  13788/opensips
unix  2      [ ]         STREAM     CONNECTED     31353  13788/opensips
unix  2      [ ]         STREAM     CONNECTED     31351  13829/opensips
unix  2      [ ]         STREAM     CONNECTED     31348  13827/opensips
unix  2      [ ]         STREAM     CONNECTED     31346  13825/opensips
unix  2      [ ]         STREAM     CONNECTED     31344  13823/opensips
unix  2      [ ]         STREAM     CONNECTED     31342  13821/opensips
unix  3      [ ]         STREAM     CONNECTED     31338  13817/opensips
unix  2      [ ]         DGRAM                    31337  13821/opensips
unix  2      [ ]         STREAM     CONNECTED     31335  13817/opensips
unix  2      [ ]         STREAM     CONNECTED     31331  13817/opensips
unix  3      [ ]         STREAM     CONNECTED     31328  13814/opensips
unix  2      [ ]         STREAM     CONNECTED     31326  13814/opensips
unix  2      [ ]         STREAM     CONNECTED     31324  13814/opensips
unix  3      [ ]         STREAM     CONNECTED     31320  13811/opensips
unix  2      [ ]         STREAM     CONNECTED     31318  13811/opensips
unix  2      [ ]         STREAM     CONNECTED     31316  13811/opensips
unix  3      [ ]         STREAM     CONNECTED     31312  13807/opensips
unix  2      [ ]         STREAM     CONNECTED     31310  13807/opensips
unix  2      [ ]         STREAM     CONNECTED     31308  13807/opensips
unix  3      [ ]         STREAM     CONNECTED     31303  13803/opensips
unix  3      [ ]         STREAM     CONNECTED     31301  13801/opensips
unix  2      [ ]         STREAM     CONNECTED     31298  13803/opensips
unix  3      [ ]         STREAM     CONNECTED     31295  13798/opensips
unix  2      [ ]         STREAM     CONNECTED     31291  13803/opensips
unix  2      [ ]         STREAM     CONNECTED     31289  13801/opensips
unix  3      [ ]         STREAM     CONNECTED     31288  13794/opensips
unix  2      [ ]         STREAM     CONNECTED     31287  13798/opensips
unix  2      [ ]         STREAM     CONNECTED     31285  13794/opensips
unix  2      [ ]         STREAM     CONNECTED     31283  13801/opensips
unix  3      [ ]         STREAM     CONNECTED     31279  13795/opensips
unix  2      [ ]         STREAM     CONNECTED     31277  13798/opensips
unix  2      [ ]         STREAM     CONNECTED     31275  13795/opensips
unix  2      [ ]         STREAM     CONNECTED     31273  13795/opensips
unix  2      [ ]         STREAM     CONNECTED     31271  13794/opensips
unix  2      [ ]         DGRAM                    31257  13788/opensips
unix  2      [ ]         DGRAM                    31256  13788/opensips


top - 07:42:58 up 2 days, 16:38,  1 user,  load average: 3.00, 3.00, 3.00
Tasks: 108 total,   2 running, 106 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
0.0%st

This is a CentOS i386 xen VM, running on CentOS i386.  The physical machine
has 8 processing cores, and the virtual has 4.

I've been keeping fairly up to date from SVN.  I'd say this problem appeared
maybe a week or so ago but I'm not certain on the exact rev number.


- Jeff





On 3/15/09 12:00 AM, "Dan Pascu" <[hidden email]> wrote:

> On Friday 13 March 2009, Bogdan-Andrei Iancu wrote:
>> Hi Jeff,
>>
>> A simple google on the topic "top command load average"
>>
>> http://www.teamquest.com/resources/gunther/display/5/
>>
>> See starting with section 2.2
>>
>> Shortly, is about the number of processes in your case.
>
> It's the number of _active_ processes. Or to be more precise it's the
> number of processes waiting in the queue to get scheduled to execute on
> the CPU. You cannot get a 3.0 load average with all processes being idle,
> so something is definitely abnormal. You need 3 processes loaded at 100%
> to reach a load average of 3.0 (with 1 CPU).
>
> But I'm curious if opensips can really trigger this, or is just a glitch
> in the OS reporting the load average?
>
> An interesting piece of information would be to know if the opensips
> processes have any incoming/outgoing network queues (seen with
> netstat -pan | grep opensips).
>


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: 3.00 load avg

Dan Pascu
On Monday 16 March 2009, Jeff Pyle wrote:
> Dan,
>
> # netstat -pan | grep opensips
> udp        0      0 ww.xx.yy.zz:5060            0.0.0.0:*

There appear to be no network queue, so the app is idle. However you
mention that you run this in a virtual machine. Do you see the problem
with a real system as well? Otherwise I wouldn't rely on the accuracy of
what the virtual machine reports. Besides, did you upgrade the emulator
as well around that time? This may be very well a bug in the emulation
software, reporting false status under certain conditions.

> top - 07:42:58 up 2 days, 16:38,  1 user,  load average: 3.00, 3.00,
> 3.00 Tasks: 108 total,   2 running, 106 sleeping,   0 stopped,   0
> zombie Cpu(s):  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,
> 0.0%si, 0.0%st
>
> This is a CentOS i386 xen VM, running on CentOS i386.  The physical
> machine has 8 processing cores, and the virtual has 4.
>
> I've been keeping fairly up to date from SVN.  I'd say this problem
> appeared maybe a week or so ago but I'm not certain on the exact rev
> number.


--
Dan

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: 3.00 load avg

Jeff Pyle
Hi Dan,

I haven't tried it on a real box yet.  With all the things it hooks into
it's not exactly portable at the moment.  Nothing has changed elsewhere on
the system, either with the emulator, kernels on host or guest, etc.

I'm not *too* worried about it only because the apps on the system (vm) seem
to be running well.  I may drop the v-cpus from 4 to 2 to see if that
adversely affects performance under low load.

For what it's worth I've excellent success with Xen under CentOS with
Openser 1.3.2 and other apps.  The reports seem to be accurate everywhere
but here.  Even mediaproxy (under moderate to low load) does fairly well,
although I'd never consider going into production with it on a virtual
machine.

Thanks for the analysis.


- Jeff



On 3/16/09 1:38 PM, "Dan Pascu" <[hidden email]> wrote:
 
> There appear to be no network queue, so the app is idle. However you
> mention that you run this in a virtual machine. Do you see the problem
> with a real system as well? Otherwise I wouldn't rely on the accuracy of
> what the virtual machine reports. Besides, did you upgrade the emulator
> as well around that time? This may be very well a bug in the emulation
> software, reporting false status under certain conditions.



_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: 3.00 load avg

Romanov Vladimir
Hi!
Why you don't want use VM in production? We have plans to use Citrix XenServer. We use rptpproxy instead of mediaproxy.

-----------------
Vladimir Romanov
Scartel Star Lab
CTO
+7 (960) 239-0853


-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Jeff Pyle
Sent: Monday, March 16, 2009 9:12 PM
To: Dan Pascu
Cc: [hidden email]
Subject: Re: [OpenSIPS-Users] 3.00 load avg

Hi Dan,

I haven't tried it on a real box yet.  With all the things it hooks into
it's not exactly portable at the moment.  Nothing has changed elsewhere on
the system, either with the emulator, kernels on host or guest, etc.

I'm not *too* worried about it only because the apps on the system (vm) seem
to be running well.  I may drop the v-cpus from 4 to 2 to see if that
adversely affects performance under low load.

For what it's worth I've excellent success with Xen under CentOS with
Openser 1.3.2 and other apps.  The reports seem to be accurate everywhere
but here.  Even mediaproxy (under moderate to low load) does fairly well,
although I'd never consider going into production with it on a virtual
machine.

Thanks for the analysis.


- Jeff



On 3/16/09 1:38 PM, "Dan Pascu" <[hidden email]> wrote:
 
> There appear to be no network queue, so the app is idle. However you
> mention that you run this in a virtual machine. Do you see the problem
> with a real system as well? Otherwise I wouldn't rely on the accuracy of
> what the virtual machine reports. Besides, did you upgrade the emulator
> as well around that time? This may be very well a bug in the emulation
> software, reporting false status under certain conditions.



_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: on a VM

Jeff Pyle
Hello,

I have no problem with using Opensips in production on a VM.  Mediaproxy (or
rtpproxy for that matter) is a different story.  I haven't done any
calculations or benchmarks, but it seems to me the sheer number of
interrupts triggered as the result of all the relayed RTP packets is
something that would scale much better on a pure hardware platform compared
to a VM of any flavor.

I suppose it depends on the application.  If I had an instance of Opensips
with mediaproxy or rtpproxy to allow my ten friends and I to establish SIP
sessions with each other, and we were all behind NAT, I'd put that on a VM
in a second.  But if we're talking about 500 concurrent sessions, that's
50000 packets per second (assuming a 20ms ptime) by the time you figure
inbound packets and outbound packets.  Again, I haven't done the testing,
but I'd think a hardware platform would respond much more favorably.

In your plans for XenServer, what are your scalability requirements?



- Jeff
 


On 3/17/09 2:03 AM, "Romanov Vladimir" <[hidden email]> wrote:

> Hi!
> Why you don't want use VM in production? We have plans to use Citrix
> XenServer. We use rptpproxy instead of mediaproxy.
>
> -----------------
> Vladimir Romanov
> Scartel Star Lab
> CTO
> +7 (960) 239-0853
>
>
> -----Original Message-----
> From: [hidden email]
> [mailto:[hidden email]] On Behalf Of Jeff Pyle
> Sent: Monday, March 16, 2009 9:12 PM
> To: Dan Pascu
> Cc: [hidden email]
> Subject: Re: [OpenSIPS-Users] 3.00 load avg
>
> Hi Dan,
>
> I haven't tried it on a real box yet.  With all the things it hooks into
> it's not exactly portable at the moment.  Nothing has changed elsewhere on
> the system, either with the emulator, kernels on host or guest, etc.
>
> I'm not *too* worried about it only because the apps on the system (vm) seem
> to be running well.  I may drop the v-cpus from 4 to 2 to see if that
> adversely affects performance under low load.
>
> For what it's worth I've excellent success with Xen under CentOS with
> Openser 1.3.2 and other apps.  The reports seem to be accurate everywhere
> but here.  Even mediaproxy (under moderate to low load) does fairly well,
> although I'd never consider going into production with it on a virtual
> machine.
>
> Thanks for the analysis.
>
>
> - Jeff
>
>
>
> On 3/16/09 1:38 PM, "Dan Pascu" <[hidden email]> wrote:
>  
>> There appear to be no network queue, so the app is idle. However you
>> mention that you run this in a virtual machine. Do you see the problem
>> with a real system as well? Otherwise I wouldn't rely on the accuracy of
>> what the virtual machine reports. Besides, did you upgrade the emulator
>> as well around that time? This may be very well a bug in the emulation
>> software, reporting false status under certain conditions.
>
>
>
> _______________________________________________
> Users mailing list
> [hidden email]
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users