Packet Loss and its Solution

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Packet Loss and its Solution

Faisal Rehman
Hi Everyone,

I am facing huge packet loss in my server, so I am here to share with you some of the output of packets losses that you can see in attached image. Secondly I have few questions that I want to discuss with:

1. How can we reduce the packet loss to a minimum in an asterisk server, I mean I just want to know more detailed info about packet loss reduction.
2. I am calculating packet loss following that link http://www.linuxjournal.com/article/9398 where there is written that 20% loss is acceptable, but if you see the attached image what will be your conclusions about packet loss here?
3. At last but not least I just want to know the responsibilities of the software & the network, I mean how much software or network is responsible for packet loss?
4. What are the best possible ways to reduce the packet loss to a minimum extent?

 
Regards,

Faisal Rehman


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Packet Loss and its Solution

Faisal Rehman
Attached is the link for image:

 
Regards,

Faisal Rehman

From: Faisal Rehman <[hidden email]>
To: Open SIPS users mailling list <[hidden email]>
Sent: Monday, February 27, 2012 7:02 PM
Subject: [OpenSIPS-Users] Packet Loss and its Solution

Hi Everyone,

I am facing huge packet loss in my server, so I am here to share with you some of the output of packets losses that you can see in attached image. Secondly I have few questions that I want to discuss with:

1. How can we reduce the packet loss to a minimum in an asterisk server, I mean I just want to know more detailed info about packet loss reduction.
2. I am calculating packet loss following that link http://www.linuxjournal.com/article/9398 where there is written that 20% loss is acceptable, but if you see the attached image what will be your conclusions about packet loss here?
3. At last but not least I just want to know the responsibilities of the software & the network, I mean how much software or network is responsible for packet loss?
4. What are the best possible ways to reduce the packet loss to a minimum extent?

 
Regards,

Faisal Rehman


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users



_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Packet Loss and its Solution

Max Mühlbronner
Hi,


20% might be true for general network usage, but not for Voice traffic. Also it depends on the codecs used, 10% packetloss at VoIP means quality will be degraded (codec could only compensate up to 5%).


http://www.voiptroubleshooter.com/problems/packetloss.html


g711 at 10% packetloss:  http://www.voiptroubleshooter.com/sound_files/male1_3_rep_20ms_10.wav

pre-sip / h323, but also example files at different packetloss rates, tune in to the 20%  g711 wav file...  :)

http://www.networkcomputing.com/1001/1001ws2.html




Best Regards

Max M.

On 02/27/2012 03:12 PM, Faisal Rehman wrote:
Attached is the link for image:

 
Regards,

Faisal Rehman

From: Faisal Rehman [hidden email]
To: Open SIPS users mailling list [hidden email]
Sent: Monday, February 27, 2012 7:02 PM
Subject: [OpenSIPS-Users] Packet Loss and its Solution

Hi Everyone,

I am facing huge packet loss in my server, so I am here to share with you some of the output of packets losses that you can see in attached image. Secondly I have few questions that I want to discuss with:

1. How can we reduce the packet loss to a minimum in an asterisk server, I mean I just want to know more detailed info about packet loss reduction.
2. I am calculating packet loss following that link http://www.linuxjournal.com/article/9398 where there is written that 20% loss is acceptable, but if you see the attached image what will be your conclusions about packet loss here?
3. At last but not least I just want to know the responsibilities of the software & the network, I mean how much software or network is responsible for packet loss?
4. What are the best possible ways to reduce the packet loss to a minimum extent?

 
Regards,

Faisal Rehman


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users




_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Packet Loss and its Solution

Muhammad Danish Moosa
Interesting discussion but network component (bandwidth quality etc) must have major role in packet losses , right?



On Mon, Feb 27, 2012 at 9:29 PM, Max Mühlbronner <[hidden email]> wrote:
Hi,


20% might be true for general network usage, but not for Voice traffic. Also it depends on the codecs used, 10% packetloss at VoIP means quality will be degraded (codec could only compensate up to 5%).


http://www.voiptroubleshooter.com/problems/packetloss.html


g711 at 10% packetloss:  http://www.voiptroubleshooter.com/sound_files/male1_3_rep_20ms_10.wav

pre-sip / h323, but also example files at different packetloss rates, tune in to the 20%  g711 wav file...  :)

http://www.networkcomputing.com/1001/1001ws2.html




Best Regards

Max M.


On 02/27/2012 03:12 PM, Faisal Rehman wrote:
Attached is the link for image:

 
Regards,

Faisal Rehman

From: Faisal Rehman [hidden email]
To: Open SIPS users mailling list [hidden email]
Sent: Monday, February 27, 2012 7:02 PM
Subject: [OpenSIPS-Users] Packet Loss and its Solution

Hi Everyone,

I am facing huge packet loss in my server, so I am here to share with you some of the output of packets losses that you can see in attached image. Secondly I have few questions that I want to discuss with:

1. How can we reduce the packet loss to a minimum in an asterisk server, I mean I just want to know more detailed info about packet loss reduction.
2. I am calculating packet loss following that link http://www.linuxjournal.com/article/9398 where there is written that 20% loss is acceptable, but if you see the attached image what will be your conclusions about packet loss here?
3. At last but not least I just want to know the responsibilities of the software & the network, I mean how much software or network is responsible for packet loss?
4. What are the best possible ways to reduce the packet loss to a minimum extent?

 
Regards,

Faisal Rehman


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users




_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users




--
Muhammad Danish Moosa

" The core of mans' spirit comes from new experiences. "___ Christopher McCandless

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Packet Loss and its Solution

Bogdan-Andrei Iancu-2
In reply to this post by Faisal Rehman
Hi Faisal,

Let me comment a bit on the loss at software - packages can be discarded at TCP/IP stack level (by kernel) if no application is reading the data  (or no reading as fast as the data comes).

You can check on this (if opensips is able to process all incoming traffic, without having the kernel to discard data because of full buffering on sockets) via some statistics from the NET class :
        http://www.opensips.org/Resources/DocsCoreStats17#toc17

An overall idea over the load in opensips (if you have idle processes or not) can be monitored via the LOAD stats:
        http://www.opensips.org/Resources/DocsCoreStats17#toc14

Regards,
Bogdan

On 02/27/2012 04:02 PM, Faisal Rehman wrote:
Hi Everyone,

I am facing huge packet loss in my server, so I am here to share with you some of the output of packets losses that you can see in attached image. Secondly I have few questions that I want to discuss with:

1. How can we reduce the packet loss to a minimum in an asterisk server, I mean I just want to know more detailed info about packet loss reduction.
2. I am calculating packet loss following that link http://www.linuxjournal.com/article/9398 where there is written that 20% loss is acceptable, but if you see the attached image what will be your conclusions about packet loss here?
3. At last but not least I just want to know the responsibilities of the software & the network, I mean how much software or network is responsible for packet loss?
4. What are the best possible ways to reduce the packet loss to a minimum extent?

 
Regards,

Faisal Rehman

_______________________________________________ Users mailing list [hidden email] http://lists.opensips.org/cgi-bin/mailman/listinfo/users


-- 
Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
http://www.opensips-solutions.com

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Packet Loss and its Solution

Muhammad Danish Moosa
Helo Bogdan

only signalling packets can be lost on opensips?

But rtp streams are faster and frequent and have high impact on voice quality. He seems to ask packet losses on rtp packets. Even if the problem is identified what are the clues to solve the problem?


On Tue, Feb 28, 2012 at 3:14 AM, Bogdan-Andrei Iancu <[hidden email]> wrote:
Hi Faisal,

Let me comment a bit on the loss at software - packages can be discarded at TCP/IP stack level (by kernel) if no application is reading the data  (or no reading as fast as the data comes).

You can check on this (if opensips is able to process all incoming traffic, without having the kernel to discard data because of full buffering on sockets) via some statistics from the NET class :
        http://www.opensips.org/Resources/DocsCoreStats17#toc17

An overall idea over the load in opensips (if you have idle processes or not) can be monitored via the LOAD stats:
        http://www.opensips.org/Resources/DocsCoreStats17#toc14

Regards,
Bogdan


On 02/27/2012 04:02 PM, Faisal Rehman wrote:
Hi Everyone,

I am facing huge packet loss in my server, so I am here to share with you some of the output of packets losses that you can see in attached image. Secondly I have few questions that I want to discuss with:

1. How can we reduce the packet loss to a minimum in an asterisk server, I mean I just want to know more detailed info about packet loss reduction.
2. I am calculating packet loss following that link http://www.linuxjournal.com/article/9398 where there is written that 20% loss is acceptable, but if you see the attached image what will be your conclusions about packet loss here?
3. At last but not least I just want to know the responsibilities of the software & the network, I mean how much software or network is responsible for packet loss?
4. What are the best possible ways to reduce the packet loss to a minimum extent?

 
Regards,

Faisal Rehman

_______________________________________________ Users mailing list [hidden email] http://lists.opensips.org/cgi-bin/mailman/listinfo/users


-- 
Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
http://www.opensips-solutions.com

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users




--
Muhammad Danish Moosa

" The core of mans' spirit comes from new experiences. "___ Christopher McCandless

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Packet Loss and its Solution

SamyGo
Hi Faisal,

Can you copy/paste the stats of "ifconfig ethN" on which traffic is terminating. I just wanted to see the error and dropped packets on physical interface. 
Hardware and physical connectivity plays major role in packet losses.On Asterisk server Jitter options might help you but this is media-proxy, I assume from the interface you are viewing, the packets are shown as lost. So could it be heavy media traffic flowing through the interface and media proxy is unable to use much CPU processing power to process all the RTPs ?

Before troubleshooting the software application I suggest start digging the networking interfaces and tweak the ethN and related properties of kernel to maximize the throughput.

This would be just how I'd go with this kind of problem.

Regards.
Sammy


On Tue, Feb 28, 2012 at 3:45 AM, Muhammad Danish Moosa <[hidden email]> wrote:
Helo Bogdan

only signalling packets can be lost on opensips?

But rtp streams are faster and frequent and have high impact on voice quality. He seems to ask packet losses on rtp packets. Even if the problem is identified what are the clues to solve the problem?



On Tue, Feb 28, 2012 at 3:14 AM, Bogdan-Andrei Iancu <[hidden email]> wrote:
Hi Faisal,

Let me comment a bit on the loss at software - packages can be discarded at TCP/IP stack level (by kernel) if no application is reading the data  (or no reading as fast as the data comes).

You can check on this (if opensips is able to process all incoming traffic, without having the kernel to discard data because of full buffering on sockets) via some statistics from the NET class :
        http://www.opensips.org/Resources/DocsCoreStats17#toc17

An overall idea over the load in opensips (if you have idle processes or not) can be monitored via the LOAD stats:
        http://www.opensips.org/Resources/DocsCoreStats17#toc14

Regards,
Bogdan


On 02/27/2012 04:02 PM, Faisal Rehman wrote:
Hi Everyone,

I am facing huge packet loss in my server, so I am here to share with you some of the output of packets losses that you can see in attached image. Secondly I have few questions that I want to discuss with:

1. How can we reduce the packet loss to a minimum in an asterisk server, I mean I just want to know more detailed info about packet loss reduction.
2. I am calculating packet loss following that link http://www.linuxjournal.com/article/9398 where there is written that 20% loss is acceptable, but if you see the attached image what will be your conclusions about packet loss here?
3. At last but not least I just want to know the responsibilities of the software & the network, I mean how much software or network is responsible for packet loss?
4. What are the best possible ways to reduce the packet loss to a minimum extent?

 
Regards,

Faisal Rehman

_______________________________________________ Users mailing list [hidden email] http://lists.opensips.org/cgi-bin/mailman/listinfo/users


-- 
Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
http://www.opensips-solutions.com

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users




--
Muhammad Danish Moosa

" The core of mans' spirit comes from new experiences. "___ Christopher McCandless

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users



_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Packet Loss and its Solution

Bogdan-Andrei Iancu-2
In reply to this post by Muhammad Danish Moosa
Hi Muhammad,

OpenSIPS only handles signaling, no media going through OpenSIPS :)

Best regards,
Bogdan

On 02/28/2012 12:45 AM, Muhammad Danish Moosa wrote:
Helo Bogdan

only signalling packets can be lost on opensips?

But rtp streams are faster and frequent and have high impact on voice quality. He seems to ask packet losses on rtp packets. Even if the problem is identified what are the clues to solve the problem?


On Tue, Feb 28, 2012 at 3:14 AM, Bogdan-Andrei Iancu <[hidden email]> wrote:
Hi Faisal,

Let me comment a bit on the loss at software - packages can be discarded at TCP/IP stack level (by kernel) if no application is reading the data  (or no reading as fast as the data comes).

You can check on this (if opensips is able to process all incoming traffic, without having the kernel to discard data because of full buffering on sockets) via some statistics from the NET class :
        http://www.opensips.org/Resources/DocsCoreStats17#toc17

An overall idea over the load in opensips (if you have idle processes or not) can be monitored via the LOAD stats:
        http://www.opensips.org/Resources/DocsCoreStats17#toc14

Regards,
Bogdan


On 02/27/2012 04:02 PM, Faisal Rehman wrote:
Hi Everyone,

I am facing huge packet loss in my server, so I am here to share with you some of the output of packets losses that you can see in attached image. Secondly I have few questions that I want to discuss with:

1. How can we reduce the packet loss to a minimum in an asterisk server, I mean I just want to know more detailed info about packet loss reduction.
2. I am calculating packet loss following that link http://www.linuxjournal.com/article/9398 where there is written that 20% loss is acceptable, but if you see the attached image what will be your conclusions about packet loss here?
3. At last but not least I just want to know the responsibilities of the software & the network, I mean how much software or network is responsible for packet loss?
4. What are the best possible ways to reduce the packet loss to a minimum extent?

 
Regards,

Faisal Rehman

_______________________________________________ Users mailing list [hidden email] http://lists.opensips.org/cgi-bin/mailman/listinfo/users


-- 
Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
http://www.opensips-solutions.com

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users




--
Muhammad Danish Moosa

" The core of mans' spirit comes from new experiences. "___ Christopher McCandless
_______________________________________________ Users mailing list [hidden email] http://lists.opensips.org/cgi-bin/mailman/listinfo/users


-- 
Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
http://www.opensips-solutions.com

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Packet Loss and its Solution

Faisal Rehman
In reply to this post by SamyGo
Hi Sammy,

Sure I will paste the stats here but will you please tell me some important steps to be performed on software side. Here I am talking about the OS, kernel upgrade & advanced system administration level stuff that can reduce the packet loss immediately. Through the final conclusions I have become to know that its not the OpenSIPS problem but it regards with physical, software & network upgrades. 
 
Regards,

Faisal Rehman

From: Sammy Govind <[hidden email]>
To: OpenSIPS users mailling list <[hidden email]>
Sent: Tuesday, February 28, 2012 10:38 AM
Subject: Re: [OpenSIPS-Users] Packet Loss and its Solution

Hi Faisal,

Can you copy/paste the stats of "ifconfig ethN" on which traffic is terminating. I just wanted to see the error and dropped packets on physical interface. 
Hardware and physical connectivity plays major role in packet losses.On Asterisk server Jitter options might help you but this is media-proxy, I assume from the interface you are viewing, the packets are shown as lost. So could it be heavy media traffic flowing through the interface and media proxy is unable to use much CPU processing power to process all the RTPs ?

Before troubleshooting the software application I suggest start digging the networking interfaces and tweak the ethN and related properties of kernel to maximize the throughput.

This would be just how I'd go with this kind of problem.

Regards.
Sammy


On Tue, Feb 28, 2012 at 3:45 AM, Muhammad Danish Moosa <[hidden email]> wrote:
Helo Bogdan

only signalling packets can be lost on opensips?

But rtp streams are faster and frequent and have high impact on voice quality. He seems to ask packet losses on rtp packets. Even if the problem is identified what are the clues to solve the problem?



On Tue, Feb 28, 2012 at 3:14 AM, Bogdan-Andrei Iancu <[hidden email]> wrote:
Hi Faisal,

Let me comment a bit on the loss at software - packages can be discarded at TCP/IP stack level (by kernel) if no application is reading the data  (or no reading as fast as the data comes).

You can check on this (if opensips is able to process all incoming traffic, without having the kernel to discard data because of full buffering on sockets) via some statistics from the NET class :
        http://www.opensips.org/Resources/DocsCoreStats17#toc17

An overall idea over the load in opensips (if you have idle processes or not) can be monitored via the LOAD stats:
        http://www.opensips.org/Resources/DocsCoreStats17#toc14

Regards,
Bogdan


On 02/27/2012 04:02 PM, Faisal Rehman wrote:
Hi Everyone,

I am facing huge packet loss in my server, so I am here to share with you some of the output of packets losses that you can see in attached image. Secondly I have few questions that I want to discuss with:

1. How can we reduce the packet loss to a minimum in an asterisk server, I mean I just want to know more detailed info about packet loss reduction.
2. I am calculating packet loss following that link http://www.linuxjournal.com/article/9398 where there is written that 20% loss is acceptable, but if you see the attached image what will be your conclusions about packet loss here?
3. At last but not least I just want to know the responsibilities of the software & the network, I mean how much software or network is responsible for packet loss?
4. What are the best possible ways to reduce the packet loss to a minimum extent?

 
Regards,

Faisal Rehman

_______________________________________________ Users mailing list [hidden email] http://lists.opensips.org/cgi-bin/mailman/listinfo/users


-- 
Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
http://www.opensips-solutions.com

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users




--
Muhammad Danish Moosa

" The core of mans' spirit comes from new experiences. "___ Christopher McCandless

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users



_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users



_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Packet Loss and its Solution

SamyGo

I don't have anything related to these troubleshooting and tweaking readily available in my mind, I already wrote the pseudo routine/steps to resolve the issue.
As Sir Bogdan has stated openSIPS dont deal with RTPs so you can safely conclude right away that its not OpenSIPS causing the packet losses.
I will still insist on focusing/troubleshooting the 1st three layers on your network or on server.


On Thu, Mar 1, 2012 at 12:04 PM, Faisal Rehman <[hidden email]> wrote:
Hi Sammy,

Sure I will paste the stats here but will you please tell me some important steps to be performed on software side. Here I am talking about the OS, kernel upgrade & advanced system administration level stuff that can reduce the packet loss immediately. Through the final conclusions I have become to know that its not the OpenSIPS problem but it regards with physical, software & network upgrades. 
 
Regards,

Faisal Rehman

From: Sammy Govind <[hidden email]>
To: OpenSIPS users mailling list <[hidden email]>
Sent: Tuesday, February 28, 2012 10:38 AM
Subject: Re: [OpenSIPS-Users] Packet Loss and its Solution

Hi Faisal,

Can you copy/paste the stats of "ifconfig ethN" on which traffic is terminating. I just wanted to see the error and dropped packets on physical interface. 
Hardware and physical connectivity plays major role in packet losses.On Asterisk server Jitter options might help you but this is media-proxy, I assume from the interface you are viewing, the packets are shown as lost. So could it be heavy media traffic flowing through the interface and media proxy is unable to use much CPU processing power to process all the RTPs ?

Before troubleshooting the software application I suggest start digging the networking interfaces and tweak the ethN and related properties of kernel to maximize the throughput.

This would be just how I'd go with this kind of problem.

Regards.
Sammy


On Tue, Feb 28, 2012 at 3:45 AM, Muhammad Danish Moosa <[hidden email]> wrote:
Helo Bogdan

only signalling packets can be lost on opensips?

But rtp streams are faster and frequent and have high impact on voice quality. He seems to ask packet losses on rtp packets. Even if the problem is identified what are the clues to solve the problem?



On Tue, Feb 28, 2012 at 3:14 AM, Bogdan-Andrei Iancu <[hidden email]> wrote:
Hi Faisal,

Let me comment a bit on the loss at software - packages can be discarded at TCP/IP stack level (by kernel) if no application is reading the data  (or no reading as fast as the data comes).

You can check on this (if opensips is able to process all incoming traffic, without having the kernel to discard data because of full buffering on sockets) via some statistics from the NET class :
        http://www.opensips.org/Resources/DocsCoreStats17#toc17

An overall idea over the load in opensips (if you have idle processes or not) can be monitored via the LOAD stats:
        http://www.opensips.org/Resources/DocsCoreStats17#toc14

Regards,
Bogdan


On 02/27/2012 04:02 PM, Faisal Rehman wrote:
Hi Everyone,

I am facing huge packet loss in my server, so I am here to share with you some of the output of packets losses that you can see in attached image. Secondly I have few questions that I want to discuss with:

1. How can we reduce the packet loss to a minimum in an asterisk server, I mean I just want to know more detailed info about packet loss reduction.
2. I am calculating packet loss following that link http://www.linuxjournal.com/article/9398 where there is written that 20% loss is acceptable, but if you see the attached image what will be your conclusions about packet loss here?
3. At last but not least I just want to know the responsibilities of the software & the network, I mean how much software or network is responsible for packet loss?
4. What are the best possible ways to reduce the packet loss to a minimum extent?

 
Regards,

Faisal Rehman

_______________________________________________ Users mailing list [hidden email] http://lists.opensips.org/cgi-bin/mailman/listinfo/users


-- 
Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
http://www.opensips-solutions.com

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users




--
Muhammad Danish Moosa

" The core of mans' spirit comes from new experiences. "___ Christopher McCandless

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users



_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users



_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users



_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users