locally generated replies

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

locally generated replies

OpenSIPS - Users mailing list
Hi Guys,

How's it possible locally generated replied triggering on_reply_route ? I ran in to issue where all opensips process gets stuck in processing same call replies and causing other traffic to get drop.


<info> /usr/sbin/opensips[27464]: Call: Reply from a NAT endpoint - S=408 D=Request Timeout F=sip:xxx@xxx T=sip:yyy
yy@x IP=a.b.c.d ID=asgasgasgas

Request process by opensips before this is an ACK request belong to the call where I don't think It'll expect a reply. Could it be an issue If I call t_on_reply on an ACK msg ?

I'm trying to figure out where the bug in my opensips routing script.  It causes all sip listerner processes to get stuck in a loop causing to generate above message. IP a.b.c.d is the sip server IP which confuse me as locally generated replies shouldn't trigger on_reply_route as per docs.

Any clue is welcome.

I'm using opensips 2.3.6

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: locally generated replies

Bogdan-Andrei Iancu-2
Hi Pasan,

Using t_on_reply() is harmless as it simply set a holder in the msg/transaction, nothing more. And a locally generated reply does NOT trigger on_reply_route - this route is triggered only by *RECEIVED* replies.

So the bottom problem for you is why the processes stuck ? do they stuck like consuming cpu (doing some crazy looping) or blocking in some I/O (no CPU usage) ?

Regards,
Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  https://www.opensips-solutions.com
OpenSIPS Summit 2019
  https://www.opensips.org/events/Summit-2019Amsterdam/
On 02/13/2019 09:36 AM, Pasan Meemaduma via Users wrote:
Hi Guys,

How's it possible locally generated replied triggering on_reply_route ? I ran in to issue where all opensips process gets stuck in processing same call replies and causing other traffic to get drop.


<info> /usr/sbin/opensips[27464]: Call: Reply from a NAT endpoint - S=408 D=Request Timeout F=<a class="moz-txt-link-freetext" href="sip:xxx@xxx">sip:xxx@xxx T=<a class="moz-txt-link-freetext" href="sip:yyy">sip:yyy
yy@x IP=a.b.c.d ID=asgasgasgas

Request process by opensips before this is an ACK request belong to the call where I don't think It'll expect a reply. Could it be an issue If I call t_on_reply on an ACK msg ?

I'm trying to figure out where the bug in my opensips routing script.  It causes all sip listerner processes to get stuck in a loop causing to generate above message. IP a.b.c.d is the sip server IP which confuse me as locally generated replies shouldn't trigger on_reply_route as per docs.

Any clue is welcome.

I'm using opensips 2.3.6


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: locally generated replies

OpenSIPS - Users mailing list
Hi Bogdan,

Thanks for the reply. writing inline replies


"Using t_on_reply() is harmless as it simply set a holder in the msg/transaction, nothing more. And a locally generated reply does NOT trigger on_reply_route - this route is triggered only by *RECEIVED* replies."

My problem was that server received 407 replies from its own source ip :/. I couldn't figure out what triggered it. Anyhow issue resolved after I removed fix_dialog_route() function which causes  BYE to send to a private ip of a NATed device. I have fix_nated_contact() before create_dialog() function but dialog data seems to have unmodified contact data which I believe is fixed in 2.4 as you mentioned.

 "So the bottom problem for you is why the processes stuck ? do they stuck like consuming cpu (doing some crazy looping) or blocking in some I/O (no CPU usage) ?"

All traffic processing workers were stuck in a loop and consume 100% CPU. I couldn't get much debug info like running opensipsctl trap when issue occured :/

Anyhow now I'm planning to upgrade thee system to 2.4 this year which has support till 2021.
my sip cluster is now  stable without fix_dialog_route() function.

On 02/13/2019 09:36 AM, Pasan Meemaduma via Users wrote:


>  
> Hi Guys,
>
>
>
>
> How's it possible locally generated replied triggering on_reply_route ? I ran in to issue where all opensips process gets stuck in processing same call replies and causing other traffic to get drop.
>
>
>
>
>
>
>
> <info> /usr/sbin/opensips[27464]: Call: Reply from a NAT endpoint - S=408 D=Request Timeout F=sip:xxx@xxx T=sip:yyy
>  
> yy@x IP=a.b.c.d ID=asgasgasgas
>
>
>
> Request process by opensips before this is an ACK request belong to the call where I don't think It'll expect a reply. Could it be an issue If I call t_on_reply on an ACK msg ?
>
>
>
>
> I'm trying to figure out where the bug in my opensips routing script.  It causes all sip listerner processes to get stuck in a loop causing to generate above message. IP a.b.c.d is the sip server IP which confuse me as locally generated replies shouldn't trigger on_reply_route as per docs.
>
>
>
>
>
> Any clue is welcome.
>
>
>
>
> I'm using opensips 2.3.6
>
>
>
>
>
>
>
> _______________________________________________
> Users mailing list
> [hidden email]
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>  



_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: locally generated replies

Bogdan-Andrei Iancu-2
Hi Pasan,

Have you checked if there is traffic on looping on your OpenSIPS ? maybe
due bad routing, OpenSIPS is keep sending to itself some traffic,
causing the loops and the load.

Regards,

Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
   https://www.opensips-solutions.com
OpenSIPS Summit 2019
   https://www.opensips.org/events/Summit-2019Amsterdam/

On 03/07/2019 03:17 PM, Pasan Meemaduma wrote:

> Hi Bogdan,
>
> Thanks for the reply. writing inline replies
>
>
> "Using t_on_reply() is harmless as it simply set a holder in the msg/transaction, nothing more. And a locally generated reply does NOT trigger on_reply_route - this route is triggered only by *RECEIVED* replies."
>
> My problem was that server received 407 replies from its own source ip :/. I couldn't figure out what triggered it. Anyhow issue resolved after I removed fix_dialog_route() function which causes  BYE to send to a private ip of a NATed device. I have fix_nated_contact() before create_dialog() function but dialog data seems to have unmodified contact data which I believe is fixed in 2.4 as you mentioned.
>
>   "So the bottom problem for you is why the processes stuck ? do they stuck like consuming cpu (doing some crazy looping) or blocking in some I/O (no CPU usage) ?"
>
> All traffic processing workers were stuck in a loop and consume 100% CPU. I couldn't get much debug info like running opensipsctl trap when issue occured :/
>
> Anyhow now I'm planning to upgrade thee system to 2.4 this year which has support till 2021.
> my sip cluster is now  stable without fix_dialog_route() function.
>
> On 02/13/2019 09:36 AM, Pasan Meemaduma via Users wrote:
>
>
>>    
>> Hi Guys,
>>
>>
>>
>>
>> How's it possible locally generated replied triggering on_reply_route ? I ran in to issue where all opensips process gets stuck in processing same call replies and causing other traffic to get drop.
>>
>>
>>
>>
>>
>>
>>
>> <info> /usr/sbin/opensips[27464]: Call: Reply from a NAT endpoint - S=408 D=Request Timeout F=sip:xxx@xxx T=sip:yyy
>>    
>> yy@x IP=a.b.c.d ID=asgasgasgas
>>
>>
>>
>> Request process by opensips before this is an ACK request belong to the call where I don't think It'll expect a reply. Could it be an issue If I call t_on_reply on an ACK msg ?
>>
>>
>>
>>
>> I'm trying to figure out where the bug in my opensips routing script.  It causes all sip listerner processes to get stuck in a loop causing to generate above message. IP a.b.c.d is the sip server IP which confuse me as locally generated replies shouldn't trigger on_reply_route as per docs.
>>
>>
>>
>>
>>
>> Any clue is welcome.
>>
>>
>>
>>
>> I'm using opensips 2.3.6
>>
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> Users mailing list
>> [hidden email]
>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>    
>


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: locally generated replies

Bogdan-Andrei Iancu-2
Hi Pasan,

I doubt that fix_route_dialog() is doing something wrong :D. Maybe there is something wrong with the signaling / cfg. After the call is established, do a "opensipsctl fifo dlg_list <callid>" to see the dialog details as learned by OpenSIPS.

Also post the incoming BYE, to see how does it "fit" to the dialog and what should be the effect of the "fixing".

Regards,
Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  https://www.opensips-solutions.com
OpenSIPS Summit 2019
  https://www.opensips.org/events/Summit-2019Amsterdam/
On 03/08/2019 11:28 AM, Pasan Meemaduma wrote:
Hi Bogdan,

Yes, that was the theory I had,  And after I remove the fix_route_dialog function it
didn't reoccur yet, so I believe fix_route_dialog caused the bad routing in my script I think.


On Friday, 8 March 2019, 1:11:38 PM GMT+5:30, Bogdan-Andrei Iancu [hidden email] wrote:


Hi Pasan,

Have you checked if there is traffic on looping on your OpenSIPS ? maybe
due bad routing, OpenSIPS is keep sending to itself some traffic,
causing the loops and the load.

Regards,

Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  https://www.opensips-solutions.com
OpenSIPS Summit 2019
  https://www.opensips.org/events/Summit-2019Amsterdam/

On 03/07/2019 03:17 PM, Pasan Meemaduma wrote:
> Hi Bogdan,
>
> Thanks for the reply. writing inline replies
>
>
> "Using t_on_reply() is harmless as it simply set a holder in the msg/transaction, nothing more. And a locally generated reply does NOT trigger on_reply_route - this route is triggered only by *RECEIVED* replies."
>
> My problem was that server received 407 replies from its own source ip :/. I couldn't figure out what triggered it. Anyhow issue resolved after I removed fix_dialog_route() function which causes  BYE to send to a private ip of a NATed device. I have fix_nated_contact() before create_dialog() function but dialog data seems to have unmodified contact data which I believe is fixed in 2.4 as you mentioned.
>
>  "So the bottom problem for you is why the processes stuck ? do they stuck like consuming cpu (doing some crazy looping) or blocking in some I/O (no CPU usage) ?"
>
> All traffic processing workers were stuck in a loop and consume 100% CPU. I couldn't get much debug info like running opensipsctl trap when issue occured :/
>
> Anyhow now I'm planning to upgrade thee system to 2.4 this year which has support till 2021.
> my sip cluster is now  stable without fix_dialog_route() function.
>
> On 02/13/2019 09:36 AM, Pasan Meemaduma via Users wrote:
>
>
>>   
>> Hi Guys,
>>
>>
>>
>>
>> How's it possible locally generated replied triggering on_reply_route ? I ran in to issue where all opensips process gets stuck in processing same call replies and causing other traffic to get drop.
>>
>>
>>
>>
>>
>>
>>
>> <info> /usr/sbin/opensips[27464]: Call: Reply from a NAT endpoint - S=408 D=Request Timeout F=sip:[hidden email] T=<a class="moz-txt-link-freetext" href="sip:yyy">sip:yyy
>>   
>> [hidden email] IP=a.b.c.d ID=asgasgasgas
>>
>>
>>
>> Request process by opensips before this is an ACK request belong to the call where I don't think It'll expect a reply. Could it be an issue If I call t_on_reply on an ACK msg ?
>>
>>
>>
>>
>> I'm trying to figure out where the bug in my opensips routing script.  It causes all sip listerner processes to get stuck in a loop causing to generate above message. IP a.b.c.d is the sip server IP which confuse me as locally generated replies shouldn't trigger on_reply_route as per docs.
>>
>>
>>
>>
>>
>> Any clue is welcome.
>>
>>
>>
>>
>> I'm using opensips 2.3.6
>>
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> Users mailing list
>> [hidden email]
>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>   
>



_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: locally generated replies

OpenSIPS - Users mailing list
Hi Bogdan,

I'll try to get this info and examine.
Thanks for the follow up :)
On Monday, 11 March 2019, 1:43:13 PM GMT+5:30, Bogdan-Andrei Iancu <[hidden email]> wrote:


Hi Pasan,

I doubt that fix_route_dialog() is doing something wrong :D. Maybe there is something wrong with the signaling / cfg. After the call is established, do a "opensipsctl fifo dlg_list <callid>" to see the dialog details as learned by OpenSIPS.

Also post the incoming BYE, to see how does it "fit" to the dialog and what should be the effect of the "fixing".

Regards,
Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  https://www.opensips-solutions.com
OpenSIPS Summit 2019
  https://www.opensips.org/events/Summit-2019Amsterdam/
On 03/08/2019 11:28 AM, Pasan Meemaduma wrote:
Hi Bogdan,

Yes, that was the theory I had,  And after I remove the fix_route_dialog function it
didn't reoccur yet, so I believe fix_route_dialog caused the bad routing in my script I think.


On Friday, 8 March 2019, 1:11:38 PM GMT+5:30, Bogdan-Andrei Iancu [hidden email] wrote:


Hi Pasan,

Have you checked if there is traffic on looping on your OpenSIPS ? maybe
due bad routing, OpenSIPS is keep sending to itself some traffic,
causing the loops and the load.

Regards,

Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  https://www.opensips-solutions.com
OpenSIPS Summit 2019
  https://www.opensips.org/events/Summit-2019Amsterdam/

On 03/07/2019 03:17 PM, Pasan Meemaduma wrote:
> Hi Bogdan,
>
> Thanks for the reply. writing inline replies
>
>
> "Using t_on_reply() is harmless as it simply set a holder in the msg/transaction, nothing more. And a locally generated reply does NOT trigger on_reply_route - this route is triggered only by *RECEIVED* replies."
>
> My problem was that server received 407 replies from its own source ip :/. I couldn't figure out what triggered it. Anyhow issue resolved after I removed fix_dialog_route() function which causes  BYE to send to a private ip of a NATed device. I have fix_nated_contact() before create_dialog() function but dialog data seems to have unmodified contact data which I believe is fixed in 2.4 as you mentioned.
>
>  "So the bottom problem for you is why the processes stuck ? do they stuck like consuming cpu (doing some crazy looping) or blocking in some I/O (no CPU usage) ?"
>
> All traffic processing workers were stuck in a loop and consume 100% CPU. I couldn't get much debug info like running opensipsctl trap when issue occured :/
>
> Anyhow now I'm planning to upgrade thee system to 2.4 this year which has support till 2021.
> my sip cluster is now  stable without fix_dialog_route() function.
>
> On 02/13/2019 09:36 AM, Pasan Meemaduma via Users wrote:
>
>
>>   
>> Hi Guys,
>>
>>
>>
>>
>> How's it possible locally generated replied triggering on_reply_route ? I ran in to issue where all opensips process gets stuck in processing same call replies and causing other traffic to get drop.
>>
>>
>>
>>
>>
>>
>>
>> <info> /usr/sbin/opensips[27464]: Call: Reply from a NAT endpoint - S=408 D=Request Timeout F=sip:[hidden email] T=sip:yyy
>>   
>> [hidden email] IP=a.b.c.d ID=asgasgasgas
>>
>>
>>
>> Request process by opensips before this is an ACK request belong to the call where I don't think It'll expect a reply. Could it be an issue If I call t_on_reply on an ACK msg ?
>>
>>
>>
>>
>> I'm trying to figure out where the bug in my opensips routing script.  It causes all sip listerner processes to get stuck in a loop causing to generate above message. IP a.b.c.d is the sip server IP which confuse me as locally generated replies shouldn't trigger on_reply_route as per docs.
>>
>>
>>
>>
>>
>> Any clue is welcome.
>>
>>
>>
>>
>> I'm using opensips 2.3.6
>>
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> Users mailing list
>> [hidden email]
>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>   
>



_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users