Crashesh with SSL

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Crashesh with SSL

Vasil Kolev
Hi,

I'm using 1.5 (trunk) with some small changes and trying to run it with
TLS. I see all kinds of weird crashes related to libssl, which I'm
pretty sure are related to a race condition somewhere, which I have hard
time finding. So I decided to try running this in non-forking mode, to
see what's happening, and turns out that in non-forking mode you can
have only one socket, which is UDP, e.g. I can't run it TCP only.
(also, in tcp-only mode the initialisation of the 'acc' module
segfaults, I'll also look into that)

I'm on revision 5334 of the trunk currently.

Any ideas on this, e.g. should I be able to run in tcp-only mode
non-forked? And does someone have such issues with SSL ?


--
Regards,
Vasil Kolev
Attractel NV
dCAP #1324, LPIC2


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Crashesh with SSL

Bogdan-Andrei Iancu
Hi Vasil,

Vasil Kolev wrote:
> Hi,
>
> I'm using 1.5 (trunk) with some small changes and trying to run it with
> TLS. I see all kinds of weird crashes related to libssl, which I'm
> pretty sure are related to a race condition somewhere, which I have hard
> time finding.
Can you post some more info on this, like logs, backtrace, anything that
might help in locating the problem?

>  So I decided to try running this in non-forking mode, to
> see what's happening, and turns out that in non-forking mode you can
> have only one socket, which is UDP, e.g. I can't run it TCP only.
> (also, in tcp-only mode the initialisation of the 'acc' module
> segfaults, I'll also look into that)
>
> I'm on revision 5334 of the trunk currently.
>
> Any ideas on this, e.g. should I be able to run in tcp-only mode
> non-forked? And does someone have such issues with SSL ?
>  
no, there is not way to start in no-fork mode with TCP/TLS, as the
TCP/TLS manager is a separate process that needs to fork :).

Regards,
Bogdan
>
>  


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Crashesh with SSL

Vasil Kolev
В 17:04 +0200 на 20.02.2009 (пт), Bogdan-Andrei Iancu написа:

> Hi Vasil,
>
> Vasil Kolev wrote:
> > Hi,
> >
> > I'm using 1.5 (trunk) with some small changes and trying to run it with
> > TLS. I see all kinds of weird crashes related to libssl, which I'm
> > pretty sure are related to a race condition somewhere, which I have hard
> > time finding.
> Can you post some more info on this, like logs, backtrace, anything that
> might help in locating the problem?


I've been debugging this for a few days, the backtraces are useless,
like this one:

(gdb) bt
#0  0x00000000 in ?? ()
#1  0xb7e495ab in lh_retrieve () from /usr/lib/i686/cmov/libcrypto.so.0.9.8
#2  0xb7dde7b2 in ?? () from /usr/lib/i686/cmov/libcrypto.so.0.9.8
#3  0xb55b6168 in ?? ()
#4  0xbf96c8e0 in ?? ()
#5  0xb7eb2ed0 in ?? () from /usr/lib/i686/cmov/libcrypto.so.0.9.8
#6  0x0000012c in ?? ()
#7  0x00000000 in ?? ()

or like this:

#0  0x00000000 in ?? ()
#1  0xb7e945ab in lh_retrieve () from /usr/lib/i686/cmov/libcrypto.so.0.9.8
#2  0xb7e297b2 in ?? () from /usr/lib/i686/cmov/libcrypto.so.0.9.8
#3  0xb5601168 in ?? ()
#4  0xbfab6340 in ?? ()
#5  0xb7efded0 in ?? () from /usr/lib/i686/cmov/libcrypto.so.0.9.8
#6  0x0000012c in ?? ()
#7  0xb7f768b8 in ?? () from /usr/lib/i686/cmov/libssl.so.0.9.8
#8  0xb7efded0 in ?? () from /usr/lib/i686/cmov/libcrypto.so.0.9.8
#9  0x00000001 in ?? ()
#10 0xb7f51767 in ssl3_clear () from /usr/lib/i686/cmov/libssl.so.0.9.8
#11 0xb7e29d5c in ?? () from /usr/lib/i686/cmov/libcrypto.so.0.9.8
#12 0xb57d6f28 in ?? ()
#13 0xb57d6f28 in ?? ()
#14 0xbfab6388 in ?? ()
#15 0xb7f58b02 in tls1_clear () from /usr/lib/i686/cmov/libssl.so.0.9.8
#16 0xb7e294c9 in CRYPTO_new_ex_data () from /usr/lib/i686/cmov/libcrypto.so.0.9.8
#17 0xb7f6660c in SSL_new () from /usr/lib/i686/cmov/libssl.so.0.9.8
#18 0x08129d48 in tls_tcpconn_init (c=0xb57c6e10, sock=24) at tls/tls_server.c:610
#19 0x080b4e79 in tcpconn_new (sock=24, su=0xbfab6518, ba=0x819fa40, type=3, state=3) at tcp_main.c:402
#20 0x080b84a0 in handle_new_connect (si=0x819fa40) at tcp_main.c:966
#21 0x080bdbef in tcp_main_loop () at io_wait.h:727
#22 0x0806e38e in main (argc=1, argv=0xbfab67d4) at main.c:847

(here when I looked at tls_tcpconn_init, it looked like something went
really wrong on the stack, as for example 'dom' was 0x52 and stuff like
that, and the line of the crash seemed the wrong one, looking at the
variables and the code flow)


After some time, I have managed to narrow it down to the following: if
the xcap_client module is present and used by presence_xml, crashes
occur, otherwise it works fine.

I'll look into the xcap client, but currently I've started using the
'integrated_xcap_server' option.

--
Regards,
Vasil Kolev
Attractel NV
dCAP #1324, LPIC2


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Crashesh with SSL

Klaus Darilion
Are you using another module which uses libssl? (e.g. identity, )
libssl is not reentrant.

Klaus

Vasil Kolev schrieb:

> В 17:04 +0200 на 20.02.2009 (пт), Bogdan-Andrei Iancu написа:
>> Hi Vasil,
>>
>> Vasil Kolev wrote:
>>> Hi,
>>>
>>> I'm using 1.5 (trunk) with some small changes and trying to run it with
>>> TLS. I see all kinds of weird crashes related to libssl, which I'm
>>> pretty sure are related to a race condition somewhere, which I have hard
>>> time finding.
>> Can you post some more info on this, like logs, backtrace, anything that
>> might help in locating the problem?
>
>
> I've been debugging this for a few days, the backtraces are useless,
> like this one:
>
> (gdb) bt
> #0  0x00000000 in ?? ()
> #1  0xb7e495ab in lh_retrieve () from /usr/lib/i686/cmov/libcrypto.so.0.9.8
> #2  0xb7dde7b2 in ?? () from /usr/lib/i686/cmov/libcrypto.so.0.9.8
> #3  0xb55b6168 in ?? ()
> #4  0xbf96c8e0 in ?? ()
> #5  0xb7eb2ed0 in ?? () from /usr/lib/i686/cmov/libcrypto.so.0.9.8
> #6  0x0000012c in ?? ()
> #7  0x00000000 in ?? ()
>
> or like this:
>
> #0  0x00000000 in ?? ()
> #1  0xb7e945ab in lh_retrieve () from /usr/lib/i686/cmov/libcrypto.so.0.9.8
> #2  0xb7e297b2 in ?? () from /usr/lib/i686/cmov/libcrypto.so.0.9.8
> #3  0xb5601168 in ?? ()
> #4  0xbfab6340 in ?? ()
> #5  0xb7efded0 in ?? () from /usr/lib/i686/cmov/libcrypto.so.0.9.8
> #6  0x0000012c in ?? ()
> #7  0xb7f768b8 in ?? () from /usr/lib/i686/cmov/libssl.so.0.9.8
> #8  0xb7efded0 in ?? () from /usr/lib/i686/cmov/libcrypto.so.0.9.8
> #9  0x00000001 in ?? ()
> #10 0xb7f51767 in ssl3_clear () from /usr/lib/i686/cmov/libssl.so.0.9.8
> #11 0xb7e29d5c in ?? () from /usr/lib/i686/cmov/libcrypto.so.0.9.8
> #12 0xb57d6f28 in ?? ()
> #13 0xb57d6f28 in ?? ()
> #14 0xbfab6388 in ?? ()
> #15 0xb7f58b02 in tls1_clear () from /usr/lib/i686/cmov/libssl.so.0.9.8
> #16 0xb7e294c9 in CRYPTO_new_ex_data () from /usr/lib/i686/cmov/libcrypto.so.0.9.8
> #17 0xb7f6660c in SSL_new () from /usr/lib/i686/cmov/libssl.so.0.9.8
> #18 0x08129d48 in tls_tcpconn_init (c=0xb57c6e10, sock=24) at tls/tls_server.c:610
> #19 0x080b4e79 in tcpconn_new (sock=24, su=0xbfab6518, ba=0x819fa40, type=3, state=3) at tcp_main.c:402
> #20 0x080b84a0 in handle_new_connect (si=0x819fa40) at tcp_main.c:966
> #21 0x080bdbef in tcp_main_loop () at io_wait.h:727
> #22 0x0806e38e in main (argc=1, argv=0xbfab67d4) at main.c:847
>
> (here when I looked at tls_tcpconn_init, it looked like something went
> really wrong on the stack, as for example 'dom' was 0x52 and stuff like
> that, and the line of the crash seemed the wrong one, looking at the
> variables and the code flow)
>
>
> After some time, I have managed to narrow it down to the following: if
> the xcap_client module is present and used by presence_xml, crashes
> occur, otherwise it works fine.
>
> I'll look into the xcap client, but currently I've started using the
> 'integrated_xcap_server' option.
>

_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Crashesh with SSL

Vasil Kolev
В 10:38 +0100 на 25.02.2009 (ср), Klaus Darilion написа:
> Are you using another module which uses libssl? (e.g. identity, )
> libssl is not reentrant.
>
> Klaus

Hm. After removing the xcap_client module, it stopped crashing, so that
might be the reason...

>  
--
Regards,
Vasil Kolev
Attractel NV
dCAP #1324, LPIC2


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Crashesh with SSL

Bogdan-Andrei Iancu
Hi Vasil,

Strange, because the xcap_client does not use libssl (it links against
xml and curl libs)...

The trace points to the place were a new TLS connection is accepted.

Maybe the problem is related to memory override........let me see....

Regards,
Bogdan

Vasil Kolev wrote:

> В 10:38 +0100 на 25.02.2009 (ср), Klaus Darilion написа:
>  
>> Are you using another module which uses libssl? (e.g. identity, )
>> libssl is not reentrant.
>>
>> Klaus
>>    
>
> Hm. After removing the xcap_client module, it stopped crashing, so that
> might be the reason...
>
>  
>>  
>>    


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Crashesh with SSL

Vasil Kolev
В 21:28 +0200 на 25.02.2009 (ср), Bogdan-Andrei Iancu написа:
> Hi Vasil,
>
> Strange, because the xcap_client does not use libssl (it links against
> xml and curl libs)...
>
> The trace points to the place were a new TLS connection is accepted.
>
> Maybe the problem is related to memory override........let me see....
>

Well, curl links to libssl... This is probably the same issue I've seen
with php+curl+pgsql, which core dumped every time pgsql was using SSL
(e.g. pgsql is linked to libssl).


--
Regards,
Vasil Kolev
Attractel NV
dCAP #1324, LPIC2


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Reply | Threaded
Open this post in threaded view
|

Re: Crashesh with SSL

Bogdan-Andrei Iancu
I see....Here comes again the old SSL problem.....

have you tried to locally compile the curl lib, but with ssl disabled?

Regards,
Bogdan

PS: I know this is a hack, but should do it until we find a way of
solving this is a general and nicer way.

Vasil Kolev wrote:

> В 21:28 +0200 на 25.02.2009 (ср), Bogdan-Andrei Iancu написа:
>  
>> Hi Vasil,
>>
>> Strange, because the xcap_client does not use libssl (it links against
>> xml and curl libs)...
>>
>> The trace points to the place were a new TLS connection is accepted.
>>
>> Maybe the problem is related to memory override........let me see....
>>
>>    
>
> Well, curl links to libssl... This is probably the same issue I've seen
> with php+curl+pgsql, which core dumped every time pgsql was using SSL
> (e.g. pgsql is linked to libssl).
>
>
>  


_______________________________________________
Users mailing list
[hidden email]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users