Mike Stacy's Blog

Communicator cannot login, 0x80090308 SEC_E_INVALID_TOKEN

Today I hit an issue at a customer where Communicator could not login with TLS.  In the event log and a Communicator trace we would see 0x80090308 errors, which translates to SEC_E_INVALID_TOKEN.  Also, communication between the existing OCS R1 and the new R2 server would fail in one direction – R1 users could not reply to R2 IMs.  The R1 servers reported the same 80090308 error.

Log Name:      Application

Source:        Communicator

Date:          5/6/2009 9:32:16 AM

Event ID:      5

Task Category: None

Level:         Error

Keywords:      Classic

User:          N/A

Computer:      <Computer Name>

Description:

Communicator could not connect securely to server <ocspool> because the certificate presented by the server was not trusted due to validation error 0x80090308.  The issuing certificate authority (CA) for the server's certificate may not be locally trusted by the client, the certificate may be revoked, or the certificate may have expired.

Upon further investigation we discovered the following error in the System event log of the R2 server:

Event Type:        Warning

Event Source:    Schannel

Event Category:                None

Event ID:              36885

Date:                     5/6/2009

Time:                     8:42:26 AM

User:                     N/A

Computer:          <Server Name>

Description:

When asking for client authentication, this server sends a list of trusted certificate authorities to the client. The client uses this list to choose a client certificate that is trusted by the server. Currently, this server trusts so many certificate authorities that the list has grown too long. This list has thus been truncated. The administrator of this machine should review the  certificate authorities trusted for client authentication and remove those that  do not really need to be trusted.

The Trusted Root certificate store had almost 300 certificates listed because this customer had been diligent in applying the root certificate updates that are published periodically.  We deleted a large number of certificates that were not needed and ended up with around 120.  Once this was done all communication completed successfully.

Subsequently we discovered a KB article related to IAS that also addresses this issue:

http://support.microsoft.com/default.aspx?scid=kb;EN-US;933430

We did not contact PSS to acquire the hotfix so I can’t speak to whether or not it would have corrected the issue.

Exchange UM Fails to Validate Certificate

Last week I hit an issue where Play on Phone was not working.  After entering the phone number in the appropriate field, Outlook would try for 30 seconds and then give the generic error indicating that it could not connect to the UM server.  As a starting point I checked the URL that was being sent to the client (https://<CAS URL>/unifiedmessaging/service.asmx) to ensure there were no connectivity or certificate issues.  Everything checked out fine there.

Next I looked at the UM server and found an event 1113 with the following description:

The Unified Messaging server failed to exchange the required certificates with an IP gateway to enable Transport Layer Security (TLS) for an incoming call. Check that this is a configured TLS peer and that the correct certificates are being used. More information: A TLS failure occurred because the certificate that was presented by the remote server is not trusted. The error code was "-2146762487" and the message was "A certificate chain processed, but terminated in a root certificate which is not trusted by the trust provider".

This particular customer has separate servers for the mailbox, hub transport, CAS, and UM roles.  Since the client connection goes through the CAS server this would indicate that the UM server was rejecting the certificate that was presented by the CAS server.  A quick check of the certificates on the CAS server via get-exchangecertificate showed the certificate that we expected, and we could successfully browse to the same /unifiedmessaging/service.asmx URL from the UM server without any issues.  A netmon trace further confirmed that the certificate presented to the UM server was the one we expected.  Despite all this, the CAS server insisted that it was not able to find a valid UM server:

Event Type: Warning
Event Source: MSExchange Unified Messaging
Event Category: Shell
Event ID: 1083
Description:
The Unified Messaging Web service was unable to process a "PlayOnPhone" request for user "Doe, John". The error was "A valid Unified Messaging server could not be found for dial plan NorthAmerica.".

Since the CAS server was configured with a Verisign certificate this really didn’t make sense, but for the sake of completeness I checked each certificate in the chain and verified that it existed in the UM server’s local computer certificate store.  What I found was that the Verisign root certificate was indeed in the Trusted Root Certification Authorities store but the intermediate CA, from which the CAS server’s certificate was issued, was not in the Intermediate Certification Authorities store.  This didn’t seem to cause any problems from IE but just to be safe I added the intermediate certificate into the store.  Sure enough, this caused UM to begin accepting the connections from the CAS server.

I haven’t had a chance to research whether or not this is documented/expected behavior but it’s certainly interesting to note this, especially as certificate vendors are beginning to use new 2048 bit based CAs.

Entrust UC Certificates

I’ve favorably referenced the UC certs from Entrust a few times over on the OCS Forums and I noticed recently that they’ve had a pretty significant price drop.  They used to be priced at $1099 for 2 years but are now available in 1, 2, 3, and 4 year versions, all of which have been reduced in price.  Definitely worth a look if you are doing an OCS installation or getting ready to renew your certificates.

Microsoft actually worked with the certificate providers when they created the UC Cert program, which includes other providers as well (check the link for the current list).  Any time I’m working with a customer on an Edge deployment I use these certificates – they are simple to obtain, well priced, and can be easily reissued if your names change.

Common Certificate issue with Vista

I got a call from an industry colleague yesterday who was trying to set up an OCS lab environment.  He hit what has become an increasingly common issue as Vista becomes more heavily adopted in the enterprise.  The scenario is almost always the same - "My XP computers can download the address book without a problem but Vista cannot."  User Account Control is commonly suspected as the problem, but in this case it has nothing to do with the actual issue.  In fact, the issue is actually a default setting in IE7:  Check for server certificate revocation.

The error that you'll see is:
"Cannot Synchronize with the corporate address book. This may be because the proxy server setting in your web browser does not allow access to the address book. If the problem persists, contact your system administrator."

The error message is a bit misleading.  What is actually happening is that IE is checking the CRL Distribution Points property of the certificate that is provided by the web conferencing server and trying to connect to the paths specified.  If it cannot connect to any of these it will fail with the above error message.  The reason this does not cause Communicator logons to fail is because the the address book download process uses the WinInet API just like IE since the connection is https.  Communicator does not use WinInet for its logon, so the logon succeeds as long as the certificate is trusted by the client.

This scenario is most common in lab environments because the lab will often use a separate AD environment but users whose machines are joined to the production AD forest will try to connect with Communicator.  Without any trusts and/or DNS replication of the test forest, the client will not be able to connect to either the LDAP or HTTP paths referenced in the CRL Distribution Points property.  Here is an example of a CRL DP from a certificate generated by a Windows CA:

[1]CRL Distribution Point

Distribution Point Name:

Full Name:

URL=ldap:///CN=EVAN-CA,CN=EVAN-DC1,CN=CDP,CN=Public%20Key%20Services,CN=Services,CN=Configuration,DC=corp,DC=evangelyze,DC=net?certificateRevocationList?base?objectClass=cRLDistributionPoint

URL=http://evan-dc1.corp.evangelyze.net/CertEnroll/EVAN-CA.crl

You can imagine by looking at the paths above that a computer without any knowledge of the namespaces used would have a problem connecting to them.  This problem can also manifest itself in a production environment if you have machines that are not members of the same forest or for any other reason cannot resolve or connect to the paths referenced in the CRL DP.  In any case there are only 3 options:

1) Disable certificate revocation checking in IE.  This can be done manually or via group policy.  The setting can be found in the Security section of the Advanced tab in Internet Options.

2) Modify your infrastructure in such a way as to make the CRL DP accessible to all users.

3) Use a public CA to issue the certificate rather than an internal CA.  This is the best solution if you have non-domain joined users connecting to your internal pool.

Exchange UM Certificates with OCS

While Exchange 2007's UM functionality is capable of integrating with OCS for voicemail, Exchange 2007 SP1 adds new options to more easily facilitate this integration.  The information to do so is outlined here.  As you can imagine, the certificate requirements add a new level of complexity for even experienced Exchange engineers.

When Exchange 2007 is installed, it generates a self-issued certificate for use with IIS, SMTP, and SIP (if you're using UM).  This certificate generally isn't ideal for Outlook and OWA clients because it's not trusted by any machines except for the Exchange server, and one of the first tasks to do is replace this certificate with one that is trusted by the user's machines.  Additionally, the certificate you use to replace it may not match the FQDN of the server since the server may be referenced by a more generic name (for example, mail.domain.com) to make it an easy to remember OWA address.  However, one of the requirements of UM/OCS connectivity is that the certificate used for UM exactly match the FQDN of the server so that MTLS is successful.  For MTLS to be successful, the certificate must be trusted by the OCS server(s).  This means you must either import the self-signed Exchange certificate to each OCS server or replace it with one that is trusted.  Most customers will want to use a certificate that is automatically trusted by OCS.  To do this, create a new certificate to be used by Exchange (outlined towards the bottom of this article) and then use the Enable-ExchangeCertificate cmdlet to assign it to the UM service.  Once this is complete and you've followed all the other steps for UM integration you should be all set.

OCS Certificate Expiration Monitoring

Many OCS admins are not aware that OCS gives you a warning as a certificate nears expiration.  While the MOM/SCOM management pack includes alerts for these events, there are plenty of folks using other monitoring systems that won't be aware of this event until it's too late.  Here is an example of the event that is triggered:

Event Type: Warning
Event Source: OCS Protocol Stack
Event Category: (1001)
Event ID: 14398
Date:  11/14/2008
Time:  8:22:53 AM
User:  N/A
Computer: SE
Description:
The default outgoing certificate configured for secure transport will expire soon.

Outgoing Certificate for (Default) edge will expire on Thursday, December 04, 2008 at 1:10:50 PM Local Time. The certificate serial number is attached for reference.

Main events to monitor:

14398 and 14342 - A certificate will expire soon.

14399, 14341, and 14393 - A certificate has expired.

So what should you do when this happens?  The UC team has a post addressing just that question.

Special thanks to Joe Flynn for his research assistance and SCOM expertise.