Client Certificate issues
Two days, and two companies looking at this problem - resolved 4 hours before go-live.
For some reason a WCF application deployed to a production server refused to accept connections. It was not accepting client certificates. Web.configs were re-written, IIS settings changed everywhere and walls head-butted.
We finally restored everything to normality but the error persisted in production only:
The HTTP request was forbidden with client authentication scheme 'Anonymous'.
I can't tell you how long I spent looking at the Anonymous access button in IIS just in case it wasn't really enabled. The tell-tell sign to the cause was when trying to open the WCF endpoint in a browser. No client certificates from the local store were available to select.
This finally lead a colleague's Google Kung Fu to find this article by Jonathan Demarks. The issue it seems is too many root certificates. A strange position to be in, especially as the root certificates are updated by Windows Update. Annoyingly the related warnings do not persist themselves in the event log on each request, so our warning was a long way down that nobody had seen.
The obvious answer is to clear out some of the root certificates - but this leaves you with the possibility of it happening again the next time they are updated. The more permanent solution is to disable the list of accepted CA's being sent to the client during the TLS handshake. This doesn't mean that any old certificate can be used, but that the client has the freedom to select any old certificate. If the server doesn't trust the CA then it still won't work.
As per KB2464556 method 3, you need to add a DWORD "SendTrustedIssuerList" to HKLM/SYSTEM/CurrentControlSet/Control/SecurityProviders/SCHANNEL with a value of 0.
Job done.