CryptoAPI and the headaches that it caused

Many moons ago I created an encryption dll for our company website. This dll was created in VB6 and using the CryptoAPI. OK, so I cheated a bit and used some encryption code that was created by Fredrik Qvarfort. But why reinvent the wheel? Or the CryptoAPI calls? Anyways, during a routine security patch deployment the encryption dll on one of the servers stopped working. The funny thing was it was working fine on the other machines. So we reinstalled the web application, uninstalled and reinstalled the COM dlls as well as some system dlls. But the dll was still not working. So I had to do what I didn’t want to do, write some debug code. I cracked open VB6 and wondered where all my intellisense, formatting and inline help went. But after a few minutes it all came back. Like an old glove, worn but it still fits.

The problem stemmed from the CryptAcquireContext function call. The CryptAcquireContext function gets a handle to the key container (where the keys are stored). Here is the code we were using:

  ‘Get handle to CSP
  If (CryptAcquireContext(hCryptProv, KEY_CONTAINER, SERVICE_PROVIDER, PROV_RSA_FULL, CRYPT_MACHINE_KEYSET) = 0) Then
    If (CryptAcquireContext(hCryptProv, KEY_CONTAINER, SERVICE_PROVIDER, PROV_RSA_FULL, CRYPT_MACHINE_KEYSET + CRYPT_NEWKEYSET) = 0) Then
    Call Err.Raise(vbObjectError + Err.LastDllError, , “CryptAcquireContext – ” & DetermineError(Err.LastDllError))
    End If
  End If

The first call to CryptAcquireContext attempts to gain access to an existing key container. The second attempts to create a new container if it cannot find the existing key container. The first call was giving me an error of -2146893802. The second call was sending an error of -2146893809. Big help. Luckily, Microsoft has a KB article on translating automation errors for VB. I was able to cross reference the error hex codes with another KB titled CryptAcquireContext() use and troubleshooting. And there is where I found my Rosetta Stone. Turns out that the errors -2146893802 (0x80090016 or NTE_BAD_KEYSET) and -2146893809 (0x80090019 or NTE_EXISTS) together implies that access to the key container is denied. Great! But where are the key containers stored? Through a bit more searching I found the answer, on the file system. Machine keys are kept at \Documents and settings\All Users\Application Data\Microsoft\Crypto\RSA\Machinekeys. User keys are kept at \Documents and settings\\Application\Data\Microsoft\Crypto\RSA\. Once we deleted the key container the dll worked perfectly! Whew, that was a close one! Dealing with the CryptoAPI directly is a nightmare and I’ve got the proof. I’m going to go crack open .Net now and regain some sanity into my life.

UPDATE: When we deleted the key container it looks like we deleted other key containers as well. This caused IIS to fail and and we had to reinstall IIS.