Pages

Sunday, February 17, 2013

Load SSL certificate from memory using cURL and CyaSSL

The cacertinmem sample in cURL shows you how to do this with OpenSSL, but it's slightly different if you use CyaSSL, and actually simpler.

From the CyaSSL manual:

To use the extension define the constant NO_FILESYSTEM and the following functions will be made available:
int CyaSSL_CTX_load_verify_buffer(...)
int CyaSSL_CTX_use_certificate_buffer(...)
int CyaSSL_CTX_use_PrivateKey_buffer(...)
int CyaSSL_CTX_use_certificate_chain_buffer(...)
Use these functions exactly like their counterparts that are named fileinstead of buffer. 
And instead of providing a filename provide a memory buffer

Use the NO_FILESYSTEM preprocessor definition when building CyaSSL, cURL, as well as in your own project.

In my case, I wanted to load a bundle of CA certificates, and would therefore use CyaSSL_CTX_load_verify_buffer() (please turn to the CyaSSL API reference).

In addition, you must be able to use the SSL callback function which isn't supported in cURL (version 7.29.0) when using SSL libraries other than OpenSSL. To solve this problem, read my article about the SSL context function with cURL and CyaSSL.

Bundle of CA root certificates

Useful when you don't know which SSL providers were used. I got it from curl.haxx.se.

I compressed the CA certificates with GZip before embedding them. Poco (C++ framework) took care of decompresson.

Reduction: 115 KB / ~45% (245 KB down to 130 KB).

Probably sensible enough if GZip (or whatever you want to use) is used in your application already elsewhere (thinking about increased code size).

main.cpp
#include "CaCert.h"

#include <curl/curl.h>
#include <cyassl/ssl.h>

#include <string>

// ...

CURLcode sslContextCallback(void* ctx) {
    // Load CA certificates from memory
    CaCert cert;
    const std::string& certData = cert.GetData();
    CyaSSL_CTX_load_verify_buffer(
        reinterpret_cast<CYASSL_CTX*>(ctx),
        reinterpret_cast<const unsigned char*>(certData.c_str()),
        certData.size(),
        SSL_FILETYPE_PEM);
    return CURLE_OK;
}

// ...
curl_easy_setopt(curl, CURLOPT_SSL_CTX_FUNCTION, *sslContextCallback);
CaCert.h
#ifndef __CaCert__
#define __CaCert__

#include <string>

class CaCert {
public:
    const std::string& GetData();
private:
    static const unsigned char m_cacert_pem_gz[];
    std::string m_cacert_pem;
};

#endif // guard
CaCert.cpp
#include "CaCert.h"
#include "cacert.pem.gz.h"
#include <Poco/InflatingStream.h>
#include <sstream>

const std::string& CaCert::GetData() {
    if (m_cacert_pem.empty()) {
        std::string gzstr(reinterpret_cast<const char*>(&m_cacert_pem_gz[0]), sizeof(m_cacert_pem_gz));
        std::istringstream istr(gzstr);
        Poco::InflatingInputStream inflater(istr, Poco::InflatingStreamBuf::STREAM_GZIP);
        std::stringstream ostr;
        ostr << inflater.rdbuf();
        m_cacert_pem = ostr.str();
    }

    return m_cacert_pem;
}
cacert.pem.gz.h

This file was generated using Hex Workshop and modified slightly.

// Generated by BreakPoint Software's Hex Workshop v6.6.1.5158
//   http://www.hexworkshop.com
//   http://www.bpsoft.com
//
//  Source File: C:\code\curltest\cacert.pem.gz
//         Time: 17.02.2013 04:55
// Orig. Offset: 0 / 0x00000000
//       Length: 133503 / 0x0002097F (bytes)
const unsigned char CaCert::m_cacert_pem_gz[133503] = { /*...*/ };

Reduce size and memory footprint

We don't always need all of the CA root certificates. We can remove all of the certificates we don't need, and any comments. In my case, I needed only StartCom certificates.

I skipped compression since the file was only 7.37 KB.

Reduction: 236.63 KB / ~96.9%.

main.cpp
#include "StartComCaCert.h"

#include <curl/curl.h>
#include <cyassl/ssl.h>

#include <string>

// ...

CURLcode sslContextCallback(void* ctx) {
    // Load CA certificates from memory
    StartComCaCert cert;
    const unsigned char* certData = cert.GetData();
    CyaSSL_CTX_load_verify_buffer(
        reinterpret_cast<CYASSL_CTX*>(ctx),
        certData,
        cert.GetSize(),
        SSL_FILETYPE_PEM);
    return CURLE_OK;
}

// ...
curl_easy_setopt(curl, CURLOPT_SSL_CTX_FUNCTION, *sslContextCallback);
StartComCaCert.h
#ifndef __StartComCaCert__
#define __StartComCaCert__

#include <string>

class StartComCaCert {
public:
    const unsigned char* GetData() const;
    unsigned int GetSize() const;
private:
    static const unsigned char m_cacert_StartCom_pem[];
};

#endif // guard
StartComCaCert.cpp
#include "StartComCaCert.h"
#include "cacert_StartCom.pem.h"

const unsigned char* StartComCaCert::GetData() const {
    return &m_cacert_StartCom_pem[0];
}

unsigned int StartComCaCert::GetSize() const {
    return sizeof(m_cacert_StartCom_pem);
}
cacert_StartCom.pem.h

This file was also generated.

// Generated by BreakPoint Software's Hex Workshop v6.6.1.5158
//   http://www.hexworkshop.com
//   http://www.bpsoft.com
//
//  Source File: C:\code\curltest\curltest\cacert_StartCom.pem
//         Time: 18.02.2013 01:23
// Orig. Offset: 0 / 0x00000000
//       Length: 7547 / 0x00001D7B (bytes)
const unsigned char StartComCaCert::m_cacert_StartCom_pem[7547] = { /*...*/ }

It'll be ridiculous to continue, but let's do it!

Looking at the certificates as they are stored in PEM format, our precious bytes are obviously wasted while it tries to look pretty to humans.

While keeping the base-64-encoded data (which also wastes space), let's go ahead and write a small Python script to minify the file.

minify_cert_pem.py

It removes everything at the beginning of the file (comments, newlines), removes repeated equal signs, and newlines mixed in the base-64-encoded data.

import re

def stripNewlines(match):
    return match.group(1) + match.group(2) + match.group(3).replace("\n", "") + match.group(4)

inFile = open("cacert_StartCom.pem", "r")
caCertData = inFile.read()

# Strip comments and everything else in the beginning of the file
result = re.sub(r"(?s)^.*?\n(\w)", r"\1", caCertData)
# Strip the newlines we can strip
result = re.sub(r"(?s)(=)+\s*(\s-----BEGIN CERTIFICATE-----\s)(.*?)(\s-----END CERTIFICATE-----\s)\s*", stripNewlines, result)
# Strip remaining newlines at the end
result = result.rstrip()
outFile = open("cacert_StartCom.min.pem", "w")
outFile.write(result)

Reduction: 175 bytes / ~2.32%.

Just for the sake of it, minify the original cacert.pem

Reduction: 7.53 KB / 3%.
Comparing old and new GZipped file: 5.98 KB / 4.59% reduction.
Total reduction: 121 KB / 49.32%.

SSL context function with cURL and CyaSSL

For some reason, cURL disables this functionality when using SSL libraries other than OpenSSL. To solve the problem, we need to dive into the cURL source code for a bit.

This code applies for cURL version 7.29.0.

Source file: cyassl.c, cyassl_connect_step1()

Declare and define variable at the top:

CURLcode retcode = CURLE_OK;

The rest of the code:

#ifndef NO_FILESYSTEM
// ...
#else
/* give application a chance to interfere with SSL set up. */
if(data->set.ssl.fsslctx) {
    retcode = (*data->set.ssl.fsslctx)(data, conssl->ctx, data->set.ssl.fsslctxp);
    if(retcode) {
        failf(data,"error signaled by ssl ctx callback");
        return retcode;
    }
}
#endif /* NO_FILESYSTEM */

Most of this was copied from ssluse.c.

Source file: url.c, Curl_setopt()

Look for the following code.

#ifdef USE_SSLEAY
    /* since these two options are only possible to use on an OpenSSL-
       powered libcurl we #ifdef them on this condition so that libcurls
       built against other SSL libs will return a proper error when trying
       to set this option! */
  case CURLOPT_SSL_CTX_FUNCTION:
    /*
     * Set a SSL_CTX callback
     */
    data->set.ssl.fsslctx = va_arg(param, curl_ssl_ctx_callback);
    break;
  case CURLOPT_SSL_CTX_DATA:
    /*
     * Set a SSL_CTX callback parameter pointer
     */
    data->set.ssl.fsslctxp = va_arg(param, void *);
    break;
  case CURLOPT_CERTINFO:
    data->set.ssl.certinfo = (0 != va_arg(param, long))?TRUE:FALSE;
    break;
#endif

Change the #ifdef:

#if defined(USE_SSLEAY) || defined(USE_CYASSL)

Project files: *.vcxproj

Add preprocessor definition if needed: USE_CYASSL

Using it

CURLcode sslContextCallback(CURL* curl, void* ctx, void* param) {
    return CURLE_OK;
}

// ...
curl_easy_setopt(curl, CURLOPT_SSL_CTX_FUNCTION, *sslContextCallback);

SSL context function with cCURLpp

The function signature for cURL can't be used with cURLpp.

Expected code (not working)

If you're familiar with cURL, you would probably try this first.

CURLcode sslContextCallback(CURL* curl, void* ctx, void* param) {
    return CURLE_OK;
}

// ...
using namespace curlpp;
Easy easy;
// ...
easy.setOpt(new options::SslCtxFunction(sslContextCallback)); // CURLOPT_SSL_CTX_FUNCTION in cURL

This would give the following error (VC++):

error C2198: 'CURLcode (__cdecl *)(CURL *,void *,void *)' : too few arguments for call

Code for cURLpp

CURLcode sslContextCallback(void* ctx) {
    return CURLE_OK;
}

// ...
using namespace curlpp;
Easy easy;
// ...
easy.setOpt(new options::SslCtxFunction(sslContextCallback)); // CURLOPT_SSL_CTX_FUNCTION in cURL

I'm not sure at this time how to get the optional extra data, but for now, I don't need it either.