Wednesday, October 21, 2015

Cross compiling libs3 for Raspberry Pi

Recently I had a need to upload files from Raspberry Pi to Amazon S3 bucket. There are at least two ways to do this: uploading through REST API and using libs3 C library. The REST API approach requires HMAC-SHA1 signature authentication. Calculating the signature is not trivial considering the various rules in constructing the signature payload. So I chose to go with libs3 approach to achieve the goal. This post explains how I cross compiled libs3 for Raspberry Pi/armhf.

Getting Source

The libs3 source is hosted in Github in the repository Clone the repository and make sure it is accessible from our build environment.

Getting Dependencies

We need header and library files of libcurl and libxml2 to build libs3. We also need library files of zlib and liblzma while building host application that uses libs3.

The easy way to get these files is to get your OS distribution packages of these libraries and extract the required files. My Raspberry Pi 2 runs Raspbian OS created from Debian Wheezy. So I downloaded Wheezy packages of these libraries for "armhf" architecture from here and extracted the files as explained here.

The hard way is to get source of all these libraries and cross compiling them for Raspberry Pi then using the resultant libraries. It is a time consuming and unnecessary exercise as the required files are readily available in the OS distribution packages. So I didn't choose to go in this way.

The libs3 can be built as static and shared library. The default build configuration builds both version. When you build libs3 static library you need the .a files from these dependencies and when you build shared library you need the .so files.

Build Environment 

Get your Raspberry Pi cross compiling environment ready as explained here. Copy the dependencies header files(i.e. "curl" and "libxml" directories) into "$HOME/raspberrypi/rootfs/usr/include". Copy the dependencies library files into "$HOME/raspberrypi/rootfs/usr/lib/arm-linux-gnueabihf".

Building The Library

Navigate to the libs3 source directory and issue the below commands to build libs3 library.

$ make CC=arm-linux-gnueabihf-gcc CURL_LIBS=$HOME/raspberrypi/rootfs/usr/lib/arm-linux-gnueabihf/libcurl.a CURL_CFLAGS=-I/opt/raspberrypi/rootfs/usr/include LIBXML2_LIBS=/$HOME/raspberrypi/rootfs/usr/lib/arm-linux-gnueabihf/libxml2.a LIBXML2_CFLAGS=-I$HOME/raspberrypi/rootfs/usr/include -f GNUmakefile

It is enough to specify the include directory "-I/opt/raspberrypi/rootfs/usr/include" once either in CURL_CFLAGS or LIBXML2_CFLAGS but if we didn't specify in both it will attempt to invoke the programs "curl-config" or "xml2-config" which we don't have.

After successfully building the library, the libs3 static and shared libraries will be available under "build/lib" directory of libs3 source directory. The directory "inc" under libs3 source directory contains the corresponding libs3 header files. Now we can either point our application build system to consume libs3 from here or copy the header and library files into the corresponding directories of "$HOME/raspberrypi/rootfs/usr" to consume them automatically via the pi.cmake toolchain file. Make sure you link zlib and lzma libraries with your application when you consume libs3, otherwise the linking process will fail with undefined symbol error.

I hope the information helps you, thanks for reading.

Thursday, December 18, 2014

Pure CMake based bin2h implementation

I had been writing CMake build script for a project that requires converting few files into C/C++ headers so that the content of those files can be embedded in the output binary.

I could've done this by executing an external bin2h program to do the conversion. But as it is a cross-platform application I need to depend on platform specific bin2h programs. A widely used approach for this is to include bin2h source in the project code base and compile it on the target platform then use the resultant executable for the bin2h conversion.

I didn't want to include the source bin2h in my project and compile it on-demand. I was thinking there should be a CMake command/function/module to do this in cross-platform way but I couldn't find anything.

So I started writing a pure CMake based bin2h implementation and end up with the following. The following module provides a CMake function "BIN2H" that can be used to convert any files into C/C++ header file. We need to specify the source file, header file and the C/C++ variable name that points to the raw bytes of the source file content. We can also specify the optional parameter 'APPEND' to append to the header file instead of overwriting and the optional parameter 'NULL_TERMINATE'  to terminate the raw bytes array with null byte.

I hope it will be useful to others.

Monday, July 07, 2014

RSA OAEP padding with SHA512 hash algorithm

Recently I wanted to encrypt a message with RSA with OAEP padding. I also wanted to use SHA512 as hashing algorithm and mask generation function(MGF) in the OAEP padding instead of SHA1. But it looks like it is not possible with the OpenSSL/libcrypto as the SHA1 hash algorithm is hard coded in OAEP padding implementation. This is confirmed by this thread in OpenSSL forum. Though the forum thread was written around 2012 but still I couldn't find a way use either SHA256 or SHA512 as my hashing algorithm and MGF in OAEP padding.

As suggested by "Dr Stephen N. Henson"(the core developer of OpenSSL) in the forum thread , I've took the implementation of RSA OAEP padding and modified to use SHA512 instead of SHA1. It is mostly just find EVP_sha1 and replace with EVP_sha512. We also need to update the usage of SHA_DIGEST_LENGTH macro to SHA512_DIGEST_LENGTH to reflect the output length of SHA512 hash. Below is the modified RSA OAEP padding implementation which uses SHA512 algorithm. Hope it helps, cheers.

Wednesday, July 02, 2014

Viewing raw RGB bitmaps with ImageMagick

I've been looking for a tool to view raw RGB/RGBA bitmaps but couldn't locate such one. It will be useful at times when we want to troubleshoot what bitmaps we are feeding to video encoder or what frames are getting produced by video decoder. The RGB bitmaps contains raw image data and doesn't carry any header info like width, height and etc. For example, if the image is 100x100 and it is a 32 bits per-pixel(RGBA) then the RGBA image will be 100 * 100 * 4 = 40,000 bytes. In this image data each pixel is represented by four bytes and the bytes present the color primary values of Red, Green, Blue & Alpha respectively.

Very recently I realized I can use ImageMagick to convert these RGB bitmaps to any viewable image formats like png or jpg and use any standard image viewer to view them. The following command uses ImageMagick's convert utility to convert the raw RGBA image to png image which can be viewed with any image viewer application.
convert.exe -depth 8 -size 2048x858 image.rgba image.png
as you can see we specify the bit depth(how many bits per pixel component) and size, without which the utility doesn't know how to interpret the image data from source image. We can also specify source and destination color space with '-colorspace' option to convert the image from one color space to another. Also the option '-alpha off' can be used if we want to ignore the alpha channel from source image.

Saturday, June 28, 2014

Enabling Direct3D fullscreen mode from non-active window

In this post I am going to tell you how can you enable Direct3D fullscreen mode from non-active window which is used as swap chain's rendering target.

Initially I didn't realize it could be challenging to enable Direct3D fullscreen mode from non-active window. I thought all I need to do is call the SetFullscreenState method of the SwapChain which is also worked fine from test application. But when I tried the same thing from the real WinForm application which was not active/foreground window at the time of enabling fullscreen, it didn't work. I was getting DXGI_ERROR_NOT_CURRENTLY_AVAILABLE error from SetFullscreenState method.

When I checked MSDN page about the SetFullscreenState method I can clearly see why it is failing.
DXGI_ERROR_NOT_CURRENTLY_AVAILABLE if the action failed. There are many reasons why a windowed-mode swap chain cannot switch to full-screen mode. For instance:
  • The application is running over Terminal Server.
  • The output window is occluded.
  • The output window does not have keyboard focus.
  • Another application is already in full-screen mode.
The third point mentioned in the MSDN page is clearly my situation. So I thought I can simply set the keyboard focus to my WinForm application window using SetForegroundWindow Win32 API. But it seems it is not as simple as it sounds, Windows impose many restrictions on SetForegroundWindow API which are explained in detail in the MSDN page. Basically a non-foreground window can't set itself or another window as foreground window. But a foreground window can set another window as foreground window.

Apparently there are ways/hacks in which we can overcome this restriction and they are explained here, here and here. I tried the first solution mentioned in the post here and it worked fine. The solution is wrapped inside the following function, the hwnd is the handle of target window which want it to be foreground/active window.

void ActivateWindow(HWND hwnd)
    DWORD foregroundThread = ::GetWindowThreadProcessId(GetForegroundWindow(), NULL);
    DWORD thisThread = ::GetCurrentThreadId();

    if (foregroundThread != thisThread)
        BOOL isAttached = ::AttachThreadInput(foregroundThread, thisThread, TRUE);

        ::ShowWindow(hwnd, SW_SHOW);

        if (isAttached)
            ::AttachThreadInput(foregroundThread, thisThread, FALSE);

The hack involves temporarily attaching our thread as the active foreground window's input processing thread and trying to bring the our window to the top most. This works as we fake our window as the active foreground window by hooking its input processing mechanism and Windows thinks we are the active foreground window and wants to change the focus. After bring our application to the top most we revert the input processing mechanism to its original thread as our goal is achieved. Once the window becomes active window then full screen mode enabled without any problem.

I hope the above description of my experience and the code snippet helps you, cheers.

Sunday, March 17, 2013

Troubleshooting .NET "module could not be found" exception with ProcessMonitor

In this post I am going to share a technique to troubleshoot "module could not be found" exception in managed/.NET application due to missing native DLL dependency. The difficulty in troubleshooting this exception is it doesn't provide any information about the missing DLL(e.g. file name) apart from the exception message "module could not be found".

The Problem
To demonstrate the problem I am going to create an managed/.NET console app which depends on C++/CLI DLL which in-turn depends on a native Win32 DLL. The dependency looks like below

and the implementation of executable and DLLs are
// ManagedApp.cs
namespace ManagedApp
    class Program
        static void Main(string[] args)
// CppCliDll.h
#pragma once

#include <NativeDll/NativeClass.h>

namespace CppCliDll
    public ref class CppCliClass
        static void Print()
// NativeDll.h
#define NATIVEDLL_API __declspec(dllexport)
#define NATIVEDLL_API __declspec(dllimport)

#include <stdio.h>

namespace NativeDll
    // This class is exported from the NativeDll.dll
    class NATIVEDLL_API NativeClass
        static void Print()
            printf("Hello from native dll");
for readability the DLLs are implemented in the header itself.

So with this dependency and implementation if we place all the three binaries(ManagedApp.exe, CppCliDll.dll and NativeDll.dll) in same directory and run the ManagedApp.exe we will get the following expected output.
C:\Demo> ManagedApp.exe
Hello from native dll
whereas if the NativeDll.dll is not present in the directory then we will get the "module could not be found" exception
C:\Demo&gt del NativeDll.dll
C:\Demo&gt ManagedApp.exe
Unhandled Exception: System.IO.FileNotFoundException: The specified module could not be found. (Exception from HRESULT: 0x8007007E)
   at ManagedApp.Program.Main(String[] args)
the exception is exactly same even if we run the ManagedApp.exe in Visual Studio debug mode

the exception doesn't provide any clue about which DLL is missing. If the application depends on small no. of DLLs(like the ManagedApp.exe) then it is relatively easy to guess which DLL could be missing. But it will be really difficult to guess if the application depends(directly and indirectly) on more no. of (e.g. 20+) DLLs. It will be even more difficult if one of our dependency DLL assumes presence of system wide DLL like libcrypto.dll(part of OpenSSL).

The strange thing here is if the managed DLL(here CppCliDll.dll) is not present in the directory then we'll get a exception with information explaining exactly which DLL is missing.
C:\Demo> del CppCliDll.dll
C:\Demo> ManagedApp.exe
Unhandled Exception: System.IO.FileNotFoundException: Could not load file or assembly 'CppCliDll, Version=1.0.4824.22063, Culture=neutral, PublicKeyToken=null' or one of its dependencies. The system cannot find the file specified.
File name: 'CppCliDll, Version=1.0.4824.22063, Culture=neutral, PublicKeyToken=null'
   at ManagedApp.Program.Main(String[] args)
But I don't understand why that crucial information is not available in the exception when the native dependency of managed app/DLL is missing. I also tried the Fusion logs but even that didn't provide any clue which DLL we are missing. I've wasted many hours trying to find the missing DLL before finding the following technique.

The Technique
The technique I finally found to troubleshoot this exception is using the ProcessMonitor. ProcessMonitor is a tool provided by Microsoft(originally SysInternals which is bought by Microsoft) and it can monitor system events like access to file, network, registry and etc. It provides various options like filtering in which it will show events related to specific process.

So to find out the missing DLL start the ProcessMonitor with the following capture options

  • Enable "Show File System Activity" and disable everything else
  • Add filter for "Process Name" field containing with value as our application name(it is ManagedApp for the demo app)
  • Add highlighting for "Path" field ending with value "dll"
  • Add highlighting for "Result" field containing with value "NOT FOUND"

now run the application and it will show the same error message that we've observed earlier. Now stop capturing events in the ProcessMonitor and look for for highlighted row starting from the latest event(bottom) to oldest event(top). When you find a highlighted row observe the corresponding DLL file name and make sure the same DLL file name is not opened successfully in the future(towards bottom) or past(towards top). You can also narrow down the search space by considering file opening attempt that happened only in the application directory(the directory where ManagedApp.exe present).

in the above screenshot the red highlighted row shows that an attempt to open "NativeDll.dll" failed and the loader tries to open the DLL in various other places like "C:\Windows\system" and etc. If you fully analyze the events you can confirm that all attempts to open the DLL would have failed. So this clearly shows that the missing DLL is "NativeDll.dll" file.

This approach of analyzing the ProcessMonitor file system events is little tricky and as well as time consuming. That is why I came up with the following python script which can analyze the ProcessMonitor file system events exported as CSV and it can report the missing DLLs.
C:\Demo> python Logfile.CSV
Attemmpt to open the following dlls failed:
As you can see, it also reported failure attempts to open other DLLs apart from NativeDll.dll. These are all false-positives, failure to open these DLLs will not affect the executability of the application. The source of the python script is

import sys
import os

if __name__ == '__main__':
    if len(sys.argv) != 2:
        print "Usage: FindMissingDll <process monitor csv exported file>"
    csv_lines = open(sys.argv[1]).readlines()
    # the following dictionary contains the status of all dlls. the key for the dictionary
    # is the dll file name(lower case) and value is the status of whether that dll successfully
    # opened once(an attempt to open the same file in some other path may fail which we shouldn't
    # consider as missing dll)
    dll_status = {}
    for line in csv_lines:
        fields = line.replace("\"", "").split(',')
        # process monitor CSV file fields are
        # "Time of Day","Process Name","PID","Operation","Path","Result","Detail"
        operation = fields[3]
        result = fields[5]
        if operation == "IRP_MJ_CREATE":
            path = fields[4]
            file_name = os.path.basename(path).lower()
            if not file_name.endswith(".dll"):
            if result == "SUCCESS":
                dll_status[file_name] = True
            elif result == "NAME NOT FOUND" and file_name not in dll_status:
                dll_status[file_name] = False

    if len(dll_status) > 0:
        print "Attemmpt to open the following dlls failed:"
        for file_name in dll_status:
            if not dll_status[file_name]:
                print file_name

So with this technique and the script I hope whoever struggling with "module could not be found" exception can easily find the missing DLL(s).

Sunday, November 25, 2012

Comparing X509 distinguished names with BouncyCastle

In this post I am going to show how to compare X509 distinguished name using BouncyCastle. Distinguished names are used to specify the subject and issuer identity in X509 certificate. Distinguished name(DN) is like a LDAP directory name and contains one or more relative distinguished name(RDN) separated by comma. The definition for distinguished name can be found in X500 specification.

The problem with comparing distinguished name is the relative distinguished names they contain can be in any order. So simple string comparison will not work and parsing the distinguished name is not straight forward.

But BouncyCastle provides a class named X509Name through which we can easily parse and compare the X509 distinguished names. The below code snippets demos the parsing and comparison of X509 distinguished names
static void Main(string[] args)
    var name1 = new X509Name("CN=person1, OU=dept1, O=org1");
    var name1Dup = new X509Name("O=org1, OU=dept1, CN=person1");
    var name2 = new X509Name("CN=person2, OU=dept2, O=org1");

    Console.WriteLine("name1 = {0}", name1);
    Console.WriteLine("name2Dup = {0}", name1Dup);
    Console.WriteLine("name2 = {0}", name2);

    Console.WriteLine("name1 == name1Dup => {0}", name1.Equivalent(name1Dup));
    Console.WriteLine("name1 == name2 => {0}", name1.Equivalent(name2));
and the output
name1 = CN=person1,OU=dept1,O=org1
name2Dup = O=org1,OU=dept1,CN=person1
name2 = CN=person2,OU=dept2,O=org1

name1 == name1Dup => True
name1 == name2 => False
The X509Name class makes it so simple to parse and manipulate the X509 distinguished names. But I encountered a situation where one of the relative distinguished name contained by X509 distinguished name is not recognizable by X509Name even though the underlying OID(object identifier) of the relative distinguished name is supported by the X509Name. The problematic relative distinguished name is "dnQualifier" whose OID is The same OID is supported and parsable by X509Name if the relative distinguished name is "DN" instead "dnQualifier".

Digging little deep into X509Name source reveals that we can make it to parse/support "dnQualifier" relative distinguished name like below

static void Main(string[] args)
    X509Name.DefaultLookup["dnqualifier"] = X509Name.DnQualifier;

    var name1 = new X509Name("CN=person1, OU=dept1, O=org1, dnQualifier=aaaabbbbccccddddeeeeffffggg=");
In the above code basically we are providing another name("dnQualifier") for X509Name.DnQualifier which is by default identified with name "DN".

Executing plink from non-interactive shell

Today I had a requirement to launch plink.exe(a command-line interface to the PuTTY back ends) from a C# code and this code can run as Windows service or from an ASP.NET application. I accomplished the task with the following code
The above line launches the plink program which I can see running in TaskManager but for some reason it is not connecting with the remote host. After some google search and experimentation with plink I finally found the reason why it is waiting and not connecting with the remote program.

Actually the plink was launched in non-interactive shell(as it is launched from Windows service/ASP.NET) and it runs under SYSTEM/ASP.NET user account. As we don't usually run plink in these user accounts plink sees the remote host's certificate as new unauthorized certificate and asking us to authorize it. As it is non-interactive shell we don't see the console and the authorization request.

After finding this I found that launching plink from another batch file with the following command line automatically answers "yes" to the plink's certificate authorization request and plink successfully able to execute the commands on remote host.
echo yes | C:\path\to\plink.exe {parameters}