## Sunday, March 17, 2013

### Troubleshooting .NET "module could not be found" exception with ProcessMonitor

In this post I am going to share a technique to troubleshoot "module could not be found" exception in managed/.NET application due to missing native DLL dependency. The difficulty in troubleshooting this exception is it doesn't provide any information about the missing DLL(e.g. file name) apart from the exception message "module could not be found".

The Problem
To demonstrate the problem I am going to create an managed/.NET console app which depends on C++/CLI DLL which in-turn depends on a native Win32 DLL. The dependency looks like below

and the implementation of executable and DLLs are
// ManagedApp.cs
namespace ManagedApp
{
class Program
{
static void Main(string[] args)
{
CppCliDll.CppCliClass.Print();
}
}
}

// CppCliDll.h
#pragma once

#include <NativeDll/NativeClass.h>

namespace CppCliDll
{
public ref class CppCliClass
{
public:
static void Print()
{
NativeDll::NativeClass::Print();
}
};
}

// NativeDll.h
#ifdef NATIVEDLL_EXPORTS
#define NATIVEDLL_API __declspec(dllexport)
#else
#define NATIVEDLL_API __declspec(dllimport)
#endif

#include <stdio.h>

namespace NativeDll
{
// This class is exported from the NativeDll.dll
class NATIVEDLL_API NativeClass
{
public:
static void Print()
{
printf("Hello from native dll");
}
};
}


So with this dependency and implementation if we place all the three binaries(ManagedApp.exe, CppCliDll.dll and NativeDll.dll) in same directory and run the ManagedApp.exe we will get the following expected output.
C:\Demo> ManagedApp.exe
Hello from native dll

whereas if the NativeDll.dll is not present in the directory then we will get the "module could not be found" exception
C:\Demo&gt del NativeDll.dll
C:\Demo&gt ManagedApp.exe
Unhandled Exception: System.IO.FileNotFoundException: The specified module could not be found. (Exception from HRESULT: 0x8007007E)
at ManagedApp.Program.Main(String[] args)

the exception is exactly same even if we run the ManagedApp.exe in Visual Studio debug mode

the exception doesn't provide any clue about which DLL is missing. If the application depends on small no. of DLLs(like the ManagedApp.exe) then it is relatively easy to guess which DLL could be missing. But it will be really difficult to guess if the application depends(directly and indirectly) on more no. of (e.g. 20+) DLLs. It will be even more difficult if one of our dependency DLL assumes presence of system wide DLL like libcrypto.dll(part of OpenSSL).

The strange thing here is if the managed DLL(here CppCliDll.dll) is not present in the directory then we'll get a exception with information explaining exactly which DLL is missing.
C:\Demo> del CppCliDll.dll
C:\Demo> ManagedApp.exe
Unhandled Exception: System.IO.FileNotFoundException: Could not load file or assembly 'CppCliDll, Version=1.0.4824.22063, Culture=neutral, PublicKeyToken=null' or one of its dependencies. The system cannot find the file specified.
File name: 'CppCliDll, Version=1.0.4824.22063, Culture=neutral, PublicKeyToken=null'
at ManagedApp.Program.Main(String[] args)
...

But I don't understand why that crucial information is not available in the exception when the native dependency of managed app/DLL is missing. I also tried the Fusion logs but even that didn't provide any clue which DLL we are missing. I've wasted many hours trying to find the missing DLL before finding the following technique.

The Technique
The technique I finally found to troubleshoot this exception is using the ProcessMonitor. ProcessMonitor is a tool provided by Microsoft(originally SysInternals which is bought by Microsoft) and it can monitor system events like access to file, network, registry and etc. It provides various options like filtering in which it will show events related to specific process.

So to find out the missing DLL start the ProcessMonitor with the following capture options

• Enable "Show File System Activity" and disable everything else
• Add filter for "Process Name" field containing with value as our application name(it is ManagedApp for the demo app)
• Add highlighting for "Path" field ending with value "dll"

now run the application and it will show the same error message that we've observed earlier. Now stop capturing events in the ProcessMonitor and look for for highlighted row starting from the latest event(bottom) to oldest event(top). When you find a highlighted row observe the corresponding DLL file name and make sure the same DLL file name is not opened successfully in the future(towards bottom) or past(towards top). You can also narrow down the search space by considering file opening attempt that happened only in the application directory(the directory where ManagedApp.exe present).

in the above screenshot the red highlighted row shows that an attempt to open "NativeDll.dll" failed and the loader tries to open the DLL in various other places like "C:\Windows\system" and etc. If you fully analyze the events you can confirm that all attempts to open the DLL would have failed. So this clearly shows that the missing DLL is "NativeDll.dll" file.

This approach of analyzing the ProcessMonitor file system events is little tricky and as well as time consuming. That is why I came up with the following python script which can analyze the ProcessMonitor file system events exported as CSV and it can report the missing DLLs.
C:\Demo> python FindMissingDll.py Logfile.CSV
Attemmpt to open the following dlls failed:
mscorrc.dll.dll
rpcss.dll
nativedll.dll
wow64log.dll

As you can see, it also reported failure attempts to open other DLLs apart from NativeDll.dll. These are all false-positives, failure to open these DLLs will not affect the executability of the application. The source of the python script is

import sys
import os

if __name__ == '__main__':
if len(sys.argv) != 2:
print "Usage: FindMissingDll <process monitor csv exported file>"
sys.exit(0)

# the following dictionary contains the status of all dlls. the key for the dictionary
# is the dll file name(lower case) and value is the status of whether that dll successfully
# opened once(an attempt to open the same file in some other path may fail which we shouldn't
# consider as missing dll)
dll_status = {}

for line in csv_lines:
fields = line.replace("\"", "").split(',')

# process monitor CSV file fields are
# "Time of Day","Process Name","PID","Operation","Path","Result","Detail"
operation = fields[3]
result = fields[5]
if operation == "IRP_MJ_CREATE":
path = fields[4]
file_name = os.path.basename(path).lower()

if not file_name.endswith(".dll"):
continue

if result == "SUCCESS":
dll_status[file_name] = True
dll_status[file_name] = False

if len(dll_status) > 0:
print "Attemmpt to open the following dlls failed:"
for file_name in dll_status:
if not dll_status[file_name]:
print file_name


So with this technique and the script I hope whoever struggling with "module could not be found" exception can easily find the missing DLL(s).

## Sunday, November 25, 2012

### Comparing X509 distinguished names with BouncyCastle

In this post I am going to show how to compare X509 distinguished name using BouncyCastle. Distinguished names are used to specify the subject and issuer identity in X509 certificate. Distinguished name(DN) is like a LDAP directory name and contains one or more relative distinguished name(RDN) separated by comma. The definition for distinguished name can be found in X500 specification.

The problem with comparing distinguished name is the relative distinguished names they contain can be in any order. So simple string comparison will not work and parsing the distinguished name is not straight forward.

But BouncyCastle provides a class named X509Name through which we can easily parse and compare the X509 distinguished names. The below code snippets demos the parsing and comparison of X509 distinguished names
static void Main(string[] args)
{
var name1 = new X509Name("CN=person1, OU=dept1, O=org1");
var name1Dup = new X509Name("O=org1, OU=dept1, CN=person1");
var name2 = new X509Name("CN=person2, OU=dept2, O=org1");

Console.WriteLine("name1 = {0}", name1);
Console.WriteLine("name2Dup = {0}", name1Dup);
Console.WriteLine("name2 = {0}", name2);
Console.WriteLine();

Console.WriteLine("name1 == name1Dup => {0}", name1.Equivalent(name1Dup));
Console.WriteLine("name1 == name2 => {0}", name1.Equivalent(name2));
}

and the output
name1 = CN=person1,OU=dept1,O=org1
name2Dup = O=org1,OU=dept1,CN=person1
name2 = CN=person2,OU=dept2,O=org1

name1 == name1Dup => True
name1 == name2 => False

The X509Name class makes it so simple to parse and manipulate the X509 distinguished names. But I encountered a situation where one of the relative distinguished name contained by X509 distinguished name is not recognizable by X509Name even though the underlying OID(object identifier) of the relative distinguished name is supported by the X509Name. The problematic relative distinguished name is "dnQualifier" whose OID is 2.5.4.46. The same OID is supported and parsable by X509Name if the relative distinguished name is "DN" instead "dnQualifier".

Digging little deep into X509Name source reveals that we can make it to parse/support "dnQualifier" relative distinguished name like below

static void Main(string[] args)
{
X509Name.DefaultLookup["dnqualifier"] = X509Name.DnQualifier;

var name1 = new X509Name("CN=person1, OU=dept1, O=org1, dnQualifier=aaaabbbbccccddddeeeeffffggg=");
Console.WriteLine(name1);
}

In the above code basically we are providing another name("dnQualifier") for X509Name.DnQualifier which is by default identified with name "DN".

### Executing plink from non-interactive shell

Today I had a requirement to launch plink.exe(a command-line interface to the PuTTY back ends) from a C# code and this code can run as Windows service or from an ASP.NET application. I accomplished the task with the following code
System.Diagnostics.Process.Start(PLINK_PATH_WITH_PARAMETERS);

The above line launches the plink program which I can see running in TaskManager but for some reason it is not connecting with the remote host. After some google search and experimentation with plink I finally found the reason why it is waiting and not connecting with the remote program.

Actually the plink was launched in non-interactive shell(as it is launched from Windows service/ASP.NET) and it runs under SYSTEM/ASP.NET user account. As we don't usually run plink in these user accounts plink sees the remote host's certificate as new unauthorized certificate and asking us to authorize it. As it is non-interactive shell we don't see the console and the authorization request.

After finding this I found that launching plink from another batch file with the following command line automatically answers "yes" to the plink's certificate authorization request and plink successfully able to execute the commands on remote host.
echo yes | C:\path\to\plink.exe {parameters}

## Wednesday, September 19, 2012

### Exporting RSA public key in PKCS#1 format

Recently I was working on task in which I had to generate a RSA key pair and export the public key to another application in PEM format. I implemented the task with help of BouncyCastle(code below) but the exported public key is not accepted by the other application.
static void ExportPublicKey(AsymmetricKeyParameter publicKey)
{
var stringWriter = new StringWriter();
var pemWriter = new PemWriter(stringWriter);
pemWriter.WriteObject(publicKey);
stringWriter.Flush();
stringWriter.Close();

File.WriteAllText(@"PublicKey.pem", stringWriter.ToString());
}

Little investigation revealed that the other application expects the public key in PKCS#1 format whereas the above code exports in PKCS#8 format(the ASN.1 structure of SubjectPublicKeyInfo). The difference between these formats is the additional field algorithm identifier in PKCS#8 format. We can clearly in there ASN.1 structure below

Also the exported PEM will have different start/end line like below

in PKCS#8
-----BEGIN PUBLIC KEY-----
...
-----END PUBLIC KEY-----

but in PKCS#1
-----BEGIN RSA PUBLIC KEY-----
...
-----END RSA PUBLIC KEY-----

Though the difference is one additional information but exporting public key without that information is not straightforward in BouncyCastle, or atleast at the time of writing. In OpenSSL(libcrypto) this is as simple as the calling the functions PEM_write_RSAPublicKey or PEM_write_bio_RSAPublicKey. In BouncyCastle the following code exports the public key in PKCS#1 format

static void ExportRsaPublicKey(AsymmetricKeyParameter publicKey)
{
var subjectPublicKeyInfo = SubjectPublicKeyInfoFactory.CreateSubjectPublicKeyInfo(publicKey);
var rsaPublicKeyStructure = RsaPublicKeyStructure.GetInstance(subjectPublicKeyInfo.GetPublicKey());
var rsaPublicKeyPemBytes = Base64.Encode(rsaPublicKeyStructure.GetEncoded());

var stringBuilder = new StringBuilder();

stringBuilder.AppendLine("-----BEGIN RSA PUBLIC KEY-----");

for (int i = 0; i < rsaPublicKeyPemBytes.Length; ++i)
{
stringBuilder.Append((char)rsaPublicKeyPemBytes[i]);

// wraps after 64 column
if (((i + 1) % 64) == 0)
stringBuilder.AppendLine();
}

stringBuilder.AppendLine();
stringBuilder.AppendLine("-----END RSA PUBLIC KEY-----");

File.WriteAllText(@"RsaPublicKey.pem", stringBuilder.ToString());
}

and the code that generates RSA key pair is
static void Main(string[] args)
{
var rsaKeyPairGenerator = new RsaKeyPairGenerator();
rsaKeyPairGenerator.Init(new KeyGenerationParameters(new SecureRandom(), 2048));
var keyPair = rsaKeyPairGenerator.GenerateKeyPair();

ExportPublicKey(keyPair.Public);
ExportRsaPublicKey(keyPair.Public);
}


## Sunday, September 16, 2012

### Constructing .NET RSACryptoServiceProvider from DER bytes

In my current task I have to construct private key object from its DER bytes. In OpenSSL I can easily do this with d2i_RSAPrivateKey function but in .NET this doesn't seems to be an easy task. I hoped in BouncyCastle this can be done in single method call but it requires the following code to achieve the functionality

byte[] privateKeyDer = File.ReadAllBytes("PrivateKey.der");
var privateKeyStructure = new RsaPrivateKeyStructure((Asn1Sequence)derSequence);

var privateCrtKeyParameters =
new RsaPrivateCrtKeyParameters(privateKeyStructure.Modulus,
privateKeyStructure.PublicExponent,
privateKeyStructure.PrivateExponent,
privateKeyStructure.Prime1,
privateKeyStructure.Prime2,
privateKeyStructure.Exponent1,
privateKeyStructure.Exponent2,
privateKeyStructure.Coefficient);

var privateKey = DotNetUtilities.ToRSA(privateCrtKeyParameters); // returns RSA object

Please note the classes Asn1InputStream, RsaPrivateKeyStructure, RsaPrivateCrtKeyParameters and DotNetUtilities are provided by BouncyCastle.

## Saturday, April 07, 2012

In this post I am going explain how Notepad++ can be scripted with Python. I came to know about this as I wanted to analyse huge amount of console logs.

First we need to install Notepad++ Python plugin to able to control Notepad++ from python code. The plugin can be installed through Notepad++'s Plugin Manager by installing "Python Script" plugin or we can download the plugin from here http://sourceforge.net/projects/npppythonscript/ and extract the files into Notepad++'s plugin directory. As of this writing the Python Script plugin version is 0.9.2.

# Invoking Python Scripts

The Notepad++ Python scripts needs to placed in particular directory so that it will recognized by the Python plugin and can be invoked from Notepad++. Usually the directory is %APPDATA%\Notepad++\plugins\config\PythonScript. The scripts can be invoked through menu Plugins->Python Script->Scripts. We can also create toolbar button for these scripts to quickly invoke them.

# Counting Words Programmatically

To demonstrate the plugin lets write python script to count the number of characters, words and lines in the current editor window of Notepad++.
from Npp import *
import re

numChars = 0
numWords = 0
numLines = 0

editorContent = editor.getText()
for line in editorContent.splitlines():
numLines += 1
for word in re.findall("[a-zA-Z0-9]+", line):
numWords += 1
numChars += len(word)

notepad.messageBox("Number of characters: %d \nNumber of words: %d \nNumber of lines: %d" % (numChars, numWords, numLines))

at line 8 we get the active editor window's text content and everything else is typical Python program except at line 15 we tell the number of chars, words and lines through Notepad++ message box.

# Bookmarking Programmatically

Lets see an another Python script which utilizes the bookmark feature of Notepad++
from Npp import *

linesBookmarked = []

def onMatch(lineNumber, match):
if lineNumber not in linesBookmarked:
lineStartPos = editor.positionFromLine(lineNumber)
editor.gotoPos(lineStartPos)
linesBookmarked.append(lineNumber)

editor.pysearch("Pos", onMatch)

the above script bookmarks all the lines that contains the word "Pos". The Editor class provides a method "pysearch" which can search for given regular expression and will call the given function for each match. Like the "pysearch" method the Editor and Notepad class object provides various helper methods to automate Notepad++ functionalities from Python script.

## Sunday, March 04, 2012

### Handling pre-shutdown notification in C# Windows service

In this post I am going to explain how pre-shutdown notification can be enabled and handled in C# Windows service.

Starting from Windows Vista on wards Windows provides an mechanism to notify applications/services about the system shutdown/reboot. Though shutdown notification was already supported through ServiceBase class's CanShutdown property and OnShutdown method their functionality is very limited.

The limitation of OnShutdown method is, limited time given for the cleanup work and there is no guarantee that other essential services(e.g. DB, EventLog) are running. The last point is the fundamental reason why Microsoft brought this pre-shutdown notification.

If we are developing a Win32 application it is really easy to enable pre-shutdown notification. All we need to do is include "SERVICE_ACCEPT_PRESHUTDOWN" flag in dwControlsAccepted field of service's SERVICE_STATUS structure.

But in .NET's ServiceBase class there is no direct way to enable pre-shutdown notification. We can achieve the goal by accessing the ServiceBase class's internals through .NET reflection.
const int SERVICE_ACCEPT_PRESHUTDOWN = 0x100;
const int SERVICE_CONTROL_PRESHUTDOWN = 0xf;

MyService
{
...

FieldInfo acceptedCommandsFieldInfo =
typeof(ServiceBase).GetField("acceptedCommands", BindingFlags.Instance | BindingFlags.NonPublic);
if (acceptedCommandsFieldInfo == null)

int value = (int)acceptedCommandsFieldInfo.GetValue(this);
acceptedCommandsFieldInfo.SetValue(this, value | SERVICE_ACCEPT_PRESHUTDOWN);
}

In the constructor of the service "MyService" we add the "SERVICE_ACCEPT_PRESHUTDOWN" flag to "acceptedCommands" private field of ServiceBase using reflection. This enables the pre-shutdown notification.

The pre-shutdown notification is delivered through the ServiceBase class's "OnCustomCommand" method. This method is called whenever a command is sent to the service which is not supported by ServiceBase. The method provides a parameter "command" which holds the command code. So to handle the pre-shutdown notification we need to check whether the "command" matches with pre-shutdown notification code.
protected override void OnCustomCommand(int command)
{
if (command == SERVICE_CONTROL_PRESHUTDOWN)
{
// do the clean-up here
}
else
base.OnCustomCommand(command);
}

So that is all we need to enable and handle pre-shutdown notification.

## Sunday, October 10, 2010

### XML Schema Validation using Xerces-C++

Recently I've been writing some serious codes in Xerces-C++, Xml-Security-C++ and OpenSSL. I planned to share my experience in a series of posts. In this post I am going to show how can we validate a XML against its schema using Xerces-C++. To demonstrate the schema validation I am going to use the example schema and XML from W3 Schools. Our schema file is shiporder.xsd and the XML file is shiporder.xml. Lets jump into the code
#include <stdio.h>

#include <xercesc/util/XMLString.hpp>
#include <xercesc/parsers/XercesDOMParser.hpp>
#include <xercesc/framework/LocalFileInputSource.hpp>
#include <xercesc/sax/ErrorHandler.hpp>
#include <xercesc/sax/SAXParseException.hpp>
#include <xercesc/validators/common/Grammar.hpp>

XERCES_CPP_NAMESPACE_USE

class WStr
{
private:
XMLCh*  wStr;

public:
WStr(const char* str)
{
wStr = XMLString::transcode(str);
}

~WStr()
{
XMLString::release(&wStr);
}

operator const XMLCh*() const
{
return wStr;
}
};

class ParserErrorHandler : public ErrorHandler
{
private:
void reportParseException(const SAXParseException& ex)
{
char* msg = XMLString::transcode(ex.getMessage());
fprintf(stderr, "at line %llu column %llu, %s\n",
ex.getColumnNumber(), ex.getLineNumber(), msg);
XMLString::release(&msg);
}

public:
void warning(const SAXParseException& ex)
{
reportParseException(ex);
}

void error(const SAXParseException& ex)
{
reportParseException(ex);
}

void fatalError(const SAXParseException& ex)
{
reportParseException(ex);
}

void resetErrors()
{
}
};

void ValidateSchema(const char* schemaFilePath, const char* xmlFilePath)
{
XercesDOMParser domParser;
{
return;
}

ParserErrorHandler parserErrorHandler;

domParser.setErrorHandler(&parserErrorHandler);
domParser.setValidationScheme(XercesDOMParser::Val_Auto);
domParser.setDoNamespaces(true);
domParser.setDoSchema(true);
domParser.setValidationConstraintFatal(true);

domParser.parse(xmlFilePath);
if (domParser.getErrorCount() == 0)
printf("XML file validated against the schema successfully\n");
else
printf("XML file doesn't conform to the schema\n");
}

int main(int argc, const char *argv[])
{
if (argc != 3)
{
printf("SchemaValidator <schema file> <xml file>\n");
return 0;
}

XMLPlatformUtils::Initialize();

ValidateSchema(argv[1], argv[2]);

XMLPlatformUtils::Terminate();

return 0;
}

The classes WStr and ParserErrorHandler are helper classes. The class WStr helps us to convert UTF-8 string to UTC-16 string with auto pointer like usage. The class ParserErrorHandler helps to show the parser errors to the console. The function ValidateSchema does the actual validation. The important lines are
domParser.setValidationScheme(XercesDOMParser::Val_Auto);
domParser.setDoNamespaces(true);
domParser.setDoSchema(true);
domParser.setValidationConstraintFatal(true);

which enables namespace processing and schema validation. It also tells the parser to consider any validation error as fatal. The command line to compile the SchemaValidator code is
$g++ SchemaValidator.cpp -o SchemaValidator -lxerces-c  Lets try our SchemaValidator with valid and invalid xml. First lets try with valid xml file $ ./SchemaValidator shiporder.xsd shiporder.xml
XML file validated against the schema successfully

and with a invalid shiporder in which I've added a element "oldprice" to the second "item"
\$ ./SchemaValidator shiporder.xsd shiporder-invalid.xml
at line 15 column 23, no declaration found for element 'oldprice'
XML file doesn't conform to the schema

You can see the schema validation fails saying "oldprice" is unknown element according to given schema(xsd). I hope I showed you how to validate XML schema using Xerces-C++. We'll see what else we can do with Xerces-C++ in the future posts.