Tuesday 3 November 2009

Building a data access layer

I recently agreed to help a student out with a software development assignment out for university. At first glance, it looks to be a pretty heavyweight peice of work for an undergrad to be given, even more so when the undergrad doesnt have a lot of experience in software development.

Essentially, the assignment is to create an Ebay-esque Windows Form app. The wording of the assignment makes it look harder than it actually is, when you get around the features that are being requested the actual technical requirement is really very simple. The assignment needs to be split into several peices, something that a lot of students probably wont be familiar with as it simply doesnt get taught at uni. The first section is the persistent data storage, I don't yet know what options are available at the moment, but the assignment spec asks for data to be retrieved from either a database or some XML. Now, I can understand why these features would be requested, it allows the application to be able to pick up its data from the local infrastructure or from a web service.

So, they key technical part here? Well, it's a data access layer. I have put together a few of these now, normally as part of an architectural migration to ease the switch over, all this assignment really needs is an appropriate DAL in order to get its data into the Windows app. It struck me that there really isnt a lot of info on how to do this rather simple thing, I think with the maturation of ORM like nHibernate and Linq-to-SQL a lot of people are over looking this for smaller projects. Anyway, I plan to put together a short howto on building your own DAL, because once you understand how the things work in the firs place, it makes implementing your own or using one of the ORM's already out there a lot more intuitive.

Sunday 11 October 2009

Old School Computing, in its most truest sense.

Recently, I posted about the excellent machines that a rejuvinated Acorn Computers are now selling on the Internet. It got me thinking, anyone who went through an English school suring the late 80's and early 90's should have at least some distant memory of these rather fine machines.

For those that don't know, Acorn was one of the UK's only computer builders, a lot of people wont even remember them these days, but the ones who do will no doubt have fond memories of the BBC Master and probably playing Chuckie Egg on it.

Acorn made computers from 1978 to 1998, when they finally succumbed to the dominance of the PC architeture. There is loads about them on Wikipedia: http://en.wikipedia.org/wiki/Acorn_Computers, there are loads of these machines about now, mainly because of the sheer amount that were installed in schools across the country for so long. The first ever computer I owned was an Acorn Electron back in the early 80's. My parents thought that it might improve my hand/eye coordination to play games on it!! I think what many people wont realise is the lasting and prominent impact Acorn Computers have had on our everyday lives. Not only did they build their own computers, but they made their own OS, architecture and hardware. Acorn are resposible for developing the ARM processor, a peice of hardware that pretty much everyone carries around in their pocket, powering their mobile phone. It would be a fair bet to make that there is also one in your set top box at home, powering Sky, Virgin Media or your freeview/sat tuner. It doesnt start and end there either, ARM processors powered another venerable computing device towards the end of the 90's, the PSION Series 5. This is a very notable device as it was the precursor to the S60 mobile phone platform, the OS that powered it was called EPOC and it pretty much turned into Symbian - now things should start to be recogniseable!

With that said, I have now in front of me an original Acorn A7000+ desktop computer from a school down in England. It should run, I have powered it on and got the desktop up and displayed on my monitor. I plan to rennovate this UK computing relic to get it back up and running properly. I remember schools were using these machines right up until 2001 in some parts of the UK, this one I have here is a clear indication as it comes with an all important ethernet adapter.

One of the notable things about these machines is that they don't really need hard drives, the entire OS and all its supporting applications are stored on ROM. When you want to upgrade the OS, you simply replace the ROMs with your new set. This isn't something we see at all these days in a desktop computer, unless you are running a setup with embedded Linux.

I recall from the last time I saw an Acorn machine in operation, that it had a fully functional web browser. So, my plan right now is to get this bad boy up and running and connected to the Internet so that I can try and do some simple things like email and facebook. The example I have here, an A7000+, has a hard drive installed, as well as a CD writer, so this means I will be able to install software to it if I need to. From what I can see from the boot up screen, this machine is running RISC OS 3.7. From what I can remember, this was the last main release of the OS prior to Acorns demise, however since then there have been three seperate releases of newer OS's. In a rather different move to the normal distributions of Windows and Linux etc that we see now, a new company was formed in 1999 to release RISC OS 4. Not long after this, a seperate company release a RISC OS 5 for a seperate line of devices, but as far as I can tell this version of the OS is only for machines made by a company called Castle. This confuses me a bit, as far as I can tell this version of the OS has never really been available seperately.

But, a further twist is this, a new OS has been recently released, RISC OS 6... I have no idea how much of a departure this will be from the RISC OS I know and love. One thing that is very notable about RISC OS is that it pretty much introduced the concept of the task bar (as far as I know). This is very familiar to Apple users and now, to Windows 7 users, its a feature that Acorn introduced in 1988 though... As soon as I get this one up and running, I will start to get some screen shots up and running.

Friday 7 August 2009

Sorting Lists with Comparison in .Net

The other day, I made a post about invoking methods using reflection. One of the things I wanted to do in my code was get a list of the methods in a class and then run them. However, what I didnt mention was that I wanted the methods added to my list in an order. Sorting a list is easy if you are working with a list of strings or numbers. But the list I generated was a list of type MethodInfo - something that .Sort wouldnt work with.

So, to overcome this, I did the following:
sectionNames.Sort(delegate(MethodInfo m1, MethodInfo m2) { return m1.Name.CompareTo(m2.Name); });

And out pops a sorted list of type MethodInfo. Short, but sweet and also provides a nice little example of how lists of different types can be sorted.

Friday 31 July 2009

Sick of my phone

I am totally sick of my phone just now. For the past five years I have been using Nokia phones, mainly S60 smartphones. However, over the last few years I have been getting more and more frustrated with the S60 OS. It sees to me that once Nokia gained control of Symbian (at least I think it did) things started getting bad.

At the present time, I use an N96 - when it wants to work. Recently, one of my friends at work got hold of the HTC Hero, a smartphone that uses Google's Android 1.5. As far as I am concerened, it out cools the iPhone. However, I dont know if I want to get another smartphone. Right now I am thinking about getting a normal phone and some form of Internet Tablet/PDA - ironically I am looking at getting a Nokia N810. My reasons for this? Well, out of the box it runs a Linux, but you can also install Android on it as well :).

PDF Generation with PDFJet

As mentioned previously, I have been working a lot on report generation. Where I work just now, we produce a lot of reports. A lot of them are produced manually and are very large. An ideal candidate for automation.

I need to produce them in a number of different formats, pdf is the first one I tackled. All of the reports I need to produce use tables, so I needed to find a .Net pdf library that had this functionality. I took a look at what was on offer, open source, closed source and commercial. I went for a lib called PDFJet. This is a fairly mature Java pdf library that has been ported over to .Net for C# http://www.pdfjet.com/. Its fairly cheap and whilst it lacks a lot of in depth examples, there are quite a few available on their website. The support is really good too - if you have a question, the guys are able to provide feedback very quickly.

It is very well featured, but I am basically using it to generate tables from datatables. I have been using it in earnest for about a week now, so I will try and post some examples of how it can be used here. There are two ways of creating a table with PDFJet, you can use a delimited text file holding the data you need or you can build a table programatically. Doing it in the code is quite tricky, to do this you need to build a list of lists of type cell - not as easy as it sounds... However, I am very impressed with this library and the things it can do.

Automatically invoke methods from a given class

Had an interesting problem at work this week. I have been working on a way to automatically generate a number of (lengthy) reports in a number of different formats. Each report has a number of sections that in turn hold a number of subsections. Each subsection holds either a chart or a table. Each report needs to be presented as either a pdf, a doc, an xls or a csv file. Some of the sections in each report are used in other reports.

I came round to designing each section as a class, in each class is a method that generates either a datatable to build a table from or a chart generated as an image. I wanted the generation of these datatables and images to be as generic as possible, the methods called provide the objects I needed but the way I call them needed to be as generic as I could get it. So, instead of putting together a method that calls all of these other methods I came up with the idea of building a list of all the methods I need to call in order to generate a report. To do this, I used reflection.

Firstly, I needed to get a list of all the method names I wanted to call.
List<MethodInfo> sectionNames = new List<MethodInfo>();
Sections s = new Sections(conn);
Type t = s.GetType();
MethodInfo[] mi = t.GetMethods();
foreach (MethodInfo method in mi)
{
sectionNames.Add(method);
}


In this example, Sections is the name of the class holding the methods I need to generate a report. It populates its datatables from a database, so the conn variable holds the connection string needed. There are a lot of sections in the report I was working on, so I split each section up as partial classes. Using GetMethods () on my class returns all of the available methods within that class. With that information, I simply create a list of type MethodInfo and add each element of the array I have created to the list. I dont really need to do this, but at some point I am going to want to sort the list, which is a little easier than sorting the array (from my perspective). I now have a list of all the method names available in my class. The next thing I do is get rid of some of the generic methods, like GetType(). I dont know if there is a cleaner way of doing this, so all I am doing right now is deleting a range from the list:


sectionNames.RemoveRange(8, 4);

Now I have almost everything I need. Next up, I need to invoke the method and capture its return. The first type of report I tackled was a pdf. The library I am using to create these pdf's uses delimited text files to create tables (it can be done programatically, but I havent mastered this yet). So, I need to create a dictionary holding the name of the text file I want to create and the name of the method to create it from. Now, I also mentioned that my report will also hold charts generated as images - so the dictionary I create needs to hold a string and an object, not a string and a datatable or any other type. It may not be the cleanest way to do it, but I can cast the object as a datatable later on, anyway...
Dictionary<string, Object> content = new Dictionary<string, Object>();


Thats my dictionary ready to go, I just need to populate it with filenames and object. This isnt as hard as it seems, because my list contains type MethodInfo I can use the Invoke method to call all of the methods in the list. Thats a bit of a mouthful really, so it is better to demonstrate:


foreach (MethodInfo method in sectionNames)
{
object obj = method.Invoke(s, null);
content.Add(method.Name + ".txt", obj);
}


I loop through each MethodInfo type in my list. I create an object called obj, this becomes the return from each of the methods I call using Invoke. Once called and invoked, I add the result to my dictionary, on success the dictionary contains a key value pair representing a string filename and an object. You can see in the example above that the filename is always a .txt - this is bad when it comes to images (I am not creating any just yet). But, this can be worked around by getting the type of the object etc. However, right now this is all I need to create my text files to use with my pdf generation. I simply loop through the dictionary creating a textfile from the contents of the object (which is a datatable dont forget), the string is the filename the text file is created with.

Its not very efficient just now, I need to manage the methods in the dictionary - I need to remove the generic ones prior to adding them and I need to sort them in order, both things I either do manually or not at all right now. But this is a nice little example of how methods can be gathered from a class and then dynamically invoked. The complete source for this example is as follows:


/// <summary>
/// Creates a Dictionary of strings and object representing the content we want to build
/// </summary>
/// <returns>a Dictionary of strings and objects holding the filename and the datatable</returns>
private Dictionary<string, Object> Contents()
{
List<MethodInfo> sectionNames = new List<MethodInfo>();
Sections s = new Sections(conn);
Type t = s.GetType();
MethodInfo[] mi = t.GetMethods();
foreach (MethodInfo method in mi)
{
sectionNames.Add(method);
}
sectionNames.RemoveRange(8, 4);
sectionNames.Sort();
Dictionary<string, Object> content = new Dictionary<string, Object>();
foreach (MethodInfo method in sectionNames)
{
object obj = method.Invoke(s, null);
Console.WriteLine(obj);
content.Add(method.Name + ".txt", obj);
}
return content;

I will also post the file creation method that works with this after the weekend as well as some bits and peices to do with the pdf library I am using. But hopefully this should be enough to get someone else on the right track if they are also considering such a soloution.

Sunday 21 June 2009

Old School Acorn Computing

Just been doing a bit of a browse for info on my vintage Acorn A7000+ desktop when I discovered that Acorn Computers are back in business. I was impressed, however a little dismayed to see that they dont use the classic RISC OS operating system any more, but instead plump for Vista, or other versions of Windows. Their site is here: http://www.acorncomputers.co.uk/index.htm, and to be honest the devices they are selling do seem to be quite attractive. They come complete with the old school Acorn logo all over them, which just adds to the appeal.

I think anyone who still has fond memories of using Acorn and BBC computers when they were at school will enjoy taking a look at the new series of products. Of course, anyone who is still hankering for a bit of RISC OS action can take a look at Ebay, you can pick up an A7000+ very cheaply these days, some even with a network adapter. Schools are literally chucking these things out with the rubbish nowadays, unfortunately they can't even give them way to charities etc. However, if you really want the latest iteration of RISC OS, currently at version 5, check out Castle's list of products here: http://www.iyonix.com/. Personally, I dont see the need to spend almost £800 on a computer like this, but if you really are an enthusiast you will snap one right up.

Sunday 7 June 2009

PGP Decryption with C#

Yesterday I posted on how I was able to encrypt some data and compress it using C#. Been thinking about how useful it would be to be able to decrypt it as well, there is no pressing need for me to do this in the situation I was working in really - once my data is encrypted and archived it would be sent off to someone else, so I dont really need to worry about decrypting it.



Thing is, I just wondered how it would work. I already know that Bouncy Castle works heavily with streams, so I guess the logic would be:


  • Open a file as a stream

  • Get the encryption keys needed to decrypt the file

  • Decrypt the file to a stream

  • Write the file(s)

I already have the code that will manage the keys for me and I already have an understanding on how Bouncy Castle's libs work with streams. I had a quick google to see if there was anything going in the way of examples on how to do this, but only managed to find one (someone had pasted the example in the Bouncy Castle source to a forum). Thing is, none of the offered examples seemed to fit my code, which was a shame. However, these things can be overcome once you understand a little more about how Bouncy Castle implements OpenPGP. The code I created to decrypt my archive isnt going to go into production (probably), its just something I wanted to try out, so it's all one big method. Note, these libs work on streams - to decrypt my archive and get to the literal data, I had to manipulate the streams to get there, I think this could put a lot of people off using this. First off, we need to get the file we wish to decrypt into a stream and determine the decoder from it:


input = PgpUtilities.GetDecoderStream(input);

The next thing we want to do is create a PGPObjectFactory. From here we can get a PGPEncryptedDataList and start creating objects from our input stream:
PgpObjectFactory pgpObjF = new PgpObjectFactory(input);
PgpEncryptedDataList enc;
PgpObject obj = pgpObjF.NextPgpObject();
if (obj is PgpEncryptedDataList)
{
enc = (PgpEncryptedDataList)obj;
}
else
{
enc = (PgpEncryptedDataList)pgpObjF.NextPgpObject();
}

Now we are able to get the first encrypted object from our stream so lets decrypt the stream using the private key:
PgpPrivateKey privKey = pgpKeys.PGPPrivateKey;
PgpPublicKeyEncryptedData pbe = null;
foreach (PgpPublicKeyEncryptedData pked in enc.GetEncryptedDataObjects())
{
if (privKey != null)
{
pbe = pked;
break;
}
}
Stream clear = pbe.GetDataStream(privKey);
PgpObjectFactory plainFact = new PgpObjectFactory(clear);
PgpObject message = plainFact.NextPgpObject();

With this done, we are now able to start inspecting the decrypted objects we are creating. We know the file we made earlier was an encrypted zip file, so at this point it is fair to assume message is compressed, which it is:


if (message is PgpCompressedData)
{
PgpCompressedData cData = (PgpCompressedData)message;
Stream compDataIn = cData.GetDataStream();

I knew that message was of the type PGPCompressedData from inspecting the object, what we need to do now is find out when message becomes literal data. To do this, we need to go a bit further, I need to create a new PGPObjectFactory:
PgpObjectFactory o = new PgpObjectFactory(compDataIn);
message = o.NextPgpObject();
if (message is PgpOnePassSignatureList)
{
message = o.NextPgpObject();
PgpLiteralData Ld = null;
Ld = (PgpLiteralData)message;
Stream output = File.Create(outputpath + "\\" + Ld.FileName);
Stream unc = Ld.GetInputStream();
Streams.PipeAll(unc, output);
}

I had thought that the literal data would come next, but it didnt - dont forget when encrypting the data previously I signed it - so I got the signature first. So, I just needed to move on to the next object which was the literal data. As you can see above, all I need to do now is write the file out to the disk and I am done. I did think when I had completed this, "What about decompressing directory structures?". Well, in my humble opinion, there is no need to do this. From what I have seen with encryption over the last few months, files get encrypted, not directories. I wrote a lib the other day that compresses into a zip, I will post it here over the next week, it also decompresses (including the directory structure). So the way I see it, if you wanted to encrypt a set of files and directories, you would create the zip first and then create an encrypted archive from that file. I know it would be two seperate operations, but this sort of suits my situation at the moment: I dont need to decrypt archives programatically, but I can if I need to (now, anyway). So with the stuff I have now, I can create a zip for internal deployment and then create a seperate, encrypted zip from that file for external deployment.

The complete code for decryption is as follows:


using Org.BouncyCastle.Bcpg.OpenPgp;
using Org.BouncyCastle.Crypto;
using Org.BouncyCastle.Security;
using Org.BouncyCastle.Utilities.IO;
using Org.BouncyCastle.Bcpg;
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Text;
using Tools.PGP.Crypto;

namespace Tools.PGP.Crypto
{
public class PGPDecrypt
{
public string _encryptedFilePath;
public string _privKeyPath;
public char[] _password;
public string _outputPath;
public PGPKeys pgpKeys;
public PGPDecrypt(string encryptedFilePath, string privKeyPath, string password, string outputPath, string pubKeyPath, long keyID)
{
_encryptedFilePath = encryptedFilePath;
_outputPath = outputPath;
_password = password.ToCharArray();
_privKeyPath = privKeyPath;
pgpKeys = new PGPKeys(pubKeyPath, privKeyPath, password, keyID);
}
public void decrypt(Stream input, string outputpath)
{
input = PgpUtilities.GetDecoderStream(input);
try
{
PgpObjectFactory pgpObjF = new PgpObjectFactory(input);
PgpEncryptedDataList enc;
PgpObject obj = pgpObjF.NextPgpObject();
if (obj is PgpEncryptedDataList)
{
enc = (PgpEncryptedDataList)obj;
}
else
{
enc = (PgpEncryptedDataList)pgpObjF.NextPgpObject();
}
PgpPrivateKey privKey = pgpKeys.PGPPrivateKey;
PgpPublicKeyEncryptedData pbe = null;
foreach (PgpPublicKeyEncryptedData pked in enc.GetEncryptedDataObjects())
{
if (privKey != null)
{
pbe = pked;
break;
}
}
Stream clear = pbe.GetDataStream(privKey);
PgpObjectFactory plainFact = new PgpObjectFactory(clear);
PgpObject message = plainFact.NextPgpObject();
if (message is PgpCompressedData)
{
PgpCompressedData cData = (PgpCompressedData)message;
Stream compDataIn = cData.GetDataStream();
PgpObjectFactory o = new PgpObjectFactory(compDataIn);
message = o.NextPgpObject();
if (message is PgpOnePassSignatureList)
{
message = o.NextPgpObject();
PgpLiteralData Ld = null;
Ld = (PgpLiteralData)message;
Stream output = File.Create(outputpath + "\\" + Ld.FileName);
Stream unc = Ld.GetInputStream();
Streams.PipeAll(unc, output);
}
else
{
PgpLiteralData Ld = null;
Ld = (PgpLiteralData)message;
Stream output = File.Create(outputpath + "\\" + Ld.FileName);
Stream unc = Ld.GetInputStream();
Streams.PipeAll(unc, output);
}
}
}
catch (Exception e)
{
throw new Exception(e.Message);
}
}
}
}


To decrypt a file, I can just call the method like so:
PGPDecrypt test = new PGPDecrypt(@"C:\test\somefile.zip",
@"C:\GnuPG\secring.gpg",
"password",
@"C:\test\test",
@"C:\GnuPG\pubring.gpg",
3699527550217851901);
FileStream fs = File.Open(@"C:\test\somefile.zip", FileMode.Open);
test.decrypt(fs,@"C:\test\test");

Saturday 6 June 2009

PGP Encryption with C#

Recently at work, I had to come up with a deployment framework for a legacy app we use. The initial requirement was to simply copy the app from a network share onto users desktops. There was also a requirement to provide a level of security around the deployment itself, so to me, an anonymous network share didnt seem to be the correct way of releasing this app... A much more simpler way to provide at least a small level of security would be make the deployment framework download the app as an archive from a webserver on the domain. This, at least, means that only users with authorisation to access the domain would be able to download the app, and they would only be able to download it whilst they were logged in.

Then I was told that there will be some instances that field workers will need to download parts of the app whilst they are out and about doing whatever it is they do. I had already planned to compress the app as an archive, but now I needed a way to secure the archive itself. One of my colleagues suggested that I put a password on the archive, its not a daft suggestion to make, it would imply that only people with the password would be able to extract the data from the archive itself. But, I pointed out that this doesnt prove that the archive was one created by us - all it does it password protect an archive. Plus, we would need to communicate the password to those that needed it in a secure manner, which in turn would mean we couldnt use the same password all the time etc. I asked if we already use some form of secure signature and got told we use PGP - result. I explained to my colleagues that yes, password protecting the archive would mean that only password holders could extract from the archive, but this wouldnt mean they are authorised to hold the password in the first place. I also explained that the password could be cracked, even if it was sufficiently complex, a persistant attacker could spend a lot of time trying to crack it.

With this in mind, I went on to say that as we already encrypt data with PGP, we could do the same thing here with the archives we are planning to create. This way, people using our app (or archives in general) would be able to see with great certainty that they did originate from us and that it hasnt been substituted by a third party.

This is part of a .Net project, so any encryption would ideally take place using the .Net framework. However, right now there is no PGP support in .Net. There is support for cryptograhy in general, for instance RSA is employed as well as DPAPI, but we already use PGP and already have public keys. So it would make more sense to me for us to use these for encryption. To do this, I am going to be using the C# cryptography libraries available from The Legion of the Bouncy Castle: http://www.bouncycastle.org/csharp/ . There are not a lot of good examples of how to use this out on the Internet, the best I could find to outline the concepts involved was here. Unfortunately, the formatting of the page makes it a little hard to understand what is happening - which is the important part

Lets get down to encrypting a file and compressing it to an archive. I am going to assume that if you are searching for PGP encryption methods in C# that you have already generated a set of PGP keys. If you havent, then I can suggest that you nip over to http://www.gnupg.org/ and download a copy of GnuPG to generate some for you (it's what I used to test through my app). The first step we need to take is to understand what we need to get the key values. We need to know the path to the public and private keys, the password for the public key to encrypt with and we need to know the key ID we want to target. The key ID is a 64bit signed integer, using GnuPG you can get this value from the command line using the command:

--list-keys --with-colons <userid>
Simply replace the user ID with the user ID attached to the key (this will probably be the email address used when generating the key itself). You will get something that looks like this back:

pub:u:1024:1:33575CB4BB8193FD:2009-06-06:::u:Your Name(RSA Sign only test) <your@email.address>::scESC:
sub:u:1024:1:31B23C6C6681F46B:2009-06-06::::::e:

The key ID is the bit in bold (or should be bold, but not quite so well - it's the fith column along). This is the key ID in hexidecimal, in the code I am about to show you it needs to be in decimal format, so you can either convert it in c# with the following:

int decAgain = int.Parse(hexValue, System.Globalization.NumberStyles.HexNumber);
Or you can use a tool to convert it for you like this one: http://www.parkenet.com/apl/HexDecConverter.html, or if you happen to know what it is anyway (like I did), you can just type it in etc.

I used RSA encryption when generating my key. I used this as it would appear to be the best method for creating keys right now, there are not many examples of how to use PGP encryption on the Internet at the moment, and the peices of code that are here and there seem to use older standards.

Right, now we know our key ID etc, we can start writing the code. I created a class called PGPKeys to do all my key related work for me. First up, we need to declare all our variables etc:
/// <summary>
/// long representing the key ID we want to target.
/// </summary>
private long _keyID;
/// <summary>
/// string to represent the path to the private key file.
/// </summary>
private string _privKeyPath;
/// <summary>
/// string to represent the path to the public key file.
/// </summary>
private string _pubKeyPath;
/// <summary>
/// string to represent the passphrase used with PGP.
/// </summary>
private string _password;
public PgpPublicKey PGPPublicKey { get; private set; }
public PgpPrivateKey PGPPrivateKey { get; private set; }
public PgpSecretKey PGPSecretKey { get; private set; }

Nice and simple. Next up, I want to be able to validate the values I am planning to use:

/// <summary>
/// Constructor to manage variables for us.
/// </summary>
/// <param name="pubKeyPath">string representing the public key path</param>
/// <param name="privKeyPath">string representing the private key path</param>
/// <param name="password">string representing the passphrase</param>
/// <param name="keyID">long representing the key ID we want to use</param>
public PGPKeys(string pubKeyPath, string privKeyPath, string password, long keyID)
{
if (!File.Exists(pubKeyPath))
{
throw new ArgumentNullException("Could not find the public key at " + pubKeyPath);
}
else
{
_pubKeyPath = pubKeyPath;
}
if (!File.Exists(privKeyPath))
{
throw new ArgumentNullException("Could not find the private key at " + privKeyPath);
}
else
{
_privKeyPath = privKeyPath;
}
if (String.IsNullOrEmpty(password))
{
throw new ArgumentNullException("The password must not be null");
}
else
{
_password = password;
}
if (keyID == 0)
{
throw new ArgumentNullException("The key ID must not be null");
}
else
{
_keyID = keyID;
}
PGPPublicKey = getPublicKey(_pubKeyPath);
PGPSecretKey = getSecretKey(_privKeyPath);
PGPPrivateKey = getPrivateKey(_password);
}
You can see here where the public, private and secret keys are going to be held. Simple so far, but now the work begins. I think one of the tough things to get your head around when using the Bouncy Castle libs is that everything is a set of streams that have things done to them. This is a slight over simplification, but the following bits of code will demonstrate this process. Lets look at getting the public key value:
/// <summary>
/// private method to get the public key value.
/// </summary>
/// <param name="_pubKeyPath">string representing the public key path</param>
/// <returns>a PGPPublicKey</returns>
private PgpPublicKey getPublicKey(string _pubKeyPath)
{
PgpPublicKey pubKey;
using (Stream keyin = File.OpenRead(_pubKeyPath))
using (Stream s = PgpUtilities.GetDecoderStream(keyin))
{
PgpPublicKeyRingBundle pubKeyBundle = new PgpPublicKeyRingBundle(s);
pubKey = pubKeyBundle.GetPublicKey(_keyID);
if (pubKey == null)
throw new Exception("The public key value is null!");
}
return pubKey;
}
This is the on ramp with PGP, getting the public key value represented in the code itself. It also shows how Bouncy Castle likes to work with streams, the first stream here opens the file in which the key is stored, the second stream then goes on to get the value from it. With that done, lets get the secret key:
/// <summary>
/// private method to get the secret key value.
/// </summary>
/// <param name="_privKeyPath">string representing the private key path</param>
/// <returns>a PGPSecretKey</returns>
private PgpSecretKey getSecretKey(string _privKeyPath)
{
PgpSecretKey secKey;
using (Stream keyin = File.OpenRead(_privKeyPath))
using (Stream s = PgpUtilities.GetDecoderStream(keyin))
{
PgpSecretKeyRingBundle secKeyBundle = new PgpSecretKeyRingBundle(s);
secKey = secKeyBundle.GetSecretKey(_keyID);
if (secKey == null)
throw new Exception("The secret key value is null!");
}
return secKey;
}

Here we pretty much do the same thing as before, but this time we are looking at the private key file path. Dont get confused here, this method gets the secret key value, to do this we point it at the private key file and supply a key ID. To get the private key itself, we use the following:

/// <summary>
/// private method to get the private key value.
/// </summary>
/// <returns>a PGPPrivateKey</returns>
private PgpPrivateKey getPrivateKey(string _password)
{
PgpPrivateKey privKey = PGPSecretKey.ExtractPrivateKey(_password.ToCharArray());
if (privKey == null)
{
return null;
}
else
{
return privKey;
}
}
This one is a lot smaller, but with it we get our private key. And with that, we are done with our PGPKeys class, lets move on to encrypting, signing and compressing using these keys. Remember what I said about streams? Well, this is how it comes into play, lets encrypt some data:
private Stream encrypt(Stream output)
{
PgpEncryptedDataGenerator pgpEncDataGen = new PgpEncryptedDataGenerator(SymmetricKeyAlgorithmTag.Twofish, new SecureRandom());
pgpEncDataGen.AddMethod(_pgpKeys.PGPPublicKey);
Stream encryptedOutput = pgpEncDataGen.Open(output, new byte[BufferSize]);
return encryptedOutput;
}
Thats it done, this will encrypt a stream only. On its own, this isnt so useful, you can see where I am specifying the encryption algorythm (I decided to use TwoFish, apparently this is a very secure algorythm) and where the public key value is applied. If you just hold in your mind for a moment that this represents a stream of encrypted data at its conclusion, we can look at how it is compressed to an archive:
private Stream compress(Stream output)
{
PgpCompressedDataGenerator pgpCompDataGen = new PgpCompressedDataGenerator(CompressionAlgorithmTag.Zip);
Stream compressedEncryptedOut = pgpCompDataGen.Open(output);
return compressedEncryptedOut;
}

The stream we just emcrypted is now passed to this method which compresses it to a zip - right now though, this is still a stream, nothing more. The next thing we need to do is apply the literal data to this stream:
private Stream literalOutput(Stream compressedOut, FileInfo file)
{
PgpLiteralDataGenerator pgpLiteralDataGen = new PgpLiteralDataGenerator();
Stream literal = pgpLiteralDataGen.Open(compressedOut, PgpLiteralData.Binary, file);
return literal;
}

Now we have a stream that has been encrypted, compressed and had its literal data applied to it. I am going to point out here that if you dont want to create a PGP encrypted file, but instead just want a PGP encrypted stream, then you could just use the encrypt method. Anyway, now we are ready to write and sign our archive:
private static void writeAndSign(Stream ouput, Stream literalout, FileStream inputFile, PgpSignatureGenerator sigGen)
{
int length = 0;
byte[] buf = new byte[BufferSize];
while ((length = inputFile.Read(buf, 0, buf.Length)) > 0)
{
literalout.Write(buf, 0, length);
sigGen.Update(buf, 0, length);
}
sigGen.Generate().Encode(ouput);
}

The last thing we want to do here, is sign the encrypted data itself:
private PgpSignatureGenerator sigGen(Stream compressedOut)
{
const bool Iscritical = false;
const bool IsNested = false;
PublicKeyAlgorithmTag tag = _pgpKeys.PGPSecretKey.PublicKey.Algorithm;
PgpSignatureGenerator pgpSigGen = new PgpSignatureGenerator(tag, HashAlgorithmTag.Sha1);
pgpSigGen.InitSign(PgpSignature.BinaryDocument, _pgpKeys.PGPPrivateKey);
foreach (string userID in _pgpKeys.PGPSecretKey.PublicKey.GetUserIds())
{
PgpSignatureSubpacketGenerator subPackGen = new PgpSignatureSubpacketGenerator();
subPackGen.SetSignerUserId(Iscritical, userID);
pgpSigGen.SetHashedSubpackets(subPackGen.Generate());
break;
}
pgpSigGen.GenerateOnePassVersion(IsNested).Encode(compressedOut);
return pgpSigGen;
}

Now we are able to encrypt, compress, sign and write our chosen file to an archive. All we need to do now is tie it up nicely:

public void EncryptSignAndZip(Stream output, FileInfo unencryptedinput)
{
if (output == null)
{
throw new ArgumentNullException("The output stream cannot be null");
}
if (unencryptedinput == null)
{
throw new ArgumentNullException("You must supply a filename to encrypt");
}
if (!File.Exists(unencryptedinput.FullName))
{
throw new ArgumentNullException(unencryptedinput + " does not exist");
}
using (Stream encryptedout = encrypt(output))
using (Stream compressedOut = compress(encryptedout))
{
PgpSignatureGenerator signature = sigGen(compressedOut);
using (Stream literalOut = literalOutput(compressedOut, unencryptedinput))
using (FileStream inputfile = unencryptedinput.OpenRead())
{
writeAndSign(compressedOut, literalOut, inputfile, signature);
}
}
}

Again, I am validating the arguments as they go in to make sure that they are not null values. You can also see here how Bouncy Castle works with the streams we are creating. However, thats it - these classes will create an encrypted zip file for you with what ever you want in them. To make use of this class, I used this in testing:
string pubKeyPath = @"C:\GnuPG\pubring.gpg";
string privKeyPath = @"C:\GnuPG\secring.gpg";
string password = "password";
long _keyID = 3699527550217851901;
PGPEncrpyt test = new PGPEncrpyt(pubKeyPath, privKeyPath, password, _keyID);
FileInfo myFile = new FileInfo(@"C:\test\app.config");
FileStream mystream = new FileStream(@"C:\test\somefile.zip", FileMode.Create, FileAccess.Write);
test.EncryptSignAndZip(mystream, myFile);

And thats it, I using the code above I was able to create an encrypted zip, which I was able to decrypt and extract using GnuPG. I have written the code to decrypt it yet as at this time, I dont really need it - the files I am encrypting will normally be sent on to other organisations etc. One thing I may make use of in future is the ability to encrypt just a stream - this may become very useful for me when it comes to sending data over a network to other parts of the business. The complete code is as follows, PGPKeys:

using Org.BouncyCastle.Bcpg.OpenPgp;
using Org.BouncyCastle.Crypto;
using Org.BouncyCastle.Security;
using Org.BouncyCastle.Utilities.IO;
using Org.BouncyCastle.Bcpg;
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Text;

namespace Tools.PGP.Crypto
{
/// <summary>
/// Nested class to get information on out PGP Keys
/// </summary>
public class PGPKeys
{
/// <summary>
/// long representing the key ID we want to target.
/// </summary>
private long _keyID;
/// <summary>
/// string to represent the path to the private key file.
/// </summary>
private string _privKeyPath;
/// <summary>
/// string to represent the path to the public key file.
/// </summary>
private string _pubKeyPath;
/// <summary>
/// string to represent the passphrase used with PGP.
/// </summary>
private string _password;
public PgpPublicKey PGPPublicKey { get; private set; }
public PgpPrivateKey PGPPrivateKey { get; private set; }
public PgpSecretKey PGPSecretKey { get; private set; }
/// <summary>
/// Constructor to manage variables for us.
/// </summary>
/// <param name="pubKeyPath">string representing the public key path</param>
/// <param name="privKeyPath">string representing the private key path</param>
/// <param name="password">string representing the passphrase</param>
/// <param name="keyID">long representing the key ID we want to use</param>
public PGPKeys(string pubKeyPath, string privKeyPath, string password, long keyID)
{
if (!File.Exists(pubKeyPath))
{
throw new ArgumentNullException("Could not find the public key at " + pubKeyPath);
}
else
{
_pubKeyPath = pubKeyPath;
}
if (!File.Exists(privKeyPath))
{
throw new ArgumentNullException("Could not find the private key at " + privKeyPath);
}
else
{
_privKeyPath = privKeyPath;
}
if (String.IsNullOrEmpty(password))
{
throw new ArgumentNullException("The password must not be null");
}
else
{
_password = password;
}
if (keyID == 0)
{
throw new ArgumentNullException("The key ID must not be null");
}
else
{
_keyID = keyID;
}
PGPPublicKey = getPublicKey(_pubKeyPath);
PGPSecretKey = getSecretKey(_privKeyPath);
PGPPrivateKey = getPrivateKey(_password);
}
/// <summary>
/// private method to get the public key value.
/// </summary>
/// <param name="_pubKeyPath">string representing the public key path</param>
/// <returns>a PGPPublicKey</returns>
private PgpPublicKey getPublicKey(string _pubKeyPath)
{
PgpPublicKey pubKey;
using (Stream keyin = File.OpenRead(_pubKeyPath))
using (Stream s = PgpUtilities.GetDecoderStream(keyin))
{
PgpPublicKeyRingBundle pubKeyBundle = new PgpPublicKeyRingBundle(s);
pubKey = pubKeyBundle.GetPublicKey(_keyID);
if (pubKey == null)
throw new Exception("The public key value is null!");
}
return pubKey;
}
/// <summary>
/// private method to get the secret key value.
/// </summary>
/// <param name="_privKeyPath">string representing the private key path</param>
/// <returns>a PGPSecretKey</returns>
private PgpSecretKey getSecretKey(string _privKeyPath)
{
PgpSecretKey secKey;
using (Stream keyin = File.OpenRead(_privKeyPath))
using (Stream s = PgpUtilities.GetDecoderStream(keyin))
{
PgpSecretKeyRingBundle secKeyBundle = new PgpSecretKeyRingBundle(s);
secKey = secKeyBundle.GetSecretKey(_keyID);
if (secKey == null)
throw new Exception("The secret key value is null!");
}
return secKey;
}
/// <summary>
/// private method to get the private key value.
/// </summary>
/// <returns>a PGPPrivateKey</returns>
private PgpPrivateKey getPrivateKey(string _password)
{
PgpPrivateKey privKey = PGPSecretKey.ExtractPrivateKey(_password.ToCharArray());
if (privKey == null)
{
return null;
}
else
{
return privKey;
}
}
}
}


And PGPEncrypt:
using Org.BouncyCastle.Bcpg.OpenPgp;
using Org.BouncyCastle.Crypto;
using Org.BouncyCastle.Security;
using Org.BouncyCastle.Utilities.IO;
using Org.BouncyCastle.Bcpg;
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Text;

namespace Tools.PGP.Crypto
{
/// <summary>
/// Public class to encrypt and compress files to a zip archive using PGP
/// </summary>
public class PGPEncrpyt
{
/// <summary>
/// instansiate a PGPKeys object
/// </summary>
private PGPKeys _pgpKeys;
private const int BufferSize = 0x10000;
public PGPEncrpyt(string _pubKeyPath, string _privKeyPath, string _password, long _keyID)
{
_pgpKeys = new PGPKeys(_pubKeyPath, _privKeyPath, _password, _keyID);
}
private static void writeAndSign(Stream ouput, Stream literalout, FileStream inputFile, PgpSignatureGenerator sigGen)
{
int length = 0;
byte[] buf = new byte[BufferSize];
while ((length = inputFile.Read(buf, 0, buf.Length)) > 0)
{
literalout.Write(buf, 0, length);
sigGen.Update(buf, 0, length);
}
sigGen.Generate().Encode(ouput);
}
private Stream encrypt(Stream output)
{
PgpEncryptedDataGenerator pgpEncDataGen = new PgpEncryptedDataGenerator(SymmetricKeyAlgorithmTag.Twofish, new SecureRandom());
pgpEncDataGen.AddMethod(_pgpKeys.PGPPublicKey);
Stream encryptedOutput = pgpEncDataGen.Open(output, new byte[BufferSize]);
return encryptedOutput;
}
private Stream compress(Stream output)
{
PgpCompressedDataGenerator pgpCompDataGen = new PgpCompressedDataGenerator(CompressionAlgorithmTag.Zip);
Stream compressedEncryptedOut = pgpCompDataGen.Open(output);
return compressedEncryptedOut;
}
private Stream literalOutput(Stream compressedOut, FileInfo file)
{
PgpLiteralDataGenerator pgpLiteralDataGen = new PgpLiteralDataGenerator();
Stream literal = pgpLiteralDataGen.Open(compressedOut, PgpLiteralData.Binary, file);
return literal;
}
private PgpSignatureGenerator sigGen(Stream compressedOut)
{
const bool Iscritical = false;
const bool IsNested = false;
PublicKeyAlgorithmTag tag = _pgpKeys.PGPSecretKey.PublicKey.Algorithm;
PgpSignatureGenerator pgpSigGen = new PgpSignatureGenerator(tag, HashAlgorithmTag.Sha1);
pgpSigGen.InitSign(PgpSignature.BinaryDocument, _pgpKeys.PGPPrivateKey);
foreach (string userID in _pgpKeys.PGPSecretKey.PublicKey.GetUserIds())
{
PgpSignatureSubpacketGenerator subPackGen = new PgpSignatureSubpacketGenerator();
subPackGen.SetSignerUserId(Iscritical, userID);
pgpSigGen.SetHashedSubpackets(subPackGen.Generate());
break;
}
pgpSigGen.GenerateOnePassVersion(IsNested).Encode(compressedOut);
return pgpSigGen;
}
public void EncryptSignAndZip(Stream output, FileInfo unencryptedinput)
{
if (output == null)
{
throw new ArgumentNullException("The output stream cannot be null");
}
if (unencryptedinput == null)
{
throw new ArgumentNullException("You must supply a filename to encrypt");
}
if (!File.Exists(unencryptedinput.FullName))
{
throw new ArgumentNullException(unencryptedinput + " does not exist");
}
using (Stream encryptedout = encrypt(output))
using (Stream compressedOut = compress(encryptedout))
{
PgpSignatureGenerator signature = sigGen(compressedOut);
using (Stream literalOut = literalOutput(compressedOut, unencryptedinput))
using (FileStream inputfile = unencryptedinput.OpenRead())
{
writeAndSign(compressedOut, literalOut, inputfile, signature);
}
}
}
}
}

Thursday 4 June 2009

Reporting made easy

I recently found out at work that the MI analysts have a monthly report running up to seventy pages that they need to collate. Nothing unusual there, lots of places have big monthly reports. Then I was told that it takes five days to collate.

I initially though this was a joke, afterall this report had been publised each month for the last two years. Once I was told no, we really do do this, I needed to go and get myself a cup of sweet tea. After I had recovered, I started to speak to some of the analysts to try and find out the complete story behind this seemingly terrifying report, once I started to get some of the details behind this I came to the conclusion that the main reason for the length of time it takes to create this report was down to analysts doing the same thing every twenty-five days by hand. A prime opportunity to automate if ever there was one - and I had been after an excuse to create a .Net DAL for the team as well :).

There is nothing outstanding about this project at all, there isnt really any cutting edge tech being employed nor is there any new, fancy way of doing things being discussed. There is rarely anything taxing when it comes to generating reports. The most complicated thing you need to do is validate all the information that you are presenting, once the formula is there you are safe in the knowledge that your report will continue to churn out the required data for ever, the only time you will ever need to test it is when a new requirement is added to the schema of the report itself.


The uber report I am going to tackle just now is intended for publication via printed media. Because of this, I am confident in the knowledge that once the report has been created, no one will need to copy and past the data from it into another application - I know it gets distributed on paper. So, I know that I dont really need to provide a spreadsheet, csv, .doc or peice of XML initially. This is good news for me, as I only need to worry about one format, and I have decided to go for PDF. I had a look through the report a few days ago, when initially this task was given to me. Basically, its just a collection of graphs and tables, very little text, this just gets better as far as I am concerned. One format to worry about and very little wording to go into it - bliss!

So, what do I need for this to work? I need:
  • A graphing lib, I am going for ZedGraph. Its open source and does everything I need.


  • A PDF generation lob, for this I am going to use PFDJet. This is a commercial product, but there is also an open source version. As I dont need to do anything that fancy, it will do for me.

And thats pretty much it, all I will need to do is get the data I need from the db and use it to create some graphs and charts which go into a PDF report. Couldnt be simpler really, could it?

After a good planning meeting with the team lead and analyst I would be working with, it turns out that there is a similar system already in place for creating the basis for some of the data needed to create this report. After this session, I found that there is a monster Oracle DB somewhere in the depths of the core business that holds all the information we need. With the interesting bits to one side, I was then told a fairly comical story about Oracle, MySql and Access all doing a funny dance...

The only challenging thing really is the design of the report. Now that people have learned that I am automating some report generation, it seems like everyone has a preference as to how it should look. Right now, I am not concerned with fancy graphics or logos, right now the only important thing is getting the correct data into the report in a logical manner...

Saturday 23 May 2009

Encrypting Connections strings in .Net with NAnt

I have been working a lot with databases over the last couple of weeks, one of the things that is a novelty to the people I am working with now is the ability to store multiple connection strings in our .config files. Its a really simple and flexible thing to do, allowing us to transport any of our apps from environment to environment with very little hassle. The only downer is that the connection strings are stored in plain-text, which naturally is a massive security hazard.



That is, of course, unless you encrypt them.



There are loads of examples on the Internet explaining how this can be done, and they are very simple to follow and implement - but I wonder how many people make use of this fairly simple feature? One thing that became apparent to me was, even though I can encrypt the file - just when and how do I want to do this? The when is simple, I want my connection strings encrypted when the application moves into a testing environment an I want them to stay encrypted from then on. The how is slightly different, there is no point in having the encryption take place as I am developing the app, as I want to have the flexibility of changing the connection strings as and when I need them. So, the ideal time for me to encrypt them would be during the build itself. Its a very straight forward thing to do, simply write a small program that encrypts the connection strings section of a web or app .config and have Visual Studio process this as a post build event. Its possibly the easiest way to do this. But, this isnt so flexible, it would only really encrypt one file relevant to a project, and whilst the code is reusable, it isnt very agile.

So, why not move this from being a post build event in the soloution file out to being a post build event in your larger build process? Using NAnt (or MSBUILD), we can create a custom task that will encrypt the connectionStrings section in any .config file. This could be used over and over again as and when requirements dictate a section in a config file should be encrypted.

There are a couple of things we need to capture:
  • The path of the .config file
  • What sort of encryption we want to use.

It would appear that .Net offers two methods to encrypt a .config file, DPAPI and RSA. DPAPI is provided by Microsoft, there is a nice article on Wikipedia discussing this encryption method. For a lot of people though, RSA may be a more suitable encryption method (its the one I am using if that is worth anything).

Back to the task. To start with, I want to get the location of the .config file and decide which encryption method I want to use. This is easy enough:


[TaskName("config-encrypt")]
class Encryption : Task
{
private string strProvider;
private string _filePath;
[TaskAttribute("configfile", Required = true)]
public string FilePath
{
get { return _filePath; }
set { _filePath = value; }
}
private string _rsa;
[TaskAttribute("RSAencryption", Required = false)]
public string RSA
{
get { return _rsa; }
set { _rsa = value; }
}
private string _dpapi;
[TaskAttribute("DPAPIencryption", Required = false)]
public string DPAPI
{
get { return _dpapi; }
set { _dpapi = value; }
}
protected override void ExecuteTask()
{
if (checkFileExists())
{
if (FilePath.Contains("web.config"))
{
encryptWebConfig();
rename();
}
else
{
encryptAppConfig();
rename();
}
}
else
{
throw new BuildException("The .config nominated does not seem to exist");
}
}

I have left the encryption methods as optional - this way, you dont need to worry about which type you need, however by doing this I need to set up a default encryption provider. A simple method like the one below will take care of that for me, I have established that I want to use RSA as the default encryption method if no other is specified:
private void chooseEncryptionProvider()
{
if (RSA == "true")
{
strProvider = "RSAProtectedConfigurationProvider";
}
else if (RSA == "" & DPAPI == "")
{
strProvider = "RSAProtectedConfigurationProvider";
}
else if(DPAPI == "true")
{
strProvider = "DataProtectionConfigurationProvider";
}
}

I also need to make sure that the file I want to apply encryption to exists in the first place. Again, this is very simple to do:


private bool checkFileExists()
{
if (File.Exists(FilePath))
{
return true;
}
else
{
return false;
}
}

Now I am all set to get encrypting. I want my task to be able to handle both web and app.config files, so I have two methods to handle each. They are pretty much identical in their operation, the only real difference is down to the way .Net handles web.config files. To encrypt a web.config I have done the following:
private void encryptWebConfig()
{
chooseEncryptionProvider();
try
{
Configuration _configFile = WebConfigurationManager.OpenWebConfiguration(FilePath);
if (_configFile != null)
{
try
{
ConnectionStringsSection _section = (ConnectionStringsSection)_configFile.GetSection("connectionStrings");
_section.SectionInformation.ProtectSection(strProvider);
_section.SectionInformation.ForceSave = true;
_configFile.Save();
}
catch (Exception e)
{
throw new BuildException("Failed to encrypt the connection strings sectiion",
e.InnerException);
}
}
}
catch (Exception e)
{
throw new BuildException(e.Message.ToString(), e.InnerException);
}
}

You can see where I am choosing the encryption provider at the start of the method. One thing I havent done here is implement any logging. For the sake of an audit trail, it would be a very wise thing to log what sort of encryption is being used. This way, you can look back across the builds you have completed for a specific environment and see which encryption provider was used. The following method encrypts an app.config:


private void encryptAppConfig()
{
string path = FilePath.Replace(".config", "");
chooseEncryptionProvider();
Configuration _configFile = ConfigurationManager.OpenExeConfiguration(FilePath);
if (_configFile != null)
{
try
{
ConnectionStringsSection _section = (ConnectionStringsSection)_configFile.GetSection("connectionStrings");
_section.SectionInformation.ProtectSection(strProvider);
_section.SectionInformation.ForceSave = true;
_configFile.Save();
}
catch (Exception e)
{
throw new BuildException("Failed to encrypt the connection strings section"
, e.InnerException);
}
}
}

As you can see, it is pretty much the same idea for this operation compared to encrypting a web.config. With all this in place, we now have all we need to automatically encrypt a .config file as part of an automated build. This can be added on to the end of a build target in a NAnt script, or better yet added to its own target to be used as a post build event. One thing I found when using this is that after the .config had been encrypted a .config.config file was created leaving the un-encrypted .config in the same directory. Naturally, this defeats the point so I put the following method together to clean up for me:
private void rename()
{
if (File.Exists(FilePath + ".config"))
{
string oldname = FilePath;
string newname = FilePath + ".config";
File.Delete(FilePath);
File.Copy(newname, oldname);
File.Delete(newname);
}
}

And there we go, all done. The syntax for the build script is insanely simple - if you dont mind using RSA encryption all you need to do is point it at the .config you want to encrypt:
</target>
<target name="encrypt">
<config-encrypt configfile="C:\test\web.config" />
</target>

I have been using this task for the past week or so, and it has been working like a dream :). The complete code for the task is below:
using System;
using System.Collections.Generic;
using System.Configuration;
using System.IO;
using System.Linq;
using System.Text;
using System.Web;
using System.Web.Configuration;
using NAnt.Core;
using NAnt.Core.Tasks;
using NAnt.Core.Attributes;

namespace Custom.NantTasks.Encryption
{
[TaskName("config-encrypt")]
class Encryption : Task
{
private string strProvider;
private string _filePath;
[TaskAttribute("configfile", Required = true)]
public string FilePath
{
get { return _filePath; }
set { _filePath = value; }
}
private string _rsa;
[TaskAttribute("RSAencryption", Required = false)]
public string RSA
{
get { return _rsa; }
set { _rsa = value; }
}
private string _dpapi;
[TaskAttribute("DPAPIencryption", Required = false)]
public string DPAPI
{
get { return _dpapi; }
set { _dpapi = value; }
}
protected override void ExecuteTask()
{
if (checkFileExists())
{
if (FilePath.Contains("web.config"))
{
encryptWebConfig();
rename();
}
else
{
encryptAppConfig();
rename();
}
}
else
{
throw new BuildException("The .config nominated does not seem to exist");
}
}
private void encryptWebConfig()
{
chooseEncryptionProvider();
try
{
Configuration _configFile = WebConfigurationManager.OpenWebConfiguration(FilePath);
if (_configFile != null)
{
try
{
ConnectionStringsSection _section = (ConnectionStringsSection)_configFile.GetSection("connectionStrings");
_section.SectionInformation.ProtectSection(strProvider);
_section.SectionInformation.ForceSave = true;
_configFile.Save();
}
catch (Exception e)
{
throw new BuildException("Failed to encrypt the connection strings sectiion",
e.InnerException);
}
}
}
catch (Exception e)
{
throw new BuildException(e.Message.ToString(), e.InnerException);
}
}

private void encryptAppConfig()
{
string path = FilePath.Replace(".config", "");
chooseEncryptionProvider();
Configuration _configFile = ConfigurationManager.OpenExeConfiguration(FilePath);
if (_configFile != null)
{
try
{
ConnectionStringsSection _section = (ConnectionStringsSection)_configFile.GetSection("connectionStrings");
_section.SectionInformation.ProtectSection(strProvider);
_section.SectionInformation.ForceSave = true;
_configFile.Save();
}
catch (Exception e)
{
throw new BuildException("Failed to encrypt the connection strings section"
, e.InnerException);
}
}
}

private void chooseEncryptionProvider()
{
if (RSA == "true")
{
strProvider = "RSAProtectedConfigurationProvider";
}
else if (RSA == "" & DPAPI == "")
{
strProvider = "RSAProtectedConfigurationProvider";
}
else if(DPAPI == "true")
{
strProvider = "DataProtectionConfigurationProvider";
}
}

private bool checkFileExists()
{
if (File.Exists(FilePath))
{
return true;
}
else
{
return false;
}
}

private void rename()
{
if (File.Exists(FilePath + ".config"))
{
string oldname = FilePath;
string newname = FilePath + ".config";
File.Delete(FilePath);
File.Copy(newname, oldname);
File.Delete(newname);
}
}
}
}

Tuesday 19 May 2009

Using log4net with C# and MySql

Logging your application is a basic unwritten requirement. I cannot think of one application I have ever worked with that does not provide any form of logging, be it a logfile or a log in a database.

Recently at work, I came across a situation where someone described a log as a "nice to have". No, it's not nice to have - its essential to have it. As I was working in .Net (c# to be exact, hence the title), I decided to take a look at log4net. This is a logging framework maintained by the Apache project and is a port of the popular log4j to .Net, I am not going to bore people with an explanation of it here, all I will say about it is that it is a very powerful and flexible logging service. You can check out the official site here: http://logging.apache.org/log4net/index.html.

I spent a few weeks working with log4net (as it apparently wasnt essential to have a log, it was a low priority task). Creating a logfile with it is incredibly easiy, as is logging to the console. There are wide range of methods for creating a log, called appenders, however the most useful for me was appending the log to a database. Whilst log4net is popular, powerful and active, there isnt that much in the way of documentation or howto's, which could put some people off using it, this is especially apparent when logging to a database. The examples that already exist on the interweb, while useful, are sometimes out of date using deprecated syntax.

For my logging scenario, I needed to maintain a log in a MySql table, this compounded my explotation of log4net, as it isnt readily obvious how this can be done. My aim is to use log4net for all of my logging needs, so I want to create an assembly that can be referenced etc wherever I need it. Due to the extremely felxible nature of log4net, this is achievable - the majority of the configuration for it lives within the app.config (or web.config if you are that way inclined). It is possible to programatically configure log4net from within your code - but what would be the point? If you do that, then you loose all of the flexibility offered. One way to highlight this feature - right now I need to log to a MySql table, but what about in future if we decide to migrate to another DB? If I configured my logger programtically, I would then need to rewrite code. Whereas if I configured it via a .config file, all I need to do is deploy that file wherever it needs to go...

Anyway, enough waffle - you are all grownups and have myriad needs. Lets take a look at how I got log4net logging to my MySql table.

Setting up log4net to log to the console or to a file is dead simple, there is very little taking place in the code, most of it is done in the config file. The same can go for logging to a db, but with one small difference - if you rely on the config file to control your logging when using a database you have to be comfortable with the fact that your connection string will be visible. I am not sure how much of a security risk that is for everyone else, but for me it wasnt something I was willing to do at all. Even if you use a seperate database for logging, you are still exposing a lot of information about your infrastructure here. So, instead of placing the connection string in the log4net config, I am going to place it in the connectionStrings section and encrypt it. Once this is done, I need to set the connection programatically. This isnt as easy as you would first imagine (well, at least I didnt find it that easy), whilst there is a lot of documentation on what log4net can do, I found it quite tricky to figure out how to make it work with MySql - and even when I had I still had one small issue... Anyway, on to getting this setup.

All of my config for my logger is going to live in my app.config - inlcuded my encrypted db connection string. I need to get this string and then provide it to log4net, this bit is done in the code. It sounds simple but there are a number of steps I need to take first. To start with I need to get the default repository that my logger is going to use with log4net:
private Hierarchy hierachy()



{



Hierarchy h = (Hierarchy)log4net.LogManager.GetRepository();



return h;



}




It is possible to use different configuration repositories with log4net - but I dont need to so I havent had to worry about this. To be honest, I am not sure that a lot of people will need to either. Once this has been done, we need to create an ADOAppeneder - it should also be pointed out by now that I am using the MySql .Net connector to work with my db.
private AdoNetAppender adoAppender()



{



Hierarchy myHierarchy = hierachy();



if (myHierarchy != null)



{



AdoNetAppender adoApp = (AdoNetAppender)myHierarchy.Root.GetAppender("ADONetAppender");



return adoApp;



}



return null;



}




With this code, I am able to get the section of my .config file that holds all of the config for logging to a db through log4net, log4net relies on one or more 'appenders' being configured for use - my db appender is simply called 'ADONetAppender', so I have specified this in the above method. With this done, I am now ready to configure my appender programtically:
private void createAdoSettings(string dbConn)



{



AdoNetAppender myAppender = adoAppender();



if (myAppender != null)



{



myAppender.ConnectionString = dbConn;



myAppender.UseTransactions = true;



myAppender.ActivateOptions();



}



}




After all of this, we are done. All I need to do is tell my logging class what my connection string is. This isnt so difficult (thankfully, is very easy):
string connstring = ConfigurationManager.ConnectionStrings["SQLLogDB"].ConnectionString.ToString();



SetUpLog _setup = new SetUpLog(connstring);




Cant get any simpler really :). All I need to do is drop this into any class I want to log from and I am set. There is only one other addition I need to make, and that is to implement the ILog interface. This is the most important bit, regardless of any db logging, without it you cant log anything at all! I got a nice little gem from here: http://www.ondotnet.com/pub/a/dotnet/2003/06/16/log4net.html that give a good tip on implementing the interface. Basically, popping this line in at the start of your class ensures that log4net reports the name of the class duing the logging process:
protected static readonly log4net.ILog log = log4net.LogManager.GetLogger(System.Reflection.MethodBase.GetCurrentMethod().DeclaringType);




This is so useful and provides a very concise log. With all this in place, we are now able to use log4net to log our application. The following code is the test app I wrote when to prove that it works:
using Logger.Setup;



using System;



using System.Configuration;



using System.Collections.Generic;



using System.Linq;



using System.Text;







namespace TestClient



{



class Program



{



protected static readonly log4net.ILog log = log4net.LogManager.GetLogger(System.Reflection.MethodBase.GetCurrentMethod().DeclaringType);







static void Main(string[] args)



{



Program test = new Program();



test.logtest();



}



private void logtest()



{



string connstring = ConfigurationManager.ConnectionStrings["LogDB"].ConnectionString.ToString();



SetUpLog _setup = new SetUpLog(connstring);



log.Info("Hello there");



}



}



}








On execution, it provides the following on the command line:
2009-05-23 23:21:37,536 [12260] INFO  Hello there 23 TestClient.Program





You can also see how I am specifying the connection string, which is also held in the .config file. This is driven by my .config file, it is just a console appender and lives inside the log4net section:

<appender name="ConsoleAppender" type="log4net.Appender.ConsoleAppender">



<layout type="log4net.Layout.PatternLayout">



<conversionPattern value="%date [%thread] %-5level %message%newline %L %C" />



</layout>



</appender>




Now I know it works, I can get down to logging the same information to my db, first I create my schema:
CREATE TABLE logdb.mylog(



`DateTime` DATETIME DEFAULT NULL,



Thread VARCHAR (20) DEFAULT NULL,



Level VARCHAR (20) DEFAULT NULL,



Message VARCHAR (20) DEFAULT NULL,



Exception VARCHAR (20) DEFAULT NULL,



line VARCHAR (20) DEFAULT NULL,



object_ref VARCHAR (20) DEFAULT NULL,



taskID VARCHAR (20) DEFAULT NULL



)



ENGINE = INNODB



CHARACTER SET latin1



COLLATE latin1_swedish_ci;





The object_ref and taskID columns are things I want to capture as part of my wider logging process - they dont really have anything to do with log4net or its configuration. With my schema in place all I need to do now is configure log4net to log my messages to my new table. This can be a bit frustrating when working with MySql - if you get something wrong in the config you will get an error message, but they dont always seem to be that helpful to me. For instance, log4net requires you to define a connection type in order to log to a db. When I first started working with log4net, my connection type was incorrect which resulted in an error. However, the error I got complained about connection strings being null - not that there was an error with the connection type I had configured. Granted, they are closely related, but it would have just been nice if I get an error back telling me that my connection type was wrong... Anyway, I got passed it. When configuring log4net there are a couple sections that you need to work with. I am going to assume that if you are reading this then you have already looked at the log4net site and looked at how it is configured. I am not going to explain how to configure log4net here in general, if you want to see how this works then there are a lot of other articles out there that do this well, one good place to start on the topic of configuration is the log4net site - even though it is very general it will give you an idea of how all the appenders look: http://logging.apache.org/log4net/release/manual/configuration.html

Now that my schema is in place and my logic is complete, all I need to do now is configure a log4net appender. Appenders in log4net control how information is formatted as it is logged, I am going to use an ADO appender and it looks like this:
<appender name="ADONetAppender" type="log4net.Appender.ADONetAppender">
<connectionType value="MySql.Data.MySqlClient.MySqlConnection, MySql.Data, Version=6.0.3.0, Culture=neutral, PublicKeyToken=c5687fc88969c44d" />
<commandText value="insert into mylog (thread,line,object_ref,taskID,message,level,datetime)
values(?thread,?line,?objectref,?taskid,?message,?level,?log_date)" />
<parameter>
<parameterName value="?thread"/>
<dbType value="String"/>
<size value="20"/>
<layout type="log4net.Layout.PatternLayout">
<conversionPattern value="%thread"/>
</layout>
</parameter>
<parameter>
<parameterName value="?line"/>
<dbType value="String"/>
<size value="20"/>
<layout type="log4net.Layout.PatternLayout">
<conversionPattern value="%L"/>
</layout>
</parameter>
<parameter>
<parameterName value="?objectref"/>
<dbType value="String"/>
<size value="20"/>
<layout type="log4net.Layout.PatternLayout">
<conversionPattern value="%C"/>
</layout>
</parameter>
<parameter>
<parameterName value="?taskid"/>
<dbType value="String"/>
<size value="20"/>
<layout type="log4net.Layout.PatternLayout">
<conversionPattern value="%property{taskid}"/>
</layout>
</parameter>
<parameter>
<parameterName value="?message"/>
<dbType value="String"/>
<size value="20"/>
<layout type="log4net.Layout.PatternLayout">
<conversionPattern value="%message"/>
</layout>
</parameter>
<parameter>
<parameterName value="?level"/>
<dbType value="String"/>
<size value="20"/>
<layout type="log4net.Layout.PatternLayout">
<conversionPattern value="%level"/>
</layout>
</parameter>
<parameter>
<parameterName value="?log_date" />
<dbType value="DateTime" />
<layout type="log4net.Layout.RawTimeStampLayout"/>
</parameter>
</appender>

The most notable thing about my config here is the use of a property. This lets me capture custom information for my log and place this in my db. This is really useful, I am sure you can all understand why and goes to show the power of log4net. With all of this in place, I am now able to log to my db - but with one small caveat, I think I have found a problem with log4net when using an ADO appender with MySql. For some reason, log4net generates an error telling me that there is a null system exception, but carries on and logs the data to the db anyway. I spent a long time trying to figure out what the problem was, but couldnt spot it. So, after some advice, I started working with the log4net source to spot where the error was being generated. Stepping through the code, I found that log4net (when used with the MySql .Net connector) seems to loose the connection string, but logs data anyway. I couldnt replicate this with SQL Server and have still been unable to resolve the error. I dont like errors hanging around like this, the data is logged, but I get this nasty message.... I have posted my findings to the log4net support group, but have not had anything back about it really - which is a shame. However, in saying that, log4net is still a truly useful tool to use in any project that requires any form of logging.