Mongrel and Docker: the power of containers

Last time we looked at how we started off the Mongrel2 webserver in a docker container. It was a very simple setup, with a single container running an instance of Mongrel with a few bits and pieces of static content.

This time, we're going to look at what makes Mongrel so interesting, and why I think that Docker suits it perfectly as a deployment mechanism. We'll shoot over the basics of handlers, and I'll summarise the handler that I created, and how Docker

Handlers

Mongrel2 doesn't deploy applications in the same way that, say, servlet containers do, and it doesn't do any processing of code itself in the way that PHP applications might. Instead, it has a construct called a handler. These are specific paths defined in the server configuration that, when requested, construct a message for the ZeroMQ message framework, and pass them to a socket. A dedicated application reads the message from that socket, takes any necessary action, and then responds to the Mongrel2 server by placing a ZeroMQ message back to a new queue.

Handler application: "thought for the day"

In this case, I've only constructed one handler - an incredibly simple one, that could have easily been managed other ways, but we'll test the water slowly. It's a simple "thought for the day" generator, that will return a json object containing a quotation, and a source for that quotation.

The code for this handler isn't checked into Github yet, but is very simple. There's a single, looping process that waits for messages, and returns one of a random set of quotations whenever it receives a message. It's just a little jar file that gets executed and stays up until terminated.

Accessing the handler

Accessing the handler from the frontend is relatively easy: I've wired up a simple AngularJS controller that just grabs the json object from the /thought path, and plugs it into some html code on the front page. Nothing too fancy.

Putting it all together

So now we have an infrastructure that looks like this:

Mongrel2 and the handler process are both running in docker containers. The communication ports for the two of them are exposed within the docker engine, but not outside of it. Mongrel's main access point (port 6767, in this case) is mapped to port 80 on the virtual machine and exposed to the outside world.

However, we're still not quite done yet. To make things even easier, we can use Docker Compose to describe this entire diagram, and suddenly we're able to build, deploy and start all of our containers from a single command. Again, we're not at a particularly complex level yet, this single file

mongrel2:
	build: ./mongrel2-main
	ports:
	- "80:6767"
	expose:
	- "5557"
	- "5558"
	samplehandler:
	build: ./sample-handler
	links:
	- mongrel2

will use the information in the two dockerfiles to build — from scratch — the entire application above and deploy it.

And that is an amazing tool, which will give us the ability to add sections to our infrastructure quickly and easily.

Getting into docker: the simple case

So last time I'd been left in the situation of moving from Vagrant over to Docker. And I found myself really appreciating what Docker was doing, and beginning to get my head around what it's capable of. There's still a long way to go in order to use it properly, but I think I'm beginning to get the basics.

I ended up in a situation where I just about had the Mongrel2 webserver being loaded up onto a dockerfile, and starting up.

I've managed to take that a little further in the right direction now.

Step 1: Getting Mongrel to start properly — and keep running
I touched on it last time, but the thing you really have to nail to get Docker to work is the ability to start a single process in a container and keep it running. This really isn't as easy as it could be, given a few of the limitations of the way Docker runs things.

A docker container will only keep running as long as the process started on id 1 keeps running. There are a few hacky ways of doing this, like piping together a series of shell scripts, and finishing by tailing a log file, but that's best avoided. The best way that I've found so far is to use the supervisor daemon, which starts on process 1 and then spools up the processes that you deem necessary. It actually turned out to be easier – with Mongrel – to fire up a second supervisor process called procer in order to manage the startup of the Mongrel server. There might be a better way of doing it, but this seems to work, and (in theory) gives a layer of resiliency to the Mongrel process by granting automatic restarts in case the process dies on us.

Step 2: Getting some static content in there
The next step was to get some static content onto the site. That was pretty easy, and we ended up with an infrastructure that looked a bit like this.

The Dockerfile controlling the Mongrel2 server pulled all the necessary files to install Mongrel2 and the dependencies, copied in a set of static files, and finally started the server.

Next steps:
This is all very well, but it seems like a lot of effort to go to in order to get some static content served up by a webserver – and it is. Next up is the first handler, the independent programs that make Mongrel2 interesting, and how I believe they are perfectly suited to running on a containerised platform.

Docker vs Vagrant - Round 1

Having kicked around Vagrant (https://docs.vagrantup.com/v2/) for a while, a colleague finally persuaded me to try out Docker instead, just to see what the competition was like.
... so, I spent a while working out ways of setting up a virtual machine / container to run an instance of the mongrel2 webserver, just to see how they compared.
The results are now in and ... well, it's a bit of a mixed bag, to be honest.
The Dockerfile and associated project are up and available on github, at https://github.com/nihilogist/docker-mongrel2, for those interested.

Setup

How easy are the two different solutions to set up?

Vagrant 

It's really easy. You write a few scripts to download and install the software you need - in this case it's a little more complex as you need to grab ZeroMQ and then make mongrel, but it's not hard at all. You copy what you need over to the VM and start the server. Great.

Docker

It took a while, I have to say. I found it a good deal harder to get my head around the way that the container works. Especially coming from Vagrant, which is purely and simply a method of managing VMs, it took some getting used to the idea that the container only persists as long as the process on PID1 is running. So this means that there were a few extra steps needed to ensure that PID1 keeps running, but happily there are plenty of tools to help with this.

Running

How do the two containers run?

Vagrant

You have a full VM running. You can ssh into it. You can see it running in VirtualBox (if that's the provider you're using). You can use it in exactly the ways you'd use any other virtual machine. But it is pretty heavy - it takes a while to boot up, but seems pretty solid once it's up, as you'd expect.

Docker

It's just a container - there are lots of limits on what you can do with it and what you can't. Well - not so much limits as recommendations. Using Vagrant I think  you could be easily tempted to start up a whole load of extra processes on the VM, just because it's easy, and if you have the machine running, then why not? Docker, on the other hand, really wants you to dedicate each container to a single process. It's really lightweight. It starts in an instant.

Which do I prefer?

Well, it's little early in the day, but I think that Docker has won me over. I've still got a heck of a lot to explore, like getting data volumes to work properly, but I'm really enjoying using it.
I'm also keen to explore the way that mongrel2 wants you to use many small services, and I think that Docker is perfectly aligned with that - each container running a single service, but running it well.

Automated deployments: What are the components of a system?

I've been thinking a lot recently about software deployments, particularly in my current project, which is a (relatively) standard Java EE application: application server, database, messaging framework, blah blah blah. As I'm sure a lot of projects do, we have a hotch-potch of assorted build scripts, deployment scripts, semi-automated configuration scripts.

Two or three members of the project team try to manage deployments to test systems, and have currently managed to write several thousand lines of code which act as a universal way of deploying the artifacts to the various environments which ... doesn't work. These issues seem to crop up on every project - certainly every developer I've worked with recounts stories of projects where weeks of time were lost trying to figure out why code that apparently worked perfectly would not even deploy to a target system.

The upshot of this was that I started trying to think about the process in a more modular fashion; to break it down into smaller components and see if there's a standard pattern that we could apply. And I think I've come up with a few things - some of which are obvious, but we may as well start with the obvious and see where it leads us.

The System
To create this mythical working system we need three things:

• Environment
• Build artifacts
• Deployment processes

Environment: the environment is the actual machines (either physical or virtual) that the software will run on. It includes all third party additions required to run the software, such as databases, web / application servers, messaging systems and so on. It also includes all necessary network configurations such as load balancers, firewalls and host configurations.

Build artifacts: the build artifacts are the final output of building the source code ready for deployment. They may be tailored to fit a specific environment by setting runtime variables, but other than that the source code itself should not need to know about environment-specific details.

Deployment processes: the deployment processes are the description of steps needed to take the build artifacts, transfer them to the relevant environments, and initialise them such that the system is available for use.

These three aspects of the system can - and should - be treated separately when it comes to automating the overall process of creating and deploying environments. For instance, we should not be trying to write code in our build artifacts that generates application servers that are then deployed alongside the application files; they are part and parcel of the environment, and that should be as static as possible for the lifetime of the system. Also, the deployment process should not be part of building the artifacts from the source code: it should be possible to build individual artifacts and deploy them as necessary.

All these principles stem from the idea of the separation of concerns. Our build process should not interfere with a deployment; our environments should not define the way our software is built from source code.

Testing times

I'm very excited about the upcoming Dev/Test Lab feature that's coming to preview soon in Azure.

A lot of time working in an agile team is spent working on testing. Even with a dedicated QA engineer, a moderate-major portion of any development work is spent writing tests, running tests, and just flat out checking that the stuff you've written works.

It's not always easy to do that just on your local workstation, even if you're lucky enough to have a stupidly powerful machine at your disposal (spoiler alert for employers: this really does make a difference :) )

On the other hand, you can't really have a bunch of test machines, with all the associated maintenance and other costs, available for the development team to use on a whim, or because the QA team are running extended regression tests on an earlier build.

Dev/Test Lab might just help with that.

There are a bunch of tools that can be used with public or private clouds to simplify the provisioning and deployment of virtual machines and - given that many production environments now run on similarly virtualised equipment - this is a very reasonable way to run your test environments. What really has my interest in this offering from Azure is the tooling around it - making it easy to bring up and shut down systems, as well as manage a team's budget. (Sadly, this is reality, and your managers don't tend to appreciate you racking up $50,000 in VM fees because you forgot to shut down the load test system before going on holiday).

So yeah - I'm very much looking forward to kicking the tires on the preview. I might even find time to write about it.

Building sandcastles part 4: Azure

Continuing the adventures in continuous integration, I'm now looking into how services like AWS or Azure could really help dev teams along. A lot has been written about this before now, but while there seem to be lots of places talking about how easy AWS makes their dev / test / deploy cycle, I haven't seen too many places talking about all the components you might need.

So, looking at it from the ground up, we need, in the simplest terms:

• Somewhere to store our code
• Somewhere to build our code
• Somewhere to test our builds and artifacts
• Somewhere to host our final builds

This translates to:

• Some kind of SCM system
• Some kind of continuous integration system
• A private set of machines to deploy our applications to (let's assume that we're building a web application, so we'll need webserver and a database, minimum)
• A public set of machines to deploy the applications to

After mucking around with the trials of Azure and AWS, I decided to run a few experiments with Azure, and have ended up - so far - with meeting the first two requirements, with virtual machines running GitLab and TeamCity respectively.
I created a rather simple web application - no database yet, and pushed it up to GitLab, and then connected TeamCity over to that to run a couple of builds.

So far, so easy.

The next step - which feels like a pretty big one - is to build a test system. Now, it's very easy to set up a virtual machine, set up a webserver and then have a script to deploy the build artifacts from TeamCity over to it, but in an ideal world - where it's not just me working on it - it would be ncie to be a lot more flexible than that. What I really want to achieve is to be able to deploy a whole test environment with the click of a button.

That means that we need to start looking at the automated provisioning of virtual machines, because the last thing that anyone wants to do is set up a whole environment every time they want a build. Happily, there are a bunch of tools that can help us with this.

For a first experiment, I'm using Vagrant. It seems to be generally straightforward, and - in theory - relatively portable. It's been somewhat of a pain to get working with Azure, but now that it's running, I seem to be able to deploy machines relatively easily. Next steps are to actually get servers running on them.

Building sandcastles, part 3: Scala

(aka: And Now For Something Completely Different)

So, this is less about the continued experimentation in integration a bunch of SCM and CI systems, and more about random experimentation, because sometimes I just roll that way ;)

Pretty much on a whim, I brought Scala into the mix of technologies used. Following the recommendations of a couple of folk I've spoken to about this, I brought it in to the unit testing layer first, so that I can give it a go without too much impact on the actual written code. To be honest, it felt like a bit of a slog (though, having said that, I did manage to get the first unit test written and passing within an hour, after a beer, so I guess it can't be that hard).

First of all, we had to install Scala and the various IntelliJ plugins. That wasn't so bad. Second was to research how to run a JUnit test in Scala. Again, not too bad - suggestions are to use scalatest, and then have your unit tests extend the JUnitSuite class. Writing the unit test? No worries. Well ... not for the real simple ones, anyway. Running the test? Now that was the trick. First of all, you need to make sure that the version of scalatest you are using is compatible with the version of scala that you're using. Then you have to make sure that all the scala setup you did in the IDE also matches the version of scala you installed. Finally, you might need to tweak the build steps in the pom.xml file. Sadly, very few of the errors that get thrown when you have these things wrong make much sense.

For future reference (for myself, but maybe someone else will find this useful as well), this build configuration in maven seems to work quite well:

    <build>
        <plugins>
            <plugin>
                <groupId>org.scala-tools</groupId>
                <artifactId>maven-scala-plugin</artifactId>
                <version>2.15.2</version>
                <executions>
                    <execution>
                        <id>scala-compile</id>
                        <goals>
                            <goal>compile</goal>
                            <goal>testCompile</goal>
                        </goals>
                        <configuration>
                            <args>
                                <arg>-dependencyfile</arg>
                                <arg>${project.build.directory}/.scala_dependencies</arg>
                            </args>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

Something to be aware of is that the -make:transitive configuration argument seems to be redundant as of scala 2.11, and actually breaks the compilation. I've seen it mentioned in a couple of places, but taking it out made everything magically work for me... YMMV.

So anyway, I now have a very very simple test committed, which is currently passing. We'll see how things go as I try to rack up the complexity, and start on mocking out various interfaces.

Building sandcastles part 2

You know, that went pretty well, all things considered.

So, there are a couple of repositories up on GitHub now:
https://github.com/nihilogist/DiceEngine
https://github.com/nihilogist/PsychicWight

I managed to set up TeamCity without too many issues, and have that observing the two repositories.
It's also observing a given set of branches of the two repositories, so that a group of remote branches can also be built, and the results tracked.

I don't have a nexus installation running properly yet - that might be part of a larger experiment, but I can certainly see the benefits of the GitFlow workflow, where feature branches / release branches / etc. are the order of the day.

The next part of the project really should be to get the war file for the PsychicWight project up and running, and get some kind of automated / semi-automated deployment going.

But given that this took me just one day to set up, I think that it's certainly a viable approach to consider at the start of a project.

Hopefully there'll be time to revisit this next week.

Building sandcastles

(or: how do you practice at being an architect?)

Figuring out a different way of running a software project is kinda hard. Usually, you come into a project when everything is already set up and running, and you either go along with it, try to tweak it a little bit, or get the hell out. Starting from scratch isn't really one of the options.
Whilst I have opinions on some of the options, I don't really have a lot of experience at the way the different systems interact. I think that the best way of trying to sort this out is probably experience, but a problem with that is that building a large system with many interacting components is a hell of a lot of work.
So what I'm trying out is a small test project, with as many interacting components as I can, as well as a series of hoops to jump through that would usually only be present on a much larger project.

Project Sandcastle:

Going back to my absolute standard project, I'm going to build a dice roller. This time it's going to be a simple web based application, but I'm going to try and compartmentalise it as well.
In terms of infrastructure, I'm aiming for:

• an SCM system (git repository hosted on GitHub)
• a CI system (TeamCity, running locally)
• an artifact storage system (haven't decided yet, probably Nexus)

In terms of projects:

• a dice rolling engine (packaged as a jar file)
• a web application to interface with the engine (packaged as a war file, possibly with a built in server)

It's about commit(ment)s

So, I've managed a few commits to the Github project already then. Not that I have a great deal to write home about, but it's actually nice and easy to get the hang of. And the nice folks there have produced a great little Windows app to help skin over the Git interface: takes a lot of the pain out of standard tasks, though I haven't really had the pleasure (?) of dealing with multiple branches and revisions yet, so I'll reserve judgement till then.

Still in very early days of implementation yet, but there's a very basic dicebag there now, with an API that is (hopefully) easy to follow. Should also be a simple case to check it out, build it as a maven artefact and then chuck it into a project. Honestly, there are loads more updates to come. Honest.

Getting to the party. Late as usual.

So, I'm going to starting experimenting over at GitHub - I've always used SVN / CVS previously, so I've resisted Git for a while: inertia is a hard thing to overcome!

Still, I like to think that I've a relatively open mind, so I gave Git a go, and I've been generally pleasantly surprised that it's not nearly as difficult to work with as I'd feared. Plus, there's GitHub.

So there's going to be a new repository going up there at some point, just so that I can noodle around with it for a while. For those who know me, it won't be a surprise that it's going to be a dice roller (some people write 'Hello World!'; I write dice rollers - don't ask me why, it's a habit.) I'd quite like this one to be a bit more generic and useful, though, so it's also planned to be a standalone jar file that can be included in damned near anything that might need a randomiser at some point.

For those interested, the repository is up at: https://github.com/nihilogist/dicebag

Currently there's very little up there, but hopefully the thing should be filled out sooner rather than later.

If you can't find someone to blame, then blame everybody...

I'm going to start this post by going off on a tangent. I promise I'll lead you back to the point :-)

But first: if you're not interesting in reading this long and occasionally rambling post, then at least go to http://stopsopaireland.com/, have a read, and then think about signing. If you do want to read the post, then I'll give you the link again at the end: it's important.

When I was back in school - and I'm sure this happened to most of the rest of you as well - there was one particular teacher known for being tough on pupils. Any minor infringement saw the offender punished to the full extent of the law. And this was fine: everyone knew where they stood. Until the day came when a paper aeroplane landed on his desk, and the perpetrator was nowhere to be seen.

"Who threw that?" came the cry from the front of the room. "Who threw that?" A stony silence filled the classroom as the entire class clammed up. "If you don't own up, the whole class will be in detention, all week." No one owned up.

We spent the rest of the week in detention, writing out, time after time, "I will not throw paper aeroplanes in class."

And you know, I think that's an important lesson. If you can't find someone to blame, then blame everyone.


OK. The point, then.

By reports, on the 26th January, the Irish government is due to pass legislation that is known as "The Irish SOPA", potentially forcing ISPs to block entire sites from the internet if a copyright holder alleges that the hosting site has content which breaches their copyright. So - in theory - if a website has a single copyright infringement on it then the entire site can be brought to court by the copyright holder and blocked by ISPs. Fantastic: copyright holders no longer need to find out who was to blame for uploading an illegal file, or even who downloaded an illegal file: they just blame everyone, and suddenly everyone is in detention, writing out a million times: I will not upload pirated content to the Internet.

OK, quick notice: uploading material to which you do not own the copyright is wrong. Pirating music is wrong. But that doesn't mean we should shut down YouTube because some people upload pirated music there. (And, you know, the only thing worse that having your music pirated onto YouTube is not having your music pirated onto YouTube...)

You know, this could turn into a really long ranty post, but I'm going to try and keep things brief: it's late, you know... I'll stick to the most concerning parts:

• The method of bringing the bill into law
• The rushed nature of the bill and the vague wording of it
• The legal implications of the bill and the potential for challenges to it

1) The bill is due to be passed by a statutory instrument, meaning that it is not to be debated in the parliament (tjmcintyre.com)
We've just seen online the outpouring of support against this type of law when it was brought up for debate in the United States. Protests saw, amongst other events, Google placing links to the debate on their homepage and Wikipedia going dark for a day. The level of reporting and engagement with this bill at the moment is miniscule: it's only two days before it's due to be passed and there is still (to my knowledge) no specific wording of the bill released, only the fact that it will be passed. It may not have been intended this way, but it feels like legislation passed by stealth, and passed by force, bypassing the usual democratic safeguards.

2) The rushed nature and vague wording
I'm writing this on the 24th January. The earliest reference to this I've found has been the statement release by the chief instigator of the bill, Sean Sherlock TD, on his website at: http://www.labour.ie/press/listing/13245642324974.html. That was released on the 22nd December. So let's say a month. I'd be very happy to know that this had been announced earlier, and that we'd all just missed it, rather than the whole thing being rushed from end to end within a month, but so far, not much evidence. An earlier draft had been around since June last year, but reports indicate that this new bill may go further.
I've been looking around for specifics of the bill, and the best I've found has been the summary on TJ McIntyre's blog, as linked above. That references the early draft, and goes into the implications of the vague wording far better than I can.

3) The legal implications and potential challenges
Other reading I've done suggests that the legal implications of these amendments haven't been properly considered. A reply from ALTO to the Deparment of Jobs, Enterprise and Innovation on the subject of the proposed changes to the copyright act points out some of the potential problems the new law may face; including the fact that previous, similar, acts in other European countries have been overturned by the courts. Likewise, in November last year the European Court of Justice ruled that it is not legal to force ISPs to block specific sites, a rule which should be in force throughout Europe.
What would happen if the ruling were to be challenged by a large organisation? A humiliating backtrack or a confrontation?

Don't take my word for it: go to http://stopsopaireland.com/ and have a read. It's worth it. If you agree with me, sign the petition there. Write to your TD. Do something.

Two lines of code can change your life

I've written about this before (and I'll most likely write about it again), so apologies in advance if I bore people to death with it. It's the issue of education and computing. There have been a few reports recently indicating that computing courses in the UK are failing their students, the worst examples being some of the specialist games development courses: just 12% of graduates having found a job in recent times.

I write code for a living, and I enjoy my job. I'm very lucky to be doing something that I really want to do, and I think that I'm doubly lucky because it's taken me a long and improbable route to get here. And fundamentally, that route started when I was about seven or eight, and I first wrote the lines:

10 PRINT "My name is David"
20 GOTO 10
RUN

Suddenly, my television screen was filled with an endlessly repeating filler of My name is David. I'd managed to get a computer to do something that I had told it to. I was hooked. Over the next few years I would often be found hunched over the beige keyboard of my trusty Acorn Electron, typing in programs and trying to get them to work. By the time I was fourteen, just typing in other people's code wasn't enough, and I began to try and write my own from scratch, starting an epic quest to produce a text adventure game.

I don't know what it was that really spurred me on, whether it was getting to hang around with some seriously talented young developers (I kid you not, these guys had clubbed together and written an email application for the school network while I was still working on drawing a circle on the screen), or that I was very much supported by the school I was in, but despite a long time not working in the IT industry, I came back to college, and picked up Java again. And it was almost like I'd never left.

The thing is, when I was younger, it seemed so much easier to start programming. The languages were so much closer to you - heck, the Electron started up with a BASIC prompt. I can well see someone these days enrolling on a programming course and being slightly taken aback when presented with their first view of an IDE or a command line.

Now, I'm not living in the UK at the moment, so I guess I'm not really qualified to talk about the way that ICT is taught there, and I'm not well enough acquainted with the details of ICT eduction in Ireland, either. But the reports that more and more computing lessons are heading towards teaching basic office skills are a little worrying. It's not that people don't need the ability to use the word processor / spreadsheet / office suite du jour, but it would be great to see other classes available for people who really do want to stretch themselves. There are plenty of resources on the web these days - with many kids it might just take a small push in the right direction, and boom - we have another killer app developer in the running.

It sounds like we might be moving in the right direction at last though, with a few new initiatives kicking around to try and get children interested in the reality of computer science from a younger age, and I really do believe that one of the ways to do that is to show them how easy it is to get started. Show how quickly you can put together a web page. Write a little java program to print stuff out on screen or a simple game. Try and get hold of some of the old computer controlled Lego Technic, or - better yet - the new Mindstorms NXT. Given the choice between working on a spreadsheet and teaching a robot to walk? I know which one I'd choose.

People keep talking about the knowledge economy. We've got to make sure that we create a generation who are going to be up to creating that knowledge, not just using the fruits of it.

Remember, sometimes all it takes to spark an interest is two lines of code. (Though next time, please don't use GOTO. Bad habits and all that ;-) )

Resolutions or revolutions

So, like everyone else in the world (approximately), I made a resolution over New Year's to keep this blog more up to date with exciting new things that I find out over the year.

However, I'm going to be realistic about this one: there are a whole load of other things that really need to be sorted out before I can get on with any projects that might interest people reading a technology and coding blog. It's going to be at least a month before any real work gets done on the great big board game project and --

Look! It's a distraction!

Maybe a photograph of a shark will keep people happy enough while I get round to writing a real post. In the mean time, feel free to not watch this space till the end of January ;-)

Harnessing Hibernate

OK, so I  finally cracked and bought a real textbook on Hibernate, rather than trying to muddle my way through a mass of random online tutorials and Java debugger statements. And, even more excitingly, it came as a Kindle file. Marvellous.*

Harnessing Hibernate is the title, and so far it's been very useful. I wouldn't recommend it to anyone except a near-total Hibernate beginner: you'd want to know a bit of Java and roughly what Hibernate is trying to achieve, and preferably you'll have seen a bit of Hibernate code flying around, just to get your eye in, as it were.

It's a little out of date now: all the examples are built using Ant rather than Maven. But still, I can't really criticise that - I still have regular and ongoing battles with Maven, and the Ant stuff means that it's been dead easy for the authors to introduce small sections at a time, and slowly upgrade from feature to feature over the chapters.

Overall, I'm impressed so far - but we'll reserve the real judgement for when I try to bring Hibernate into my own projects!

* I don't own a Kindle (yet) but the Kindle app for Windows does what it says on the tin. Not too sure about trying to read books on an Android phone but sure, I'll give it a go some day. For now, screen reading is grand. And yet another reason why a second monitor is an essential development tool, not just a nice extra.

Test Driven Development - Review 1

OK, so.

I have my first few classes down using the new test-driven method. It's been written about a thousand times before (at least), so I shan't bore you with the details, but a nice quick summary is here (linky!).

I like it, actually. It hasn't been getting in the way too much, and I've gotten a lot further with the project this time - when I've been writing tests about what to expect from the code before touching the code itself - than the first time I tried it, when I ended up bogged  down in complexity and confusion.

Coverage of the code by the tests is still nice and high (thanks in no small part to EclEmma!) and motivation is also pretty good.

No real demonstrable product to show yet, though... that'll be a while before we've anything that even remotely resembles a game system, and I haven't even looked at an interface yet. Lots of fun to come.

Note to self: should also put together a website.

Progress so far: 1,172 instructions; 1090 instructions covered by tests.

Something... somthing... somthing... test-driven.

OK, so it's been a good long while since the last post to this - which, in turn, was probably a post muttering about how long it had been since the one before.

However, this time, I have a plan. There's a wee idea I've had kicking around for a  little while, and I'm finally going to actually give it a shot. What's more, I'm going to have a stab at putting all this talk of "test driven development" into actual  use.

The basic idea is to write a board game: a wargame simulating starship combat. It's going to try to incorporate actual approximations of vector physics, and generally as much sci fi goodness as I can cram in, before it falls apart under is own weight (I am at least realistic on my chances of finishing this!).

What I'll try to do is keep a running tally of the tests that I've written and gotten to pass, and maybe a log of the difficulties I find on the way. Fun for all the family!

An ode to the Acorn Electron, or: Thoughts on Computing in Education

*beep*

Acorn Electron

BASIC

>

Ahh, nostalgia. I think my folks still have the first computer I ever used somewhere, up in the attic. An Acorn Electron, with a whole thirty-two kilobytes of RAM, and no disk drives - you had to plug in a temperamental tape deck and wait fifteen agonising minutes loading a game before you found out that you'd set something up wrong and you'd have to start all over again. But I loved it. And nearly as much as the games (Citadel! Palace of Magic! Frenzy!), I loved the idea that you could write your own programs for it, too. I spent hours trying to write my own versions of the games that I played, by typing in programs that I found in magazines, and then spending just as many hours trying to figure out which of the semi-colons or quotation marks that I'd typed wrongly.

I sometimes stop to think about how I would have turned out born just a few years later. The Electron, by booting straight into this (admittedly crude) programming environment, really made the point that you were supposed to try and do your own thing with the computer rather than just accepting what other people had written.

I don't know a great deal about the current state of computing in education, but it's something that I should find out more about. I remember that IT lessons were adventures in controlling a turtle onscreen, or (if we were lucky) controlling actual Lego connected to the serial port. It really opened your eyes to the possibilities of computers as tools. I'd like to think that there are some quick and easy programming languages for kids still out there. Whatever happens in the future, we can't just have computing lessons being restricted to learning how to use the office software suite du jour. What a way to kill any enthusiasm.

There's no shortage of resources for programming education. And the thing is, many of them are free: look at Greenfoot, for Java. It hides a lot of the complexity of the language: but that's a great way to start. It's got a graphical interface: that's great.

The problem is, of course, that to teach Java, you need to know some Java in the first place. And how do we get that kind of knowledge imparted? How many teachers are there in schools with any kind of programming knowledge? Not nearly enough. Following on from that, these programs also need to be installed, set up, and maintained. Who's going to take care of that? Much easier to just teach kids how to use the internet, a word processor and a spreadsheet.

Perhaps a special kind of week-long code camp could be run for children with any kind of interest in programming: do a week of classes in the summer holidays covering the very basics - and give pointers on places to look for further information. It might whet someone's interest, and - who knows - might produce the programmers of tomorrow.

Another note to self

Right - it's probably time to take a serious look at normalising the fitness factors of a genetic algorithm; otherwise you end up with absolutely mental fitness values that aren't a great deal of use to anyone. :/

Or I might just need to figure out a different way of picking parents.

So, it works.

After another little while pottering away this afternoon, I've managed to get the code for the big GA into some kind of shape. All the essentials are there, as it were:

- chromosomes and genes

- a basic fitness function

- a basic parent-selection method

- a basic crossover system

- a basic mutation system

I've knocked together a rough initial dataset, and it's generating the sort of results that I'd expect; trending from worse to better. This is good :-)

Now we move onto the actual tricky part, which is working in the various other bits of the requirements, like the quality of service restrictions and so on. First up will be fixing the fitness function to be a bit more standardised.

Step 1: types of QoS attributes

First thing to consider is whether we should have different methods of combining attributes over the service. My initial reaction is "yes", and here's why:

For something like "cost", all we need to do to work out the total cost of a particular set of services is to add up the cost of each of them. Easy. But for something like "Availability", expressed in a percentage, we can't really just add up all the availability scores, can we? That'd produce an oddly skewed result at the end.

eg: we have three services, each of which has a 90% chance of being available. If we add those up, we get an availability score of 270 for the operation. We could divide that by the number of services to get back to the average availability, but that still only takes us to 90% -- in fact, the aggregate availability (the chance that all three services will be available during the invocation) is only 73%.

So, which will serve the algorithm better - the average availability of the services, or the actual computed aggregate availability for the whole set? I'm not sure. Possibly I should test both :-)

Step 2: weighting factors

Also, we should take into account a couple of other factors when calculating the fitness of a chromosome: partly a user-defined weighting (eg: if they would rather the algorithm brought the cost of the service composition down, rather than the execution time), and partly a dynamic weighting to try and meet any particular requirements (eg: if the maximum cost of the service composition is set to £10, and none of the current solutions match that, then the weighting factor assigned to the cost of the service composition should be increased).

What this means is that I'll have to take the fitness calculation of a chromosome out of the chromosome itself, and into the main population object; that way I can store a series of weighting factors that can be changed quickly and easily throughout the execution time.