Automated deployments: What are the components of a system?

I've been thinking a lot recently about software deployments, particularly in my current project, which is a (relatively) standard Java EE application: application server, database, messaging framework, blah blah blah. As I'm sure a lot of projects do, we have a hotch-potch of assorted build scripts, deployment scripts, semi-automated configuration scripts.

Two or three members of the project team try to manage deployments to test systems, and have currently managed to write several thousand lines of code which act as a universal way of deploying the artifacts to the various environments which ... doesn't work. These issues seem to crop up on every project - certainly every developer I've worked with recounts stories of projects where weeks of time were lost trying to figure out why code that apparently worked perfectly would not even deploy to a target system.

The upshot of this was that I started trying to think about the process in a more modular fashion; to break it down into smaller components and see if there's a standard pattern that we could apply. And I think I've come up with a few things - some of which are obvious, but we may as well start with the obvious and see where it leads us.

The System
To create this mythical working system we need three things:

• Environment
• Build artifacts
• Deployment processes

Environment: the environment is the actual machines (either physical or virtual) that the software will run on. It includes all third party additions required to run the software, such as databases, web / application servers, messaging systems and so on. It also includes all necessary network configurations such as load balancers, firewalls and host configurations.

Build artifacts: the build artifacts are the final output of building the source code ready for deployment. They may be tailored to fit a specific environment by setting runtime variables, but other than that the source code itself should not need to know about environment-specific details.

Deployment processes: the deployment processes are the description of steps needed to take the build artifacts, transfer them to the relevant environments, and initialise them such that the system is available for use.

These three aspects of the system can - and should - be treated separately when it comes to automating the overall process of creating and deploying environments. For instance, we should not be trying to write code in our build artifacts that generates application servers that are then deployed alongside the application files; they are part and parcel of the environment, and that should be as static as possible for the lifetime of the system. Also, the deployment process should not be part of building the artifacts from the source code: it should be possible to build individual artifacts and deploy them as necessary.

All these principles stem from the idea of the separation of concerns. Our build process should not interfere with a deployment; our environments should not define the way our software is built from source code.

Building sandcastles, part 3: Scala

(aka: And Now For Something Completely Different)

So, this is less about the continued experimentation in integration a bunch of SCM and CI systems, and more about random experimentation, because sometimes I just roll that way ;)

Pretty much on a whim, I brought Scala into the mix of technologies used. Following the recommendations of a couple of folk I've spoken to about this, I brought it in to the unit testing layer first, so that I can give it a go without too much impact on the actual written code. To be honest, it felt like a bit of a slog (though, having said that, I did manage to get the first unit test written and passing within an hour, after a beer, so I guess it can't be that hard).

First of all, we had to install Scala and the various IntelliJ plugins. That wasn't so bad. Second was to research how to run a JUnit test in Scala. Again, not too bad - suggestions are to use scalatest, and then have your unit tests extend the JUnitSuite class. Writing the unit test? No worries. Well ... not for the real simple ones, anyway. Running the test? Now that was the trick. First of all, you need to make sure that the version of scalatest you are using is compatible with the version of scala that you're using. Then you have to make sure that all the scala setup you did in the IDE also matches the version of scala you installed. Finally, you might need to tweak the build steps in the pom.xml file. Sadly, very few of the errors that get thrown when you have these things wrong make much sense.

For future reference (for myself, but maybe someone else will find this useful as well), this build configuration in maven seems to work quite well:

    <build>
        <plugins>
            <plugin>
                <groupId>org.scala-tools</groupId>
                <artifactId>maven-scala-plugin</artifactId>
                <version>2.15.2</version>
                <executions>
                    <execution>
                        <id>scala-compile</id>
                        <goals>
                            <goal>compile</goal>
                            <goal>testCompile</goal>
                        </goals>
                        <configuration>
                            <args>
                                <arg>-dependencyfile</arg>
                                <arg>${project.build.directory}/.scala_dependencies</arg>
                            </args>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

Something to be aware of is that the -make:transitive configuration argument seems to be redundant as of scala 2.11, and actually breaks the compilation. I've seen it mentioned in a couple of places, but taking it out made everything magically work for me... YMMV.

So anyway, I now have a very very simple test committed, which is currently passing. We'll see how things go as I try to rack up the complexity, and start on mocking out various interfaces.

Building sandcastles part 2

You know, that went pretty well, all things considered.

So, there are a couple of repositories up on GitHub now:
https://github.com/nihilogist/DiceEngine
https://github.com/nihilogist/PsychicWight

I managed to set up TeamCity without too many issues, and have that observing the two repositories.
It's also observing a given set of branches of the two repositories, so that a group of remote branches can also be built, and the results tracked.

I don't have a nexus installation running properly yet - that might be part of a larger experiment, but I can certainly see the benefits of the GitFlow workflow, where feature branches / release branches / etc. are the order of the day.

The next part of the project really should be to get the war file for the PsychicWight project up and running, and get some kind of automated / semi-automated deployment going.

But given that this took me just one day to set up, I think that it's certainly a viable approach to consider at the start of a project.

Hopefully there'll be time to revisit this next week.

Building sandcastles

(or: how do you practice at being an architect?)

Figuring out a different way of running a software project is kinda hard. Usually, you come into a project when everything is already set up and running, and you either go along with it, try to tweak it a little bit, or get the hell out. Starting from scratch isn't really one of the options.
Whilst I have opinions on some of the options, I don't really have a lot of experience at the way the different systems interact. I think that the best way of trying to sort this out is probably experience, but a problem with that is that building a large system with many interacting components is a hell of a lot of work.
So what I'm trying out is a small test project, with as many interacting components as I can, as well as a series of hoops to jump through that would usually only be present on a much larger project.

Project Sandcastle:

Going back to my absolute standard project, I'm going to build a dice roller. This time it's going to be a simple web based application, but I'm going to try and compartmentalise it as well.
In terms of infrastructure, I'm aiming for:

• an SCM system (git repository hosted on GitHub)
• a CI system (TeamCity, running locally)
• an artifact storage system (haven't decided yet, probably Nexus)

In terms of projects:

• a dice rolling engine (packaged as a jar file)
• a web application to interface with the engine (packaged as a war file, possibly with a built in server)

It's about commit(ment)s

So, I've managed a few commits to the Github project already then. Not that I have a great deal to write home about, but it's actually nice and easy to get the hang of. And the nice folks there have produced a great little Windows app to help skin over the Git interface: takes a lot of the pain out of standard tasks, though I haven't really had the pleasure (?) of dealing with multiple branches and revisions yet, so I'll reserve judgement till then.

Still in very early days of implementation yet, but there's a very basic dicebag there now, with an API that is (hopefully) easy to follow. Should also be a simple case to check it out, build it as a maven artefact and then chuck it into a project. Honestly, there are loads more updates to come. Honest.

Harnessing Hibernate

OK, so I  finally cracked and bought a real textbook on Hibernate, rather than trying to muddle my way through a mass of random online tutorials and Java debugger statements. And, even more excitingly, it came as a Kindle file. Marvellous.*

Harnessing Hibernate is the title, and so far it's been very useful. I wouldn't recommend it to anyone except a near-total Hibernate beginner: you'd want to know a bit of Java and roughly what Hibernate is trying to achieve, and preferably you'll have seen a bit of Hibernate code flying around, just to get your eye in, as it were.

It's a little out of date now: all the examples are built using Ant rather than Maven. But still, I can't really criticise that - I still have regular and ongoing battles with Maven, and the Ant stuff means that it's been dead easy for the authors to introduce small sections at a time, and slowly upgrade from feature to feature over the chapters.

Overall, I'm impressed so far - but we'll reserve the real judgement for when I try to bring Hibernate into my own projects!

* I don't own a Kindle (yet) but the Kindle app for Windows does what it says on the tin. Not too sure about trying to read books on an Android phone but sure, I'll give it a go some day. For now, screen reading is grand. And yet another reason why a second monitor is an essential development tool, not just a nice extra.

Test Driven Development - Review 1

OK, so.

I have my first few classes down using the new test-driven method. It's been written about a thousand times before (at least), so I shan't bore you with the details, but a nice quick summary is here (linky!).

I like it, actually. It hasn't been getting in the way too much, and I've gotten a lot further with the project this time - when I've been writing tests about what to expect from the code before touching the code itself - than the first time I tried it, when I ended up bogged  down in complexity and confusion.

Coverage of the code by the tests is still nice and high (thanks in no small part to EclEmma!) and motivation is also pretty good.

No real demonstrable product to show yet, though... that'll be a while before we've anything that even remotely resembles a game system, and I haven't even looked at an interface yet. Lots of fun to come.

Note to self: should also put together a website.

Progress so far: 1,172 instructions; 1090 instructions covered by tests.

Platformer blues

"I've got a great idea for an app - but what phones should I write it for?"

It's an interesting question and - as it turns out - the answer is harder than I thought. Now, in an ideal world, you'd just write and release it for all the major smartphone OSs - Symbian, iOS, Android and RiM. But that's quite a lot of effort for just one person! So how do you maximise your returns or exposure for minimal effort?

First of all, it's probably an idea to think about what language you're comfortable working in, and what operating systems that the language plays well with.

• Nokia -- a whole variety of languages supported, so you'll pretty much have to pick the one you want and grab the relevant package for it.
• RiM Blackberry -- Java
• Android -- Java
• iOS -- one of the C languages. You'll also need (yes, really - it's mandatory) a Mac.

Next up, let's look at the different platform distributions (data from Gartner).

• Symbian: 41%
• RiM: 18%
• Android: 17%
• iOS: 14%

It might be worth noting, though, that the market share for Android jumped from 1% in 2009 to 17% in 2010 - obviously this might not be indicative of future trends, but Google have put a lot of effort into getting it onto as many handsets as possible. Also of note is that Oracle are currently taking legal action against Google for unlicensed use of the Java platform - though the future of that action is somewhat uncertain.

Lastly, we'll need to have a look at the different distribution systems for the different operating systems. (This is just a summary, though - there are multiple delivery platforms for most operating systems, so you don't have to follow the well-trodden path - it might just be a lot easier! Also, having not personally explored them inch by inch, there may still be pitfalls to discover...)

• iOS: Sign up for an AppStore developer account - this is $99 per year, and lets you distribute your apps through the AppStore, after they've been checked out by Apple.
• Android: Sign up as an Android Market developer for $25 (one-off fee), and this lets you distribute your apps through the Android Marketplace. However, there are caveats here based on countries - the list of countries where you can charge for your apps is currently quite small.
• Nokia's Ovi store: Sign up for $50 to register yourself as a publisher - this lets you upload content for distribution via the Ovi store.
• RiM's Blackberry AppWorld: I can't find a price for signing up as a developer, but again, there's a central hub for app distribution.

Most seem to offer a standard 30% distribution charge.

So, where does this leave us? Personally, I'm going to carry on working with Android for now - I think that it has a good future ahead of it, and I'm far more comfortable working in Java than with C. At some point, I'd like to give the iOS development thing a go, but it would mean buying another computer, which would be a little annoying.

Android app: details

Seventh Sea dice-rolling application
This is something that I've been working on for a little while. The dice rolls in the role-playing game Seventh Sea are not the most intuitive or easily calculated sometimes, particularly later in the game when you can be sorting through handfuls of dice, frantically adding some - but not all - of them together to get a total. This isn't exactly adding to the fast and frenetic nature of the genre. Therefore, I've been working on some kind of application to help speed this up.

History
It started out as a purely Swing based Java application, and since developed into a Java applet for ease of access. Finally, it has evolved into an application for Android platform phones, purely as a development challenge...
The development has taught me a good deal about developing for Android - and mainly that it's actually not that hard (particularly for something as simple as this)! There are plenty of good tools available, and the UI creation tools are a breeze - certainly much easier to get into than Swing.

UI
The UI is very simple for this app: there is a main menu screen with a few options available. The only one of any real practicality is the Roller screen, which contains the nuts and bolts of the application. There are two keypads, one of which selects the number of dice that you want to roll, and the other selects the number of dice that you want to keep. There are also buttons available to choose whether the dice "explode" or if and Drama Dice are added to the roll.

• Splash screen
• Menu screen
• Roller screen: Portrait 1 -- Portrait 2 -- Portrait 3
• Roller screen: Landscape 1 -- Landscape 2

The layout files are all quite simple XML files, detailing the page elements available to each page of the app. Each layout file comes in two "flavours", one for the portrait orientation of the phone, and one for the landscape. On a change of handset orientation, the phone switches between the layouts. (This does, however, call the onCreate() method of the layout again, resetting any options that have been chosen.)

Problems
There haven't really been many problems in development at all. The most challenging part was to develop the object-oriented approach to the dice-rolling, as there are some slightly awkward options and corner-cases to be handled.

Future tweaks
There are still some tweaks that I'd like to make to this app, though.
Firstly, it would be nice if changing the orientation of the phone didn't cause the app to forget any dice rolls that it had just made - this, I assume would be handled by a new implementation of the onCreate() or onResume() methods of the particular activity.
Secondly, it'd be nice to check out the RNG for the engine as a whole. This probably isn't essential, as I'm using the Random class, which should be more than enough of a random generator - each single die is given a 48-bit seed based on the system time when it's created - but it might be an interesting experiment to run some tests on how random it actually is.
Lastly, it'd be nice to add some extra touches from the mobile interface itself - perhaps some (optional!) sound effects. On the other hand, they are rather gimmicky, and would probably become extremely irritating rather quickly. (On the other hand, it would be a good reason to start working with storing user settings, so that's a good reason to implement them by itself!)

An applet is born

Having found a pretty useful little article on the deployment of applets the other day, I promoted by 7th Sea dice roller up the priority ratings as an experiment to see what they can do. So there's now an unlinked page on my regular site which I'll be using as a sandbox (here) to test them. The current applet is a little crude, but serves to demonstrate that it works.

Next steps

The next steps will be to implement the option to change the number of dice that are kept on a roll, and to set up the other options mentioned on the dice roller development page.

(The article is here, and I found it very useful, even if it is a little old!)