Building sandcastles part 4: Azure

Continuing the adventures in continuous integration, I'm now looking into how services like AWS or Azure could really help dev teams along. A lot has been written about this before now, but while there seem to be lots of places talking about how easy AWS makes their dev / test / deploy cycle, I haven't seen too many places talking about all the components you might need.

So, looking at it from the ground up, we need, in the simplest terms:

• Somewhere to store our code
• Somewhere to build our code
• Somewhere to test our builds and artifacts
• Somewhere to host our final builds

This translates to:

• Some kind of SCM system
• Some kind of continuous integration system
• A private set of machines to deploy our applications to (let's assume that we're building a web application, so we'll need webserver and a database, minimum)
• A public set of machines to deploy the applications to

After mucking around with the trials of Azure and AWS, I decided to run a few experiments with Azure, and have ended up - so far - with meeting the first two requirements, with virtual machines running GitLab and TeamCity respectively.
I created a rather simple web application - no database yet, and pushed it up to GitLab, and then connected TeamCity over to that to run a couple of builds.

So far, so easy.

The next step - which feels like a pretty big one - is to build a test system. Now, it's very easy to set up a virtual machine, set up a webserver and then have a script to deploy the build artifacts from TeamCity over to it, but in an ideal world - where it's not just me working on it - it would be ncie to be a lot more flexible than that. What I really want to achieve is to be able to deploy a whole test environment with the click of a button.

That means that we need to start looking at the automated provisioning of virtual machines, because the last thing that anyone wants to do is set up a whole environment every time they want a build. Happily, there are a bunch of tools that can help us with this.

For a first experiment, I'm using Vagrant. It seems to be generally straightforward, and - in theory - relatively portable. It's been somewhat of a pain to get working with Azure, but now that it's running, I seem to be able to deploy machines relatively easily. Next steps are to actually get servers running on them.

Building sandcastles, part 3: Scala

(aka: And Now For Something Completely Different)

So, this is less about the continued experimentation in integration a bunch of SCM and CI systems, and more about random experimentation, because sometimes I just roll that way ;)

Pretty much on a whim, I brought Scala into the mix of technologies used. Following the recommendations of a couple of folk I've spoken to about this, I brought it in to the unit testing layer first, so that I can give it a go without too much impact on the actual written code. To be honest, it felt like a bit of a slog (though, having said that, I did manage to get the first unit test written and passing within an hour, after a beer, so I guess it can't be that hard).

First of all, we had to install Scala and the various IntelliJ plugins. That wasn't so bad. Second was to research how to run a JUnit test in Scala. Again, not too bad - suggestions are to use scalatest, and then have your unit tests extend the JUnitSuite class. Writing the unit test? No worries. Well ... not for the real simple ones, anyway. Running the test? Now that was the trick. First of all, you need to make sure that the version of scalatest you are using is compatible with the version of scala that you're using. Then you have to make sure that all the scala setup you did in the IDE also matches the version of scala you installed. Finally, you might need to tweak the build steps in the pom.xml file. Sadly, very few of the errors that get thrown when you have these things wrong make much sense.

For future reference (for myself, but maybe someone else will find this useful as well), this build configuration in maven seems to work quite well:

    <build>
        <plugins>
            <plugin>
                <groupId>org.scala-tools</groupId>
                <artifactId>maven-scala-plugin</artifactId>
                <version>2.15.2</version>
                <executions>
                    <execution>
                        <id>scala-compile</id>
                        <goals>
                            <goal>compile</goal>
                            <goal>testCompile</goal>
                        </goals>
                        <configuration>
                            <args>
                                <arg>-dependencyfile</arg>
                                <arg>${project.build.directory}/.scala_dependencies</arg>
                            </args>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

Something to be aware of is that the -make:transitive configuration argument seems to be redundant as of scala 2.11, and actually breaks the compilation. I've seen it mentioned in a couple of places, but taking it out made everything magically work for me... YMMV.

So anyway, I now have a very very simple test committed, which is currently passing. We'll see how things go as I try to rack up the complexity, and start on mocking out various interfaces.

Building sandcastles part 2

You know, that went pretty well, all things considered.

So, there are a couple of repositories up on GitHub now:
https://github.com/nihilogist/DiceEngine
https://github.com/nihilogist/PsychicWight

I managed to set up TeamCity without too many issues, and have that observing the two repositories.
It's also observing a given set of branches of the two repositories, so that a group of remote branches can also be built, and the results tracked.

I don't have a nexus installation running properly yet - that might be part of a larger experiment, but I can certainly see the benefits of the GitFlow workflow, where feature branches / release branches / etc. are the order of the day.

The next part of the project really should be to get the war file for the PsychicWight project up and running, and get some kind of automated / semi-automated deployment going.

But given that this took me just one day to set up, I think that it's certainly a viable approach to consider at the start of a project.

Hopefully there'll be time to revisit this next week.

Building sandcastles

(or: how do you practice at being an architect?)

Figuring out a different way of running a software project is kinda hard. Usually, you come into a project when everything is already set up and running, and you either go along with it, try to tweak it a little bit, or get the hell out. Starting from scratch isn't really one of the options.
Whilst I have opinions on some of the options, I don't really have a lot of experience at the way the different systems interact. I think that the best way of trying to sort this out is probably experience, but a problem with that is that building a large system with many interacting components is a hell of a lot of work.
So what I'm trying out is a small test project, with as many interacting components as I can, as well as a series of hoops to jump through that would usually only be present on a much larger project.

Project Sandcastle:

Going back to my absolute standard project, I'm going to build a dice roller. This time it's going to be a simple web based application, but I'm going to try and compartmentalise it as well.
In terms of infrastructure, I'm aiming for:

• an SCM system (git repository hosted on GitHub)
• a CI system (TeamCity, running locally)
• an artifact storage system (haven't decided yet, probably Nexus)

In terms of projects:

• a dice rolling engine (packaged as a jar file)
• a web application to interface with the engine (packaged as a war file, possibly with a built in server)

It's about commit(ment)s

So, I've managed a few commits to the Github project already then. Not that I have a great deal to write home about, but it's actually nice and easy to get the hang of. And the nice folks there have produced a great little Windows app to help skin over the Git interface: takes a lot of the pain out of standard tasks, though I haven't really had the pleasure (?) of dealing with multiple branches and revisions yet, so I'll reserve judgement till then.

Still in very early days of implementation yet, but there's a very basic dicebag there now, with an API that is (hopefully) easy to follow. Should also be a simple case to check it out, build it as a maven artefact and then chuck it into a project. Honestly, there are loads more updates to come. Honest.

Getting to the party. Late as usual.

So, I'm going to starting experimenting over at GitHub - I've always used SVN / CVS previously, so I've resisted Git for a while: inertia is a hard thing to overcome!

Still, I like to think that I've a relatively open mind, so I gave Git a go, and I've been generally pleasantly surprised that it's not nearly as difficult to work with as I'd feared. Plus, there's GitHub.

So there's going to be a new repository going up there at some point, just so that I can noodle around with it for a while. For those who know me, it won't be a surprise that it's going to be a dice roller (some people write 'Hello World!'; I write dice rollers - don't ask me why, it's a habit.) I'd quite like this one to be a bit more generic and useful, though, so it's also planned to be a standalone jar file that can be included in damned near anything that might need a randomiser at some point.

For those interested, the repository is up at: https://github.com/nihilogist/dicebag

Currently there's very little up there, but hopefully the thing should be filled out sooner rather than later.

If you can't find someone to blame, then blame everybody...

I'm going to start this post by going off on a tangent. I promise I'll lead you back to the point :-)

But first: if you're not interesting in reading this long and occasionally rambling post, then at least go to http://stopsopaireland.com/, have a read, and then think about signing. If you do want to read the post, then I'll give you the link again at the end: it's important.

When I was back in school - and I'm sure this happened to most of the rest of you as well - there was one particular teacher known for being tough on pupils. Any minor infringement saw the offender punished to the full extent of the law. And this was fine: everyone knew where they stood. Until the day came when a paper aeroplane landed on his desk, and the perpetrator was nowhere to be seen.

"Who threw that?" came the cry from the front of the room. "Who threw that?" A stony silence filled the classroom as the entire class clammed up. "If you don't own up, the whole class will be in detention, all week." No one owned up.

We spent the rest of the week in detention, writing out, time after time, "I will not throw paper aeroplanes in class."

And you know, I think that's an important lesson. If you can't find someone to blame, then blame everyone.


OK. The point, then.

By reports, on the 26th January, the Irish government is due to pass legislation that is known as "The Irish SOPA", potentially forcing ISPs to block entire sites from the internet if a copyright holder alleges that the hosting site has content which breaches their copyright. So - in theory - if a website has a single copyright infringement on it then the entire site can be brought to court by the copyright holder and blocked by ISPs. Fantastic: copyright holders no longer need to find out who was to blame for uploading an illegal file, or even who downloaded an illegal file: they just blame everyone, and suddenly everyone is in detention, writing out a million times: I will not upload pirated content to the Internet.

OK, quick notice: uploading material to which you do not own the copyright is wrong. Pirating music is wrong. But that doesn't mean we should shut down YouTube because some people upload pirated music there. (And, you know, the only thing worse that having your music pirated onto YouTube is not having your music pirated onto YouTube...)

You know, this could turn into a really long ranty post, but I'm going to try and keep things brief: it's late, you know... I'll stick to the most concerning parts:

• The method of bringing the bill into law
• The rushed nature of the bill and the vague wording of it
• The legal implications of the bill and the potential for challenges to it

1) The bill is due to be passed by a statutory instrument, meaning that it is not to be debated in the parliament (tjmcintyre.com)
We've just seen online the outpouring of support against this type of law when it was brought up for debate in the United States. Protests saw, amongst other events, Google placing links to the debate on their homepage and Wikipedia going dark for a day. The level of reporting and engagement with this bill at the moment is miniscule: it's only two days before it's due to be passed and there is still (to my knowledge) no specific wording of the bill released, only the fact that it will be passed. It may not have been intended this way, but it feels like legislation passed by stealth, and passed by force, bypassing the usual democratic safeguards.

2) The rushed nature and vague wording
I'm writing this on the 24th January. The earliest reference to this I've found has been the statement release by the chief instigator of the bill, Sean Sherlock TD, on his website at: http://www.labour.ie/press/listing/13245642324974.html. That was released on the 22nd December. So let's say a month. I'd be very happy to know that this had been announced earlier, and that we'd all just missed it, rather than the whole thing being rushed from end to end within a month, but so far, not much evidence. An earlier draft had been around since June last year, but reports indicate that this new bill may go further.
I've been looking around for specifics of the bill, and the best I've found has been the summary on TJ McIntyre's blog, as linked above. That references the early draft, and goes into the implications of the vague wording far better than I can.

3) The legal implications and potential challenges
Other reading I've done suggests that the legal implications of these amendments haven't been properly considered. A reply from ALTO to the Deparment of Jobs, Enterprise and Innovation on the subject of the proposed changes to the copyright act points out some of the potential problems the new law may face; including the fact that previous, similar, acts in other European countries have been overturned by the courts. Likewise, in November last year the European Court of Justice ruled that it is not legal to force ISPs to block specific sites, a rule which should be in force throughout Europe.
What would happen if the ruling were to be challenged by a large organisation? A humiliating backtrack or a confrontation?

Don't take my word for it: go to http://stopsopaireland.com/ and have a read. It's worth it. If you agree with me, sign the petition there. Write to your TD. Do something.