Monday, 9 May 2016

Moved blog platforms and a new post

I've moved blog platforms.  I'm now using Markdown, Jekyll and GitHub pages.  This blog is now here.  There is a new entry about using Protobufs With Go

So long Blogger.

Thursday, 26 March 2015

Logging from Go

I recently wrote about some of the features of Go that I really like for getting code into production.  After my initial excitement there were a couple of things I had to check before being certain Go would be a winner.  The first was logging.  Can I easily get log messages from my app to where I can see them?  For us that means the excellent Logentries service.

I've written before about reconfiguring Java syslogging in Mule to send to Logentries.  It was hard won ground and not an experience I would like to repeat.  Logging from Java is a mess.  Throw in multi line strack traces caused by exceptions "bubbling up the stack" and getting a readable message into a syslog server is suddenly an exercise in frustration.  There is an excellent blog about this from ZeroTurnaround - The State of Logging in Java 2013.  An unsurprising finding is that 87% of respondents to a questionnaire have no real-time way of seeing logs from their applications in production.  In 2013 there was already a confusing array of Java logging frameworks and facades. Figuring out how to make them play nicely with your application, your dev, test, and prod environments, and your syslog server is a problem that quickly goes into the basket marked Too Hard.  By 2015 things in Java land have not got simpler.  If you're in that 87% of developers I sympathize - there is little to make logging from your application as easy as it should be.

It turns out logging from Go is so easy I was left wondering how it ever got so hard in that other language.  There are two core Go packages.

  • log - a simple logging package.
  • log/syslog - an interface to the system log service.

Check the Go example for using the log package.  It writes to stderr, if you're a Twelve-Factor App acolyte then you are nearly done (you will need to switch stderr to stdout).  We like to go a step further and have an application in production send it's log messages straight to Logentries.

This is easy to achieve using the log and log/syslog packages using TCP or UDP:

The Logentries docs suggest the use of TLS for untrusted networks.  When we're running code in The Cloud and sending log messages to somewhere else in The Cloud, that means all networks.  By using the crypto packages (also in the Go core) this is easy to achieve.  It's made even easier by the Go code being open source - I can largely copy the experts by reading the syslog code.

log/logentries is a Go package I wrote that makes it easy to reconfigure log to send messages to Logentries.  Before you jump in it's worth understanding my requirements:

  • I'm usually most interested in log messages during app start up (when configuration errors tend to show up).
  • Once an app is up and running we use metrics (not log messages) to track application performance.
  • I don't ever want an app to block on logging.
  • If Logentries is not available for a some time I'm happy to log to stderr and then manually retrieve the logs later if I really need them.  An alternative would be to store and forward.
  • A logger is for logging and a debugger is for, well, debugging.  If you have to use logging for debugging then DELETE those calls before you commit.
  • I want to deploy applications without having to set up a syslog server as well.  Syslog servers and their configuration are an arcane art best left to a specialist

If you need more features, the code is open source.  I hope it makes a useful starting point for you.

Sending our log message to Logentries from Go is now a simple case of calling one method during init.  The rest of the application code is unchanged.  During development we set an empty Logentries token.  This causes the Init method to no-op and the app continues to log only to stderr.  Here's an example app, using the package, that will log to Logentries every five seconds.  Create your own Logentries token and try it out.

This is how easy logging should be.  It's important, I need it, but it shouldn't be a battle.  Go makes it easy to log what I need and spend my energy focusing on the business problem.

Sunday, 8 March 2015


Last year I was trying out some of the cool features in Postgres and Postgis for generating JSON and GeoJSON in the database.  I wanted to try these features for making a web service and I decided to try a new programming language for the task. 

Here's a teaser of the conclusion - more open data from GeoNet.

The image shows a slow slip event or silent earthquake recorded at the Gisborne GPS site.

Going Go

I had already run through the Tour of Go.  A lot has been written about Go syntax and language features.  I liked what I saw well enough to do some investigation where the hard work often is - putting code into production. 

The Web Server

We've been embedding Jetty in our Java apps and deploying them via RPMs for a long time.  I wanted to write a web service in Go and assumed I would need some sort of additional server application to deploy a Go app, or at the very least something to run in front of it.  Nope!  First mistake.  There is a fully functioning web server in the core libs.


So I'm bound to need some kind of runtime on production, right?  Nope!  Go apps compile to a single binary. 

Deploying a Go web application looks like this: Drop binary on system. Start it up. Job Done.

Dependencies and Compilation

There are a lot of powerful features in the core Go packages but external dependencies are inevitable at some point.  I expected some binary package management tool.  Dependency repos in the clouds.  Broken meta data.  Version skew in production and general gnashing of teeth.  Nope, nope, and nope!  The Go compiler is so fast you compile all your code and its dependencies from source in single digit seconds or less.  It's straightforward to get or vendor drop dependencies for your code. 

Given the C-like syntax, surely I was going to have to write a Makefile?  Noooope.  If you organise your code as suggested the go tool can build it for you with no further help.

Sold on Go

By this point I was hooked.  No Russain Doll JVM+Container+App deployment.  No dependency hell.  No compile and package time taking so long that I could forget what I was doing while I waited.  It is so easy to compile and deploy Go code.  It looks like someone really cares about making our jobs easier.

A Real Project

I turned the web services experiment into a real project - FITS (Field Time Series) - storing low data rate field data collected for the GeoNet project.  FITS code is available on Github.

The fledgling first version of the FITS api is available to use now.

Here's those JSON and GeoJSON functions in action. They return site information as GeoJSON.

Visualizing the data easily from FITS is important.  I used the excellent svgo package.  SVG browser support is now very good and an SVG image handles well in a responsive web page.  The image at the top of this post is SVG straight from the FITS api.

Deployment - Docker

Having gone trendy with the code, I went full trendy with the deployment.  It's deployed in a Docker container.  The Docker build, including compiling the Go code in the container, is done in  The build is triggered from Github commits.  The FITS Quay repo is available.  Getting code to production this way is embarrassingly easy.  Go and containers work together like two things that go really well together... for example, a glass of whisky and another glass of whisky.

FITS runs in production in Amazon Web Services (AWS) in the Sydney region.  FITS uses AWS Elastic Beanstalk and AWS Relational Database Service running Postgres+Postgis for the database. 

Going Forward

I've done a few projects in Go now.  I still really like it.  Go makes building and deploying code as easy as it should be.  Most importantly to me - I'm really enjoying programming like I javan't done for quite a while.

Tuesday, 13 May 2014

Logging - What's Your App Doing Right Now?

Knowing how your application is performing in production should be really important to you.  I've written before about gathering application metrics.  This post is about gathering log messages.  I've started writing this post twice before and it quickly turned into an unhelpful rant about the state of Java logging.  Third time's a charm for sticking to the useful stuff without throwing my toys.

There is a list of what I want out of logging at the end.  This may help you come up with a list of your own and also understand some of the decisions I've made.

At the moment I'm focused on logging from Mule ESB applications.  This means I'm using log4j 1.x.  I'm guessing you are as well.    I want per application log files and the short host name to appear in the logs.  I don't want to repackage the application as I move it through dev, test, and prod.

I'm using for my log collector as they provide many of the features on my list.  Importantly, their Java collector is really nice and open source.  If nothing else use that so you get multline logging - no more mangled stack traces in your logs.  As an added bonus their pricing is really nice. There are other SAAS options out there, or you could roll your own infrastructure with something like  For me this is not worth the time or overhead and Logentries provide a far nicer service than I could ever set up in the time I have available.

So now to logging from a Mule ESB application.  There are a couple of ways to shave this yak.  The main thing I've had to work around is that log4j 1.x can include system properties in the logging setup but can't read environment variables.  The approach I've taken is:
  1. Write Spring config to reconfigure logging inside a Mule application.
  2. Provide a Logentries token and other properties to the application via properties files.
  3. Use a Spring profile to enable the logging in production.

1. The Spring Config

This is reusable between Mule applications so I pulled it out into its own library.  It's available on Github and our public Maven repo.  Looking at the config it does the following;
  • Provides a bean to look up the host name.
  • Provides a log4j pattern.
  • Removes any existing log4j appenders.
  • Adds an appender to send to Logentries.
  • Adds a appender per application rolling file system log  to mimic the one that Mule usually provides.
Before you load this config your application needs to have the following properties defined:
  • le.token - a valid token for
  • - the name of the Mule application.
  • app.version - the version of the Mule application.
A shout out to Harry Lime for his answer on Stack Overflow that got me started on reconfiguring log4j with Spring.

2. Providing the Properties


We use the same approach with all our applications.  We bundle default and empty properties with the application and then load an option set of overrides from the file system.  The override file is provided by Puppet in production and contains, amongst other things, the correct Logentries token for the environment. and app.version

We build our Mule applications using our own Gradle plugin which writes a file into the application zip for us.  The application version comes from our Gradle build version plugin.  If you don't want to take the same approach then you could write them into your mule-config as global properties.

Loading these properties then looks like this.  The load order is important to ensure that the file from /etc/sysconfig overrides the one bundled in the Mule app.  Also setting ignore-unresolvable="true" is important as we don't have the same properties in all our files.

See the MuleSoft Blog for a different approach to providing application properties.

3. Spring Profile

Once the properties are set all that's left is to import the logging configuration.  We do this in a Spring profile which is enabled in production.


Here's what some log messages look like at Logentries.  Awesome stuff for two dependencies and a few lines of XML.  The log messages are from our testing version of the back end services for a new version of the mobile quake applications that we are working on.  Multiple location options and faster notifications coming soon to a phone near you.

Appendix 1 - Enabling the Spring Profile and Logs from Mule

I also want to see the logs from the Mule server itself at Logentries as well as be able to turn on the Spring profile in production.  I want multi line logging so I'm going to use the logentries-appender again.  Here's one way to do this.  It requires repackaging your Mule server and adding some files on disk in production.  The goal is to pass some extra System properties to the Mule JVM.

In the Mule distribution:

  • Add the logentries-appender.jar to lib/boot
  • Add an appender to conf/  This refers to System properties that we will set in a moment:

  • In the startup script bin/mule (adjust for your distro or O/S) set some additional environment variables:

  • Where /etc/sysconfig/mule contains:

  • Finally, edit conf/wrapper.conf so that the environment variables are passed to the Mule JVM as System properties.  Remember to increment the properties appropriately for your config file.

You can now repackage the Mule server for however you deploy it.

To enable logging from the Mule server to logentries I need to provide a file /etc/sysconfig/logentries.token the only contents of which are a valid logentries token.  If you don't want the Mule server to log to logentries then remove this file or leave it empty (in which case the logentries-appender will throw debug errors).  Alternatively, provide all environments with a separate logging token.

To set the Spring profile add the profile(s) required to the file /etc/sysconfig/spring.profile (e.g. 'production') and restart the Mule server.

Appendix 2 - What I Want From Logging

  • log messages in one place.
  • search.
  • alerting.
  • developers involved in logging from their apps, not sysadmins having to bolt it on for production.
  • multi line logging - once you've had this for Java stack traces it's hard to go back to syslog mangled error messages.
  • a log message is a line of text - don't force pseudo formats on me.
  • log direct from an application (no faffing with syslog please).
  • don't block the application while logging.
  • don't lose messages if possible.
  • logging that plays well with running in the cloud or on saas.
  • log to the local file system as well in case the central collector goes away.
  • a log per application.
  • search by host occasionally.
  • easy reconfiguration between dev, test, and prod.
  • no requirement to repackage my app for each environment. 
  • some nice graphs would be good.
  • log everything that maybe useful.
  • the option to archive logs for longer periods of time.
  • security.
  • application deployment markers.
  • fast ingestion of log messages into the collector.
  • someone else to run the collector for me.
  • beatings for people that use logging when they should use a debugger (and vis versa).
  • The Moon on a stick.
This list is pretty long already.  Fortunately I don't also have to deal with:
  • regulatory requirements for logging.
  • private data in the log messages. 

Appendix 3 - A Challenge

If this is like any other blog written about logging from Java then people reading this will doubtless mention a different framework or facade that I can throw in here to make my life magically easier.   Logging in Java is already confusing enough.  Simply name checking another approach does not improve the problem.  I need to get a small amount of text onto the network and into a collector.  I shouldn't have to perform multilayer classpath and configuration surgery to achieve that.  If you must suggest alternatives then please do it with concrete examples on how your suggestion actually improves logging from Java in my situation.  I doubt you will fit convincing, tested, and proven examples into blog comments.  

Better yet - help get log4j2 over the line to a production release and into use on some popular projects.  It has some really nice looking features that I think actually will make logging from Java simpler.  We've tested it in dev and like what we saw.  I expect it will be used our next non-Mule project.


Monday, 10 March 2014

Metrics - What's Your App Doing Right Now?

It's been a long time between updates, work, busy, mumble mumble mumble.

In a minute I'm going to talk about monitoring Java and that's enough reason to stop reading right now.  Fair enough.  But before you go:

  • How is your application performing in production right now?  
  • Are you meeting demand?  
  • If you add more load to a server will it cause application problems? 
  • Do you know why things fail when they do?  

Developers - you are not excused, DevOps is the new now.  Code that is not in production and performing well is worthless.  If instrumenting your code to collect metrics about it's performance in production is not already part of your regular work it soon will be.  If Java's not your thing but you get a bit mumble mumble on the topic of how your code performs in production then scan the pictures below and then head on over to Librato Metrics or Hosted Graphite and get busy.  They don't care what metrics you're collecting they just make it so easy that you have no excuses not get on with it right now.

Here's a picture.  It's performance metrics for earthquake messages going through the Mule ESB from the SeisComP3 earthquake location system to the web.  There is also a heart beat message tracked so that we know everything is still working between earthquakes.  There is a lot of information here.  Not least of which is it takes about 10 times longer to insert messages into the a database (akapp01) than it does to read them from the file system (akeqp01).  This is the sort of information that makes targeting performance improvement, trouble shooting, or capacity planning easy.  Without this sort of information you're left guessing about what to do when something is not right with your application.  Guessing only rarely leads to success.

The picture above is a Librato Metrics dashboard.  Here's one of similar information sent to Hosted Graphite.  

Librato Metrics and Hosted Graphite both have strengths and weaknesses.  Try both and see which suits your needs the best.  One of the biggest differences, if you need it, is the ability to alert on your metrics.

So to the Java bit.  Monitoring the JVM and processes running in it usually involves using JMX.  Accessing information via JMX is easy and secure sucks. Running in the cloud, with servers coming and going at a moments notice, makes this problem worse.  The obvious answer is to turn the metric gathering problem around and have a JVM agent push metrics to you.  There are services available that do this.  There is the awesome New Relic and others like it.  However, they all come at a cost.  Enough of a cost that I ended up rolling our own agent for collecting JVM metrics from Mule, Jetty, and Tomcat.

I didn't have to do much.  Librato Metrics is a great data store with fantastic visualisations for what I want.  Getting data out of JMX is the only hard work and fortunately the Jolokia project removes all the pain by providing an HTTP-JMX bridge.  I started off with a Jolokia agent being queried with a Perl script.  Once I was happy with the moving parts I've written agents the wrap Jolokia to run as an application in Mule or a Servlet container and no external script is needed.  The applications themselves periodically gather and send metrics to Librato or Hosted Graphite.  Getting metrics is as simple as adding some config properties to a server and dropping an app into Mule, Jetty, or Tomcat.  Gathering metrics is deploying an application - no fire wall changes, no adding servers to a remote collector process, very little pain at all.  

Here's some Jetty metrics for Jetty and the JVM it's running in under test load.  It looks to me that with a little bit of tuning I could handle a lot more requests with this server.

These are the projects, they are open source on Githib:
If Librato or Graphite are not your thing then Mule Metrics or App Server Metrics should be pretty easy to extend (implement one method).  There is a lot more that could be done with Jolokia and JMX beyond using it to extract metrics.  As we find places where we need more detail about an application then I think we will start to look seriously at using Coda Hale's Metrics as we need it.  We're also getting far more sophisticated about how we use logging but that's a different topic.  For now, if you haven't got it, then work on getting some insight into your applications in production and avoid having to mumble mumble mumble when problems arise.

Wednesday, 21 August 2013

The Java Hammer and Learning Functional Programming

Ribbing other developers about their programming language of choice is all part of the culture.  I nearly succumbed myself today.

The obvious joke being surely if you debug PHP you just end up with Perl6.  Usually just a bit of a larf, but often dismissive humor is a quick way to avoid having to understand and learn something new.  Occasionally, as in the case of the PHP Hammer, things can go perhaps a little to far.

This is my hammer:

I made it when I used to big wall climb a lot.  Where I learnt to climb you couldn't walk into the local climbing store and buy a hammer so I made one (along with a lot of my other gear).  I bought a geologist's hammer, cut part of the pick off, ground in a notch for pulling pitons and drilled a hole for attaching a funkness for ripping out stubborn pitons and busted bolts.  It worked really well and I helped a couple of friends make their own hammers.

I don't big wall climb any more but I hung onto the hammer.  I still use it when I work around the house or garden and it works just fine to whack in nails.  The handle is a bit longer and the head is heavier than a typical framing hammer so you can really punch a nail home with it.  Of course because it's heavier my feeble geek arms get tired quicker than they might with a smaller hammer.  If I help a friend out on a project they always take the time to make fun of my hammer.  However, it's when I need to pull a nail out that the real problems start.  I can't just flip this hammer over and use the claws on the back, I have to resort to this tool:

Which is fine until the nails get large and then I have to break out Fat Max:

Somehow I've ended up with three tools to do the job of one.

I could go get a new hammer that's more appropriate for the job, but when I just need to put in a few nails I never bother making the trip to the hardware store.  I'd have to wade through shiny nail guns and battery powered impact drivers just to get one simple tool to do a straight forward job, so I stick with what I've got.

I realised today that this is exactly how I feel about Java.  This is the Java Hammer; not always the right tool but I reach for it because it's familiar.

Like most people who have been programming for a while I'm truly proficient (be honest) in a small number of languages, can muddle my way through in a few more, and I've written hello world in more languages than I can remember.  I haven't properly learnt a new language every year because it often ends up being relearning object oriented programming with different syntax.

Lately I've become interested in functional programming, particularly Scala.  I can see that it will suit a lot of the programming problems that I have.  What really interests me is the paradigm shift to functional programming.  The trouble is, it's hard to break the habit of the familiar especially when there is work to be done.  I've been to presentations and tutorials about Scala and functional programming and the most useful for me was this one by Dianne Marsh.  Dianne's talk helped steer me off the inevitable path of being a functional fascist: behaviour that would make adoption at work difficult.  Instead, she suggested ways to start using Scala in daily coding, beginning with non production code like tests and simple scripts and building out from there.  Also, Dianne encouraged me to start out using Scala in an object oriented way, and to worry about writing functional code later. Sound advice for making a daunting change!

When the world stops shaking or erupting and I have the energy to look at a computer in the evenings, I'm going to finish working my way through Atomic Scala by Dianne Marsh and Bruce Eckel.  It's really great and I'm pretty sure that if I stick at it that one day I will be able to look at some code and realise that I can write in a truly functional style.  It will be a nice break from the Java Hammer.

Footnote: This blog is about me learning something new that suits the programming problems I have.  Don't bother flaming, trolling, extolling the virtues of your favorite language, or any of that other pointless behavior.  I won't be listening.  Put your energy into learning something new yourself.

Disclaimer: I have never coded in PHP and have no opinions about it.  I do notice that it's used for a lot of things on the web so it must be good for getting stuff done.

Tuesday, 30 July 2013

Notifying You

The GeoNet Project provides a range of ways that you can stay up to date with geohazards information for New Zealand.  This includes smart phone applications that provide near real time earthquake push notifications for both Android and iOS devices.  I've worked with the internet for a long time but some days it still amazes me - receiving an earthquake notification on a phone in my pocket within moments of an earthquake occurring is one of those times.  It's both disarmingly simple and complex at the same time.  Here's how we get the earthquake location to the phone in your hand.

Getting push notifications to your phone involves services in several countries.  The numbered steps are described below.

1. The Application on Your Phone

To receive near real time push notifications you have to have the GeoNet app for Android or iOS installed on your phone, it has to be connected to the internet, and you have to have your quake preferences set appropriately.

2. Pushing to the Phone

We don't connect to your phone directly to send the push notifications, Apple or Google do this for us via either the Apple Push Notification Service or Google Cloud Messaging. When it's on the internet (and you've given it permission) your phone maintains a connection to the Apple or Google servers and receives push notification messages over that connection.    When your phone isn't connected those servers store messages to send to your phone next time it connects to the internet.  Using these services also allow the application on the phone to receive messages even when it's in the background.

We use Urban Airship to send push notifications to the Apple and Google messaging services.  Urban Airship give us a single API to send messages to and do all the hard work of maintaining the connection to the Apple and Google service as well as staying up to date with any changes to those services.  Urban Airship also provide some handy reporting tools for us.

3. Managing Your Preferences

We run a back end application to store your quake notification preferences.  When a quake message is received the preferences are used to decide which phones should be notified about this quake and a push message is sent to Urban Airship.  We run the back end application in AWS Elastic Beanstalk in the Sydney region.  This gives us very easy deployment, high availability, and easy scaling as demand increases (like it did after the recent earthquakes).

4. Locating the Quake

This is done by GeoNet Rapid, automatically in near real time and reviewed by the duty officer as required.  It's using all that great data coming from the remote seismic sensor network.

The Internet is Amazing!

When the notification arrives on my phone it has started from a message about an earthquake location in New Zealand, been sent to Sydney, forwarded onto America, and sent back to my phone in New Zealand.  This all happens within moments of the earthquake occurring.  A journey of over 25,000 km passing through several cloud services.  This might not be magic but every time it happens it seems pretty close to it to me.