In the last article, I talked a little about geopotential height and the data we’re going to use for this analysis. Earlier, I talked about how to read data from the NetCDF files that the NCEP reanalysis data comes in. Now we’re going to take a look at some of the features in the data set to get some idea of what we might see in our analysis. In order to do this, we’re going to have to produce some plots. As I’ve said before, I tend not to be very dogmatic about what software to use for plotting – for simple things (scatter plots, line plots, and so on) there are lots of tools that will do the job (including some Haskell tools, like the Chart library), but for more complex things, it tends to be much more efficient to use specialised tools. For example, for 3-D plotting, something like Paraview or Mayavi is a good choice. Here, we’re mostly going to be looking at geospatial data, i.e. maps, and for this there aren’t really any good Haskell tools. Instead, we’re going to use something called NCL (NCAR Command Language). This isn’t by any stretch of the imagination a pretty language from a computer science point of view, but it has a lot of specialised features for plotting climate and meteorological data and is pretty perfect for the needs of this task (the sea level pressure and plots in the last post were made using NCL). I’m not going to talk about the NCL scripts used to produce the plots here, but I might write about NCL a bit more later since it’s a very good tool for this sort of thing.
In this article, we’re going to look at some of the details of the data that we’re going to be using in our study of non-diffusive flow in the atmosphere. This is still all background material, so there’s no Haskell code here!
As I said in the last article, the next bit of this data analysis series is going to attempt to use Haskell to reproduce the analysis in the paper: D. T. Crommelin (2004). Observed nondiffusive dynamics in large-scale atmospheric flow. J. Atmos. Sci. 61(19), 2384–2396. Before we can do this, we need to cover some background, which I’m going to do in this and the next couple of articles. There won’t be any Haskell code in any of these three articles, so I’m not tagging them as “Haskell” so that they don’t end up on Planet Haskell, annoying category theorists who have no interest in atmospheric dynamics. I’ll refer to these background articles from the later “codey” articles as needed.
I never really intended the FFT stuff to go on for as long as it did, since that sort of thing wasn’t really what I was planning as the focus for this Data Analysis in Haskell series. The FFT was intended primarily as a “warm-up” exercise. After fourteen blog articles and about 10,000 words, everyone ought to be sufficiently warmed up now…
Instead of trying to lay out any kind of fundamental principles for data analysis before we get going, I’m just going to dive into a real example. I’ll talk about generalities as we go along when we have some context in which to place them.
All of the analysis described in this next series of articles closely follows that in the paper: D. T. Crommelin (2004). Observed nondiffusive dynamics in large-scale atmospheric flow. J. Atmos. Sci. 61(19), 2384–2396. We’re going to replicate most of the data analysis and visualisation from this paper, maybe adding a few interesting extras towards the end.
It’s going to take a couple of articles to lay out some of the background to this problem, but I want to start here with something very practical and not specific to this particular problem. We’re going to look at how to gain access to meteorological and climate data stored in the NetCDF file format from Haskell. This will be useful not only for the low-frequency atmospheric variability problem we’re going to look at, but for other things in the future too.
Here’s a mixed bag of interesting links, some sciencey, some mathsy, some miscellany:
Network Rail Virtual Archives: OK, this might not, at first sight, sound like something interesting, but it really is. This site has original Victorian-era engineering drawings for a whole range of British railway infrastructure. Bridges, viaducts, stations, tunnels. All rendered in lovely 19th Century penmanship. The Forth Bridge is particularly nice.
open.NASA: A couple of years ago, NASA started a project to open-source code and data from their Earth observing and planetary missions. Open.NASA is gateway to these resources. I’ve not had a chance to look at it in huge detail yet, but there is a lot of stuff there. The list of projects on the code.NASA part looks particularly entertaining.
Game of Primes: Giganotosaurus is a science fiction site that publishes one (longish) short story each month. They’re often very good, and this one was particularly striking – it’s quite beautifully done, full of mystery, and feels like it could be a part of something much larger and deeper.
Surprising connections in mathematics: This one is a bit more technical, from the Math Overflow Q&A website. A lot of the connections people mention are very technical, but some are more accessible, for instance the link between algebra and geometry developed by Descartes and others in the 17th Century. This is something we learn about in school, and something that we don’t think about too much because it seems “obvious”. Only obvious in retrospect, of course, since it took hundreds of years for the connection to be discovered!
De Bruijn grids and tilings: Another technical one, but very interesting. Aperiodic tilings of the plane, like Penrose tilings, are slightly mysterious. This article gives a really clear description of one systematic method for generating such tilings. It’s a very odd and intriguing little bit of mathematics.
Atul Gawande on end-of-life care: Atul Gawande is one of my favourite writers on medical and ethical issues. This article is quite long, but well worth a read.
by Eric Schlosser
My reading list recently has been chock-full of light-hearted and mood-lifting material: some Irvine Welsh novels (always guaranteed to shed a gentle light on all that’s best about the human condition), a long book about clinical depression, M. R. Carey’s interesting sort-of-zombie apocalypse/extreme mycology novel, The Girl With All The Gifts, de Becker’s The Gift Of Fear, a book all about fear and violence, and Piper Kerman’s prison memoir, Orange Is The New Black (which did spoil the mood a little having a few sparks of hope in among the gloom).
Among all this bleakness and blackness, Command and Control somehow manages to stand out as a particularly grim monument to human folly and our collective crimes against all sense and reason. It’s a book about nuclear weapons, so it never really had much chance of being too jolly, but even so, Schlosser’s decision to focus in parallel on US nuclear doctrine and nuclear weapons safety makes for some horrifying reading. It’s something of a mystery how we made it through the Cold War without either a “hot” war or at least some sort of unintended detonation of a nuclear weapon.
I’ve been doing quite a bit of reading lately, so I have 28 novels to review! All but one are from series of novels, so that’s not quite as daunting as it sounds. Still, I’ll split this into two posts to make it manageable.
In particular, getting from where you are now to where you want to be, in terms of your career.
As a result of an email I sent to the Haskell-Cafe mailing list a couple of weeks ago looking for someone to take over a contract I had been working on, someone contacted me asking for career advice. Clearly not someone who knew me at all, otherwise they would have known what a crazy idea that was. Anyway, this person was asking about one of the fundamental problems when you’re starting out in more or less any profession: how do you acquire the experience you need to apply for jobs that say “experience required”, which is more or less all of them?
They asked: “What is the path to getting involved in this stuff? How do I bridge the gap from just playing around with these technologies to having real world experience? It seems that most opportunities are for people with experience.” And this is exactly right. Particularly for contracting, no-one wants to hire someone they think will have to learn on the job. You need to know what you’re doing, which means getting experience somehow. And it would of course be nice to be able to eat and have a life while getting that experience.
I wrote an epic email in reply, and was told that it would have worked better as a blog post (or perhaps a short novel). So here I am, turning it into a blog post!
It’s more than two months since I last wrote a blog article. I’ve been ridiculously busy since then and things are only just now calming down. It now looks as though I’m going to try something new, at least for three months or so, and that should provide more time for blogging. I had to drop more or less all of my personal projects for the last couple of months, which has been frustrating (no work on my data analysis book, no work on
arb-fft, very little work on C2HS, a huge backlog of technical reading piling up and up and up like some Tower of Techno-Babel). Things should get back to something more like normal from now on though.
One benefit of working like a donkey for the last couple of months is that I now have a bit of money in the bank, and I’m planning to use that financial window to push some personal projects forwards. I have a few ideas, starting with “finishing”
arb-fft and getting back to some work on my book. I’ll do a couple of days of paid work a week, do a bit of open-source stuff (C2HS and Hackage mostly) and work on those personal projects. And blogging. There will be blogging.
Starting tomorrow. Now though, I’m going to go outside and lie myself down in the sunshine.