The "Aha" Moment: How to Onboard an API Service and Get Active Users
Introducing Serverless Data Feeds
Share Data Without Sharing Credentials: Introducing Pipe-level Permissions
How to Embed a Live, Refreshable D3.js Chart into GitHub Pages
A 90 Degree Tilt: Introducing Vertical Pipes
A Simple Pipe Routing Example: HTML Upload to HTML Display
Introducing our API and Command Line Interface: Flex.io for Developers
Just Binge-Listened to 95 SaaStr Podcasts, Here's What I Learned
Stormy Weather: Open Data Doesn't Mean Accessible Data
Governments have made a lot of data available in the last few years. In the U.S. alone, Data.gov hosts over 140,000 data sets, and many other cities and states have launched open data initiatives as well.
However, a critical problem often arises when people actually try to find the specific data relevant to their needs. Governments often don’t tell people where to get the data.
It doesn’t matter how useful data sets are if you can’t find them.
Everything from the simplest to the most sophisticated data project can be crippled by an inability to locate the relevant data source. It’s the unavoidable first step for data seekers of all stripes. For my money, easy data accessibility is the most important hurdle for governments to clear for open data initiatives right now.
“But really,” you might be thinking, “How bad can it be? These are government programs, after all, and when has a government program ever been confusing or inefficient?” And besides, aren’t there private or commercial data sources that might be better alternatives anyway?
Well, let me show you, my friend.
My project was simple. So far in my life, I’ve had nineteen birthdays – a number I hope will continue to rise – and I wanted to find Chicago’s weather data for each of them.
All I really wanted for this was high and low temperature, precipitation, wind speed, etc. – basic stuff, right? Now, before this project I had no experience at all working with weather data. I’ve used various weather sites to get forecasts, of course, but that’s about it. So this was basically a first-time user experience.
Like any good Internet user, I started with a Google search. Specifically, I searched for “historical weather data.” Among the first results was a page from Weather.gov promising me that I could “find past weather…fast.”
“Doesn’t sound even remotely sleazy,” I thought, as I clicked the link.
Upon arriving I was treated to, well, whatever this thing is.
This bizarre document promised me that after a couple initial steps, there would be a link to a page called “Climate” in yellow text on the left side of the page. That turned out to be untrue, but there was a link to “Climate and Past Weather” in blue text toward the right side of the page. Close enough.
This page, naturally, wasn’t helpful either. Only very recent data is stored there, and if Weather.gov hosts data going further back, it’s certainly not easily accessible.
What do these bureaucrat punks think they’re doing, anyway? Enough! I can’t endure it anymore. Bad air! Bad air! [Editor’s Note: several hundred words of incoherent ranting removed here.]
- Frustration rating: Stepped on a Lego
A bit weary of government at this point, I moved on to the Weather Underground to help plot a leftist revolution see if they have anything easier to access in their historical weather data archives.
This site does have a really nice tool for searching for reports on dates going back at least to 1950, but there’s no way to get data spanning more than about a year and a half.
At this point, I was pretty annoyed. Weather is a data-rich topic by its very nature, and it’s not as if it’s some obscure subject like, say, penguin migration patterns. (Side note: anyone know where I can find data on penguin migration patterns?)
Surely there must be some data set somewhere that serves my needs, right?
- Frustration rating: Bad golfer
As it turns out, a solution does exist. My third attempt was to try the “Past Weather” page on the National Oceanic and Atmospheric Administration’s website.
Unlike the rambling, inaccurate mess that is the NWS’s guide, however, this page consisted of a single sentence directing me to the National Climactic Data Center website. From there, one click took me to the Climate Data Online tool. Finally I was getting somewhere.
The Climate Data Online search tool was exactly what I was looking for. I simply had to request daily information, set the date range I wanted, and choose a weather station. After that, I was able to customize the data elements I wanted and the format—PDF (for humans), CSV (for machines), or text (for cyborgs). Five minutes later, the relevant data was sitting in my e-mail inbox. All that was left was to filter out the records for all non-birthday dates, and at long last I had exactly the data I wanted.
Now, the CDO search isn’t perfect. One annoyance, which didn’t affect me personally, is that you’re only allowed 5,000 station years of data per request. No problem at all for someone looking for the weather at one station, but people doing complex projects over a large region of the country might struggle under such a restriction. But despite this quibble, it’s a fantastic tool: customizable but not overwhelming.
- Frustration rating: Lingering
I chose not to start my search for weather data at Data.gov, but if I had, I would have had an even more difficult time.
The Climate Data Online tool is accessible from Data.gov, but you need to know its name in order to reach it. A search for “historical weather data” does not yield this tool on the first page of results. What is on the first page?
Aerial photographs of Guam from 1944, for one. (Final verdict: it still looks like an island.)
Providing better search results on Data.gov won’t resolve the issue entirely, but it would be a good first step in making government data more easily accessible. In addition, it would be helpful to include clear guides on how to find relevant data and to modernize webpages that provide outdated or inaccurate information.
Open data is exciting in part because it is such a new development. We don’t really know its potential yet. The possibilities seem endless, and innovations are coming in left and right.
But the newness of open data can also make it hard to use. New things are never perfect right away. Whether you’re looking for the weather on your birthday or some other piece of data, if you’re not either very savvy or very lucky, you’ll probably have trouble finding the dang thing.
The sheer magnitude of data, the inevitable difficulty finding it, and the possibility of quality issues combine to make open data quite intimidating to a potential first-time user. Perhaps this is part of why only 5% of Americans believe the federal government shares data “very effectively.”
Ultimately, open data will have its greatest impact if as many citizens as possible are able to use it. While we may not currently be able to address every issue with open government data, surely we can at least make it easier to find. And that’d be a huge step in the right direction.
Header image by: MattysFlicks