In November we announced that we were retiring our current data store. Here’s a quick update on our plans to ensure continued access to all of the local data we’ve published. Including a preview of our new datastore.
Originally our plan was to switch to a static publishing solution, using github to host the data files. We’ve been using this approach for a while for some our geospatial data and the council had also prototyped this as an option.
It was easy to create archival copies of our datasets ready to publish via this route. Github provide an effective and free way to distribute static data. And tools like JKAN and Octopub provide user and publishing interfaces onto this type of static catalogue.
But two of our most popular datasets are the live parking data and the regularly updated air quality data. While these could be published as frequently updated static files, it’s not an ideal approach. So we knew we needed a means to ensure that at least some of the datasets were published in other ways.
To support this, the council have deployed an instance of geoserver, an open source platform for publishing geospatial data using a standard set of APIs. It’s not quite ready for public use, so we’re not sharing a URL to it yet. But it will provide them with a more flexible way to publish spatially located data like the car park occupancy and air quality readings, as well as other geospatial data like local boundaries.
With a plan in place for continued access to the live and geospatial data from the council, we just needed to decide how to provide a user interface onto the catalogue and whether we were going to offer any other APIs.
Mark Owen has made that whole decision easier by developing us a shiny new data portal!
You can see a preview of the new datastore.
The portal is based on new custom code that Mark has developed over the last few weeks. It currently includes:
- Search and browse by collection and tags. It should contain all of our published data
- Page per dataset with full metadata and embedded data tables
- APIs providing access to the catalogue and dataset metadata as well as the data held in each dataset
- APIs to import, update and maintain data behind the scenes
It’s still at an early stage, but its looking great. Mark is planning to add more features over time and will open source the code once its a bit more stable. Having a home-grown solution gives us a bit more flexibility and control over the features we need.
If you’re looking to migrate to the new APIs, then have a look at the API links on the relevant dataset page. For example here’s a preview of the live car park occupancy data.
Depending on timing of DNS changes, the new store will be available at data.bathhacked.org on Tuesday or Wednesday next week. Redirects are in place for the older URLs, so hopefully we won’t have any broken links to the data.
Mark has also pointed out that there’s a easter egg hidden in the visual design of the new site. See if you can spot it!
If you have any feedback or questions, then leave a comment or ask us on twitter.