The perils of sharing different types of data at once

Organisations of all sizes have data that is of benefit to users both inside and outside their boundaries (both in the private and public sectors). Therefore having the tools and processes to find and share this data should make things run more efficiently and effectively... hopefully.

In a recent client conversation, we discussed the creation of a data portal / platform for the easier discovery and sharing of data. This inevitably led to the discussion about the types of data that the organisation wanted to share and who they wanted to share it with. Leading to the realisation that they actually had the need for sharing data that ranged from the very secure (restricted customer or commercially sensitive information) through to Open Data (information that they wanted to share for free outside of the organisation). They also had requirements for sharing data that sat somewhere in the murky area between those two extremes (information limited either by license or by access / user) including some they wanted to monetize.

Most data platform projects I have worked on previously have focused on the sharing of particular types of data (e.g. just Personally Identifiable user information or a mix of limited /restricted and Open data), but not the need for sharing different data sources from across The Data Spectrum:

https://www.theodi.org/about-the-odi/the-data-spectrum/

So creating a single data sharing technology platform that can share any sort of organisation data creates a number of issues:

Privacy and security:
Balancing accessibility and protection is tricky. You want your service to be easy to use, but with more sensitive data, strict access controls and strong security measures are vital.

Data standardisation:
Different users across an organizations often format their data differently. Such a service needs to be able to handle these inconsistencies or offer tools to convert data to a common format for sharing.

Traceability & reusability:
Data consumers need to understand how the data was collected and what it can be used for. Your service should provide clear audit trails and data provenance, as well as ways of licensing and charging for the data - especially if the data consumer is an external user.