What is a data marketplace?

A data marketplace (or data mart) is an online portal or website where data can be easily found and shared.

Although the name indicates that data is only bought and sold via a Data Marketplace, other data can also be made available such as Open Data (free to use, re-use or redistribute) or Shared Data (limited by license or end-usage).

Data marketplaces typically offer various types of data for different markets and from different sources. Making them a focal point for providing and accessing information about specific sectors or industries.

There can be a wide range of data formats and technologies provided via Data Marketplaces. However the widespread adoption of APIs for integration has meant that these are often the chosen modern way of publishing data sources now.

Wow, what a year 2019 was

I've had some pretty interesting years in my life (so far), but 2019 surely has to rank up there with busiest and most productive from a work perspective.

Aside from managing a Digital Strategy, technology and Marketing consultancy for the 12th year.... throughout 2019 I managed to pull together various transport / mobility and data specialists to form the Open Transport initiative. They worked to deliver two draft API specifications, one a transport account data interoperability standard and the other the design for a centralised transport operator information look-up (a directory service, giving the URL of each transport / mobility provider’s account API).

On the 3rd January 2020, following a wide-reaching peer review and feedback period, these specifications are now due to become ratified Open Standards and made publicly available. Thus, allowing the transportation and mobility industry to freely use them without conditions.
Yes.... I know. I am giving away a load of intellectual property that I've built up over many months of work from different subject-matter-experts... but the industry needs it (well, it actually needed it many years ago, but nobody ever got around to doing the work).

This work is available here (https://opentransport.co.uk) 

The aim of the initiative is that over time every transport authority (central and local government), transport operating company and their software vendors all recognise the customer value that this openness brings to the entire transport industry and therefore implements it at early as is feasibly possible….and possibly before it becomes legalisation.

My personal thanks goes out to all those who have helped. 

Loyalty & a fragmented transport network won't work!

I recently read an article on Wired.com about how USA-based transit providers are offering loyalty rewards to encourage usage of their services.

In my opinion royalty rewards are offered for only two purposes:
1. In exchange for customer data (e.g. give us your email for 10% off first purchase)
2. To change user behaviour (e.g. get 20% off rail fares for travelling off-peak)

However, this article got me thinking about how a multi-modal (rail, bus, subway, ferry, etc.)  transport network could never hope to implement an effective loyalty scheme if it only had one mode... in other words in a fragmented area, such as Scotland, could this work?

So here's my article on Linkedin explaining this in more detail:
https://www.linkedin.com/pulse/supporting-loyalty-rewards-fragmented-transport-hayden-sutherland/

In reality... of course I know it won't work unless:

• All operators share the same technology/software/account application (and I know how unlikely that would be in a privatised industry)
• There are ways for all these accounts to integrate together, so that an aggregated view of all accounts is possible in one place.
  And I think it is for this purpose that an Open Transport API should work towards.

Launching the Open Transport website

Today I have launched the website for Open Transport:
https://opentransport.co.uk/

This website explains the aim of creating an online account open data standard across the transport industry.

I have seen the start of the benefits that Open Banking has has in the Financial Services industry and I think the same thing needs to happen for the transit and transportation industry in Scotland, The UK, Europe and wider afield.

Perhaps this standard may even be adopted in other industries too.. perhaps the travel one.
(Can you imagine the possible benefits of all airlines being able to link their accounts, so that a customer could use just one login to manage all their details and see the data about their flights on different carriers?)

When not to use an API

In an earlier post, I covered how there are now "APIs with everything"

However, it is not always that a system designer would use an API for integration. Here are some scenarios where you would consider NOT using an API:

• When an API is not available, for example the vendor has not developed one yet
• When an API does not meet the necessary non-functional requirements (for example: security, performance, volume, availability, etc.)
• When you already have a tightly coupled integration between two systems and there is no need to change/upgrade this 

Have I missed any?

How HCI and API design are similar

All software needs an interface. Having one makes it possible to use the functionality and consume the data that lives within that software.

Humans use interfaces for software all the time, with the World Wide Web arguably being the biggest interface there is (OK, perhaps some websites aren't the best examples of usefulness in this mass experiment).  But put basically... they allow human-to-system interaction, also known as Human Computer Interaction or HCI for short.

Human Computer Interfaces have the following principles:

1. They remove complexity:
   By making it clear what each function does (e.g. do not have two ways of doing similar things)
2. They follow standards:
   By following established conventions (hyperlinks, buttons, tick boxes, etc.) they provide consistency.
3. They make interaction easier:
   By enabling swift and efficient relationship with the underlying data & processes and providing feedback or a response when something happens.

And Application Programming Interfaces (APIs) play a very similar role, but in a slightly different system-to-system manner. They:

1. Remove complexity, by (hopefully) providing a single request & response for each individual function
2. Follow standards, usually in the form of an API specification, to allow consistent development against them
3. Make interaction with the underlying systems easier via a standard set of methods (GET, POST, DELETE, etc.)

Ultimately, whether interacting with a human or another system... a well designed interface benefits both parties.

API's with everything

If you work in and around any sort of company that is undergoing a digital transformation, you will almost certainly hear the term "API".

Short for Application Programming Interface, put simply it provides a way for one system to be used by another in an agreed way. In other words, you can run some specific functionality or process elsewhere (e.g. within you organisation or across the Internet) via a set of published instructions.

APIs aren't a particularly new idea (I've been working with them since the 1990s), but they have grown in prominence and usage as other technologies and approaches such as Digital and SaaS (software as a service) have become more popular. It is therefore very likely that at some point or another most large organisations will have built APIs for their customers, or for their internal use.

Some  might therefore possibly state that we have currently "reached peak API use"... were it not for the likelihood that API development and adoption is set to increase further as more companies look to standardise and expand their digital integration capabilities.

It is also one of the reason's I called our company "Ideal Interface".

API granularity

Deciding how to design a digital architecture that provides optimum agility and scale-ability for your organisation is a challenge for many architects in the midst of a digital transformation. Deciding if you should use APIs in your integration layer is perhaps much less of a dilemma (hint: of course you must). But then working out how to design those APIs and the services behind them can be a new problem to solve.

I think this is an especially important decision, as it is quite hard to undo an architectural choice. Issues, can subsequently occur such as:

• performance bottlenecks (e.g. from too course a design)
• too much 'chatter' (e.g. from too granular a design) 
• the inclusion of too much (any?) business logic

All of which could have major implications in the future.

The Four Technical Pillars of Digital Transformation

There's no doubting that the path to successful Digital Transformation involves changes to: people (e.g.skills), processes, products (e.g. the creation or improvement of customer facing software) and proposition. The temptation is therefore to assume the technology will just "sort itself out" without an investment in thought, effort and finances. 

So what technology is now supporting the transformation to digital?

Here's my top 4:

The move to cloud

The use of online cloud-based services such as Amazon (AWS) and Microsoft (Azure) means that issues such as the hosting and the scaling of digital platforms becomes an on-demand Operational Expenditure (Opex) rather than a Capital Expenditure (CapEx) cost. With this high cost barrier now radically changed (as we create a shift from one column to the other on the Finance Department's spreadsheet), this means that demand and growth of online services are easier to deal with.

The digitisation of services
The conversion of the physical into software has been happening for some while. We've had digital media players and MP3 collections for many years now and you only have to look at how many of our daily tools are on our mobile devices, including: cameras, credit cards, health meters, maps, messengers and travel tickets. Now, with increased processing capability everywhere, what else can now move from being tangible to tap-able?

The creation and use of APIs

Organisations increasingly want simpler user interfaces that present and collate functionality and content from multiple systems behind the scenes. Your users don't care if your systems are having to pull together multiple source of data to present their online information in the way they want it, if you don't they will get frustrated (and consequently look to go elsewhere). Building Application Programming Interfaces (APIs) for each software system  enables this flexibility by providing the means for others to remotely invoke your applications functionality in a system-to-system way.

The adoption of DevOps

As the speed and complexity of digital delivery increases, companies realise they must integrate software development and IT operations. DevOps is the newer approach to this, where continual deployment becomes the norm and the ability of your tech team evolves form just being able to create stable code, to also deploying this code to a stable managed (typically cloud) environment.

The Wider CIO Role In A Digital Age

The role of the CIO has evolved over the last decade or so. In my opinion this is primarily as the technology services, digital products and online marketing industries have changed considerably over that period.

Cloud, Apps, SaaS, micro services, and a bunch of other technologies & paradigms now means that:

• Hosting a online service with 99.9% availability is just a £10/month subscription rather than a £100,000 per year investment
• Developers no longer go to offices and work where work is, but instead sit in their bedrooms and offer themselves to the most innovative opportunities
• Your customer (or customer's customer) is now online at any time... and more than ever before likely to be viewing your services via a smartphone
• The new Chief Digital Officer role came & went, with a lot of their work being picked-up by the CIO (and some by the CMO)
• Vendor lock-in is less of a concern than a vendor that doesn't have an API

All this also means the CIO role in the digital age has become more strategic than ever to the CEO. 

This is because, as software replaces more and more things, every business is becoming a technology business.

Organisational eBusiness Maturity

Thankfully, a growing number of organisations are looking to improve their digital channels. Consequently they are looking around for others who have already made a step forward and to learn from their innovations (without hopefully copying their mistakes).  Consequently I have seen certain trends appear over the years that may act as a model of not only where companies have come from, but also where they can grow and develop in the future.

Note: Like most of my work on this blog, this model is a ‘work in progress’ where I post my thoughts before they are completely refined and documented. It is therefore submitted with the aim that it will not only be refined by my own further understanding and application, but by wider feedback (either via comments on this blog or by other means).

 

Here’s how I see the organisational maturity of a company progressing (typically in the retail, financial services and travel markets, but potentially in others where this model can be applied):

Individual
Online initiatives originally sprung up thanks to the innovation and inspiration of specific people. Historically this may well have been a young-ish or passionate person who saw the opportunity to utilise some form of digital technology to improve something or interest to them.  Based in the IT, Marketing or other part of the company, they would initially have had very little influence, but potentially the opportunity to create and learn by themselves.

Department
As the individual has grown in their knowledge, they may well have caught the eye of senior individuals. Aligned with a growing understanding of the possible benefits of digital channels for communication, acquisition, commerce and engagement…  this one-man initiative may have grown into a team of people who have specialist knowledge of digital (marketing, eCommerce, User Experience, Analytics, etc.). From experience this has usually been the ‘land grab’ stage, with different high-powered players staking their claim to know all about modern technologies and taking this department under their wing.

Enterprise
Eventually the rest of the company wakes up and realises it is not just the digital team that either needs to understand and use digital tech, but that the whole organisation has become an e-ebusiness, with a digital eco-system around it… enabling everything from new customer influence and marketing through to existing customer self-service and HR connectivity (e.g. automatic postings to job sites, etc.)

Extended
For a company to truly ‘live’ digital however, it needs to move beyond the connected state (e.g. creating a bunch of fixed  digital connections with its customers and suppliers) it needs to extend some of its functions outside the organisation and embrace co-creation as a way to generate a flow of sustainable new ideas and talent.  API’s and XML interfaces enable these companies to allow 3^rd parties (individual developers, agencies and sometimes even entire industries) to build upon their data and functionality, to reach a new audience or to connect to other web services in ‘mash-ups’. Don’t get me wrong, this isn’t easy stuff and very few companies have the will, ability and braveness to venture into this territory. But for those who do (e.g. Amazon, Google, etc.) the rewards are obvious.

the corporate layers - an idea developed

In a recent post I started to work through my thoughts around the development of McKinsey's company software layer concept. I mentioned that I thought there was at least one obvious omission (e.g. web services) and that a single layer to explain it all was too simple.

So now I find myself putting forward an evolved version of this idea, unashamedly taking McKinsey's model as the basis of it.

Corporate layers

View more PowerPoint from Hayden Sutherland

As you can see from my embedded presentation there are now five proposed layers rather than just two.

1. Core business processes
2. Web services & API’s
3. Owned media such as website(s), Apps, Kiosks, etc.
4. Paid media such as online advertising (PPC, etc.)
5. Earned media such as Social, Word-of-mouth, etc.

Although I believe the lines between owned, paid and earned are now becoming increasingly blurred, there is a place for each of the different communication media in my new model.

Companies and the services layers

I recently received a report from McKinsey [link] that covered the topic of the 'corporate software layer'. This wasn't just another publication talking about application development standards, it referred to a metaphorical layer of services that surrounds the core 'hardware' processes and functions of an organisation. This software layer is an all-encompassing wrapper that the author described as including APIs, digital customer touchpoints and even Social Media.

Coincidentally I was chatting with someone in the Digital industry only a few days ago (thanks Wyndham) about how the use of web-based services can allow a company to incrementally develop its inner and external functions to remain agile & responsive to changes. I explained that via a Service Orientated Approach (SOA), a company could continually evolve its client-facing functionality (internal and external clients), wrap legacy functions and get the most from its digital agencies... by getting them to utilise any existing services.
Note:
We also questioned why digital agencies never seemed to develop work for clients using their own suite of web services to quickly deliver more intelligent work for their clients, but that's another topic for this blog sometime).

Anyhow.... Having thought about this some more, I do think there's the opportunity to not only merge the two topics mentioned above, but to then represent this in a similar (but perhaps more complex) way than McKinsey have done.

Here's what I mean:

1. McKinsey (perhaps in an effort to de-techie their report) haven't mentioned SOA and it's obvious benefit.

2. The 'software layer' is a little too catch-all for me.

3. Their diagram doesn't explain the big difference between the more technical machine-to-machine interfaces and the softer services that involve human interaction.

I guess what I'm saying here is that there might not be just one layer around the business, but possibly several service layers..... And the diagram needs to reflect this.

Does Facebook Places mean the end for Foursquare?

As regular readers and followers of my Twitter stream will know, I'm a frequent user of the Geo-location based application Foursquare. I've been using it for most of the year now and I like its combination of gaming (I am currently having a competition with a fellow nearby user to be the Major of our local Pizza restaurant) and tips... where I have been know to give the odd negative review.

However, Mark Zuckerberg made the announcement last week that Facebook will now extended the availability of its geo-tagging service Facebook Places to the UK (and Japan).

Facebook Places has up until this week only been available in the US, but it is now live here in the UK and already seems to be building up users (if my friends are anything to go by). The service is already built into the iPhone Facebook application, however other mobile users (Blackberry, Android, etc.) will have to wait for their developers to catch-up and for now can only use the service via the 'touch' version of the site accessed with a mobile browser.

But with the launch of this service, I am sure I'm not the only Foursquare UK user who thought "hey, why use two applications when one will now do?". Or to put it anther way, given the 500 million Facebook users, does this spell the beginning of the end for Foursquare?

Possibly... but I have an idea... sell the product to a bigger company right now!

But who? Well.... if:

• Google has Latitude & Places
• Facebook has errrr Places
• Yahoo has..... mmmmm... actually Yahoo hasn't done much since it failed to by Foursquare back in April. But it did go and buy a little Indonesian Geo-location service called Koprol which it has integrated, plus it has a 'geocoding API' called.... PlaceFinder

So who's left to buy up little old Foursquare?

Microsoft?

Perhaps. Although it also failed to buy Foursquare in April for the assumed $100m+ price tag, you can't help but think that Foursquare's value may have decreased a little now that Facebook has entered the market.

Social Graph API

There has been a lot of activity recently devoted to opening up the social networks (E.g. Open Social) and linking them together so a person can find their friends or family in multiple places.
This is related to the terminology called the 'Social Graph', a representation of an individual's network.

Here's some nice chap at Google explaining how the stuff comes together: