This article, Inside American Express' Big Data Journey, explores what is needed to make the shift to using big data effectively.

You need to be prepared for a journey. The changes needed won't happen overnight.

You need buy-in from the top of the organisation. Without that, you won't get the mandate for the investment and the culture shift that is required.

At the moment, big data technologies are still being developed, so you need to adapt to using new tools, and be prepared to upgrade them regularly.

You also need to recruit and retain new talent. There are not that many people around with experience of big data, so you may need to get creative and be flexible. Once you have recruited the talent, you need to work out how to hang on to them, as they are likely to want to move on to the next challenge.

You need to be prepared for a process of continuous improvement and iterative learning. This applies to both development and marketing. This sounds like Agile to me, which means the whole company will need to adapt to working in an Agile way, becoming tolerant of a trial-and-error approach.

American Express also made the use of big data available to more decision-makers within the company, empowering people to "act locally", enabling new ways for customers to use their services. It also allowed them to get rid of obsolete services, and reduce levels of fraud, saving huge amounts of money and time. The benefits from fraud improvement alone have paid for their investment in big data.

Where to start?

The first step in the process of analysing, using, and benefiting from big data is being able to process large volumes of data. For American Express, the ability to process large amounts of data has allowed them to make credit decisions based on a bigger data sample, which significantly decreases the risks of lending.