This article, Inside American Express' Big Data Journey, explores what is needed to make the shift to using big data effectively.
You need to be prepared for a journey. The changes needed won't happen overnight.
You need buy-in from the top of the organisation. Without that, you won't get the mandate for the investment and the culture shift that is required.
At the moment, big data technologies are still being developed, so you need to adapt to using new tools, and be prepared to upgrade them regularly.
You also need to recruit and retain new talent. There are not that many people around with experience of big data, so you may need to get creative and be flexible. Once you have recruited the talent, you need to work out how to hang on to them, as they are likely to want to move on to the next challenge.
You need to be prepared for a process of continuous improvement and iterative learning. This applies to both development and marketing. This sounds like Agile to me, which means the whole company will need to adapt to working in an Agile way, becoming tolerant of a trial-and-error approach.
American Express also made the use of big data available to more decision-makers within the company, empowering people to "act locally", enabling new ways for customers to use their services. It also allowed them to get rid of obsolete services, and reduce levels of fraud, saving huge amounts of money and time. The benefits from fraud improvement alone have paid for their investment in big data.
Where to start?
The first step in the process of analysing, using, and benefiting from big data is being able to process large volumes of data. For American Express, the ability to process large amounts of data has allowed them to make credit decisions based on a bigger data sample, which significantly decreases the risks of lending.
AMEX’s Gupta highlights three challenges the organization has faced along the way. First, adoption of new and immature technologies requires significant organizational adaptation and cultural transformation. Old processes become obsolete. New approaches require fresh skills and approaches. Second, AMEX needed to recruit new talent, with skills in Big Data solutions and approaches. This challenge is complicated by the scarcity of Big Data talent, and compounded by two additional factors: a) the need to always understand “business context”, which comes from experience, and b) the tendency for millennial Big Data talent to continually seek new challenges, creating a retention challenge. Lastly, Mr. Gupta cites the “marketing process journey”, which he characterizes as a process of continuous improvement intended to consistently refashion customer experience in a positive way.