Big data refers to data sets that are too large to handle using conventional database management tools. Felix Tan, a lecturer in information systems, technology and management at theAustralian School of Business (ASB), says these sets can include traditional structured data, as well as unstructured data such as social media, smart meters, point-of-sale systems and machine data from geo-satellites.
“There’s a lot of buzz in the market about the potential of new data technologies,” says Tan, who is part of an ASB team collaborating on big data research with European technology vendor SAP and some of its Australian clients.
“We’re hearing about data speeds now that are 20,000 to 30,000 times faster than before. And storage space is much cheaper. In the past, one terabyte of data may have cost thousands of dollars for a business. But now anyone can walk into JB Hi-Fi and buy a terabyte of space for less than $100.”
From New York to Sydney, Beijing and beyond, a lot is happening in the big data space. US retailer Macy’s says data analytics have helped to boost store sales by 10% and to automate critical reports, saving half a million dollars a year. Financial services firm Capital One has become a Fortune 500 company using big data to grow its credit card service and cut costs. French bank BNP Paribas now completes critical reports in three seconds, rather than three minutes, and business intelligence storage needs have been cut by 75%. Chinese water distiller Nongfu Spring says operational reports now only take 3.8 seconds, rather than 24 hours, making the company more stable and more scientific in decision-making.
According to Tan, technologies such as mobile and cloud computing and social media are throwing up new business cases for real-time big data, which is characterised by the three Vs – volume, velocity and variety.
“Mobile devices and social media give you access to data, but it’s what you do almost immediately after you get the data that’s crucial. If you can analyse and respond to customer and supplier data almost instantaneously, the better informed your business processes will be. That’s the challenge,” Tan says.
Opportunities for Start-Ups
In Australia, corporates such as Woolworths, Wesfarmers, Australia Post, the Commonwealth Bank and AGL are variously processing big data for customer analytics, marketing strategies, risk management, audit reports and supply chain management. The Australian Public Service Big Data Strategy was released in 2013, to improve government competence in analytics.
But big data is not just for large players. Even start-ups can now afford to create new applications (apps). New York City has created a website for program developers to design apps around official city data to combat traffic and other problems. Big data has been “democratised” – and there’s more to come. American IT research and advisory firm Gartner estimates that by 2020, about 75% of workers will use analytics, up from 30% today.
Capitalising on this trend, SAP in 2012 launched its global Startup Forum, where apps can be built using its in-memory real-time platform, SAP HANA. Six Australian companies joined the forum last year and 400 solutions are in development globally.
“About 30 of them are market-ready,” says Ryan Blackwood, national director of SAP analytics, database and technology for SAP Australia & New Zealand. “There’s a bit of a revolution occurring in the way users use data. We’re seeing the business case now for really strong adoption by small players who can see the promise of big data and what they can do with it.”
Speed is the New Currency
One forum member is goCatch, the Australian mobile taxi app which allows users to book and pay for cabs using smartphones.
“If we’d tried to launch this app two or five years ago, it wouldn’t have been possible,” says Andrew Campbell, the co-founder of goCatch. About 150,000 passengers have downloaded the app and 15,000 taxi drivers have registered. “It’s gone viral. Our job volumes are growing by about 20% per month.”
In an upcoming iteration, “goCatch will offer extra points to taxi drivers to increase the probability of passengers being picked up”, Campbell says, noting that the service will crunch data in real time about how many taxis are in an area and other variables. “With big data, a report takes only milliseconds instead of days. It allows us to process massive amounts of data in real time.”
According to Tan, the speed factor is all-important. “Speed these days is the new currency. In the past we always talked about information as the currency, but right now with big data, I think it’s speed of information.”
Tan sees the taxi app as a good example of how big data can benefit the little guys. “It gives them the opportunity to deliver a product that’s innovative and unique in this market,” he says.
With an ASB team, Tan attended this year’s SAP forum and is working with goCatch – and soon other forum members – to research how start-ups use new technologies to create disruptive innovations. ASB's school of information systems, technology and management is part of the global SAP University Alliances program and, with service providers in the region, will roll out big data content to classrooms next year.
Other universities are also investing in big data. Canberra’s Australian National University plans to switch on its A$23 million cloud-based Australian National Computational Infrastructure service in October to crunch data in areas such as climate change and earth-system science. University of Technology, Sydney has set up its Advanced Analytics Institute, which is running data projects in areas ranging from airports to great white sharks.
Retail is another major sector. Tel-Aviv-based alcohol systems developer WeissBeerger has launched its Alcohol Analytics platform to give brewers and hotel chains real-time data from hundreds of installed flow meters on beer taps in bars and restaurants, which helps manage supply chains.
Australian retailer Woolworths paid about A$20 million in May this year for a 50% stake in data analytics company Quantium. Also in Australia, Wesfarmers is ramping up data analytics across retail and industrial supply chains to capitalise on data from its rewards scheme FlyBuys.
“We’re seeing a resurgence in retail for these types of analytics, particularly when it comes to customer intimacy,” says Blackwood.
Traction Takes Time
Companies are aware that they're not using their data properly. In a 2013 survey of world marketers by global analytic solutions company Teradata, 50% reported data as their most under-utilised asset. Barriers often cited included a shortage of qualified data specialists, security concerns and implementation problems. A survey by US research firm Forrester found only 7% of IT executives and 9% of business leaders feel they’ve gained a true return on big data.
It’s still early days for big data, as Gartner assessed this year. The IT research group said it would be five to 10 years before big data reached mainstream adoption – not the two to five years it previously estimated. Gartner sees two factors driving big data down: tools and techniques are being adopted ahead of learned expertise and any maturity/optimisation, which is creating confusion; and then there’s "the inability to spot big data opportunities by the business, formulate the right questions and execute on the insights".
Blackwood believes it takes time for new technologies to gain traction. “I think there’s a fear of these sorts of new concepts. But we’re now seeing organisations [such as] Australia Post, which was one of the more traditional types of IT operations, looking intently at the value they can unlock from big data,” he says.