Digital Agency Turbocharges Social Listening and Insights with the Infochimps Cloud
Today’s top contenders in the digital marketing and advertising market have a new set of challenges in creating and managing brands. Now, with an abundance of data at their fingertips, agencies are learning how to leverage their customers’ internal data to message and communicate brands and products more effectively. To further strengthen their efforts, they are tapping into technologies that allow them to leverage massive data streams from social and other web sources.
Meet Agency X*, one of the world’s top ad agencies with global brand management of a number of Fortune 500 accounts. Agency X has a proprietary social media listening and insights offering, powered by technologies like Radian6 among others that allows them to acquire and process data to visualize and present brand performance metrics and statistics. Agency X’s social media listening gathers information from social sites, like Facebook, Twitter, Tumblr and more, while the insights offering aggregates metrics from a number of sources including Google Analytics, Facebook Insights and Omniture. Ultimately, the organization was looking to unify their data platforms and offer their team members push-button reporting.
“We had realized that the most profitable 20% of our projects were reporting and data related. It was clear we needed to double down in this area and we quickly discovered Infochimps was the right partner to build it out.” - VP - Analytics, Agency X
Repeatability. Agency X is considered a leader in out-of-the-box thinking when it comes to new projects, it’s their bread and butter. However, this can put a lot of stress on the technical de-velopment team responsible for implementing these projects.
Every project seems like it’s breaking new technical ground and there’s always a tight deadline, which means code gets written in a one-off fashion, without long-term, repeated use in mind. And because actions can only be done ad-hoc and by a technical resource, the account teams can usually be left on their own when it comes to creating new reports through one esoteric tool after another.
Led by Agency X’s senior management, and spearheaded by its CTO, the organization decided it needed to develop a Platform - a consistent set of tools and processes for executing similar projects. Its data and reporting projects were a natural first place to start, as many of the workflows and scripts involved were ripe for consolidation and automation. It was clear that clients and account teams would benefit greatly from push-button reporting and analyti¬cal capabilities, saving the technical teams the effort, and getting insights to their clients in near real-time.
Diversification. Like many of the top agencies, Agency X is always looking for ways to offer more services to differentiate themselves from their peers and create more value. The best and largest agencies don’t focus on just one aspect of the advertising landscape, but have a whole portfolio of tools and services they can bring to bear - creating more value for their clients, and greater revenue opportunities for the agency, beyond hourly billing for creative work.
Early on Agency X realized the opportunity to differentiate themselves and serve their clients better by creating a social media practice. Social media strategy and reporting are by now just one of the many ways Agency X operates a diversified portfolio of services that can help their clients. After clients started asking for more metrics and measurement around their social media campaigns, Agency X adopted Radian6 as a third party tool to help them with reports. Realizing the power of such a system to gener¬ate real value for themselves and their clients, Agency X began looking for ways to have their own proprietary solution which they could fur-ther customize and give themselves greater capabilities.
Profitability. With every new digital project that Agency X took on, technical and human risk had the potential creep up and eat into that project’s margins. Stacking enough of these together meant Agency X could face signifi¬cant barriers to growing revenues and profits, as resources were tied up in projects that took longer than they should have and suck up more resources than budgeted.
Revenues and profits are always on senior management’s mind at Agency X. Leading up to working with Infochimps, Agency X realized that a significant portion of their repeatable profits came from data pipelines and reporting related projects. By Agency X’s own account, these were the 20% of projects that generated 80% of their profits. It was clear to them they should double their efforts in this area.
Creativity. At Agency X, one argument against bringing technology platforms in-house was that the focus on locking down toolsets and process¬es could restrict the creative possibilities for the organization and the kinds of projects they could take on. Certain employees at Agency X had experience with using legacy data plat¬forms, being familiar with Business Intelligence cubes and Relational Database Management Systems, and bemoaned their rigidity and the negative effects it would have on their client projects.
The CTO at Agency X knew that philosophi¬cally, these arguments were wrong. Embrac¬ing a few constraints and simplifications in one area of a business meant that other areas can be even more freely creative in their solutions - they can depend on the repeatable results and build around them. On a practical level the CTO knew the old-school Business Intelligence tools wouldn’t work for his needs, and he began evaluating the various big data technologies out there to see what could offer the flexibility he would still need within a solution.
Scalability and Elasticity. As content increased to “Big Data” scale, existing SQL databases were pushed to their limits.
Analytics Feature Development. Senti¬ment analysis, influence scores, and other derivative data are important features that would greatly add to the overall value of the listener and insights platform. However, there wasn’t a common platform and tool set for these features to be developed and scaled out.
Workflows. Much of the work that Agency X’s existing data platforms perform were done with manual tasks. Agency X needed a solution that enabled the automation of particular data workflows, starting with Sentiment Analysis and Influencer Reporting.
Client Customization. Customizing and preparing reporting and analytics for any given customer was largely a manual process. The ability to create customizations easily and efficiently, as well as cater analytics on a per-customer basis, would greatly enhance the efficiency of Agency X teams and increase the speed with which teams could deliver insights to customers.
Challenges. Beyond the complex organizational challenges in the way of moving Agency X into the future, there were a number of technical challenges that needed solved. As an example, for their social media reporting services their clients began asking for greater and greater insight and simplification, including real-time analytics and sentiment analysis. Their teams struggled with scaling the underlying technology to deliver these solutions.
Key challenges preventing the scaling of these systems for Agency X were:
Sourcing and Sustainability. Agency X relied on building custom scrapers and data integrations and soon found that sustaining these custom connections with analytics and application layers was messy and ad-hoc at best. Regardless of how an agency obtained their data, whether powered by homemade or third-party aggregators like Gnip and DataSift, the components and collected data were nearly impossible to maintain indefinitely as customer and project priorities shifted.
Solution. Infochimps helped Agency X strengthen and extend their listening and insights platform by leveraging the Infochimps cloud, tools, and expertise to unify data sources, process data streams, and visualize insights in real-time, all while satisfying on-demand internal reporting needs.
System Unification within a Flexible Big Data Infrastructure. By taking advantage of Big Data tools and Ironfan as the provisioning, configuration, and updates foundation, Agency X can scale to any volume of applications, reporting, and data collection. All applications, reporting, and data are connected as part of one unified, modular system.
Peace of Mind. Infochimps hosts, maintains, and monitors the underlying system architecture and tools to greatly minimize the operational resources and mindshare that Agency X must allocate to infrastructure support. Simply put - it just works.
Push Button Reporting. By leveraging the Hadoop suite of tools for batch analytics, as well as the Infochimps Data Delivery System (based on Storm and Kafka) for real-time streaming analytics, Agency X can develop analysis and subsequent reports on the fly for high customer responsiveness. The power and simplicity of the Infochimps analytics tools make it easier to develop and productionize analysis and reporting.
Modular Analytics Development. By using the Infochimps Cloud::Streams (based on Storm and Kafka) and “decorators,” Agency X can add any number of additional analytics enhancements as their feature set grows. These enhancements are modular, meaning Agency X can customize features depending on the customer’s needs, and new analytics can be added without disruption to previously developed metrics.
Working with Infochimps.
Infochimps has built a plug-and-play platform, provides all the data you could want from our data marketplace at your fingertips, and we’ve got the experts to make it all just work.
Big Data Experts. We built a uniquely powerful data system that could ingest and deliver massive quantities of data while performing complex, resource-intensive analytics in order to create and maintain our market-leading catalog of thousands of data sets and APIs. Now, com-panies looking to gain insights big data come to us to leverage our platform, experience and expertise. Underlying those offerings are a set of cutting edge Infochimps’ technologies and tools, including Ironfan and Wukong, as well as key partnerships.
Integrated. Infochimps’ experience and tool development in all the other aspects of Big Data, from ingestion and conditioning to low-friction delivery (in the form of our API cloud), means that the team understands and is equipped with the tools to successfully navigate and troubleshoot the entire Big Data ecosystem of an organization.
Infochimps cloud Benefits
The Infochimps cloud provided these key benefits to Agency X:
Diminished operations burden allowing them to focus on building products, not machines. Sustainable, scalable systems powered by best-in-class technologies like Hadoop, Flume, Elasticsearch, and more. Infochimps hosting, maintenance and administration, support, skills transfer, and Big Data strategy and methodology.
A system that is easy to change, grow, and manage. Built on a foundation of open-source technologies. Agile infrastructure development - their infrastructure is defined with descriptive code. This enables fast iterations, testability, and flexibility. Modularity and customizability, allowing them to respond dynamically to changing business or customer requirements.
Powerful analytics capabilities. Batch Big Data analytics with Hadoop and related tools. Streaming (real-time) Big Data analytics powered by Storm and Kafka and enhancements added by Infochimps’ Cloud::Streams.
Flexible Cost. Infochimps helps companies manage costs with infrastructure agnosticism. Want to pay for just what you need? We can leverage the flexibility of Infrastructure as a Service (IaaS) providers such as Amazon Web Services and the Rackspace Cloud. Now, all infrastructure costs can be operating expenses (use what you need) and not capital expenditures (pay for infrastructure whether you need it or not). Switching from CapEx to OpEx can dramatically lower the funding barrier to adopting Big Data internally in an enterprise. Got bare metal? We’ll set you up with OpenStack and Ironfan so you can create your own flexible, highly-efficient infrastructure environment.
Context. Perhaps best of all, the Infochimps cloud can be used to provide context to enterprise’s internal data, whether through public opinion mining (via social networks), geo-located information of many varieties, word corpus training for machine learning, and other commonly useful (but difficult to accumulate) data. All those capabilities combine to make us a great choice for providing Big Data services to the budget and process-conscious enterprise customer.