Are you going to O’Reilly’s Strata Conference next week from February 26^th – 28^th at the Santa Clara Convention Center in Santa Clara, California?

Here at Infochimps, we’ve been planning for the big event, excited to see it all come together next week, and eager to talk Big Data.

As a proponent of empowering businesses through Big Data and bringing together thought-leaders from around the globe, Infochimps will be exhibiting at Strata along with a session given by Co-Founder and CTO Flip Kromer:

Title: “How Hadoop in the Cloud Affects Developer-Friendly Decision Making”
Date: Thursday, February 28, 2013
Time: 10:40a Pacific/12:40p Central/1:40p Eastern
Track: Hadoop in Practice
Location: Ballroom CD
Why: In this talk, Flip Kromer will walk you through a series of decision trees outlining why Hadoop in the cloud can be a powerful combination, helping to make clusters cheaper and developers happier.

Stop by the Innovators Pavilion, Booth P5, to chat about Big Data, set up a 1-1 meeting with one of our Big Data experts, or to just pick-up a free hip t-shirt. See you there!

The Ideal Big Data Application Development Environment

Lets assume that your entire organization had access to the following building blocks:

• Data: All sources of data from the enterprise (at rest and in motion)
• Analytics: Any/All Queries, Algorithms, Machine Learning Models
• Application Business Logic: Domain specific use-cases / business problems
• Actionable Insights: Knowledge of how to apply analytics against data through the use of application business logic to produce a positive impact to the business
• Infrastructure Configuration: High scalable, distributed, enterprise-class infrastructure capable of combining data, analytics, with app logic to produce actionable insights

Imagine if your entire organization was empowered to produce data-driven applications tailored specifically for your vertical use-cases?

Data-Driven Vertical Apps

You are a regional bank who is under heavier regulation, focused on risk management, and expanding your mobile offerings. You are seeking ways to get ahead of your competition through the use of Big Data by optimizing financial decisions and yields.

What if there was an easy and automated way to define new data sources, create new algorithms, apply these to gain better insight into your risk position, and ultimately operationalize all this by improving your ability to reject and accept loans?

You are a retailer who is being affected by the economic downturn, demographic shifts, and new competition from online sources. You are seeking ways of leveraging the fact that your customers are empowered by mobile and social by transforming the shopping experience through the use of Big Data.

What if there was an easy and automated way to capture all customer touch points, create new segmentation and customer experience analytics, apply these to create a customized cross-channel solution which integrates online shopping with social media, personalized promotions, and relevant content?

You are a fixed line operator, wireless network provider, or fixed broadband provider who is in the middle of convergence of both services and networks, and feeling price pressures of existing services. You are seeking ways to leverage cloud and Big Data to create smarter networks (autonomous and self-analyzing), smarter operations (improving working efficiency and capacity of day-to-day operations), and ways to leverage subscriber demographic data to create new data products and services to partners.

What if there was an easy and automated way to start by consuming additional data across the organization, deploy segmentation analytics to better target customers and increase ARPU?

It Starts With The “Infrastructure Recipe”

OK. You are a member of the application development team. All you have to do is create a data-driven application “deploy package.” It’s your recipe of all the data sources, analytics, and application logic needed to insert into this magical cloud service that produces your industry and use-case specific application. You don’t need to be an analytics expert. You don’t need to be a DBA, an ETL expert or even a Big Data technologist. All you need is a clear understanding of your business problem, and you can assemble the parts through a simple-to-use “recipe” which is abstracted from the details of the infrastructure used to execute on that recipe.

Any Data Source

Imagine an environment where your enterprise data is at your fingertips – no heavy ETL tools, no database exports, no Hadoop flume or sqoop jobs. Access to data is as simple as defining “nouns” in a sentence. Where your data lives is not a worry. You are equipped with the magic ability to simply define what the data source is and where it lives and accessing it is automated. You also care less whether the data is some large historic volume living in a relational database or whether it is real-time streaming event data.

Analytics Made Easy

Imagine a world where you can pick from literally thousands of algorithms and apply them to any of the above data sources in part or in combination. You create one algorithm and can apply it to years of historic data and/or a stream of live real-time data. Also, imagine a world where configuring your data in a format that your algorithms can consume is made seamless. Lastly, your algorithms execute on infrastructure in a parallel, distributed, highly scalable way. Getting excited yet?

Focus on Applications With Actionable Insights

Now lets embody this combination of analytics and data in a way that can actually be consumed and acted upon. Imagine a world where you can produce your insights and report on them with your BI tool of choice. That’s kind of exciting.

But what’s even more exciting is the ability to deploy your insights operationally through an application that leverages your domain expertise and understanding of the business logic associated with the targeted use-case you are solving. Translation – you can code up a Java, Python, PHP, or Ruby application that is light, simple, and easy to build/maintain. Why? Because the underlying logic normally embedded in ETL tools, separate analytics software tools, MapReduce code, NoSQL queries and stream processing logic is pushed up into the hands of application developers. Drooling yet?  Wait, it gets better.

Big Data, Cloud and The Enterprise

Lets take this entire application paradigm and automate it within an elastic cloud service purpose-built for the organization. You have the ability to submit your application “deploy packages” to be instantly processed without having to understand the compute infrastructure and, better yet, without having to understand the underlying data analytic services required to process your various data sources in real-time, near real-time or in batch modes.

Ok…if we had such an environment, we’d all be producing a ton of next-generation applications…data-driven, highly intelligent and specific to our industry and use-cases.

I’m ready…are you?

Jim Kaskade serves as CEO of Austin-based Infochimps, the leading Big Data Platform-as-a-Service provider. Jim is a visionary leader within both large as well as small company environments with over 25 years of experience building hi-tech businesses, leading startups in cloud computing enterprise software, software as a service (SaaS), online and mobile digital media, online and mobile advertising, and semiconductors from their founding to acquisition.

Essentials of Big Data Architecture (with Think Big Analytics) Thurs, Feb 21 @ 11a P, 1p C, 2p E

Join us to discuss the essentials of a Big Data architecture and how technologies are being used to influence different decisions. Scott Fleming, Engineering Practice Manager of Think Big Analytics and Tim Gasper, Product Manager of Infochimps will share essentials they’ve seen in their experience of standing multiple Big Data projects. We will also discuss the skills and processes required to make a successful implementation work.

This webinar will also discuss:

• What kinds of technologies make up today’s “Big Data” architecture
• What role do the components play in the analytics data pipeline
• How to decide which technology right for your use case
• Skills and people requirements
• Data governance

The webinar will be recorded, and emailed after the event to all who register.

Register Today>>

As the uses of Big Data continue to evolve with the creation of platforms and dashboards that promise in-the-moment marketing feedback, much skepticism arises as to whether or not they can deliver on their promise: real-time decision making analytics that are actionable and accessible without a team of data-scientists.

Though advancements are being made every day and with an infinite future of refinement to come, there naturally exists some uncertainty around Big Data, and what it can actually offer marketers. Here are a few of the most common misconceptions fueling such apprehension:

Myth: Campaigns take weeks, if not months to execute.

Truth: Big Data makes real-time campaigns a reality.

As stated simply by GigaOm’s Ravi Mhatre, “Big Data is useless unless it’s also fast”. At a time when social and mobile walk hand-in-hand, marketing departments must be agile, and capable of acting at the drop of a hat (or tweet). The fact is that Big Data has entered an era where “real-time” is possible, and business dashboards power crucial decisions in-the-moment.

Myth: Marketers must still rely on “gut” decisions, which may or may not be reliable.

Truth: Big Data powers success through simple data-driven decisions.

Another common theme among skeptics is the notion that the only people equipped to understand Big Data insights are the data scientists that are siloed in departments and organizations far away from management and marketing. This is simply not the case.

Widely available tools allow marketers and other business experts to derive data-driven insight without having the technical expertise of a data scientist. Marketers can now perform sophisticated analytics to deliver truly actionable information about efficiencies (or lack thereof) within a business, as well as tangible insights about customers.

Myth: Much data is useless.

Truth: All data is powerful; Big Data makes it possible for a business to find unexpected stories and insights.

With traditional techniques, data storage is expensive, and therefore finite. Consequently, companies have had to pick and choose which data is important enough to keep, and have thrown away data which actually could have yielded valuable insight. With new Big Data technologies drastically reducing the storage and management price tag, companies now have the freedom to save and analyze everything – those who do will quickly begin to uncover gems.

As buzz builds around potential enterprise Big Data use-cases, so does hesitation and concern that this is just a fleeting trend; but this couldn’t be farther from the truth.

While we’ve finally reached the point where it is feasible for businesses to tap and start to understand the data that streams from their market, operations, and customers, there’s still much room for refinement. Although one can justifiably state that we’ve entered the era of Big Data, where campaigns can be executed quickly and insights can be pulled in real-time, we’ve just hit the tip of the iceberg in terms of the untapped potential of these technologies.

Amanda McGuckin Hager is a high-tech marketing professional with over 17 years of experience focused on driving demand through strategic marketing programs. She is the Director of Marketing at Infochimps. Follow Amanda on Twitter.

Image Source: brightonwoman.blogspot.com

Our partners at *IE. would like to introduce you to the exclusive Big Data Innovation Summit in San Francisco on April 11 & 12, 2013.

Take a look at the agenda here >>

The Big Data Innovation Summit is the largest gathering of Fortune 500 business executives leading Big Data initiatives.

It will help your business understand & utilize data-driven strategies and discover what disciplines will change because of the advent of data. With a vast amount of data now available, modern businesses are faced with the challenge of storage, management, analysis, visualization, security and disruptive tools & technologies.

The Summit includes 6 tracks:

• Big Data Innovation
• Big Data in Healthcare
• Big Data in Finance
• Big Data in Government
• Hadoop Innovation
• Women in Big Data

Confirmed speakers include:

• Principal Data Scientist, eHarmony
• Senior Vice President, Strategy, Comcast
• Data Science, Analytics, Facebook
• Director, Global Customer Insights & Analytics, Walmart
• Data Scientist, Groupon
• Team Lead, Mobile Data Science, LinkedIn
• Vice President, Advanced Analytics, Orbitz
• WorldwideSenior Computer Engineer, General Electric
• Principal Data Scientist, Director of Data Science, eBay
• Data Engineer, Amazon.com
• Director, Enterprise Warehousing, NBC Universal
• Vice President, Strategy & Engagement, Salesforce.com

& many more…

Register Today >>

If you want to meet up with us at the Big Data Innovation Summit and talk shop about your Big Data project objectives, design, infrastructure, tools, etc, let us know at info[at]infochimps[dot]com. If you’re interested in scheduling a demo, go to: Infochimps.com/demo

Traditionally, systems configuration involves a time-consuming process that is vulnerable to human error. Infochimps leverages the power and simplicity of Ironfan as its provisioning and deployment layer, allowing end-users to easily launch and orchestrate repeatable infrastructure. If you’re interested in digging deeper into the details of Ironfan, take advantage of:

Ironfan: A Community Discussion Thursday, January 31 @ 10a P, 12p C, 1p E

Join Nathaniel Eliot, DevOps Engineer and lead on Ironfan, in this community discussion. Ironfan is a lightweight cluster orchestration toolset, built on top of Chef, which empowers spinning up of Hadoop clusters in under 20 minutes. Nathan has been responsible for Ironfan’s core plugin code, cookbooks, and other components to stabilize both Infochimps’ open source offerings, and internal architectures.

Register Today >>

It’s no secret that enterprises today face an increasingly competitive and erratic global business environment, and that Big Data is more than just another IT project – it’s truly a finger on the pulse of the business. To say that in 2013 Big Data is “mission critical” is to put it mildly – organizations that ignore the insights that Big Data can deliver are flying blind. So, it is all the more disconcerting that 55% of Big Data projects don’t get completed, and many others fall short of their objectives.

In order to understand the reasons for this, Infochimps partnered with SSWUG.org, one of the largest enterprise technology-focused, community-driven sites and a source for answers to IT-related questions and professional growth for more than 570,000 members. Together we got survey responses from over 300 IT department staffers – 58% of whom have current Big Data projects underway – on what they most wanted their CIOs to know about the process of implementing Big Data projects.

Read the full report here. >>

Key findings are summarized in the following infographic:

While the findings reveal many reasons for Big Data project failure, undoubtedly one of the biggest factors is lack of communication between top managers, who provide the overall project vision, and the data scientist and other IT staff charged with actually implementing it. Far too frequently their opinions are taken as an afterthought, and consequently considered only when projects veer off-course.

Given the stakes, it’s imperative that CIOs have a 360-degree view of all that a Big Data project will involve – not just the various Big Data technologies that are so frequently at the forefront of Big Data discussions.

The insight we gleaned reveals much about both enterprise technology and enterprise culture. In order for companies to succeed with Big Data, executives will need to rethink long-held notions of how diverse departments should function together. In the past “breaking down silos” was a nice mantra. Now, it is imperative. Additionally, CIOs and other enterprise executives may find it necessary to educate their organizations on the advantages of new Big Data applications and processes that will give them better customer insights, make their jobs infinitely easier and give their departments the elasticity needed to meet virtually any business need in real-time.

We hope this report will serve not only as a source of insight, but also be a reminder to seek the invaluable perspective of IT staff as early as possible in the process of developing new, technology-intensive projects.

Read the press release here. >>

I’ve been reading Flip’s book, Big Data for Chimps: A Guide to Massive Scale Data Processing, available for pre-order now from O’Reilly. While I’m no data engineer, I am able to follow along. After reading a bit, it comes as no surprise that Flip helped to found Infochimps with the philosophy of making the world’s knowledge accessible to anyone.  The content is unexpected and engaging. Take, for example, the story of Chimpanzee and Elephant Start a Business, from The Stream Chapter:

Chimpanzee and Elephant Start a Business

As you know, chimpanzees love nothing more than sitting at typewriters processing and generating text. Elephants have a prodigious ability to store and recall information, and will carry huge amounts of cargo with great determination. The chimpanzees and the elephants realized there was a real business opportunity from combining their strengths, and so they formed the Chimpanzee and Elephant Data Shipping Corporation. They were soon hired by a publishing firm to translate the works of Shakespeare into every language. In the system they set up, each chimpanzee sits at a typewriter doing exactly one thing well: read a set of passages, and type out the corresponding text in a new language. Each elephant has a pile of books, which she breaks up into “blocks” (a consecutive bundle of pages, tied up with string).

Read the full chapter (available here: The Stream Chapter) to understand how this example, combined with pig latin, simple streamers, and running Hadoop jobs have to do with each other. You’ll also get two exercises and a Ruby helper section containing tips and tricks.

Amanda McGuckin Hager is a high-tech marketing professional with over 17 years of experience focused on driving demand through strategic marketing programs and is the Director of Marketing at Infochimps. Follow Amanda on Twitter.

Here at Infochimps, we value the developer community. From Founder and CTO Flip Kromer being named by GitHub as one of the Top 100 Contributors in 2012, to hosting community discussion webinars, Infochimps tries to keep developers’ needs and interests in mind.

Here are some resources catered towards the developer community:

In case you missed it: Knowing analytics tools designed for developers’ needs are in high demand, Derrick Harris writes an article highlighting the top 12 Big Data tools developers need to know: “A programmer’s guide to big data: 12 tools to know”

Pre-Order Now – Big Data for Chimps: In addition to being a prolific code contributor and one of the nations’ leading data scientists, Flip Kromer is the author of Big Data for Chimps, A Guide to Massive Scale Data Processing, published by O’Reilly, and available for pre-order now.

Upcoming Webinar: Ironfan – A Community Discussion
Thursday, January 31 @ 10a P, 12p C, 1p E

Join Nathaniel Eliot, DevOps Engineer and lead on Ironfan, in this community discussion. Ironfan is a lightweight cluster orchestration toolset, built on top of Chef, which empowers spinning up of Hadoop clusters in under 20 minutes. Nathan has been responsible for Ironfan’s core plugin code, cookbooks, and other components to stabilize both Infochimps’ open source offerings, and internal architectures.

Flip Kromer, Infochimps Founder and CTO, also known as MrFlip, was named by GitHub as one of the Top 100 Contributors in 2012. Flip made over 2,300 contributions to the global, open source developer community.

And he’s in good company. Also on the list are: Linus Torvals of Linux, Erik Michaels-Ober, and Dr. Nic Williams.

In addition to being a prolific code contributor and one of the nations’ leading data scientists, Flip is the author of Big Data for Chimps, A Guide to Massive Scale Data Processing, published by O’Reilly, and available for pre-order now.

About GitHub: Github, a Forbes’ Top Tech Company of 2012 and the largest code host in the world, was founded in 2008 and is leading enterprises to adopt open source technology. Github, known for social coding, was founded as a place for developers to code together, as teams and individuals.

About Infochimps: The Infochimps Platform for Big Data combines leading data technologies with managed cloud services, a strong partner network to empower customers with unprecedented speed, scale and flexibility in their Big Data initiatives. Infochimps is a privately held, venture-backed company with offices in Austin, TX and the Silicon Valley. Follow @infochimps on Twitter.