Guest Blogger Series

Becoming a Believer in Artificial Intelligence

Big Think originally published this transcript of Eric Siegel’s own words. The article relates to his book, Predictive Analytics: The Power to Predict Who Will Click, Buy, Lie, or Die.

Why I Became a Believer in Artificial Intelligence

I’ve been asked periodically for a couple of decades whether I think artificial intelligence is possible.  And I taught the artificial intelligence course at Columbia University.  I’ve always been fascinated by the concept of intelligence.  It’s a subjective word.  I’ve always been very skeptical. And I am only now newly a believer.

Now this is subjective: my opinion is that IBM’s Watson computer is able to answer questions, and so, in my subjective view, that qualifies as intelligence.  I spent six years in graduate school working on two things.  One is machine learning and that’s the core to prediction – learning from data how to predict.  That’s also known as predictive modeling. And the other is natural language processing or computational linguistics.

Working with human language really ties into the way we think and what we’re capable of doing and that does turn out to be extremely hard for computers to do.  Now playing the TV quiz show Jeopardy means you’re answering questions – quiz show questions.  The questions on that game show are really complex grammatically.  And it turns out that in order to answer them Watson looks at huge amounts of text, for example, a snapshot of all the English speaking Wikipedia articles.  And it has to process text not only to look at the question it’s trying to answer but to retrieve the answers themselves.  Now at the core of this it turns out it’s using predictive modeling.  Now it’s not predicting the future but it’s predicting the answer to the question.

The core technology is the same.  In both cases it involves learning from examples.  In the case of Watson playing the TV show Jeopardy it takes hundreds of thousands of previous Jeopardy questions from the TV show having gone on for decades and learns from them.  And what it’s learning to do is predict whether this candidate answer to this question is likely to be the correct answer.  So it’s going to come up with a whole bunch of candidate answers, hundreds of candidate answers, for the one question at hand at any given point in time.  And then amongst all these candidate answers it’s going to score each one.  How likely is it to be the right answer?  And, of course, the one that gets the highest score as the highest vote of confidence – that’s ultimately the one answer it’s going to give.

READ 300x80 Becoming a Believer in Artificial Intelligence



Eric Siegel, Ph.D., founder of Predictive Analytics World and Text Analytics World, and Executive Editor of the Predictive Analytics Times, makes the how and why of predictive analytics understandable and captivating. In addition to being the author of Predictive Analytics: The Power to Predict Who Will Click, Buy, Lie, or Die, Eric is a former Columbia University professor, and a renowned speaker, educator, and leader in the field. 

406b489e b14e 4684 bbd3 c316b533aea8 Becoming a Believer in Artificial Intelligence

[Guest Blog] Jim Kaskade on Requirements for Effective Analytics Projects

Data Informed 300x62 [Guest Blog] Jim Kaskade on Requirements for Effective Analytics ProjectsInfochimps CEO, Jim Kaskade, recently guest blogged for Data Informed, the leading resource for business and IT professionals looking for news, in-depth case studies, expert insight, best practices, and product analyses to plan and implement their data analytics and management strategies.

In his blog post, “Effective Analytics Projects Require Blend of 3 Approaches, in Concert“, he mentions “arranging technologies to address a particular use case is a little bit like arranging instruments in a symphony”.

“Each has particular qualities and plays a different role in the end product, but that end result should be a seamless meshing of the individual elements to achieve the desired effect such that the unique qualities of each one fade into the background. The same is true of Big Data use cases.”

Read the full article, here. >>

Thank you Data Informed for giving decision makers perspective on how they can apply Big Data concepts and technologies to their business needs.

406b489e b14e 4684 bbd3 c316b533aea8 [Guest Blog] Jim Kaskade on Requirements for Effective Analytics Projects

5 Reasons to Not Care About Predictive Analytics

predictive analytics 5 Reasons to Not Care About Predictive AnalyticsTechnology: complex and alienating, or promising and fascinating?

I’ve seen plenty of people roll their eyes and give all sorts of reasons they don’t pay much attention to predictive analytics, the increasingly common technology that makes predictions about what each of us will do—from buying, thriving, and donating, to stealing and crashing your car. Here are 5 reasons to go ahead and ignore this prognostic power… or not—you may choose to pay close attention after all.

1. Predictive computers don’t affect me. Not true. You are predicted every day by companies, government, law-enforcement, hospitals, and universities. Their computers say, “I knew you were going to do that!” These institutions seize upon newfound power, predicting whether you’re going to click, buy, lie, or even die. Their technology foresees who will drop out of school, cancel a subscription or get divorced, in some cases before they are even aware of it themselves. Although largely unseen, predictive proaction is omnipresent, determining whom to call, mail, investigate, incarcerate, set up on a date, or medicate.

2. Corporations invade privacy with data and prediction. This is sometimes true. Predicting human behavior is a new “super power” that combats financial risk, fortifies healthcare, conquers spam, toughens crime-fighting, boosts sales, and wins votes. Organizations gain this power by predicting potent yet—in some cases—sensitive insights about individuals. Companies ascertain untold, private truths—Target figures out that some customers are pregnant and Hewlett-Packard deduces who’s about to quit his or her job. We must each make our own judgment about judges and parole boards who rely every day on crime-predicting computers to decide who stays in prison and who goes free.

3. Prediction is impossible. Not so fast. Nobody knows the future, but putting odds on it to lift the fog just a bit off our hazy view of tomorrow—that’s paydirt. Organizations win big by predicting better than guessing, and they are continually cranking up the precision of predictive technology. Per-person prediction is the key to driving improved decisions, guiding millions of per-person actions. For healthcare, this saves lives. For law enforcement, it fights crime. For business, it decreases risk, lowers cost, improves customer service, and decreases junkmail and spam. It was a contributing factor to the reelection of the U.S. president. Predictive analytics is one of this century’s most important emerging applied sciences.

4. Science is boring—I drive a car but I don’t care how it works. Think again. Cars are simple: little explosions push them. But a computer that learns to predict? That’s a conceptual revolution. There’s an inevitable parallel to be drawn between how a computer learns and how a person learns that only gets more interesting as you examine the details of the machine learning process. It gets even more exciting when you see the heights this technology can reach, such as that achieved by IBM’s Watson computer, which defeated the all-time human champions on the TV quiz show Jeopardy! by “predicting” the answer to each question.

5. I hate math. That’s OK. You don’t need formulas to see how this fascinating science works. Predictive analytics learns by example. The process is not so mysterious: If people who go to the dentist most often pay their bills on time, this factoid is noted and built upon to help predict bill payments. At its core, this technology is intuitive, powerful and awe-inspiring—learn all about it!

Eric Siegel, Ph.D., is the founder of Predictive Analytics World (—coming in 2013 and 2014 to Toronto, San Francisco, Chicago, Washington D.C., Boston, Berlin, and London—and the author of Predictive Analytics: The Power to Predict Who Will Click, Buy, Lie, or Die (February 2013, published by Wiley). For more information about predictive analytics, see the Predictive Analytics Guide (

b0bae296 90b0 4bfe 8177 b5ac72be71c6 5 Reasons to Not Care About Predictive Analytics

Big Data: The Marketer’s (potential) Goldmine

Gold 300x225 Big Data: The Marketers (potential) GoldmineIn a world littered with seemingly endless streams of data, marketing has some interesting challenges.  What exactly are we supposed to do with terabytes, even petabytes, of data?  Unfortunately, there is still a lot of buzz in the phrase, “big data”.  Most believe it’s still just a marketing term that provides little value in the long run.

However, big data is quite powerful.  While its relevance may seem obvious for certain industries that are heavily structured around data, for most organizations, big data still represents an unknown.  Most marketing organizations have now implemented some form of customer relationship management (CRM) systems, but most of these programs have been poorly executed.  Storing huge amounts of consumer data may seem appealing, but how exactly can marketing teams use this data to the company’s advantage?

There are numerous examples of analytical tools and campaign software.  How does big data fit into this mix?  How can marketers leverage these tools to notice trends in a timely manner that would allow us to properly direct resources, thereby eliminating waste and improving ROI? How can big data provide insights that might otherwise be overlooked?

Raw data can tell us what, but it can’t tell us how or why.  Insights are key.  It can seem like searching for a needle in a haystack.  But as more advanced technology surfaces, big data can help marketers turn information into insights at a relatively break-neck pace.  This offers substantial competitive advantages.

Here are a few areas where big data can help marketers gain a competitive edge:
Predictions:  Big data is fundamentally about predicting outcomes and behavior.  By analyzing data from multiple channels and multiple points in time, big data can potentially supply marketers with information about customers’ buying and behavior patterns.  We now have highly advanced real-time analytical models to ease these challenges.  Instead of taking data from only the past, we can now use big data to find patterns based on real-time data, not simply averages over certain periods of time.

Better Decisions:  When we have accurate predictive models in place, we can make better decisions about all aspects of a marketing campaign.  And more importantly, these decisions can now be based on evidence, rather than guesswork.  How many times have you seen marketing and sales teams make decisions based on gut feelings?  Gut instinct is a nice way of saying gambling.  As creative marketers, we certainly like intuition and creativity, but, for the most part, decisions in this context need to be based on evidence.  If we can know what, how, why and when, there is no need to rack our brains on “what if” scenarios.

Social Media:  When social media began to surface, it made us realize that companies can interact and engage with customers (and potential customers) immediately.  And with further development of analytics, it’s becoming more important to evaluate those interactions to determine customer opinions and perceptions, and identify ways to improve.  Social media has given the public a platform to voice their opinions, which can go viral in the blink of an eye.  Depending on what has been said, there may be a need for serious damage control.  Never before has the public had the power to make or break a company with an expression of their thoughts and experience.  Big data can help gather the information and make predictions based on that data.  Just as governments, NGOs and other organizations have used data to predict patterns in crime, health epidemics, and even financial events and fraud patterns, companies can utilize the huge amounts of data in social platforms to not only observe customer thoughts, but to analyze their online behavior to predict the success of a company’s product or service.

Customer Experience Management:  Consumer data is extremely valuable.  It’s the marketer’s currency.  And with recent advancements in technology, there are now ways that data can help us discern what is valuable to the customer, what influences customer loyalty and what makes them return.  This is relevant because having this insight can help marketers tailor advertising campaigns to current and perspective customers.

Automation:  Marketing automation is now expanding into more advanced functionality, partially with the help of machine learning.  Big data is giving marketers the ability to create advertising and content that is highly personalized.  When actions are automated, machines and humans can form a partnership and the platforms can eventually decide what is released to the customer, why it’s released, and how.  This can be very powerful because the more relevant the material for consumers, the more likely they are to make a purchase.

Marketing Evaluation and Performance:  One of the most valuable aspects to big data is that the technology can be used to measure nearly everything.  It not only gives marketers insight into their customers, but also into their own organization’s efficiency.  Big data makes it possible for marketers to monitor the productivity of marketing programs through constant analysis and evaluation.  Campaigns are virtually useless if their performance isn’t being measured.

The Bottom Line:
Big data is often a missed revenue opportunity.  If marketing teams can incorporate data management into their strategy, the payoff is potentially huge.  Think of data as the essential ingredient to an overall strategy.  The combination of insights from big data, decisions based on those insights, and the actions taken will ultimately prove worthwhile.

Jessica Marie is a consultant, writer and recovering banker.  After nearly 10 years holding various positions in commercial banking and finance, she moved on to pursue her passion for enterprise technology and corporate strategy.  In her spare time, she is a volunteer with non-profits and a concert pianist by night.  Reach her via Twitter @jessicamariemba.

        b0bae296 90b0 4bfe 8177 b5ac72be71c6 Big Data: The Marketers (potential) Goldmine


Image Source:

Tell Your Children to Learn Hadoop

I spent some time with several vendors and users of Hadoop, the formless data repository that is the current favorite of many dot coms and the darling of the data nerds. It was instructive. Moms and Dads, tell your kids to start learning this technology now. The younger the better.

I still know relatively little about the Hadoop ecosystem, but it is a big tent and getting bigger. To grok it, you have to cast aside several long-held tech assumptions. First, that you know what you are looking for when you build your databases: Hadoop encourages pack rats to store every log entry, every Tweet, every Web transaction, and other Internet flotsam and jetsam. The hope is that one day some user will come with a question that can’t be answered in any way other than to comb through this morass. Who needs to spend months on requirements documents and data dictionaries when we can just shovel our data into a hard drive somewhere? Turns out, a lot of folks.

Think of Hadoop as the ultimate in agile software development: we don’t even know what we are developing at the start of the project, just that we are going to find that proverbial needle in all those zettabytes.

Hadoop also casts aside the notion that we in IT have even the slightest smidgen of control over our “mission critical” infrastructure. It also casts aside that we turn to open source code when we have reached a commodity product class that can support a rich collection of developers. That we need solid n.1 versions after the n.0 release has been debugged and straightened out. Versions which are offered by largish vendors who have inked deals with thousands of customers.

No, no, no and no. The IT crowd isn’t necessarily leading the Hadooping of our networks. Departmental analysts can get their own datasets up and running, although you really need skilled folks who have a handle on the dozen or so helper technologies to really make Hadoop truly useful. And Hadoop is anything but a commodity: there are at least eight different distributions with varying degrees of support and add-ons, including ones from its originators at Yahoo. And the current version? Try something like 0.2. Maybe this is an artifact of the open source movement which loves those decimal points in their release versions. Another company has released its 1.0 version last week, and they have been at it for several years.

And customers? Some of the major Hadoop purveyors have dozens, in some cases close to triple digits. Not exactly impressive, until you run down the list. Yahoo (which began the whole shebang as a way to help its now forlorn search engine) has the largest Hadoop cluster around at more than 42,000 nodes. And I met someone else who has a mere 30-node cluster: he was confident by this time next year he would be storing a petabyte on several hundred nodes. That’s a thousand terabytes, for those that aren’t used to thinking of that part of the metric system.

Three years ago I would have told you to teach your kids WordPress, but that seems passé, even quaint now. Now even grade schoollers can set up their own blogs and websites without knowing much code at all, and those who are sufficiently motivated can learn Perl and PHP online. But Hadoop clearly has captured the zeitgeist, or at least a lot of our data, and it poised to gather more of it as time goes on. Lots of firms are hiring too, and the demand is only growing.

Infochimps has some great resources to get you started here >>

David Strom is a world-known expert on networking and communications technologies. Whether you got your first PC at age 60 or grew up with an Apple in your crib, Strom can help you understand how to use your computers, keep them secure, and understand how to create and deploy a variety of Internet applications and services. He has worked extensively in the Information Technology end-user computing industry and has managed editorial operations for trade publications in the network computing, electronics components, computer enthusiast, reseller channel and security markets.

6fefa857 2e95 4742 9684 869168ac7099 Tell Your Children to Learn Hadoop

Image Source:

Is Big Data the Tail Wagging the Data Economy Dog?

Business Dog 300x213 Is Big Data the Tail Wagging the Data Economy Dog?Segmenting the overall IT market horizontally typically results in five sub-markets: Semiconductors, hardware, software, telecommunications, and professional services.  But an anomaly buried in the usual segmentation has existed for several decades, glossed over because it was such a slender slice of IT.   That hidden slice has widened considerably post-2000 however, and the time has come to give those IT suppliers, for want of a better term we will call them “data providers,” their fair due – recognition of their own market space which I refer to as the “Data Economy.”

Even though many data providers are not-for-profit, if one aggregates the revenues of all the data providers the “Data Economy” market now runs in excess of $100 billion in annual revenues.  By comparison, ESG estimates the software and core services revenues associated with the BI-Analytics platform market at around $20 billion.  Even if you add all the adjunct products and services required for big data, such as servers, storage, networking and professional services, it probably still slightly trails the Data Economy in terms of market size.  And ESG believes the Data Economy is growing even faster than big data.  Who are these data providers?  Let’s barely scratch the surface of some of the Data Economy players.

You are probably familiar with some of the world’s largest data providers like the multi-billion dollar Acxiom and Lexis Nexis.  Unless you pay close attention to the securities arm of the financial services industry you may not have heard of Interactive Data, a nearly $1b firm, and similarly if you are quite interested in channel data you might have heard of Zyme.  Not all data providers focus on a particular industry or role, for example DataLab USA offers data spanning insurance, credit, healthcare and real estate.  If you have ever been wondering about how best to classify industries to optimize search you might try WAND, and of course the U.S. Department of Labor’s Bureau of Labor Statistics will help you with understanding job taxonomies and data thereof.  And if you really want to span the globe in terms of data you might want to start at which acts as a portal for governmentally-sourced data from 39 states, 41 other countries, and a host of other governmental organizations, as part of the movement to “democratize data.”  The Open Archives Initiative is another data democratization example.  While not-for-profits are important participants in the Data Economy, the United States Postal Service offers data products for a price to help try to offset its rather notorious non-profitability.

Most data providers don’t simply provide data.  Particularly commercial providers like Lexis Nexis offer a variety of products for understanding the data they offer, in terms of data attributes, how to best ingest and use the data, and even tools to perform data analysis a la big data.  Almost all data providers offer information about the metadata, or at least how to interpolate the metadata, for the data they distribute.  Data providers generally gather, aggregate, qualify, refine and distribute (preferably with value-added ease) data.  Data has been referred to as “the new oil,” and while I might extend the metaphor to all kinds of mining and agricultural activities as well, the basic idea that data increasingly acts as the caloric source for an increasing number of modern pursuits, business, governmental, and consumer, is the fundamental driver behind the Data Economy.

If you are a business professional, or a data analyst, or a CIO, why should you care about the Data Economy?  First, your competitors may have already jumped ahead of you by tapping into the Data Economy.  For example, if you are highly dependent on channel partners, but have little visibility as to their performance in terms of reselling your products other than some simple monthly reports and word-of-mouth, you may be over or under-investing in various channels.  Your competitor, however, working with the aforementioned Zyme, might have a far clearer grasp of what is and isn’t working in the channel, and is making workflow and investment decisions accordingly.  In that example if you are not plugged into the Data Economy, what you don’t know may indeed be hurting you.

As a data analyst, you, with help from your IT department, may have done a fantastic job culling all the internal data available for business intelligence and analytical purposes.  However, that internal data may lack context, or perhaps could be further enriched with 3rd party data.  Some big data BI-Analytics platform vendors, like Alteryx, make it really easy to tap into data providers by offering relevant built-in data services from those providers.  To the data analyst using such a feature the 3rd party data looks just like an internal data source – except you may have to pay for the data.  Regardless, however, the data analyst or scientist who regularly scans and potentially uses external data for business intelligence and analytical model development is using a best practice.  Data analysts who only look to internal data sources are potentially overlooking major insight opportunities.

CIOs should think of the Data Economy as another external resource, like public cloud or professional services, which may be brought to bear to help the IT department deliver the best possible information technology for the business.  If the CIO has a CDO, Chief Data Officer, the CDO should certainly track potential external sources for the data needs of the business.  If the business has a CDO, Chief Digital Officer, that CDO should likewise be tapping into, as applicable, 3rd party data, and perhaps should consider using the company’s internal data as a revenue-generating asset; perhaps your company could be a data provider in the Data Economy too.

The big data movement has largely been technology based.  But the innovation for much of the core technology for big data, like Hadoop, was originally developed by Web 2.0 companies, like Yahoo and Google, for business purposes.  BI-Analytics platforms offer the tools to gain deeper insights into business performance, market opportunities, research and development, and customer understanding.  But they are merely the tools, like an automobile.  The fuel they need to run on is data, and increasing that data will come from outside of the firewall.   For IT departments, being your organization’s steward of technology is no longer enough.  IT and its data professional partners in the lines of business also increasingly carry the responsibility for ensuring that the company has ALL the right data, from inside and outside, to help guide the business from daily tactical operations through strategic decision-making.

Evan Quinn is Senior Principal Analyst at Enterprise Strategy Group (ESG), an integrated IT research, analysis, and strategy firm that is world-renowned for providing actionable insight and intelligence to the global IT community.

6fefa857 2e95 4742 9684 869168ac7099 Is Big Data the Tail Wagging the Data Economy Dog?

Image source: