elemental links

Lessons from Googlenomics: Data abundance, Insight Scarcity

“"What's ubiquitous and cheap?" [Google’s Hal] Varian asks. "Data." And what is scarce? The analytic ability to utilize that data.”

The June issue of Wired has an excellent article by Steven Levy, entitled Secret of Googlenomics: Data-Fueled Recipe Brews Profitability.  The article delves into the history and algorithms behind Google’s auction based ad system, highlighting the significance of engineering, mathematics, economics, and data mining in Google’s success.

On the economics front, the article explains Hal Varian’s role as Chief Economist at Google, including why Google needs a chief economist:

“The simplest reason is that the company is an economy unto itself. The ad auction, marinated in that special sauce, is a seething laboratory of fiduciary forensics, with customers ranging from giant multinationals to dorm-room entrepreneurs, all billed by the world's largest micropayment system.

Google depends on economic principles to hone what has become the search engine of choice for more than 60 percent of all Internet surfers, and the company uses auction theory to grease the skids of its own operations. All these calculations require an army of math geeks, algorithms of Ramanujanian complexity, and a sales force more comfortable with whiteboard markers than fairway irons.”

After reading the article, Varian’s economic view of data ubiquity and analytic scarcity really stuck with me.  The quote I opened the post with isn’t directed at software availability or processing power.  It refers to the scarcity of people qualified to churn abundant data into economic value.  

What follows are some excerpts “about harnessing supply and demand”.  The sub-headers and emphasis are mine.

Enter Econometricians

"The people working for me are generally econometricians—sort of a cross between statisticians and economists," says Varian, who moved to Google full-time in 2007 (he's on leave from Berkeley) and leads two teams, one of them focused on analysis.

"Google needs mathematical types that have a rich tool set for looking for signals in noise," says statistician Daryl Pregibon, who joined Google in 2003 after 23 years as a top scientist at Bell Labs and AT&T Labs. "The rough rule of thumb is one statistician for every 100 computer scientists."

Ubiquitous Data

“As the amount of data at the company's disposal grows, the opportunities to exploit it multiply, which ends up further extending the range and scope of the Google economy…

Keywords and click rates are their bread and butter. "We are trying to understand the mechanisms behind the metrics," says Qing Wu, one of Varian's minions. His specialty is forecasting, so now he predicts patterns of queries based on the season, the climate, international holidays, even the time of day. "We have temperature data, weather data, and queries data, so we can do correlation and statistical modeling," Wu says. The results all feed into Google's backend system, helping advertisers devise more-efficient campaigns.”

Continuous Analysis

“To track and test their predictions, Wu and his colleagues use dozens of onscreen dashboards that continuously stream information, a sort of Bloomberg terminal for the Googlesphere. Wu checks obsessively to see whether reality is matching the forecasts: "With a dashboard, you can monitor the queries, the amount of money you make, how many advertisers you have, how many keywords they're bidding on, what the rate of return is for each advertiser."”

Behavioral Based Insights

“Wu calls Google "the barometer of the world." Indeed, studying the clicks is like looking through a window with a panoramic view of everything. You can see the change of seasons—clicks gravitating toward skiing and heavy clothes in winter, bikinis and sunscreen in summer—and you can track who's up and down in pop culture. Most of us remember news events from television or newspapers; Googlers recall them as spikes in their graphs. "One of the big things a few years ago was the SARS epidemic," Tang says. Wu didn't even have to read the papers to know about the financial meltdown—he saw the jump in people Googling for gold. And since prediction and analysis are so crucial to AdWords, every bit of data, no matter how seemingly trivial, has potential value.”

Rise of the Datarati

“Varian believes that a new era is dawning for what you might call the datarati—and it's all about harnessing supply and demand. "What's ubiquitous and cheap?" Varian asks. "Data." And what is scarce? The analytic ability to utilize that data. As a result, he believes that the kind of technical person who once would have wound up working for a hedge fund on Wall Street will now work at a firm whose business hinges on making smart, daring choices—decisions based on surprising results gleaned from algorithmic spelunking and executed with the confidence that comes from really doing the math.”

Now, a few questions I think folks should consider:

1. Who does that math in your organization? 
2. Does your analytics / active information strategy suffer from information processing richness and insight scarcity?
3. Who are, or should be, your datarati? 

Conversation with Steve Goldman of the CME Group on CEP as Enterprise Platform & StreamBase

Late in May, Mark Palmer, CEO of StreamBase, piqued the event processing community’s curiosity with this tweet: “Today I signed what I think is the most exciting CEP deal of 2009 - corporate selection by a household name…”. 

While many household names use Complex Event Processing products, the products are acquired solve a particular business problem, or perhaps, a handful of scenarios within a business unit.  In his tweet, Mark signaled an adoption pattern shift, from CEP as application enabler, to CEP as enterprise technology platform. 

For the event processing community -- vendors, researchers, early adopters and advocates -- this shift has been long overdue.  Of course, as a fact based community, we require a little more information than a 140-character tweet.

That information became public this week, as StreamBase announced that the “household name” is the CME Group:

“StreamBase today announced that CME Group, the world’s largest and most diverse derivatives exchange, has selected StreamBase Complex Event processing solution for enterprise-wide deployment.  After a comprehensive evaluation, CME Group chose StreamBase as its internal standard Complex Event Processing (CEP) platform, and will be initially deploying it for their options pricing applications.

“CME Group is one of the most demanding technology environments in the world, processing millions of orders a day in milliseconds, and disseminating market data in a reliable and low latency manner is critical to our customers,” said Steve Goldman, Director, Enterprise Architecture, CME Group.  “Their high-performance multi-threaded server and easy to use modeling tools met our requirements and will enable the exchange to quickly react to the ever changing needs of our customers.”

…As an international marketplace, CME Group brings buyers and sellers together on the CME Globex electronic trading platform and on trading floors in Chicago and New York. By acting as the buyer to every seller and the seller to every buyer, CME Clearing virtually eliminates counterparty credit risk. CME Group offers the widest range of benchmark products available across all major asset classes, including futures and options based on interest rates, equity indexes, foreign exchange, energy, agricultural commodities, metals, and alternative investment products such as weather and real estate. More information can be found at www.cmegroup.com or via Twitter @cmegroup.”

Earlier this month, I had the opportunity to speak with Steve Goldman, Director of Enterprise Architecture at the CME Group, about event processing at the CME Group and their selection of StreamBase.   We had a great conversation that substantiated Mark’s proclamation “of the most exciting CEP deal of 2009”. 

One administrative note before I jump into the highlights from our conversation.  What follows are an edited and summarized version of my notes from the call.  In other words, these are not direct quotes. 

Business Scenarios

The CME Group’s first StreamBase use case is generating options settlement prices.  The daily settlement process involves complicated calculations based on a number of market data feeds.  For an idea of the complexity and product line variations, here are some details from the CME’s Daily Settlement Procedures (pdf):

“Equity Options: Exchange staff identifies “seed strikes” that include the at-the-money straddle and several out-of-the-money calls/puts. The midpoints of the bid/ask quotes in the seed strikes on Globex are used to create an implied volatility skew. The skew is adjusted based upon the underlying settlement price to automatically generate the out-of-the money settlement prices, and the in-the-money options are settled automatically, using the method referenced on page 4 of this document. For longer dated options for which no Globex data exists, market participants provide bid/ask data for the seed strikes. Adjustments may be made to incorporate relevant pit data.

Non-Treasury Interest Rate Options: Similar to the procedure used in equity options, settlements in the front year of expirations are generated based on the skew derived from taking the midpoint of the bid/ask quotes in Exchange-designated seed strikes from the pit and from Globex. The skew is adjusted based upon the underlying settlement price. The additional guidelines referenced on page 3 of this document are also utilized. All other contract months are settled by Exchange officials based upon input from market participants.

Agricultural Options: Market participants provide quotes in Exchange-designated seed strikes which are used to generate the implied volatility skew and the skew is adjusted to the underlying futures settlement price. Dairy products are settled using a flat volatility determined by the at-the-money straddle.

Weather Options: Option trades are converted to “standard deviations” using a model based on Stephen Jewson’s model for pricing Weather. This standard deviation creates prices in the entire options series which is then applied to the open strikes.

Housing Futures and Options: The futures are settled to the last trade or better bid/offer on Globex. Absent a trade or better bid/offer, the prior day settlement is used. The options are settled using volatility skews derived from the midpoints of the bid/ask in a given strike, tied to a futures level.

Metal Options: Exchange officials, in consultation with market participants, establish the at-the-money volatility and create the volatility surface for the out-of-the money puts and calls for all option series based on traded/quoted outrights and spreads, which is entered into an options pricing model to determine the settlements for all strikes. Settlements may be adjusted in accommodate relevant orders.”

[For more on the CME Group’s business, see Mark Palmer’s Innovation by the Numbers post.]

Event-Driven Organization

Goldman shared that the exchange has been an event-driven organization for a long time, at least since they began electronic trading.  Goldman described CEP as the epitome.  CEP introduces an engine to process thousands and thousand of real-time events, with a simple way to instruct the engine on what to do with those events.

Goldman emphasized the productivity benefits for business users.  Business users will be able to build, dynamically change and test models.  Once the business scenario is resolved, the business hands off the models to technology personnel who focus on implementation aspects, such as scale, reliability and monitoring.

[Weather Options Settlement Example in StreamBase, Click on Picture to enlarge]

 

By adding StreamBase, they now have a powerful and flexible tool to work with market data. To maximize this flexibility, the solution is being architected to receive all market data within the exchange, as well as many external data sources. 

Future use cases include real-time risk analysis and the margining aspect of the business.   

Selection Process

In respect to the selection process, Goldman spoke of mature enterprise architecture practices and deep business participation.  They started by developing an enterprise architecture framework that looked into the entire settlement process.  This resulted in a design, which ultimately led to Complex Event Processing.

Goldman outlined an evaluation process that continually narrows the field via introductory briefings, RFI responses, follow-on meetings, proof-of-concepts, gap analysis, and business terms.  During the CEP evaluation, the CME Group looked at four vendors, and ended with two finalists.

The team determined that both finalists could do the job, meeting functional, performance, scale and monitoring requirements.  Ultimately, the usability of the StreamBase Studio won the day.

The product’s ease of use, Goldman believes, also contributed to the business team’s deep engagement in the proof-of-concept and involvement in the final decision-making. 

Return on Investment

Goldman projects the CEP engine investment will pay-off in less than a year.  The alternative to purchasing a CEP engine was a custom solution.  A custom solution would have required more development time and delayed the introduction of business capability, which the CME Group needs now. 

In addition, a custom solution would have included manual processing and “taped together” third party tools.  Besides cost and time, this path introduces more opportunities for error.

Real-time World

Speaking to opportunities outside of capital markets, Goldman spoke of the importance of real-time business in an increasingly real-time world.  The ability to see and process orders, data, risk and regulatory compliance in real-time ultimately results in more business.  More business results in more profits, now.

 

[Disclosure: StreamBase is not a client of my company, Elemental Links.  Nor do I have the skill to trade on the CME Group’s exchanges.]

@ Enterprise 2.0 Evening in the Cloud Panel discussion

[update: 6.23.2009 - "MIT CIO Symposium Organizer" is Christopher Reichert]

David Berlind is our Evening in the Cloud host.  David says the discussion shouldn’t be about cloud computing definition, it should be about cloud computing benefits.  The benefits will lead to the ‘right’ definition.

Evening format is a panel discussion, followed by networking and speed geeking demos to win over our virtual $1 million.

I’ll blog the panel tonight.  Probably offer commentary on the speed geeking via twitter and follow-on posts.

Panel Format, each panelist has 8 minutes to “pitch us” as though they were visiting our organization.

Mike Feinberg, Senior VP, Cloud Infrastructure, EMC

Mike ask us to imagine: Infinite on-demand resources, economies of scale, servicing multiple industries

Commitment that is proportional to investment.  Can IT organizations deliver cloud architectures?  Is cloud synonymous with Internet access?

Customers are concerned with putting 100% of data in external cloud; customers want choice, control & flexibility, one size doesn’t fit all. 

There will be a federation of Internal & External clouds.  “internet access to capability and local access to capability”

Cloud computing is a storage and data problem.  If you are going to compute on something, need access to data and information. Refers to Atmos product.  Atmos has policy control to annotate data – confidentiality, etc.  Policy used to manage where data is stored – Atmos in datacenter vs. online.

Not a statement about internet access, but architecture.

Rajen Sheth, Senior Product Manager (and Inventor), Google Apps

3 questions he is frequently asked:

1. what applications should i move to cloud?

2. what does google offer?

3. is anyone doing this?

benefits from Google are scale, innovation, cost.

4 layers: infrastructure, platform, identity & administration, applications

calls out fallacy of just moving existing applications to the cloud and being able to take advantage of all cloud capabilities

Google started with applications: gmail, talk, calendar, docs, sites etc. 

Extended google platform to allow people to run their own applications – Google AppEngine & Web Toolkit; Gadgets; Google Secure Data Connector – secure bridge between Google cloud & enterprise, or other clouds

Value starts at Google Platform

Examples:

- Presidential Town Hall meeting, question submission, huge spike at submission deadline

- Cast Iron’s 360 degree customer view – pulled from Salesforce, Remedy

Companies that deployed apps – slide with 12-20 logos, not Fortune 500 companies

Sean Poulley, VP Online Collaboration Services, IBM

Agree with Rajen, running your infrastructure is a drag.  Slide of IBM in the cloud, working with customers on private and public clouds.

“The cloud is a new way to solve old problems”.  Problems won’t go away.  Need to protect customer’s investment. 

What is shifting to the cloud?

- workloads move to the cloud – analytics, collaboration, desktops

- cloud systems – IBM Cloud Burst – hardware, networking, systems management, virtualization for enterprises to build private cloud that has many of the capabilities of public cloud

Walking through LotusLive as intercompany collaboration platform: web conferencing, collaboration, email.  Good example for Enterprise 2.0 crowd.

Closing with IBM as trusted partner – remember, they are supposed to be pitching.

“Customer Response”

Besides the audience, the panel has two customers who will now respond to the pitches.

Q: Christopher Reichert, MIT CIO Symposium Organizer, missed name: What’s your take/response to the McKinsey study on true costs of cloud?

Mike, EMC: confusing internet access with architecture. cloud is interesting – cost savings from efficiency and late binding.

Sean, IBM: reality is not all applications will move.  Need to make economic decision, just like always.  Not just economic though, need to consider business policy, risk management trade-offs.

Doug Cornelius, Chief Compliance Officer Beacon Capital Partners – self described as “the no guy”. 

Doug’s question, “How do you deal with the geography issues” – legal, regulatory aspects of where cloud (data) is physically located.

Rajen, Google – absolutely an issue.  What will be interesting is when/how policies, law catch up with technology.  Listed a ton of policies, governmental organizations Google works with to ensure compliance.

Mike, EMC: Policy approach of Atmos allows organizations to manage this.

Sean, IBM: Tricky, technology is moving faster than the law.  Always been issues of selling/deploying technology to other countries based on U.S. policy.  Everything has cost, benefit, risk equation to be evaluated.  If you want to be ultimately safe, you would do any of this stuff, nor get out of bed in the morning.

Doug Cornelius: Records management, really talking about records destruction.  Really destroyed, not on backup tapes somewhere.  Concerned about redundancy, good for back, bad for records destruction.

Sean, IBM: No CIO in right mind would not be looking at Cloud.  Compliance though is the big concern.  This often drives private clouds.  Existing data centers are often chaotic.  There is opportunity to create “cloud environment” internally, reduce data center sprawl.

Mike, EMC: IT knows how to run datacenter, but the question is how efficient is that data center running. Opportunities for both internal efficiency and taking advantage of public capabilities.

Q: issues moving data to cloud –

General discussion that we (the industry) have been talking about this for 7-8 years, however conversation has changed, not technology issue as much as philosophical/trust issue. 

Rajen, Google mentions: security, legal contracts/data ownership, policy setting & enforcement

Q: Christopher Reichert, MIT CIO Symposium: Three bands of cloud adopters – small companies, little data to shift, large companies that have means to outsource the cloud shift (applications and data), middle group, don’t have means to shift, have critical, yet poorly performing applications.

Now I feel like I’m in vendor meeting, the question asked isn’t being answered (yet).

Rajen says good question on midmarket.  Calls out Gartner research that suggested organizations not move data when move email to cloud.  Interesting trend in midmarket is that companies just do a complete switch over.  Obviously, doesn’t work for all types of applications.

Also mentions standards based Google App Engine, development language familiarity eases adoption.

Doug Cornelius: If cloud vendor doesn’t know SAS 70, talk to someone else, if can’t negotiate terms of service, throw them out of room.  See this post of Doug’s for more tips on clouds and compliance.

@ Enterprise 2.0 Cloud Roadmaps Panel

Jake Sorofman, rPath, James Duncan, Joyent and Chet Kapoor, Sonoa Systems chat with Alistair Croll on the futures of cloud.  These companies offer software, products that are adjacent to, or run on, the cloud.  They are not cloud operators.

Jake: rPath is adjacent to cloud.  Platform to package applications for deployment and automated maintenance for many environments, inclusive of clouds.

Chet Kapoor: Visibility, Control and Scale for APIs, feeds and services.  Sonoa Systems provides technology for providers and consumers of cloud services.

James Duncan: Company had internal IaaS cloud, took that knowledge and packaged as company called Reasonably Smart, which was acquired by Joyent. 

Q: VM/AMI Standards?

Chet – standards need to move to higher level, not just VM level, but up at layer 7.

James – To predict the precise mechanism, shape or form, it’s too early.  Agrees they need to be higher level.  Standards need to reflect innovation, not run ahead.

Chet – It’s hard to get momentum around standard without real-life adoption (working code as example).  Doesn’t want to see WS* again. 

Q: Enterprises want to dip toe-in, how to do that, without being in “big cloud”.

Brings up the discussion that cloud technology is not limited to “off-premise” use.

Chet – The next 12-18 months need to resolve security concerns.  For enterprise adoption.

James – Security in the cloud is a trust issue, rather than a security issue.  Over next 12 months there will be increase in familiarity, not increase in security related products and protocols.

Q: How much of cloud computing is intrinsically linked to the client?  Organizations will want to have some code on client.

James – developers, engineers are exposed to the costs of what they are doing like never before.  That visibility will be even more granular over time.  What is the cost of any given API, or URL?  Clients will become more important in that realm.  Costs will be factor in software engineering.

Chet – RIA client to DB picture is wrong.  RIA client will fan out calls to dozens or hundreds of applications 90% of the time, only 10% will be RIA client to DB.

Will every cloud eventually be a service cloud?

Chet – comes down to one question, “is the organization comfortable sharing fate”.  What business are they in, what is their familiarity?

Jake – Companies will always want to package up the applications that contain “their secret sauce”.  These applications will never be developed on PaaS. 

Q: When all clouds are interoperable, doesn’t economy of scale win?

James – obvious comparison we try to make is to electricity.  A better commodity to compare is to is corn, pork bellies, something that needs to be moved around.  Excepting the “rot factor”.

Jake – Electricity moves more rapidly that data, but not efficiently.

Chet – Customers will self-select based on business need, application portfolio, etc.

Chet – how many enterprise CIOs actually write checks to Amazon, not many at all.  Enterprises more often use a Rackspace.

Jake – Sees use case specific cloud deployment.  Metadata enriched policies will be packaged with applications stating where that application can run.

James – Governance, Service Desk, ITIL model is concern in enterprise cloud adoption.

Q: Is data portability the new Open Source debate?

Chet – Open Source resolved accessibility problem, not “free as in beer”.  The bazaar model is far better than cathedral, offers more innovative ideas.  Customers don’t get up in the morning thinking about data portability, or getting down to 5 vendors.  Customers think about who they are betting on, and how to manage those relationships if something goes bad.  [Chet ran Glue Code prior to sale to IBM]

Management Issues

Jake – cloud will reduce barrier to application deployment.  Number of units deployed will go through the roof.  That will make the number of units to be managed go through the roof.  Tendency to deploy and forget.  This is a problem enterprises need to face.  Throwing people at the problem isn’t the solution.

One big prediction – 12 Months

James – Governance.  Governance around acquisition.  Take all those Amazon EC2 charges off of developer’s credit card expense report.

Chet – Familiarity will happen over the next 12 months.  This is as big, or bigger, than web computing.  Will see enterprise adoption.

Jake – Will see examples of high profile applications leaking to the cloud.  Will see some rogue applications, breaches happen.  Will force controls, formalized strategies, governance.

@ Enterprise 2.0: Alistair Croll on Moving to the cloud

Which cloud you go to, depends on what you are moving.  Move machines, code, processes or content.  This is the clearest way to determine what type of cloud an operator is offering, ask them “what do I move to you – machines, code, processes or content”.

Alistair is qualifying “move”, in that some code might need to tweaked for the features/functions/services of the cloud, or perhaps re-written in the case of tightly coupled legacy code.

Clouds as a utility

Nick Carr, Big Switch.  Common need set, not a differentiator, economies of scale.  When these things happen, this pattern exists, you get a utility model.

Alistair says the real question is, “is computing like electricity?”  Now referencing Chris Anderson’s to be released book, Free.  These points are consistent with Alistair’s Interop talk, which I blogged about previously.

One aspect of utility is decreasing cost.  Another is degree of elasticity. 

Computing is not like electricity, because (according to Forrester’s James Staten) no interoperability standards, and only fits loosely coupled applications (SOA). 

Other reasons from Alistair, can’t have electricity breach, electricity doesn’t have to be monitored for HIPAA compliance, electricity doesn’t lock you in (plugs, interfaces).  Electricity outages have easy back-up plan, batteries and generators.  Electricity doesn’t suffer from utility sprawl. 

Issues with cloud computing

Alistair says “cloud bursting is nonsense”.  (IBM won’t like that).  It’s all about the data.  The biggest cost is moving the data.  It’s also a time issue.  Can the data be moved quick enough to support unplanned spike?  If not, data needs to be kept synched with cloud (burst destination).  This is expensive.  Might require all data to be moved to hosted environment connected to the cloud.

Who owns the metadata?  Typically, it’s the cloud provider.  This is issue when crowd sourcing results in metadata that provides value to your business.

Too much choice right now.  A lot of people are sampling, but not a lot are buying.  Alistair implies there will be more buying when there is less choice, because industry will be settling out.

Cloud can dump you.  Cloud provider has freedom in relationship, can discontinue service even if consumers hasn’t broken terms-of-service.

Taxation issues.  State of NY wants to tax transactions done in environments hosted in NY, even if business (cloud consumer) isn’t subject to NY tax.  As a former retailer, I’d call this Nexus.

What is you are a Cloud?  Has the ship already sailed?

If you aren’t thinking in terms of shipping containers, instead of racks, you shouldn’t be considering cloud computing to compete purely on economy of scale.

Beyond shipping containers, can you build by dam, or buy nuclear power plant.

Answer, specialize.  One potential market, legal drivers that force local (geographic) data storage.  Canadian government won’t store data, use applications, in US based clouds.  (Patriot Act implications)

Others: offer different contract terms, user interface, support – more help to onboarding.  Niche markets. 

Cloud Futures – Why Clouds are Cool

Clouds are cool not for themselves, but for what flows over it.

Enterprises today think of clouds as peripherals.  Something on the edge, that you plug in.

Enterprises need to think about disruptive aspects – perform tasks couldn’t before, speed up the organization, etc.

What will matter more as computing becomes less expensive:

- data synch

- portability

- always connected

- transcoding in the cloud

- situational awareness

Alistair gives example of how cloud computing, and the ability to have split-second auctions, will be the downfall of NYC taxi medallion program in favor of “just-in-time” town car arrangements.

He says this is one example of how cloud computing will change our everyday lives. 

[I’m not convinced that it’s cloud computing that changes everything.  I think the internet, cellular, smart handhelds, constant connectedness changed everything.  The cloud is an extension of prior works.  But, not all internet applications need to be cloud resident.]