elemental links

Business Ecology: Optimization for Innovation

As I mentioned in my 2010 plans, one of my projects this year is writing and advocacy for the OMG’s Business Ecology Initiative (BEI).  This morning, we launched the new Business Ecology Initiative blog, and I published the following overview post on Business Ecology, the Business Ecology Initiative and Business Ecology enablers. 

As you read the post, you’ll notice several themes common to my on-going writings (soapboxes), which is why getting involved with the BEI is a no-brainer for me.  The original post follows.

What is Business Ecology?

Business Ecology is a business-technology imperative focused on streamlining business processes, removing waste from technology portfolios, and adjusting resource consumption, to optimize business operations and foster business innovation.

As the world economy emerges from a painful recession, organizations are confronted with the challenge of retaining bottom-line diligence, while pursuing market sustaining and gaining innovation. 

For many organizations, the answer lies in harvesting savings and trapped value from existing processes, resources and capabilities.  To accomplish this, organizations are turning to Business Ecology. 

Business Ecology is not a one-time fix, but rather a management philosophy concerned with business vitality over time, balancing current conditions, optimization and innovation focus areas, resource allocations, and longer-term business motivations, capabilities and outcomes.

An important enabler of Business Ecology is the use of technology beyond automation. Business Ecology practitioners employ technology to identify, measure, model and drive business change. 

Business Ecology adoption and execution requires a cross functional team, comprised of business and information technology professionals. Key team capabilities include business process thinking, business performance measurement and analysis, financial analysis, IT architecture, portfolio management, service delivery and iterative project management.

Successful Business Ecology initiatives adopt the principles and values of Business Technology.  The team communicates with each other, sponsors and constituents in a common, business based language.  Measurement is defined and reported in business terms.  Funding and governance decisions and mechanisms reflect a shared resource, investment portfolio approach.

As Business Ecology teams progress, technology and business savvy is exchanged, fostering a greater understanding of each other’s challenges, skills and tools, which leads to a break in the longstanding, constraining, business and IT divide.  A byproduct of successful Business Ecology is business-IT integration.

The Chief Information Officer (CIO), given his/her unique position to view business processes, resource consumption, and technology portfolios across the organization, most often champions Business Ecology adoption and execution.  C-level executives, including the CEO, CFO and COO, sponsor Business Ecology initiatives.

What is the Business Ecology Initiative?

OMG, via the Business Ecology Initiative, leads the drive towards Business Ecology. 

The Business Ecology Initiative provides education, advocacy and member programs to enable organizations to achieve Business Ecology success, employ Actionable Architecture^TM, and carve a path to business-IT integration.

Actionable Architecture^TM brings transparent business methodology to the definition and delivery of common IT infrastructure, platforms and services.  Emphasized business attributes include quality, efficiency, compliance, agility, value, effectiveness, ease of use, sustainability and business goal traceability.

Business-IT Integration is the organizational model for business and information technology convergence.  This model promotes collaborative strategy, planning, architecture and execution; shared decision-making, business-tech savvy personnel and service delivery at the point of value generation.   

What enables Business Ecology?

Techniques: LEAN, Six Sigma, BPM, Value Chain Analysis, Actionable Architecture^TM , Business Technology, Agile, Modeling, Simulation, Business Measurement, and Sustainability Analysis

Technology: SOA, BPM, Cloud Computing, Event Processing, Analytics, Master Data Management (MDM), and Open Standards

Measurement Models: Business Process Maturity Model (BPMM), Sustainability Assessment Model (SAM)

People: C-level executives, business and information technology professionals, who embrace the philosophy of Business Ecology

 

[Disclosure: The Business Ecology Initiative is a client of my firm, Elemental Links.]

Julian Hyde in ACM on In-Flight Data Processing with Streaming SQL Technology

Julian Hyde, chief architect of SQLstream, and lead developer of Mondrian, has a great article in Communications of the ACM on Data in Flight.  The article provides an overview of streaming query engines, demonstrates simple queries using a clickstream example, compares streaming query engines to relational database technology, discusses the advantages of streaming, and concludes with additional streaming applications, including CEP.

On streaming query engines vs. relational database technology:

“The streaming query engine is a new technology that excels in processing rapidly flowing data and producing results with low latency. It arose out of the database research community and therefore shares some of the characteristics that make relational databases popular, but it is most definitely not a database. In a database, the data arrives first and is stored on disk; then users apply queries to the stored data. In a streaming query engine, the queries arrive before the data. The data flows through a number of continuously executing queries, and the transformed data flows out to applications. One might say that a relational database processes data at rest, whereas a streaming query engine processes data in flight.”

On CEP and Streaming, Hyde states:

“Application areas include complex event processing (CEP), monitoring, population data warehouses, and middleware. A CEP query looks for sequences of events on a single stream or on multiple streams that, together, match a pattern and create a "complex event" of interest to the business. Applications of CEP include fraud detection and electronic trading.

CEP has been used within the industry as a blanket term to describe the entire field of streaming query systems. This is regrettable because it has resulted in a religious war between SQL-based and non-SQL-based vendors and, in overly focusing on financial services applications, has caused other application areas to be neglected.”

Hyde concludes his article as follows:

“Streaming query engines are based on the same technology as relational databases but are designed to process data in flight. Streaming query engines can solve some common problems much more efficiently than databases because they match the time-based nature of the problems, they retain only the working set of data needed to solve the problem, and they process data asynchronously and continuously.

Because of their shared SQL language, streaming query engines and relational databases can collaborate to solve problems in monitoring and realtime business intelligence. SQL makes them accessible to a large pool of people with SQL expertise.

Just as databases can be applied to a wide range of problems, from transaction processing to data warehousing, streaming query systems can support patterns such as enterprise messaging, complex event processing, continuous data integration, and new application areas that are still being discovered.”

If you are even remotely interested in event processing, active information strategies and/or stream processing, I highly recommend reading Hyde’s article.

Rise of Event Processing / Active Information Picks

Earlier this month, I made my first and only prediction for 2010, that we would (finally) see the “Rise of Event Processing”.  Often, when I speak of Event Processing, I refer to a broader context of Active Information and (when relevant) the establishment of an Active Information Tier.  This past week, I read several articles that are applicable to the event processing / active information space.  I’ve included excerpts and links to 5 of those articles.  [Emphasis is my own.]

Briefly… I was particularly heartened to see the MIT Technology Review article dispel the notion that real-time, and the real-time web, is solely the domain of Twitter and related social media technologies.  The Economist SIS and Fast Company Corventis pieces highlight interesting sense, analyze and respond use cases in the real (physical) world.  The Carol Bartz piece, also in The Economist, discusses leadership traits in the age of information deluge.  Finally, the Progress Software predictions echo my sentiment in “Rise of Event Processing”, which is “You can’t change what you can’t see”.

Rise Event Processing / Active Information Picks:

1. MIT Technology Review: Startups Mine the Real-Time Web, There's more to it than microblog posts and social network updates.

“The "real-time Web" is a hot concept these days. Both Google and Microsoft are racing to add more real-time information to their search results, and a slew of startups are developing technology to collect and deliver the freshest information from around the Web.

But there's more to the real-time Web than just microblogging posts, social network updates, and up-to-the-minute news stories. Huge volumes of data are generated, behind the scenes, every time a person watches a video, clicks on an ad, or performs just about any other action online. And if this user-generated data can be processed rapidly, it could provide new ways to tailor the content on a website, in close to real time.”

… “Richard Tibbetts, CTO of StreamBase, explains that financial markets make up about 80 percent of his company's customers today. Web companies are just starting to adopt the technology.

"You're going to see real-time Web mashups, where data is integrated from multiple sources," Tibbetts says. Such a mashup could, for example, monitor second-to-second fluctuations in the price of airline tickets and automatically purchase one when it falls below a certain price.”

… Real-time applications, whether using traditional database technology or Hadoop, stand to become much more sophisticated going forward. "When people say real-time Web today, they have a narrow view of it--consumer applications like Twitter, Facebook, and a little bit of search," says StreamBase's Tibbetts.”

2. The Economist: The World in 2010 – Big SIS (Societal Information-Technology Systems) is watching you

…“Thanks to Moore’s law (a doubling of capacity every 18 months or so), chips, sensors and radio devices have become so small and cheap that they can be embedded virtually anywhere. Today, two-thirds of new products already come with some electronics built in. By 2017 there could be 7 trillion wirelessly connected devices and objects—about 1,000 per person.

Sensors and chips will produce huge amounts of data. And IT systems are becoming powerful enough to analyse them in real time and predict how things will evolve. IBM has developed a technology it calls “stream computing”. Machines using it can analyse data streams from hundreds of sources, such as surveillance cameras and Wall Street trading desks, summarise the results and take decisions.

Transport is perhaps the industry in which the trend has gone furthest. Several cities have installed dynamic toll systems whose charges vary according to traffic flow. Drivers in Stockholm pay between $1.50 and $3 per entry into the downtown area. After the system—which uses a combination of smart tags, cameras and roadside sensors—was launched, traffic in the Swedish capital decreased by nearly 20%.

More importantly, 2010 will see a boom in “smart grids”. This is tech-speak for an intelligent network paralleling the power grid, and for applications that then manage energy use in real time. Pacific Gas & Electric, one of California’s main utilities, plans to install 10m “smart meters” to tell consumers how much they have to pay and, eventually, to switch off appliances during peak hours.

Smart technology is also likely to penetrate the natural environment. One example is the SmartBay project at Galway Bay in Ireland. The system there draws information from sensors attached to buoys and weather gauges and from text messages from boaters about potentially dangerous floating objects. Uses range from automatic alerts being sent to the harbourmaster when water levels rise above normal to fishermen selling their catch directly to restaurants, thus pocketing a better profit.

Yet it is in big cities that “smartification” will have the most impact. A plethora of systems can be made more intelligent and then combined into a “system of systems”: not just transport and the power grid, but public safety, water supply and even health care (think remote monitoring of patients). With the help of Cisco, another big IT firm, the South Korean city of Incheon aims to become a “Smart+Connected” community, with virtual government services, green energy services and intelligent buildings…”

3. Fast Company: Corventis's PiiX Monitor Promises to Predict Heart Failure

… “The company's first product, PiiX, is a wireless, water-resistant sensor that sticks to a patient's chest like a large Band-Aid and monitors heart rate, respiratory rate, bodily fluids, and overall activity. It transmits the data to a central server for analysis and review by doctor and patient.

The basic technology platform has already received FDA approval, but Corventis envisions the PiiX as much more than a simple monitoring system. The company is working to generate algorithms that can predict, for instance, when a patient is on the verge of heart failure by comparing trends in his or her vital signs to other cases. "When you apply it in the real world, the algorithm begins to learn," says CEO Ed Manicka. "Not from 5 or 10 patients, but from hundreds of thousands of patients, as the system is applied across the planet."

… "What Corventis is trying to do is fundamentally create a new type of machine intelligence that serves to manage the patient's overall health," he says. "It moves from the reactive approach of practicing medicine that is prevalent today to something that is much more proactive, preventative, and individualized."”

4. The Economist: The World in 2010 - Leadership in the information age, by Carol Bartz, CEO of Yahoo!

… “The second obligation that information creates for executives is to identify and mentor thought leaders. In the past, seeking out “high potential” employees typically meant looking for those who could climb the next rung of the management ladder. That remains important. But equally pressing is finding those employees who, though perhaps not the best managers, have the ability to digest and interpret information for others. Grooming these in-house ideas people helps foster a culture of openness to fresh thinking—the greatest energy an organisation can have.

The deluge of information is only going to rise. Leadership will increasingly mean leveraging that information, clarifying it, and using it to advance your strategy, engage customers and motivate employees. Business stakeholders are interested not only in your products and services, but also in your ideas.

So, welcome the information flood. Those who learn how to keep their head above it will be the most effective leaders.”

5. 2010: IT set to move from evolution to quiet revolution, predicts Progress Software

“Based on feedback from customers, as well as its own research and development, Progress Software sees five key technology trends set to shake up computing in 2010.

1. Real-time insight and business control will become a must-have, as organizations can ill-afford to lose money and customer through being slow to notice problems in delivery. In 2009, our research found that 67% of businesses only become aware of problems when customers report them. 80% of companies already have critical business events they need to monitor in real time. In 2010, insight into these events, powered by the right technology, will be essential to success.

2. Event-driven computing will accelerate, driven by business needs, and impacting both the way applications are built and how they are deployed in the enterprise. Architectures are increasingly being built around ‘events’, and this will increase to deal with both new sources of events appearing within the enterprise as well as external event sources from partners and customers.”

2010: The Rise of Event Processing

I don’t typically engage in predictions, but here’s mine for 2010, fresh from my tweet stream:

2010: Event Processing transcends niche status, to well-recognized & adopted business technique for real-time visibility & responsiveness.

I can list tons of reasons why, but it boils down to this:  you can’t change what you can’t see.

Lessons from Googlenomics: Data abundance, Insight Scarcity

“"What's ubiquitous and cheap?" [Google’s Hal] Varian asks. "Data." And what is scarce? The analytic ability to utilize that data.”

The June issue of Wired has an excellent article by Steven Levy, entitled Secret of Googlenomics: Data-Fueled Recipe Brews Profitability.  The article delves into the history and algorithms behind Google’s auction based ad system, highlighting the significance of engineering, mathematics, economics, and data mining in Google’s success.

On the economics front, the article explains Hal Varian’s role as Chief Economist at Google, including why Google needs a chief economist:

“The simplest reason is that the company is an economy unto itself. The ad auction, marinated in that special sauce, is a seething laboratory of fiduciary forensics, with customers ranging from giant multinationals to dorm-room entrepreneurs, all billed by the world's largest micropayment system.

Google depends on economic principles to hone what has become the search engine of choice for more than 60 percent of all Internet surfers, and the company uses auction theory to grease the skids of its own operations. All these calculations require an army of math geeks, algorithms of Ramanujanian complexity, and a sales force more comfortable with whiteboard markers than fairway irons.”

After reading the article, Varian’s economic view of data ubiquity and analytic scarcity really stuck with me.  The quote I opened the post with isn’t directed at software availability or processing power.  It refers to the scarcity of people qualified to churn abundant data into economic value.  

What follows are some excerpts “about harnessing supply and demand”.  The sub-headers and emphasis are mine.

Enter Econometricians

"The people working for me are generally econometricians—sort of a cross between statisticians and economists," says Varian, who moved to Google full-time in 2007 (he's on leave from Berkeley) and leads two teams, one of them focused on analysis.

"Google needs mathematical types that have a rich tool set for looking for signals in noise," says statistician Daryl Pregibon, who joined Google in 2003 after 23 years as a top scientist at Bell Labs and AT&T Labs. "The rough rule of thumb is one statistician for every 100 computer scientists."

Ubiquitous Data

“As the amount of data at the company's disposal grows, the opportunities to exploit it multiply, which ends up further extending the range and scope of the Google economy…

Keywords and click rates are their bread and butter. "We are trying to understand the mechanisms behind the metrics," says Qing Wu, one of Varian's minions. His specialty is forecasting, so now he predicts patterns of queries based on the season, the climate, international holidays, even the time of day. "We have temperature data, weather data, and queries data, so we can do correlation and statistical modeling," Wu says. The results all feed into Google's backend system, helping advertisers devise more-efficient campaigns.”

Continuous Analysis

“To track and test their predictions, Wu and his colleagues use dozens of onscreen dashboards that continuously stream information, a sort of Bloomberg terminal for the Googlesphere. Wu checks obsessively to see whether reality is matching the forecasts: "With a dashboard, you can monitor the queries, the amount of money you make, how many advertisers you have, how many keywords they're bidding on, what the rate of return is for each advertiser."”

Behavioral Based Insights

“Wu calls Google "the barometer of the world." Indeed, studying the clicks is like looking through a window with a panoramic view of everything. You can see the change of seasons—clicks gravitating toward skiing and heavy clothes in winter, bikinis and sunscreen in summer—and you can track who's up and down in pop culture. Most of us remember news events from television or newspapers; Googlers recall them as spikes in their graphs. "One of the big things a few years ago was the SARS epidemic," Tang says. Wu didn't even have to read the papers to know about the financial meltdown—he saw the jump in people Googling for gold. And since prediction and analysis are so crucial to AdWords, every bit of data, no matter how seemingly trivial, has potential value.”

Rise of the Datarati

“Varian believes that a new era is dawning for what you might call the datarati—and it's all about harnessing supply and demand. "What's ubiquitous and cheap?" Varian asks. "Data." And what is scarce? The analytic ability to utilize that data. As a result, he believes that the kind of technical person who once would have wound up working for a hedge fund on Wall Street will now work at a firm whose business hinges on making smart, daring choices—decisions based on surprising results gleaned from algorithmic spelunking and executed with the confidence that comes from really doing the math.”

Now, a few questions I think folks should consider:

1. Who does that math in your organization? 
2. Does your analytics / active information strategy suffer from information processing richness and insight scarcity?
3. Who are, or should be, your datarati?