elemental links

Technical Debt Management: Pay the Interest or Pay down the Principal?

Let’s face it, Enterprise Architects and CIOs don’t always see eye-to-eye.  In my experience, the most common point of contention is “get it right” versus “get it done”.  And of course, this is never an academic conversation, it’s typically a hasty phone call, email or conversation trigged by a “new finding” at a “crucial moment” for a “critical project”. 

At the risk of over generalizing, in these “emergencies”, the deciding factors are time, short-term cost, security risk and short-term technical risk (availability, reliability).  So, if bypassing the architecture gains time, but creates a security or short-term technical risk, then no bypass is granted.  But, if these risks aren’t a factor, well, “get it done”. 

The problem, of course, is that most bypasses are never revisited, and accumulate as expensive, high risk, Technical Debt.  Even worse, most organizations don’t recognize the existence of, nor manage, that technical debt. 

So, how do you raise awareness about mounting technical debt, without being labeled “technology purist”?  Use the inherent “debt metaphor” to build your case in business terms.  Start with a point from Martin Fowler’s recent Technical Debt Quadrant Bliki entry:  “The decision of paying the interest versus paying down the principal…”

The interest is the recurring cost of maintaining a subpar solution that is outside the architecture.  The principal is the subpar solution itself.  Principal is paid down by investing resources to remediate the subpar solution.

Organizations with existing technical debt, need to add technical debt management practices to periodically review the costs and risks of continuing interest payments, versus the costs, risks and opportunities of paying down the principal. 

And for those organizations staring down a “new finding” at a “crucial moment” for a “critical project”, add technical debt to your decision making.  Can you afford to incur additional technical debt?  If so, for how long?

“Pay the interest, pay down the principal, manage your credit line.”  That sounds like a good conversation starter to me...

@ Virtualization, Cloud Computing & Green IT Summit: Report from Trenches: What’s Working in Virtualization & Green IT

Reports from the Trenches: What’s Working in Virtualization and Green IT, moderated by Larry Hale, Director, Office of Infrastructure Optimization, GSA

Panelists:

• Jack Baxter, Manager, IT&S, Government Printing Office
• Richard Fichera, Director, Blade Systems Strategy, HP
• Bernard Golden, CEO, HyperStratus
• Dale Wicklizer, US Public Sector CTO, NetApp

Larry Hale has some starter questions for the panel:

1. Biggest challenges in adopting virtualization?

Jack Baxter: Greatest challenges: application qualification, funding, hardware and how it’s going to be used.  Heterogeneous environment calls for a lot of up-front research.

Dale: Moving to cloud computing requires a mental shift.  Iron huggers need to switch from managing stuff to managing service-level agreements.

Richard Fichera: Distinction between virtualization and cloud.  Virtualization is well adopted.  Cloud is very early.  Adding to Dale and Jack, there are internal process barriers.  Applying current processes to virtual machine environment will lose advantage of shared service environment. 

Bernard Golden: Breaking down of traditional IT silos, forces people to work together who never have, need to optimize / change vertical processes for horizontal view.

2. Is there one government agency that best illustrates benefits of virtualization?

Dale: Federal government is Navy / Marines.  In rest of public sector, great work going on in universities.  Grant providers don’t want money going to IT, want it applied to research.  Universities are starting consortiums of shared IT resources, using virtualization.

Richard: School district work as well.  Virtual desktops that follow students around school.  Others that provide shared resources across schools in a district.

Bernard: Refers to prior session, says Casey Coleman was modest in her description of Apps.gov, and he points to Apps.gov as the best example.

3. Role of virtualization in government in next 5 years?

Jack: Downside of virtualization is ease of creating a server, results in server sprawl.  Still need a datacenter plan and management.  Calls out VMWare as an important management tool.

Bernard: Virtualization will be the defacto way servers work.  Will be packaged with every server and default configuration.

Dale: Most organizations will have hybrid cloud environment.  Legacy will remain as is.  Core, mission critical applications will run on internal (private) cloud.  Other mission applications will run on public clouds.

Richard: Not focused on government.  Financial services industry is further ahead, virtualization is mainstream in commercial space.  Emphasizes he is saying virtualization, not cloud.  There is still a layer of physical machine management.  In future, integrated management of these two control planes. 

Need to think about overall reliability, not reliability of single elements.  This can be achieved with a collection of cheaper infrastructure, don’t need to rely on expense high availability infrastructure.  But, need to plan and manage for overall (application layer) reliability, not physical elements.

Dale: Points out that cheaper infrastructure models, such as Google and Amazon, require more physical infrastructure and creates server (and therefore datacenter) sprawl.  This is fine for users of Google and Amazon, but not necessarily for private datacenters and private clouds. (Please note, Dale is with NetApp).

Richard: Coming to Dale’s defense, reminds everyone that infrastructure from vendors like NetApp is not nearly as expensive now, as it has been.  This price efficiency will only increase.  At the same time, systems integrators will become more sophisticated in delivering low cost infrastructure solutions.  [Of course, you need to pay the SIs].  [Please note, Richard is with HP]

Bernard: Going forward, applications will be written with different assumptions, one of which is that hardware costs won’t be constraining. 

4. Primary security considerations with virtualization?

Dale – starts with cloud, not virtualization.  Calls out Terremark and FISMA certification.  States that many virtual private cloud environments have better security than organizations currently have in place, on premise.

Bernard – In addition to FISMA, organizations need to be concerned with privacy regulations.  Privacy regulations are not up-to-date with technology advances.

Richard – The “spectacular security breaches” in cloud computing will have same root causes as today, passwords on post-its, using dog’s name for server password, etc.

The panel continued with a good discussion on virtualization, product or journey.  The answer, as you know, is journey.  The steps expressed, standardization, consolidation etc. are well known, so I didn’t capture again here.

An important note, is that this panel focused heavily on virtualization as being a step to the clouds, leading cloud thinkers, say otherwise.  I’ll post in-depth on this another time.  Think bundles, patterns and frameworks.

@ Virtualization, Cloud Computing & Green IT Summit: Progress Report on Cloud Computing in Government

Cloud Computing in Government: A Progress Report, moderated by Robert Ames of IBM

Panelists:

• Doug Bourgeois, Director, National Business Center, Dept. of the Interior
• Casey Coleman, CIO, GSA
• William (Bill) Turnbull, Associate CIO for Advanced Technology and Systems Integration, Department of Energy

Robert Ames of IBM opens, sets context for panel and quips on the 4th Cloud Delivery model, “Hype as a Service”, via David Vap.

Casey Coleman, opens saying hard to give progress report at this early stage, but there is a lot of momentum for cloud computing in the government.  A big part of that momentum is the recently launched Apps.gov site. 

Goals of CIO Council and Apps.gov work: establish governance, procurement Leadership, technology innovation, Implementation and adoption, enable sustainable and cost-effective computing, operate as a service provider, and conduct outreach activities.

Doug Bourgeois of the National Business Center is up next.  National Business Center is a service provider to other government agencies.  Doug begins with starting point operational challenges: security, infrastructure sprawl (datacenter growth containment), limited flexibility/adaptability to change.

How to overcome this: 1. security program, 2. service-oriented architecture to add standardization, interoperability and business process management into organization, 3. virtualization 4. dual data center (risk management, failover).  These provided preparedness.

Made business decision to employ cloud technology, refer to their cloud site for more information, including roadmap (pdf).

Now, William Turnbell of the Department of Energy.  DOE’s efforts include (1) data center virtualization, (2) private clouds (Argonne National Lab, Lawrence Berkeley National Lab, National Renewable Energy Lab (green focus)), (3) business applications and (4) research areas: security, HPC, authentication in the cloud.

William states that not all applications are suitable to general purpose clouds.  Experimenting with different private cloud approaches to suit different business needs, including high performance computing.

Q: Biggest Challenges?

Doug: Challenges that rise to top: Governance model for the cloud.  From roles and responsibilities from security standpoint.  Shared multi-tenancy model, who decides, who pays, when it is time to upgrade.  (This is private cloud issue).

Federal government is moving more quickly than OEM suppliers.  OEM (software) licenses are not very cloud friendly. 

Casey: Security is big challenge that people talk about.  Not comprised security, but a different approach is required.

Apps.gov changes purchasing model.  Right now, need purchasing authority to participate.  But, it could (eventually) move to the users of applications and IT services in federal government.  Would need to change purchasing model, authorities and governance.

William: Not all applications move smoothly to the cloud.  Some require re-writing, this is a significant challenge.

Opened to Audience Q&A

Q: Audience question directed at Doug, why did NBC build out the infrastructure, when it could have been commercially acquired?

Doug: Didn’t implement a bunch of new infrastructure to implement cloud, moved existing infrastructure to cloud, added virtualization, management controls: provisioning, metering etc.  The decision was based on economy of scale.  Leveraged existing investment, able to spread that investment out to greater number of consumers (agency support).

Q: Culture Issues?

Casey: Disruptive technology adoption always has culture issues.  Early uses are collaboration and commodity IaaS.  As move to mission critical support, there will be more private cloud adoption.

Q: Security/Risk

Doug: Mission is sensitive but unclassified information (personally identifiable information).  Implementing 3 security zones, dev/test, production, PII.

Q: NASA with Nebula is creating an open source cloud.  What is the role of open source in your agencies?

William: Open source can be looked at as another competitor.  Run TCO models on both commercial and open source offerings.  Key that there is a community supporting the open source software (project).

Doug: Looked at from economic perspective.  Viable from risk perspective.  Are using some open source products at O/S level.

Robert Ames, IBM: Notes really interesting / innovative work going on in the open source community, such as Hadoop.

Q: Can existing datacenter assets be used to create IaaS services?  Especially, with efficiency gains of virtualization etc.  What happens to underlying assets?

Casey: Enormous legacy datacenter investment.  Great question.  Are currently asking themselves how this can be repurposed.  Some newer assets and applications may be candidates to be IaaS providers.  But some older assets and applications may never be cloud friendly, cloud ready.

Q: Low hanging fruit / general characteristics of good cloud candidates? Hybrids models (mix of public and private clouds)?

Doug: Can be very theoretical question / answer.  Best advice is to look at commodity stuff, things with low security model, things with low interoperability requirements.

Casey: USA.gov citizen portal is moved to a cloud solution.  Cost savings approx 75%.  Another high potential area is email and collaboration.  Agency requirements are similar.

William: Collaboration is one area. 

Q: Working with communities?

All of the panelists mentioned working on security models, and working with researchers on security models.  Robert Ames added information sharing, agencies who use / require the same information, and opportunities for new findings (disease prevention) via access and updates to a shared information source.

Q: Apps.gov and IaaS, will there be agency offerings, such as NBC, with commercial offerings?

Casey: Good question.  Very early stages right now.  Could evolve to a competitive marketplace over time.

Q: Platform as a Service?

Casey: Interesting aspect is that government applications developed on a government provided PaaS could then be shared amongst other government agencies, the IP would be a public asset.  Robert points out this could be “jumpstart for innovation”.  Both Casey and Doug agree.

Q: What about compliance issues like HIPAA and Sarbanes Oxley?

Casey: Great many compliance issues.  For example, agency video on YouTube also needs to be agency website where it is captioned.  So, compliance is about the offering (video), not necessarily the delivery platform (YouTube). 

Doug: Cloud users have a responsibility, nothing (technical) prevents them from bringing HIPAA type data to non-compliant platform, so there needs to be education, governance and communication.

In Closing…

Doug: Try it, start small, low risk.

Casey: Despite hype, cloud is real.  Reduce friction, increase capability delivery.

@ Virtualization, Cloud Computing & Green IT Summit: Federal Sustainability Success Stories / Electronics Stewardship

Federal Sustainability Success Stories / Electronics Stewardship, moderated by Cate Berard, US EPA

Panelists:

• Jeff Eagan, Program Manager, Federal Electronics Challenge, Department of Energy
• Holly Elwood, Electronics Stewardship Coordinator, Office of Environmental Policy and Assistance, EPA
• Johnny Lopez, Chief, IT Division, OUSD (P&R IM), Dept. of Defense

Holly Elwood opens, talking about EPA’s electronics product registry initiative.  From the EPEAT website:

“EPEAT is a system that helps purchasers evaluate, compare and select electronic products based on their environmental attributes. The system currently covers desktop and laptop computers, thin clients, workstations and computer monitors.

Desktops, laptops and monitors that meet 23 required environmental performance criteria may be registered in EPEAT by their manufacturers in 40 countries worldwide. Registered products are rated Gold, Silver or Bronze depending on the percentage of 28 optional criteria they meet above the baseline criteria. EPEAT operates an ongoing verification program to assure the credibility of the registry.”

Johnny Lopez from the DoD is up next.  He has integrated EPEAT into DoD’s lifecycle management of electronic assets.  Another initiative he’s charged with is power management.  By identifying “power vampires” (green lights on over night) in their office, they were able to reduce electricity consumption by 13%.  They’ve also adopted a related continuous improvement program.

Jeff Eagan of the Dept of Energy is up now.  First organization to obtain green status on electronic stewardship.  (Federal Electronics Challenge winner).  Used mixture of incentives and mandates to achieve Green IT success.  Cradle to crave program: acquisition, use to disposal.  95% of purchases are EPEAT registered, most are silver, some are gold.  93% of end user seats are engaged in power management program.  High equipment re-use, recycle rates [think he said 98%, don’t quote me].

Mentions new executive order, signed yesterday, mandating that US Federal Government agencies lead by example in the reduction of greenhouse gas emissions.

Panel Discussion

Holly Elwood speaks to EPEAT as an information source for IT acquisition, from silver and gold level specifications to contract language.  New standards in development for TV and imagining devices.

In response to question on power management challenges, Johnny Lopez mentions the human factor.  Getting people to remember to shutdown non-essential peripherals over night.  CPUs are left up during the week for systems management purposes, everything is shutdown on the weekends.  Monday morning restart is a little longer, but the payback is worth it.

Jeff Eagan echoes the human issue.  The Department of Energy has automated switching computers off at set times, turn back on for systems management.  A useful ally is the cyber security team.  A shut off machine is a safer machine.

Jeff Eagan talks about the composition of Green teams – facility operators, environmentalists and IT.  Looking forward to recycling standards for electronics equipment, standards are on the way.  Mentions still deployed older technology that continues to serve its purpose and is better for environment.  Not introducing new electronics.  [wonder about the power / efficiency trade-offs]

Holly Elwood points out the Green Electronics Council as a green steward and resource.

Jeff Eagan shares that the new executive order has new standards for data center management. 

A theme here is that everything starts with visibility.  Johnny Lopez used a Watt’s Up power meter device.  Ran tests with different combinations of on/off devices.  Realized shutdown of CPU wasn’t worth productivity loss (30 minutes) on daily startup.

Cate Berard of the EPA mentions a new software program that combines power management, systems management and security.  By coordinating these processes, the EPA can reduce energy consumption and provide better, safer, more consistent support for the people engaged in EPA’s mission.

Holly Elwood “the best way to green your product purchases is to not buy; extend the life of your existing products”.  Looking at complete lifecycle – cradle to grave – extending life is better than buying new.  [Answers my power efficiency question noted earlier].

In response to question, Holly Elwood points out the challenge of operating systems overriding machine set green standards.  Interoperability issues might prevent an operating system from knowing that the imaging system is pre-set, and should remain set, to duplex. 

This has been an interesting panel.  Lots of good resources pointed out by the panelists.  Certainly relevant beyond the government sector.  Personally, I turn off my the power strips to my office’s power vampires every night.  Good for the environment and my electric bill.  [Sustainability starts with simple steps.]

@ Virtualization, Cloud Computing & Green IT Summit: Tod Nielsen, VMWare

Energize and Save, Tod Nielsen, Chief Operating Officer, VMWare, opening keynote

Todd starts by talking about all of the government agencies VMWare works with.  The list is extensive.  One example, the CIA is virtualizing 4000 servers, saving millions in capital and energy costs.

Gartner factoid, 89% of virtualized apps run on VMWare (dated December 2008).

Early in virtualization journey, 0 – 20% virtualized, the big savings is Capex.  These projects are typically for IT owned assets, file and print, mail etc.  Next phase, is line of business applications, the concerns here shift from just capex to “speeds and feeds”, business continuity, up-time.  Often, these are less critical, tier 2 and 3 applications.  Todd shares anecdote of organization where CIO was concerned that tier 2 and 3 had better up-time than tier 1, response tiers 2 – 3 are virtualized, was driver for tier 1 virtualization as well.  So, capex, energy and up-time.

Now, Todd is talking about IT complexity, and how a reduction in IT complexity (increase IT agility) will increase business agility.  This gets us to the cloud.  “Business Infrastructure Virtualization”. 

VMWare saves energy.  Financial energy, capital and operations expense.  Human energy, shift from serving hardware to serving the business. Earth’s energy, can get up to 80% savings on data center energy costs with virtualization.

Tod has moved on to the Platform, talking about vMotion and consolidation.  Vsphere moves from hypervisor to helping build a “giant software mainframe”.  [begs the question, why not an actual mainframe, but I digress]

TechTonic shift slide up now, Cloud is in top right position: frictionless deployment, self-service environment (and more).

“Cloud computing is not a destination, but a way of doing computing”. 

Tod mentions desktop management product coming in November, solves the “PC over IP” issue.

Now, the internal cloud (private cloud).  Doesn’t see internal datacenters and clouds going away.  Does see internal extending to external via virtual private clouds.

Tod mentions VMWare’s announcement today: “VMware, Inc., the global leader in virtualization solutions from the desktop through the datacenter and to the cloud, today announced the opening of a new green IT datacenter in East Wenatchee, Washington.”  VMWare will share Green Data Center lessons with customers.

Tod is taking questions from the audience.  Interesting that the questions focus on virtualizing the desktop, not servers.  A little (friendly) Apple bashing.