I have gotten a lot of response from the post on my thoughts on Data Center regulation.   Many of the comments in a response to an Infoworld article focused on the disbelief of regulations particularly targeting data centers.  A Greener Computing article felt that because the current administration is very tech-savvy they wouldn’t do anything to hurt data centers.  In fact the exact quote was:

I can understand Manos’ concerns, but I think he’s on the wrong track. The federal government is very unlikely to issue strict green regulations related to data centers. And if they do regulate them in some way, the regulations will no doubt be reasonable. The current administration is very technology-savvy — after all, the current Secretary of Energy Steven Chu was recently the director of the Lawrence Berkeley National Laboratory, whose work was heavily dependent on its data center. Chu did some great work related to Green IT when at the labs. He knows what can and can’t be done — and will make sure that data centers aren’t hamstrung with unnecessary regulation.

I guess for clarity sake I should state unequivocally that I do not believe that Data Centers will specifically be targeted or singled out for regulation.   Domestically here in the United States the EPA has kicked off its Energy Star Data Center evaluation which looks to study data centers as a sector, and something may come out of that, but in all honesty that wont be for some time.  I think the more immediate threat is in the efforts around Carbon Cap and Trade.  As the Greener Computing Article calls out, it was front and center at the G8 meetings.   With the UK leading the charge and the only real legislation on the books in this space, it would be hard for the other countries not to use it as the base for their programs.   My previous post focuses specifically on the fact that Data Centers will end up being significant contributing factors to Carbon metrics for companies.  Data Center Managers just aren’t thinking about it, and wont be until its far too late. » Continue

First let me start out with the fact that the need for datacenter measurement is paramount if this industry is to be able to manage itself effectively.   When I give a talk I usually begin by asking the crowd three basic questions

1) How many in attendance are monitoring and tracking electrical usage?

2) How many in attendance measure datacenter efficiency?

3) How many work for organizations in which the CIO looks at the power bills?

The response to these questions has been abysmally low for years, but I have been delighted by the fact that slowly but surely, the numbers have been rising.  Not in great numbers mind you, but incrementing.  We are approaching a critical time in the development of the data center industry and where it (and the technologies involved) will go. » Continue

As an industry, we have been talking about it for some time.  Some claimed it would never come and it was just a bunch of fear mongering. Others like me said it was the inevitable outcome of the intensifying focus on energy consumption.   Whether you view this to be a good thing or bad thing its something that you and your company are going to have to start planning for very shortly.  This is no longer a drill.

CRC – its not just a cycle redundancy check

I have been tracking the energy efficiency work being done in the United Kingdom for quite some time and developments in the Carbon Reduction Commitment (CRC).  My recent trip to London afforded me the opportunity to drive significantly harder into the draft and discuss it with a user community (at the Digital Realty Round table event) who will likely be the first impacted by such legislation. For those of you unfamiliar with the initiative let me give a quick overview of the CRC and how it will work. 

The main purpose of the CRC is a mandatory carbon reduction and energy efficiency scheme aimed at changing energy use behaviors and further incent the adoption of technology and infrastructure.  While not specifically aimed at Data Centers (its aimed at everyone) you can see that by its definition Data Centers will be significantly affected.  It was introduced as part of the Climate Change Act 2008.

In effect it is an auction based carbon emissions trading scheme designed to operate under a Cap and Trade mechanism.  While its base claim says that it will be revenue neutral to the government (except of course for penalties resulting from non-compliance), it provides a very handy vehicle for future taxation and revenue.  This is important, because as data center managers you are now placed in a position where you have primary regulatory reporting responsibilities for your company.  No more hiding under the radar, your roles will now be front and center.                             

All organizations including governmental agencies who consume more than 6000 MWh in 2008 are required to participate.  The mechanism is expected to go live in April 2010.  Please keep in mind that this consumption requirement is called out as MWh and not Megawatts.  What’s the difference? Its energy use over time for your whole company.  If you as a data center manager run a 500 kilowatt facility you account for almost 11% of the total energy consumption.  You can bet you will be front and center on that issue. Especially when the proposed introductory price is £12/tCO2 (or $19.48/tCO2).  Its real money.  Again, while not specifically focused on data centers you can see that they will be an active contributor and participant in the process.  For those firms with larger facilities, lets say 5MW of data center space – dont forget to add in your annual average PUE – the data centers will qualify all to themselves.

For more information of the CRC you can check out the links below:

• A Good Deck with a high level overview can be found here
• The wikipedia definition
• For alot of additional detail the CRC user guide is a great tool

While many of you may be reading this and feel poorly for your brothers and sisters in Great Britain while sighing in relief that its not you, keep in mind that there are already other mechanisms being put in place.  The EU has the ETS, and the Obama Administration has been very public about a similar cap and trade program here in the United States.  You can bet that the US and other countries will be closely watching the success and performance of the CRC initiative in the UK. They are likely to model their own versions after the CRC (why invent the wheel over again, when you can just localize to your country or region).  SO it might be a good idea to read through it and start preparing how you and your organization will respond and/or collect.

I would bet that you as a Data Center Manager have not been thinking of this, that your CIO has not thought about this, the head of your facilities group has not thought about this.  First you need to start driving awareness to this issue.    Next we should heed to a call to arms.

One of the items that came out during the Roundtable discussions was how generally disconnected government regulators are to the complexities of the data center.   They want to view Data Centers as big bad energy using boxes that are all the same.  When the differences in what is achievable from small data centers to mega-scale facilities are great.  Achieving PUEs of 1.2x might be achievable for large scale Internet firms who control the entire stack from physical cabling to application development,  banks and financial insitutions are mandated to redundancy requirements which force them to maintain scores of 2.0. 

Someone once decried to me that data centers are actually extremely efficient as they have to integrate themselves into the grid, they generally purchase and procure the most energy efficient technologies, and are incented from an operating budget perspective to keep costs low.  Why would the government go after them before they went after the end users who typically do not have the most energy efficient servers or perhaps the OEMs that manufacture them.  The simple answer is that data centers are easy high energy concentration targets.   Politically going after users is a dicey affair and as such DCs will bear the initial brunt.

As an industry we need to start involving ourselves in educating and representing  the government  and regulatory agencies in our space.   While the Green Grid charter specifically forbids this kind of activity, having a Data Center industry lobby group to ensure dumb things wont happen is a must in my opinion.  

Would love to get your thoughts on that.

/Mm

I was very pleased at the great many responses to my data center capacity planning chat.  They came in both public and private notes with more than a healthy population of those centered around my comments on power capping and their potential disagreement on why I don’t think the technology/applications/functionality is 100% there yet. So I decided to throw up an impromptu ad-hoc follow-on chat on Power Capping.  How’s that for service?

What’s your perspective?

In a nutshell my resistance can be summed up and defined in the exploration of two phrases.  The first is ‘prime time’ and how I define it from where I come at the problem from.  The second is the definition of the term ‘data center’ and in what context I am using it as it relates to Power Capping.

I think to adequately address my position I will answer it from the perspective of the three groups that these Chiller Side Chats are aimed at namely, the Facility side, the IT side, and ultimately the business side of the problem.

Let’s start with the latter phrase : ‘data center’ first.  To the facility manager this term refers to the actual building, room, infrastructure that IT gear sits in.   His definition of Data Center includes things like remote power panels, power whips, power distribution units, Computer Room Air Handlers (CRAHs), generators, and cooling towers.   It all revolves around the distribution and management of power.

From an IT perspective the term is usually represented or thought of in terms of servers, applications, or network capabilities.  It sometimes blends in to include some aspects of the facility definition but only as it relates to servers and equipment.   I have even heard it used to applied to “information” which is even more ethereal.  Its base units could be servers, storage capacity, network capacity and the like.

From a business perspective the term ‘data center’ is usually lumped together to include both IT and facilities but at a very high level.  Where the currency for our previous two groups are technical in nature (power, servers, storage, etc) – the currency for the business side is cold hard cash.   It involves things like OPEX costs, CAPEX costs, and Return on Investment.

So from the very start, one has to ask, which data center are you referring to?  Power Capping is a technical issue, and can be implemented at either of the two technical perspectives.   It also will have an impact on the business aspect but it can also be a barrier to adoption.

We believe these truths to be self-evident

Here are some of the things that I believe to be inalienable truths about data centers today and in some of these probably forever if history is any indication.

1. Data Centers are heterogeneous in the make up of facilities equipment with different brands of equipment across the functions.
2. Data Centers are largely heterogeneous in the make up of their servers population, network population, etc.
3. Data Centers house non-server equipment like routers, switches, tape storage devices and the like.
4. Data Centers generally have differing designs, redundancy, floor layout, PDU distribution configurations.
5. Today most racks are unintelligent, those that are not, are vendor specific and/or proprietary-also-Expensive versus bent steel.
6. Except in a very few cases, there is NO integration between asset management, change management, incident management, problem management systems between IT *AND* facilities systems.

These will be important in a second so mark this spot on the page as it ties into my thoughts on the definition of prime time.  You see, to me in this context, Prime Time means that when a solution is deployed it will actually solve problems and reduce the number of things a Data Center Manager has to do or worry about.   This is important because notice I did not say anything about making something easier.  Sometimes, easier doesn’t solve the problem.

There is some really incredible work going on at some of the server manufacturers in the area of power capping.   After all they know their products better than anyone.  For gratuitous purposes because he posts and comments here, I refer you to the Eye on Blades blog at HP by Tony Harvey.  On his post responding to the previous Chiller-side chat, he talked up the amazing work that HP is doing and is already available on some G5 boxes and all G6 boxes along with additional functionality available in the blade enclosures.

Most of the manufacturers are doing a great job here.  The dynamic load stuff is incredibly cool as well.    However, the business side of my brain requires that I state that this level of super-cool wizardry usually comes at additional cost.   Lets compare that with Howard, the every day data center manager who does it today, who from a business perspective is a sunk cost.   Its essentially free.   Additionally, simple things like performing an SNMP poll for power draw on a box (which used to be available in some server products for free) have been removed or can only be accessed through additional operating licenses.  Read as more cost.    So the average business is faced with getting this capability for servers at an additional cost, or make Howard the Data Center manager do it for free and know that his general fear of losing his job if things blow up is a good incentive for doing it right.

Aside from that, it still has challenges in Truth #2.  Extremely rare is the data center that uses only one server manufacturer.  While its the dream of most server manufacturers, its more common to find DELL Servers, along side HP Servers, alongside Rackable. Add to that fact that even in the same family you are likely to see multiple generations of gear.  Does the business have to buy into the proprietary solutions of each to get the functionality they need for power capping?  Is there an industry standard in Power Capping that ensures we can all live in peace and harmony?  No.  Again that pesky business part of my mind says, cost-cost-cost.  Hey Howard – Go do your normal manual thing.

Now lets tackle Truth #3 from a power capping perspective.   Solving the problem from the server side is only solving part of the problem.   How many network gear manufacturers have power capping features? You would be hard pressed to find a number on one hand.   In a related thought, one of the standard connectivity trends in the industry is top of rack switching.  Essentially for purposes of distribution, a network switch is placed at the top of the rack to handle server connectivity to the network.     Does our proprietary power capping software catch the power draw of that switch?  Any network gear for that matter?  Doubtful.  So while I may have super cool power capping on my servers I am still screwed at the rack layer –where data center managers manage from as one of their base units.   Howard may be able to have some level of Surety that his proprietary server power capping stuff is humming along swimmingly, he still has to do the work manually.  Its definitely simpler for Howard, to get that task done potentially quicker, but we have not actually reduced steps in the process.   Howard is still manually walking the floor.

Which brings up a good point, Howard the Data Center manager manages by his base unit of rack.  In most data centers, racks can have different server manufacturers, different equipment types (servers, routers, switches, etc), and can even be of different sizes.    While some manufacturers have built state of the art racks specific for their equipment it doesn’t solve the problem.  We have now stumbled upon Truth #5.

Since we have been exploring how current power capping technologies meet at the intersection of IT and facilities it brings up the last point I will touch on regarding tools. I will get there by asking some basic questions as to the operations of a typical data center.  In terms of Operations does your IT asset management system provide for racks as an item of configuration?  Does your data center manager use the same system?  Does your system provide for multiple power variables?  does it track power at all?  Does the rack have power configuration associated with it?  Or does your version of Howard use spreadsheets?  I know where my bet is on your answers.  Tooling has a long way to go in this space.   Facilities vendors are trying to approach it from their perspective, IT tools providers are doing the same, along with tools and mechanisms from equipment manufacturers as well. There are a few tools that have been custom developed to do this kind of thing, but they have been done for use in very specific environments.  We have finally arrived at Power Capping and Truth #6.

Please don’t get me wrong, I think that ultimately power capping will finally fulfill its great promise and do tremendous wonders.  Its one of those rare areas which will have a very big impact in this industry.   If you have the ability to deploy the vendor specific solutions (which are indeed very good), you should. It will make things a bit easier, even if it doesn’t remove steps.   However I think ultimately in order to have real effect its going to have to compete with the cost of free.   Today this work is done by the data center managers with no apparent additional cost from a business perspective.   If I had some kind of authority I would call for there to be a Standard to be put in place around Power Capping.  Even if its quite minimal it would have a huge impact.   It could be as simple as providing three things.  First provide for free and unfiltered access to an SNMP Mib that allows access to the current power usage information of any IT related device.  Second, provide a Mib, which through the use of a SET command could place a hard upper limit of power usage.  This setting could be read by the box and/or the operating system and start to slow things down or starve resources on the box for a time.  Lastly, the ability to read that same Mib.    This would allow for the poor cheap Howard’s to take advantage of at least simplifying their environments.  tremendously.  It would still provide software and hardware manufacturers to build and charge for the additional and dynamic features they would require.

\Mm

I recently had a conversation with the CIO of a well respected company who informed me that his “data center people” had completely mismanaged his data center space which was now causing them to look at having to lease additional capacity or more aggressively pursue virtualization to solve the problem.   Furthermore he was powerless to drive and address change as that data center facilities people worked for a different organization.  To top it off it frustrated him to no end that in his mind they simply did not understand IT equipment or technologies being deployed.   Unfortunately its a common refrain that I hear over and over again.  It speaks to the heart of the problems with understanding data center issues in the industry.

How Data Center Capacity Planning Really works!

Data Center managers are by their very nature extremely conservative people.   At the root of this conservatism is the understanding that if and when a facility goes down, it is their bottoms on the bottom line.   As such, risk takers are very few and far between in this industry.   I don’t think I would get much argument from most business-side managers, who would readily agree to that in a heart beat.   But before we hang the albatross around the neck of our facilities management brethren lets take a look at some of the challenges they actually have.

Data Center Capacity Planning is a swirling vortex of science, art, best guess-work, and bad information with a sprinkling of cult of personality for taste.  One would think that is should be a straight numbers and math game, but its not.   First and foremost, the currency of Data Center Capacity Management and Planning is power.  Simple right?  Well we shall see.

Lets start at a blissful moment in time, when the facility is shiny and new. The floor shines from its first cleaning, the VESDA (Very Early Smoke Detection and Alarm) equipment has not yet begun to throw off false positives, and all is right with the world.   The equipment has been fully commissioned and is now ready to address the needs of the business.

Our Data Center Manager is full of hope and optimism. He or she is confident that this time it will be much different than the legacy problems they had to deal with before.  They now have the perfect mix of power and cooling to handle any challenge to be thrown at them.   They are then approached by their good friends in Information Services with their first requirement.   The business has decided to adopt a new application platform which will of course solve all the evils of previous installations.

Its a brand new day, a new beginning.   The Data Center Manager asks the IT personnel how many servers are associated with this new deployment.   They also ask how much power those servers will draw so that the room can be optimized for this wonderful new solution.   The IT personnel may be using consultants, or maybe they are providing the server specifications themselves.  In advanced cases they may even have standardized the types and classes of servers they use.  How much power?   Well, the nameplate on the server says that each of these bit crunching wonders will draw 300watts a piece.   As this application is bound to be a huge draw on resources, they inform the facilities team that there are approximately 20 machines at 300watts that are going to be deployed.

The facilities team knows that no machine ever draws its nameplate ratings once ‘in the wilds’ of the data center and therefore for capacity planning purposes they ‘manually’ calculate a 30% reduction into the server deployment numbers.  You see, its not that they don’t trust the IT folks, its just that they generally know better.   So that nets out to a 90 watt reduction per server bringing the “budgeted power allocation” down to 210 watts per server.  This is an important number to keep in mind.  You now have two ratings that you have to deal with.   Nameplate, and Budgeted.  For advanced data center users they may use even more scientific methods of testing to derive their budgeted amount.   For example they may run test software on the server designed to drive the machine to 100% CPU utilization, 100% disk utilization, and the like.   Interestingly even after these rigorous tests, the machine never gets close to nameplate.  Makes you wonder what that rating is even good for, doesn’t it?  Our data center manager doesn’t have that level of sophistication, so he is using a 30% reduction.  Keep these numbers in mind as we move forward.

The next question is typically are these servers dual or single corded?  Essentially will these machines have redundancy built into the power supplies so in the event of a power loss they might still operate through another source of electricity?  Well as every good business manager, IT professional, and data center manager knows – This is a good thing.  Sure lets make them double corded.

The data center manager, now well armed with information begins building out the racks, pulls the power whips from diverse PDU (power distribution units) to the location of those racks to ensure that the wonders of dual cording can come to full effect.

The servers arrive, they are installed and in a matter of days the new application is humming along just fine, running into all kinds of user adoption issues, unexpected hick-ups,  budget over-runs, etc.  Okay maybe I am being a bit sarcastic and jaded there but I think it works for many installations.   All in all a successful project right?  I say sure.  But do all parties agree today? tomorrow? 3 years from now?

Lets break this down a bit more on the data center side.   The data center manager has to allocate power out for the deployment.   He has already de-rated the server draw but there is a certain minimum amount of infrastructure he has to deploy regardless.   The power being pulled from those PDUs are taking up valuable slots inside that equipment.   Think of your stereo equipment at home, there are only so many speakers you can connect to your base unit no matter how loud you want it to get.  The data center manager had to make certain decisions based upon the rack configuration.   If we believe that they can squeeze 10 of these new servers into a rack, the data center manager has pulled enough capacity to address 2.1 kilowatts per rack (210watts*10 servers).  With twenty total servers that means he has two racks of 2.1kilowatts of base load.  Sounds easy right? Its just math.  And Mike – you said it was harder than regular math.  You lied.  Did I? Well it turns out that physics is physics and as Scotty from the Enterprise taught us, “You cannot change the laws of Physics, Jim!”  Its likely that the power capacity being allocated to the rack might actually be a bit over the 2.1kilowatts due to the nature of what sized circuits might be required. For example he or she may have only needed enough power for 32 amps of power, but because of those pesky connections he had to pull two 20 amp circuits.  Lets say for the sake of argument that in this case he has to reserve 2.5 kilowatts as a function of the physical infrastructure requirements.  You start to see a little waste right?  Its a little more than one servers expected draw, so you might think its not terribly bad.  As a business manager, your frustrated with that waste but you might be ok with it.  Especially since its a new facility and you have plenty of capacity.

But wait!  Remember that dual cording thing?   Now you have to double the power you are setting aside.  You have to ensure that you have enough power to ensure you can maintain the servers.  Usually this is from another PDU so that you can survive a single PDU failure.  Additionally you need to reserve that each side (each cord) has enough power to failover.  In some cases the total load of the server is divided between the two power supplies, in some cases, power is drawn from the primary with a small trickle of draw from the redundant connection.  If the load is divided between both power supplies you are effectively drawing HALF of the total reserved power.   If its the situation where they draw full load off one, and have a trickle draw off the second power supply, you are actually drawing the correct amount on one leg, and dramatically less than HALF on the second.   Either way the power is allocated and reserved and I bet its more than you thought when we started out this crazy story.  Well hold tight, because its going to get even more complicated in a bit.

Right now, any data center manager in the world is reading this and screaming at the remedial nature of this post.  This is Facilities Management 101 right after the seminar entitled ‘The EPO button is rarely your friend’ .   In fact, I am probably insulting their intelligence because there are even more subtleties than what I have outlined here.    But my dear facilities comrade,  its not to you I am writing this section to.  Its the business management and IT folks.   With physics being its pesky self combined with some business decisions, you are effectively taking more power down that you initially thought.   Additionally you now have a tax on future capacity as that load associated with physics and redundancy is forever in reserve.  Not to be touched without great efforts if at all.

Dual Cording is not bad. Redundancy is not bad.  Its a business risk, and that’s something you can understand, and in fact, as a business manager its something I would be willing to bet you do every day in your job.  You are weighing the impact of the outage of the business to actual cost.  One can even easily calculate the cost of such a decision by taking proportional allocations of your capital cost from an infrastructure perspective and weigh it against the economic impact of not having certain tools and applications available.   Even when this is done and its well understood, there is a strange phenomena of amnesia that sets in and in a few months/years the same business manager may look at the facility and give the facilities person a hard time for not utilizing all the power.    To Data Center Managers – Having sat as a Data Center Professional for many years – I’m sad to say, you can expect to have this “Reserved” power conversations multiple times with your manager over and over again, especially when things get tight in terms of capacity left.  To business managers, book mark this post and read it about every 6 months or so.

Taking it up a notch for a moment…

That last section introduced the concept of Reserved Power.  Reserved Power is a concept that sits at the Facility level of Capacity Planning.   When a data center hall is first built out there are three terms and concepts you need to know.  The first is Critical load (sometimes called IT load).  This is the power available to IT and computer equipment in your facility.  The second is called Non-Critical load, which has to do with the amount of power allocated to things like lighting, mechanical systems and your electrical plant, generators, etc.  What I commonly call ‘Back of the house’ or ‘Big Iron’.  The last term is Total load.  Total load is the total amount of power available to the facility and can usually be calculated by adding Critical and Non-Critical loads.

A Facility is born with all three facets.  You generally cannot have one without the other.  In plan on having a future post called ‘Data Center Metrics for Dummies’ which will explore the interconnection between these.  For now lets keep it really simple.

The wonderful facility we have built has a certain amount of IT gear that it will hold.   Essentially every server we deploy into the facility will subtract from the total amount of Critical Load available for new deployments.  As we deduct the power from the facility we are allocating that capacity out.    In our previous example we deployed two racks at 2.5kilowatts (and essentially reserved capacity for two more for redundancy).   With those two racks we have allocated enough power for 5 kilowatts of real draw and have reserved 10 kilowatts in total.

Before people get all mad at me, I just want to point out that some people don’t count the dual cording extra because they essentially de-rate the room with the understanding that everything will be dual corded.    I’m keeping it simple for people to understand what’s actually happening.

Ok back to the show – As I mentioned those racks would really only draw 2.1kW each at full load (we have essentially stranded 400watts per rack of potential capacity and combined its almost 800 watts per rack).  As a business we already knew this but we still have to calculate it out and apply our “Budgeted Power” to the room level.   So, across our two racks we have an allocated power of 5 kilowatts, with a budgeted amount of 4.2 kilowatts.

Now here is where our IT friends come into play and make things a bit more difficult.   That wonderful application that was going to solve world hunger for us and was going to be such a beefy application from a resources perspective is not living up to its reputation.    Instead of driving 100% utilization, its sitting down around 8 percent per box.   In fact the estimated world wide server utilization number for  servers sits between 5-14%.  Most server manufacturers have built their boxes in a way where they draw less power at lower utilizations.  Therefore our 210watts per server might be closer to 180watts per server of “Actual Load”.  That’s another 30watts per server.   So while we have allocated 2.5kilowatts, and reserved 2.1kilowatts, we are only drawing 1.8kilowatts of power.   We have two racks, we so double it.  So now we are in a situation where we are not using 700watts per rack or 1.4 kilowatts across our two racks.  Ouch that’s potentially 28% of power wasted!

The higher IT and applications drive the utilization rate, the lower amount of waste you will have.   Virtualization can help here, but its not without its own challenges as we will see.

Big Rocks Little Rocks…

Now luckily, as a breed, data center managers are a smart bunch and they have a various of ways to try and reduce this waste.   It goes back to our concept of budgeted or reserved power combined with our “stereo jacks” in the PDU.   As long as we have some extra jacks, the Data Center Manager can return back to our two racks and artificially set a lower power budget per rack.  This time after metering the power for some time he makes the call to artificially limit the racks allocation to 2 kilowatts – he could go to 1.8kilowatts, but remember he is conservative and wants to still give himself some cushion.  He can then deploy new racks or cabinets and pretend that the extra 200 watts to the new racks.  He can continue this until he runs out of power or out of slots on the PDU.   This is a manual process that is physically managed by the facilities manager.   There is an emerging technology called power capping which will allow you to do this in software on a server to server basis which will be hugely impactful in our industry, its just not ready for prime time yet.

This inefficiency in allocation creates strange gaps and holes in data centers.  Its a phenomena I call Big Rocks, Little Rocks, and like everything in this post is somehow tied to physics.

In this case it was my freshman year physics class in college.  The professor was at the front of the class with a bucket full of good sized rocks.   He politely asked if the the bucket was full.   The class of course responded in the affirmative.  He then took out a smaller bucket of pebbles and poured and rigorously sifted and shook the heck out of the larger bucket until every last pebble was emptied into that bucket with the big rocks.  He asked again, “Now is it full?”  The class responded once more in the affirmative and he pulled out a bucket of sand.  He proceeded to re-perform the sifting and shaking, etc and emptied the sand into the bucket.   “Its finally full now right?”   The class shook their heads one last time in the affirmative and he produced a small bucket of water and poured it into the bucket as well.

That ‘Is the bucket full exercise’ is a lot like the capacity planning that every data center manager eventually has to get very good at.   Those holes in capacity at a rack level or PDU level I spoke out are the spaces for servers and equipment to ultimately fit into.  At first its easy to fit in big rocks, then it gets harder and harder.    You are ultimately left trying to manage to those small spaces of capacity.   Trying to utilize every last bit of energy in the facility.

This can be extremely frustrating to business managers and IT personnel.   Lets say you do a great job of informing the company how much capacity you actually have in your facility, if there is no knowledge of our “rocks” problem you can easily get yourself into trouble.

Lets go back for a second to our example facility.   Time has now passed and our facility is now nearly full.  Out of the 1MW of total capacity we have been very aggressive in managing our holes and still have 100kilowatts of capacity.  The IT personnel have a new application that is database intensive that will draw 80 kilowatts and because the facility manager has done a good job of managing his facility, there is every expectation that it will be just fine.  Until of course they mention that these servers have to be contiguous and close together for performance or even functionality purposes.   The problem of course is that you now have a large rock that you need to try and squeeze into small rock places.  It wont work.  It may actually even force you to either move other infrastructure around in your facility impacting other applications and services, or cause you to get more data center space.

You see the ‘Is it full exercise’ does not work in reverse.  You cannot fill a bucket with water, then sand,  then pebbles, then rocks.   Again lack of understanding can lead to ill-will or the perception that the data center manager is not doing a good job of managing his facility when in fact they are being very aggressive in that management.    Its something the business side and IT side should understand.

Virtualization Promise and Pitfalls…

Virtualization is a topic unto itself that is pretty vast and interesting, but I did want to point out some key things to think about it.  As you hopefully saw, server utilization has a huge impact on power draw.  The higher the utilization the better performance from a power perspective.   Additionally many server manufacturers have certain power ramps built into their equipment where you might see an incrementally large jump in power consumption from 11 percent to 12 percent for example.  It has to do with throttling of the power consumption I mentioned above. This is a topic that most facility managers have no experience and knowledge of as it has more to do with server design and performance.   If your facility manager is aggressively managing your facility as in the example above and virtualization is introduced, you might find yourself tripping circuits as you drive the utilization higher and it crosses these internal utilization thresholds.  HP has a good paper talking about how this works.  If you pay particular attention to page 14, The lower line is the throttled processor as a function utilization.  The upper line is full speed as a function of utilization and then their dynamic power regulation feature is the one that jumps up to full speed a 60% utilization.  This gives the box performance only at high utilizations.  Its a feature that is turned on by default in HP Servers.  Other manufacturers have similar technologies built into their products as well.  Typically your Facilities people would not be reading such things. Therefore its imperative that when considering virtualization and its impacts – it should be something that the IT folks and Data Center managers should work on jointly.

I hope this was at least partially valuable out there and hopefully explained some things that may have been considered black box or arcane data center challenges in your mind.   Keep in mind with this series I am trying to educate on all sides the challenges we are facing together.

/Mm

To most outsiders the datacenter industry may seem to be quite parochial.   A niche blend at the intersection of building and IT technologies.   Even to many senior business managers both of these areas in and of themselves are not necessarily considered business critical. As such, little if any time is spent concentrating on the issues this space.  That is of course until it becomes business critical.  At that time many business managers struggle to navigate this complex world.  That complexity can drive frustration, confusion, and is prone to misinformation ultimately leading to bad business decisions.  In many cases it might even be the first time they speak to those in the organization responsible for these assets.

Conversely, those who have had responsibility for the care and maintenance of the data center facilities themselves are now thrust into the spotlight and into a foreign world of business-speak.  Usually these professionals report in through Corporate Real Estate services, where skill sets tend to lean towards more generic building issues.  Real Estate groups tend to minimize the complexity of these facilities and apply what they know rather than what is truly happening. It leads to an interesting time for data center professionals who tend to be more engineering focused.  Normally quite content to talk at an engineering and operations level, they now find themselves thrust into a world of business jargon, return on investments, capital planning, and business impacts.  Out of their comfort zone and frustrated by the inability for management to grasp the engineering aspects of the issues at hand, cynicism increases and tempers can flare.

To make it even more confusing the IT organization, normally the largest user of those facilities and the primary advocate and buffer between business management and the data center organization rarely understands the issues within the facility itself or the challenges the advancements in server and IT technology may actually have in the environment in which their services run.  Even when they believe themselves to be assisting their data center comrades they may unintentionally be causing more harm than good.  Data Center professionals (at the facility level) rarely understand server and application technologies which are increasingly placing strain on their facilities.

Three worlds have collided and its never pretty.   In my experience and in conversations with many customers in all three categories its a time that fosters frustration, mistrust, and stress.  Its also a wonderful time for less than scrupulous vendors, contractors, and consultants to take advantage of the situation and cause poor decisions to be made.   I am not saying that all consultants are bad or ill intentioned, in fact, there are some phenomenal organizations and products out there.   Its just that you need to be aware of the biases and “religious” debates in this space.   If your a business manager think about the debates around out-sourcing versus in-sourcing, off-shoring versus near-shoring.  If you are an IT professional think back to the times of Ethernet versus Token Ring, Oracle versus SAP, Windows versus Netware or Windows versus Linux.  While it may it may seem strange to people in IT and business management, the data center world suffers from many of the same things.  Concepts like AC versus DC, batteries versus rotary-based UPS systems raised floor versus on-slab.  Different firms have different biases and religious affiliations.

With this post, I am kicking off a series of posts in which my sincerest wish is to help all three groups during these stressful times.   Having spent significant time in all three camps I will offer up my own personal take on the issues at hand.  I am calling them Chiller-Side Chats.   From time to time I will post my thoughts on various issues aimed at bridging the communication between these organizations.  I strongly encourage anyone reading these posts to drop comments or offer up suggestions so we can have a lively discussion on these topics.

\Mm

This post is a portion of my previous post that I broke into two.   It more clearly defines where I think the market is evolving to and why companies like Digital Trust Realty will be at the heart of the change in our industry.

The Birth of Utilities and why a Century ago will matter today and forward…

I normally kick off my vision of the future talk with a mention first to history (my former Microsoft folks are probably groaning at this moment if they are reading this).   I am a huge history buff.  In December of 1879, Thomas Edison harnessed the power of electricity for the first time to light a light bulb.  What’s not apparent is that this “invention” was in itself not complete.  To get this invention from this point to large scale, commercial application required a host of other things to be invented as well.   While much ado is made about the successful kind of filament used to ensure a consistent light source, there were no less than at least seven other inventions to make electric light (and ultimately the electric utility) practical for everyone.  Invention of things like the parallel circuit, an actual durable light bulb, an improved dynamo, underground conductor networks, devices to maintain constant voltage, insulting materials and safety fuses, the light socket, the on/off switch, and a bunch of other minor things.   Once all these things were solved, the creation of the first public electricity utility was created.  On September of 1882, the first commercial power station, located on Pearl Street in lower Manhattan opened its doors and began providing light and electrical power to all customers within the “massive” area of one square mile.   This substation was a marvel of technology staffed with 10s of technicians, maintaining the complex machinery to exacting standards.   The ensuing battle between Direct Current and Alternating Current was then created and in some areas still continues today. More on this in a bit.

A few years earlier a host of people were working on what would eventually become known as the telephone.   In the United States this work is attributed to Alexander Graham Bell and its that story I will focus on here for a second.  Through trial and error Bell and his compatriot Watson, accidently stumbled across a system to transfer sound in June of 1875.  After considerable work on refinement the product launched (There is an incredibly interesting history of this at SCRIBD), and after ever more additional trial and error the first telephonic public utility was created with the very first central office coming online in January of 1878 in New Haven, Connecticut.  This first central office was a marvel to behold.  Again the extremely high tech equipment with a host of people ensuring that telephonic utility was always available and calls were transferred appropriately.  Interestingly by 1881 only 9 cities with populations above 10,000 were without access to the telephone utility and only 1 above 15,000!  That is an adoption rate that remains boggling even by today’s standards. 

These are significant moments in time that truly changed the world in the way we live every day.   Today we are the birth of another such utility.  The Information Utility.   Many people I have spoken to claim this “Information Utility” is something different. It’s more of a product, because it uses existing utility services. Some maintain that its truly not revolutionary because its not leveraging new concepts.   But the same can be said of those utilities as well.   The communications infrastructure we use today whether telephone or data has its very roots in the telegraph.  The power utilities have a lot to thank the gas-lamp utilities of the past for solving early issues as well.  Everything old is new again and everything gets refined into something newer and better.  Some call this new Information Utility the “cloud”, others the Information-sphere, others just call it the Internet.  Regardless what you call it, access to information is going to be at your finger tips more today and tomorrow than it has ever been before.

Even though this utility is built upon existing services, this utility too will have its infrastructure.  Just as the electric utility has its sub-stations and distribution yards, and the communication utilities have central offices, so too will data centers become the distribution mechanism for the the Information Utility.   We still have a lot of progress to make as well.   Not everything is invented or understood yet.  Just as Edison had to invent a host of other items to make electricity practical, and Bell and Watson have to develop the telephone, the telephone ringer (or more correctly, thumper or buzzer), so to does our information Utility have a long way to go.  In some respects its even more complicated than its predecessors as their was not burdened with legislation and government involvement that would affect its early development.  The “Cloud” does.

And that innovation does not always come from a select few.  Westinghouse and his alternating current eventually won out over direct current because it found its killer app and business case.   Alternating current was clearly the technically superior and better for distribution. They had even demonstrated generating power at Niagara Falls and successfully transferred that power all the way to Buffalo, New York! Something direct current was unable to do.  In the end, Westinghouse worked with appliance manufacturers to create devices that used alternating current.  By driving his killer app (things like refrigerators), Edison eventually lost out.  So too will the cloud have its killer apps.  The pending software and services battle will be interesting to note.  However what is interesting to me is that it was the business case that drove adoption and evolution here.  This also modified how the utility was used and designed.   DC substations gave way to AC substations, what used to take scores of people to support has dwindled to occasional visitations and pre-scheduled preventative maintenance.   At the data center level, we cannot afford to think that these killer applications will not change our world.    Our killers applications are coming and it will forever change how our world does business.  Data Centers and their evolution are at the heart of our future.

On Fogs, Mist, and the Clouds Ahead . . .

After living in Seattle for close to 10 years, you learn you become an expert in three things.  Clouds, rain, and more clouds.   Unlike the utilities of the past, this new Information Utility is going to be made up of lots of independent cloudlets full of services.  The Microsoft’s, Google’s and Amazon’s of the world will certainly play a large part of the common platforms used by everyone, but the applications, products, content, customer information, and key development components will continue to have a life in facilities and infrastructure owned or controlled by companies providing those services.    In addition, external factors are already beginning to have a huge influence on cloud infrastructure.  Despite the growing political trend of trans-nationalism where countries are giving up some of their sovereign rights to participate in more regionally-aware economics and like-minded political agendas, that same effect does not seem to be taking place in the area of taxation and regulation of cloud and information infrastructure.  Specifically as it relates to electronic or intellectual property entities that derive revenue from infrastructure housed in those particular countries or do so (drive revenue) off of online activity of citizens of those nations.

There are numerous countries today that have, or are, seriously engaged in establishing and managing their national boundaries digitally online.  What do I mean by that?  There is a host of legislation across the globe that is the beginning to govern the protection and online management of their their citizens through legislation and mandates in accordance with their own laws.   This is having (and will continue to have) a dramatic impact on how infrastructure and electronic products and services will be deployed, where that data is stored,  and how revenue from that activity can and will be taxed by the local country.  This level of state exercised control can be economically, politically, or socially motivated and cloud services providers need to pay attention to it.   A great example of this is Canada which has passed legislation in response to the U.S. Patriot Act.   This legislation forbids personally identifiable information (PII) of Canadian citizens to be housed outside of the boundaries of Canada or perhaps more correctly, forbids its storage in the United States.    There are numerous laws and legislation making their way across Europe and Asia as well.   That puts an interesting kink in the idea of a world wide federated cloud user-base where information will be stored “in the cloud”.  From an infrastructure perspective it will mandate that there are facilities in each country to house that data.  While the data storage and retention challenge is an interesting software problem to solve the physical fact that the data will need to remain in a local geography will require data centers and components of cloud infrastructure to be present.  I expect this to continue as governments become more technically savvy and understand the impact of the rate of change being caused by this technology evolution. Given the fact that data centers are extremely capital intensive only a few players will be able to deploy private global infrastructures.  This means that the “information sub-station” providers will have an even more significant role in driving the future standards of this new Information Utility. One might think that this could be a service that is ultimately provided by the large cloud providers as a service.   That could be a valid assumption however, there is an interesting wrinkle developing around taxation or more correctly exposure to double taxation or multiple-country taxation that those large providers will face.   In my opinion the federation of “information substation” providers will provide the best balance of off-setting taxation issues and still providing a very granular and regionally acceptable way to service customers. That is where companies like Digital Realty Trust are going to come in and drive significant value and business protection.

I watch a lot of these geo-political and economic developments pretty closely as it relates to Data Center and infrastructure legislation and will continue to do so.  But even outside of these issues, the “cloud” or whatever term you like will continue to evolve and the “channels” created by this paradigm will continue to drive innovation at the products and services level.  Its at this level where the data center story will continue to evolve as well.   To start we need to think about the business version of the IT “server-hugging” phenomena. For the uninitiated, “Server Huggers” are those folks in an IT department who believe that the servers have to be geographically close  in order to work on them. In some cases its the right mentality, in others, where the server is located truly doesn’t matter.   It’s as much a psychological experiment as a technical one.   At a business level, there is a general reluctance to house the company jewels outside of corporate controlled space.  Sometimes this is regulated (like banks and financial institutions), most often its because those resources (proprietary applications, data sets, information stores, etc) are crucial to the success of the company, and in many instances ARE the company. Not something you necessarily want to outsource to others for control.  Therefore wholesale adoption of cloud resources is still a very very long way off.  That is not to say that this infrastructure wont get adopted into solutions that companies ultimately use to grow their own businesses.  This is going to drive tons of innovation where businesses evolve their applications , create new business models, and join together in mutually beneficial alliances that will change the shape, color, and feel of the cloud.  In fact, the cloud or “Information Utility” becomes the ultimate channel distribution mechanism.

The first grouping I can see evolving is fraternal operating groups or FOGs.  This is essentially a conglomeration of like minded or related industry players coming together to build shared electronic compute exchanges or product and service exchanges.  These applications and services will be highly customized to that particular industry. They will never be sated by the solutions that the big players will be putting into play, they are too specialized.  This infrastructure will likely not sit within individual company data centers but are likely to be located in common ground facilities or leased facilities with some structure for joint ownership.   Whether large or small, business to business, or business to consumer, I see this as an evolving sector.  There will be definitely companies looking to do this on their behalf, but given the general capital requirements to get into this type of business these FOG Agreements may be just the answer to find a great trade off between capital investment and return on the compute/service.

The next grouping builds off of the “company jewels” mindset and how it could blend with cloud infrastructure.  To continue the overly used metaphor of clouds,I will call them Managed Instances Stationed Territorially or MISTs.   There will likely be a host of companies that want to take advantage of the potential savings of cloud managed infrastructure, but want the warm and fuzzy feeling knowing its literally right in their backyard.  Imagine servers and infrastructure deployed at each customer data center, but centrally managed from cloud service providers.   Perhaps its owned by the cloud provider, perhaps the infrastructure has been purchased by the end-user company.   One can imagine the container-based server solutions being dropped into container-ready facilities or jury-rigged in the parking lot of a corporate owned or leased facility.  This gives companies the ability to structure their use of cloud technologies and map them into their own use case scenarios.  What makes the most sense for them.  The recent McKinsey paper talked about how certain elements of the cloud are more expensive than managing the resources through traditional means.  This is potentially a great hybrid scenario where companies can integrate as they need to using those services.  One could even see Misty FOGs or Foggy Mists.  I know the analogy is getting old at this point, but hopefully you can see that the future isn’t as static as some would have you believe.  This ability to channelize the technologies of the cloud will have a huge impact on business costs, operations, and technology.   It also suggests that mission critical infrastructure is not going to go away but become even more important and potentially more varied.  This is why I think that the biggest infrastructure impact will occur in the “information substation provider” level.  Data Centers aren’t going away, they might actually be growing in terms of demand, and the one thing definitely for sure is that they are evolving today and will continue to evolve as this space matures.  Does your current facility allow for this level of interconnectivity?  Do you have the ability to have a mixed solution management providers in your facility?  Lots of questions lots of opportunities to develop answers.

The last grouping is potentially an evolution of modern content delivery infrastructure or edge computing capabilities.  I will quit with the cutesy cloud names and call this generically Near Cloud Content Objects.   Given that products, services, and data will become the domain of those entities owning them, and a general reluctance to wholesale store them in someone else’s infrastructure, one could see that this proprietary content could leverage the global cloud infrastructures through regional gateways where they will be able to maintain ownership and control of their asset.  This becomes even more important when you balance into this the economic and geo-political aspects emerging in cloud compute.

In the end the cloud approach is going to significantly drive data center demand and cause it to evolve even further.  It will  not as some would like to project end the need for corporate data centers.  Then there is that not so little issue of the IT Applications and internal company services we use everyday.  This leads me into my next point . . .

The Continued and Increasing Importance of Enterprise Data Centers

This post has concentrated a lot on the future of cloud computing, so I will probably tick off a bunch of cloud-fan-folk with this next bit, but the need for the corporate data centers is not going away.  They may change in size, shape, efficiency, and the like, but there is a need to continue to maintain a home for those company jewels and to serve internal business communities.  The value of any company is the information and intellectual property developed, maintained, and driven by their employees.   Concepts like FOGs and MISTs and such still require ultimate homes or locations for that work to be terminated into or results sent to.  Additionally look at the suite of software each company may have in its facilities today supporting their business.  We are at least a decade or more away before those could be migrated to a distributed cloud based infrastructure.  Think about the migration costs of any particular application you have, then compound that with having the complexity of having your data stored in those cloud environments as well.  Are you then locked into a single cloud provider forever? It obviously requires cloud interoperability, which doesn’t exist today with exception of half-hearted non-binding efforts that don’t actually include any of the existing cloud providers.   If you believe as I do that the “cloud”  will actually be many little and large channelized solution cloudlets, you have to believe that the corporate data center is here to stay.  The mix of applications and products in your facilities may differ in the future, but you will still have them.  That’s not to say the facilities themselves will not have to evolve.  They will.  With changing requirements around energy efficiency and green reporting, along with the geo-political and other regulations coming through the pipeline, the enterprise data center will still be an area full of innovation as well. 

/Mm

Here, we make the Myth-of-the-Month video on data center density available via Podcast.

• 
• Click to play or Right-Click | Save As to download: The Myth of Data Center Density