I Promise Not To Quote That Old Annoying Dell PC Slogan

I’ve been buying Dell computers for decades, but not really because I loved them. Sure, my XPS laptop was awesome, but it burned out its motherboard and I never really touched the RMA replacement, having bought a MacBook Pro in the meantime. Enterprise buyers seem to have the same ambivalence about Dell. They buy it, but I’m not sure they really “buy” the company as an IT partner.

I’m sure you’ve heard the same comments as me: “Dell just assembles off-the-shelf components and sells them in volume” or “Dell’s a follower, not an innovator.” There seems to be a great deal of respect for Dell’s ability to produce competitive products and sell them at reasonable cost. Truly, most of their competitors would love to have this kind of reputation. But most of their competitors also have a reputation for partnership, innovation, and solution selling.

Dell Is Making An Effort

It seems clear that Dell would like to change this attitude, and they are investing serious resources to make it happen. While acquisitions like Compellent and Force10 raised eyebrows in storage and networking, it is the activity I see behind the scenes that paints the clearest picture. Dell isn’t just buying into new markets, they’re investing to change the company.

When Dell acquired EqualLogic in 2008, many assumed it was a tactical investment to increase margins over the (resold) EMC storage equipment the company was then pushing. Pundits were similarly dismissive of the acquisition of Perot Systems in 2009, calling it a “me too” effort after HP acquired rival EDS. Regardless of the motivations, however, Dell was becoming more of a serious challenger to HP and IBM every day.

After failing to acquire 3PAR in 2010, then picking up Compllent shortly after, accusations that Dell was “mini me” to HP were rampant. But HP stumbled mightily in 2011, and many in IT quickly lost confidence in that company’s management. All the while, Dell moved forward, increasing in-house IP and expanding enterprise offerings.

What Is The Result?

Today, one sees a very different landscape than just last year. Dell’s acquisitions focused on some of the ripest spots in storage and networking, and no one would disagree that the company has the ability strongly to push these products. Compellent and Force10 went from interesting startups to serious contenders overnight.

Dell really pulled out all the stops to tell us they are "Fluid by Design"

More importantly, Dell has retained much of the innovation these companies offered, from employees to support programs. Last week, I attended the Dell Storage Forum in London, an event initiated by Compellent prior to the acquisition. At the event, I talked to many Dell employees who came to the company through acquisition but had now been given power to challenge the status quo in their respective areas.

If Dell really intended only to push product, why retain marketing personnel? Why invest in the Dell Storage Forum? Why continue Compellent’s beloved Co-Pilot support program?

Then there are the products. Dell leveraged its investment in Ocarina Networks to create a deduplicating backup appliance, the new DR4000. They salvaged file system startup ExaNet and are beginning to bring scale out technology to market. The latest revision of the Compellent software finally brings it to parity in terms of VMware support. And Dell is really working to sell their DX Object Store.

This is the sort of activity one would expect from a contender, not a “box pusher”.

Stephen’s Stance

In the words of Malcolm Reynolds, my days of not taking Dell seriously are certainly coming to a middle. Dell is investing in product IP, innovative marketing and PR events, customer support, and personnel. This does not mean that Dell is instantly a player in the enterprise storage and networking markets, or that all this work will pay off. But I don’t laugh when I hear Dell boast that they intend to be a “top three” enterprise storage company in a few years. It could happen.

Disclaimer: Dell sponsored two Tech Field Day events in 2011, paid me as a speaker at two DX events, and paid for my trip to Dell Storage Forum in London. But no one can buy a post on this site, and I did similar business with IBM, HP, Cisco, and many other companies. This is my opinion.

© sfoskett for Stephen Foskett, Pack Rat, 2012. |
Why Should Anyone Take Dell Seriously in Enterprise Storage?


This post was categorized as Computer History, Enterprise storage, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you’d like to filter out or focus on posts like this.

© Stephen Foskett for Gestalt IT, 2012. | Why Should Anyone Take Dell Seriously in Enterprise Storage?
Read more posts categorized as Storage

When designing VDI environments, a couple of things that you should consider are the .vswp file and the GuestOS paging file. In this article I am going to focus on the Paging file and hopefully in the not so distant future I will write a post about the .vswp file in a VDI environment.

What is point of the paging file (also known as the pagefile.sys)?

RAM is a limited resource. Virtual memory was introduced to help remove that limit.

Most modern operating system now use Virtual Memory. Virtual memory is a memory management technique. Applications running on a GuestOS reference memory using virtual memory addresses which are then automatically translated into RAM addresses by the hardware. These virtual memory address spaces are divided in pages or block, usually of 4KB.

If RAM resource is exhusted, the operating system will move 4KB pages of the virtual memory onto the computers hard disk to free up the physical memory (RAM) for other processes. In Windows operating systems, these pages are stored in the pagefile.sys.

A good way to think of this is;

Imagine a restaurant that has just open for the evening. When customers (Processes) arrive they get allocated a table (RAM) to sit and eat at. As the night goes on the restaurant get busier and free tables (RAM) begin to run out for the new customers (Processes) coming through the door. To free up spare tables (RAM) the waiter asks customers (Processes) who have finished eating if they wouldn’t mind moving to the bar (Virtual Memory) where they can continue drink.

Without the paging file, if the physical memory becomes full, applications including the operating system will have to waiting until physical memory becomes available before it can be stored in RAM ready for the CPU to process. As you can imagine this causes massive performance problems.

In summary, you NEED to have a paging file.
Read the rest of this entry »

© Simon for Gestalt IT, 2012. | VDI Paging Files – Big? Small? Or Non At All?
Read more posts categorized as Desktop, Server Virtualization

It's not going to be this easy to bridge Fibre Channel and Ethernet!

Before the holidays, I posed a question on Google+ that generated quite a bit of interest and feedback. Now that it has settled down a bit I’d like to summarize the unresolved elements to make FCoE truly a world-class storage interconnect.

Setting the Stage

FCoE has been a controversial topic in both storage and networking, and for good reason. No one would deny that Ethernet is not an ideal transport mechanism for block storage I/O. “Porting” Fibre Channel to run on Ethernet networks has been a supreme technical challenge, and many companies and individuals have labored long and hard to make FCoE a reality.

Now that FCoE is specified in the standard and has been deployed in production environments, the question turns to its future. Will it take off and seize the mantle of dominance currently held by what I like retroactively to call “Fibre Channel over Fibre Channel?” Will they coexist for the next decade, with FCoE mainly deployed in “block” environments such as Cisco UCS? Or will FCoE ultimately fail to catch on, displaced by some other storage protocol like plain FC, iSCSI, NFS, or something entirely different?

The data center needs a flexible new protocol to meet the needs of virtual environments, and convergence of storage and data networking makes a great deal of sense in these environments. This was the root of my question, and I ask it in all earnestness.

My question: What elements remain unresolved to make FCoE truly world-class? What should the vendors be prioritizing? Here are the answers I received.

Technical Considerations

Link Aggregation on CNA’s

Converged network adapters (CNA’s) allow multiple protocols to access a single Ethernet connection, but some also include multiple ports that can be aggregated. In traditional Ethernet networks, link aggregation is a respectable approach for performance and availability. But storage networks have traditionally relied on host-based MPIO software, and these features are mutually exclusive. The zeitgeist seems to be a recommendation to avoid link aggregation on CNA’s that are used for storage networks.

How Do You Handle Virtual Machine Mobility?

As I described recently, virtual machine mobility is a major technical challenge for existing networks. The VMware proposal, the VXLAN, seems to be gaining traction right now. But this is only a solution for data networking. How will FCoE SANs handle virtual machine mobility? This remains unresolved as far as I can tell, though Ethernet switch vendors have come up with their own answers. Brocade demonstrated just such a solution at Networking Field Day 2, and I know that others have answers as well. But will there be an interoperable industry solution?

How Should FCoE Be Implemented Over Longer Distances?

Fibre Channel has traditionally relied on routers and other protocols (FCIP and iFCP) to span distances, but FCoE raises the possibility of native traversal. While it is certainly possible to span distances with FCoE, this is definitely not a recommended or supported idea. Without TCP/IP, or any routing mechanism, it’s just a bad idea. But I imagine that it won’t be long before vendors decide to give it a go anyway.

Implementation Considerations

Is TRILL Required for FCoE Networks?

This has been one of my own questions since the very beginning. Clearly, edge only FCoE works just fine without TRILL. But as networks become more complicated, and virtual machines move, it seems an awfully good idea to have some protocol to alleviate East-West routing concerns. I feel much better with TRILL (or some similar Ethernet fabric technology) in a complicated FCoE network.

Should All Switches Be Full FC Forwarders?

There are number of ways to implement FCoE on Ethernet network, and not all involve building a full Fibre Channel stack in each switch. While many (including myself) assumed that FCoE implied Fibre Channel forwarding in all switches, this is clearly not the direction taken by vendors, at least initially. Perhaps the current “Ethernet forwarding” approach is only a stepping stone, or perhaps it will emerge as the dominant FCoE standard.

How Will OpenFCoE and LoM Be Used?

OpenFCoE is a software solution allowing FCoE to be run without a CNA. If this became popular, it wouldn’t be long before data center architects began looking at LAN on Motherboard (LoM) and even 10GBASE-T as a potential SAN alternative. Will this be used in the long run? It could happen, but it’s certainly not something that’s here at the moment. But OpenFCoE is a real player, especially with Intel’s backing.

How Will Technologies like Zoning Interoperate?

Many networkers are just now beginning to see the true complexity of Fibre Channel SANs. Although interoperability of higher-level Fibre Channel functions between vendors has never been a priority in “FC over FC” SANs, Ethernet could change things. I would not be at all surprised to see a groundswell of customer support demanding greater levels of interoperability from FCoE than from FC, and zoning and VSAN is the likely first beachhead.

The Big Question: When Will We See the “Killer App” For FCoE

Just about everyone agreed that the real challenge for FCoE is market acceptance. Customers aren’t yet demanding FCoE, and vendors are finding it hard to articulate a compelling case to move from “tried-and-true” FC. Convergence, cost savings, and performance have all been put forth, but customers aren’t biting. Perhaps they just need a little time and a little more proof.

This post relies extensively on feedback from a number of people, including Ivan Pepelnjak, Tony Bourke, J Metz, Dmitri Kalintsev, Derick Winkworth, David Hardaker, Juan Lage, and Corey Hines.

© sfoskett for Stephen Foskett, Pack Rat, 2012. |
Eight Unresolved Questions About FCoE


This post was categorized as Enterprise storage, Everything, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you’d like to filter out or focus on posts like this.

© Stephen Foskett for Gestalt IT, 2012. | Eight Unresolved Questions About FCoE
Read more posts categorized as Storage

Note: I realize that they are doing this already, but until recently I hadn’t been close enough to a product that used it to worry much about it.

Then I started getting into Exchange 2010.  PowerShell for managing E-mail from almost all aspects of the process is a damn fine idea.  Now I have a reason to learn more than a few simple commands, because I might actually put them to use.

Where I have been

I signed up for an account at PowerShell.com to get my feet wet and hopefully participate in a community. I have found this to be somewhat useful in the past and am hoping that it allows me an outlet to go dig around in other peoples scripts to see if I can comprehend just what they are doing.

I am on the fence about ISE… I know at some point notepad will become a burden to use, but just starting out, I don’t think I need to worry about that just yet.  If anyone has any suggestions for things to look at in terms of ISE or just good resources for learning PowerShell, please post them in the comments.  I am anxious to get a jump on this thing in the hopes that it will be worth the effort to understand.

Tons to learn

I have tried a few things that I found while binging my way around the web and it has been interesting to see what is out there, surely I haven’t even scratched the surface yet.  I would think a PowerShell magazine or some newsletter type offering would be a huge benefit to the PowerShell community.  Maybe the guys at Redmond Magazine would consider getting something wild like that off the ground??

For now, I will begin re-perusing the books I have on PowerShell and dig into the Internet on the subject further.  Maybe there will be an event near me in the future that will help me learn… I will keep my eyes peeled for that for sure.

© derek for Gestalt IT, 2012. | PowerShell… What an awesome tool
Read more posts categorized as Desktop, Server Virtualization

Windows 8 server editions will include a filter driver for NTFS for data deduplication

The next version of Microsoft Windows Server includes integrated data deduplication technology. Microsoft is positioning this as a boon for server virtualization and claims it has very little performance impact. But how exactly does Microsoft’s de-duplication technology work?

Introducing Windows 8 Deduplication

Let’s make one thing clear right from the start: Microsoft started from a clean sheet and invented their own deduplication technology. This is not a licensed, cloned, or copied feature as far as I can tell. There are some clever aspects to it, along with a few head scratchers for folks like me who’ve seen lots of different deduplication approaches.

Microsoft’s deduplication is layered onto NTFS in Windows 8, and will be a feature add-on for Server users. It is implemented as a filter driver on a per volume basis, with each volume a complete, self describing unit. It is cluster aware, and fully crash consistent on all operations. This is a pretty neat trick: As is typical for Microsoft, deduplication will be a simple, transparent feature.

Now let’s talk for a moment about what Windows 8 deduplication is not.

• It is a server-only feature, like so many of Microsoft’s storage developments. But perhaps we might see it deployed in low-end or home servers in the future.
• It is not supported on boot or system volumes.
• Although it should work just fine on removable drives, deduplication requires NTFS so you can forget about FAT or exFAT. And of course the connected system must be running a server edition of Windows 8.
• Although deduplication does not work with clustered shared volumes, it is supported in Hyper-V configurations that do not use CSV.
• Finally, deduplication does not function on encrypted files, files with extended attributes, tiny (less than 64 kB) files, or re-parse points.

Some Technical Details on Deduplication in Windows 8

Microsoft Research spent 2 years experimenting with algorithms to find the “cheapest” in terms of overhead. They select a chunk size  for each data set. This is typically between 32 KB and 128 KB, but smaller chunks can be created as well. Microsoft claims that most real-world use cases are about 80 KB. The system processes all the data looking for “fingerprints” of split points and selects the “best” on the fly for each file.

After data is de-duplicated, Microsoft compresses the chunks and stores them in a special “chunk store” within NTFS. This is actually  part of the System Volume store in the root of the volume, so dedupe is volume-level. The entire setup is self describing, so a deduplication NTFS volume can be read by another server without any external data.

There is some redundancy in the system as well. Any chunk that is referenced more than x times (100 by default) will be kept in a second location. All data in the filesystem is checksummed and will be proactively repaired. The same is done for the metadata. The deduplication service includes a scrubbing job as well as a file system optimization task to keep everything running smoothly.

Windows 8 deduplication cooperates with other elements of the operating system. The Windows caching layer is dedupe-aware, and this will greatly accelerate overall performance. Windows 8 also includes a new “express” library that makes compression “20 times faster”. Compressed files are not re-compressed based on filetype, so zip files, Office 2007+ files, etc will be skipped and just deduped.

New writes are not deduped – this is a post-process technology. The data deduplication service can be scheduled or can run in “background mode” and wait for idle time. Therefore, I/O impact is between “none and 2x” depending on type. Opening a file is less than 3% greater I/O and can be faster if it’s cached. Copying a large file can make some difference (e.g. 10 GB VHD) since it adds additional disk seeks, but multiple concurrent copies that share data can actually improve performance.

Stephen’s Stance

Although I am intrigued by Microsoft’s new deduplication technology in Windows 8 server, I still have many questions about its usefulness and impact on performance. Concentrating duplicate data in the system volume makes sense from a technical perspective, but could lead to an I/O hotspot on the disk. This is especially true for external caching storage systems, since there is no integration between Microsoft deduplication and storage array features. I am particularly concerned about the use of deduplication with VHD files in Hyper-V, since it could eat up valuable system RAM and impact I/O performance.

If you would like to try Microsoft deduplication for yourself, I am happy to report that it is included in the developer preview of Windows 8 that is available on Dev Center. Here are a few commands to get you started, and read Rick Vanover’s post too!

Import-Module ServerManager
Add-WindowsFeature -name FS-Data-Deduplication
Import-Module Deduplication
Enable-DedupVolume E:
get-dedupvolume

Note: I am a Microsoft MVP and Microsoft briefs me on upcoming technologies under NDA. This post is based on a Microsoft briefing from November which was said at the time not to be covered by any NDA. All of this information could be gleaned by experimenting with the Windows 8 developer preview, but it’s much easier to just go to the source.

© Stephen Foskett for Gestalt IT, 2012. | Microsoft Adds Data Deduplication to NTFS in Windows 8
Read more posts categorized as Server Virtualization, Storage

It isn’t always easy to get where you need to go!

Consider the following situation: You go to lunch with your good friends, John and Mary. Halfway through a rousing discussion of the latest Hollywood movie, Mary starts talking about the fantastic action sequences while John criticizes the romantic angle. You realize something mine-bending has happened: John now has Mary’s personality, and vice versa. It’s like they have switched brains or something!

This truly weird situation isn’t likely to happen in person, but occurs all the time in the data center. Virtualization of server, network, and storage services illuminates the link between physical resources and functional applications. A running virtual machine can instantly move from one server, network adapter, HBA, or LUN to another. And when it happens, traditional components have no idea how to react.

The Challenges of Mobility

Mobility is perhaps the “killer app” of virtualization, but it is also the killer of traditional IT systems. Let’s consider the challenges of this “Twilight Zone” moment.

• The operating system expects a consistent hardware environment, which is exactly what the hypervisor creates
• The LAN must be prepared to redirect all network traffic instantly and seamlessly to one or more new physical interfaces
• The SAN similarly must be able to reroute all I/O to a new pair of HBA’s without missing a beat
• The storage array must be able to re-present capacity to a new physical device, and must maintain snapshots and other configurations
• The backup system must also be able to maintain consistency over time even as machines relocate to different server and storage locations

All of this must be done while maintaining quality of service (QoS), access control, reporting, and appropriate segmentation at all levels. This is an incredibly challenging task, and no conventional protocol (IP, Ethernet, NFS, SCSI, Fibre Channel, etc.) is anymore ready then you are when you’re good friends switch personalities.

Two Paths

So much of the development that is currently taking place in IT focuses on accommodating this “mobility issue”. Two key approaches have emerged to take on this challenge:

• “In a vacuum” technologies (like VXLAN) assume that no other changes will be made, so the focus is on maintaining complete compatibility in front and behind
• “Clean sheet” technologies (usually from startups) take a different approach, throwing out compatibility in favor of technical elegance

Both of these approaches have merit. Attempting to maintain compatibility only works so far (just ask a Windows API programmer), but it leverages the existing environment and recognizes that most people are not ready for wholesale change. Clean sheet designs always make more sense, but they rarely attain mass acceptance. Nearly every technology we rely on today is full of bolt-ons in the name of compatibility. Some, like Ethernet and x86, actually work pretty well, too.

The Stack of Lies

The difference between virtualization and cloud computing is exactly this same distinction. Hypervisors, NPV, NAT, thin provisioning, and so many other virtualization technologies exist mainly to maintain compatibility in a vacuum. In contrast, true cloud computing dispenses with the entire stack and creates a new platform for applications.

This is, perhaps, the reason that cloud computing is not taken off in the enterprise. Simply put, IT is not prepared to ditch everything they have ever used even in the face of a demonstrably superior alternative. Currently, the highest use of cloud is behind gateways and virtualization engines that bring it back down to earth.

This brings us to the stack of lies called server virtualization. Any “modern” virtualized data center is built on lie after lie, with each level telling the other what it wants to hear. The volume manager lies to the operating system, hypervisor lies to the volume manager, and the storage array lies to the hypervisor. The same sad state of affairs allows networking and even memory to function in a virtual world.

But these shaky stacks of lies have difficulty adapting to motion, since no level truly “knows” the reality of the world around. The depressing truth is that a bowl of spaghetti like VXLAN is perhaps the highest form of art we can expect in a virtual data center.

Stephen’s Stance

As a techie, I am always drawn to clean sheet designs that offer technical elegance along with functionality. But I know that, realistically, products that assume nothing about the world around them and bend over backward to maintain compatibility are more likely to succeed. Still, I maintain hope that the issues of virtual machine mobility will be solved in an elegant way, rather than adding to the “stack of lies”.

© Stephen Foskett for Gestalt IT, 2012. | The Terrifying True Story Of Virtual Machine Mobility
Read more posts categorized as Server Virtualization, Storage

This is Tom. Ask him about NAT!

After reading the post, I had some thoughts come to mind that I just had to throw into a reaction post.

New Culture

As new generations of individuals grow up and mature, it is expected that cultural shifts will take place. What I do not understand is how a culture of technological availability has morphed into an expectation that an individual can bring anything into the corporate environment and expect to use it for their job.

Too many times, I am approached by users bringing their personal laptop into the office and wanting to know how to connect it to the internal network. Or, users that want to connect their iPhones to the network so they can use Spotify or YouTube without using their cellular data plans… as though the corporate infrastructure and services are there to do their bidding.

This new culture developing assumes that everything in the outside world must be the same as in the corporate world. Their iPad can connect to GMail, so why not just connect it to the Exchange server?

What the user sees.                            What IT sees!

Unknown/Unowned Devices

The IT ecosystem is a carefully designed and tightly guarded world.

None shall pass!

Systems are selected carefully to ensure a proper balance between functionality, supportability, and stability. The discovery of an unknown device is enough to throw an IT professional into a fit of rage. The environment has been compromised in some fashion and there is potential to throw off the carefully designed balancing act.

The presence of an unknown device opens up a venerable Pandora’s Box and raises a huge red flag. Suddenly, the corporate environment is now vulnerable to a machine or device infested with trojans, a honeypot of virus infections, access to corporate resources, and not managed by IT.

IT has been assigned a critical role in modern businesses… provide tools that enable the business to function. Traditionally, this included the workstation, network, monitors, servers, etc… With more people feeling as though it is acceptable to provide their own devices, who is responsible for supporting them? What happens when the “S” key breaks off or the monitor is too blue for their liking. When IT owns and manages a device, IT is responsible. When the users owns the device, but is using it in a corporate environment, the answer is much foggier. An IT persons says the user is responsible. However, the true answer lies somewhere in the depths of politics and policy.

Unknown devices also introduce the loss of data control. The moment a user is allowed to bring in a USB drive, iPod, access GMail, or Dropbox, the data is no longer under any control of the company.

Corporate IT Adaptation

First and foremost, IT has a responsibility to the company to ensure the protection and function of corporate technological resources and systems.

However, with that said, IT needs to acknowledge the changing ways of technology. Anyone who has been in IT longer than 1 month knows that times have a way of changing and the minute you buy your phone, it is obsolete. That is the way of the world and 42 is the ultimate answer to the ultimate question of live, the universe, and everything.

Is Google “Deep Thought”?

IT departments need to be cognoscente of what exists in the marketplace, impacts (both positive and negative) to overall productivity/security, and the long term viability of those entities. A tablet, for example, may seem like a large phone (cough iPad cough). However, for an executive that spends more time meeting customers and reading email, it is a perfect tool to enable them to get their job done without needing a laptop… but how is it secured?

Security becomes one of the most important concerns for IT in a time where users have expectation of providing their own devices. NAC/NAP/Port Security ensures authorized devices are allowed on the network. Remote technologies (Application Presentation (XenApp/RemoteApp) and VDI (View, XenDesktop)) allow users to interact with applications running on protected and trusted infrastructure from unknown endpoints. Proper backups, snapshotting, and antivirus on the server and storage side ensure the data consistency is proper and recoverable in the event of a break in security.

Finally, IT needs to engage with the business to keep them abreast of concerns. Open dialogue with the business will help ensure technological expectations meet some sort of equilibrium between what IT feels is appropriate and what the business feels is necessary.

What do you really think, Bill?!

I whole heartedly do not like the idea of users bringing in their own devices for business use. Maybe I am cruisin for a bruisin (politically speaking), but I see my environment as known and trusted. The introduction of a new device takes some planning and testing because I have a responsibility to the company to provide a stable and operational environment. The introduction of a Mac laptop into my environment is not smooth. Exchange and SharePoint support is so horrible that Mac users need to use a Fusion VM running Windows 7 to fully function.

However, while it is possible to be completely restrictive and be more like “The Man”, I feel that the best way to manage the user owned devices converging on my environment is more political.

- I encourage the business to adapt corporate policies addressing the need to not bring personal devices into the environment.

- I encourage the business to develop a stricter definition of who needs email outside of the office, partial compensation for use of personal devices OR providing a company owned and managed phone, and which devices are supported.

- Have an open and friendly dialogue with those users that approach IT for assistance with personal devices. Being honest and frank about not supporting devices, needing management approval, and being unsure as to the functionality/operation of the device goes a long way.

I love the idea of new devices and new technology in the workplace. But, I want the introduction to be more structured and tested.

Tom – Thanks for the awesome post. Definitely food for thought and got my wheels spinning!

© Bill for Gestalt IT, 2011. | BYO(a)D Reaction
Read more posts categorized as Desktop, Featured

I have been waiting for some time to get some ESXi 5.0 awesomeness going on in my environment. vCenter has been sitting on v5 for some time and I have been deploying ESXi 5 in a couple stand-alone situations without any issues. So, now that I have more compute capacity in the data center, it is time to start rolling the remaining hosts to ESXi 5… or so I thought!

I downloaded ESXi 5.0.0 Kernel 469512 a while back and have been using that on my deployments. So far, so good… until today. Update Manager configured with a baseline –> Attach –> Scan –> Remediate –> back to business. Surely, Update Manager processes should take more time than the actual upgrade. About 30 minutes after starting the process, vCenter was showing that the remediation progress was a mere 22% complete and the host was unavailable. I used my RSA (IBM’s version of HP ILO or Dell DRAC) to connect to the console. Sure enough, it was stuck at loading some kernel modules. About 20 minutes later IT WAS STILL THERE!

Restarting the host did not resolve the issue. During the ESXi 5 load screen, pressing Alt + F12 loads the kernel messages. It turns out that iSCSI was having issues loading the datastores in an acceptable amount of time. I was seeing messages similar to:

A little research turned me onto the following knowledgebase article in VMware’s KB: ESXi 5.x boot delays when configured for Software iSCSI (KB2007108)

To quote:

This issue occurs because ESXi 5.0 attempts to connect to all configured or known targets from all configured software iSCSI portals. If a connection fails, ESXi 5.0 retries the connection 9 times. This can lead to a lengthy iSCSI discovery process, which increases the amount of time it takes to boot an ESXi 5.0 host.

So, I have 13 iSCSI stores on that specific host and multiple iSCSI VMkernel Ports (5). So, calling the iSCSI lengthy is quite the understatement.

The knowledgebase states that the resolution is applying ESXi 5.0 Express Patch 01. Fine. I can do that. And… there is a work around described in the article that states you can reduce the number of targets and network portals. I guess that is a workaround… after you have already dealt with the issue and the ridiculously long boot.

Finally, to help mitigate the issue going forward, VMware has released a new .ISO to download that includes the patch. However, this is currently available in parallel with the buggy .ISO ON THE SAME PAGE! Seriously. Get this… the only way to determine which one to download is:

As a virtualization admin, I know that I am using the Software iSCSI initiator in ESXi. But, why should that even matter at all?! There is a serious flaw in the boot process in version 469512  and that should be taken offline. Just because someone is not using Software iSCSI at the current time does not mean they are not going to in the future. So, if they download the faulty .ISO, they are hosed in the future. Sounds pretty crummy to me!

My Reaction

I am quite shocked that this made it out of the Q/A process at VMware in the first place. My environment is far from complex and I expect that my usage of the ESXi 5.0 hypervisor would be within any standard testing procedure. I try to keep my environment as vanilla as possible and as close to best practices as possible. 1.5 hours for a boot definitely should have been caught before release to the general public.

Additionally, providing the option to download the faulty ISO and the fixed ISO is a complete FAIL! As mentioned on the download page, this is a special circumstance due to the nature of the issue. I would expect that if this issue is as serious as the download page makes it out to be, the faulty ISO should no longer be available. There has to be a better way!

Conclusion

I have since patched the faulty ESXi 5.0 host to the latest/safest version, 504890, and boot times are back to acceptable. I will proceed with the remainder of the upgrades using the new .ISO and have deleted all references to the old version from my environment.

I have never run into an issue like this with a VMware product in my environment and I still have all the confidence in the world that VMware products are excellent. In the scheme of things, this is a bump in the road.

© Bill for Gestalt IT, 2011. | ESXi 5.0–1.5 Hour Boot Time During Upgrade
Read more posts categorized as Server Virtualization

The great battle of enterprise storage is on!

The time has come to take sides on the core question of storage for virtual servers: Do you want storage intelligence to live in the hypervisor or the array? Most administrators are already lining up on one side or the other, unintentionally casting their vote while the rest flounder. But the storage industry must wake up and embrace the divide.

Hypervisor Huggers Unite!

Do you “heart” virtualization?

VMware’s vSphere dominates the world of enterprise server virtualization and has become the core element of the modern open systems datacenter. Microsoft recognizes this but has been unable to field a competitive hypervisor ecosystem for the virtual datacenter. Today, vSphere is the state of the art and nowhere is this more apparent than in storage.

In just a few years, VMware has delivered and updated a host of advanced storage functionality, from provisioning to migration and load balancing to backup and data protection. vSphere 5 includes an advanced and scalable storage virtualization layer, delivering everything a datacenter needs. VMFS sculpts basic block storage into a shared resource for virtual machines, with snapshots, policy-based layout and movement, and flexible allocation and thin provisioning.

Most VMware administrators are “server guys” and relish these features. They have never experienced an automated “storage service” like this, and the enterprise storage world has never been able to construct anything remotely as flexible, user-friendly, and functional. And Hypervisor Huggers don’t need complex enterprise storage arrays to do it: They can use basic iSCSI or Fibre Channel devices to provide performance and capacity and let VMware do the rest!

Storage DRS is exemplary of the new virtual datacenter world. Introduced in vSphere 5 (and restricted to the pricey Enterprise Plus license), Storage DRS uses the core technology of Storage vMotion to dynamically balance I/O and capacity across a diverse pool of storage. Storage DRS even uses Policy-Driven Storage and VASA to enforce tiered storage and data placement strategy. This kind of virtualization has been a “holy grail” quest for the enterprise storage industry, but they’ve never delivered on their promises.

Cheers for Storage Stalwarts!

Do you wish all this virtualization nonsense would just go away?

But not every IT environment wants be 100% vSphere focused, and many aren’t convinced that dumb storage is the smartest place for their data. These Storage Stalwarts want smarter and better-integrated storage arrays, and VMware is innovating here as well.

VMware’s Storage APIs for Array Integration (VAAI) technology is squarely aimed at this market. VAAI allows vSphere to hand off heavy storage operations to the high-end storage arrays from the major players. It works transparently, too, coordinating cloning without the kind of scripting and hair-pulling that used to require. VAAI in vSphere 4.1 also includes block zeroing support and something called “atomic test and set” which we’ll get to in a moment. Microsoft announced their own cloning integration, ODX, but it won’t ship until Windows Server 8 appears sometime next year.

But cloning is just the tip of the iceberg when it comes to Hypervisor-to-array integration. The rising army of NAS users have seen the glory of more-complete array integration for a while, and they’re not quiet about it. They love that VMware’s NFS protocol support makes storage “disappear” in vCenter, becoming just another resource with integrated thin provisioning and flexible allocation and movement.

VMware is moving aggressively to please their Storage Stalwarts, adding more VAAI support for block and file in vSphere 5. But, as the company laid out at VMworld 2011, neither access method is ideal for virtual servers. So VMware has been pushing the enterprise array vendors for ever-greater integration. They see a future where a VAAI-based protocol enables arrays to de-multiplex I/O streams from the hypervisor and intelligently handle per-VM data.

Stephen’s Stance

You can spot a Hypervisor Hugger by their big LUNs: They would rather treat storage as a bulk commodity, and array vendors should be lining up to get their business. Storage Stalwarts will jump on each new VMware innovation, finally making use of the capabilities they have spent over a decade paying for but not utilizing. The only untenable stance is trying to keep a foot in both worlds: It’s foolish to buy an enterprise array and use it as bulk storage!

© Stephen Foskett for Gestalt IT, 2011. | Are You a Hypervisor Hugger or a Storage Stalwart?
Read more posts categorized as Featured, Server Virtualization, Storage

The PXE environment components included with the VCVA are not configured and turned off by default. So, if you’re ready to configure your VCVA for PXE, time to roll up your sleeves, crack those knuckles, and get ready to get your hands dirty.

Before we get started, though, and little caution (and disclaimer so I can sleep better at night):

I know nothing about your environment. You are following these instructions at your own risk. This setup will impact DHCP functionality on your network. Follow these instructions at your own risk and make the appropriate adjustments to work in your environment.
Additionally, I do not know everything about everything. So, you are going to need to rely upon your sleuthing abilities to help resolve issues that may arise.

These instructions assume some knowledge of CLI-based file editing (vi). So, please research how to use it if you are unsure.

Overview

A PXE environment via the VCVA requires the following components in your environment

• DHCP server
• TFTP server
• Web Server (for kickstart scripts)
• SYSLINUX (for pxeboot.0)
• Access to an ESXi 5.0 installation CD (perhaps you created on using my Image Builder tutorial)
• vCenter Virtual Appliance deployed
• Blank server to PXE boot and install ESXi 5.0 on (aka – the client)
• ESXi 5.0 installation .ISO
• HTTP server on the network (for hosting kickstart files – customization during installation)

For this exercise:

• Network: 192.168.226.0/24
• VCVA: 192.168.226.21
• DHCP Range: 192.168.226.200 – 254
• Default Gateway: 192.168.226.1

Configuration

0 – Log into the appliance as ‘root’

1 – Configure DHCP

‘dhcpd‘ will listen to IP address requests, provide an IP to use, direct the client to the “next-server” to continue PXE booting, and which file (filename) to download from the server.

• cd /var/lib/dhcp/etc
• cp -a dhcpd.conf dhcpd.conf.orig
• vi dhcpd.conf

Once inside of the file, ensure the following exists (highlighted for your ease of identification)

ddns-update-style ad-hoc;
allow booting;
allow bootp;

#gPXE options
option space gpxe;
option gpxe-encap-opts code 175 = encapsulate gpxe;
option gpxe.bus-id code 177 = string;
class “pxeclients”{
match if substring(option vendor-class-identifier, 0, 9) = “PXEClient”;
next-server 192.168.226.21;
filename “pxelinux.0″;
}
subnet 192.168.226.0 netmask 255.255.255.0 {
range 192.168.226.200 192.168.226.254;
}

Save the file and exit (hint: :wq)

2 – Configure TFTP

TFTP services are provided by the ‘atftpd’ daemon

• cd /etc/sysconfig
• cp –a atftpd atftpd.orig
• vi atftpd

Once inside the file adjust the “ATFTP_OPTIONS” line to read: “–daemon –user root”. Typically, the atftpd daemon runs as ‘nobody’. However, the TFTP root (/tftpboot/) is configured as owned by the ‘root’ user.

Save and exit the file.

3 – Get the SYSLINUX packages on the server

There is one package missing to make the PXE installation process work: ‘pxelinux.0′. ‘pxelinux.0‘ is an executable that is downloaded by the client in order to properly continue the PXE process (aka – download the files, execute the installer, etc…). ‘pxelinux.0‘ is provided by the SYSLINUX package. In order for PXE to work properly with the ESXi 5.0 installation, SYSLINUX version 3.86 (or higher) is needed.

• cd /tmp
• wget http://www.kernel.org/pub/linux/utils/boot/syslinux/3.xx/syslinux-3.86.tar.gz
• gunzip syslinux-3.86.tar.gz
• tar xvf syslinux-3.86.tar

Note: you can use YUM or copy the files to the server another way if you’d like. Regardless, get the files there. This example will continue to use the /tmp file as the landing area for the SYSLINUX files.

Copy the pxelinux.0 file to your TFTP root

• cp /tmp/syslinux-3.86/core/pxelinux.0 /tftpboot

4 – Prep the TFTP root for PXE

The TFTP root configured on the VCVA is located at /tftpboot. We are going to need to get the directory structure built out to support PXE.

• cd /tftpboot
• mkdir esxi50

By adding a directory, we are able to organize the TFTP server and support additional versions of ESXi going forward.

5 – Get the ESXi 5.0 CD contents onto the server

Seeing as the VCVA is a virtual appliance, it is easy to get the contents of the installation media onto the server.

• Mount the installation CD to the VCVA as a CD-ROM drive using the vSphere Client.
• mount /dev/cdrom /media
• cp –a /media* /tftpboot/esxi50/
• umount /dev/cdrom

6 – Configure PXELINUX

pxelinux is the utility that enables the PXE functionality. As mentioned before, pxelinux.0 is an executable that the server downloads. The executable provides functionality to parse a menu system, load kernels, options, customizations, modules, etc…, and boot the server. Since PXE can be used by multiple physical servers for multiple images, we need to configure pxelinux for this specific image.

• cd /tftpboot
• mkdir pxelinux.cfg
• cd pxelinux.cfg

pxelinux.0 looks for configuration files in the TFTP:/pxelinux.cfg directory.

pxelinux looks for a large number of configuration files… specific to a default/generic value. This allows server administrators to define a file based on a complete MAC address, partial MAC address, or none at all to determine which image to boot from. Since this is the first configuration on the VCVA, we are going to configure a default. Do your research if you want to adjust this from the default value.

The installation media contains a file called isolinux.cfg. We can use this as the basis for our file called ‘default’. Copy it from the installation media and start customizations:

• cp –a /tftpboot/esxi50/isolinux.cfg default
• chmod a+w default
• vi default

Ensure the appropriate lines match the following lines:

DEFAULT /esxi50/menu.c32
KERNEL /esxi50/mboot.cfg
APPEND -c /esxi50/boot.cfg

Save and Exit

7 – Configure the Kickstart file

Using a kickstart file, we can configure ESXi 5.0 automatically during installation. This requires that a file be placed on a server that is available to the client.  Sadly, the HTTP areas on the VCVA are not readily available… and, they may be erased during future upgrades. So, we need to use an external HTTP server somewhere on your network. (Note: NFS and FTP are options as well).

Add the following contents:

# Accept the EULA
vmaccepteula

#Set root password
rootpw supersecretpassword

#Install on first local disk
install –firstdisk –overwritevmfs

#Config initial network settings
network –bootproto=dhcp –device=vmnic0

 

In this example, we are saving the file to:

• HTTP Server: 192.168.226.1
• Directory: ks
• File: esxi50.txt
• URL: http://192.168.226.1/ks/esxi50.txt

8 – Configure the installation files

The CD installation media for ESXi 5.0 assumes a single installation point. Thus, all the files are placed at the root of the image. However, since we want to actually organize our installation root, we added the ‘/tftpboot/esxi50‘ directory and copied the files into it. We need to adjust the installation files in /tftpboot/esxi50 to reflect the change.

• cd /tftpboot/esxi50
• cp -a boot.cfg boot.cfg.orig
• vi boot.cfg
• Using the following picture as reference, add “/esxi50” to the paths for ‘kernel’ and ‘modules’

Save and quit

9 – Restart services to load the service configurations and configure to start with server

• /etc/init.d/dhcpd restart
• /etc/init.d/atftpd restart
• chkconfig –add dhcpd
• chkconfig –add atftpd

10 – Take a break

You made it this far… great job. At this time, we have configured DHCP, TFTP, pxelinux, copied installation media to the TFTP root, and configured the installation for our organizational purposes.

11 – Start your host and install away

[BELOW] Reading the Kickstart Script. No need to enter customization info anymore.

[BELOW] Checking contents of Kickstart file. You will see errors here if errors in file.

© Bill for Gestalt IT, 2011. | vSphere 5–PXE Installation Using vCenter Virtual Appliance
Read more posts categorized as All, Featured, Server Virtualization, Top Story