George Crump, of Storage Switzerland, published an interesting post recently titled Cost Effectively Solving Oracle Performance Problems. Crump explains the challenges of solving Oracle storage performance problems (including several Oracle instances) while keeping Oracle data in shared storage.

In his analysis, Crump details three solid-state storage solutions that address Oracle performance:

• Augmentation to existing mechanical storage via tiering or caching;
• Using SSD on Oracle’s application server itself to cache data;
• Using forklift upgrade solutions or database machines such as Oracle Exadata.

I agree with him on many points but disagree on others. First, here is what he says about Flash devices such as Kaminario K2:

Here are my thoughts in response:
The challenge is that typically these vendors have limited experience in delivering the types of storage services that Oracle administrators have become accustomed to.
This is no longer true. Today, Kaminario K2 offers sophisticated SAN features important to Oracle administrators including lightening fast snapshots and non-disruptive operations.

They also use deduplication and compression to optimize a flash only investment, which can, in some cases, hinder overall system performance.
I agree that deduplication and compression can slow down Oracle performance and might not be a good solution for all Oracle workloads. However, K2 does not currently use deduplication and compression for database applications. Honestly, I keep asking myself why someone would use SAN compression when you can use database compression instead. I would let the RDBMS handle the compression. This, perhaps, is a topic for a future post.

Introducing any new product has a learning curve, but we are proud of the K2’s simple installation requirements (no tuning required). We have seen customers get up and running very quickly. For them, installation and integration has not been a significant issue at all.

Crump then talks about the option of keeping the existing storage and adding a flash read-only cache appliance. So, why am I against this?
The idea of caching is to quickly serve data that was served before. Most Oracle read performance problems are random read or single blocks. This is where mechanical disk storage is limited. If Oracle needs to read a block, that block needs to be in the cache appliance to improve performance. But will that block be in the cache? Only if that block is being used a lot (was served before). We call these hot-blocks. BUT, if these blocks are hot, they will already be in the Oracle internal cache and therefore Oracle will not read them. You’d end up double caching without a lot of improvements, unless the entire SAN data is placed in the caching appliance.

Some applications are dynamic in nature with real random access. You will not get much improvement from a caching solution compared with placing the entire database on a Flash array. We have seen this with Oracle Flash Cache vs. placing the entire tablespace on SSD.

I agree that caching will improve writes BUT not as much as placing an entire database on Kaminario K2. There is no doubt there.

Finally, if you really like your existing SAN and want to keep it, use a solution like Kaminario K2 mirrored to the existing SAN. Use ASM or OS mirroring to assure that writes go to both storages, but reads are served only from K2 (prefer-read option). Then you will get both read and the write improvements. This solution will work for every Oracle application.

If you are having problems with your Oracle database performance, you should consider participating in our Oracle Database Performance Assessment. It is free and you will learn a lot about your Oracle database behavior from the experience.

(photo credit)

Another interesting seminar at Stephen Foskett’s Tech Field Day, hosted by Nigel Poulton, addressed the best architecture for an SSD array. Participants included Thomas Isakovich, CEO and Founder of Nimbus Data Systems, Umesh Maheswhari, CTO and founder of Nimble Storage, Jonathan Goldick, software CTO at Violin Memory, and Dave Wright, founder and CEO of SolidFire. It was a pretty lively debate, with Goldick grinning broadly through much of it and taking jabs at the others. One couldn’t help but wonder what he was grinning about.

The truth is, there was a lot of disagreement over the best array architecture and sometimes the argument got a tad heated and personal, much to the delight of the audience. However, there were three things everyone could agree to. First, the best architecture is one that provides an ideal balance of scalability, share ability, reliability, and performance, not performance alone. Second, for all but the few most performance- and latency-sensitive applications, it’s more important to provide consistent, predictable performance for an array of applications, than to provide the absolute best performance. And third, the best architecture is a mix of commodity hardware and a software architecture designed from the ground up for SSD. Sound familiar?

Interesting first question: “Do array vendors have their heads up their rears putting SSD on the network rather than inside the server (not my language)?” The answer: An SSD array brings latency from about 10 or 20 milliseconds to a few hundred microseconds. The extra 50 to 100 microseconds saved by a server-based architecture doesn’t matter for all but the absolute most performance hungry applications and you’re constrained by a single server. Goldick countered that latency was important, but acknowledged that most applications didn’t need the lowest latency.

Then came the question of all Flash vs. hybrid disk/Flash. Umesh argued that a mix of Flash and disk provides a balance of performance and capacity. Wright touted an architecture that can balance capacity and performance for each application, including a mix of high- performance and less-costly, lower-performance Flash. What about the cache solution? Goldick dismissed Flash cache’s inability to support writes and its tendency to break down every time a change is made or an application is restarted. Isakovich favored tiering at the application layer to allow for best of breed solutions in each tier, rather than an expensive single tiered storage solution.

Next came a question about the impact of inline deduplication on performance. Isakovich was openly skeptical that deduplication could be nondisruptive, pointing out snidely that “none of the vendors who make that claim allow you to test their solutions with dedupe turned off.” Goldick promoted immediate asynchronous deduplication, rather than in-line, as the way to minimize disruption. Most of the members agreed that solutions should allow dedupe for the applications it benefits and let you turn off dedupe for those that don’t.

To achieve nondisruptive dedupe, Wright argued for a scale-out architecture, similar to that of the K2, that has a high ratio of CPU to memory and doesn’t put excessive pressure on a single or dual controller. Goldick countered that in a scale out architecture every controller has to be informed of any deduplication that takes place and so there is an impact on performance.

As I said, they all agreed that the future belongs to a combination of off-the-shelf hardware and a software architecture geared for SSD. Also that individual disk format SSD’s do not provide the consistent performance users need. Uh, we agree.

Click here to see the members face off yourself:

Last week, Kaminario CEO Dani Golan presented at Stephen Foskett’s Tech Field Day in San Jose. The event afforded us the opportunity to have conversations with several data storage bloggers including Howard Marks, Nigel Poulton, Chris Evans, Ray Lucchesi, Robin Harris and Hans DeLeenheer to name a few.

I was fortunate to be in the room, and the feedback I heard is gratifying. In fact, DeLeenheer published his thoughts about Kaminario in his blog and said we were worth watching.

In addition to providing a K2 product overview, Golan shared insights about the SSD market and where Kaminario fits. “Solid-state storage is the biggest storage revolution in the last 30 years…one of the biggest in IT since virtualization.”

One of the effects of this revolution is that definitions for high end, mid range and low end are being turned on their head. Improving price/performance and increasing application requirements make it tough to distinguish among tier 0, tier 1, tier 2 etc. I would argue “that our software stack is far superior to a $3 million high-end [HDD] array,” Golan said.

Interestingly, Golan noted that when the company began speaking with potential customers early on, the most surprising thing was the level of interest among startups. Startups typically do not have legacy SAN environments and large IT staffs. These organizations wanted SSD performance rather than investing in older HDD technology and wanted something simple to deploy.

Poulton and others asked questions about Kaminario’s market focus and why the company has not yet integrated deduplication.

“We are aiming at the enterprise market,” Golan said. “That is our focus.” With regard to dedupe, he said that is coming, but Kaminario first wanted to focus on fundamental capabilities such as scale-out architecture and data protection so that the K2 would fit in mission-critical environments for OLTP, OLAP and other applications on both physical and virtual servers.

Golan also said that “Kaminario is 100 percent a software company.” This delivers many customer benefits because it can ease scalability and adapt well to hardware innovations such as increased density and new networking capabilities. There is a real power in being a software play in this space because of the flexibility it offers. Kaminario is only at the beginning of qualifying all kinds of applications and hardware combinations that ultimately will deliver value to customers, especially enterprises that have multi-storage vendor environments.

Watch Golan’s presentation in full below for additional details plus highlights of Kaminario’s SPEAR and DataProtect technologies.

The press has picked up on indications that EMC is about to acquire future SSD array vendor XtremeIO. This is an interesting development coming on the heels of EMC’s Project Lightning and Thunder announcements. As everyone knows, EMC is the mother of all disk storage vendors and has, until now, touted SSD primarily as a cache solution fronting and accelerating scores of legacy EMC disk storage arrays. Project Lightning and Thunder reflect this strategy, with Lightning providing a server based PCI SSD read cache solution and Project Thunder looking to do the same thing with a storage array.

XtremeIO is in prerelease semi-secretive mode right now, but has said clearly that it aims to produce pure SSD arrays to compete with the likes of Kaminario, Violin Memory, and all the other usual SSD array suspects. An EMC acquisition of such a vendor indicates that holes have developed in EMC’s SSD cache armor and the disk storage giant feels forced to validate SSD arrays as a large, viable, growing market competing with disk. It will be interesting to see how EMC integrates XtremeIO’s technology into its strategy without eating into its bread and butter disk array product line.

Just a quick note on this development. Pure Storage recently published a blog that comments on the EMC acquisition and portrays Pure as the only Tier 1 SSD array vendor, relegating Kaminario and the rest to Tier 0 niche solutions that compete solely on performance and lack scalability and price competiveness. Excuse me Pure, but the K2 is a highly scalable, price competitive solution used for many mainstream Tier 1 applications. In fact the Kaminario K2 is the only solution on the market that can scale to two or more enclosures for 100 TB yet still be managed as one from a central console.

The recent introduction of Dataprotect high availability and reliability capabilities indicates clearly that Kaminario is aiming for mainstream enterprise storage offering much more than just performance. And every feature we’ve included or added to the K2 is price conscious, with the ability to mix and match faster DRAM and more affordable Flash, numerous techniques for ensuring very long Flash life (see Built for Speed and Endurance), and RAID 10HD’s deployment of less expensive Flash or Disk for backup and reliability. Check out What You Need to Know about SSD HA and Data Protection for more information on how we ensure disaster recovery while holding down cost and take a look at these case studies to see how many mainstream applications are running on the K2 in every business market.

Welcome to the party, EMC. Thanks for the validation.

You’ve probably heard about the endurance limitations of Flash–particularly MLC–and the hoops manufacturers jump through to lengthen life expectancy. If you really want to understand what this issue is all about and how SSD vendors handle it, check out Eric Slack’s Storage Switzerland post entitled Why Flash Wears Out and How to Make it Last Longer.

Slack provides a very thorough explanation of how and why NAND Flash degrades, why MLC degrades faster than SLC, what actually happens during that degrading process, and all the tricks SSD manufacturers employ to slow it down. Techniques include sophisticated error correction, spare blocks of NAND flash that take over when one block degrades, and wear leveling, which distributes write operations across available blocks to ensure that a single block doesn’t wear out prematurely. Vendors also embed advanced technologies, such as digital signal processing, in their SSD controllers to reduce bit errors and reduce the workload on the error correcting (ECC) engine, and employ sophisticated read level adjustments to recognize data on a degraded Flash block. Some SSD controllers can also make sophisticated adjustments to the way a cell is read and written to minimize wear.

Kaminario’s Flash cards come from PCIe Flash card market leader Fusion-io and incorporate many of the best technologies to minimize MLC wear. I’d like to point out a few other measures Kaminario takes, however, that have just as much or even more impact on the superior life expectancy of our Flash array solutions.

First, Kaminario’s sophisticated SPEAR operating system distributes all reads and writes not only across Flash blocks but across entire Flash data nodes. The benefit is not only superior availability and performance, but extended life for each card and each individual block.

Second, Kaminario’s DataProtect availability and disaster recovery capabilities employ a unique technology called RAID 10HD, (HD stands for Hybrid Distributed), which backs up all writes to either hard disk or less expensive SSD, taking the backup write burden off of its production level Flash storage. Hard disks in particular don’t have any of the same wear issues as SSD, so there is virtually no Flash wear from HA and DR.

Finally, Kaminario’s K2 is the only solution on the market that lets you mix DRAM and Flash SSD in the same unit. Not only is DRAM a superior medium for write performance and latency compared to Flash, but it also has none of Flash’s write endurance issues. By placing your write heavy data on a small section of DRAM and leaving Flash for read-heavy applications, you can add years to your Flash life expectancy. So the K2 is not only faster, simpler, and safer, it also lasts longer, something to think about when considering SSD solution TCO.

Does your array have a single point of failure?
Single points of failure are a liability for your HA solution. Some storage arrays still have them especially if they have only one controller. If that is the case, you might need two systems mirrored to prevent complete outages. Depending on the implementation, there could also be single points of failure in servers or Flash cards.

Kaminario K2 is fully N+1 redundant. There is NO single point of failure allowing the K2 to withstand any single failure.

Are your hardware components including Flash hot-swappable?
Can you swap a FRU (Field Replaceable Unit) while the system is operational? Many vendors claim their arrays are hot-swappable but they are not usable during the swap unless it is fully mirrored. Those products require downtime to allow opening a server so Flash cards can be exchanged. This process increases the time needed to return to full speed.

All hardware FRUs in the Kaminario K2 are hot swappable. Kaminario demonstrated this capability in a video.

Does your array support snapshots natively?
Most flash-based storage solutions do not include a native snapshot feature. This is an important feature because third-party snapshot utilities often come with additional cost and performance penalties.

Kaminario K2 offers native snapshots as part of its DataProtect software suite. These are high performance snaps designed to deliver the proper functionality to a flash based storage system without degrading performance.

Does your array support asynchronous replication?
Flash-based storage solutions generally do not offer native asynchronous replication. This capability is an important high-availability component to help customers withstand disasters such as power loss or physical damage.

While some systems include this type of replication, they are typically designed for HDDs — that negates the performance benefits that SSDs bring.

Kaminario K2 will include native asynchronous replication later this year.

Ask your storage vendor to answer these questions for you before you purchase an all SSD storage array. Then try these features for yourself. Many vendors claim these capabilities but few can deliver them. Kaminario just does it!

(photo credit)

In his recent StorageSearch blog, High Availability Enterprise SSD’s, Zsolt Kerekes discusses why building a do-it-yourself HA solution typical of data center Fibre Channel or IP SAN hard disk array installations is not a viable option for an enterprise SSD array. The reasons, according to Zsolt, boil down to performance, flexibility of use, risk, complexity, and scalability. I would add Murphy’s Law.

Zsolt points out that any home-grown HA solution sitting in front of an SSD storage controller is likely to introduce considerable latency and time to the recovery process. When it comes to the mission critical applications typically running on SSD arrays, such as online transaction processing, time really is money and simply not something you want to sacrifice. “Wrap around” HA, as Zsolt calls it, also introduces architectural complexity and controller configuration issues that can gum up the works.

Risk, or Murphy’s Law, comes into play when this beautiful HA solution you assembled suddenly becomes part of the problem during an actual outage instead of part of the solution–a more common scenario than you might think according to Zsolt. Built-in HA is designed by vendor architects who know their own products inside out and have spent a lot of time thinking about and testing the HA solution, so failure is a lot less likely.

We understand the criticality of built-in high-performance HA, which is why we designed the K2’s DataProtect with an internal clustered N+1 architecture and instant failover and recovery capabilities. If you want to see just how fast the K2 can recover from a failed data node, take a look at the Kaminario K2 High Availability Demonstration video. When someone pulls out an entire storage node, instantly the K2 operating system detects the outage, fails over to a spare node, and reconfigures the array, all without data loss and barely missing a beat in terms of performance. We’ve also added super-fast snapshot and replication capabilities, with no impact on overall performance, so you’re protected from data corruption and local outages.

You have to ask yourself, why take on the risk of a homegrown availability solution, when a fast, inexpensive one is already designed into the Kaminario K2.

Most of you know that SSD is a great way to eliminate I/O bottlenecks, but did you know that it can also eliminate servers in the data center–with all the requisite hardware, power, cooling, and real estate savings?

In SSDs and Server Consolidation, Jim Handy of Objective Analysis demonstrates how the use of SSD eliminates the need for server-hungry I/O acceleration techniques, such as running the Open Source caching program memcached on its own server or sharding, which splits large databases into smaller parallel data sets running on multiple servers.

Add SSD, either as local or shared storage–including the Kaminario K2 mentioned in Jim’s blog (Thanks Jim)–and you can get the same or better performance results without all that unsightly excess server hardware. And you get less complexity and better reliability to boot. I’ll be back.

Analyst and Media Reaction to the DataProtect Launch
It has been just over a month since Kaminario launched the DataProtect™ high availability and data protection capabilities for the Kaminario K2 product line. Analyst and media reaction has been very positive — highlighting the advantages and challenges that we have ahead.

After reviewing the DataProtect coverage, three messages stood out:

• High availability and data protection features such as DataProtect are needed for SSDs to be accepted as a HDD replacement in the data center;
• Kaminario is moving beyond the high-performance storage niche segment and aiming squarely at primary storage;
• DataProtect gives Kaminario some advantages but market competition is very aggressive.

Just to recap the coverage highlights, I’d like to share a selection of comments that capture much of the feedback we have observed.

Arun Taneja
Taneja Group
(as quoted in searchsolidstatestorage.com)

“In the first volley of all-flash or all-DRAM arrays it was crucial to focus on performance,” he said. “But as soon as that baseline was set, customers started asking why there wasn’t HA or protection and why could they not manage several arrays from a common management tool, just like they do a regular array. As an industry, we are already in the second phase of maturity. This rounding out of what I call ‘soft features’ will give Kaminario a solid standing.”

Dan Kusnetzky
ZDNet
“For the most part, solid state storage has been added on as a caching mechanism, but not as part of the standard set of storage architectures traditional storage servers are designed to support. So, most of today’s storage systems do not offer the range of performance, ease of management or availability offered by newer generation storage systems, such as those offered by Kaminario.  I expect that over time, all storage system suppliers will offer the same capabilities that Kaminario is offering today.”

George Crump
Storage Switzerland
“This is a significant upgrade for Kaminario. The high availability, the uniqueness of RAID 10HD and the addition of data protection services like snapshots and replication clearly move the Kaminario solution out of the niche performance market and into the primary storage market. The solution is now just as viable for accelerating a virtualized server infrastructure as it is for accelerating a high end Oracle or Oracle RAC databases.”

Stephen Foskett
Pack Rat
“Kaminario is wise to move from the small but lucrative “maximum performance” market to the broader but more competitive “disk replacement” segment. DataProtect adds important features required by mainstream buyers, but Kaminario still lacks capacity optimization features (deduplication, compression, and thin provisioning) that their competitors boast.”

(Editor’s note: Kaminario argues that today dedupe, compression and similar features negatively impact latency and therefore have not, up until now, focused on adding them.)

David Hill
Network Computing
“Although with K2 Kaminario is not necessarily alone at the top of the high-performance SSD storage mountain, it has a clear view of where it is going and what benefits high-performance-demanding customers need and are willing to pay for. And that is a good position to be in.”

So What Does this Mean?
My conclusions are that:

• Customers require performance and advanced data protection features, something that, until our DataProtect launch, they could not get anywhere else in SSD-based arrays;
• Performance alone is insufficient to drive SSD adoption in the data center. Our DataProtect launch responds to those customers with a solid commitment to data protection;
• Our Scale-out Performance Storage Architecture (SPEAR) truly enables our continued innovation around data protection, as well as with scalability and other areas.

What Data Protection Features do Customers want Most?
The launch taught me that data protection conversations need to continue. Our customers are demanding it and we expect that they are not alone. One way we are facilitating this is by conducting customer research about their views on the data protection features they want/need most, see previous post Which Data Protection Features Would You Like to See. This project is just beginning, but we’ll share the results online to foster healthy debate.

(photo credit)

Kaminario is embarking on a research project to learn more about what data protection features customers want in their SSD SAN storage arrays. We will be sharing the results with you to continue conversations about SSD adoption in the enterprise data center.

We want your opinion too! Tell us in the blog comments. Also, feel free to publish your thoughts on your blog. Send us the link.