Database Customer Benchmarketing Reports

A few weeks ago I read Curt Monash’s report on interpreting the results of data warehouse proofs-of-concept (POCs) and I have to say, I’m quite surprised that this topic hasn’t been covered more by analysts in the data warehousing space. I understand that analysts are not database performance engineers, but where do they think that the performance claims of 10x to 100x or more come from? Do they actually investigate these claims or just report on them? I can not say that I have ever seen any database analyst offer any technical insight into these boasts of performance. If some exist be sure to leave a comment and point me to them.

Oracle Exadata Performance Architect Kevin Closson has blogged about a 485x performance increase of Oracle Exadata vs. Oracle Exadata and his follow-up post to explain exactly where the 485x performance gain comes from gave me the nudge to finish this post that had been sitting in my drafts folder since I first read Curt’s post.

Customer Bechmarketing Claims

I thought I would compile a list of what the marketing folks at other database vendors are saying about the performance of their products. Each of these statements have been taken from the given vendor’s website.

  • Netezza: 10-100 times faster than traditional solutions…but it is not uncommon to see performance differences as large as 200x to even 400x or more when compared to existing Oracle systems
  • Greenplum: often 10 to 100 times faster than traditional solutions
  • DATAllegro: 10-100x performance over traditional platforms
  • Vertica: Performs 30x-200x faster than other solutions
  • ParAccel: 20X – 200X performance gains
  • EXASolution: can perform up to 100 times faster than with traditional databases
  • Kognitio WX2: Tests have shown to out-perform other database / data warehouse solutions by 10-60 times

Certainly seems these vendors are a positioning themselves against traditional database solutions, whatever that means. And differences as large as 400x against Oracle? What is it exactly they are comparing?

Investigative Research On Netezza’s Performance Claims

Using my favorite Internet search engine I came across this presentation by Netezza dated October 2007. On slide 21 Netezza is comparing an NPS 8150 (112 SPU, up to 4.5 TB of user data) server to IBM DB2 UDB on a p680 with 12 CPUs (the existing solution). Not being extremely familiar with the IBM hardware mentioned, I thought I’d research to see exactly what an IBM p680 server consists of. The first link in my search results took me to here where the web page states:

The IBM eServer pSeries 680 has been withdrawn from the market, effective March 28, 2003.

Searching a bit more I came across this page which states that the 12 CPUs in the pSeries 680 are RS64 IV microprocessors. According to Wikipedia the “RS64-IV or Sstar was introduced in 2000 at 600 MHz, later increased to 750 MHz”. Given that at best, the p680 had 12 CPUs running at 750 MHz and the NPS 8150 had 112 440GX PowerPC processors I would give the compute advantage to Netezza by a significant margin. I guess it is cool to brag how your most current hardware beat up on some old used and abused server who has already been served its end-of-life notice. I found it especially intriguing that Netezza is boasting about beating out an IBM p680 server that has been end-of-lifed more than four years prior to the presentation’s date. Perhaps they don’t have any more recent bragging to do?

Going back one slide to #20 you will notice a comparison of Netezza and Oracle. Netezza clearly states they used a NPS 8250 (224 SPUs, up to 9 TB of user data) against Oracle 10g RAC running on Sun/EMC. Well ok…Sun/EMC what??? Obviously there were at least 2 Sun servers, since Oracle 10g RAC is involved, but they don’t mention the server models at all, nor the storage, nor the storage connectivity to the hosts. Was this two or more Sun Netra X1s or what??? Netezza boasts a 449x improvement in a “direct comparison on one day’s worth of data”. What exactly is being compared is up to the imagination. I guess this could be one query or many queries, but the marketeers intentionally fail to mention. They don’t even mention the data set size being compared. Given that Netezza can read data off the 224 drives at 60-70 MB/s, the NPS 8250 has a total scan rate of over 13 GB/s. I can tell you first hand that there are very few Sun/EMC solutions that are configured to support 13 GB/s of I/O bandwidth. Most configurations of that vintage probably don’t support 1/10th of that I/O bandwidth (1.3 GB/s).

Here are a few more comparisons that I have seen in Netezza presentations:

  • NPS 8100 (112 SPUs/4.5 TB max) vs. SAS on Sun E5500/6 CPUs/6GB RAM
  • NPS 8100 (112 SPUs/4.5 TB max) vs. Oracle 8i on Sun E6500/12 CPUs/8 GB RAM
  • NPS 8400 (448 SPUs/18 TB max) vs. Oracle on Sun (exact hardware not mentioned)
  • NPS 8100 (112 SPUs/4.5 TB max) vs. IBM SP2 (database not mentioned)
  • NPS 8150z (112 SPUs/5.5 TB max) vs. Oracle 9i on Sun/8 CPUs
  • NPS 8250z (224 SPUs/11 TB max) vs. Oracle 9i on Sun/8 CPUs

As you can see, Netezza has a way of finding the oldest hardware around and then comparing it to its latest, greatest NPS. Just like Netezza slogan, [The Power to ]Question Everything™, I suggest you question these benchmarketing reports. Database software is only as capable as the hardware it runs on and when Netezza targets the worst performing and oldest systems out there, they are bound to get some good marketing numbers. If they compete against the latest, greatest database software running on the latest, greatest hardware, sized competitively for the NPS being used, the results are drastically different. I can vouch for that one first hand having done several POCs against Netezza.

One Benchmarketing Claim To Rule Them All

Now, one of my favorite benchmarketing reports is one from Vertica. Michael Stonebraker’s blog post on customer benchmarks contains the following table:

vertica_benchmark_table.png

Take a good look at the Query 2 results. Vertica takes a query running in the current row store from running in 4.5 hours (16,200 seconds) to 1 second for a performance gain of 16,200x. Great googly moogly batman, that is reaching ludicrous speed. Heck, who needs 100x or 400x when you do 16,200x. That surely warrants an explanation of the techniques involved there. It’s much, much more than simply column store vs. row store. It does raise the question (at least to me): why Vertica doesn’t run every query in 1 second. I mean, come on, why doesn’t that 19 minute row store query score better than a 30x gain? Obviously there is a bit of the magic pixie dust going on here with, what I would refer to as “creative solutions” (in reality it is likely just a very well designed projection/materaizied view, but by showing the query and telling us how it was possible would make it less unimpressive [sic]).

What Is Really Going On Here

First of all, you will notice that not one of these benchmarketing claims is against a vendor run system. Each and every one of these claims are against existing customer systems. The main reason for this is that most vendors prohibit benchmark results being published with out prior consent from the vendor in the licensing agreement. Seems the creative types have found that taking the numbers from the existing, production system is not prohibited in the license agreement so they compare that to their latest, greatest hardware/software and execute or supervise the execution of a benchmark on their solution. Obviously this is a one sided apples to bicycles comparison, but quite favorable for bragging rights for the new guy.

I’ve been doing customer benchmarks and proof of concepts (POCs) for almost 5 years at Oracle. I can guarantee you that Netezza has never even come close to getting 10x-100x the performance over Oracle running on a competitive hardware platform. Now I can say that it is not uncommon for Oracle running on a balanced system to perform 10x to 1000x (ok, in extreme cases) over an existing poorly performing Oracle system. All it takes is to have a very unbalanced system with no I/O bandwidth, not be using parallel query, not use compression, poor or no use of partitioning and you have created a springboard for any vendor to look good.

One More Juicy Marketing Tidbit

While searching the Internet for creative marketing reports I have to admit that the crew at ParAccel probably takes the cake (and not in an impressive way). On one of their web pages they have these bullet points (plus a few more uninteresting ones):

  • All operations are done in parallel (A non-parallel DBMS must scan all of the data sequentially)
  • Adaptive compression makes disks faster…

Ok, so I can kinda, sorta see the point that a non-parallel DBMS must do something sequentially…not sure how else it would do it, but then again, I don’t know any enterprise database that is not capable of parallel operations. However, I’m going to need a bit of help on the second point there…how exactly does compression make disks faster? Disks are disks. Whether or not compression is involved has nothing to do with how fast a disk is. Perhaps they mean that compression can increase the logical read rate from a disk given that compression allows more data to be stored in the same “space” on the disk, but that clearly is not what they have written. Reminds me of DATAllegro’s faster-than-wirespeed claims on scan performance. Perhaps these marketing guys should have their numbers and wording validated by some engineers.

Do You Believe In Magic Or Word Games?

Creditable performance claims need to be accounted for and explained. Neil Raden from Hired Brains Research offers guidance for evaluating benchmarks and interpreting market messaging in his paper, Questions to Ask a Data Warehouse Appliance Vendor. I think Neil shares the same opinion of these silly benchmarketing claims. Give his paper a read.

15 comments

  1. Donald K. Burleson

    Hi Greg,

    >> Do You Believe In Magic Or Word Games?

    Nobody does, and everyone knows that you can lie with statistics, and everyone with a grain of sense knows about vendor “puffery”!

    >> silly benchmarketing claims.

    Yup. You describe the problem very well, now what’s the solution?

    – Do you think that the TPC benchmarks can be made “word game” proof?

    – What is Oracle doing to standardize their performance claims?

  2. Greg Rahn

    @Don
    Do you think that the TPC benchmarks can be made “word game” proof?
    I don’t think that TPC benchmarks suffer from the same issues as customer benchmarketing claims. First, TPC benchmarks are audited. Second, there is always a full disclosure report (FDR) provided that gives all the details of what was run and under what environment it was run. With these customer benchmarketing claims, there is no audit, and no explanation of what was done. There is also no FDR of either of the environments being compared. Obviously there will never be, but let’s get over the shock and awe(ful) claims.

    I’m hoping that some day the TPC-DS spec will be agreed upon. Last I knew it was hung up in a political battle because some vendors currently don’t support some of the analytical SQL syntax used, among other things.

    What is Oracle doing to standardize their performance claims?
    I can’t speak for Oracle, the company. Those decisions are made above my pay grade.

  3. Dave Menninger

    Greg,

    Let me begin with full disclosure that I work for Vertica. So I’d like to offer some additional information on Vertica’s benchmark claims. We publish a number of customer benchmark comparisons. Generally, we do not have any involvement with the existing query times. So we do not have any way in which to distort those times. The customer and/or the incumbent vendor have presumably tuned the database to the best of their ability. Then we do the same or in some cases the customer does the same and we publish the results. For example, you can listen to George Chalissery, CEO of hMetrix describe his company’s performance comparisons of Vertica v. a popular open source database here:

    http://www.vertica.com/product/resourcelibrary/hmetrix

    On slide 7 at 15:00 into the presentation George describes their apples to apples comparison as “Vertica installed by us versus a relational database installed by us”. You can see his claim of performance improvement on query 2 is even more “ludicrous” than the claim above — 40,400x improvement — but it’s his comparison of his data on his hardware. If you listen, even George couldn’t believe the difference and asked the engineers to check and see if they had the units right.

    One other piece of information in an effort of full (or at least more) disclosure is the following blog post that breaks down the orders of magnitude differences between row stores and column stores to their constituent parts.

    http://www.databasecolumn.com/2008/12/debunking-yet-another-myth-col.html

    Dave Menninger
    Vertica

  4. Pingback: The Best Benchmarketing I’ve Seen Yet: Measure BI Queries In Milliseconds | Structured Data
  5. Phil Francisco

    Greg,
    As was the case with Dave Menninger, I’ll preface my comments with the full disclosure that I work for Netezza Corporation.

    Netezza is not in the business of misleading prospective customers, but in demonstrating the performance, value and simplicity that our data warehouse appliances have to offer our customers. The performance numbers quoted by us are the findings of prospective Netezza users and Netezza is NOT involved in the running of the tests on other vendors’ systems.

    With regard to the competitive systems under test in these customer benchmark (or POC) tests, the tests nearly always include results achieved by the customer running the same test set on its deployed system. Sometimes that system is at or approaching its end-of-life, but it is also frequently a relatively new deployment (less than 2 years) that has reached some critical constraint such as scalability, operational complexity and/or inability to serve particular applications needs without a major refresh. In any event, that incumbent system is typically as finely-tuned as it must be for production purposes, completely under the control of the customer. It’s true that the incumbent system’s performance is the most common set of comparative results for Netezza to be privy to.

    However it is also true that such POC tests usually involve at least one competitor system, which I can only presume is the best possible system available from the competition to respond to the customer’s system & purchase constraints.

    As for the customer benchmark/POC testing process itself, performance numbers quoted by us are the findings of prospective Netezza users. Because our systems are so easily deployed, we encourage prospective buyers to test our systems at their site, using their queries, running against their data based on their own workload.

    Unlike vendors who attempt to hide their systems’ installation, tuning and operational complexity by running tests in their own controlled test/demo environment, Netezza provides the customer visibility into the POC process, from delivery on the customer’s loading dock to installation, data loading and through completion of the final customer benchmark tests. This type of benchmark testing allows the customer to more accurately predict how the various competing systems will behave in their own production environment. Netezza is also not the only vendor in the industry to believe the currently agreed-upon TPC suite of benchmarks for data warehouse systems is not at all a good predictor for actual performance in a customer’s deployed environment.

    Unfortunately, while the Netezza PowerPoint deck you referenced was from October ’07, the customer benchmark data included in it was from much earlier. The NPS models from those examples represent samples from Netezza’s 1st and 2nd generation systems; neither of which, themselves have been sold since March 2006 at the latest and some of the NPS systems in those results were from as far in the past as 2003 or early 2004. For instance, Netezza stopped selling the NPS 8100 and 8400 model systems as early as the 1st half of 2004.

    While one could understandably quibble with the data shown in those old charts, one can hardly argue with the results achieved. Netezza continues to enjoy a very high success rate in terms of prospect-to-customer conversions using the on-site customer benchmark process and certainly we compete against all of the major data warehouse vendors in the marketplace for this business, Oracle included. I’ll ensure that the terminology we use on the Netezza website continues to be clear and accurate, but Netezza feels comfortable in its claims to 10-100X relative performance. Thanks.

    Regards,
    –Phil Francisco
    VP, Product Management & Marketing
    Netezza Corporation

  6. Pingback: The Netezza Community: Thoughts from Inside the Box: Setting the Record Straight
  7. Greg Rahn

    @Phil

    Thanks for the lengthy comment and pitch.

    The reason I (and many others) feel these benchmarketing claims are silly is because they are not a comparison of equals. I don’t see Car and Driver compare the 2009 Subaru Impreza WRX to the 2000 Honda Civic Si. They compare the same model year. Most any database company can claim they are 10-100x faster than something. The strength of the claim will obviously depend on what that something is. Thus far the Netezza list is very unimpressive and quite dated as well (as you mentioned).

    So why is it that there are no NPS 10000 series performance claims? Even Netezza presentations I’ve seen given in 2008, still reference pre NPS 10000 series hardware which arrived May 2005 and was then extended in December 2006, so the full lineup has been in place for two years.

    In most every customer benchmark I’ve done there has been a 10-100x or more performance increase over the existing system. Sometime query performance tops 1000x (yes, really!). In fact, some numbers from a recent customer POC on the HP Oracle Database Machine were well above 100x (like several times more). I guess the Oracle marketing team can use the 10-100x or more claim as well, as could most anyone (and seems they are!). I believe Oracle’s 10x pitch is a bit more believable than the overly optimistic 100x pitch. Who knows, after more numbers are available, that number could go up. After all, the beta customers saw up to 72x on less than half of a HP Oracle Database Machine.

    By the way, thanks for the free advertising of the HP Oracle Database Machine on the Netezza website. A good effort in spreading FUD, but cool. I guess that’s the marketing games companies play.

  8. Kevin Closson

    Like I said in my blog post:

    There is no reason to be mystified. If DW Appliance vendor XYZ is spouting off about a query processing speed-up of, say, X, just plug the values into the following magic decoder ring. Quote me on this, performance increase X is the product of:

    1. Executing on a platform with X-fold storage bandwidth
    2. Executing on a platform with X-fold processor bandwidth
    3. The query being measured manipulated 1/Xth the amount of data
    4. Some combination of items 1 through 3

    Any reasonable vendor will gladly itemize for you where they get their magical performance gains. Just ask them, you might learn more about them than you thought.

    BTW, has anyone bothered to read the Oracle EULA wording regarding citing of benchmark results? I have.

    The views expressed in this comment are my own and do not necessarily reflect the views of Oracle. The views and opinions expressed by others on this comment thread are theirs, not mine.

  9. Mark Boyd

    Greg –

    This is a very interesting and thought provoking article. I wanted to provide some perspective as a former Oracle DBA and a current Netezza customer. The company I work for recently migrated a 12TB Oracle database running a Sun 4900 with 8 CPUs to a Netezza NPS 10100 system. We have seen an average increase in query performance of 54x in our environment. That being said, the comparison of an 8 CPU system to a 108 CPU system isn’t a fair comparison in my mind. The real driving factor for us to move away from Oracle (we are otherwise a completely Oracle “shop”) was the combination of performance improvement and cost. In order for us to have upgraded our Oracle database server to something that would have provided the same level of performance that the NPS provides, we would have had to spend on the order of 5x the amount of money. To our thinking, that just didn’t make sense. So, while I definitely agree that I would like to see benchmarks between Netezza and the “latest and greatest” from other vendors, I would suggest that cost (and I would be the first to say “Total Cost”, including migration costs, support costs, etc.) should also be a factor. If it costs 5x the money to build an Oracle system that provides the same level of performance that an NPS provides, that would be worth understanding. I would like to emphasize, as well, that our purchase was made well before the announcement of the Oracle Exadata product, though my understanding is that box would have still cost at least 2x what we spent on Netezza and I have no idea how performance would compare. Lastly, to be clear, I am not affiliated with Netezza in any way, other than being a reasonably satisfied customer. Thanks for the article and causing me to think a lot about these issues.

    Mark Boyd

  10. Pingback: Doug's Oracle Blog
  11. Greg Rahn

    @Mark

    Thanks for the comments. Always good to get experiences first hand.

    An average 54x improvement on a NPS 10100 (108 SPUs) over a Sun Fire E4900 Server with 8 UltraSPARC IV processors. As you mentioned, I would not call that much of a comparison. It does really make you think what they are comparing to when they claim 100x, 200x or 400x. Data warehousing has really become an arms race of compute power and I/O bandwidth.

    I’d be interested to know if the PowerPC processors that Netezza uses (they currently use the 440GX) have gotten much faster since 2003. The reason I wonder, is that in April 2003, the AMD Opteron™ processor was introduced which contained the x86-64 instruction set (now known as AMD64). This processor allowed 64-bit Linux to run on a commodity server. Since then, we’ve seen Intel get into the 64-bit space with EMT64 (now known as Intel 64) as well as CPUs go from single core, to muti-core. These platforms have made Linux a significant player in the enterprise. Quad core processors are now available and eight core will be on its tails. This has drastically increased the compute power that one can get in a server, and it will keep increasing. Intel is starting to roll out its latest-generation microarchitecture (Nehalem) and the Nehalem-EP processors will arrive in 2009 (Intel Roadmap Overview August 20th 2008). The initial benchmark findings are starting to leak out. In fact, TechRadar says :

    …we can exclusively reveal that the upcoming dual-socket server variant of the Nehalem architecture is so fast it’s almost silly.

    I hope Netezza is ready for the race. It sure looks like it will be a good one.

  12. Kevin Closson

    Mark said, “average” is 54x and I’m not surprised one bit. I should think there are at at least a few queries at well over 100x considering the comparison. The data ingestion bandwidth improvement alone should be a minimum of 3.5X and the CPU count is a 13.5X. Throw in any amount of “brainy” DW features and combine it with that much “brawny” bandwidth increase and 54X actually seems light.

    If I was a betting man I’d bet that the I/O ingestion bandwidth was likely more on the order of 9.5X (2 4GFC HBAs vs 108 SPUS)… but I don’t know. Nonetheless, there is no way 8 USIV procs can ingest more than 2GB/s. When I say ingest I mean running an Oracle query with filtration/projection and even a light join as opposed to some I/O subsystem stimulation trick like dd(1).

    As an aside, just because a SPU has an old 5w PPC doesn’t put it at a deficit. The total package in the SPU is well suited to data-rich processing given the limited rate at which data is scanned (simple SATA drive). They might have a balance issue with these components scanning, say, a 15K RPM 6Gb SAS drive. Time will tell.

    The views expressed in this comment are my own and do not necessarily reflect the views of Oracle. The views and opinions expressed by others on this comment thread are theirs, not mine.

  13. Pankaj Shaju

    This is a great thread here. I agree, all of the claims of performance gains are just pure jumbled nonsense. I’m a DB Architect and have worked on Netezza, Oracle and Teradata installations (amongst others). Listening to all of these claims is like watching the presidential race all over again. With all the nonsense and double speak, comments and numbers taken out of context.

    The only true common denominator is the all mighty DOLLAR. Take a million dollar oracle system against a million dollar ‘X’ vendor. I’m talking hardware, software, environmentals… total cost of ownership…etc. Then you will truly see price/performance. I don’t care if one vendor has 100 more processors… if the cost was the same!

    I think what you might find is a software-based company like a Greenplum, will come out in the wash on top. Someone that leverages commodity hardware, where CPU speed and innovation is out-pacing the speed of which a hardware vendor like Netezza and Teradata can compete with.

    Software vendors will ALWAYS have an edge. Netezza’s hardware is 2-4 years old at best. Don’t get me wrong, it’s a good product and has exposed monopolistic DB pricing, but it’s not the way. Proprietary hardware will never win out in my opinion.

    At the end of the day, this exciting new market has made the industry that much stronger and has given many options to customers.

    P.S.

  14. Joseph

    Kevin >> Proprietary hardware will never win out in my opinion.

    Nope, commodity hardware will win out. If the proprietary hardware is better, it will be incorporated into the commodity machines (over time) and be accessible from software that it replaces. Anyway, the line between hardware and software blurs with configurable firmware (e.g. FPGAs and hybrid ASICs). Chips are getting cheaper to create or to merge IP cores. Definitely not true in graphics (despite some vendor claims to the contrary) or physics processors.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s