Well I have to tell you that this is a very important post for both sides 
In my opinion :
- Oracle is very good in database technologies (better the EMC)
- EMC is very good in storage technologies (better then Oracle)
But let’s see the all the picture: Oracle Exadata is a solution for Oracle software suites and according to my understanding with better usage results for Oracle Databases performance rather then using “traditional” storages like CX4, DMX3 or from IBM DS5300 or DS8300.
So where is the issue ? I can’t use the Exadata for my File Server, I can’t use the Exadata to boot RedHat Linux from this etc. Currently Exadata is a response to the Oracle market (big market by the way) related to better performance and low price without too much noise related to tuning the Storage, OS, Database in order to get performance.
Maybe EMC can get better results with a specific setup, but in order to get those results they are spending 5 – 15 days or more for tune all the system (storage, fc switches, os, database etc). And maybe a similar result I can have-it with 2-3 days installation of Exadata – more economically.
The performance & reliability of the application is what everyone is looking at the end. And more and more for lower prices.

So what is the storage?
- Hardware and
- Software
In my perspective the software is the one giving more value to a storage right now – 2009 (staring from controllers os, replication, snaps etc).
Coming back to Exadata I didn’t see any documents related with Disaster Recovery, Performance Tools, Update mechanism, security etc – where I think Oracle Exadata haze a very big gap against competition.

Let’s see what will be next XIV ? – what IBM will do with this, what EMC will develop related with the better understanding of the data store in their disk systems etc.

Best regards,
Florin

I initially wrote:

In the white paper Deploying EMC CLARiiON CX4-960 for Data Warehouse/Decision Support System (DSS) Workloads EMC reports a drive scan rate [for a BI/DW workload] of 20 MB/s using 8+1 RAID-5 and 33 MB/s using a 2+1 RAID-5 LUN configuration. Oracle Exadata delivers drive scan rates around 85 MB/s, a difference of 2.5X to 4.25X.

I was also (correctly) asserting that the CX4-960 head was only capable of 3GB/s. Note this passage in that same paper:

Ninety drives of the 300 GB organized as “thin RAID” will deliver in theory 3000 MB/s (90 x 33.3 MB/s)

Now, it seems way too coincidental that the array head has a max throughput of 3GB/s and that is exactly the number worked out. The 33MB/s number would appear to be an observed number (by someone), but I also do recognize that FCAL drives are capable of more, but for whatever reason (a discussion for another day) the workload didn’t exhibit higher rates.

On to the hardware comparison…
Each HP Oracle Exadata Storage Server can scan the 12 SAS drives at 1GB/s (technically it’s just a bit more, but 1GB is a nice round marketing number), so 25% more than the AX4-5 can do. Exadata scans the SATA drives at 750 MB/s. The IB can easily deliver that data to the database grid at those rates as well, however, in nearly all cases the data sent to the database grid is much less. Exadata only sends rows and columns required for the query. So unless you query every column and every row, the data sent will be much, much less than what is read from disk. This dramatically reduces the amounts of data sent from the storage (to the DB grid) and number of rows that the DB grid has to deal with, so compared to FC attached storage, less DB CPU is required to do the same work. This is where “smart scans” and “query processing closer to the data” comes into play and is what differentiates Exadata from other storage.

Go read it carefully, please. I’ll be waiting for your comments …

So, here are some ACTUAL NUMBERS you may want to go chew on:

A given disk drive sitting in a CX (or any other well-designed storage array) will deliver between 85MB/sec (SATA) and up to 150MB/sec (15k FC). Flash, considerably more, but that’s a separate discussion.

Add drives into a controller until you hit the bandwidth limit of the controller.

Here are some rules of thumb to help you:

AX4-5 is ~800MB/sec
CX4-240 is ~1.1GB/sec
CX4-960 is ~2.8-3GB/sec

Now, let me please point out — this is without the overhead of disk mirroring (mandatory in your approach), or being forced to use SATA drives, or only being able to get 6 usable drives per server. Maybe you can deliver great performance, but at what cost?

So, mentally I’m seeing the closest substitute here in the EMC portfolio being an AX4-5 with about 12 SATA drives (let’s add two for parity, shall we?) — so 14 drives delivering approx. ~800MB/sec.

Your approach would require either two bricks with 12 each to equate usable capacity (wasting 12 takeway 2 or 10 drives in the process), and the cost of two beefy HP servers as compared to a single low-cost AX4-5 controller. We’re talking several thousands of dollars difference at the “brick” level — and, by the time we get really big, we’re talking hundreds of thousands of dollars difference. Not to mention power, cooling, etc.

I’d suggest take a look at what one of these looks like — it sits in a rack just like your server-based storage brick.

Can your two mirrored bricks deliver ~800MB/sec sustained sequential throughput? I’m just assuming that’s the case, because — if not — your approach gets even more expensive. Hint: your bottleneck will either be the HP I/O card in the server, or the IB connection.

Dude, it’s not even a fair fight.

For the thin LUN (two-way striped), it takes 3 ms to move the head, and 12 ms to move the 512 KB worth of bits. The net MB/s from that drive is therefore 33 MB/s (1000 x 512 KB /(3+12)).

It is not my number. If I have taken it out of context, do provide technical clarity.

If my math is incorrect, do provide the correct equations and calculations, assumptions, etc. None of the array spec sheets have a max observable I/O bandwidth number. They used to contain the max throughput from cache numbers, which was a great marketing number, but a horrible engineering number.

Based on my information DATAllegro v3 was able to max out a CX3 head at 1200MB/s with just 12 SATA drives (plus 1 spare). Greenplum’s numbers might be interesting, but Vertica and ParAccel are probably less interesting as they are more memory based column databases where drive scan throughput is probably much less.

The passage you obviously picked up on was a theoretical discussion OF A SINGLE DRIVE, not an array.

To make matters even more interesting, you then do all this nice math on a bogus misinterpretation. Utter rubbish.

Stick to software, guys …

I said “play fair” and don’t include flash drives. I’ll be back to you soon with the data.

Oh, BTW, just be aware that the scan rates are very different for Oracle as compared with, say, something that’s optimized for the role, such as Vertica, Greenplum, ParAccel, DatAllegro et. al. Oracle keeps getting server-limited, the others can go much father/faster.

Be back soon with the goodies!

– Chuck

I find it a bit entertaining that you have mentioned that Exadata is a “horribly inefficient use of hardware resources” and that floorspace can not be discounted. On what technical information is that based on? Given that Exadata scans HDDs 2.5x faster (85MB/s vs. 33MB/s) than an EMC CX4 for a db workload, it would seem that a CX4 is “horribly inefficient”. But let’s just say for arguments sake that a CX4 was driven to scan HDDs at 85MB/s – the same rate that Oracle Exadata scans HDDs. If we compare storage that can be scanned at 3GB/s (based on the aforementioned EMC paper the CX4-960 array head is capable of this) one could use 3 HP Oracle Exadata Storage Servers containing a total of 36 HDDs and 6 Quad-Core Intel Xeon processors taking up 6 rack units or a CX4-960 array head with 4 Quad-Core Intel processors (2 in each SP/server processor) and 3 DAEs with a total of 36 HDDs taking up 15 rack units (2U SPS, 4U SPE and 3U for each DAE). So even when we assume the same number of HDDs and the same scan rate, the EMC CX4 solution takes up much more space (9U more). Looks like Exadata is more efficient and takes up less space by the numbers. BTW Chuck, if you (or EMC engineers) have performance numbers and data that the EMC CX4-960 array head can push more than 3GB/s, do let me know. It’s a good move up from the 1.2GB/s that the CX3 array heads max out at which is not even enough to max out the 4×4Gb FCP its plumbed to the hosts with.

There probably will be some IT infrastructure guys (SAN admins) will balk at Exadata, just in the same way that they balked at ASM, but I think that the business people will be very pleased when their queries run 10x to 100x faster (or more) on Exadata and the SAN admins will eventually come around. Business has this way of driving technology. Just by pure data bandwidth capacity a single HP Oracle Database Machine probably has more than 4x as much (14GB/s) as I/O bandwidth the most SAN installations out there (to the DW database grid or host). I havent seen too many SAN configurations where they push more than 2GB/s let alone 3.5GB/s. If there are such customers, feel free to showcase them. May I also suggest you read the beta customers reports about their 1/2 HP Oracle Database Machine experiences. Two of them use EMC storage today. With only 6 HP Oracle Exadata Storage Servers they are seeing up to 72x performance gains. Hard to argue with those numbers.

If you come across any customers that are doubtful, be sure tell them to see their Oracle account rep about doing a POC on Exadata, but warn them – it may be a bit like driving the Ariel Atom (2m:50s) compared to their current setup which is probably connected to a SAN.

You may have the hearts and minds of a subset of the Oracle DBA world, but this approach is not winning any fans with the crew that’s responsible for the infrastructure.

With the customers I’ve spoken to, it’s boiled down to two key objections:

1 — Horribly inefficient use of hardware resources, which translates into multiple costs: acquisition costs, operational costs, etc. You can discount the upfront costs, but I doubt that you can cover maintenance, floorspace, power, cooling, etc.

2 — Can’t be operationalized in a consistent fashion. IT infrastructure guys like to have one way of doing things: management, backup, DR, provisioning, etc. With this approach, you’re foisting yet another stovepipe on them, and they don’t like it. They’d much prefer to have standard infrastructure for DW/BI that they can manage like the rest of their landscape.

If you could somehow make enough of a case that the benefits outweigh the pains, maybe you’d stand a chance.

But you haven’t — most of what’s out there is hand-waving. And I see customer after customer tell me they’ve taken a close look at this approach, and said “no thanks”.

I’m sure a few unlucky souls will start down this path, and we’ll be reading about these few in breathless press releases. But, right now, it looks like this one has stalled on the launch pad, so to speak.

Best regards!

– Chuck