Comments on: Preprocessor For External Tables

By: Neat Oracle Database 11g Release 2 Feature: Preprocessing External Tables » Eddie Awad's Blog

Thu, 24 Mar 2011 18:36:57 +0000

[...] Greg Rahn: [...]

By: The Core Performance Fundamentals Of Oracle Data Warehousing – Data Loading

Fri, 23 Apr 2010 16:01:51 +0000

[...] to provide the desired loading rate) and consider using compressed files in conjunction with the external table preprocessor. Using the preprocessor is especially useful if there are proportionally more CPU [...]

By: New External Table Functionality - PreProcessing and Data Pump (The Data Warehouse Insider)

Wed, 25 Feb 2009 17:00:10 +0000

[...] For some more info on the actual syntax, I noticed Greg wrote a full example in his posting here. [...]

By: Bob

Bob — Sun, 23 Nov 2008 18:43:14 +0000

Regarding security, the user running the load needs to be granted EXECUTE permission on “exec_dir” (in this example). One recommendation is that rather than granting execute on exec_dir which in this example points to /bin, is simply to create another directory that holds just the execuatbles (or scripts) that the preprocessor will use.
EXECUTE is a new permission that can be grant to a directory in 11.1.0.7.

By: Greg Rahn

Greg Rahn — Sun, 23 Nov 2008 05:16:04 +0000

@Jens

I’m not sure what is so great about listing files in a directory via an external table that it warrants a paper, but whatever…

Restricting “create directory” or limiting who has execute on the directory (database directory, not filesystem) are options. Most utilities would be in /bin or /usr/local/bin or similar so only root could put files there to be used in the preprocessor.

By: Jens Petersen

Jens Petersen — Sat, 22 Nov 2008 18:41:49 +0000

From a developers perspective this is a great new feature and I’m just about to write a paper how to use this to list all files in a directory.
But from a security perspective this sounds worryingly like opening up new huge holes.
Is there any way to restrict the usage of the preprocessor clause to certain users or disable it at all?

By: Log Buffer #124: a Carnival of the Vanities for DBAs

Log Buffer #124: a Carnival of the Vanities for DBAs — Fri, 21 Nov 2008 17:31:15 +0000

[...] Rahn of Structured Data explained the use of a preprocessor for external tables. He writes, “Before External Tables existed in the Oracle database, loading from flat files [...]

By: Nigel Thomas

Nigel Thomas — Fri, 21 Nov 2008 10:50:42 +0000

@Carey – excellent idea – of course we can make a proxy file for any data source (which could contain the real parameters for the pre-processor – like source URL for example)

@Stewart – yes, the main benefit of streaming is (a) for integration with a non-Oracle source and (b) so that data-in-flight (maybe from SQL Server to DB/2) doesn’t *have* to be physically staged in the (Oracle) transformation hub before it is delivered to its final destination. In the old days we did that with a PL/SQL pipeline using UTL_FILE in and out, then piping into SQL*Loader; nowadays we can have the source treated as an external table and do the transformation as part of the SELECT from that.

By: Greg Rahn

Greg Rahn — Fri, 21 Nov 2008 02:18:10 +0000

@Stewart

The preprocessor can be uncompress/gunzip or any other command that can output text to stdout.

By: Stewart Bryson

Stewart Bryson — Fri, 21 Nov 2008 02:14:21 +0000

Greg:

Are you constrained by uncompress-type commands, or could the preprocessor call other sorts of command-line entities, such as decrypt commands, etc?

Nigel:

If you are loading an Oracle database with data from a remote Oracle database… there’s no need to trouble yourself with all of this. The INSERT /*+ APPEND */ into SELECT * FROM… across a DBLINK will perform similarly. This initiates the direct path API, very similar to SQL-Loader’s DIRECT mode. However, using the direct-path API inside of SQL gives some benefits that SQL-Loader never had: the MERGE statement, multi-table insert, etc.

Your point about using this scenario to load from other databases is not lost on me, however: that would be very valuable indeed. That’s what Informatica does to speed up loading into Oracle databases. Since it cannot support the direct-path API from inside SQL, it actually streams data through SQL-Loader in a similar way.

Regards,

Stewart