Comments on: There Is No Time Like '%NOW%' To Use Dynamic Sampling

By: Rainer Stenzel

Rainer Stenzel — Fri, 13 Jun 2008 13:55:51 +0000

Hello Greg, unfortunately I did not see your hint to 11.1.0.6 first time and can not test on 11g by myself currently.I did some tests on 10g again and found: 1) if using literals (or bind peeking ?) the optimizer 1.1) is applying s(column NOT LIKE pattern)=1-s(column LIKE pattern) as expected and 1.2) is using different fixed selectivities for wildcard prefixed patterns and not wildcard prefixed patterns. 1.3) is using different fixed selectivities for non wildcard prefixed patterns with varying length. 2) is using the same selectivity (5%) for LIKE :v as for NOT LIKE :v when using bind variables and (however) not using bind peeking. May be things changed again with using 11g and adaptive cursor sharing, ==> but at least on 10g I would expect a better cardinality estimation by the optimizer than presented from your example. Have a look at the different cardinality assumptions at the example below:

select * from T where UID not like 'LMN%'
union all
select * from T where UID like 'LMN%'
union all
select * from T where UID not like '%LMN%'
union all
select * from T where UID like '%LMN%'
union all
select * from T;

Ausführungsplan
----------------------------------------------------------
   0      SELECT STATEMENT Optimizer=FIRST_ROWS (Cost=14476 Card=6013368 Bytes=234521352)
   1    0   UNION-ALL
   2    1     TABLE ACCESS (FULL) OF 'T' (TABLE) (Cost=3624 Card=2004453 Bytes=78173667)
   3    1     TABLE ACCESS (BY INDEX ROWID) OF 'T' (TABLE) (Cost=1 Card=3 Bytes=117)
   4    3       INDEX (RANGE SCAN) OF 'PIT' (INDEX (UNIQUE)) (Cost=1 Card=3)
   5    1     TABLE ACCESS (FULL) OF 'T' (TABLE) (Cost=3624 Card=1904233 Bytes=74265087)
   6    1     TABLE ACCESS (FULL) OF 'T' (TABLE) (Cost=3618 Card=100223 Bytes=3908697)
   7    1     TABLE ACCESS (FULL) OF 'T' (TABLE) (Cost=3609 Card=2004456 Bytes=78173784)

variable b1 varchar2(20);

select * from T where UID not like :b1
union all
select * from T where UID like :b1
union all
select * from T;

Ausführungsplan
----------------------------------------------------------
   0      SELECT STATEMENT Optimizer=FIRST_ROWS (Cost=7929 Card=2204902 Bytes=85991178)
   1    0   UNION-ALL
   2    1     TABLE ACCESS (FULL) OF 'T' (TABLE) (Cost=3618 Card=100223 Bytes=3908697)
   3    1     TABLE ACCESS (BY INDEX ROWID) OF 'T' (TABLE) (Cost=702 Card=100223 Bytes=3908697)
   4    3       INDEX (RANGE SCAN) OF 'PIT' (INDEX (UNIQUE)) (Cost=10 Card=18040)
   5    1     TABLE ACCESS (FULL) OF 'T' (TABLE) (Cost=3609 Card=2004456 Bytes=78173784)

By: Greg Rahn

Greg Rahn — Thu, 12 Jun 2008 18:20:19 +0000

@Rainer

Q: Why the optimizer is not regarding the NOT operator ?
In my opinion the estimated selectivity for the negated LIKE predicate should be:
s(NOT LIKE)=1-s(LIKE)

A: Because that is not currently the logic in the code. They are not equivalent today.

I only performed my tests on 11.1.0.6.

If you check out Cost-Based Oracle Fundamentals by Jonathan Lewis, page 133, table 6-6 “Fixed Percentages Used for Selectivity on Character Expressions”, you can see his observations.

By: Rainer Stenzel

Rainer Stenzel — Thu, 12 Jun 2008 13:05:17 +0000

additional question and remark after some tests:
Was the presented scenario investigated with Oracle 9i ?
With Oracle 10g I found modified selectivities
with literals:
5% for column LIKE wild_card_prefixed_pattern
50% for column NOT LIKE wild_card_prefixed_pattern
respective with bind variables:
1% for column LIKE pattern_bind_variable
5% for column NOT LIKE pattern_bind_variable
undocumented and consequently quite wired.
Regards,
Rainer Stenzel

By: Rainer Stenzel

Rainer Stenzel — Thu, 12 Jun 2008 11:02:49 +0000

my first question was (and still is), why the optimizer is not regarding the NOT operator ?
In my opinion the estimated selectivity for the negated LIKE predicate shoud be:
s(NOT LIKE =1-s( LIKE )
but instead it seems to be equal.
Applying this to the presented query the total reduction for those four predicates should only be 95%^4 (about 0.8) instead of 5%^4 resulting into an about 100.000 times greater cardinality estimation.
Regards,
Rainer Stenzel

By: Todor Botev

Todor Botev — Wed, 23 Apr 2008 12:25:19 +0000

OK, so playing out with the level seems to be worth the effort.

As for level 1 – could it be that level 1 (OPTIMIZIER HINT) is only for the unanalyzed tables and it is missing in the docs?

By: Greg Rahn

Greg Rahn — Wed, 23 Apr 2008 12:02:43 +0000

Todor-

I had chosen 4 because I had initially used optimizer_dynamic_sampling = 4. I do recognize that they are not equals. I did just run a quick test and ran through all 10 levels of the hint. Here are the results:

DS Level  Rows
-------- -----
1            9
2         355K
3        1874K
4        1606K
5        1249K
6         654K
7        1419K
8         953K
9        1232K
10       1324K

Interesting enough, a larger sample does not always mean a more accurate estimate. This is probably data dependent. For some reason, level 1 did not produce a note in the explain plan that dynamic sampling was used, and thus the low estimate of 9. I’ll investigate this.

By: Todor Botev

Todor Botev — Wed, 23 Apr 2008 09:58:10 +0000

Great post! I liked the “methodology” of adding the predicates one at a time.

Why did you choose level 4 for dynamic sampling? According to the docs (11g), when you use a hint like in your case, level 1 could be enough.

By: Choosing An Optimal Stats Gathering Strategy | Structured Data

Choosing An Optimal Stats Gathering Strategy | Structured Data — Wed, 26 Mar 2008 08:05:04 +0000

[...] for DSS and data warehouse databases, where queries run for minutes, not seconds. See my post There Is No Time Like ‘%NOW%’ To Use Dynamic Sampling for a real-world [...]

By: David Aldridge

David Aldridge — Tue, 18 Mar 2008 20:54:35 +0000

I find that dynamic sampling to be very seductive. For quite a while I’ve been telling report developers who have problems with execution plans to turn up the dynamic sampling setting through a hint to see if the plan improves — sometime the default sampling rate is not high enough, and frankly it’s often an easy win.

If they’d just learn to apply dynamic sampling by default then I’d be practically out of a job.

By: Greg Rahn

Greg Rahn — Fri, 07 Mar 2008 03:43:45 +0000

You are completely spot on Steve. Sometimes sampling data is the only way to get the true relationship between the columns.

Currently multi-column/extended statistics only work for single equality predicates so dynamic sampling may be one of the only ways to get the cardinality estimate close.