powered by simpleCommunicator - 2.0.53     © 2025 Programmizd 02
Форумы / Oracle [игнор отключен] [закрыт для гостей] / Периоды, объединение
25 сообщений из 37, страница 1 из 2
Периоды, объединение
    #33605188
Калина
Скрыть профиль Поместить в игнор-лист Сообщения автора в теме
Участник
Имеется табличка
Код: plaintext
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
CREATE TABLE TST3
(
  F0     VARCHAR2( 10  BYTE),
  F1     VARCHAR2( 10  BYTE),
  SDATE  DATE,
  EDATE  DATE
);
Insert into TST3
   (F0, F1, SDATE, EDATE)
 Values
   ('F0', 'F1', TO_DATE('03/01/2006 00:00:00', 'MM/DD/YYYY HH24:MI:SS'), TO_DATE('03/02/2006 00:00:00', 'MM/DD/YYYY HH24:MI:SS'));
Insert into TST3
   (F0, F1, SDATE, EDATE)
 Values
   ('F0', 'F1', TO_DATE('03/02/2006 00:00:00', 'MM/DD/YYYY HH24:MI:SS'), TO_DATE('03/03/2006 00:00:00', 'MM/DD/YYYY HH24:MI:SS'));
Insert into TST3
   (F0, F1, SDATE, EDATE)
 Values
   ('F0', 'F2', TO_DATE('03/03/2006 00:00:00', 'MM/DD/YYYY HH24:MI:SS'), TO_DATE('03/04/2006 00:00:00', 'MM/DD/YYYY HH24:MI:SS'));
Insert into TST3
   (F0, F1, SDATE, EDATE)
 Values
   ('F0', 'F1', TO_DATE('03/04/2006 00:00:00', 'MM/DD/YYYY HH24:MI:SS'), TO_DATE('03/05/2006 00:00:00', 'MM/DD/YYYY HH24:MI:SS'));
Insert into TST3
   (F0, F1, SDATE, EDATE)
 Values
   ('F0', 'F1', TO_DATE('03/05/2006 00:00:00', 'MM/DD/YYYY HH24:MI:SS'), TO_DATE('03/08/2006 00:00:00', 'MM/DD/YYYY HH24:MI:SS'));
Insert into TST3
   (F0, F1, SDATE, EDATE)
 Values
   ('F0', 'F1', TO_DATE('03/08/2006 00:00:00', 'MM/DD/YYYY HH24:MI:SS'), TO_DATE('03/09/2006 00:00:00', 'MM/DD/YYYY HH24:MI:SS'));
COMMIT;
Необходимо объединить периоды для которых комбинация f0,f1 неизменна
, тоесть

F0|F1|01.03.2006|03.03.2006
F0|F2|03.03.2006|04.03.2006
F0|F1|04.03.2006|09.03.2006
Понимаю, что вроде бы как аналитические ф-ции, но немогу понять, сделать такое окно .
Может кто подтолкнет в нужную сторону?
Исходная табличка - 60 млн


------------------------------
Not affilated with VAZ
...
Рейтинг: 0 / 0
Периоды, объединение
    #33605287
Фотография Elic
Скрыть профиль Поместить в игнор-лист Сообщения автора в теме
Участник
КалинаНеобходимо объединить периоды для которых комбинация f0,f1 неизменна
Код: plaintext
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
select f0, f1, min(sdate) as sdate, max(edate) as edate
  from
  ( select f0, f1, sdate, edate, sum(start_of_group) over (partition by f0, f1 order by sdate) as group_no
      from
      ( select f0, f1, sdate, edate,
            decode(lag(edate) over (partition by f0, f1 order by sdate), sdate,  0 ,  1 ) as start_of_group
          from tst3
      )
  )
  group by f0, f1, group_no
  order by  3 
;

F0         F1         SDATE             EDATE
---------- ---------- ----------------- -----------------
F0         F1          01 . 03 . 06   00 : 00 : 00   03 . 03 . 06   00 : 00 : 00 
F0         F2          03 . 03 . 06   00 : 00 : 00   04 . 03 . 06   00 : 00 : 00 
F0         F1          04 . 03 . 06   00 : 00 : 00   09 . 03 . 06   00 : 00 : 00 
КалинаИсходная табличка - 60 млнIMHO, умрёт.
...
Рейтинг: 0 / 0
Периоды, объединение
    #33605310
Q u a d r o
Скрыть профиль Поместить в игнор-лист Сообщения автора в теме
Участник
Код: plaintext
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
SQL> select f0, f1, min(sdate) sdate, max(edate) edate
   2     from (
   3       select f0, f1, sdate, edate, sum(window_start) over (order by sdate) window
   4         from (
   5           select f0, f1, sdate, edate,
   6               case
   7                 when lag(f0,  1 , f0) over (order by sdate) != f0
   8                   or lag(f1,  1 , f1) over (order by sdate) != f1 then  1  else  0 
   9               end window_start
  10             from tst3)
  11     ) group by f0, f1, window
  12       order by sdate;

F0         F1         SDATE     EDATE
---------- ---------- --------- ---------
F0         F1          01 -MAR- 06   03 -MAR- 06 
F0         F2          03 -MAR- 06   04 -MAR- 06 
F0         F1          04 -MAR- 06   09 -MAR- 06 
...
Рейтинг: 0 / 0
Периоды, объединение
    #33605349
Jannny
Скрыть профиль Поместить в игнор-лист Сообщения автора в теме
Участник
КалинаИсходная табличка - 60 млн
А расскажите потом, что она скажет на группировку? Мы в свое время не добились, чтобы группировка такого же объема проходила. ;(
...
Рейтинг: 0 / 0
Периоды, объединение
    #33605383
Калина
Скрыть профиль Поместить в игнор-лист Сообщения автора в теме
Участник
Q u a d r o
Код: plaintext
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
SQL> select f0, f1, min(sdate) sdate, max(edate) edate
   2     from (
   3       select f0, f1, sdate, edate, sum(window_start) over (order by sdate) window
   4         from (
   5           select f0, f1, sdate, edate,
   6               case
   7                 when lag(f0,  1 , f0) over (order by sdate) != f0
   8                   or lag(f1,  1 , f1) over (order by sdate) != f1 then  1  else  0 
   9               end window_start
  10             from tst3)
  11     ) group by f0, f1, window
  12       order by sdate;

F0         F1         SDATE     EDATE
---------- ---------- --------- ---------
F0         F1          01 -MAR- 06   03 -MAR- 06 
F0         F2          03 -MAR- 06   04 -MAR- 06 
F0         F1          04 -MAR- 06   09 -MAR- 06 


еще и полей типа f1,f0 - дофига, 13 штук
...
Рейтинг: 0 / 0
Периоды, объединение
    #33605434
Q u a d r o
Скрыть профиль Поместить в игнор-лист Сообщения автора в теме
Участник
...может быть что-нибудь в этом духе?...
( если у Вас может быть "разделитель" )

Код: plaintext
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
select f0, f1, min(sdate) sdate, max(edate) edate
  from (
    select f0, f1, sdate, edate, sum(window_start) over (order by sdate) window
      from (
        select f0, f1, sdate, edate,
            case
              when lag(f0||'/'||f1,  1 , f0||'/'||f1) over (order by sdate) != f0||'/'||f1
              then  1  else  0 
            end window_start
          from tst3)
  ) group by f0, f1, window
    order by sdate;
...
Рейтинг: 0 / 0
Периоды, объединение
    #33605729
Фотография Elic
Скрыть профиль Поместить в игнор-лист Сообщения автора в теме
Участник
Q u a d r o
Код: plaintext
1.
2.
3.
4.
   6               case
   7                 when lag(f0,  1 , f0) over (order by sdate) != f0
   8                   or lag(f1,  1 , f1) over (order by sdate) != f1 then  1  else  0 
   9               end window_start
Кстати, этот вариант не совсем "объединяет периоды", т.к. на пропусках даёт ложное объединение.
...
Рейтинг: 0 / 0
Периоды, объединение
    #33606138
Фотография SY
Скрыть профиль Поместить в игнор-лист Сообщения автора в теме
Участник
Since Исходная табличка - 60 млн, I would write a pipelined table function:

Код: plaintext
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.
34.
35.
36.
37.
38.
39.
40.
41.
42.
43.
44.
45.
46.
47.
48.
49.
50.
51.
52.
53.
54.
55.
56.
57.
58.
59.
60.
61.
62.
63.
64.
65.
66.
67.
68.
69.
70.
71.
72.
73.
74.
75.
76.
77.
78.
79.
80.
SQL> create or replace
   2     type tst3_obj_type
   3       is object(
   4                 f0    varchar2( 10  byte),
   5                 f1    varchar2( 10  byte),
   6                 sdate date,
   7                 edate date
   8                );
   9   /

Type created.

SQL> create or replace
   2     type tst3_tbl_type
   3       is table of tst3_obj_type;
   4   /

Type created.

SQL> create or replace
   2     function combine
   3       return tst3_tbl_type
   4       pipelined
   5       is
   6           cursor cur
   7             is
   8               select  *
   9                 from  tst3
  10                 order by f0,
  11                          f1,
  12                          sdate;
  13           comb_tst3 tst3_obj_type := tst3_obj_type(null,null,null,null);
  14           curr_tst3 tst3_obj_type := tst3_obj_type(null,null,null,null);
  15       begin
  16           open cur;
  17           fetch cur
  18             into comb_tst3.f0,
  19                  comb_tst3.f1,
  20                  comb_tst3.sdate,
  21                  comb_tst3.edate;
  22           if cur%found
  23             then
  24               loop
  25                 fetch cur
  26                   into curr_tst3.f0,
  27                        curr_tst3.f1,
  28                        curr_tst3.sdate,
  29                        curr_tst3.edate;
  30                 exit when cur%notfound;
  31                 if     comb_tst3.f0 = curr_tst3.f0
  32                    and
  33                        comb_tst3.f1 = curr_tst3.f1
  34                    and
  35                        comb_tst3.edate = curr_tst3.sdate
  36                   then
  37                     comb_tst3.edate := curr_tst3.edate;
  38                 else
  39                   pipe row(comb_tst3);
  40                   comb_tst3 := curr_tst3;
  41                 end if;
  42               end loop;
  43               pipe row(comb_tst3);
  44           end if;
  45           close cur;
  46           return;
  47   end combine;
  48   /

Function created.

SQL> select  *
   2     from  table(combine)
   3   /

F0         F1         SDATE     EDATE
---------- ---------- --------- ---------
F0         F1          01 -MAR- 06   03 -MAR- 06 
F0         F1          04 -MAR- 06   09 -MAR- 06 
F0         F2          03 -MAR- 06   04 -MAR- 06 

SQL> 

In such case we deal with one ORDER BY versus windowing/partitioning + group by + order by. And if there is an index on F0,F1,SDATE,EDATE it might be not that bad.

SY.
...
Рейтинг: 0 / 0
Периоды, объединение
    #33606160
Фотография SY
Скрыть профиль Поместить в игнор-лист Сообщения автора в теме
Участник
One more thing - I assume intervals do not cross, otherwise it would be a bit more complex logic.

SY.
...
Рейтинг: 0 / 0
Периоды, объединение
    #33606870
Q u a d r o
Скрыть профиль Поместить в игнор-лист Сообщения автора в теме
Участник
SY I would write a pipelined table function

how about "test it"?

let's see how it goes.
...
Рейтинг: 0 / 0
Периоды, объединение
    #33606889
Фотография SY
Скрыть профиль Поместить в игнор-лист Сообщения автора в теме
Участник
Q u a d r ohow about "test it"?

Well, I provided test results against Калина's small sample. If you mean volume testing, provide a test case and I'll gladly do it. In any case, as I already noted, PL/SQL solution uses a single pass through a sorted table comparing to window sort, group by and order by.

SY.
...
Рейтинг: 0 / 0
Периоды, объединение
    #33606893
Q u a d r o
Скрыть профиль Поместить в игнор-лист Сообщения автора в теме
Участник
tkprof the analytics

tkprof the pipelined function

compare them

pipelined mean "write a lot of code" - we need to justify this one.
...
Рейтинг: 0 / 0
Периоды, объединение
    #33606911
Фотография SY
Скрыть профиль Поместить в игнор-лист Сообщения автора в теме
Участник
Q u a d r otkprof the analytics

tkprof the pipelined function

compare them

You do not need tkprof to compare

Код: plaintext
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
select f0, f1, min(sdate) as sdate, max(edate) as edate
  from
  ( select f0, f1, sdate, edate, sum(start_of_group) over (partition by f0, f1 order by sdate) as group_no
      from
      ( select f0, f1, sdate, edate,
            decode(lag(edate) over (partition by f0, f1 order by sdate), sdate,  0 ,  1 ) as start_of_group
          from tst3
      )
  )
  group by f0, f1, group_no
  order by  3 
;

and

Код: plaintext
1.
2.
3.
4.
            select  *
              from  tst3
              order by f0,
                       f1,
                       sdate;

And tkprof can not evaluate PL/SQL portion of table function.

Q u a d r opipelined mean "write a lot of code" - we need to justify this one.

If avoiding partitioninig+window sort and group by on 60,000,000 rows is not enough, then I do not know what justification you are looking for. And BTW, I do not see any complexity in that table function.

Anyway, below is my test:

Код: plaintext
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.
34.
35.
36.
37.
38.
39.
40.
41.
42.
43.
44.
45.
46.
47.
48.
49.
50.
51.
52.
53.
54.
55.
56.
57.
58.
59.
60.
61.
62.
63.
64.
65.
66.
67.
68.
69.
70.
71.
72.
73.
74.
75.
SQL> truncate table tst3;

Table truncated.

Elapsed:  00 : 00 : 00 . 01 
SQL> insert into tst3
   2   select 'f0','f1',trunc(sysdate) - rownum,trunc(sysdate) - rownum +  1  from dba_objects,dba_users
;

 980463  rows created.

Elapsed:  00 : 01 : 21 . 02 
SQL> commit;

Commit complete.

Elapsed:  00 : 00 : 00 . 00 
SQL> select f0, f1, min(sdate) as sdate, max(edate) as edate
   2     from
   3     ( select f0, f1, sdate, edate, sum(start_of_group) over (partition by f0, f1 order by sdate) 
as group_no
   4         from
   5         ( select f0, f1, sdate, edate,
   6               decode(lag(edate) over (partition by f0, f1 order by sdate), sdate,  0 ,  1 ) as start_
of_group
   7             from tst3
   8         )
   9     )
  10     group by f0, f1, group_no
  11     order by  3 
  12   ;

F0         F1         SDATE     EDATE
---------- ---------- --------- ---------
f0         f1          22 -OCT- 79   16 -MAR- 06 

Elapsed:  00 : 02 : 22 . 04 
SQL> select  *
   2     from  table(combine)
   3   /

F0         F1         SDATE     EDATE
---------- ---------- --------- ---------
f0         f1          22 -OCT- 79   16 -MAR- 06 

Elapsed:  00 : 01 : 41 . 09 
SQL> select f0, f1, min(sdate) as sdate, max(edate) as edate
   2     from
   3     ( select f0, f1, sdate, edate, sum(start_of_group) over (partition by f0, f1 order by sdate) 
as group_no
   4         from
   5         ( select f0, f1, sdate, edate,
   6               decode(lag(edate) over (partition by f0, f1 order by sdate), sdate,  0 ,  1 ) as start_
of_group
   7             from tst3
   8         )
   9     )
  10     group by f0, f1, group_no
  11     order by  3 
  12   ;

F0         F1         SDATE     EDATE
---------- ---------- --------- ---------
f0         f1          22 -OCT- 79   16 -MAR- 06 

Elapsed:  00 : 02 : 11 . 07 
SQL> select  *
   2     from  table(combine)
   3   /

F0         F1         SDATE     EDATE
---------- ---------- --------- ---------
f0         f1          22 -OCT- 79   16 -MAR- 06 

Elapsed:  00 : 01 : 40 . 01 
SQL> 

SY.
...
Рейтинг: 0 / 0
Периоды, объединение
    #33606922
Q u a d r o
Скрыть профиль Поместить в игнор-лист Сообщения автора в теме
Участник
SY And tkprof can not evaluate PL/SQL portion of table function.

Yes it can??

Anyway, you have only one combination of f0 and f1 - not a fair comparsion

how about:

Код: plaintext
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.
34.
35.
36.
37.
38.
39.
40.
41.
42.
43.
44.
45.
46.
47.
SQL> truncate table tst3;

Table truncated.

SQL> insert into tst3
   2    select mod(level,  1000 ),
   3      mod(level,  100 ),
   4      trunc(sysdate) - level, trunc(sysdate) - level +  1 
   5     from dual
   6     connect by level <=  1000000 ;

 1000000  rows created.

SQL> commit;

Commit complete.

SQL> alter session set sql_trace=true;

Session altered.

SQL> select count(*) from
   2   (
   3   select f0, f1, min(sdate) as sdate, max(edate) as edate
   4     from
   5     ( select f0, f1, sdate, edate, sum(start_of_group) over (partition by f0, f1 order by sdate) 
as group_no
   6         from
   7         ( select f0, f1, sdate, edate,
   8               decode(lag(edate) over (partition by f0, f1 order by sdate), sdate,  0 ,  1 ) as start_
of_group
   9             from tst3
  10         )
  11     )
  12     group by f0, f1, group_no
  13     order by  3 
  14   );

  COUNT(*)
----------
    1000000 

SQL> select count(*) from table(combine);

  COUNT(*)
----------
    1000000 

Код: plaintext
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.
34.
35.
36.
37.
38.
39.
40.
41.
42.
43.
44.
45.
46.
47.
48.
49.
50.
51.
52.
53.
54.
55.
56.
57.
58.
59.
60.
select count(*) 
from
 table(combine)


call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse         1        0 . 00         0 . 00            0            0            0             0 
Execute       1        0 . 00         0 . 00            0            0            0             0 
Fetch         2       31 . 40        32 . 79            0            0            0             1 
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total         4       31 . 40        32 . 79            0            0            0             1 

Misses in library cache during parse:  0 
Optimizer mode: ALL_ROWS
Parsing user id:  61   

Rows     Row Source Operation
-------  ---------------------------------------------------
       1   SORT AGGREGATE (cr= 1979  pr= 0  pw= 0  time= 37253670  us)
 1000000    COLLECTION ITERATOR PICKLER FETCH COMBINE (cr= 1979  pr= 0  pw= 0  time= 59936188  us)

select count(*) from
(
select f0, f1, min(sdate) as sdate, max(edate) as edate
  from
  ( select f0, f1, sdate, edate, sum(start_of_group) over (partition by f0, f1 order by sdate) as group_no
      from
      ( select f0, f1, sdate, edate,
            decode(lag(edate) over (partition by f0, f1 order by sdate), sdate,  0 ,  1 ) as start_of_group
          from tst3
      )
  )
  group by f0, f1, group_no
  order by  3 
)

call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse         1        0 . 00         0 . 00            0            1            0             0 
Execute       1        0 . 00         0 . 00            0            0            0             0 
Fetch         2        9 . 88         9 . 64            0         1912            0             1 
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total         4        9 . 88         9 . 65            0         1913            0             1 

Misses in library cache during parse:  1 
Optimizer mode: ALL_ROWS
Parsing user id:  61   

Rows     Row Source Operation
-------  ---------------------------------------------------
       1   SORT AGGREGATE (cr= 1912  pr= 0  pw= 0  time= 9647574  us)
 1000000    VIEW  (cr= 1912  pr= 0  pw= 0  time= 12059974  us)
 1000000     HASH GROUP BY (cr= 1912  pr= 0  pw= 0  time= 10059970  us)
 1000000      VIEW  (cr= 1912  pr= 0  pw= 0  time= 11792996  us)
 1000000       WINDOW BUFFER (cr= 1912  pr= 0  pw= 0  time= 8792992  us)
 1000000        VIEW  (cr= 1912  pr= 0  pw= 0  time= 7898566  us)
 1000000         WINDOW SORT (cr= 1912  pr= 0  pw= 0  time= 4898560  us)
 1000000          TABLE ACCESS FULL TST3 (cr= 1912  pr= 0  pw= 0  time= 1000087  us)


pipelined is tad slower

so "it depends"
...
Рейтинг: 0 / 0
Периоды, объединение
    #33607020
SkyNet77
Скрыть профиль Поместить в игнор-лист Сообщения автора в теме
Гость
But "group by" clause is not neccessary, we can use "over(partition by " on external query instead of "group by" and decrease sort operations
...
Рейтинг: 0 / 0
Периоды, объединение
    #33607149
Q u a d r o
Скрыть профиль Поместить в игнор-лист Сообщения автора в теме
Участник
benchmark it.

group by doesn't imply "sort", look above - hash group by - for you see
...
Рейтинг: 0 / 0
Периоды, объединение
    #33607179
Фотография Elic
Скрыть профиль Поместить в игнор-лист Сообщения автора в теме
Участник
2 SY & Q u a d r o
Спасибо за сравнение разных алгоритмов.
Лично мне интуитивно гораздо больше нравится (исключительно с точки зрения потенции в производительности) вариант SY. Более того, в нём имеется запас для оптимизации путём уменьшения переключений контекста при помощи bulk-fetch (а в 10-ке и вообще просто for-select-loop).

______________________________________________________________________

2 SkyNet77
Ты путаешь агрегацию с сортировкой.
...
Рейтинг: 0 / 0
Периоды, объединение
    #33607252
Калина
Скрыть профиль Поместить в игнор-лист Сообщения автора в теме
Участник
Мне лично вариант с Pipelined кажется не очень , но и SY и Elic - люди серьезные, в понедельник сделаю тестовый прогон на 1- млн , сравню все варианты
------------------------------
Not affilated with VAZ
...
Рейтинг: 0 / 0
Периоды, объединение
    #33608173
Фотография SeaGate
Скрыть профиль Поместить в игнор-лист Сообщения автора в теме
Участник
Вот, вроде еще не предлагали, для десятки. Работает медленно, зато красиво выглядит :))
Код: plaintext
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
Connected to Oracle Database 10g Enterprise Edition Release  10 . 2 . 0 . 1 . 0  
Connected as digger


SQL> create table tmp (db date,de date);

Table created

SQL> insert into tmp (select sysdate,sysdate+ 1  from dual union all select sysdate+ 1 ,sysdate+ 2  from dual union all select sysdate+ 3 ,sysdate+ 4  from dual); 

 3  rows inserted

SQL> commit;

Commit complete

SQL> select min(connect_by_root(t.db)) db,
   2   t.de
   3   from tmp t
   4   where connect_by_isleaf =  1 
   5   connect by t.db = prior t.de
   6   group by t.de
   7   order by db;

DB          DE
----------- -----------
 17 . 03 . 2006    19 . 03 . 2006 
 20 . 03 . 2006    21 . 03 . 2006 
...
Рейтинг: 0 / 0
Периоды, объединение
    #33609055
Фотография SY
Скрыть профиль Поместить в игнор-лист Сообщения автора в теме
Участник
Well, I tried Quadro's test on my laptop:

Код: plaintext
1.
2.
3.
Dell Inspirion I1150
Pentium  4 
CPU  2 .8GHz
RAM 512MB
And I came up with opposite results:

Код: plaintext
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.
34.
35.
36.
37.
38.
39.
40.
41.
42.
43.
44.
45.
46.
47.
48.
49.
50.
51.
52.
53.
54.
55.
56.
57.
58.
59.
60.
61.
62.
63.
64.
65.
66.
67.
68.
69.
70.
71.
72.
73.
74.
75.
76.
77.
78.
79.
80.
81.
82.
83.
84.
85.
86.
87.
88.
89.
90.
91.
92.
93.
94.
95.
96.
97.
98.
99.
100.
101.
102.
103.
104.
105.
106.
107.
108.
109.
110.
111.
112.
113.
114.
115.
116.
117.
118.
119.
120.
121.
122.
123.
124.
125.
126.
127.
128.
129.
130.
131.
132.
133.
134.
135.
136.
137.
138.
139.
140.
141.
142.
143.
144.
145.
146.
147.
148.
149.
150.
151.
152.
153.
154.
155.
156.
157.
158.
159.
160.
161.
162.
163.
164.
165.
166.
167.
168.
169.
170.
171.
172.
173.
174.
175.
176.
177.
178.
179.
180.
181.
182.
183.
184.
185.
186.
187.
188.
189.
190.
191.
192.
193.
194.
195.
196.
197.
198.
199.
200.
201.
202.
203.
204.
205.
206.
207.
208.
209.
210.
211.
212.
213.
214.
215.
216.
217.
218.
219.
220.
221.
222.
223.
224.
225.
226.
227.
228.
229.
230.
231.
232.
233.
234.
SQL*Plus: Release  10 . 2 . 0 . 1 . 0  - Production on Fri Mar  17   09 : 30 : 47   2006 

Copyright (c)  1982 ,  2005 , Oracle.  All rights reserved.


Connected to:
Oracle Database 10g Enterprise Edition Release  10 . 2 . 0 . 1 . 0  - Production
With the Partitioning, OLAP and Data Mining options

SQL> set timing on
SQL> connect scott/tiger
Connected.
SQL> drop table tst3;

Table dropped.

Elapsed:  00 : 00 : 01 . 99 
SQL> CREATE TABLE TST3
   2   (
   3     F0     VARCHAR2( 10  BYTE),
   4     F1     VARCHAR2( 10  BYTE),
   5     SDATE  DATE,
   6     EDATE  DATE
   7   );

Table created.

Elapsed:  00 : 00 : 00 . 30 
SQL> insert into tst3
   2    select mod(level,  1000 ),
   3      mod(level,  100 ),
   4      trunc(sysdate) - level, trunc(sysdate) - level +  1 
   5     from dual
   6     connect by level <=  1000000 ;

 1000000  rows created.

Elapsed:  00 : 00 : 09 . 97 
SQL> commit;

Commit complete.

Elapsed:  00 : 00 : 00 . 02 
SQL> connect / as sysdba
Connected.
SQL> shutdown
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> startup
ORACLE instance started.

Total System Global Area   167772160  bytes
Fixed Size                   1247900  bytes
Variable Size               75498852  bytes
Database Buffers            88080384  bytes
Redo Buffers                 2945024  bytes
Database mounted.
Database opened.
SQL> connect scott/tiger
Connected.
SQL> alter session set sql_trace=true;

Session altered.

Elapsed:  00 : 00 : 00 . 05 
SQL> select count(*) from (
   2   select f0, f1, min(sdate) as sdate, max(edate) as edate
   3     from
   4     ( select f0, f1, sdate, edate, sum(start_of_group) over (partition by f0, f1 order by sdate) 
as group_no
   5         from
   6         ( select f0, f1, sdate, edate,
   7               decode(lag(edate) over (partition by f0, f1 order by sdate), sdate,  0 ,  1 ) as start_
of_group
   8             from tst3
   9         )
  10     )
  11     group by f0, f1, group_no
  12     order by  3 
  13   )
  14   ;

  COUNT(*)
----------
    1000000 

Elapsed:  00 : 01 : 25 . 70 
SQL> connect / as sysdba
Connected.
SQL> shutdown
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> startup
ORACLE instance started.

Total System Global Area   167772160  bytes
Fixed Size                   1247900  bytes
Variable Size               75498852  bytes
Database Buffers            88080384  bytes
Redo Buffers                 2945024  bytes
Database mounted.
Database opened.
SQL> connect scott/tiger
Connected.
SQL> alter session set sql_trace=true;

Session altered.

Elapsed:  00 : 00 : 00 . 03 
SQL> select count(*) from (select * from table(combine));

  COUNT(*)
----------
    1000000 

Elapsed:  00 : 00 : 58 . 87 
SQL> connect scott/tiger
Connected.
SQL> drop table tst3;

Table dropped.

Elapsed:  00 : 00 : 01 . 96 
SQL> CREATE TABLE TST3
   2   (
   3     F0     VARCHAR2( 10  BYTE),
   4     F1     VARCHAR2( 10  BYTE),
   5     SDATE  DATE,
   6     EDATE  DATE
   7   );

Table created.

Elapsed:  00 : 00 : 00 . 29 
SQL> insert into tst3
   2    select mod(level,  1000 ),
   3      mod(level,  100 ),
   4      trunc(sysdate) - level, trunc(sysdate) - level +  1 
   5     from dual
   6     connect by level <=  1000000 ;

 1000000  rows created.

Elapsed:  00 : 00 : 08 . 63 
SQL> commit;

Commit complete.

Elapsed:  00 : 00 : 00 . 02 
SQL> exec dbms_stats.gather_table_stats('SCOTT','TST3');

PL/SQL procedure successfully completed.

Elapsed:  00 : 00 : 00 . 95 
SQL> connect / as sysdba
Connected.
SQL> shutdown
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> startup
ORACLE instance started.

Total System Global Area   167772160  bytes
Fixed Size                   1247900  bytes
Variable Size               75498852  bytes
Database Buffers            88080384  bytes
Redo Buffers                 2945024  bytes
Database mounted.
Database opened.
SQL> connect scott/tiger
Connected.
SQL> alter session set sql_trace=true;

Session altered.

Elapsed:  00 : 00 : 00 . 03 
SQL> select count(*) from (
   2   select f0, f1, min(sdate) as sdate, max(edate) as edate
   3     from
   4     ( select f0, f1, sdate, edate, sum(start_of_group) over (partition by f0, f1 order by sdate) 
as group_no
   5         from
   6         ( select f0, f1, sdate, edate,
   7               decode(lag(edate) over (partition by f0, f1 order by sdate), sdate,  0 ,  1 ) as start_
of_group
   8             from tst3
   9         )
  10     )
  11     group by f0, f1, group_no
  12     order by  3 
  13   )
  14   ;

  COUNT(*)
----------
    1000000 

Elapsed:  00 : 01 : 17 . 99 
SQL> connect / as sysdba
Connected.
SQL> shutdown
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> startup
ORACLE instance started.

Total System Global Area   167772160  bytes
Fixed Size                   1247900  bytes
Variable Size               75498852  bytes
Database Buffers            88080384  bytes
Redo Buffers                 2945024  bytes
Database mounted.
Database opened.
SQL> connect scott/tiger
Connected.
SQL> alter session set sql_trace=true;

Session altered.

Elapsed:  00 : 00 : 00 . 04 
SQL> set timing on
SQL> select count(*) from (select * from table(combine));

  COUNT(*)
----------
    1000000 

Elapsed:  00 : 00 : 57 . 36 
SQL> connect scott/tiger
Connected.
SQL> 

As you can see, with or without stats, pipelined table function is faster. Below are TKPROF results:


Код: plaintext
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.
34.
35.
36.
37.
38.
39.
40.
41.
42.
43.
44.
 ********************************************************************************

select count(*) from (
select f0, f1, min(sdate) as sdate, max(edate) as edate
  from
  ( select f0, f1, sdate, edate, sum(start_of_group) over (partition by f0, f1 order by sdate) as group_no
      from
      ( select f0, f1, sdate, edate,
            decode(lag(edate) over (partition by f0, f1 order by sdate), sdate,  0 ,  1 ) as start_of_group
          from tst3
      )
  )
  group by f0, f1, group_no
  order by  3 
)

call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse         1        0 . 02         0 . 06            2            2            0             0 
Execute       1        0 . 00         0 . 00            0            0            0             0 
Fetch         2       16 . 53        81 . 90        32505         6552           95             1 
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total         4       16 . 55        81 . 97        32507         6554           95             1 

Misses in library cache during parse:  1 
Optimizer mode: ALL_ROWS
Parsing user id:  54   

Rows     Row Source Operation
-------  ---------------------------------------------------
       1   SORT AGGREGATE (cr= 6552  pr= 32505  pw= 28932  time= 81908378  us)
 1000000    VIEW  (cr= 6552  pr= 32505  pw= 28932  time= 78197087  us)
 1000000     HASH GROUP BY (cr= 6552  pr= 32505  pw= 28932  time= 73197078  us)
 1000000      VIEW  (cr= 6552  pr= 27332  pw= 23759  time= 70310715  us)
 1000000       WINDOW BUFFER (cr= 6552  pr= 27332  pw= 23759  time= 66310704  us)
 1000000        VIEW  (cr= 6552  pr= 22856  pw= 19299  time= 66602881  us)
 1000000         WINDOW SORT (cr= 6552  pr= 22856  pw= 19299  time= 61602868  us)
 1000000          TABLE ACCESS FULL TST3 (cr= 6552  pr= 3556  pw= 0  time= 8020228  us)




********************************************************************************


Код: plaintext
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.
34.
35.
36.
37.
38.
39.
40.
41.
42.
43.
44.
45.
46.
47.
********************************************************************************

select count(*) 
from
 (select * from table(combine))


call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse         1        0 . 04         0 . 06            1          255            1             0 
Execute       1        0 . 00         0 . 00            0            0            0             0 
Fetch         2       27 . 95        30 . 00            0            0            0             1 
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total         4       27 . 99        30 . 06            1          255            1             1 

Misses in library cache during parse:  1 
Optimizer mode: ALL_ROWS
Parsing user id:  54   

Rows     Row Source Operation
-------  ---------------------------------------------------
       1   SORT AGGREGATE (cr= 4033  pr= 7811  pw= 3974  time= 55452110  us)
 1000000    COLLECTION ITERATOR PICKLER FETCH COMBINE (cr= 4033  pr= 7811  pw= 3974  time= 52225624  us)

********************************************************************************

********************************************************************************

SELECT * 
FROM
 TST3 ORDER BY F0, F1, SDATE


call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse         1        0 . 01         0 . 03            1            1            0             0 
Execute       1        0 . 00         0 . 00            0            0            0             0 
Fetch    1000001        8 . 01        23 . 41         7351         3841           27       1000000 
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total    1000003        8 . 02        23 . 45         7352         3842           27       1000000 

Misses in library cache during parse:  1 
Optimizer mode: ALL_ROWS
Parsing user id:  54      (recursive depth:  1 )



********************************************************************************


Код: plaintext
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.
34.
35.
36.
37.
38.
39.
40.
41.
42.
********************************************************************************

select count(*) from (
select f0, f1, min(sdate) as sdate, max(edate) as edate
  from
  ( select f0, f1, sdate, edate, sum(start_of_group) over (partition by f0, f1 order by sdate) as group_no
      from
      ( select f0, f1, sdate, edate,
            decode(lag(edate) over (partition by f0, f1 order by sdate), sdate,  0 ,  1 ) as start_of_group
          from tst3
      )
  )
  group by f0, f1, group_no
  order by  3 
)

call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse         1        0 . 01         0 . 01            0            0            0             0 
Execute       1        0 . 00         0 . 00            0            0            0             0 
Fetch         2       16 . 67        77 . 30        36463         3841           94             1 
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total         4       16 . 68        77 . 31        36463         3841           94             1 

Misses in library cache during parse:  1 
Optimizer mode: ALL_ROWS
Parsing user id:  54   

Rows     Row Source Operation
-------  ---------------------------------------------------
       1   SORT AGGREGATE (cr= 3841  pr= 36463  pw= 32611  time= 77304018  us)
 1000000    VIEW  (cr= 3841  pr= 36463  pw= 32611  time= 76480138  us)
 1000000     HASH GROUP BY (cr= 3841  pr= 36463  pw= 32611  time= 71480129  us)
 1000000      VIEW  (cr= 3841  pr= 27223  pw= 23371  time= 67697064  us)
 1000000       WINDOW BUFFER (cr= 3841  pr= 27223  pw= 23371  time= 62697054  us)
 1000000        VIEW  (cr= 3841  pr= 22747  pw= 18911  time= 64215930  us)
 1000000         WINDOW SORT (cr= 3841  pr= 22747  pw= 18911  time= 59215918  us)
 1000000          TABLE ACCESS FULL TST3 (cr= 3841  pr= 3835  pw= 0  time= 21089208  us)




********************************************************************************

Код: plaintext
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.
34.
35.
36.
37.
38.
39.
40.
41.
42.
43.
44.
45.
46.
47.
48.
49.
50.
51.
52.
53.
********************************************************************************

select count(*) 
from
 (select * from table(combine))


call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse         1        0 . 03         0 . 08            1          255            1             0 
Execute       1        0 . 00         0 . 00            0            0            0             0 
Fetch         2       27 . 28        29 . 05            0            0            0             1 
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total         4       27 . 31        29 . 13            1          255            1             1 

Misses in library cache during parse:  1 
Optimizer mode: ALL_ROWS
Parsing user id:  54   

Rows     Row Source Operation
-------  ---------------------------------------------------
       1   SORT AGGREGATE (cr= 3967  pr= 7906  pw= 4066  time= 36384922  us)
 1000000    COLLECTION ITERATOR PICKLER FETCH COMBINE (cr= 3967  pr= 7906  pw= 4066  time= 46619098  us)

********************************************************************************

********************************************************************************

SELECT * 
FROM
 TST3 ORDER BY F0, F1, SDATE


call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse         1        0 . 00         0 . 00            0            0            0             0 
Execute       1        0 . 00         0 . 00            0            0            0             0 
Fetch    1000001        8 . 23        24 . 23         7901         3841           34       1000000 
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total    1000003        8 . 23        24 . 24         7901         3841           34       1000000 

Misses in library cache during parse:  1 
Optimizer mode: ALL_ROWS
Parsing user id:  54      (recursive depth:  1 )

Rows     Row Source Operation
-------  ---------------------------------------------------
 1000000   SORT ORDER BY (cr= 3841  pr= 7901  pw= 4066  time= 11519767  us)
 1000000    TABLE ACCESS FULL TST3 (cr= 3841  pr= 3835  pw= 0  time= 4037466  us)




********************************************************************************
...
Рейтинг: 0 / 0
Периоды, объединение
    #33610160
Q u a d r o
Скрыть профиль Поместить в игнор-лист Сообщения автора в теме
Участник
Well...

you have a little low workarea size :-)

see - here is the problem:

Код: plaintext
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
SELECT * 
FROM
 TST3 ORDER BY F0, F1, SDATE


call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse         1        0 . 00         0 . 00            0            0            0             0 
Execute       1        0 . 00         0 . 00            0            0            0             0 
Fetch    1000001        8 . 23        24 . 23         7901         3841           34       1000000 
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total    1000003        8 . 23        24 . 24         7901         3841           34       1000000 

7901 reads from disk

Код: plaintext
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
select count(*) from (
select f0, f1, min(sdate) as sdate, max(edate) as edate
  from
  ( select f0, f1, sdate, edate, sum(start_of_group) over (partition by f0, f1 order by sdate) as group_no
      from
      ( select f0, f1, sdate, edate,
            decode(lag(edate) over (partition by f0, f1 order by sdate), sdate,  0 ,  1 ) as start_of_group
          from tst3
      )
  )
  group by f0, f1, group_no
  order by  3 
)

call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse         1        0 . 01         0 . 01            0            0            0             0 
Execute       1        0 . 00         0 . 00            0            0            0             0 
Fetch         2       16 . 67        77 . 30        36463         3841           94             1 
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total         4       16 . 68        77 . 31        36463         3841           94             1 

36463 reads from disk - you are swapping to temp like mad in this test

see above - in my tests the disk was zero.

analytics grabs more memory for hashing/sorting - and you don't have enough.

Definitley it is not for laptops with 512 megs of ram :-)
...
Рейтинг: 0 / 0
Периоды, объединение
    #33610441
Фотография SY
Скрыть профиль Поместить в игнор-лист Сообщения автора в теме
Участник
Q u a d r o36463 reads from disk - you are swapping to temp like mad in this test.
Definitley it is not for laptops with 512 megs of ram :-)

Absolutely. And it is, IMHO, good since if reflects (don't know how accurate though) scalability issue. In this case table has 1,000,000 rows while real one will have 60 times more. So even if just 1GB of memory takes care of 1,000,000 rows 60 million rows would need a bit more than that :).

Q u a d r osee above - in my tests the disk was zero.

Which tells me your test was not 100% cosher - you should have at least as many "disk" as it takes to read the table itself. Most likely you executed SQL more than once, so table data was already cached.

SY.
...
Рейтинг: 0 / 0
Периоды, объединение
    #33610799
Q u a d r o
Скрыть профиль Поместить в игнор-лист Сообщения автора в теме
Участник
SY good since if reflects (don't know how accurate though) scalability issue

I think one would better say: analytics is a tradeoff - they trade memory for speed

SY you should have at least as many "disk" as it takes to read the table itself

no i should not - given i had buffer cache large enought

you do realize that

Код: plaintext
1.
2.
3.
4.
5.
6.
SQL> insert into tst3
   2    select mod(level,  1000 ),
   3     mod(level,  100 ),
   4     trunc(sysdate) - level, trunc(sysdate) - level +  1 
   5    from dual
   6    connect by level <=  1000000 ;

is a conventional insert? It will already cache blocks for us.

consider:

Код: plaintext
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
SQL> truncate table tst3;

Table truncated.

SQL> insert into tst3
   2    select mod(level,  1000 ),
   3     mod(level,  100 ),
   4     trunc(sysdate) - level, trunc(sysdate) - level +  1 
   5    from dual
   6    connect by level <=  1000000 ;

 1000000  rows created.

SQL> commit;

Commit complete.

SQL> alter session set sql_trace=true;

Session altered.

SQL> select count(*) from tst3;

  COUNT(*)
----------
    1000000 

and the tkprof

Код: plaintext
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
select count(*) 
from
 tst3


call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse         1        0 . 00         0 . 00            0            1            0             0 
Execute       1        0 . 00         0 . 00            0            0            0             0 
Fetch         2        0 . 14         0 . 14            0         1912            0             1 
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total         4        0 . 15         0 . 14            0         1913            0             1 


zero PIO - for you see. Data was already cached for us by insert.


btw - in 10GR2 you don't need to bounce as there is alter system flush buffer_cache .
...
Рейтинг: 0 / 0
Периоды, объединение
    #33612634
Фотография SY
Скрыть профиль Поместить в игнор-лист Сообщения автора в теме
Участник
Q u a d r ono i should not - given i had buffer cache large enought

you do realize that
It will already cache blocks for us.

I do realize it. And that was exactly what I talking about. In real test there will not be a preceding insert, select or any other case that caches table. And taking into consideration 60,000,000 rows and multiuser system I would not count on 0 "disk".

SY.
...
Рейтинг: 0 / 0
Периоды, объединение
    #34042286
4uko
Скрыть профиль Поместить в игнор-лист Сообщения автора в теме
Гость
Jannny КалинаИсходная табличка - 60 млн
А расскажите потом, что она скажет на группировку? Мы в свое время не добились, чтобы группировка такого же объема проходила. ;(

Мне тоже интересно. Кто нибудь ответит ?
...
Рейтинг: 0 / 0
25 сообщений из 37, страница 1 из 2
Форумы / Oracle [игнор отключен] [закрыт для гостей] / Периоды, объединение
Целевая тема:
Создать новую тему:
Автор:
Закрыть
Цитировать
Найденые пользователи ...
Разблокировать пользователей ...
Читали форум (0):
Пользователи онлайн (0):
x
x
Закрыть


Просмотр
0 / 0
Close
Debug Console [Select Text]