Sortering fra A-Z. Henrik Dorf Chefkonsulent SAS Institute

Transkript

1 Sortering fra A-Z Henrik Dorf Chefkonsulent SAS Institute

2 Hvorfor ikke sortering fra A-Å? Det er for svært

3 Hvorfor ikke sortering fra A-Å?

4 Hvorfor ikke sortering fra A-Å?

5 Hvorfor ikke sortering fra A-Å?

6 Hvorfor ikke sortering fra A-Å?

7 Derfor ikke A-Å Og ikke meget om NLS

8 Sortere hvorfor sortere? Fordi orden er godt Og effektivt Et lille korttrick:

9

10

11

12

13 Hurtigere adgang til data Fordi data ligger samlet efter Dato Afdelinger Varegrupper. Så er det hurtigere at læse data fra disk, fordi man ikke skal så mange forskellige steder hen. ( Ikke IKEA-lager)

14 Hvad sker der, når data skal sorteres? Når data sorteres, ordnes det i et arbejdsareal i hukommelsen på computeren

15

16

17

18

19 Hvad sker der? Ved større datamængder bliver arbejdsarealet for lille Derfor oprettes et arbejdsområde på disk: Utility-filer

20

21

22

23

24

25

26

27

28

29

30 Hvad sker der? Utility-filer er smarte, men hvad betyder de for performance? Mere disk I/O: Tager mere tid, men kan jeg vise det?

31 SAS-session og arbejdsområder D:\WORK\_TD7160_KOHDOW72_

32 SAS jobs kan ikke måle sig selv Local Work

33 Workset Monitor En måleapp som holder øje med udvalgte ressourcer Startes fra jobbet Afslutter og leverer statistik SAS kan levere analysen naturligvis

34 Workset Monitor SAS Work SAS utility folder Input sas table Output SAS table

35 Workset Monitor Current size (bytes) Max size (bytes) Stat output CSV

36 Workset Monitor - output

37 Analysen

38 Eksempel på måling

39

40 Bedre anvendelse af hukommelse

41

42

43

44 Elaps-tider

45 Memory skal anvendes Computer memory (16-96GB) 16 GB

46 SAS options proc options group=memory ; run ; Group=MEMORY SORTSIZE= SUMSIZE=0 MAXMEMQUERY=0 MEMBLKSZ= MEMMAXSZ= LOADMEMSIZE=0 MEMSIZE= REALMEMSIZE=0 Specifies the amount of memory that is available to the SORT procedure. Specifies a limit on the amount of memory that is available for data summarization procedures, when class variables are active. Specifies the maximum amount of memory that is allocated for procedures. Specifies the memory block size for Windows memory-based libraries. Specifies the maximum amount of memory to allocate for using memory-based libraries. Specifies a suggested amount of memory that is needed for executable programs loaded by SAS. Specifies the limit on the amount of virtual memory that can be used during a SAS session. Specifies the amount of real memory SAS can expect to allocate.

47 Memory Computer memory Memsize=1G 16 GB Memsize=2G Memsize=0

48 Memory Computer memory Sortering memory 16 GB Memsize=2G Sortsize=1G

49 Memory Computer memory Sortering memory Memsize=6G Sortsize=4G 16 GB

50 Hvad sker der? MEMSIZE SORTSIZE SAS does not automatically reserve or allocate the amount of memory that you specify in the MEMSIZE system option. SAS uses only as much memory as it needs to complete a process. For example, a DATA step might require only 20M of memory, so even though MEMSIZE is set to 500M, SAS uses only 20M of memory. The SORTSIZE system option can reduce the amount of swapping SAS must do to sort the data set. If PROC SORT needs more memory than you specify, it creates a temporary utility file in your SAS work directory in which to store the data. The SORT procedure's algorithm can swap unneeded data more efficiently than Windows can.

51 Memory - deles mellem mange jobs Computer memory Sortering memory Memsize=6G Sortsize=5G 16 GB

52 Memory options Memsize 4G Sortsize 3G skal sættes før SAS start kan sættes undervejs i jobbet

53

54 Mulige alternativer til meget memory Proc SQL order by : Proc sort data=a out=b TAGSORT; By.. Proc sort via index

55 SQL Order by SQL 25 Sekunder

56 Alternativer Tagsort spar på pladsen i utility-området

57

58

59

60

61

62

63 Index sort Hvis en tabel er indekseret på den (ene) variabel, der skal sorteres, kan indekset anvendes til at læse data i sorteret rækkefølge, uden at udføre en egentlig sortering.

64

65

66

67

68

69 Sort via Index - grundtabellen med Index 3739 data a(index=(b)); 3740 length c $100; 3741 do i=1 to ; 3742 b=ranuni(0); 3743 c=repeat("a",100); 3744 output; 3745 end; 3746 run; NOTE: The data set WORK.A has observations and 3 variables. INFO: Multiple concurrent threads will be used to create the index. NOTE: Simple index B has been defined. NOTE: DATA statement used (total process time): real time 4.48 seconds user cpu time 6.02 seconds system cpu time 1.17 seconds memory k OS Memory k Timestamp 25/09/ f.m.

70 By clause via Index 3758Options msglevel=i; 3759DATA c ; 3760 SET a ; 3761 BY b ; 3762 RUN; INFO: Index b selected for BY clause processing. NOTE: There were observations read from the data set WORK.A. NOTE: The data set WORK.C has observations and 3 variables. NOTE: DATA statement used (total process time): real time 4:38.62 user cpu time system cpu time 3:18.77 memory k seconds OS Memory k Timestamp 25/09/ e.m.

71 Sorteret tabel 3747proc sort data=a out=b ; 3748 by b ; 3749 run; NOTE: There were observations read from the data set WORK.A. NOTE: SAS threaded sort was used. NOTE: The data set WORK.B has observations and 3 variables. NOTE: PROCEDURE SORT used (total process time): real time 7.73 seconds user cpu time 6.49 seconds system cpu time 1.84 seconds memory k OS Memory k Timestamp 25/09/ f.m.

72 Sortering og standard by 3759Data c ; 3760 set b ; 3761 by b; 3762 run ; NOTE: There were observations read from the data set WORK.B. NOTE: The data set WORK.C has observations and 3 variables. NOTE: DATA statement used (total process time): real time seconds user cpu time 1.15 seconds system cpu time 1.49 seconds memory k OS Memory k Timestamp 25/09/ e.m.

73 Resultat INFO: Index b selected for BY clause processing 00:04:38.62 Proc sort + dataudtræk 00:00: :00:11.40 ========== 00:00:19.13

74 Sortering datasæt metadata-markering Sorted by sættes af SAS ved sortering Proc sort data=master ; by kundenummer dato ; run; Sorted by kan sættes af programmet Data master(sortedby=(kundenummer dato) ); Merge master transactions ; By kundenummer dato ; Run;

75 Usorterede data Proc contents data=a ; run; Data Set Name WORK.A Observations Member Type DATA Variables 3 Engine V9 Indexes 1 Created 25/09/ :04:00 Observation Length 120 Last Modified 25/09/ :04:00 Deleted Observations 0 Protection Compressed NO Data Set Type Sorted NO Label Data Representation Encoding WINDOWS_64 wlatin1 Western (Windows)

76 Validated sort proc sort data=a out=b(index=(b)) ; by b ; run; Proc contents data=b ; run; Data Set Name WORK.B Observations Member Type DATA Variables 3 Engine V9 Indexes 0 Created 25/09/ :37:57 Observation Length 120 Last Modified 25/09/ :37:57 Deleted Observations 0 Protection Compressed NO Data Set Type Sorted YES Label Data Representation WINDOWS_64 Encoding wlatin1 Western (Windows) Sort Information Sortedby Validated Character Set b YES ANSI

77 Unvalidated sort Data Set Name WORK.D Observations Member Type DATA Variables 3 Engine V9 Indexes 0 Created 25/09/ :46:23 Observation Length 120 Last Modified 25/09/ :46:23 Deleted Observations 0 Protection Compressed NO Data Set Type Sorted YES Label Data D(sortedby=b); SET B ; BY B ; RUN; Proc contents data=d ; run ; Data Representation Encoding WINDOWS_64 wlatin1 Western (Windows) Sort Information Sortedby Validated Character Set b NO ANSI

78 Kontrol af en ikke-valideret sortedby= Validering af sorteringen: Data _null_ ; Set master ; By kundenummer dato ; Run; options sortvalidate; Proc sort data=master ; By kundenummer dato ; Run; Options nosortvalidate; NOTE: There were observations read from the data set WORK.A. NOTE: DATA statement used (Total process time): ERROR: BY variables are not properly sorted on data set WORK.A. a=1 b= FIRST.b=1 LAST.b=1 _ERROR_=1 _N_=1 NOTE: The SAS System stopped processing this step because of errors. NOTE: There were 2 observations read from the data set WORK.A. NOTE: Sort order of input data set has been verified. NOTE: There were observations read from the data set WORK.A. NOTE: Input data set is already sorted, no sorting done. NOTE: PROCEDURE SORT used (Total process time): NOTE: Input data set is not in sorted order. NOTE: There were observations read from the data set WORK.A. NOTE: SAS sort was used. NOTE: The data set WORK.A has observations and 2 variables. NOTE: PROCEDURE SORT used (total process time): *) Sorteringen er nu valideret og sortedby markeringen ændret

79 Sorteringer Er gode at have Kræver omsorg og omtanke Nogle problemer løses nemmest ved sorteringer Bedste sted at overveje performance forbedringer Minimer antal sorteringer Undgå sortering og gensortering Brug hukommelsen Undgå INFO: Index b selected for BY clause processing

80 Hvorfor ikke sortering fra A-Å?

81 proc sort data=navne out=navne1 ; by navn; run ; Navn a.txt aa.txt b.txt bb.txt c.txt cc.txt Æ.txt å.txt æ.txt ø.txt proc sort data=navne out=navne2 danish ; by navn; run ; Navn a.txt aa.txt b.txt bb.txt c.txt cc.txt æ.txt Æ.txt ø.txt å.txt proc sort data=navne out=navne3 sortseq=linguistic ; by navn; run ; Navn a.txt b.txt bb.txt c.txt cc.txt Æ.txt æ.txt ø.txt å.txt aa.txt

82 Spørgsmål? A B C D E F H G I J K L M N O P Q R S T U V W X Y Z