Is your System Management Facility (SMF) a collection of valuable and useful information or just a bunch of irrelevant data waiting to expire? Can users access the valuable data they need, or are they bogged down by too much useless information?
SMF data is voluminous. It is critical to tuning and performance efforts, data security reconstruction and event reporting, resource consumption and chargeback billing, and much more. But SMF data can also be redundant or long outdated.
This article proposes specific actions to safely reduce the clutter in your SMF datasets, transforming your collection of SMF data from trash heap to valuable system resource. Actions are grouped into three levels of control: system level, data security package level, and program product level. Before making permanent changes to any level, it is important to review existing control settings, determine user requirements, and test any proposed changes.
Figure 1. Flow of records from source to archival tape.
Figure 1 illustrates the concepts discussed throughout this article. MVS, your security system, and your programs issue "write to SMF" requests. Requests are filtered by SMFPRMxx controls in SYS1.PARMLIB: TYPE(nn) or NOTYPE(nn), REC(PERM) for SMF17 records, and the IEFU8x exits. If they pass these controls, records are written to the daily SYS1.MANx dataset. SMFDUMP then reads these records and filters them through the SMFDUMP exit, which examines records to determine if they should be kept. SMFDUMP parameters split records into various application datasets, or the bit bucket.
Jump to: MVS Level || Security Product || Product Level || Conclusion || Feedback Form
SMF controls at the system level have the final word on which SMF records to keep and which to discard. This level of control is similar to the Message Processing Facility (MPF), which was introduced a few years ago to reduce console message traffic in MVS. From the MPF, systems programmers and operations analysts learned that it was possible to substantially reduce their volume of console messages.
While reducing console message clutter is beneficial, suppressing vital informational messages can be catastrophic. MPF has SYSLOG to capture erroneously suppressed messages, but SMF has no such provision. If you suppress a record without making your own backup provisions, its data is lost and cannot be recovered.
To avoid this situation, prepare for SMF record reduction by reviewing what controls are in place, which records are being used, and what is being done with those records. Specifically:
The first step is to ascertain which records are actually used, since it is counterproductive to suppress these records, and could even cost you revenue. For example, you don't want to suppress SYSOUT records if your chargeback system bills users for SYSOUT activity. Other critical users of SMF data include DASD management, ICF catalog recovery, scheduling packages, and data security. Meet with the responsible parties in each area to ensure that their required records are preserved. Also review documentation for the SMF requirements of products, such as scheduling and automated operations packages.
Next, review the options that control recording and suppressing of SMF records in the SMFPRMxx member (see Figure 2). SMF parameters tell you what records can be written, especially when dealing with non-IBM SMF records. This member also defines active SMF exits, three of which are record filters: IEFU83, IEFU84, IEFU85. For a complete set of currently active SMF parameters, issue the D SMF,O command for a hardcopy listing of the current controls to SYSLOG.
NOPROMPT (or PROMPT(LIST))
SYS(NOTYPE(16:19,32,63) EXITS(IEFU83,IEFU84))
REC(PERM)
/* 06/13/95 MSH - make a change */
Figure 2 - Sample SMFPRMxx member
SMFPRMxx also controls the operator's or systems programmer's ability to dynamically modify SMF parameters from the console. The normal recommended setting is NOPROMPT, which rejects the operator commands SETSMF and SET SMF=xx. Since NOPROMPT requires an IPL to effect changes, consider changing the setting to PROMPT(LIST) for the duration of your testing. PROMPT(LIST) and PROMPT(ALL) allow operator intervention using the SETSMF commands.
To counter any objections by auditors for loosening the PROMPT controls, report on all uses of the SETSMF commands using SMF90 records. In extreme cases, activate command limiting provided by IBM's RACF. Remember to specify NOPROMPT within SMFPRMxx after testing is completed to prevent the operator or systems programmer from making changes to the SMF environment without an IPL. Syntax problems override this specification and enable the operator to correct errors at IPL.
Finally, review a few current listings from SMFDUMP, program name IFASMFDP, to see exactly what SMF records are being written. Since IPLs are not usually a daily event, not all records are written daily. This output can help you determine what records are actually being written, as well as answer a number of other questions. From what applications do your installation SMF records come? Are those records being used or are they just taking up space?
Once you have reviewed your existing controls, you can begin testing changes to determine what records to suppress. Use the SMF comment facility to record changes within the SMFPRMxx member. Bracket your comments with /* and */ as shown in Figure 2. Include the date, who made the change, and why in your comments.
In addition to recording changes, it is necessary to build your own "safety net" to ensure that your testing does not adversely affect required SMF data. This is accomplished by modifying the SMFDUMP parameters to produce a temporary SMF archival tape that is an exact copy of the production archival tape. Then, if your new test tapes are missing records, they can be recovered from the temporary tape. With this arrangement, an oversight is merely inconvenient, not disastrous.
Testing enables you to determine what records can be safely suppressed. The most efficient means of suppressing SMF records is by updating the SMFPRMxx parameter members of SYS1.PARMLIB. To suppress all occurrences of a particular SMF record, include its record number when setting the NOTYPE(nn) parameter.
As a starting point, write all IBM records (SMF00-127) prior to determining which ones to suppress. IBM products, such as DB2, assume that you write their SMF records; suppressing them makes the product difficult to optimize or troubleshoot. Furthermore, there is no need to suppress IBM records if their system component is not installed and the record generator is absent. For example, without installing and running DB2, suppressing its SMF100 record offers no relief because specific SMF records are not being written. If you do suppress the record and then install DB2, you must restore the setting to write SMF100 records or face complications during installation and testing.
Suppressing redundancies - In SMF recording, redundancies occur when records repeat information that is already carried in other records. This varies with your MVS release level. For example, SMF30 records repeat as well as expand upon the information carried in SMF20 (Job Initiation), SMF05 (Step Termination), and SMF04 (Job Termination).
To find redundancies for suppression, examine the output from IFASMFDP, as shown in Figure 3. This count of records read and records written provides statistical information about your SMF recording. In this case, there are over 37,000 (4.3%) SMF04, 05, and 20 records that have their data repeated in 41,000 (4.8%) SMF30 records. Before suppressing these redundant records, the chargeback system or any other application referencing them must be using SMF30. Otherwise, chargeback revenue may drop.
While looking for redundancies, the following SMF records may appear to repeat what is being recorded by your security system logging facility: SMF14 (Open Dataset for Input), SMF15 (Open Dataset for Output), SMF17 (Delete Dataset), SMF18 (Rename Dataset), SMF62 (VSAM Component Opened), SMF63 (VSAM Catalog Entry Defined), SMF64 (VSAM Component Status), SMF67 (VSAM Catalog Entry Delete), SMF68 (VSAM Catalog Entry Renamed), and SMF69 (VSAM Data Space Defined, Extended, or Deleted).
Record Count of Percent of
No. Records Total
4 23,326 2.68 % <<== Job accounting
5 6,317 .73 % <<== Job accounting
14 77,696 8.94 %
15 78,665 9.05 %
17 11,997 1.38 %
18 10 .00 %
...
20 8,144 .94 % <<== Job accounting
...
30 41,481 4.77 % <<== NEW job accounting
...
60 52,194 6.00 %
61 13,727 1.58 %
62 18,330 2.11 %
64 37,772 4.34 %
65 11,278 1.30 %
66 2,762 .32 %
70 92 .01 % <<== tuning records
71 92 .01 %
72 4,600 .53 %
73 92 .01 %
74 460 .05 %
75 460 .05 %
77 92 .01 %
78 184 .02 % <<== tuning records
80 323,068 37.16 % <<== RACF records
...
240 3,249 .37 % <<== installation record
Figure 3 - Partial output from SMFDUMP
As they say, appearances may be deceiving. Where security records events requested on a user or dataset basis, these SMF records record all actions, regardless of security definitions. Because they may hold information that is useful for dataset audits, suppressing them is not recommended.
However, you may suppress SMF17 records just for temporary datasets using the REC(PERM) parameter. Temporary datasets are not critical resources in most environments. They exist for a single job step and are not catalogued. Suppressing their SMF17 records can result in tremendous savings in SMF data.
Finally, under no circumstances is the suppression of the following IBM SMF records recommended: SMF00 (IPL), SMF07 (SMF Data Lost), SMF30 (Common Address Space Work), SMF70-SMF79 (Resource Management Facility or RMF), and SMF90 (System Environment). These records relate directly to critical portions of the MVS environment and have no redundant copies.
When suppressing every occurrence of an SMF record is not your goal, you can selectively suppress occurrences with a record or dump exit. A record exit intercepts the record before it is written to the SYS1.MANx DASD datasets. A dump exit intercepts the record before it is archived.
Record exits - Each record is presented to a record exit to determine if it should be written to an SMF dataset. Record exits include IEFU83 (Normal Entry), IEFU84 (Branch Entry), and IEFU85 (Cross Memory Entry). Under MVS 5.1 or greater, these exits are updated without an IPL using the dynamic exit support PROGxx in SYS1.PARMLIB. To review the status of an exit, issue the operator command D PROG,SMF.
Record exits enable the suppression of records before they are written to your SYS1.MANx datasets. This reduces the load on your I/O subsystem and saves DASD space. Record exits must be very well-coded; any abnormal ends (ABENDs) can affect your entire MVS system image. Because they gain control before the record is written, over-zealous exits can damage or even destroy an audit trail. Exits must also be very fast since they are driven for every record written to SMF, which can be millions of times per day.
Dump exits - An alternative to coding IEFU8x record exits is writing an SMFDUMP exit, which examines appropriate fields within selected records to determine if writing the record to your SMF archive is necessary (see the flow of records from IEFU8x through the SMFDUMP exit in Figure 1). Although there is no reduced load on your I/O subsystem, the SMFDUMP exit is a safer alternative because the record is already written to the SMF dataset, ABENDs are isolated to the SMFDUMP program, and the response time consideration is much less severe. An additional benefit is that a backup copy of the record exists to recover erroneously suppressed records.
In addition to reducing the amount of data recorded to SMF, there is a side benefit to using SMFDUMP. It accelerates your processes by splitting archive files into pieces. Splitting off selected records reduces the need for your security, tuning, and performance reports and your chargeback system to read an entire SMF file when they need to read only some SMF records. A typical scenario may find your daily security reports reading 500,000 records when it really needs to be reading only 15,000.
Splitting records is illustrated in the SMFDUMP processing of Figure 1. The archival tape holds all records, the security file contains the records needed by security reporting, and the system tuner has a file with RMF-required records. This set-up requires only one pass of the complete SMF file, instead of three or more. Note that the archival tape does not need to record every record; it is possible to consign unused records to the bit bucket.
One of the largest (ab)users of SMF recording is your security system. With hundreds or even tens of thousands of security records written per day, meaningful security analysis and review is impossible without reducing the volume of data being reported.
Each security system reserves a single SMF record number. As mentioned under MVS level controls, to suppress all occurrences of an SMF record, include its record number in the NOTYPE(nn) parameter. Therefore, do not specify the security system SMF record number when setting this parameter or all security event logging will be suppressed.
Security uses SMF records for three types of events:
There are two possible outcomes to each of these events: success, which produces logging records, or failure, which produces violation records.
Data security offers many opportunities to impact logging -- most of them negative. This is because users control logging of resource or dataset access, and many of them decide to log all accesses, "just in case." This needlessly clutters the SMF dataset and security reports.
To avoid this clutter, exercise caution when setting audit definitions. Although every access failure needs recording and follow-up, every authorized access does not. In decentralized security environments, train users or audit their definitions to record only relevant, successful accesses. For example, while it is not necessary to record all successful READ accesses of SYS1.PARMLIB, all successful WRITE accesses should be journalled. This is accomplished in RACF by setting AUDIT(SUCCESS(UPDATE)). In ACF2, specify R(A) W(L).
In some cases, you will need to use AUDIT(SUCCESS(READ)) in RACF or R(L) settings in ACF2 to record READ accesses. For example, sensitive files, like the payroll master file or the digitized signature of the CEO, should have all accesses logged and reviewed since their misuse could have a very negative impact.
In addition to logging irrelevant accesses, another abuse of SMF in security is the needless tracing of users. Though it may be necessary to briefly track all accesses by a user, turn off tracing when it is no longer needed. The largest set of information seen in a user trace is the accessing of his or her own datasets -- hardly a security risk.
Finally, most security packages offer a "don't track accesses" attribute: TRUSTED in RACF, MAINT in ACF2. Before suppressing their logging, review users' existing records and leave SMF14, SMF15, and SMF6x turned on for recovery backup copies of logging records.
Your installation's DASD management package is one example that can benefit from the TRUSTED or MAINT attribute. As part of its normal function, the DASD manager reads innumerable datasets specified by the installation. Logging these reads on a daily basis only serves to clutter SMF. RACF OPERATIONS and ACF2 READALL attributes add to the clutter by logging datasets not explicitly authorized for DASD manager. This makes your DASD manager a perfect candidate for TRUSTED or MAINT. Besides, it is very likely your DASD manager records its actions in a log file of its own.
Scheduling packages are not recommended candidates for TRUSTED or MAINT. These packages update file content, where DASD management moves data without changing the content of files. SMF logging for scheduling packages can be reduced by writing proper authorizations for the needed datasets.
Numerous products may use SMF to record their actions and provide other information to the installation. This is a benefit of the product if the data is used. But if the data is never reviewed, it is just more clutter that can be cleared from your SMF archive.
As a starting point at the product level, suppress all user records (SMF128-255). This ensures that forgotten or unnecessary records do not clutter SMF. Turn records back on only as they are determined to be necessary.
Some products allow the installation to select which events are written to SMF. Carefully review definitions to ensure that an SMF record is written anytime the product changes the operating system environment or other critical resource.
Redundancy may occur if a product calls for authorization checking from external security, and both the product and security write to SMF.
What about products that write their own SMF records for reasons other than security? Some products log user sign-on, functions executed and resource consumption by users, and other statistics. Are their SMF records being used, or are they simply written to DASD and then copied to tape? If there is no use for the information, suppress the records. If the product issues a console message protesting the suppression of its records (some products will squawk: UNABLE TO WRITE TO SMF each time an attempt is made), record the records to DASD and use SMFDUMP to suppress the records from the archival tape.
SMF can be likened to an "all-you-can-eat buffet." Take all you want, but eat all you take. In other words, record all the SMF data you need, but if it goes unused, suppress it or flush it from the archival tapes. Hopefully, these suggestions will help ensure that SMF is more than a collection of information waiting to expire, but a valuable tool that benefits your installation.