====== Using INCLude/EXCLude Rules====== ===== Notice===== For creating Include and Exclude rules the [[https://www.ibm.com/docs/en/storage-protect/8.1.23?topic=list-include-exclude-options|TSM/ISP documentation]] provides excessive examples. Therefore, only the most important statements and options are summarized: **NOTICE:** These examples are for Unices, but the statements are somehow identical on Windows. (''/'' => ''\'') The configuration therefore will be in "dsm.opt" instead of "dsm.sys". The configuration of Include, but especially **''EXCLUDE''** rules requires the highest attention as wrongly placed or wrong defined rules will lead to files excluded from backup! Subsequent changes may discard already existing backups! **//The GWDG//** therefore can only give suggestions and recommendations for creating rules, **//but is not taking any responsibility for the correctness of examples//**! Please verify your chosen rules before use, either by a backup with the option //preview// in the GUI or calling //dsmc backup preview//! ===== Using Wildcards and Special Chrarcters ===== TSM allows the usage of Wildcards in In- and Exclude rules: * ''?'' representing exactly one random character * ''*'' representing any number of random characters * ''/.../'' representing any number of directory levels /home/.../public Representing all directories ''public'' within ''/home'', independent of the directory level /srv/.../*.zip Is representing all ''.zip'' files within ''/srv''. * Groups of characters can be summed as a regular expression : ''[]''. You can use a dash to enter ranges. Example: exclude /Volumes/Vol[1-4]/.../*.cpp is selecting all files with the extension ''.cpp'' from the directories ''Vol1'',''Vol2'', ''Vol3'' and ''Vol4'' within ''/Volumes''. exclude /home/[a-c,A-C]*/.../*.cpp excludes all files from home directory starting with an ''a'',''b'',''c'',''A'',''B'' or ''C''. If a file or directory name includes special characters, they have to be masked * Usually whitespaces can be captured in quotation marks:"My Files" * The special characters * ? : [ ] have to be masked by a [\ . To exclude the file Text[2].txt following rule needs to be set:exclude /.../Text[\[]2[\]]. * To Exclude the directory /usr/src/classes/project::tmp following rule needs to be set:exclude.dir /usr/src/classes/project[\:][\:]tmp ===== Outsource Include/Exclude Rules ===== In- and exclude rules can get extensiv fast and therefore will substantially affect the configuration readability. In this state or if you want to group rules you may find it helpful to outsource these rules in another or multiple other files. These files can be imported into the "dsm.sys" using the parameter ''INCLExcl ''. Example: INCLExcl inclexcl.user INCLExcl inclexcl.linux INCLExcl inclexcl.gwdg Using ''INCLExcl'' to import a non existing file an error will occur. Please validate "dsm.sys" by testing. ===== Hierarchy of InExClude ===== * Generally the TSM client works through the ''dsm.sys'' file **last to first**. * The **rules which match first will be applied**, rules matching later may won't affect (See Example1). === Example1: === include /Users/user01/Documents/includefile.cpp exclude /Users/user01/Documents/.../* => The file ''includefile.cpp'' **will not be backed up** caused by the previous exclusion of the directory ''/Users/user01/Documents''. === Example2: === exclude /Users/user01/Documents/.../* include /Users/user01/Documents/includefile.cpp => The file ''includefile.cpp'' **will be backed up**, since it was explicitly included by using ''include'' before the exclusion of the entire directory ''/Users/user01/Documents''. * Rules which exclude entire //filespaces// (''exclude**.fs**'') or //directories// (''exclude**.dir**'') will be applied before all other rules. * Rules in a separate file will be applied at that time where the ''INEXClude '' is placed in the ''dsm.sys''. ===== Management Classes ===== The Management Class on the TSM server, which should be used for the backup can be managed also by the include rules: include This feature applies only in individual cases at the backup service by GWDG. If so, we will contact you personally. ===== Compression ===== All data will be compressed due to sever side default settings which enable the data transfer compression. Compressed data (e.g. ''.zip'',''.gz'' as well as ''.jpg'', etc.) will be tried to be compressed again. This will most likely result in a high computational load rather than result in data reduction. To avoid this unnecessary load all compressed data should be excluded from compression. The relevant rules should be applied to the end of the file ''dsm.sys''(Unices) or ''dsm.opt''(Windows). (See [[en:services:storage_services:backup:tsm:inclexcl#hierarchy_of_inexclude|Hierarchie]] above) //Suggestion for exclusion of compressed files:// exclude.compression /.../*.z exclude.compression /.../*.Z exclude.compression /.../*.gz exclude.compression /.../*.GZ exclude.compression /.../*.bz exclude.compression /.../*.BZ exclude.compression /.../*.bz2 exclude.compression /.../*.BZ2 exclude.compression /.../*.tgz exclude.compression /.../*.TGZ exclude.compression /.../*.gif exclude.compression /.../*.GIF exclude.compression /.../*.jpg exclude.compression /.../*.JPG exclude.compression /.../*.mpg exclude.compression /.../*.MPG exclude.compression /.../*.mp4 exclude.compression /.../*.MP4 exclude.compression /.../*.wmv exclude.compression /.../*.WMV exclude.compression /.../*.mp3 exclude.compression /.../*.MP3 exclude.compression /.../*.aac exclude.compression /.../*.AAC exclude.compression /.../*.wma exclude.compression /.../*.WMA exclude.compression /.../*.zip exclude.compression /.../*.ZIP exclude.compression /.../*.rar exclude.compression /.../*.RAR exclude.compression /.../*.pdf exclude.compression /.../*.PDF exclude.compression /.../*.cdr exclude.compression /.../*.CDR exclude.compression /.../*.ppt exclude.compression /.../*.PPT exclude.compression /.../*.pst exclude.compression /.../*.PST exclude.compression /.../*.rpm exclude.compression /.../*.deb \\ ===== Encryption ===== TSM supports file encryption before sending the data to a server. Beside the indisputable advantages the encryption comes with some disadvantages as well: * The encryption is preventing every access to the data. Even from the TSM administrator. * There is no way to decrypt the data if the password is lost! Some disadvantages can be neglected: * The compression rate is usually low. * Functions as //subfile backup// or //client side deduplictaion// and //server side deduplication// can't be used with encryption -- both are not offered by GWDG in context of the file back up anyway. Encryption can be used by defining rules for the files which should be encrypted. Therefore, you have to use the keyword //encrypt// with an include statement, e.g.: include.encrypt /home/.../* exclude.encrypt /home/.../public/.../* \\ ===== Excluding TSM files ===== Backing up the TSM directories is in general not very useful. Therefore, it could be excluded by ''exclude.dir'' rule. Somehow, if the configuration files should be backed up, two approaches are possible: - The files could be saved at a separate path and only have a symlink / shortcut in the TSM client directory. The TSM directory is excluded as mentioned above. - Using wildcards instead of ''exclude.dir'' to exclude the TSM directory. The explicit include for the TSM configuration files must then be set: exclude /opt/tivoli/tsm/.../* include /opt/tivoli/tsm/client/ba/bin/dsm.sys include /opt/tivoli/tsm/client/ba/bin/dsm.opt include /opt/tivoli/tsm/client/ba/bin/tsmjbbd.ini \\ ===== Server Side Include-Exclude Rules ===== Server sided in- exclude rules are overriding the client side configuration. The GWDG largely dispenses server sided rules since only the administrator can usefully define the files to be excluded or encrypted. At the present time (date of this text) only the management class //3COPIES// is specified to back up the //system state// (Windows only) to limit the amount of //system state// versions by the number of three. Relevant only for //Bare Metal// restores, the //system state// uses an excessive amount of storage. The system restore points in //Windows// providing an easier way for restoring older system states on running systems.