Differences

This shows you the differences between two versions of the page.

Link to this comparison view

en:services:storage_services:backup:tsm:anleitungen:inclexcl [2015/10/27 15:24] – created bnachtwen:services:storage_services:backup:tsm:anleitungen:inclexcl [2024/09/09 10:57] (current) – [Notice] jbruene
Line 1: Line 1:
 +====== Using INCLude/EXCLude Rules======
  
 +===== Notice=====
 +
 +For creating Include and Exclude rules the [[https://www.ibm.com/docs/en/storage-protect/8.1.23?topic=list-include-exclude-options|TSM/ISP documentation]] provides excessive examples. Therefore, only the most important statements and options are summarized:
 +
 +**NOTICE:**
 +These examples are for Unices, but the statements are somehow identical on Windows. (''/'' => ''\'')
 +The configuration therefore will be in "dsm.opt" instead of "dsm.sys".
 +
 +<WRAP center round important 90%>
 +The configuration of Include, but especially **''EXCLUDE''** rules requires the highest attention as wrongly placed or wrong defined rules will lead to files excluded from backup! Subsequent changes may discard already existing backups!
 +
 +**//The GWDG//** therefore can only give suggestions and recommendations for creating rules, **//but is not taking any responsibility for the correctness of examples//**!
 +
 +Please verify your chosen rules before use, either by a backup with the option //preview// in the GUI or calling //dsmc backup preview//!
 +</WRAP>
 +
 +===== Using Wildcards and Special Chrarcters =====
 +
 +TSM allows the usage of Wildcards in In- and Exclude rules:
 +  * ''?'' representing exactly one random character
 +  * ''*'' representing any number of random characters
 +
 +  * ''/.../'' representing any number of directory levels
 +<code>/home/.../public</code>
 +Representing all directories ''public'' within ''/home'', independent of the directory level
 +<code>/srv/.../*.zip</code>
 +Is representing all ''.zip'' files within ''/srv''.
 +
 +  * Groups of characters can be summed as a regular expression : ''[<list of characters>]''. You can use a dash to enter ranges. Example: <code>exclude /Volumes/Vol[1-4]/.../*.cpp</code> is selecting all files with the extension ''.cpp'' from the directories ''Vol1'',''Vol2'', ''Vol3'' and ''Vol4'' within ''/Volumes''. <code>exclude /home/[a-c,A-C]*/.../*.cpp</code> excludes all files from home directory starting with an ''a'',''b'',''c'',''A'',''B'' or ''C''.
 +
 +
 +If a file or directory name includes special characters, they have to be masked 
 +  * Usually whitespaces can be captured in quotation marks:<code>"My Files"</code>
 +  * The special characters <code> * ? : [ ]</code> have to be masked by a <code> [\<Symbol]</code> . To exclude the file <code>Text[2].txt</code> following rule needs to be set:<code>exclude /.../Text[\[]2[\]]</code>.
 +  * To Exclude the directory <code>/usr/src/classes/project::tmp</code> following rule needs to be set:<code>exclude.dir  /usr/src/classes/project[\:][\:]tmp</code>
 +
 +===== Outsource Include/Exclude Rules =====
 +
 +In- and exclude rules can get extensiv fast and therefore will substantially affect the configuration readability.
 +In this state or if you want to group rules you may find it helpful to outsource these rules in another or multiple other files.
 +These files can be imported into the "dsm.sys" using the parameter ''INCLExcl <Filename>''.
 +Example:
 +  INCLExcl  inclexcl.user
 +  INCLExcl  inclexcl.linux
 +  INCLExcl  inclexcl.gwdg
 +
 +
 +<WRAP center round important 90%>
 +Using ''INCLExcl'' to import a non existing file an error will occur. Please validate "dsm.sys" by testing.
 +</WRAP>
 +
 +=====  Hierarchy of InExClude =====
 + 
 +  * Generally the TSM client works through the ''dsm.sys'' file **last to first**.
 +  * The **rules which match first will be applied**, rules matching later may won't affect (See Example1).
 +<WRAP center round tip 90%>
 +=== Example1: ===
 +
 +  include /Users/user01/Documents/includefile.cpp
 +  exclude /Users/user01/Documents/.../*
 +=> The file ''includefile.cpp'' **will not be backed up** caused by the previous exclusion of the directory ''/Users/user01/Documents''.
 +
 +=== Example2: ===
 +
 +  exclude /Users/user01/Documents/.../*
 +  include /Users/user01/Documents/includefile.cpp
 +=> The file ''includefile.cpp'' **will be backed up**, since it was explicitly included by using ''include'' before the exclusion of the entire directory ''/Users/user01/Documents''.
 +</WRAP>
 +
 +
 +  * Rules which exclude entire //filespaces// (''exclude**.fs**'') or //directories// (''exclude**.dir**'') will be applied before all other rules.
 +  * Rules in a separate file will be applied at that time where the ''INEXClude <file>'' is placed in the ''dsm.sys''.
 +===== Management Classes =====
 +
 +The Management Class on the TSM server, which should be used for the backup can be managed also by the include rules:
 +  include <path or file | search pattern> <Management Class>
 +
 +This feature applies only in individual cases at the backup service by GWDG.
 +If so, we will contact you personally.
 +
 +===== Compression =====
 +
 +All data will be compressed due to sever side default settings which enable the data transfer compression.
 +Compressed data (e.g. ''.zip'',''.gz'' as well as ''.jpg'', etc.) will be tried to be compressed again.
 +This will most likely result in a high computational load rather than result in data reduction.
 +
 +To avoid this unnecessary load all compressed data should be excluded from compression.
 +The relevant rules should be applied to the end of the file ''dsm.sys''(Unices) or ''dsm.opt''(Windows).
 +(See [[en:services:storage_services:backup:tsm:inclexcl#hierarchy_of_inexclude|Hierarchie]] above)
 +
 +//Suggestion for exclusion of compressed files://
 +  exclude.compression     /.../*.z
 +  exclude.compression     /.../*.Z
 +  exclude.compression     /.../*.gz
 +  exclude.compression     /.../*.GZ
 +  exclude.compression     /.../*.bz
 +  exclude.compression     /.../*.BZ
 +  exclude.compression     /.../*.bz2
 +  exclude.compression     /.../*.BZ2
 +  exclude.compression     /.../*.tgz
 +  exclude.compression     /.../*.TGZ
 +  exclude.compression     /.../*.gif
 +  exclude.compression     /.../*.GIF
 +  exclude.compression     /.../*.jpg
 +  exclude.compression     /.../*.JPG
 +  exclude.compression     /.../*.mpg
 +  exclude.compression     /.../*.MPG
 +  exclude.compression     /.../*.mp4
 +  exclude.compression     /.../*.MP4
 +  exclude.compression     /.../*.wmv
 +  exclude.compression     /.../*.WMV
 +  exclude.compression     /.../*.mp3
 +  exclude.compression     /.../*.MP3
 +  exclude.compression     /.../*.aac
 +  exclude.compression     /.../*.AAC
 +  exclude.compression     /.../*.wma
 +  exclude.compression     /.../*.WMA
 +  exclude.compression     /.../*.zip
 +  exclude.compression     /.../*.ZIP
 +  exclude.compression     /.../*.rar
 +  exclude.compression     /.../*.RAR
 +  exclude.compression     /.../*.pdf
 +  exclude.compression     /.../*.PDF
 +  exclude.compression     /.../*.cdr
 +  exclude.compression     /.../*.CDR
 +  exclude.compression     /.../*.ppt
 +  exclude.compression     /.../*.PPT
 +  exclude.compression     /.../*.pst
 +  exclude.compression     /.../*.PST
 +  exclude.compression     /.../*.rpm
 +  exclude.compression     /.../*.deb
 + \\
 +===== Encryption =====
 +
 +TSM supports file encryption before sending the data to a server. Beside the indisputable advantages the encryption comes with some disadvantages as well:
 +  * The encryption is preventing every access to the data. Even from the TSM administrator.
 +  * There is no way to decrypt the data if the password is lost!
 +Some disadvantages can be neglected:
 +  * The compression rate is usually low.
 +  * Functions as //subfile backup// or //client side deduplictaion// and //server side deduplication// can't be used with encryption -- both are not offered by GWDG in context of the file back up anyway.
 +
 +Encryption can be used by defining rules for the files which should be encrypted. Therefore, you have to use the keyword //encrypt// with an include statement, e.g.:
 +
 +  include.encrypt /home/.../*
 +  exclude.encrypt /home/.../public/.../*
 +
 +\\
 +===== Excluding TSM files =====
 +
 +Backing up the TSM directories is in general not very useful.
 +Therefore, it could be excluded by ''exclude.dir'' rule.
 +Somehow, if the configuration files should be backed up, two approaches are possible:
 +  - The files could be saved at a separate path and only have a symlink / shortcut in the TSM client directory. The TSM directory is excluded as mentioned above.
 +  - Using wildcards instead of ''exclude.dir'' to exclude the TSM directory. The explicit include for the TSM configuration files must then be set:
 +
 +  exclude /opt/tivoli/tsm/.../*
 +  include /opt/tivoli/tsm/client/ba/bin/dsm.sys
 +  include /opt/tivoli/tsm/client/ba/bin/dsm.opt
 +  include /opt/tivoli/tsm/client/ba/bin/tsmjbbd.ini
 +\\
 +===== Server Side Include-Exclude Rules =====
 +
 +Server sided in- exclude rules are overriding the client side configuration.
 +The GWDG largely dispenses server sided rules 
 +since only the administrator can usefully define the files to be excluded or encrypted.
 +
 +
 +At the present time (date of this text) only the management class //3COPIES// is specified to back up the //system state// (Windows only) to limit the amount of //system state// versions by the number of three.
 +Relevant only for //Bare Metal// restores, the //system state// uses an excessive amount of storage.  
 +The system restore points in //Windows// providing an easier way for restoring older system states on running systems.