Edit File: lvm_fault_handling.txt
LVM device fault handling ========================= Introduction ------------ This document is to serve as the definitive source for information regarding the policies and procedures surrounding device failures in LVM. It codifies LVM's responses to device failures as well as the responsibilities of administrators. Device failures can be permanent or transient. A permanent failure is one where a device becomes inaccessible and will never be revived. A transient failure is a failure that can be recovered from (e.g. a power failure, intermittent network outage, block relocation, etc). The policies for handling both types of failures is described herein. Users need to be aware that there are two implementations of RAID1 in LVM. The first is defined by the "mirror" segment type. The second is defined by the "raid1" segment type. The characteristics of each of these are defined in lvm.conf under 'mirror_segtype_default' - the configuration setting used to identify the default RAID1 implementation used for LVM operations. Available Operations During a Device Failure -------------------------------------------- When there is a device failure, LVM behaves somewhat differently because only a subset of the available devices will be found for the particular volume group. The number of operations available to the administrator is diminished. It is not possible to create new logical volumes while PVs cannot be accessed, for example. Operations that create, convert, or resize logical volumes are disallowed, such as: - lvcreate - lvresize - lvreduce - lvextend - lvconvert (unless '--repair' is used) Operations that activate, deactivate, remove, report, or repair logical volumes are allowed, such as: - lvremove - vgremove (will remove all LVs, but not the VG until consistent) - pvs - vgs - lvs - lvchange -a [yn] - vgchange -a [yn] Operations specific to the handling of failed devices are allowed and are as follows: - 'vgreduce --removemissing <VG>': This action is designed to remove the reference of a failed device from the LVM metadata stored on the remaining devices. If there are (portions of) logical volumes on the failed devices, the ability of the operation to proceed will depend on the type of logical volumes found. If an image (i.e leg or side) of a mirror is located on the device, that image/leg of the mirror is eliminated along with the failed device. The result of such a mirror reduction could be a no-longer-redundant linear device. If a linear, stripe, or snapshot device is located on the failed device the command will not proceed without a '--force' option. The result of using the '--force' option is the entire removal and complete loss of the non-redundant logical volume. If an image or metadata area of a RAID logical volume is on the failed device, the sub-LV affected is replace with an error target device - appearing as <unknown> in 'lvs' output. RAID logical volumes cannot be completely repaired by vgreduce - 'lvconvert --repair' (listed below) must be used. Once this operation is complete on volume groups not containing RAID logical volumes, the volume group will again have a complete and consistent view of the devices it contains. Thus, all operations will be permitted - including creation, conversion, and resizing operations. It is currently the preferred method to call 'lvconvert --repair' on the individual logical volumes to repair them followed by 'vgreduce --removemissing' to extract the physical volume's representation in the volume group. - 'lvconvert --repair <VG/LV>': This action is designed specifically to operate on individual logical volumes. If, for example, a failed device happened to contain the images of four distinct mirrors, it would be necessary to run 'lvconvert --repair' on each of them. The ultimate result is to leave the faulty device in the volume group, but have no logical volumes referencing it. (This allows for 'vgreduce --removemissing' to removed the physical volumes cleanly.) In addition to removing mirror or RAID images that reside on failed devices, 'lvconvert --repair' can also replace the failed device if there are spare devices available in the volume group. The user is prompted whether to simply remove the failed portions of the mirror or to also allocate a replacement, if run from the command-line. Optionally, the '--use-policies' flag can be specified which will cause the operation not to prompt the user, but instead respect the policies outlined in the LVM configuration file - usually, /etc/lvm/lvm.conf. Once this operation is complete, the logical volumes will be consistent. However, the volume group will still be inconsistent - due to the refernced-but-missing device/PV - and operations will still be restricted to the aformentioned actions until either the device is restored or 'vgreduce --removemissing' is run. Device Revival (transient failures): ------------------------------------ During a device failure, the above section describes what limitations a user can expect. However, if the device returns after a period of time, what to expect will depend on what has happened during the time period when the device was failed. If no automated actions (described below) or user actions were necessary or performed, then no change in operations or logical volume layout will occur. However, if an automated action or one of the aforementioned repair commands was manually run, the returning device will be perceived as having stale LVM metadata. In this case, the user can expect to see a warning concerning inconsistent metadata. The metadata on the returning device will be automatically replaced with the latest copy of the LVM metadata - restoring consistency. Note, while most LVM commands will automatically update the metadata on a restored devices, the following possible exceptions exist: - pvs (when it does not read/update VG metadata) Automated Target Response to Failures: -------------------------------------- The only LVM target types (i.e. "personalities") that have an automated response to failures are the mirror and RAID logical volumes. The other target types (linear, stripe, snapshot, etc) will simply propagate the failure. [A snapshot becomes invalid if its underlying device fails, but the origin will remain valid - presuming the origin device has not failed.] Starting with the "mirror" segment type, there are three types of errors that a mirror can suffer - read, write, and resynchronization errors. Each is described in depth below. Mirror read failures: If a mirror is 'in-sync' (i.e. all images have been initialized and are identical), a read failure will only produce a warning. Data is simply pulled from one of the other images and the fault is recorded. Sometimes - like in the case of bad block relocation - read errors can be recovered from by the storage hardware. Therefore, it is up to the user to decide whether to reconfigure the mirror and remove the device that caused the error. Managing the composition of a mirror is done with 'lvconvert' and removing a device from a volume group can be done with 'vgreduce'. If a mirror is not 'in-sync', a read failure will produce an I/O error. This error will propagate all the way up to the applications above the logical volume (e.g. the file system). No automatic intervention will take place in this case either. It is up to the user to decide what can be done/salvaged in this senario. If the user is confident that the images of the mirror are the same (or they are willing to simply attempt to retreive whatever data they can), 'lvconvert' can be used to eliminate the failed image and proceed. Mirror resynchronization errors: A resynchronization error is one that occurs when trying to initialize all mirror images to be the same. It can happen due to a failure to read the primary image (the image considered to have the 'good' data), or due to a failure to write the secondary images. This type of failure only produces a warning, and it is up to the user to take action in this case. If the error is transient, the user can simply reactivate the mirrored logical volume to make another attempt at resynchronization. If attempts to finish resynchronization fail, 'lvconvert' can be used to remove the faulty device from the mirror. TODO... Some sort of response to this type of error could be automated. Since this document is the definitive source for how to handle device failures, the process should be defined here. If the process is defined but not implemented, it should be noted as such. One idea might be to make a single attempt to suspend/resume the mirror in an attempt to redo the sync operation that failed. On the other hand, if there is a permanent failure, it may simply be best to wait for the user or the automated response that is sure to follow from a write failure. ...TODO Mirror write failures: When a write error occurs on a mirror constituent device, an attempt to handle the failure is automatically made. This is done by calling 'lvconvert --repair --use-policies'. The policies implied by this command are set in the LVM configuration file. They are: - mirror_log_fault_policy: This defines what action should be taken if the device containing the log fails. The available options are "remove" and "allocate". Either of these options will cause the faulty log device to be removed from the mirror. The "allocate" policy will attempt the further action of trying to replace the failed disk log by using space that might be available in the volume group. If the allocation fails (or the "remove" policy is specified), the mirror log will be maintained in memory. Should the machine be rebooted or the logical volume deactivated, a complete resynchronization of the mirror will be necessary upon the follow activation - such is the nature of a mirror with a 'core' log. The default policy for handling log failures is "allocate". The service disruption incurred by replacing the failed log is negligible, while the benefits of having persistent log is pronounced. - mirror_image_fault_policy: This defines what action should be taken if a device containing an image fails. Again, the available options are "remove" and "allocate". Both of these options will cause the faulty image device to be removed - adjusting the logical volume accordingly. For example, if one image of a 2-way mirror fails, the mirror will be converted to a linear device. If one image of a 3-way mirror fails, the mirror will be converted to a 2-way mirror. The "allocate" policy takes the further action of trying to replace the failed image using space that is available in the volume group. Replacing a failed mirror image will incure the cost of resynchronizing - degrading the performance of the mirror. The default policy for handling an image failure is "remove". This allows the mirror to still function, but gives the administrator the choice of when to incure the extra performance costs of replacing the failed image. RAID logical volume device failures are handled differently from the "mirror" segment type. Discussion of this can be found in lvm2-raid.txt.