I have a multi-node or clustered environment, How can multiple sosreports be generated simultaneously? How can I collect a sosreport from each node?
Starting with CentOS/RHEL 7.6, the sos-collector package has been added to the base repository to facilitate easier collection of sosreports from multiple nodes. This allows system administrators to execute a single command that generates a single archive containing sosreports from the desired nodes.
On either RHEL 7 or RHEL 8, sos-colelctor can be installed with:
# yum install sos-collector
– On RHEL 7, ‘sos-collector’ is in the base repository.
– On RHEL 8, ‘sos-collector’ is in appstream.
Note: The sos-collector packages does not need to be installed on every node. It only needs to be installed on the node it is being run from.
sos-collector may be run either from a node in the environment you wish to collect from, or from a workstation. The only requirement for running is that the system that runs sos-collector has network connectivity to each node that a sosreport will be collected from.
By itself sos-collector does not need to be run as the root user, however, it will need to become the root user, or use sudo, on the remote nodes to generate the actual sosreport.
Running in the environment
Running sos-collector on a “master” node (a node capable of listing the other nodes in the cluster) is straight-forward as cluster-type detection is done automatically:
# sos-collector [...] Cluster type set to rhv The following is a list of nodes to collect from: rhev-manager.example.com rhev-hypervisor1.example.com rhev-hypervisor2.example.com
From here, sos-collector will open an SSH session to each remote node and run a sosreport command on each of them. The sosreport command run on each node is controlled by the user, the technology in use (RHV, pacemaker, etc…) and the node’s individual sosreport capabilities (in the case of varying versions of sos).
Running from a local workstation
As long as a system has network access to the nodes, it can be used to run sos-collector and provide the final archive of sosreports on the local node. To do so, use the –master option to specify a node that is capable of enumerating the other nodes in a cluster:
# sos-collector --master=rhev-manager.example.com
SSH keys, passwords, and users
sos-collector defaults to attempting to use SSH-keys to connect, and by default all keys in the running user’s keyring are tried.
To use a password to connect to the nodes, use the –password option. This will then prompt the user for the SSH password. It is assumed that this password is valid for all nodes discovered; if this is not the case, it is recommended to deploy SSH keys in the environment.
Additionally, by default the root user is used for SSH sessions since sosreport must run as root. You can change this with the –ssh-user option. Doing so will cause sos-collector to prompt for a sudo password for the specified user, unless the –become option is also specified, in which case you will be prompted for the root user password on the remote nodes.
Using Cluster Options
sos-collector supports multiple types of clusters, and options for those clusters that affect how sos-collector will function. You can view the available options using “sos-collector -l“:
# sos-collector -l The following cluster options are available: Cluster Option Name Type Default Description rhv no-database bool False Do not collect a database dump rhv cluster str Only collect from hosts in this cluster rhv datacenter str Only collect from hosts in this datacenter rhv no-hypervisors bool False Do not collect from hypervisors pacemaker online bool True Collect nodes listed as online pacemaker offline bool True Collect nodes listed as offline openshift label str Filter node list to those with matching label openshift role str Filter node list to those with matching role
These options can be enabled using the syntax “-c cluster_name.option_name=value“. For example, to disable database collection for a RHV environment, use the following:
# sos-collector -c rhv.no-database=True
Modifying the sosreport command on remote nodes
sos-collector has tight integration with the sosreport command, and most options that can be used for sosreport can be used in the exact same manner for sos-collector. For instance, running “sos-collector -e ovirt -n” logs will in turn enable the ovirt plugin and disable the logs plugin on each node.
It should also be noted that sos-collector also supports sos plugin options using the same format – E.G. “-k plugin_name.option_name=value“.
Additionally, sos-collector is capability aware on a per-node basis. This means that in the event of an invalid sosreport option being specified (for example, when there are varying versions of sos installed on the various nodes) will not prevent the collection of a sosreport from the node – the invalid options are simply filtered out.