Smart way to run commands on many *nix machines from (not only) Windows client

Having a need to run the same set of commands on many machines is a very common task for IT engineers of all kind. If you want to do it efficiently and in a smart way (e.g. to run it only on machines with specific set of conditions), I have an example, that could be added to your toolbox.

To have a real-life scenario, let’s define few assumptions:

1. You work on Windows machine with MobaXTerm installed.
2. You deal with many *nix server machines and you need to run some commands on all/big bunch of them.
3. More specifically – there are many Oracle databases running on each server and you want to run something on all/big bunch of 1st instances of those databases.
4. Ideally – it would be nice to run the command depending on the conditions found in each database.
5. To make things even a little bit more complicated – while setting up environment for your oracle user (e.g. inside .profile and/or .bash_profile), interactive input is needed (e.g. to choose which database you want to work with), so you cannot simply use sudo su - oracle in your solution.

To avoid typing password for every database, you can use Oracle Wallet. How to configure it is out of scope here, but you can check it for example in this nice blog post by Ludovico Caldara.

Assuming you have the wallet working, please find below an example, showing how to perform some actions with all above assumptions in mind. In this specific case, we are enabling Block Change Tracking in all databases running on all nodes listed in nodes.txt file, but only for ones where this feature is currently disabled. We are connecting only to first instances to avoid doing it more than once for RAC databases.

1. Open new MobaXTerm session.
2. To avoid typing your OS user password for every machine, run read -s SSHPASS, type your password and press ENTER. Option -s disables output on the screen, so nobody will be able to see your password directly. But please be aware that it will be stored in clear text under SSHPASS variable, so you may want to unset it afterwards.
3. Run below commands to do your changes in smart way (replace [USERNAME] and [ORACLE_HOME_PATH] with proper values).

for node in `cat nodes.txt` ; do
    SSHPASS=$SSHPASS sshpass -e \
        ssh -qt [USERNAME]@${node} \
            'export ORACLE_HOME=[ORACLE_HOME_PATH]; \
                for i in `ps -ef | grep ora_smon | grep -vw grep | sed "s/.*_smon_//;" | grep "1$"`; do \
                    STATUS=`echo "ORACLE_HOME=$ORACLE_HOME ORACLE_SID=${i} $ORACLE_HOME/bin/sqlplus -S / as sysdba <<EOF
                                    set pages 0
                                    set head off
                                    set feed off
                                    select trim(status) from v\\\\\\$block_change_tracking;
EOF" | sudo su oracle`;
                    if [ "${STATUS}" == "DISABLED" ]; then
                        echo -e "\e[1;41m`hostname`-${i}\e[0m"
                        echo "ORACLE_HOME=$ORACLE_HOME ORACLE_SID=${i} $ORACLE_HOME/bin/sqlplus -S / as sysdba <<EOF
                                    set pages 0
                                    set head off
                                    alter database enable block change tracking;
EOF" | sudo su oracle
                    fi;
                done'
done

As an output you should get nice, coloured node and database name along with output from SQL*Plus, looking like example one below:

If your target machines are not running Linux, but other *nix system having less “sophisticated” echo command (e.g. AIX), you can change:

echo -e "\e[1;41m`hostname`-${i}\e[0m"

to

echo "\033[1;41m`hostname`-${i}\033[0m"

in order to get proper colouring.

I know that special characters escaping looks like a nightmare (it is), but once you learn how to properly do it for the most commonly used characters (e.g. one ' has to be escaped with '"'"'), then you should be fine. Of course above solution is not limited only to Windows clients. You should be able to run it similarly for example from other *nix boxes. You may need some minor tweaks depending on your client and target environment, but above has been tested on Windows 10 with MobaXTerm v9.4 and IBM AIX 7.1 target machines, as well as macOS High Sierra with Oracle Enterprise Linux running on target.

I hope it helps…but please don’t forget that having such a possibility you have to be extremely careful, not to damage your environment, so please think at least twice before running anything that way :).