while doing the grid patch prereq checks Using the dbaascli Utility on Exadata Cloud@Customer , ran into below error:
Execution of validate_source_home failed [FATAL] Inventory location can't be null
Command tried as below:
root ]$ dbaascli grid patch --targetVersion 188.8.131.52.0 --executePrereqs DBAAS CLI version 184.108.40.206.0 Executing command grid patch --targetVersion 220.127.116.11.0 --executePrereqs Job id: 5d9235e2-7c21-499f-a2ce-2f5116751ddd Session log: /var/opt/oracle/log/grid/patch/dbaastools_2023-06-17_08-13-13-AM_362649.log Loading PILOT… Session ID of the current execution is: 3303 Log file location: /var/opt/oracle/log/grid/patch/pilot_2023-06-17_08-13-15-AM_363066 ----------------- Running initialization job Completed initialization job ----------------- Running validate_nodes job Completed validate_nodes job ----------------- Running validate_target_version job Completed validate_target_version job ----------------- Running validate_backup_locations job Completed validate_backup_locations job ----------------- Running validate_source_home job Execution of validate_source_home failed [FATAL] Inventory location can't be null *** Executing jobs which need to be run always... *** ----------------- Running restore_opatch_prereqs_only job Completed restore_opatch_prereqs_only job ******** PLUGIN EXECUTION FAILED ********
Possible Causes of the Error and troubleshooting you could try:
1 Verifying Inventory Location and check its Configuration by comparing with other good nodes 2 Checking User Permissions and Privileges of inventory files 3 Examining Network Connectivity 4 check if scp is working
Cause in our case:
When scanning through the logs mention in the command output we see below command says succesful to scp the oraInst.loc file from remote node
[2023-06-17 08:40:42.571 EDT][pool-172-thread-1][DefaultCommandExecutor$2.call:316] Command constructed for node (localnode):/usr/bin/su oracle -c 'export LC_ALL=en_US.UTF-8;/usr/bin/scp -q -o PasswordAuthentication=no -o StrictHostKeyChecking=no -r node02:/etc/oraInst.loc /tmp/node02/27191-2023-06-17_08-40-42_438' INFO:[2023-06-17 08:40:42.820 EDT][pool-1-thread-2][DefaultCommandExecutor.run:381] Execution of /usr/bin/scp -q -o PasswordAuthentication=no -o StrictHostKeyChecking=no -r node02:/etc/oraInst.loc /tmp/node02/27191-2023-06-17_08-40-42_438 script is successful on nodes : [localnode]
but the check for existence of files after scp fails
INFO: [2023-06-17 08:40:42.828 EDT][pool-1-thread-2][DefaultCommandExecutor.run:381] Execution of /usr/bin/ls /tmp/node02/27191-2023-06-17_08-40-42_820/oraInventory/ContentsXML script is failed/unexecuted on nodes : [localnode] Errors : [/usr/bin/ls: cannot access /tmp/node02/27191-2023-06-17_08-40-42_820/oraInventory/ContentsXML: No such file or directory
We tried various possible ways of manually scp the oraInst.loc file as root user and checked on the file permissions on the remote node and gave full permissions on /tmp directory. Even then the actual dbaascli command was failing.
Then , We noticed few lines above that this is being tried as oracle user .
INFO: [2023-06-17 08:40:42.562 EDT][pool-1-thread-2][RemoteFileSystemInfo.transferFileToNodes:2407] Transferring file /etc/oraInst.loc as file /tmp/node02/27191-2023-06-17_08-40-42_438 from node node02 to nodes [localnode] as user ORACLE_USER
so, when we tried scp as oracle user , it errored saying “~/.ssh/id_rsa” key file’s permissions were too open and the actual scp of the file was not happening.
we had rectified the permission of the rsa key file to 0600 and then the issue was resolved.