dbaascli grid patch executeprereqs reports “Inventory location can’t be null” error

while doing the grid patch prereq checks Using the dbaascli Utility on Exadata Cloud@Customer , ran into below error:

Execution of validate_source_home failed
[FATAL] Inventory location can't be null

Command tried as below:

root ]$ dbaascli grid patch --targetVersion 19.18.0.0.0 --executePrereqs
DBAAS CLI version 23.1.2.0.0
Executing command grid patch --targetVersion 19.18.0.0.0 --executePrereqs
Job id: 5d9235e2-7c21-499f-a2ce-2f5116751ddd
Session log: /var/opt/oracle/log/grid/patch/dbaastools_2023-06-17_08-13-13-AM_362649.log
Loading PILOT…
Session ID of the current execution is: 3303
Log file location: /var/opt/oracle/log/grid/patch/pilot_2023-06-17_08-13-15-AM_363066
-----------------
Running initialization job
Completed initialization job
-----------------
Running validate_nodes job
Completed validate_nodes job
-----------------
Running validate_target_version job
Completed validate_target_version job
-----------------
Running validate_backup_locations job
Completed validate_backup_locations job
-----------------
Running validate_source_home job
Execution of validate_source_home failed
[FATAL] Inventory location can't be null
*** Executing jobs which need to be run always... ***
-----------------
Running restore_opatch_prereqs_only job
Completed restore_opatch_prereqs_only job
******** PLUGIN EXECUTION FAILED ********

Possible Causes of the Error and troubleshooting you could try:

1 Verifying Inventory Location and check its Configuration by comparing with other good nodes
2 Checking User Permissions and Privileges of inventory files
3 Examining Network Connectivity
4 check if scp is working

Cause in our case:

When scanning through the logs mention in the command output we see below command says succesful to scp the oraInst.loc file from remote node

[2023-06-17 08:40:42.571 EDT][pool-172-thread-1][DefaultCommandExecutor$2.call:316] Command constructed for node (localnode):/usr/bin/su oracle -c 'export LC_ALL=en_US.UTF-8;/usr/bin/scp -q -o PasswordAuthentication=no -o StrictHostKeyChecking=no -r node02:/etc/oraInst.loc /tmp/node02/27191-2023-06-17_08-40-42_438'

INFO:[2023-06-17 08:40:42.820 EDT][pool-1-thread-2][DefaultCommandExecutor.run:381] Execution of /usr/bin/scp -q -o PasswordAuthentication=no -o StrictHostKeyChecking=no -r node02:/etc/oraInst.loc /tmp/node02/27191-2023-06-17_08-40-42_438 script is successful on nodes : [localnode]

but the check for existence of files after scp fails

INFO: [2023-06-17 08:40:42.828 EDT][pool-1-thread-2][DefaultCommandExecutor.run:381] Execution of /usr/bin/ls /tmp/node02/27191-2023-06-17_08-40-42_820/oraInventory/ContentsXML script is failed/unexecuted on nodes : [localnode]
 Errors  : [/usr/bin/ls: cannot access /tmp/node02/27191-2023-06-17_08-40-42_820/oraInventory/ContentsXML: No such file or directory

Solution:

We tried various possible ways of manually scp the oraInst.loc file as root user and checked on the file permissions on the remote node and gave full permissions on /tmp directory. Even then the actual dbaascli command was failing.

Then , We noticed few lines above that this is being tried as oracle user .

 INFO: [2023-06-17 08:40:42.562 EDT][pool-1-thread-2][RemoteFileSystemInfo.transferFileToNodes:2407] Transferring file /etc/oraInst.loc as file /tmp/node02/27191-2023-06-17_08-40-42_438 from node node02 to nodes [localnode] as user ORACLE_USER

so, when we tried scp as oracle user , it errored saying “~/.ssh/id_rsa” key file’s permissions were too open and the actual scp of the file was not happening.
we had rectified the permission of the rsa key file to 0600 and then the issue was resolved.