CRS: pullup() dependency test

Introduction

We were recently creating some custom CRS resources that we need to automatically startup and shutdown to follow the Primary database in a DataGuard configuration. To achieve this, we were going to add start, stop and pullup dependencies between the database service resource and our custom resource.

However, the behaviour of the pullup dependency as described in the Oracle documentation is counter-intuitive and so I decided to do some testing around it to see the results and reassure myself that the documentation was correct.

In short, the documentation states that if resource A has aSTART_DEPENDENCY=pullup(B) dependency then resource B will pull up resource A(i.e. starting B will start A).

This test confirms whether this is indeed the case.

1. Create the test scripts

Firstly, we need to create the necessary action scripts that will be used by our test resources. let’s create a generic action script which can start, stop and check a given script/command:

File: /oracle/dbadmin/generic_action.ksh

#!/bin/bash
# ----------------------------------------------------------------
# A Generic action script for use with custom CRS resources
# which will perform a start/stop/check on the given script/cmd.
#
#   syntax: generic_action.ksh <action> <cmd>
#
# NOTE: remember to quote <cmd> when calling the script if the
#       command contains spaces.
# ----------------------------------------------------------------

ACTION=$1
CMD=$2

PID1=`ps -ef | grep "${CMD}" | grep -v generic_action | grep -v grep | awk '{ print $2 }'`
LOG_DIR=/tmp
BASENAME=`basename ${CMD} 2>/dev/null`

case ${ACTION} in
  'start')
    if [ "${PID1}" != "" ]
    then
      status_p1="running"
    else
      umask 002
      nohup ${CMD} >${LOG_DIR}/${BASENAME}.log 2>&1 &
      status_p1="started"
    fi
    echo "${BASENAME}: ${status_p1}"
    ;; 

  'stop')
    if [ "${PID1}" != "" ] ; then
      kill -9 ${PID1} && echo "${BASENAME} daemon killed"
    else
      echo "${BASENAME}: no running Process!"
    fi
    ;;

  'check')
    if [ "${PID1}" != "" ] ; then
      echo "running"
      exit 0
    else
      echo "not running"
      exit 1
    fi
    ;;

  *)
    echo "Usage: "`basename $0`" start|stop|check <cmd>"
    ;;
esac

Now we need a process that can be safely started and stopped, but which doesn’t do anything exciting. The sleep command is a good candidate, so let’s put that in a script.

File: /oracle/dbadmin/sleep_test.ksh

#!/bin/ksh
sleep 86400

We are going to have two CRS application resources, TEST_ONE and TEST_TWO. Each resource will need it’s own action script (which is very simple as we’re utilizing the generic_action.ksh script above).

When calling sleep_test.ksh we append a seond parameter which is not used by the script, but will help us identify which sleep command belongs to which resource.

File: /oracle/dbadmin/test_one.ksh

#!/bin/ksh
./generic_action.ksh $1 "/oracle/dbadmin/sleep_test.ksh TEST_ONE"
return $?

File: /oracle/dbadmin/test_two.ksh

#!/bin/ksh
./generic_action.ksh $1 "/oracle/dbadmin/sleep_test.ksh TEST_TWO"
return $?

2. Create the CRS resources

Now that we have all the necessary scripts to start and stop our test processes, we need to register them as resources in CRS.

crsctl add resource ROB_TEST1 -type application -attr "ACTION_SCRIPT=/oracle/dbadmin/test_one.ksh,AUTO_START=never,PLACEMENT =restricted,HOSTING_MEMBERS=`hostname`"

crsctl add resource ROB_TEST2 -type application -attr "ACTION_SCRIPT=/oracle/dbadmin/test_two.ksh,AUTO_START=never,PLACEMENT =restricted,HOSTING_MEMBERS=`hostname`"

3. Test the pullup() dependency behaviour

Let’s add a startup dependency to ROB_TEST1 which references ROB_TEST2:

crsctl modify resource ROB_TEST1 -attr "START_DEPENDENCIES='pullup:always(ROB_TEST2)'"

And test to see whether starting ROB_TEST1 starts ROB_TEST2, or vice-versa.

<pre>crs_stat -t | grep ROB

ROB_TEST1      application    OFFLINE   OFFLINE
ROB_TEST2      application    OFFLINE   OFFLINE

Start ROB_TEST:

crsctl start resource ROB_TEST1

crs_stat -t | grep ROB

ROB_TEST1      application    ONLINE    ONLINE    wycloram001
ROB_TEST2      application    OFFLINE   OFFLINE

So at this point, we have confirmed that starting ROB_TEST1 does not “pullup”ROB_TEST2. Let’s shutdown the resource and try it in reverse.

crsctl stop resource ROB_TEST1

crs_stat -t | grep ROB

ROB_TEST1      application    OFFLINE   OFFLINE
ROB_TEST2      application    OFFLINE   OFFLINE

And this time try starting ROB_TEST2

crsctl start resource ROB_TEST2

<pre>crs_stat -t | grep ROB

ROB_TEST1      application    ONLINE    ONLINE    wycloram001
ROB_TEST2      application    ONLINE    ONLINE    wycloram001

So starting ROB_TEST2 does “pullup” ROB_TEST1.

Conclusion

So creating resource A with -attr "START_DEPENDENCIES='pullup(B)'" will cause resource A to be started by resource B, not the other way around.

Personally I find this counter intuitive, and would have preferred pullup() to have been called pullup_by() or start_with().

Clean up

Finally, we need to clean up after ourselves and remove all resources and files created.

crsctl stop resource ROB_TEST1
crsctl stop resource ROB_TEST2

crsctl delete resource ROB_TEST1
crsctl delete resource ROB_TEST2

rm /oracle/dbadmin/generic_action.ksh
rm /oracle/dbadmin/test_one.ksh
rm /oracle/dbadmin/test_two.ksh
rm /oracle/dbadmin/generic_action.ksh

crs_stat -t | grep ROB