1.1Create the Dbs Group File

1Configure Oracle DBFS on Exadata

1.1Create the dbs_group file

As the user root, create a text file in the home directory of the root user called dbs_group. This file will contain the names of both the X5-2 compute nodes.

We will be using the DCLI utility to run commands on both compute nodes in the Exadata box and this file will be used for that purpose when we run the ‘dcli –g’ command.

[root@exdb1db01 ~]# dcli -g dbs_group -l root hostname

exdb1db01: exdb1db01.gavin.com.au

exdb1db02: exdb1db02.gavin.com.au

1.2Add the oracle user to the fuse group

[root@exdb1db01 ~]# dcli -g ~/dbs_group -l root usermod -a -G fuse oracle

1.3Add the user_allow_other option to the fuse.conf file

root@exdb1db01 ~]# dcli -g ~/dbs_group -l root "echo user_allow_other > /etc/fuse.conf"

[root@exdb1db01 ~]# dcli -g ~/dbs_group -l root chmod 644 /etc/fuse.conf

Note – on the Exadata servers, the required fuse RPM packages are installed by default.

1.4Create the mount points and give appropriate permissions

On both compute nodes we will create mount points which will be used to mount the DBFS file system.

Since the objective is to have multiple mount points where each mount point is dedicated to separate database or environment, we will create the mount point with the naming convention /dbfs/<Database Name>.

Change the ownership of the mount points to the oracle user

dcli -g ~/dbs_group -l root mkdir /dbfs/dev2

dcli -g ~/dbs_group -l root chownoracle:oinstall /dbfs/dev2

1.5Create tablespace and users

As the user SYS, we will create two tablespaces which will be used to store the LOB objects associated with the DBFS file system.

We will create the dbfs_gg_dirtmptablespace with the recommended NOLOGGING attribute as it will be used to store the contents of the GoldenGatedirtmp directory.

Note: The size of the tablespace will depend on the amount of trail files which are expected to be generated as well as the required retention period for those trail files.

While the example shows the DBFS_DG ASM disk group being used for the hosting the DBFS related tablespaces, any ASM disk group with the required amount of free disk space can be used.

The DBFS_USER Oracle database user will be the owner of the DBFS related database objects and we create the user and grant the appropriate privileges especially the DBFS_ROLE database role.

createbigfiletablespacedbfs_gg_dirsrc

datafile '+DBFS_DG' size 32g autoextend on next 2g

LOGGING EXTENT MANAGEMENT LOCAL AUTOALLOCATE

SEGMENT SPACE MANAGEMENT AUTO;

createbigfiletablespacedbfs_gg_dirtmp

datafile '+DBFS_DG' size 10g autoextend on next 2g

NOLOGGING EXTENT MANAGEMENT LOCAL AUTOALLOCATE

SEGMENT SPACE MANAGEMENT AUTO;

create user dbfs_user identified by Oracle#123

defaulttablespacedbfs_gg_dirsrc

temporarytablespace temp

quota unlimited on dbfs_gg_dirsrc

quota unlimited on dbfs_gg_dirtmp;

GRANT create session, create table, create view,create procedure, dbfs_role TO dbfs_user;

1.6Create the DBFS file system

We will next connect as the DBFS_USER and run the dbfs_create_filesystem.sql script to create the necessary DBFS related database objects.

The dbfs_create_filesystem.sql takes two parameters – the tablespace_name and the DBFS file system name.

SQL> conn dbfs_user/<dbfs user password>

Connected.

SQL> @?/rdbms/admin/dbfs_create_filesystemdbfs_gg_dirsrc ogg_dev2

No errors.

------

CREATE STORE:

begindbms_dbfs_sfs.createFilesystem(store_name => 'FS_OGG_DEV2', tbl_name =>

'T_OGG_DEV2', tbl_tbs => 'dbfs_gg_dirsrc', lob_tbs => 'dbfs_gg_dirsrc',

do_partition => false, partition_key => 1, do_compress => false, compression =>

'', do_dedup => false, do_encrypt => false); end;

------

begindbms_dbfs_content.registerStore(store_name=> 'FS_OGG_DEV2',

provider_name => 'sample1', provider_package => 'dbms_dbfs_sfs'); end;

------

MOUNT STORE:

begindbms_dbfs_content.mountStore(store_name=>'FS_OGG_DEV2',

store_mount=>'ogg_dev2'); end;

------

CHMOD STORE:

declare m integer; begin m := dbms_fuse.fs_chmod('/ogg_dev2', 16895); end;

No errors.

SQL>@?/rdbms/admin/dbfs_create_filesystemdbfs_gg_dirtmp ogg_dirtmp_dev2

No errors.

------

CREATE STORE:

begindbms_dbfs_sfs.createFilesystem(store_name => 'FS_OGG_DIRTMP_DEV2',

tbl_name => 'T_OGG_DIRTMP_DEV2', tbl_tbs => 'dbfs_gg_dirtmp', lob_tbs =>

'dbfs_gg_dirtmp', do_partition => false, partition_key => 1, do_compress =>

false, compression => '', do_dedup => false, do_encrypt => false); end;

------

begindbms_dbfs_content.registerStore(store_name=> 'FS_OGG_DIRTMP_DEV2',

provider_name => 'sample1', provider_package => 'dbms_dbfs_sfs'); end;

------

MOUNT STORE:

begindbms_dbfs_content.mountStore(store_name=>'FS_OGG_DIRTMP_DEV2',

store_mount=>'ogg_dirtmp_dev2'); end;

------

CHMOD STORE:

declare m integer; begin m := dbms_fuse.fs_chmod('/ogg_dirtmp_dev2', 16895);

end;

No errors.

1.7Verify the DBFS LOB segment attributes

SQL> SELECT table_name, segment_name, logging, cache

2 FROMdba_lobs WHERE tablespace_name like 'DBFS%';

TABLE_NAMESEGMENT_NAME LOGGING CACHE

------

T_OGG_DEV2 LOB_SFS$_FST_1 YES NO

T_OGG_DIRTMP_DEV2 LOB_SFS$_FST_11 NONO

1.8Edit and customize the Oracle supplied mount-dbfs.sh script

Download the file mount-dbfs-20160215.zip from the MOS note 1054431.1 (Configuring DBFS on Oracle Exadata Database Machine).

Copy the file to a temporary directory on one of the database compute nodes and as the user root, extract the file.

We will now have two files – mount-dbfs.conf and mount-dbfs.sh.

Copy the mount-dbfs.sh to mount-dbfs_<database name>.sh

[root@exdb1db01 ~]# cd /tmp

[root@exdb1db01 tmp]# cp mount-dbfs.sh mount-dbfs_dev2.sh

[root@exdb1db01 tmp]# cp mount-dbfs.conf mount-dbfs_dev2.conf

Edit the mount-dbfs_<database name>.sh script to reference the customized CONFIG file

[root@exdb1db01 tmp]# vi mount-dbfs_dev2.sh

### Ensure that when multiple mounts are used, there are separate copies

###of mount-dbfs.sh that reference separate CONFIG file pathnames

CONFIG=/etc/oracle/mount-dbfs_dev2.conf

1.9Edit and customize the Oracle supplied mount-dbfs.conf script

Change the values for :

DBNAME
MOUNT_POINT
DBFS_USER
ORACLE_HOME
GRID_HOME
DBFS_PASSWORD

### Database name for the DBFS repository as used in "srvctl status database -d $DBNAME"

### If using PDB/CDB, this should be set to the CDB name

### Database name for the DBFS repository as used in "srvctl status database -d $DBNAME"

### If using PDB/CDB, this should be set to the CDB name

DBNAME=DEV2

### Mount point where DBFS should be mounted

MOUNT_POINT=/dbfs/dev2

### Username of the DBFS repository owner in database $DBNAME

DBFS_USER=dbfs_user

### RDBMS ORACLE_HOME directory path

ORACLE_HOME=/u01/app/oracle/product/11.2.0/shieldnp_1

### GRID HOME directory path

GRID_HOME=/u01/app/12.1.0/grid_1

###########################################

### If using password-based authentication, set these

###########################################

### This is the plain text password for the DBFS_USER user

DBFS_PASSWD=Oracle#123

1.10Copy the modified files to $GRID_HOME/crs/script as well as /etc/oracle and grant appropriate privileges

dcli -g ~/dbs_group -l root -d /u01/app/12.1.0/grid_1/crs/script -f /tmp/mount-dbfs_dev2.sh

dcli -g ~/dbs_group -l root chownoracle:oinstall /u01/app/12.1.0/grid_1/crs/script/mount-dbfs_dev2.sh

dcli -g ~/dbs_group -l root chmod 750 /u01/app/12.1.0/grid_1/crs/script/mount-dbfs_dev2.sh

dcli -g ~/dbs_group -l root -d /etc/oracle -f /tmp/mount-dbfs_dev2.conf

dcli -g ~/dbs_group -l root chownoracle:oinstall /etc/oracle/mount-dbfs_dev2.conf

dcli -g ~/dbs_group -l root chmod 640 /etc/oracle/mount-dbfs_dev2.conf

1.11Create the script for mounting the DBFS File System

We will create the add-dbfs-resource _<database name>.sh script. This script will be used to create the clusterware resource for mounting the DBFS file system.

Note that the add-dbfs-resource script will be sourcing the customized mount-dbfs_<database name>.sh script which we had created earlier.

[root@exdb1db01 tmp]# cd /u01/app/12.1.0/grid_1/crs/script

[root@exdb1db01 script]# vi add-dbfs-resource_dev2.sh

##### start script add-dbfs-resource_dev2.sh

#!/bin/bash

ACTION_SCRIPT=/u01/app/12.1.0/grid_1/crs/script/mount-dbfs_dev2.sh

RESNAME=dbfs_mount_dev2

DBNAME=DEV2

DBNAMEL=`echo $DBNAME | tr A-Z a-z`

ORACLE_HOME=/u01/app/oracle/product/11.2.0/shieldnp_1

PATH=$ORACLE_HOME/bin:$PATH

export PATH ORACLE_HOME

/u01/app/12.1.0/grid_1/bin/crsctl add resource $RESNAME \

-type local_resource \

-attr "ACTION_SCRIPT=$ACTION_SCRIPT, \

CHECK_INTERVAL=30,RESTART_ATTEMPTS=10, \

START_DEPENDENCIES='hard(ora.$DBNAMEL.db)pullup(ora.$DBNAMEL.db)',\

STOP_DEPENDENCIES='hard(ora.$DBNAMEL.db)',\

SCRIPT_TIMEOUT=300"

##### end script add-dbfs-resource_dev2.sh

Change the ownership of the script to oracle

[root@exdb1db01 script]# chownoracle:oinstall add-dbfs-resource_dev2.sh

1.12As the OS user oracle run the add-dbfs-resource script to create the resource

[root@exdb1db01 script]# su - oracle

[oracle@exdb1db01 ~]$ cd /u01/app/12.1.0/grid_1/crs/script

[oracle@exdb1db01 script]$ ./add-dbfs-resource_dev2.sh

1.13As oracle start the resource using crsctl – this will mount the DBFS file system

[oracle@exdb1db01 ~]$ cd /u01/app/12.1.0/grid_1/bin

[oracle@exdb1db01 bin]$ ./crsctl start resource dbfs_mount_dev2

CRS-2672: Attempting to start 'dbfs_mount_dev2' on 'exdb1db01'

CRS-2672: Attempting to start 'ora.dev2.db' on 'exdb1db02'

CRS-2676: Start of 'dbfs_mount_dev2' on 'exdb1db01' succeeded

CRS-2676: Start of 'dbfs_mount_dev2' on 'exdb1db02' succeeded

1.14Check the status of the resource

[oracle@exdb1db01 bin]$ ./crsctl stat res dbfs_mount_dev2

NAME=dbfs_mount_dev2

TYPE=local_resource

TARGET=ONLINE , ONLINE

STATE=ONLINE on exdb1db01, ONLINE on exdb1db02

[oracle@exdb1db01 bin]$ exit

logout

1.15As root create the Application VIP

[root@exdb1db01 script]# cd /u01/app/12.1.0/grid_1/bin

[root@exdb1db01 bin]# ./appvipcfg create -network=1 -ip=10.100.24.28 -vipname=ogg_vip_dev2 -user=root

[root@exdb1db01 bin]# ./crsctlsetperm resource ogg_vip_dev2 -u user:oracle:r-x

[root@exdb1db01 bin]# ./crsctlsetperm resource ogg_vip_dev2 -u user:grid:r-x

[root@exdb1db01 bin]# ./crsctl start resource ogg_vip_dev2

CRS-2672: Attempting to start 'ogg_vip_dev2' on 'exdb1db02'

CRS-2676: Start of 'ogg_vip_dev2' on 'exdb1db02' succeeded

We can see that the VIP is running on exdb1db02 -we can relocate it to exdb1db01

[root@exdb1db01 bin]# ./crsctl relocate resource ogg_vip_dev2

CRS-2673: Attempting to stop 'ogg_vip_dev2' on 'exdb1db02'

CRS-2677: Stop of 'ogg_vip_dev2' on 'exdb1db02' succeeded

CRS-2672: Attempting to start 'ogg_vip_dev2' on 'exdb1db01'

CRS-2676: Start of 'ogg_vip_dev2' on 'exdb1db01' succeeded

Now check the status of the resource – we can see it running on exdb1db01

[root@exdb1db01 bin]# ./crsctl status resource ogg_vip_dev2

NAME=ogg_vip_dev2

TYPE=app.appvipx.type

TARGET=ONLINE

STATE=ONLINE on exdb1db01

1.16Check if the DBFS file systems for each database environment are mounted and directories are present

[root@exdb1db01 bin]# df -k |grepdbfs

dbfs-dbfs_user@:/ 56559616 232 56559384 1% /dbfs_dev2

[root@exdb1db01 bin]# cd /dbfs_dev2/

[root@exdb1db01 dbfs_dev2]# ls -l

total 0

drwxrwxrwx 3 root root 0 Feb 25 11:56 ogg_dev2

drwxrwxrwx 3 root root 0 Feb 25 11:57 ogg_dirtmp_dev2

2Configure Grid Infrastructure Agent

2.1Create the directories on the DBFS file system

[oracle@exdb1db01 ogg_dev2pd]$ pwd

/dbfs/dev2pd/ogg_dev2pd

[oracle@exdb1db01 ]$ mkdirdirpcs

[oracle@exdb1db01 ]$ mkdirdirchk

[oracle@exdb1db01 ]$ mkdirdirdat

[oracle@exdb1db01 ]$ mkdirdirprm

[oracle@exdb1db01 ]$ mkdirdircrd

[oracle@exdb1db01 ]$ mkdir BR

[oracle@exdb1db01 dev2pd]$ cd ogg_dirtmp_dev2pd

[oracle@exdb1db01 ogg_dirtmp_dev2pd]$ pwd

/dbfs/dev2pd/ogg_dirtmp_dev2pd

[oracle@exdb1db01 ogg_dirtmp]$ mkdirdirtmp

2.2On each compute node rename the existing directories in the GoldenGate software home

[oracle@exdb1db01 dev2]$ mkdir BR

[oracle@exdb1db01 dev2]$ mvdirchkdirchk.bkp

[oracle@exdb1db01 dev2]$ mvdirdatdirdat.bkp

[oracle@exdb1db01 dev2]$ mvdirpcsdirpcs.bkp

[oracle@exdb1db01 dev2]$ mvdirprmdirprm.bkp

[oracle@exdb1db01 dev2]$ mvdircrddircrd.bkp

[oracle@exdb1db01 dev2]$ mvdirtmpdirtmp.bkp

2.3Create the symbolic links

[oracle@exdb1db01 dev2]$ ln -s /dbfs/dev2pd/ogg_dev2pd/dirdatdirdat

[oracle@exdb1db01 dev2]$ ln -s /dbfs/dev2pd/ogg_dev2pd/dirchkdirchk

[oracle@exdb1db01 dev2]$ ln -s /dbfs/dev2pd/ogg_dev2pd/dirpcsdirpcs

[oracle@exdb1db01 dev2]$ ln -s /dbfs/dev2pd/ogg_dev2pd/dirprmdirprm

[oracle@exdb1db01 dev2]$ ln -s /dbfs/dev2pd/ogg_dev2pd/BR BR

[oracle@exdb1db01 dev2]$ln -s /dbfs/dev2pd/ogg_dev2pd/dircrddircrd

[oracle@exdb1db01 dev2]$ln -s /dbfs/dev2pd/ogg_dirtmp_dev2pd /dirtmpdirtmp

2.4Download Oracle Grid Infrastructure Agent

From the URL below download the file: xagpack_7b.zip

2.5Copy the downloaded xagpack_7b.zip file to Grid user $HOME and unzip

[grid@exdb1db01 ~]$ ls

xagpack_7b.zip

[grid@exdb1db01 ~]$ unzip xagpack_7b.zip

Archive: xagpack_7b.zip

creating: xag/

inflating: xag/xagsetup.bat

creating: xag/lib/

inflating: xag/lib/facility.lis

inflating: xag/agcommon.pm

inflating: xag/agjdeas.pm

creating: xag/bin/

inflating: xag/bin/oerr.pl

inflating: xag/xagsetup.sh

…

inflating: xag/mesg/xagus.be

inflating: xag/mesg/xagus.msg

inflating: xag/mesg/xagus.msb

inflating: xag/agmysqlmonas.pm

inflating: xag/readme.txt

inflating: xag/agwl.pm

2.6Two directories will be created -xag and xagent

[grid@exdb1db01 xag]$ pwd

/home/grid/xag

[grid@exdb1db01 xag]$ cd ..

[grid@exdb1db01 ~]$ ls

xagxagent xagpack_7b.zip

2.7Run the xagsetup.sh script (as the Grid Infrastructure owner)

Note – this will install the Grid Infrastructure Agent files in the xagent directory (on both compute nodes)

[grid@exdb1db01 xag]$ ./xagsetup.sh --install --directory /u01/app/grid/xagent --all_nodes

Installing Oracle Grid Infrastructure Agents on: exdb1db01

Installing Oracle Grid Infrastructure Agents on: exdb1db02

If we try and install the Grid Infrastructure Agents under the $GRID_HOME we will see an error as shown below:

[grid@exdb1db01 xag]$ ./xagsetup.sh --install --directory /u01/app/12.1.0/grid_1/xagent --all_nodes

Installation directory cannot be under Clusterware home.

2.8As oracle we run the AGCTL command to create the GoldenGate resource

[root@exdb1db01 bin]# su - oracle

[oracle@exdb1db01 ~]$ cd /u01/app/grid/xagent/bin

[oracle@exdb1db01 bin]$ ./agctl add goldengate ogg_dev2 --gg_home /u01/app/oracle/product/gg12.2/dev2 --instance_type source --nodes exdb1db01,exdb1db02 --vip_name ogg_vip_dev2 --filesystems dbfs_mount_dev2pd --databases ora.dev2pd.db --oracle_home /u01/app/oracle/product/11.2.0/shieldnp_1

2.9Start and Stop Goldengate using AGCTL

[oracle@exdb1db01 bin]$ ./agctl status goldengate ogg_dev2

Goldengate instance 'ogg_dev2' is not running

[oracle@exdb1db01 bin]$ ./agctl start goldengate ogg_dev2

[oracle@exdb1db01 bin]$ ./agctl status goldengate ogg_dev2

Goldengate instance 'ogg_dev2' is running on exdb1db01

If we check via GGSCI, we can see the manager process is now up and running on compute node exdb1db01

[oracle@exdb1db01 bin]$ cd -

/u01/app/oracle/product/gg12.2/dev2

[oracle@exdb1db01 dev2]$ ./ggsci

Oracle GoldenGate Command Interpreter for Oracle

Version 12.2.0.1.1 OGGCORE_12.2.0.1.0_PLATFORMS_151211.1401_FBO

Linux, x64, 64bit (optimized), Oracle 11g on Dec 12 2015 00:54:38

Operating system character set identified as UTF-8.

GGSCI (exdb1db01.gavin.com.au) 1> info all

Program Status Group Lag at Chkpt Time Since Chkpt

MANAGER RUNNING

Note that manager is stopped on compute node exdb1db02

GGSCI (exdb1db02.gavin.com.au) 3> info all

Program Status Group Lag at Chkpt Time Since Chkpt

MANAGER STOPPED

2.10Relocate GoldenGate using AGCTL

[oracle@exdb1db01 bin]$ ./agctl relocate goldengate ogg_dev2

[oracle@exdb1db01 bin]$ ./agctl status goldengate ogg_dev2

Goldengate instance 'ogg_dev2' is running on exdb1db02

Now manager is running on exdb1db02

GGSCI (exdb1db02.gavin.com.au) 3> info all

Program Status Group Lag at Chkpt Time Since Chkpt

MANAGER RUNNING