Build high availability for Amazon RDS Custom for Oracle using read replicas

<section class="blog-post-content">A high availability solution for the database stack is an important aspect to consider when migrating or deploying Oracle databases in the AWS Cloud to help ensure that the architecture can meet the service level agreement (SLA) of the application. Unavailability of a critical database can lead to application outage, interruption to business operations, inaccessibility to critical data, revenue loss, and loss of customer trust.In this post, we discuss how to enable high availability for an <a href="https://aws.amazon.com/rds/custom/" target="_blank" rel="noopener noreferrer">Amazon Relational Database Service (Amazon RDS) Custom</a> for Oracle instance using read replicas and best practices to follow during role transition and failover scenarios.You can achieve high availability (HA) and disaster recovery (DR) for Oracle databases using a physical standby database by creating and maintaining a physical copy of the primary database. The standby instance can be hosted in a location different and far enough from the primary instance, such as a <a href="https://aws.amazon.com/about-aws/global-infrastructure/regions_az/" target="_blank" rel="noopener noreferrer">different Region or Availability Zone</a>, to maintain the availability of the database when the primary instance is impacted by unplanned incidents. <a href="https://docs.oracle.com/en/database/oracle/oracle-database/19/sbydb/index.html" target="_blank" rel="noopener noreferrer">Oracle Data Guard</a> provides a comprehensive set of services to create, maintain, and monitor one or more standby databases. Data Guard maintains a copy of the data in a standby database that is continuously updated with changes from the primary database. Data Guard validates the changes before they’re applied to the standby database, to help prevent physical corruptions that occur in the storage layer from causing data loss and downtime. Data Guard also helps repair corrupted blocks in the primary database in real time by fetching the “good” version of the corrupted block from an <a href="https://www.oracle.com/technetwork/database/availability/active-data-guard-ds-12c-1898869.pdf" target="_blank" rel="noopener noreferrer">Active Data Guard</a> physical standby database. Additionally, Data Guard can also help improve availability of the database during planned maintenance activities such as database patching using <a href="https://docs.oracle.com/en/database/oracle/oracle-database/19/sbydb/upgrading-patching-downgrading-oracle-data-guard-configuration.html" target="_blank" rel="noopener noreferrer">Standby First Patching</a> strategy.<a href="https://aws.amazon.com/rds/custom/" target="_blank" rel="noopener noreferrer">Amazon Relational Database Service (Amazon RDS) Custom</a> is a managed database service for legacy, custom, and packaged applications and any application that requires access to the underlying OS and DB environment. RDS Custom for Oracle provides the flexibility to customize your database, underlying server, and operating system configurations to support applications that require such customizations. RDS Custom for Oracle supports read replicas that are built on Data Guard technology, and you can use these replicas to offload reads from the primary and for disaster recovery. As of this writing, RDS Custom for Oracle doesn’t support the <a href="https://aws.amazon.com/blogs/aws/multi-az-option-for-amazon-rds-oracle/" target="_blank" rel="noopener noreferrer">Multi-AZ option</a>, which is available for <a href="https://aws.amazon.com/rds/oracle/" target="_blank" rel="noopener noreferrer">Amazon RDS for Oracle</a> as an HA solution using storage replication.<h2>Solution overview</h2>With RDS Custom for Oracle, you can create up to five read replicas, which are managed physical standby databases using Data Guard configuration. These read replicas created in mounted mode can be opened in read-only mode and used for reporting purposes. Read replicas in RDS Custom use asynchronous log shipping mode with MaxPerformance Data Guard configuration, and are restricted to the same Region as the primary instance (as of this writing). Because these read replicas are managed by AWS, most of the heavy lifting activities in maintaining a typical Data Guard configuration are automated, including the following tasks:<ul><li>Creating and maintaining a Data Guard broker configuration</li><li>Shipping redo changes from the primary instance to the standby instance (read replica)</li><li>Resolving archivelog gaps</li><li>Purging archivelogs from the primary and standby when they’re no longer needed</li></ul>The following diagram illustrates a read replica in RDS Custom for Oracle.<img class="alignnone wp-image-24730 size-full" src="https://d2908q01vomqb2.cloudfront.net/887309d048beef83ad3eabf2a79a64a389ab1c9f/2022/09/21/image001.png" alt="Solution Overview" width="1182" height="640" />However, this default configuration of read replicas doesn’t provide a Recovery Point Objective (RPO) of zero because the redo is shipped asynchronously from the primary instance to the replica. Furthermore, the failover operation is manual because there’s no automated failure detection or failover mechanism, and rebuilding the former primary instance after a failover operation needs to be done manually by creating a new read replica.With shell access to the underlying <a href="https://aws.amazon.com/ec2/" target="_blank" rel="noopener noreferrer">Amazon Elastic Compute Cloud</a> (Amazon EC2) instance of the RDS Custom for Oracle instance, the Data Guard configuration of the read replica can be customized to meet various customer-defined HA and DR requirements. The following are a few customizations you can make to the default read replica configuration to help you meet your Recovery Time Objective (RTO) and RPO requirements:<ul><li>Modify log shipping mode from asynchronous to synchronous to help you achieve an RPO of zero</li><li>Enable Data Guard <a href="https://docs.oracle.com/en/database/oracle/oracle-database/21/sbydb/oracle-data-guard-protection-modes.html#GUID-16298839-9B8A-4E58-8EC9-27912E57F1CE" target="_blank" rel="noopener noreferrer">protection mode</a> to maximum availability</li><li>Configure a Data Guard observer process to enable <a href="https://www.oracle.com/technical-resources/articles/smiley-fsfo.html" target="_blank" rel="noopener noreferrer">Fast-Start Failover (FSFO)</a> to facilitate automated failure detection and failover</li><li>Enable the flashback database feature on both the primary and standby instances for automated and fast reinstatement of the former primary database after a failover activity</li></ul>The following diagram illustrates a customized read replica configuration to help you meet your HA requirements.<img class="alignnone wp-image-24731 size-full" src="https://d2908q01vomqb2.cloudfront.net/887309d048beef83ad3eabf2a79a64a389ab1c9f/2022/09/21/image002-3.png" alt="Customized read replica" width="1257" height="763" />In addition to these managed replicas, you can also create self-managed standby databases in various locations, such as a different AWS Region, on-premises, or a different cloud environment, to help you meet your disaster recovery requirements. You can manage those self-managed standby databases and managed read replicas with a single Data Guard configuration.In this post, we discuss the following topics:<ul><li>Create a read replica for an existing RDS Custom for Oracle instance</li><li>Change Data Guard protection mode</li><li>Convert a replica from mounted state to open read-only</li><li>Role transition (switchover)</li><li>Data Guard failover</li><li>Promote the read replica</li><li>Configure FSFO</li><li>Seamless Oracle Client connectivity</li><li>Cross-Region Data Guard configuration</li></ul>In this post, we refer to different resources such as Database Identifier in AWS Console, Database Unique Name (<code>db_unique_name</code> instance parameter), Database name in Data Guard broker configuration and Net service name in <code>tnsnames.ora</code> to illustrate a sample configuration. The following table serves as a reference to the names of different resources used in this post.<table class="c5" border="1px" style="width: 730px;"><tbody><tr class="c4"><td>.</td><td>Primary</td><td>Read Replica</td></tr><tr><td>DB Identifier</td><td>orcl1</td><td>orcl2</td></tr><tr><td>DB Unique Name</td><td>ORCL_A</td><td>ORCL_B</td></tr><tr><td>DB Name in Data Guard Broker</td><td>orcl_a</td><td>orcl_b</td></tr><tr><td>Oracle Net Service Name</td><td>RDS_CUSTOM_ORCL_A</td><td>RDS_CUSTOM_ORCL_B</td></tr></tbody></table><h2>Create a read replica for an existing RDS Custom for Oracle instance</h2>You can create a read replica for an existing RDS Custom for Oracle instance either from the <a href="http://aws.amazon.com/console" target="_blank" rel="noopener noreferrer">AWS Management Console</a> or using the <a href="http://aws.amazon.com/cli" target="_blank" rel="noopener noreferrer">AWS Command Line Interface</a> (AWS CLI).To create a read replica from the Amazon RDS console, complete the following steps:<ol><li>On the Amazon RDS console, choose Databases in the navigation pane.</li><li>Select the database you want to add a read replica for.</li><li>On the Actions menu, choose Create read replica. Read replica is a copy of your source database, and automatically receives and applies changes received from source database.</li><li>For Replica source, enter your source DB instance.In this example, we use the existing RDS Custom for Oracle instance <code>orcl1</code> in the Singapore Region as the source to create our read replica.</li><li>For DB instance identifier, enter your DB instance (for this post, <code>orcl2</code>).<a href="https://d2908q01vomqb2.cloudfront.net/887309d048beef83ad3eabf2a79a64a389ab1c9f/2022/09/21/image003-3.jpg"><img class="alignnone size-full wp-image-24732 c6" src="https://d2908q01vomqb2.cloudfront.net/887309d048beef83ad3eabf2a79a64a389ab1c9f/2022/09/21/image003-3.jpg" alt="create replica screenshot1" width="758" height="271" /></a></li><li>Choose the database instance class necessary for the performance requirements of the replica instance.<a href="https://d2908q01vomqb2.cloudfront.net/887309d048beef83ad3eabf2a79a64a389ab1c9f/2022/09/21/image004-5.jpg"><img class="alignnone size-full wp-image-24733 c6" src="https://d2908q01vomqb2.cloudfront.net/887309d048beef83ad3eabf2a79a64a389ab1c9f/2022/09/21/image004-5.jpg" alt="create replica screenshot2" width="752" height="243" /></a></li><li>For Storage type, choose your storage type.We recommend that you use the same storage type as the source DB instance for the read replica used as a failover target.<img class="alignnone size-full wp-image-24734 c6" src="https://d2908q01vomqb2.cloudfront.net/887309d048beef83ad3eabf2a79a64a389ab1c9f/2022/09/21/image005-3.jpg" alt="create replica screenshot3" width="736" height="259" /></li><li>Accept the suggested values under Connectivity, and choose a different Availability Zone from the source database to protect from Availability Zone failure.<img class="alignnone size-full wp-image-24735 c6" src="https://d2908q01vomqb2.cloudfront.net/887309d048beef83ad3eabf2a79a64a389ab1c9f/2022/09/21/image006-2.jpg" alt="create replica screenshot4" width="586" height="96" /></li><li>For IAM instance profile, choose the instance profile for your RDS Custom for Oracle DB instance.<img class="alignnone size-full wp-image-24736 c6" src="https://d2908q01vomqb2.cloudfront.net/887309d048beef83ad3eabf2a79a64a389ab1c9f/2022/09/21/image007-3.jpg" alt="create replica screenshot4" width="734" height="169" /></li><li>Choose Create read replica to create the read replica <code>orcl2</code>.</li></ol>Alternatively, you can create a read replica from the AWS CLI, as shown in the following example:<h2>Choose your database protection mode</h2>Oracle Data Guard provides three protection modes to configure your Data Guard environment based on your availability, protection, and performance requirements. The following table summarizes these three modes.<table border="1px"><tbody><tr class="c4"><td>Protection Mode</td><td>Redo Transport Setting</td><td>Description</td></tr><tr><td>MAXIMUM PERFORMANCE</td><td>ASYNC</td><td>For transactions happening on the primary database, redo data is asynchronously transmitted and written to the standby database redo log, therefore the performance impact is minimal.<code>MaxPerformance</code> can not provide RPO=0 due to asynchronous log shipping.</td></tr><tr><td>MAXIMUM PROTECTION</td><td>SYNC+AFFRIM</td><td>For transactions on the primary database, redo data is synchronously transmitted and written to the standby database redo log on disk before the transaction is acknowledged. If the standby database becomes unavailable, the primary database shuts itself down to ensure transactions are protected.</td></tr><tr><td rowspan="2">MAXIMUM AVAILABILITY</td><td>SYNC+AFFRIM</td><td>Similar to <code>MaxProtection</code> mode, except when no acknowledgement from standby database, it operates as if it were in <code>MaxPerformance</code> mode to preserve the primary database availability until it’s able to write its redo stream to a synchronized standby database again.</td></tr><tr><td>SYNC+NOAFFRIM</td><td>For transactions on the primary database, redo is synchronously transmitted to the standby database, and the primary waits only for acknowledgement that the redo has been received on the standby, not that it has been written to standby disk. This mode, which is also known as <code>FastSync</code>, can provide a performance benefit at the expense of potential exposure to data loss in a special case of multiple simultaneous failures.</td></tr></tbody></table>Read replicas in RDS Custom for Oracle are created with maximum performance protection mode, which is also the default protection mode for Oracle Data Guard. It provides the lowest performance impact on the primary database, which can help you meet the RPO requirement measured in seconds. With FSFO <a href="https://docs.oracle.com/en/database/oracle/oracle-database/19/dgbkr/oracle-data-guard-broker-properties.html#GUID-DFE7305F-318B-4DE2-99D5-76E4696843D7" target="_blank" rel="noopener noreferrer">FastStartFailoverLagLimit</a> configuration, you can control the failover to the standby database based on the lag between primary and standby to further help you meet your RPO requirements.To work to achieve a zero data loss (RPO=0) objective, you can customize the Data Guard protection mode to <code>MaxAvailability</code> with the <code>SYNC+NOAFFIRM</code> setting for redo transport for better performance. Because commits on the primary database are acknowledged only after the corresponding redo vectors are successfully transmitted to the standby database, the network latency between the primary instance and replica can be crucial for commit-sensitive workloads. Deploying the read replica in the same Availability Zone as the primary database provides lower network latency compared to a read replica in a different Availability Zone. However, the same Availability Zone read replicas may not meet your availability requirements because in the unlikely event of Availability Zone unavailability, both the primary instance and read replica instance are impacted. It’s a recommended practice to perform load testing for your workload to assess the performance impact when the read replica is customized to run in <code>MaxAvailability</code> mode.Follow these steps to change Data Guard protection mode to <code>MaxAvailability</code>.<ol><li>Access the Data Guard broker configuration on Primary (<code>orcl1</code>) via the Data Guard Command Line Interface (DGMGRL).You can find the net service name for your database in the <code>tnsnames.ora</code> file located in the <code>$ORACLE_HOME/network/admin</code> directory. RDS Custom for Oracle automatically populates these entries for your primary database and your read replicas. The password for the <code>RDS_DATAGUARD</code> user is stored in AWS Secrets Manager, with secret name <code>do-not-delete-rds-custom-+<<RDS Resource ID>>+-dg</code>. Refer to Connecting to your RDS Custom DB instance using SSH for more information on how to connect to a RDS Custom EC2 instance using the SSH key retrieved from Secrets Manager.</li><li>Verify the current protection mode and log transport setting:</li><li>Change log transport mode to <code>FastSync</code>, corresponding to the redo transport setting <code>SYNC+NOAFFIRM</code>. Change it for both the primary database (<code>orcl_a</code>) and standby database (<code>orcl_b</code>) to check that you have valid settings after the role switch.</li><li>Change the protection mode to <code>MaxAvailability</code>:</li></ol><h2>Change replica status from mounted to read-only</h2>Using a replica in read-only mode requires the Active Data Guard option which comes at an extra cost because it is a separately licensed feature of the Oracle database Enterprise Edition.Having your read replica in read-only mode provides additional benefits:<ul><li>Offloads read-only workloads from the primary database</li><li>Enables automatic repair of corrupted blocks by retrieving healthy blocks from the standby database using the Active Data Guard feature</li><li>Uses the far sync capability to keep the remote standby database in sync without the performance overhead associated with long-distance redo log transmission</li></ul>The read replica is created in MOUNT mode by default. Follow these steps to open it in read-only mode.<ol><li>Access the Data Guard broker configuration on Read Replica (<code>orcl2</code>) via <code>DGMGRL</code>.</li><li>Stop redo apply for the standby database <code>orcl_b</code>.</li><li>Connect to the standby database <code>orcl_b</code> using the TNS entry <code>RDS_CUSTOM_ORCL_B</code> and open it in read-only mode.</li><li>Connect to Data Guard broker configuration on read replica using DGMGRL and enable redo apply.</li></ol><h2>Role transition (Data Guard switchover)</h2>The primary database and one of the read replica databases can perform role transition. During this process, the databases in the Data Guard configuration continue to function in their new roles. Typical use cases for Data Guard switchover are disaster recovery drills, scheduled maintenance activities on databases, and <a href="https://docs.oracle.com/en/database/oracle/oracle-database/19/sbydb/upgrading-patching-downgrading-oracle-data-guard-configuration.html#GUID-A5226768-DB6B-4714-BB9A-0A3EF17A01C8" target="_blank" rel="noopener noreferrer">Standby-First rolling patches</a>.Follow these steps to perform Data Guard switchover with a read replica.<ol><li>Pause RDS Custom database automation for both <code>orcl1</code> and <code>orcl2</code>.There might be database restarts during the switchover; this depends on the existing Data Guard role and database open mode. Refer to <a href="https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/custom-managing.html#custom-managing.pausing" target="_blank" rel="noopener noreferrer">Pausing and resuming RDS Custom automation</a> for steps on how to pause and resume RDS Custom database automation. Although the RDS Custom automation framework doesn’t interfere with the role transition process, it’s a good practice to pause automation during Data Guard switchover.</li><li>Use <code>DGMGRL</code> to log in to database <code>orcl1</code> with user <code>RDS_DATAGUARD</code>, and check the Data Guard status.</li><li>On the Amazon RDS console, check the instance role (primary and replica) in the database’s Replication section on the Connectivity & security tab.The primary role should match the Data Guard primary database, and the replica role should match the Data Guard physical standby database. Note that the instance role shown on the Amazon RDS console is only updated correctly for read replicas managed by RDS Custom.<a href="https://d2908q01vomqb2.cloudfront.net/887309d048beef83ad3eabf2a79a64a389ab1c9f/2022/09/21/image008-2.jpg"><img class="alignnone size-full wp-image-24737 c6" src="https://d2908q01vomqb2.cloudfront.net/887309d048beef83ad3eabf2a79a64a389ab1c9f/2022/09/21/image008-2.jpg" alt="role transition screenshot1" width="1070" height="233" /></a></li><li>Perform the switchover:</li><li>Verify Data Guard status.You might see <code>WARNING</code> instead of <code>SUCCESS</code> for the configuration status if you run the show configuration command right after switchover. In an ideal scenario, the configuration returns to a healthy state within a minute.The Amazon RDS console reflects the new roles after you perform the role switch. It may take a few minutes for the Replication state to update from empty to Replicating.<a href="https://d2908q01vomqb2.cloudfront.net/887309d048beef83ad3eabf2a79a64a389ab1c9f/2022/09/21/image009-1.jpg"><img class="alignnone size-full wp-image-24738" src="https://d2908q01vomqb2.cloudfront.net/887309d048beef83ad3eabf2a79a64a389ab1c9f/2022/09/21/image009-1.jpg" alt="role transition screenshot 2" width="1061" height="221" /></a></li><li>Resume RDS Custom database automation for <code>orcl1</code> and <code>orcl2</code>.</li><li>Change your application configuration to use the new primary database endpoint for the read/write workloads, or refer to the best practices discussed later in this post to avoid such changes.</li></ol><h2>Data Guard failover</h2>During the Data Guard failover process, the standby database (read replica) is converted to a primary database when the original primary database fails. This is a manual process when the Fast-Start Failover (FSFO) is not configured and there may or may not be data loss depending upon on the Data Guard configuration. The failover process doesn’t remove the primary database from the Data Guard broker configuration and it allows the former primary database to be reinstated when the Flashback database feature is enabled.If you want to test failover without having to restore from backup or re-create the replica after failover, enable the flashback database as explained in the following steps.<ol><li>Pause RDS Custom for Oracle database automation for databases <code>orcl1</code> and <code>orcl2</code>, and disable FSFO if it has been enabled previously.</li><li>Log in to the Amazon RDS for Custom underlying EC2 instances running <code>orcl1</code> and <code>orcl2</code> and create a directory using user <code>rdsdb</code>.</li><li>Enable the flashback database on both databases while the database is in MOUNT state:The code has the following parameters:<ul><li>dg_broker_start=false – Disables Data Guard broker automatic start to prevent the broker from automatically opening the database for the primary database and starting the managed recovery process for the standby database.</li><li>db_recovery_file_dest – The Oracle database stores database flashback logs in the fast recovery area (FRA) defined by this parameter.</li><li>db_flashback_retention_target – How far (in minutes) we need the database to rewind. The default value is 1440. To support our FSFO use case, 60 minutes usually is sufficient with maximum availability protection mode.</li><li>db_recovery_file_dest_size – The size of the FRA. Use the column <code>ESTIMATED_FLASHBACK_SIZE</code> in the view <code>V$FLASHBACK_DATABASE_LOG</code> to estimate the flashback database log space requirement during the peak workload.</li></ul></li><li>Enable Data Guard broker automatic start to allow the broker process to automatically open the database for the primary database and start the managed recovery process for the standby database:</li><li>When the process is complete, resume RDS Custom database automation for <code>orcl1</code> and <code>orcl2</code>, and enable FSFO if it was disabled previously.</li></ol>To test failover, or in the event of primary database failure, you can fail over to the read replica manually using the following steps:<ol><li>Pause RDS Custom database automation for <code>orcl2</code>, the read replica instance that acts as the target for failover.</li><li>Use DGMGRL to log in to database <code>orcl_b</code> with user <code>RDS_DATAGUARD</code>, and check the Data Guard status. Make sure you log in to the target database, which is the failover target.</li><li>Start the failover:With the IMMEDIATE option, no attempt is made to apply any unapplied redo that has been received. This helps in the event that the failover command fails with <code>Error: ORA-16472: failover failed due to data loss</code>.</li><li>Check the Data Guard status:The Data Guard configuration is in a <code>WARNING</code> state because there’s no working physical standby for the new primary database. There are a number of ways to reinstate the standby database, including the following:<ul><li>Use RMAN backup and restore or RMAN DUPLICATE from the active database.</li><li>Use the Oracle flashback database option if enabled.</li><li>On the Amazon RDS console, drop the old primary database (<code>orcl1</code>) and recreate a replica for <code>orcl2</code>. Note that in this case, the unique DB name takes the next running sequence name if possible. It might become <code>ORCL_C</code> instead of reusing the existing <code>ORCL_A</code>.</li></ul></li><li>Resume RDS Custom database automation for <code>orcl2</code>.</li><li>Change your application configuration to use the new primary database endpoint for the read/write workloads, or refer to the best practices discussed later in this post to avoid such changes.</li></ol><h2>Promote the read replica</h2>The read replica can be promoted to a standalone database that can support read/write workloads by choosing the Promote option on the Amazon RDS Console, the AWS CLI command <a href="https://docs.aws.amazon.com/cli/latest/reference/rds/promote-read-replica.html" target="_blank" rel="noopener noreferrer">promote-read-replica</a>, or the <a href="https://docs.aws.amazon.com/AmazonRDS/latest/APIReference/API_PromoteReadReplica.html" target="_blank" rel="noopener noreferrer">PromoteReadReplica</a> Amazon RDS API operation.As part of this read replica promotion, Amazon RDS Custom for Oracle removes Data Guard configuration for <code>orcl2</code> in both the primary (<code>orcl1</code>) and the Read Replica (<code>orcl2</code>). The promotion process completes even if the primary (<code>orcl1</code>) is not in “Available” status. Amazon RDS Custom for Oracle removes Dataguard the configuration for orcl2 in the former primary (<code>orcl1</code>) once it comes back to “Available” status.To promote an RDS Custom for Oracle replica (<code>orcl2</code>) using the Amazon RDS Console:<ol><li>On the Amazon RDS console, choose Databases in the navigation pane.</li><li>Choose the RDS Custom for Oracle replica (<code>orcl2</code>) that you want to promote.</li><li>For Actions, choose Promote.</li><li>On the Promote Oracle Replica page, enter the backup retention period and the backup window for the newly promoted DB instance, then choose Promote Read Replica.</li></ol><img class="alignnone wp-image-24739 size-full c6" src="https://d2908q01vomqb2.cloudfront.net/887309d048beef83ad3eabf2a79a64a389ab1c9f/2022/09/21/image010-3.png" alt="promote replica" width="783" height="844" />After a few minutes, the role for <code>orcl2</code> will change from Replica (RDS Custom) to Instance (RDS Custom). If <code>orcl2</code> was the only replica for the Primary (<code>orcl1</code>), the role for <code>orcl1</code> will change from Primary (RDS Custom) to Instance (RDS Custom) too.Alternatively, you can promote a read replica from the AWS CLI, as shown in the following example:<h2>Configure Fast-Start Failover</h2>FSFO monitors the health of instances in a Data Guard configuration using an observer process and automatically initiates failover to a standby database in the event of a primary instance failure. When there are multiple standby databases for the primary database, you can specify the list of databases as failover candidates via <a href="https://docs.oracle.com/en/database/oracle/oracle-database/19/dgbkr/oracle-data-guard-broker-properties.html#GUID-63108DF8-4DAD-40F1-88B1-458C3FCB6C84" target="_blank" rel="noopener noreferrer">FastStartFailoverTarget</a> configuration.An observer process is a <code>DGMGRL</code> client component, typically running in a different machine from the primary and standby databases. The ORACLE HOME installation for the observer can be an Oracle Client Administrator installation, or you can install either an Oracle Database Enterprise Edition or Personal Edition. Refer to <a href="https://docs.oracle.com/en/database/oracle/oracle-database/19/dgbkr/using-data-guard-broker-to-manage-switchovers-failovers.html#GUID-11EF3897-8FCA-4A54-B63B-E8C1668AE21B" target="_blank" rel="noopener noreferrer">Installing and Starting the Observer</a> for observer installation for your database release. To provide high availability for the observer process, you might want to do the following:<ul><li>Enable <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-recover.html" target="_blank" rel="noopener noreferrer">EC2 instance auto-recovery</a> for the EC2 instance running your observer. You need to automate the observer startup process as part of the OS startup.</li><li>Additionally, deploy an observer in the EC2 instance and configure an Amazon EC2 Auto Scaling group with size one (1). In the event of EC2 instance failure, the Auto Scaling group automatically spins up another EC2 instance. You need to automate the observer startup process as part of the OS bootstrap and startup.</li><li>For Oracle 12c Release 2 and above, you can deploy multiple (up to three) observers. One of them is the primary observer, and the rest are backup observers. When the primary observer fails, one of the backup observers resumes the primary role.</li></ul>Follow these steps for configure FSFO for an RDS Custom for Oracle instance with read replicas:<ol><li>Use <code>DGMGRL</code> from the observer host, which is configured with <code>tnsnames.ora</code> entries for the primary and standby databases:Although it’s a supported configuration to enable FSFO with maximum performance protection mode and to allow automatic failover as long as data loss is within the <code>FastStartFailoverLagLimit</code> configuration (value in seconds), you need to use maximum availability protection mode to work to achieve zero data loss (RPO=0).</li><li>Specify one or more standby databases as the failover target. In a configuration with only one standby database (read replica), this step is optional.</li><li>Specify the <a href="https://docs.oracle.com/en/database/oracle/oracle-database/19/dgbkr/oracle-data-guard-broker-properties.html#GUID-B61FE085-699E-4C00-BF09-794D42161C8F" target="_blank" rel="noopener noreferrer">FastStartFailoverThreshold</a> value in seconds, which indicates the duration the observer process waits when the primary database becomes unavailable before initiating the failover.The default value is 30 seconds in Oracle 19c, and the minimum value is 6 seconds. A lower value can potentially shorten the RTO during the failover, whereas a higher value helps reduce the chance of unnecessary failover transient errors on the primary database. The RDS Custom for Oracle automation framework monitors database health and performs corrective actions every few seconds. Therefore, we recommend setting FastStartFailoverThreshold to a value higher than 10 seconds.In the following example, configure the threshold value to 45 seconds:</li><li>Enable FSFO:If your database doesn’t have flashback database enabled, the warning message ORA-16827 appears. Although the flashback database feature is optional for FSFO, it helps automatically reinstate failed primary databases using a flashback database to a point in time before failover if the <a href="https://docs.oracle.com/en/database/oracle/oracle-database/19/dgbkr/oracle-data-guard-broker-properties.html#GUID-824E97C0-EEB0-4E1B-BD4A-F5AE282CEA28" target="_blank" rel="noopener noreferrer">FastStartFailoverAutoReinstate</a> configuration property is set to <code>TRUE</code> (which is the default).</li><li>Check the configuration to confirm FSFO is enabled. The warning message is expected because you haven’t started the observer process.</li><li>Start the observer.You can start the observer before or after you enable FSFO. If FSFO is already enabled, the observer immediately begins monitoring the status and connections to the primary and target standby databases. If FSFO is not enabled, the observer only starts monitoring after FSFO is enabled.To avoid the observer stopping when the terminal running the command is closed, we recommend wrapping the command in a script and starting with the <code>nohup</code> command. From Oracle 12c Release 2 onwards, it’s possible to start an observer in the background. This feature uses Oracle Wallet to store the credentials to log in to target databases. In the following example, <code>rds_custom_orcl_a</code> and <code>rds_custom_orcl_b</code> are the keys for Oracle Wallet to retrieve the credentials for the observer. Because it’s going to run in background mode, you can press Enter to get the <code>DGMGRL</code> command prompt after the start observer command is complete.The following are the quick steps to configure Oracle Wallet on the machine running your Oracle Data Guard observer. For complete steps, refer to <a href="https://docs.oracle.com/en/database/oracle/oracle-database/19/dbseg/configuring-authentication.html#GUID-3EA07020-A9F3-4FF9-9518-E1AEA3BDDBBE" target="_blank" rel="noopener noreferrer">Configuring a Client to Use the External Password Store</a>.The first step is to create Oracle Wallet and credentials. For this post, <code>rds_custom_orcl_a</code> is your connection identifier, and <code>rds_dataguad</code> is the user name. Enter the password in the command when prompted.Add the following lines to <code>$ORACLE_HOME/network/admin/sqlnet.ora</code>:</li><li>Check the observer status:</li><li>Check the Data Guard and FSFO status, and use <code>VERBOSE</code> to get detailed information about Data Guard configuration and status:</li><li>Optionally, perform a failover test by manually stopping the EC2 instance for <code>orcl1</code>. Before stopping the EC2 instance, use the tail command to monitor the observer log file.</li></ol>Use <code>DGMGRL</code> to log in to database <code>orcl2</code> (with DB unique name <code>orcl_b</code>) with user <code>RDS_DATAGUARD</code>, and check the Data Guard status. It shows that <code>orcl2</code> is our new primary database.If the flashback database has been enabled on <code>orcl1</code> prior to failover, after the former primary database returns online and starts in <code>MOUNT</code> state, the observer reinstates it into a new standby database. The reinstated database acts as the FSFO target for the new primary database. The observer log is as follows:<h2>Seamless Oracle Client connectivity</h2>You can avoid application configuration changes during a role transition by using a TNS entry that contains both the primary and standby database endpoints in the configuration. You can define two role-based database services to support both read/write and read-only workloads. In the following example, <code>orcl_rw</code> is the read/write service that’s active on the primary database. <code>orcl_ro</code> is the read-only service, and is active on the standby database that has been opened in read-only mode.Run the following SQL statements in the primary database to create these services:Run the following SQL statement in the primary database to start the <code>orcl_rw</code> service:Run the following SQL statement in standby databases running in read-only mode to start the <code>orcl_ro</code> service:To automatically start these services while the database is restarted, create the following trigger in the primary database:The following is a sample configuration for a read/write workload:The following is a sample configuration for a read-only workload:Refer to <a href="https://docs.oracle.com/en/database/oracle/oracle-database/19/dgbkr/using-data-guard-broker-to-manage-switchovers-failovers.html#GUID-DFFDAA2B-A889-49AD-AB85-747D73FF0FF5" target="_blank" rel="noopener noreferrer">Oracle Data Guard Specific FAN and FCF Configuration Requirements</a> for other Oracle client considerations to improve failover time during failover and switchover operations.<h2>Cross-Region Data Guard configuration</h2>To help you meet your disaster recovery requirements, you may need to deploy a standby database in a different Region or in an environment outside of AWS, such as on-premises or other cloud services. For such configurations, you can manually set up a self-managed physical standby database (managed replicas can’t be created outside your primary Region as of this writing).However, managed replicas and self-managed standby databases can be managed by a single Data Guard broker configuration. The following architecture shows one such configuration for a primary RDS Custom for Oracle instance with a managed replica in the same Region and a cross-Region self-managed physical standby database.<img class="alignnone wp-image-24740 size-full" src="https://d2908q01vomqb2.cloudfront.net/887309d048beef83ad3eabf2a79a64a389ab1c9f/2022/09/21/image011-2.png" alt="cross region" width="1624" height="741" />Refer to <a href="https://d1.awsstatic.com/whitepapers/enabling-high-availability-with-data-guard-on-amazon-rds-custom-for-oracle.pdf" target="_blank" rel="noopener noreferrer">Enabling High Availability with Data Guard on Amazon RDS Custom for Oracle</a> for more details on creating an RDS Custom for Oracle standby database in a remote Region and maintaining it as a self-managed physical standby database for a primary RDS Custom for Oracle instance.Refer to the <a href="https://docs.oracle.com/en/database/oracle/oracle-database/19/dgbkr/index.html" target="_blank" rel="noopener noreferrer">Oracle Data Guard Broker guide</a> and <a href="https://docs.oracle.com/en/database/oracle/oracle-database/19/sbydb/index.html" target="_blank" rel="noopener noreferrer">Data Guard Concepts and Administration</a> for details on how to configure and manage a Data Guard environment with multiple standby databases.<h2>Conclusion</h2>Amazon RDS Custom for Oracle allows you to create read replicas that are built on Oracle Data Guard technology to offload read-only workloads from your primary instance. In this post, we discussed options to customize the default configuration of Oracle Data Guard deployed by a read replica to help you meet your specific RTO and RPO requirements for your critical applications hosted on RDS Custom for Oracle. We also reviewed best practices and step-by-step instructions to follow during role transition and failover scenarios.If you have any comments or questions, leave them in the comment section.<h3>About the author</h3><a href="https://d2908q01vomqb2.cloudfront.net/887309d048beef83ad3eabf2a79a64a389ab1c9f/2022/09/21/Jobin-Joseph.jpg"><img class="size-full wp-image-24744 alignleft" src="https://d2908q01vomqb2.cloudfront.net/887309d048beef83ad3eabf2a79a64a389ab1c9f/2022/09/21/Jobin-Joseph.jpg" alt="" width="100" height="133" /></a>Jobin Joseph is a Senior Database Specialist Solution Architect based in Dubai. With a focus on relational database engines, he assists customers in migrating and modernizing their database workloads to AWS. He is an Oracle Certified Master with 20 years of experience with Oracle databases.<a href="https://d2908q01vomqb2.cloudfront.net/887309d048beef83ad3eabf2a79a64a389ab1c9f/2022/09/22/Donghua-Luo.png"><img class="size-full wp-image-24789 alignleft" src="https://d2908q01vomqb2.cloudfront.net/887309d048beef83ad3eabf2a79a64a389ab1c9f/2022/09/22/Donghua-Luo.png" alt="" width="100" height="133" /></a>Donghua Luo is a Senior RDS Specialist Solutions Architect. He works with AWS customers to migrate and achieve higher flexibility, scale, and resiliency with database services in AWS cloud.</section>

Fish AI Reader

FishAI

联系邮箱 441953276@qq.com

相关标签