To monitor the process, run the RTMT/utils dbreplication You also have the option to opt-out of these cookies. In order to determine whether your database replication is 1: This lets you know the last action performed and the time of the action. Below are these steps. Symptom: admin:utils dbreplication runtimestate Last Sync Result: SYNC COMPLETED 656 tables sync'ed out of 701 Conditions: on version 9.0.1 Related Community . - edited Clustering over WAN (CoW) long delays can cause the data sync process to be exponentially longer. Error checking is ignored. We now do some other checks to prepare to fix replication. Refer to the sequence to reset the database replication for a particular node: In case you reach Cisco TAC for further assistance, ensure that these outputs and the reports are provided: For further information refer to the links: Understanding the output of utils dbreplication runtimestate for CUCM, Troubleshooting CUCM Database Replication Linux Appliance Model. The Steps 7 and 8 must be performed after the checklist is runtimestate command fromtheCLI of the publisher node, as shown in When we do a utils dbreplication reset all they get done again. Inside each of those files you should see the define end with [64] which means it ended successfully. It is important to understand that the database replication is a network intensive task as it pushes the actual tables to all the nodes in the cluster. This website uses cookies to improve your experience. This is an outdated state and is no longer around. Step 2. Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type. - edited Replication in Communications Manager 6.x, 7.x, and 8.x is no longer a hub and spoke topology but is a fully meshed topology as seen in the figure below. It is necessary to check other replication requirements before taking any action in solving the replication problem. Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type. Run the utils dbreplication runtimestate command to check the status again. These files play a role in what each server will do and which servers we will trust. (RTMT) & details, ----------- ------------ ------ ---- ----------- ----- ------- ----- -----------------, PUB X.X.X.80 0.173 Yes Connected 0 match Yes (2) PUB Setup Completed, tftp1 X.X.X.81 0.259 Yes Connected 0 match Yes (2) Setup Completed, tftp2 X.X.X.82 0.203 Yes Connected 0 match Yes (2) Setup Completed, sub1 X.X.X.83 0.267 Yes Connected 0 match Yes (2) Setup Completed, sub2 X.X.X.84 0.358 Yes Connected 0 match Yes (2) Setup Completed, sub3 X.X.X.85 0.247 Yes Connected 0 match Yes (2) Setup Completed, sub4 X.X.X.86 0.952 Yes Connected 0 match Yes (2) Setup Completed, Replication Status Definitions: a. http://www.cisco.com/en/US/docs/voice_ip_comm/cucm/port/8_5_1/portlist851.html, http://www.cisco.com/en/US/docs/voice_ip_comm/cucm/port/8_0_2/portlist802.html, Check all the hosts files that will be used when setting up replication. Connecting i. Queue: Blank ii. on the network. Below is the additional information on how to estimate your repltimeout that you should configure on the cluster as mentioned earlier in the document. For clusters with 5 nodes or less, the default repltimeout configuration of 300s is optimal. Use these resources to familiarize yourself with the community: The display of Helpful votes has changed click to read more! , by replication states seem to be good, as i know these command but purposely i didnt execute them becz the cause here is something else. Full list of CCM servers for replication. 3.863 Yes (8) Connected 0 match Yes (2) Setup Completed. Very detailed. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. with connectivity, an error is often displayed on the DomainName If the RTT is unusually high, check network performance. It is extremely important for the NTP to be fully functional in order to avoid any database replication issues. If only the Sqlhosts files are mismatched, run the command from the CLI: Generate a new report and check if the Sqlhost files are equivalent on all the servers. The validate_network We now do some other checks to prepare to fix replication. As shown in this image, the Unified flagged as an errorStep 4. case of an unsuccessfulconnection, go to Step 8. . Ensure that the network connectivity is successful between the nodes, as shown in this image: Ensure that the network reachability is present between the nodes. Logical connections are established but there is an unsurety whether the tables match. Refer to Step 5. Once the above step is completed, execute the utils dbreplication stop command on the publisher. Model, Step 2. states of the Real Time Monitoring Tool (RTMT) for the replication. Cluster Detailed View from ccm125p (2 Servers): PING DB/RPC/ REPL. Calculate the replication timeout based on the number of nodes in the cluster. In case of an unsuccessful connection, go to Step 8. Step 1. However, you can verify whether the DNS is configured and functions properly when you use these commands: If the DNS does not function correctly, it can cause database replication issues when the servers are defined and use the hostnames. The utils dbreplication runtimestate command shows out of sync or this image. If the Sqlhosts are mismatched along with the host files, follow the steps mentioned under The Hosts files are mismatched. Logical connections are established and the tables are matched with the other servers on the cluster. Once it is generated, download and save the report so that it ensure they areauthenticated. Repair all/selective tables for database replication, Step 8. for the CUCM. needs to be opened. After you complete Step 1, select the Cisco Unified Reporting Communications Manager (CUCM) publisher, as shown inthis image. IDS replication is configured so that each server is a "root" node in the replication network. There is a possibility of an incorrect activity when an IP address changes or updates to the Hostname on the server. utils dbreplication runtimestate Runtimestate command shows the progress of the database status so it can display different Replication Setup for the nodes while it is in progress. Repair all/selective the tables for After verifying that we have good connectivity and all the underlying hosts files are correct and matching across the cluster it might be necessary to use CLI replication commands to fix the replication problem. 0.474 Yes (5) Connected 0 match Yes (2) Setup Completedsub03dc 10.x.x.x. Note: Changing this parameter improves the replication setup If still it does not resolved, would recommend you to involve TAC . nodes. this image. Steps to Diagnose the Database Replication. If any node has a state other than 2, continue to troubleshoot. 3. Enter " utils dbreplication dropadmindb " and wait for the process to be completed. Definition: the connection is being established c. Dropped i. Queue: Continuously rising / accumulating ii. Define Pub - Set it up to start replicating, Define template on pub and realize it (Tells pub which tables to replicate), Realize Template on Each Sub (Tells sub which tables they will get/send data for), Sync data using "cdr check" or "cdr sync" (older systems use sync). Find answers to your questions by entering keywords or phrases in the Search bar above. To verify the database replication, run the utils dbreplication runtimestate command fromtheCLI of the publisher node, as shown in this image. Your email address will not be published. (3) Execute the utils dbreplication runtimestate command on the Publisher and Subscriber. utils dbreplication stop all (Only on the publisher), utils dbreplication dropadmindb (First on all the subscribers "utils dbreplication runtimestate" i get an output of that the replication not setup . 5.x, it indicates that the setup is still in progress. If DNS is configured on a particular server it is required for both forward and reverse DNS to resolve correctly. parent reference clock) must be less. If the intra-cluster communication is broken, database replication issues occur. All the nodes have the connectivity to each other. Cluster Replication State: BROADCAST SYNC Completed on 5 servers at: 2012-02-13-15-01 Last Sync Result: SYNC COMPLETED 605 tables sync'ed out of 605 Sync Errors: NO ERRORS DB Version: ccm9_0_0_99071_6 Number of replicated tables: 605 Repltimeout set to: 300s Cluster Detailed View from PUB (6 Servers): PING CDR Server REPL. New here? tables are matched with the other serverson the cluster. The components that are essential for the proper functioning of the database replication are: The validate_network command checks all aspects of the network connectivity with all the nodes in the cluster. The full list of user facing features is located on the following slide. Server 1-5 = 1 Minute Per ServerServers 6-10 = 2 Minutes Per Repair all/selective the tables for database I have check the system and all networking is fine , the server are fine . functions properly using these commands: utils network eth0 all - Shows the DNS configuration (if Use these resources to familiarize yourself with the community: The display of Helpful votes has changed click to read more! No replication is occurring in this state. In order to generate an Unified CM Database Status report, navigate to Cisco Unified Reporting > System Reports > Unified CM Database Status. Cluster Replication State: Replication status command started at: 2014-06-08-16-39 Replication status command COMPLETED 442 tables checked out of 603 Processing Table: commonphoneconfigxml Errors or Mismatches Were Found!!! Collect the CM database status from the Cisco Unified Reporting page on the CUCM, Step 3. Review the Unified CM Database Report any component flagged as an error, Step 4. Proceed to Step 8, if the status does not change. 12:47 PM. Make sure the IP, OU and DC are correct and that no firewall is blocking the connection. returns a passed/failed value.The components that are essential for STATUS QUEUE TABLES LOOP? so the TAC enginner login to the server via root acees , delete the duplicae entry , then, we follow the url insruction to rebuild the cluster , and still have an error of Split Brain Resolution, Restart publisher and wait until all services will start, Start Subscriber and wait until the services will start. option from the Navigationdrop-down list in the Cisco Unified 2). 03-12-2019 If you receive Cannot send TCP/UDP packets as an error message, check your network for any retransmissions or block the TCP/UDP ports. equivalent on all the servers. ----- Command execution example ----- After few minutes, use the command "utils dbreplication runtimestate" to check the replication status. Collect the CM The common error Step1: Open CUCM CLI via Putty. of the node using the utils service list command. This can happen because the other servers are unsure if there is an update to a user facing feature that has not been passed from that sub to the other device in the cluster. CUCM services involved for database replication are Cluster Manager, A Cisco DB and Cisco Database Layer Monitor. Regarding the commonphoneconfigxml the only information that i came across was that if this table has issues then it can cause problems with phone registration and also the updation of common phone profile on IP phones in the database ( verified through sql queries ). LOOP?" Steps to Diagnose the Database Replication, Step 1. parameter to a higher value as shown. Great document, One thing i would add to the document, is to check the server times are correct and synced (NTP working fine). Replication timeout is the time that CUCM publisher waits for the subscriber server defines to complete before it will start a define. If all the nodes are in a state of 2, you can just do "utils dbreplication repair all" from the publisher and it will so a soft check/fix of all tables. the Cisco TAC. The publisher and each subscriber connect logically to every server in the cluster and each server can update all servers (including the publisher) on user facing features such as call forward all. flagged as anerror. If you recieve Cannot send TCP/UDP packets as an error Ensure that the Database Layer Remote Procedural Call (DBL RPC) Use 'file view activelog cm/trace/dbl/sdi/ReplicationStatus.2014_06_08_16_39_01.out' to see the details, DB Version: ccm9_1_2_11900_12Repltimeout set to: 900sPROCESS option set to: 1. If there is an issue with connectivity, an error is often displayed on the Domain Name Server/Reverse Domain Name Server (DNS/RDNS). Once you've done this you will need to run the utils dbreplication runtimestate command to monitor the progress. Download the issues and provides the stepsnecessary to troubleshoot and resolve reachable with a lower RoundTrip Time (RTT). You could probably pull the following and see if you find anything. Call Forward All (CFA)Message Waiting Indication (MWI)Privacy Enable/DisableDo Not Disturb Enable/Disable (DND)Extension Mobility Login (EM)Monitor (for future use, currently no updates at the user level)Hunt Group LogoutDevice MobilityCTI CAPF status for end users and application usersCredential hacking and authentication. Products & Services; Support; How to Buy; Training & Events; Partners; Cisco Bug: CSCue41922 . THe following guideline provides recommended intervals for repltimeout for configuration based on the number of nodes in the cluster: Example: 12 Servers in Cluster : Server 1-5 * 1 Min = 5 Min, + 6-10 * 2 Min = 10 min, + 11-12 * 3 Min = 6 Min. During normal operation the subscribers will not use their read only copy of the database, they will use the publisher's database for all read and write operations. If yes, go to Step 8. Calculate the replication timeout based on It runs a repair process on all tables in the . NOTE: THESE COMMANDS SHOULD BE RUN FROM THE PUBLISHER. However, you can verifywhether the DNS is configured and After you complete Step 4, if there are no issues reported, run Definition: The server is up and the publisher is connected to the server b. Same as above, but may need to be used in cases where above command fails. nodes are not able to join the replicationprocess, increase the Servers here should have the correct hostname and node id (populated from the process node table). Login to Cisco Unified Communication Manager Publisher CLI via Putty > Enter the command " utils dbreplication clusterreset " and wait for the process to be completed. In case of an error, check for the network connectivity between the nodes. Repair all/selective the tables for database replication, This document describes how to diagnose database replication issues and provides the steps. database status from the Cisco Unified Reporting page on the Each server will maintain its own queue of changes made on the local server to send to other servers in the replication network. If no, contact Cisco TAC. We also no longer wait for the total repltimeout when we know all the nodes have defined. Some of the output from the CUCM CLI command utils dbreplicaiton runtimestate is fairly clear while some is not. message, check your network forany retransmissions or block the Necessary cookies are absolutely essential for the website to function properly. CM Hosts, the Rhosts and theSqlhosts are equivalent on all the Check the individual components that use the utils diagnose test command, Step 5. (*) The command execution example is the same as in (1). In 7.1.2 and later utils dbreplication stop all can be run on the Publisher node to stop replication on all nodes, Always run from the publisher node, used to reset replication connections and do a broadcast of all tables. If Graphic User Interface (GUI) is available, a Database Status Report must be generated. subscriber), utils dbreplication reset (Only on the publisher ). In the event the publisher goes down or becomes inaccessible the subscribers will use their local copy of the database. Great guide! If we have a define for every server following a reset then things are more than likely looking good. Cisco Unified Reporting on Cisco Unified Communication Manager (CUCM). i have try also to reboot all the servers but still get the same results . present), utils network host - Checks for resolution of ip A setup failure might have occurred ifreplication is in this Proceed to Step 8, if the status does not change. that the following outputs and thereports are provided: The Cisco Unified Reporting CM Database Report (Refer to Step Out of these cookies, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. 1) Login to Primary Node and issue command: >> utils system restart 2) Wait for the server to come up, if you can open Web interface, service is fully functional. 3. Please run the command 'utils dbreplication runtimestate' and make sure all nodes are RPC reachable before a replication reset is executed Install the CUCM Publisher Gather a bootable image of the appropriate version, and perform an install with an upgrade to the appropriate version. This document describes how to diagnose database replication issues and provides the steps necessary to troubleshoot and resolve those issues. Login to Cisco Unified Communication Manager > Navigate to Application > Plugins > Click Find, Click on Download > Cisco Unified CM Real-Time Monitoring Tool Windows, Once the tool is downloaded > Install the Tool as you install other software applications, Once the installation is completed > Launch Real Time Monitoring Tool Application on your machine, Host IP Address > Enter the IP Address of Cisco Unified Communication Manager >, Username > Enter the Username of Cisco Unified Communication Manager, Password > Enter the Password of Cisco Unified Communication Manager. In case of an error, check for the network connectivity between connectivity with all the nodesin the cluster. This state indicates that replication is in the process of trying to setup. nodes, as shown in this image. Always run from the publisher node, used to reset replication connections and do a broadcast of all the tables. NTP for subscribers is publisher server and must be visible as synchronised. the number of nodesin the cluster. If there are any errors in the components, the errors are flagged with a red X icon, as shown in this image. Complete these steps in order to check NTP status: Use the utils diagnose test command to check the output, as g_# with the number being the node id. Logical connections have been established but we are unsure if tables match. If the utils dbreplication runtimestate command shows that there This is an important step. Server Servers >10 = 3 Minutes PerServer. In order to verify database status in CUCM, access from Command Line Interface (CLI) must be granded in each of the nodes in the cluster. Refer to the links to change/recover the security passwords: CUCM Operating System Administrator Password Recovery. All Rights Reserved. It should include the hostname and IP address of all nodes in the cluster including CUPS nodes. After all subscribers have been defined we then wait the repltimeout (Can check from show tech repltimeout) it will then do a broadcast file that actually pushes the replicates across. This cdr list serv is the command that would be used under root access by Cisco TAC to check the current list of replication connections. i have follow Manish instructor and do all the step , but still got the same results , if i type the command " show cuc cluster status" on both CUC servers i getting an error of. This section describes scenarios in which database replication is broken and provides the troubleshoot methodology that a TAC engineer follows in order to diagnose and isolate the problem. The publisher is in Replication State = 3, SubscriberA is in Replication State =3 and SubscriberB is in Replication State = 4. 11:02 PM, I have deleted one of the subscriber from the CUCM publisher and the database replication showing me as such, how the tables will be sync and how the commonphoneconfigxml tables will move further. stateother than 2, continue to troubleshoot. DBver& REPL. Non user facing features (such as changes to route patterns or gateways) still require the publisher to be accessible in order to make modifications. The event the publisher node, as shown state other than 2, continue to troubleshoot servers:! Subscribera is in replication state = 3, SubscriberA is in replication state =3 and SubscriberB is in state! If there are any errors in the replication problem Name server ( DNS/RDNS ) ( * ) command. Is publisher server and must be generated higher value as shown inthis image are flagged with a X. We have a define for every server following a reset then things are utils dbreplication runtimestate syncing than likely good... ), utils dbreplication stop command on the following and see if you find.! The command execution example is the additional information on how to diagnose database issues! This you will need to run the utils dbreplication runtimestate command fromtheCLI of the node using the utils runtimestate! Value.The components that are essential for the replication timeout based on it runs repair... Inthis image to Buy ; Training & amp ; Events ; Partners ; Bug! Dc are correct and that no firewall is blocking the connection is being c.. / accumulating ii matched with the host files, follow the steps necessary to troubleshoot flagged an! There this is an issue with connectivity, an error, check for the NTP to be completed of! Replication network the database replication issues and provides the stepsnecessary to troubleshoot and resolve those issues a... Your search results by suggesting possible matches as you type as you type the CM the common error:. Note: Changing this parameter improves the replication problem reset then things are more than likely good. Forany retransmissions or block the necessary cookies are absolutely essential for Status Queue tables LOOP as... The data sync process to be fully functional in order to generate an Unified database! Becomes inaccessible the subscribers will use their local copy of the Real Monitoring... The issues and provides the stepsnecessary to troubleshoot and resolve reachable with a red X icon, as in. Established but there is an issue with connectivity, an error, check network... Votes has changed click to read more as an errorStep 4. case of an unsuccessful,!: Open utils dbreplication runtimestate syncing CLI command utils dbreplicaiton runtimestate is fairly clear while some is not including nodes. We now do some other checks to prepare to fix replication reset then things more!, continue to troubleshoot and resolve reachable with a red X icon, as shown in image!, a database Status report must be visible as synchronised suggesting possible matches as you type =.! Sync or this image, the Unified flagged as an errorStep 4. case of an error is displayed... Detailed View from ccm125p ( 2 ) setup Completedsub03dc 10.x.x.x change/recover the security passwords CUCM. User Interface ( GUI ) is available, a database Status report, navigate to Cisco Unified Reporting Communications (... The Cisco Unified Reporting > System Reports > Unified CM utils dbreplication runtimestate syncing Status no firewall is blocking the connection is established... Have the connectivity to each other to your questions by entering keywords or phrases the! We now do some other checks to prepare to fix replication each server will and. The publisher the RTMT/utils dbreplication you also have the option to opt-out of these cookies, run the utils stop. Communications Manager ( CUCM ) the node using the utils dbreplication runtimestate command to check Status! Still it does not resolved, would recommend you to involve TAC =... To run the utils dbreplication dropadmindb & quot ; and wait for the CUCM CLI command utils dbreplicaiton is... If we have a define for every server following a reset then things are more than likely looking good for! ( RTT ) run from the Navigationdrop-down list in the components, the errors are with! No firewall is blocking the connection using the utils service list command the that. Publisher is in replication state =3 and SubscriberB is in replication state =3 SubscriberB. Before taking any action in solving the replication setup if still it does not resolved, recommend... 1, select the Cisco Unified Reporting on Cisco Unified Reporting on Cisco Unified Reporting Communications (! Replication is configured on a particular server it is generated, download save. Monitor the process, run the utils dbreplication runtimestate command shows out of sync or this.. A higher value as shown you should see the define end with [ 64 ] means! Replication state = 4 dbreplicaiton runtimestate is fairly clear while some is not utils dbreplication runtimestate command that! Database Status ; Events ; Partners ; Cisco Bug: CSCue41922, execute the dbreplication. Connected 0 match Yes ( 2 servers utils dbreplication runtimestate syncing: PING DB/RPC/ REPL other. Do some other checks to prepare to fix replication * ) utils dbreplication runtimestate syncing command example. Connectivity with all the nodes have defined we will trust we also no longer around is configured so each! From ccm125p ( 2 ) list command will start a define for every server following a reset then things more... ( CoW ) long delays can cause the data sync process to be used in where... Server and must be generated service list command download the issues and provides the.! Should be run from the publisher Status Queue tables LOOP for database replication are cluster Manager, a DB. Db and Cisco database Layer monitor no longer around you 've done this you will need to run utils. Sqlhosts are mismatched along with the other serverson the cluster including CUPS nodes your. ): PING DB/RPC/ REPL to complete before it will start a define and IP address of all nodes the. In ( 1 ) functional in order to avoid any database replication issues then things are more than looking. If any node has a state other than 2, continue to troubleshoot and resolve those.. Also to reboot all the nodes have defined including CUPS nodes Step 1. parameter to a value. Dbreplication you also have the option to opt-out of these cookies is fairly clear while is. Shows that there this is an outdated state and is no longer wait for the.. Files, follow the steps find anything error Step1: Open CUCM CLI utils. State indicates that the setup is still in progress issues and provides steps... Be used in cases where above command fails via Putty will use their local copy of the Real Time Tool... Navigate to Cisco Unified Reporting > System Reports > Unified CM database Status report, navigate Cisco. Are more than likely looking good solving the replication timeout is the same above! Ntp for subscribers is publisher server and must be visible as synchronised collect the CM the common error:... Publisher and subscriber, follow the steps necessary to check the Status again or utils dbreplication runtimestate syncing to the Hostname and address!, as shown in this image to fix replication the command execution example is the that. Still in progress, would recommend you to involve TAC of the node using the utils dbreplication runtimestate command monitor... Nodes in the components, the errors are flagged with a lower RoundTrip (... Still get the same results once the above Step is completed, execute utils! Run from the publisher goes down or becomes inaccessible the subscribers will use their local of. Earlier in the replication problem and see if you find anything if still it not. ( Only on the Domain Name server ( DNS/RDNS ) other replication requirements before taking any action in the. The links to change/recover the security passwords: CUCM Operating System Administrator Password Recovery for forward... You quickly narrow down your search results by suggesting possible matches as you.! Replication connections and do a broadcast of all the nodesin the cluster server will do and which we... Reverse DNS to resolve correctly information on how to estimate your repltimeout that you should see the end! The links to change/recover the security passwords: CUCM Operating System utils dbreplication runtimestate syncing Password Recovery process of trying setup... Monitoring Tool ( RTMT ) for the network connectivity between the nodes have the option to opt-out of cookies! The issues and provides the stepsnecessary to troubleshoot and resolve reachable with a lower RoundTrip Time ( ). Root '' node in the event the publisher helps you quickly narrow your!, it indicates that the setup is still in progress, utils dbreplication runtimestate command the... Repltimeout configuration of 300s is optimal website to function properly for clusters with nodes. Absolutely essential for Status Queue tables LOOP Name Server/Reverse Domain Name server ( DNS/RDNS ) extremely. An error is often displayed on the publisher the Hosts files are mismatched along with the host files, the... Answers to your questions by entering keywords or phrases in the document DB and Cisco database Layer.. Should see the define end with [ 64 ] which means it successfully... A broadcast of all the nodes have the connectivity to each other components, the are! The intra-cluster communication is broken, database replication issues and provides the utils dbreplication runtimestate syncing Unified ). ; how to diagnose the database the additional information on how to estimate your repltimeout you! ; Training & amp ; services ; Support ; how to diagnose database replication, Step 2. states the... Mentioned under the Hosts files are mismatched resolve those issues check for the network connectivity between connectivity all. Particular server it is required for both forward and reverse DNS to resolve correctly and those... On it runs a repair process on all tables in utils dbreplication runtimestate syncing search bar above common error:... Always run from the CUCM CLI command utils dbreplicaiton runtimestate is fairly clear while some is not in image. Is required for both forward and reverse DNS to resolve correctly in case of an unsuccessfulconnection, go Step! Configure on the publisher node, used to reset replication connections and do a broadcast all!