Voice Operating Systems Hot Issues from Cisco TAC

Critical BIOS Update for MCS 7825-I2 fails to boot after Oct 1, 2009 , Fixed CSCta43460

Symptom: Starting October 1, 2009 the MCS 7825-I2 servers will not boot properly. These may run any of the Cisco Unified Communications products.
Conditions: Customer is running an affected product on MCS 7825I2 with BIOS version 1.43 or earlier. details of hardware model and BIOS version can be found by running platform admin CLI command "show hardware".
Workaround: For Windows 2003 based products obtain upgrade from Cisco. For Linux based appliance products obtain upgrade from Cisco. For Windows 2000 based products Cisco will not be releasing a patch as those products have passed End of Software Maintenance in the End of Life process.
Further Information CM5.1(3): If upgrading CM5.1(3) and currently running an Engineering Special (ES) obtain the latest 5.1(3)ES here: http://tools.cisco.com/support/downloads/go/ReleaseType.x?optPlat=&isPlatform=Y&mdfid=280735907&sftType=Unified+Communications+Manager+Updates&treeName=Voice+and+Unified+Communications&modelName=Cisco+Unified+Communications+Manager+Version+5.1&mdfLevel=Software%20Version/Option&treeMdfId=278875240&modifmdfid=null&imname=&hybrid=Y&imst=N Version 5.1(3.7109.1) under the All Releases, Engineering Specials Section. Note this download is a patch only, denoted by "k9-patch-es" in the filename, and will only patch earlier 5.1(3) versions. 5.1(3) train has a special issue, CSCsm37017, that is fixed in 5.1(3)ES but not fixed in 5.1(3g). This fix is present in all 5.1(3)ES after 5.1.3.2104-1. Attempting to upgrade from a 5.1(3)ES that includes this fix to 5.1(3g) will fail with this error: error in installdb_l2.log: 09/24/2009 09:39:27.543 installdb|*ERROR* Error executing "insert into processnodeservice (devicenamemonitorflag,devicetypemonitorflag,enable,filetraceflag,fkprocessnode,includenondevicetraces,isactive,maxfilesize,numfiles,numlines,numminutes,outputdebugstringflag,pkid,priority,restrictserver,tkservice,tracelevel,usercategories,usexml) values ('F','T','T','T','6e85a7a0-669d-4ec1-9d67-1bf4c811d797','F','T',1,10,10000,1440,'F','68e7183f-1340-4f94-928b-3f385de70fae',0,'F',93,63,15,'F'): [Informix][Informix ODBC Driver][Informix]Missing key in referenced table for referential constraint (informix.tk_processnodeservice_tkservice).| There are other fixes that may cause similar failures. Those failures may appear as: 10/03/2009 01:15:05 AppInstall|cmdExec: Command: /partB/usr/local/bin/base_scripts/env.sh nice -n 19 /usr/sbin/chroot /partB /usr/local/cm/script/5.1.3.8000-4/cm-dbl-install L2 PostInstall 5.1.3.8000-4 5.1.3.7107-1 /usr/local/cm/ /partB/usr/local/cm/ /var/log/install/install_log_091003004108.log -- failed with failure code 1| 10/03/2009 01:15:05 AppInstall|Internal Error, File:instMain.c:1501, Function: handlePhase(), Failed to exec command: "/partB/usr/local/bin/base_scripts/env.sh nice -n 19 /usr/sbin/chroot /partB /usr/local/cm/script/5.1.3.8000-4/cm-dbl-install L2 PostInstall 5.1.3.8000-4 5.1.3.7107-1 /usr/local/cm/ /partB/usr/local/cm/ /var/log/install/install_log_091003004108.log"| 10/03/2009 01:15:05 upgrade_install.sh|Cleaning up download...| In both cases the appropriate action is to upgrade to the CUCM ES 5.1(3.7109.1) or later.

CUCM 7.1.2 file system read-only mode after EXT3 Journal aborted error , Fixed CSCta73022

Symptom: File system goes to read only mode. The following may be seen in the (RTMT-System Logs) or "messages" file on the local server: ------- kern 2 kernel: EXT3-fs error (device sda2): ext3_journal_start_sb: Detected aborted journal kern 2 kernel: Remounting filesystem read-only ------- You could also see the following in the (RTMT-Application Logs) or "CiscoSyslog" file: ------- SyslogSeverityMatchFound events generated: SeverityMatch - Critical kernel: EXT3-fs error (device sdb1) in start_transaction: Journal has aborted or SyslogSeverityMatchFound Detail: At Tue Sep 01 23:01:03 CDT 2009 on node 10.120.200.33, the following SyslogSeverityMatchFound events generated: SeverityMatch - Critical kernel: EXT3-fs error (device sdb1) in start_transaction: Journal has aborted ------------------------------ To determine whether the root (/) partition is still writable, enter the following from CLI: utils iothrottle enable You should see the following message: "I/O throttling has been enabled" utils iothrottle disable You should see the following message: "I/O throttling has been disabled" To determine whether the /common partition is still writable, enter the following from CLI every 2 minutes: file list activelog syslog/* detail You should see the size of the file "CiscoSyslog" change, for e.g. 589,219 CiscoSyslog and then approx. 2 minutes later: 589,550 CiscoSyslog
Conditions: Following hardware models are susceptible to the problem when installed or upgraded to CUCM or Unity Connection release 7.1.2.20000-2: MCS 7835-I2 servers MCS 7845-I2 servers or customer-provided IBM xSeries x3650 CUCM/UC release can be identified from the product banner displayed on system console. Another way to find out CUCM/UC release is to run platform admin CLI command "show version active". Hardware model can be identified by running platform admin CLI command "show hardware".
Workaround: If experiencing above problem for specified CUCM/Unity Connection release and hardware models, then apply the following workaround: 1. Boot the system using the recovery disk (version 7.1.2.10000-16 or later) 2. From the recovery CD menu select option 'f' to run file system check 3. When completed select option 'q' to quit recovery CD 4. eject the CD when prompted and reboot the system Additional Information: ***If you think you are running into this defect, please collect the following: 1) show hardware 2) active syslog/* files 3) "utils create report hardware" (DSA) Any customer that have had the r/o problem on the /common partition and recovered via reboot and/or Recovery CD should consider doing an upgrade to a CUCM or UC version w/ the CSCta73022 fix, then doing a DRS backup, fresh install the same CUCM or UC version, and restore. Reason being is the /common partition is common to both the inactive and active (upgraded) UCM and as such, will still potentially have broken files and or directories. The fresh install would remove this possibility. If the customer doesn't known which partitions were affected and no record exists, the above procedure should be followed verbatim.

7828-H3 server goes down with Journal Aborted error , Fixed CSCsv49493

Symptom: Phone services will go down, and server will only be semi-responsive. Local console access will show the following error constantly scrolling across the screen. EXT3-fs error (device sd(8,6)) in start_transaction: Journal has aborted
Conditions: During normal operation services will go down. Reboot will bring services back up for a while, anywhere between a couple hours and a couple days. Seen most frequently on MCS7828-H3-K9/BE but has been reported on MCS7825-H2-IPC1 and MCS7825-H3. This defect is for the MCS7828-H3 servers only.
Workaround: Shut down the server, and remove the first hard drive until a final fix is available. If server still fails, try switching to the other drive. Watch during boot up for any errors which might indicate hardware failure (SMART errors in particular). If server stills fails on 2nd drive, leave one drive in, and reinstall CUCM.
Further Information This issue was tracked to a firmware problem on the hard drives. A firmware update is available on cisco.com. Please download the readme and iso for 7828h3-hddfwupdate-v11 under the MCS 7828-H3 Unified Communications Manager Appliance. Direct link: http://tinyurl.com/mkrbxa

Cluster Manager on subscriber node log info alarm every 3 min , Fixed CSCtc27081

Symptom: Following alarm is seen in the CUCM Application Logs CiscoSyslog every 3 min on all of the subscriber nodes. Sep 23 14:59:53 CBNA-CUCM-S11 local7 6 clm: 1354: Sep 23 19:59:53.212 UTC : %CCM_CLUSTERMANAGER-CLUSTERMANAGER-6-CLM_PeerState: Current ClusterMgr session state. Node's Name or IP:CBNA-CUCM-P Node's State:POLICY_INJECTED App ID:Cisco Cluster Manager Cluster ID: Node ID:CBNA-CUCM-S11
Conditions: Every 3 min all of the subscriber nodes in a CUCM cluster will run a connectivity test to the Publisher node via the Cluster Manager service. At the end of this test the subscriber nodes logs an Informational alarm unnecessarily as there is no change of Policy states. The alarm is informational only and no action needs to be taken on it.
Workaround: To avoid the excessive alarms and log entries the connectivity test may be turned off: 1) obtain a remote account 2) Stop Cluster Manager from admin CLI: "utils service stop Cluster Manager" 3) Stop the automated connectivity test using the remote account: "/usr/local/platform/bin/clm/clm_ctl set clm_network_test_timer 0" 4) Start Cluster manager from the admin CLI: "utils service start Cluster Manager" This does *NOT* turn off the alarms if the network (pub or sub) goes down. The only thing the automated connectivity test does is to scan the network for possible packet loss to the private ports. In an established environment that usually occurs with the introduction of a new firewall into the network. "utils network connectivity test" can still be issued manually.
Further Problem Description:

IBM Director Agent reports defunct drive - false RAID alert , Open CSCtb66354

Symptom: IBM ServeRAID defunct drive messages seen in system syslog file.
Conditions: Under normal operations IBM Director Agent reports a drive to be defunct, and logical drive to be offline. A bit later, the IBM Director Agent reports logical drive to be in normal state again. This may be seen intermittent, sometimes a few times a day, sometimes once every week or month.
Workaround: None as issue is in the IBM driver. Need newer RAID driver to avoid the false alerts.
Further Problem Description: This issue is only a cosmetic issue and does not have any impact on the system other than triggering alerts. Note that IBM Director Agent may also see a real drive issue and report that, so it's recommended to closely monitor the system if you see these messages to make sure both drives in the logical drive is working properly. The normal message usually appears within a few minutes after the incorrectly reported defunct drive message.

HardwareFailure alert is raised due to iLO 2 Communications Error , Fixed CSCsl74589

Symptom: HardwareFailure alert is raised as a result of the following message seen in the System Event logs (messages) On Tue Dec 11 00:46:52 EST 2007 on node 172.18.10.10 , the following HardwareFailure events generated: hwStringMatch - hpasmxld[4500]: Failed GET SENSOR READING, sensor 9 hwStringMatch - hpasmxld[4500]: iLO 2 Communications Error - Attempting synchronization!
Conditions: This alert is raised as a result of the following logs Nov 14 16:16:32 claire daemon 3 hpasmxld[4545]: Failed GET SENSOR READING, sensor 9 Nov 14 16:16:32 claire daemon 3 hpasmxld[4545]: iLO 2 Communications Error - Attempting synchronization! Nov 14 16:17:17 claire daemon 3 hpasmxld[4545]: check_ilo2: Failed to get BMC Device Id! Nov 14 16:18:02 claire daemon 3 hpasmxld[4545]: The iLO 2 reset / synchronization process has FAILED! Nov 14 16:18:02 claire daemon 3 hpasmxld[4545]: CRITICAL ERROR: Resetting HP Advanced System Management Stack! Nov 14 16:18:02 claire daemon 3 hpasmxld[4545]: The iLO 2 Management Processor is not responding! Nov 14 16:18:02 claire daemon 3 hpasmxld[4545]: Nov 14 16:18:02 claire daemon 3 hpasmxld[4545]: CRITICAL ERROR: Resetting HP Advanced System Management Stack! iLO 2 on this server has 1.35 Firmware revision. Initially the above logs were thought to occur due to an iLO firmware issue and this defect updated the iLO firmware to the latest version at the time. However this was not the root cause. The root cause was later identified as the same root cause that lead to CSCsr23111 - "HP SNMP Agent stops unexpectedly leading to HardwareFailure alert". The root cause was the fact that HP SNMP agents were unexpectedly stopping without generating a core file. As a defensive mechanism to ensure SNMP agents are maintained and restarted in case they unexpectedly stop a watchdog was implemented via CSCsm98609 - "Platform CLI show environment temperature fails due to missing hpasmd"
Workaround: None, this particular instance of HardwareFailure Alert doesn't indicate a critical hardware failure on the system that could impact call processing. CSCsr23111 addresses the root cause of this defect and is first fixed in CUCM ES version 6.1(2.1108-1) CSCsm98609 first fixed in ES 6.1(2.1104-1)

CDR SSH user incorrect causing wrong file permissions in CUCM (GGSG Req) , Fixed CSCsz99841

Symptom: Cluster Manager will update ccmservice user SSH security keys whenever a new node is added or removed from the cluster
Conditions: CDR Agent cannot send CDR files from sub to pub using sftp. This is because SFTP is initiating a connection using the user 'ccmservice'
Workaround: i) Stop CDR Agent service on Call Processing Node ii) Gain root access via remote access support account. And then Copy over the authorized keys in the specific user's SSH directory: /home/sftpuser/.ssh/authorized_keys file to /home/ccmservice/.ssh/authorized_keys Root CLI command: cp /home/sftpuser/.ssh/authorized_keys /home/ccmservice/.ssh/authorized_keys iii) Start CDR Agent service on Call Processing Node.
Further Problem Description: This was caused by changes that were made to tighten up file permissions around the CUCM file structure.

DST: Update CM to Olson version 2009p , Fixed CSCtc85618

Symptom: CM needs updated to Olson version 2009p
Conditions: UCM 4.3(2) - 7.1(3)
Workaround: For Argentina (primary TZ currently affected) - place phones into Greenland TZ for the affected period.
Further Problem Description: Updates will be made available shortly for 7.1(3) which has a built in TZ Update feature. Prior releases do not have this TZ update functionality.

SLP_srvreg process spikes CallManager CPU , Fixed CSCta21676

Symptom: slp_srvreg process consumes 90% of the CPU power.
Conditions: During normal operation.
Workaround: "utils snmp hardware-agents restart " on the Admin prompt of the CallManager server experiencing the high CPU Restart Call Manager if it doesn't help.

Installing respin on ES should not be allowed , Fixed CSCsr88057

Symptom: CUCM L2 Upgrade fails with message "Internal Error".
Conditions: #1 The active CUCM version is an ES version. For example, 5.1.3.2105 #2 The upgrade version is a base version (respin). For example 5.1.3.4000.
Workaround: You may try one of the following options: #1 If the previous active version was a base version, try to switch back to that base version and upgrade to the new base version: a) On CLI, type "show version active". If the inactive version is a base version (e.g. 5.1.3.2000), continue to step b, otherwise try option #2. b) On CLI, type "utils system switch-version" to switch to the inactive version. System will reboot to base version (e.g. 5.1.3.2000) c) Upgrade to the new base version (e.g. 5.1.3.4000) Please note: This option will wipe out any ES fixes. If this is not the desired outcome, please use option 2 below. This could also potentially lose configuration changes committed since the last upgrade. #2 Open a TAC case to get the ES (Engineering Special) file based on the new base version. For example, ES 5.1.3.4105 (or above).
Further Problem Description: Any build number ending with 000 (for example, 5.1.3.2000, 5.1.3.4000, etc.) is a base version. A base version may not contain all the ES fixes in the previous base version. For example, 5.1.3.4000 may not contain all ES fixes in 5.1.3.2105. A build number increased in 1000th place is called "respin". A respin may contain security patches (PSIRT fixes) and select critical fixes.

7.1.2a subscriber install fails to connect to publisher , Fixed CSCta66767

Symptom: Subscriber fails to connect to publisher during install.
Conditions: Defect condition appears when publisher is installed from factory boot image with an upgrade to 7.1.2. IPM|(CAPTURE) isftp using password failed (12). Trying sftp using keys. or IPM|(CAPTURE) isftp using password failed (15). Trying sftp using keys. can be seen in the install.log of the subscriber affected OR Defect condition can also appear when you hit SKIP instead of PROCEED when installing a publisher WITH the upgrade during install option - in this scenario, the connectivity test to the publisher will fail during the installation of the subscriber. In this particular scenario, there is no way to check the install log on the subscriber, just check for the conditions mentioned in the workaround below - same workaround applies. You can view the attached screenshots for more details.
Workaround: Contact TAC to perform workaround. Workaround is as follows: 1) Login to pub w/ remote support account. 2) cp -p /usr/local/platform/conf/platformConfig.xml /partB/usr/local/platform/conf 3) Continue with the subscriber install.

publisher NTP down if configured NTP down or unreliable , Fixed CSCsk70971

Symptom: The subscriber CUCMs NTP will not sync to the publisher when the external NTP server is unreachable. Other symptoms include failing database replication or failure to setup database replication. 15 minutes clock variation on any replication reset will prevent replication setup.
Conditions: The external NTP server configured on the publisher is unreachable and the cluster has been rebooted. This can also be triggered if there is intermittent connectivity to the publisher. Fluctuation in round trip time to the publisher can also cause this.
Workaround: Restore connectivity to an external NTP source. Stabilize IP connectivity between publisher and all subscribers in the cluster. Stabilize the round trip time between publisher and all subscribers.

Cannot upgrade from CUCM 5.1.3 to CUCM 7.0.1 , Fixed CSCsl21150

Symptom: Upgrade from 5.1.3.4111-1 or lower to 7.0.1 fails, giving the following error: 'The directory was located and searched but no valid options or upgrades were available. Note, a machine cannot be downgraded so option and upgrade files for previous releases were ignored.'
Conditions: Upgrading from 5.1.3.4111-1 or lower to 7.0.1 using an SFTP/FTP server installed on Windows.
Workaround: 1. Use an SFTP server installed on UNIX/LINUX. 2.Burn the unzipped directory to a DVD and perform a local DVD upgrade. For e.g. for upgrading from a 5.1.3.ES to 7.0.1.ES: a. unzip cisco-ipt-k9-patch7.0.1.11101-1.sgn.zip which will result in a directory "cisco-ipt-k9-patch7.0.1.11101-1". b. Copy this directory in its entirety to the DVD and then from the "Software Installation/Upgrade" option, select "DVD/CD" as the source. Additional Workaround: In the event that neither of the above options is possible, please contact TAC so they can provide you with a patch.

sysappl mib is not responding , Fixed CSCsh72473

Symptom: SNMP MIB SYSAPPL table wont respond on CCM 6.0.
Conditions: After installing CCM 6.0 freshly.
Workaround: Contact Cisco TAC for workaround

CM6.0:Co-Res:Kernel panic : No init found try passing init , Terminated CSCsh78294

Symptom: On the console the error keeps coming up frequently which doens't allow the customer to login. *EXT3-fs error (device sd(8,6)) in start_transaction : Journal has aborted.*
Conditions: MCS 7825 H3 servers running CUCM 5.1 or 6.0 Server Not respoding due to File System Errors and not accessible via GUI/CLI.
Workaround:None. Contact TAC Hard reboot to recover the server
Further Problem Description:

HP SNMP Agent stops unexpectedly leading to HardwareFailure alert , Fixed CSCsr23111
Symptom: HardwareFailure alert is raised after detecting the following message in the System Event Logs On Tue Dec 11 00:46:52 EST 2007 on node 10.1.1.1, the following HardwareFailure events generated: hwStringMatch - hpasmxld[4500]: Failed GET SENSOR READING, sensor 9 hwStringMatch - hpasmxld[4500]: iLO 2 Communications Error - Attempting synchronization! Upon this alert other functions that depends on the HP SNMP agents stops to work. These include responses to SNMP get requests against the various CPQ MIBs doesn't return a response. Platform CLI "show environment" gives error.
Conditions: HP SNMP agent unexpectedly stops which triggers the above symptoms. Platform CLI command "utils snmp hardware-agents status" command indicates hpasmxld service is stopped.
Workaround: Restart the platform snmp hardware agents using "utils snmp hardware-agents restart"

'utils network connectivity' reports false negatives stuck in bad state , Fixed CSCsq34466
Symptom: 'utils network connectivity' reports failure on subscriber server even though connectivity exists. The test continues for give false negatives.
Conditions: Rebooting the server in the middle of running the command can cause the test to get stuck in this error state. The network connectivity test is also run automatically every 3 min in order to proactively test network connectivity failure events. If the server is rebooted in the middle of the automatic test it can also cause future tests to get stuck in this error state. cluster manager logs show connectivity check running continuously: 11/24/2008 10:27:36.003 clm|Connectivity test is Running| 11/24/2008 10:27:36.004 clm|Connectivity test is Running| 11/24/2008 10:33:36.005 clm|Connectivity test is Running| 11/24/2008 10:33:36.005 clm|Connectivity test is Running| 11/24/2008 10:39:36.007 clm|Connectivity test is Running| 11/24/2008 10:39:36.007 clm|Connectivity test is Running| 11/24/2008 10:45:36.001 clm|Connectivity test is Running| (continues repeating)
Workaround: Cisco TAC can reset the status of this test if it is stuck in this state. The workaround is the following: 1) Get a root admin account (remote_account) 2) Stop the "Cluster Manager" service from the admin cli: utils service stop Cluster Manager 3) Reset test from the root cli: Enter: /usr/local/platform/bin/clm/clm_ctl set clm_connectivity_test disable Enter: /usr/local/platform/bin/clm/clm_ctl set clm_connectivity_test_results 1 4) Start the "Cluster Manager" service from the admin cli: utils service start Cluster Manager

CUCM upgrade fails with message "unable to mount the local file system" , Fixed CSCsm55596
Symptom: Upgrading from 5 or 6 to 6.1 version
Conditions: Upgrading to 6.x versions. 7.x and above are not affected.
Workaround: Must obtain a remote account and run the /usr/sbin/updfstab system utility. Then verify there is a /mnt/cdrom and /etc/fstab contains cdrom.

Memory leak causing many processes to be killed by OOM Killer , Fixed CSCsz11007
Symptom: Many CUCM processes are killed some causing things like Web access and alerting to be out. This may also impact AMC/RTMT, DRS backups and the ability to collect CDR records. The OOM Killer reports some stats when it kills processes. One of the stats seem to point to the depletion of lowmem memory (as shown below): ... Apr 5 10:46:54 server1 kern 4 kernel: Normal free:3696kB min:3728kB low:7456kB high:11184kB active:204kB inactive:276kB present:901120kB pages_scanned:396 all_unreclaimable? yes ... Apr 5 10:46:54 server1 kern 4 kernel: Normal: 0*4kB 0*8kB 1*16kB 1*32kB 1*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 3696kB ...
Conditions: Memory leak eventually causes the system OS to start killing processes.
Workaround: Disabling CSA (utils csa disable) prevents this problem. After this problem occurs, rebooting the system will restore the system to operational state.
Further Problem Description: A process with a memory leak starts to allocate too much lower memory. In order to protect the operating system kernel, the OS starts to kill processes using lower memory. This will cause various services to fail depending on which ones are killed by the OS.

IBM SNMP agent cimserver process has apparent memory leak , Fixed CSCsl71487
Symptom: RTMT and perfmon counters show the cimserver process to consume increasing amounts of memory on IBM MCS servers. cimserver gradually consumes the majority of available virtual memory over a period of several weeks and eventually causes the server to hang.
Conditions: The cimserver memory leak can only occur on IBM MCS servers. All existing field cases have occurred on the 7825-I3. Cisco Dev Eng. and IBM PE team have reproduced the memory leak on other IBM server models. The likelihood of encountering a cimserver memory leak is fairly slim. The current field cases have occurred on CUCM versions 5.1.2.3000-2, 5.1.3.1000-12, 6.0.1.1000-37, 6.0.1.2000-3, and 6.0.1.2000-4, and on UC 2.0. It's believed the cimserver memory leak can occur on any 5.1+ or 6.x version of CUCM, CUP 1.x-6.x, and UC 2.x.
Workaround: A workaround is implemented with CSCsm71662. Download cop file ciscocm.CSCsm71662.cimserver.snmp.leak.cop.sgn from cisco.com: 1. Navigate to Software Search: http://www.cisco.com/kobayashi/sw-center/index.shtml 2. In the search box enter 'CSCsm71662' The download is presented in the search results. Next, in the Cisco Unified OS Administration GUI, select Software Upgrades->Install/Upgrade, Remote Filesystem, enter the directory, remote server of the cop file, a user name and password to the client machine with the cop file, select sftp transfer protocol, and proceed with the upgrade.
Further Problem Description: IBM has provided a fix for this memory leak, which has been integrated into CUCM 5.1(3d), 6.1(3), 7.0(1), and will be available in the next ES release of 6.1(1) and 6.1(2) as of June 10, 2008. Newer releases of CUCM will incorporate this fix. Upgrades to other CUCM versions from versions for which the cop file was applied, will require that the cop file be re-applied after the upgrade. The cop file attached to this defect is a workaround that automatically restarts the IBM DIrectory Agent SNMP cimserver process (dacimom) every Sunday at midnight. It also automatically restarts the dependent net-snmp.org snmpd process. Restarting these processes results in the recovery of previously used memory back to the OS. How do I determine whether this issue is affecting my server? The cimserver memory leak problem can be determined by inspecting Perfmon RIS logs or RTMT. The Perfmon logs will show cimserver memory use exceeding 350 KB. RTMT will report an error alert indicating that available virtual memory has fallen below 30% and that cimserver is using the majority of virtual memory. E.G., "%CCM_RTMT-RTMT-3-RTMT-ERROR-ALERT: This alert is generated by RTMT AlertMgr. Alert Name:LowAvailableVirtualMemory Alert Detail: On Tue Sep 18 03:40:53 MDT 2007 on node edmunity. Available virtual memory below 30 Percent. cimserver (1552 MB) uses most of the memory." Monitoring software such as CUOM, IBM Director, or HP Insight Manager may generate an event indicating the cimserver process stopped and subsequently started. Any such events occuring at midnight Sunday are attributable to the automatic restart set up by this cop file. Also, please note that CSCta99636 is the defect to track the fix for this issue on MCS-7828 servers.

add ability to collect IBM DSA logs from platform CLI , Fixed CSCsv59702
Symptom: For hardware failure IBM service needs DSA report from IBM servers. Currently there is no way to generate this from platform CLI.
Conditions: IBM hardware platform experiencing hardware failures. For HP platforms the CLI command utils recreate report hardware generates inventory for HP service. This is an enhancement request to have 'utils create report hardware' generate DSA report when the platform is IBM.
Workaround: Obtain a bootable disk from IBM that includes the DSA utility.
Further Problem Description: utils create report hardware was created for HP platforms as part of CSCsi30296. This is an enhancement to generate similar report for IBM service.

IBM DA tier1slpinst proc has erroneous LD_LIBRARY_PATH , Fixed CSCsj81116
Symptom: Library path of IBM Director Agent was errorneous. This problem exhibited itself in syslog/messages as invalid pointers in free calls.
Conditions: Problem was exhibited on IBM servers.
Workaround: NA.
Further Problem Description:

Unified OS - Browser hangs trying display certificate with 4096-bit key , Fixed CSCsv32209
Symptom: Cisco Unified OS Admin web page hangs when attempt to display a certificate with 4096-bit key is made. (CUCM,Connection and CUP)
Conditions: Cisco Unified OS v6.0(2), v6.0(3), v6.0(4),v7.0.(2),v7.1.(2)
Workaround: Use certificates that are created with 1024-bit key Additional Info: Once you are in this state, you cannot view any other certs. unless you restart Tomcat service from CLI: utils service restart Cisco Tomcat

Error/status msg indicating mismatched keys in the CSR & identity cert , Fixed CSCtc77187
Symptom: Following is the error/status message displayed when mismatched keys in the CSR and identity certificate. This happens when the generate CSR tab is hit more that once. The requested command [sudo/usr/local/platform/bin/CertMgmt.pydecodeop:importtype:certsunit:tomcatsrc-cert:%2Fusr%2Flocal%2Fplatform%2Fupload%2Fcerts%2F%2Ftomcat.dercert-dir:%2Fusr%2Flocal%2Fplatform%2F.security%2Ftomcat%2Fcertskey-dir:%2Fusr%2Flocal%2Fplatform%2F.security%2Ftomcat%2FkeysrootCA-cert:PENNONI-COLO-PKI-01-CA.pemtrust-dir:%2Fusr%2Flocal%2Fplatform%2F.security%2Ftomcat%2Ftrust-certslogfile:%2Fvar%2Flog%2Factive%2Fplatform%2Flog%2Fcert-mgmt.logresultfile:%2Fvar%2Flog%2Factive%2Fplatform%2Flog%2Fcertde-info.xmldescription:Self-signed+certificate] could not be executed.
Conditions: This happens when the generate CSR tab is hit more that once.
Workaround: Will need to get the CSR signed from the CA and upload certs again

Subscriber NTP Reference in ntp.conf not updated on Pub Hostname Change , Fixed CSCsy71974
Symptom: Subscriber's NTP reference to Publisher on OS Admin page still shows former publisher hostname.
Conditions: This can occur after changing the Publisher hostname.
Workaround: Contact TAC to manually fix the ntp.conf file.

Need a cop file for 5.1.3e to fix upg rules for L2ing to a 7.1(x) vers , Fixed CSCsy23628
Symptom: Cannot upgrade from 5.1.3.x release because system will not see a 7.1(x) upgrade file as a valid upgrade file.
Conditions: If attempting to upgrade to 7.1(x) from one of the following versions: 5.1.3.6000-2 5.1.3.6101-1 5.1.3.6102-1 5.1.3.6103-1 system will not see a 7.1(x) upgrade file as a valid file due to a bug in the upgrade path logic.
Workaround: 1.) Upgrade to a 5.1.3 ES greater than or equal to 5.1.3.6104 and then upgrade to desired 7.1(x) release. or 2.) Upgrade to the 5.1.3.7000 respin when available or 3.) Upgrade to a 7.0.x release and then upgrade to desired 7.1(x).
Further Problem Description:

CN of CSR and CA signed cert is case sensitive. , Fixed CSCsz51678
Symptom: CUCM 7.0 will do a case sensitive comparison when comparing a certificate's subject name to that of a CSR that was generated on CUCM.
Conditions: If the hostname is "SERVER", the CSR will have "SERVER" as the subject CN, and the certificate will have a subject CN of "server". This will cause an error and not allow you to upload the certificate since the CSR subject doesn't match the certificate's subject due to a case sensitive comparison.
Workaround: None

CUCM cluster clock syncronization drift if Publisher NTP server is down , CSCsw14629
Symptom: NTP Clock synchronization is off within a cluster. Services such a Database is reporting time is not synchronized.
Conditions: When a cluster has external NTP clock configured and that clock source is unavailable for some reason Publisher node will stop synchronizing its clock with the rest of the cluster nodes. This causes all nodes to drift independently of each other. This condition could lead to major DB replication failures that depends on all nodes part of the DB replication network clocks' to be in sync all the time. Other services such as AMC, or RISDC could also be impacted due to the clocks being out of sync. The following can be seen in the Informix Database logs (ccm.log) indicating the cluster nodes clocks are out of sync. 17:40:48 CDR NIF site: 9 clocks differ by 17 seconds 17:40:48 Warning - The operating system time of day clock at the remote site differs from the local site. This can slow down the data-sync apply performace at the target server. This happens only if timestamp or stored procedure conflict resolution rules are being used.
Workaround: No way to keep the cluster's clock in sync when the Publisher node can not reach the external NTP clock source. You can how ever disable NTP synchronization to an external source that way at least the cluster's clock can stay in sync.
Further Problem Description:

Platform CLI show environment temperature fails due to missing hpasmd , Fixed CSCsm98609
Symptom: show environment command gives the following error admin:show environment fans Internal CLI error detected, please try again.
Conditions: This error occurs when of one of the necessary platform agents is not running. Running utils snmp hardware-agents status could indicate if the following is seen hpasmxld is stopped...[ OK ] Then the error occurs
Workaround: Restart the platform snmp hardware agents and try the command again. use utils snmp hardware-agents restart

ES install fail on Sub after cluster IP addresses are changed , Fixed CSCsj35064
Symptom: - 3 node Cluster, Pub & 2 Subs - CCM 5.1.1.3113-1 - hosts & .rhosts verified to have the correct IP's & names Attempting to install ES 5.1.2.1101-4 from 5.1.1.3113-1 via SFTP or FTP fail after the Pub & Sub's have their IP Addresses changed. The ES may be installed via DVD. Upgrade logs show the Sub is attempting to SFTP to the Pub's old IP Address to get the platformConfig.xml. 06/20/2007 11:34:35 primNodeVerify.sh|Download /usr/local/platform/conf/platformConfig.xml file from 10.13.100.10|
Conditions: Publisher has previously changed IP address.
Workaround: Contact Cisco TAC and provide remote access for manual workaround.

Unable to create ssl cert with friendly hostname , Fixed CSCsr68571
Symptom: End user receiving a warning about an invalid certificate when connecting to the CCMUser pages.
Conditions: End users attempts to connect to CCMUser web page via a "friendly" URL, which has a different host.domain name than the actual publisher server.
Workaround: CUCM 6.x cert creation process is described in documentation here: http://www.cisco.com/en/US/docs/voice_ip_comm/cucm/cucos/6_1_1/cucos/iptpch6.html#wp1046223 Unlike in CUCM 4.x, this process does not allow the administrator to change the CN of the cert. Workaround would be to have the end-user connect to the actual CUCM host.domain name. Potential alternative workaround would be to change the server to the "friendly" hostname, generate the CSR, and then change the hostname back to the original.

catalina.out file gets too big when there is problem to start tomcat , Fixed CSCso50705
Symptom: Cisco Tomcat service log catalina.out file grows very large in size and stops to get rotated. The file could
Conditions: Under certain error conditions such as Database problems or Certificate problems Tomcat services or applications running within Tomcat we could be logging excessive amount of data in the catalina.out log file. This excessive logging error condition interferes with the log file management system and interrupts file rotation.
Workaround: None to restore the log file rotation or clean up the excessive catalina.out file manually. Contact TAC who can assist further.

network flapping fails clustermanager and causes replication failure , Fixed CSCsx07438
Symptom: In 'show open ports regexp 8500' it is seen that another process then clm (cluster manager) is bound to UDP port 8500. The correct output should be similar to : admin:show open ports regexp 8500 Executing.. please wait. clm 4575 root 6u IPv4 17966 UDP *:8500 clm 4575 root 7u IPv4 17967 TCP *:8500 (LISTEN) Incorrect is when another process then clm is bound to 8500, such as : modprobe 2305 root 6u IPv4 12116 UDP *:8500 modprobe 2305 root 7u IPv4 12117 TCP *:8500 (LISTEN)
Conditions: This failures occurs in CM 6.x. This is triggered by loss of connectivity between nodes or restarting clustermanager(clm). CM 7.x should not experience this failure.
Workaround: Only known workaround is to reboot the server on which this is seen.
Further Information: The fix implemented prevents clm from reloading iptables which prevents removing and reinserting kernel modules. This fix should address this issue and CSCsx72862 where kernel hangs during kernel module remove/insert and is recovered by ASR.

CM 5.1 - Tomcat web certificate regeneration fails , CSCsi51295
Symptom: The customer cannot access the CCM admin web page or connect to the the server using RTMT as both use SSL
Conditions: New Installation. The customer did not put in the FQDN in during install. Go to Cisco Unified OS administration, Security -> Certificate management. Regenerate certificate and choose Regenerate Self-Signed Cert and pick Tomcat
Workaround: None

HP agents v7.8.1 incorrectly reports fan status degraded , Terminated CSCsu99994
Symptom: Customers using HP agents v7.8.1 which were added via CSCsr23111 may see false reports of fan status degraded or thermal warnings on 7835H-3.0GHz servers.
Conditions: Only seen on HP DL380 G3 servers running HP agents v7.8.1 which is included in some versions of CallManager 5.1(3)es, 6.1(1)es, and 6.1(2)es code.
Workaround: Revert to a version of software with an older set of HP agents, such as 7.8.0

CUCM reboots by ASR with intermittent connectivity between nodes , CSCsx72862
Symptom: Some servers in the cluster (normally NOT all servers) may be automatically rebooted if ASR is enabled, or hang if ASR is disabled. When server hung with ASR disabled, no console ouput from the server, no network connectivity to the server and cold boot of the server is the only way to recover it. Netdump is setup but did not collect any kernel panic when the problem happened. If ASR is enabled, server will be automatiically rebooted by ASR in the frequency as a few times per day to once/twice per two days. file view system-management-log 0157 Critical 08:13 02/11/2009 08:13 02/11/2009 0001 LOG: ASR Detected by System ROM In CiscoSyslog (application logs), the following messages have been continueously repeating throughout the logs every few seconds. Feb 18 18:33:41 clay-voip-cm0 local7 6 : 41778: Feb 18 07:33:41.451 UTC : %CCM_CLUSTERMANAGER-CLUSTERMANAGER-6-CLM_PeerState: Current ClusterMgr session state. Node's Name or IP:clay-voip-pres0 Node's State:INITIATOR App ID:Cisco Cluster Manager Cluster ID: Node ID:clay-voip-cm0 Feb 18 18:33:49 clay-voip-cm0 local7 6 : 41779: Feb 18 07:33:49.131 UTC : %CCM_CLUSTERMANAGER-CLUSTERMANAGER-6-CLM_PeerState: Current ClusterMgr session state. Node's Name or IP:clay-voip-pres0 Node's State:POLICY_INJECTED App ID:Cisco Cluster Manager Cluster ID: Node ID:clay-voip-cm0 The above log messages were ONLY observed in the problem/rebooting servers in the cluster, NOT in the working servers. Alternatively the last messages that appear in the system log file (messages file) may be: Feb 23 01:39:59 myserver kern 4 kernel: ip_tables: (C) 2000-2002 Netfilter core team Feb 23 01:39:59 myserver kern 4 kernel: ip_conntrack version 2.1 (16384 buckets, 131072 max) - 304 bytes per conntrack Feb 23 01:39:59 myserver local7 5 iptables_orig: succeeded Feb 23 01:51:50 myserver syslog 6 syslogd 1.4.1: restart (remote reception). Feb 23 01:51:50 myserver local7 5 syslog: syslogd startup succeeded
Conditions: it has been observed, but not limited to: A unused CUP server (presence installed and inserted into cluster, but not used or deployed) enter into "READ-ONLY" like state, iptable failed to be open and cluster manager on presence keep talking to cluster manager in callmanager to inject policy.
Workaround: 1. reboot the problem presence server. 2. take the unused problem presence server offline.
Further Problem Description: Replace the whole server did not help the situation, During investigation, it has been observed that the unused presence servers in the newtork have caused these significant amount of messages in callmanager cluster (more than 30000 cluster manager messages per day in application log). it is also observed that these unused presence servers are in state simlar to a READ-ONLY file system. cluster manager on the presence servers failed to open iptable to inject policy,.As a result, cluster manager of CUP have been bounced between INITIATOR and POLICY_INJECTED state due to failed to open/update iptable, this has caused cluster manager on callmanager keep starting/stopping iptables and reloading kernel module.

L2 Upgrade takes too long/hangs , Fixed CSCsz22187
Symptom: L2 upgrade from 6.1.2 to 6.1.3 takes too long.
Conditions: Inactive partition has version 6.1.1 or lower with CDRREP logs in excess of 500,000.
Workaround: Wait for upgrade to finish (it will finish eventually), or, manually delete the CDRREP logfiles.

Incorrect SNMP sysOID for 7825H4 server model , Fixed CSCta35604
Symptom: CER unable to discover phones registered to one callmanager
Conditions: Incorrect SNMP sysOID reported for callmanager installed on 7825H4 server. SNMP sysOID for all 7825H models (including 7825-H4) should be 1.3.6.1.4.1.9.1.583, but the below example output shows the final digits as 853. admin:utils snmp get 1 test 1.2.3.4 .1.3.6.1.2.1.1.2.0 This command may temporarily impact CPU performance. Continue (y/n)?y iso.3.6.1.2.1.1.2.0 = OID: iso.3.6.1.4.1.9.1.853
Workaround: Create a remote account, contact Cisco TAC, and provide remote access to implement workaround.

commonName field in generated Tomcat CSR does not contain domain name , Open CSCtd36375
Symptom: When you 1. configure the domain by issuing the command "set network domain exampledomain.local" -> this will reboot the node 2. generate a tomcat CSR 3. Verify the CSR (from root) -> "openssl req -noout -in /usr/local/platform/.security/tomcat/keys/tomcat.csr -text" => you will notice that the CN Field only contains the hostname and not the FQDN as configured.
Conditions: Verified on 6.1.4.1211-1 (issue not seen in 6.1.3.2000-1)
Workaround: 1. Generate the tomcat certificate through OS Admin page. This self signed certificate will be generated with CN=FQDN and will be stored in .tomcat.keystore. 2. Contact TAC to create a root account using the remote support tool, get TAC to connect to the node by ssh and execute following commands. cd /usr/local/platform/.security/tomcat/keys cat .tomcat.passphrase | keytool -certreq -alias tomcat -sigalg SHA1withRSA -keysize 1024 -file .tomcat.csr -keystore ../certs/.tomcat.keystore >>> this will generate new csr. chown certbase:ccmbase .tomcat.csr chmod 755 .tomcat.csr 3. openssl -req -in .tomcat.csr -text .>>>> check subjectName to verify CN=FQDN and other fields also. 4. Download CSR through OS Admin page and get it signed by CA and upload it again.

Not able to import certificate due to special character in CN of Cert. , Fixed CSCsv76010
Symptom: When trying to upload a certificate on OS Administration web page, getting error: "\/usr/bin/openssl\ terminated with error 1"
Conditions: There are two consecutive special characters in CN attribute. For example CN = "//" is considered as a special character because target filename is getting created from CN.
Workaround: None. Upgrade to a version that has this bug fixed.

servM process Increases in memory use with every backup , Fixed CSCsk62547
Description: ServM process Vmsize increases on a daily basis over a period of time Symptom: ServM process Vmsize increases on a daily basis over a period of time irrespective of whether a backup(DRF) is done or not.
Conditions: Normal running condition(irrespective of whether a DRF is done or not) Work-around: Unknown.

FTP incompatibility causes "Not enough disk space" upgrade failure , Fixed CSCsi91410
Symptom: An upgrade fails indicating there is not enough disk space to complete the upgrade even though there is enough disk space.
Conditions: An upgrade is done using remote method via FTP.
Workaround: Use a different upgrade method (such as SFTP or LOCAL) or use different FTP server software.
Further Problem Description: The problem here is that the FTP server software returns back a negative size value when CUC's FTP client issues a LIST FTP directive to the server. This causes CUC upgrade software to believe the size of the file is much larger than it really is and it causes the software to think there is not enough disk space to contain the file. The negative size value is most likely due to the fact that the FTP server software cannot deal with file characteristics where the file size is >2GB. CUC development has noticed this in CUC software since files of that size are not commonly downloaded using FTP.

process core due to File size limit exceeded , Fixed CSCsk21012
Symptom: Services started by servM trying to write the servM's deleted log file. RTMT alarms for coredump file generated.
Conditions: When a chatty child process of servM writes to a deleted log file, and the file limit crosses a limit, the process terminates. This can affect any process started by ServM including ccm, cti, ac, and others. For ccm process this condition is exacerbated by failed device registration attempts. 2 vg248s with all 48 ports failing to register will cause CM to core within a 14 day period.
Workaround: Restart the service.
Further Problem Description: utils core analyze cli command against result core file includes the following as part of the output:: Program terminated with signal 25, File size limit exceeded

CUC Unable to mount local file system when attempting to install locales , Fixed CSCsr20224
Symptom: CUC Install Software page reports "Unable to mount locale file system" and console shows "ide-scsi: hdc: unsupported command in request queue (0)" when attempting to install locale files.
Conditions: Unity Connection 2.1 on a MCS7845I2-K9-UCA1.
Workaround: Log on as root and issue the following commands (ignore errors if any): rm -f /dev/cdrom mknod /dev/sr0 b 11 0 ln -sf /dev/sr0 /dev/cdrom ** Changes will be lost on reboot and need to be reapplied **
Further Problem Description: This issue is not observed on HP servers when using the same install media.

MCS 7816-H3 servers have performance issues showing High % I / 0 Wait , Fixed CSCsm83603
Symptom: MCS-7816-H3 servers will experience HIGH %IO WAIT
Conditions: MCS-7816-H3 servers running any VOS (Voice Operating System) based version of an Application, like CM 6.1, will experience performance issues due to high %IO WAIT. This is caused by malfunctioning BIOS settings meant to enable the Hard Drive Write Cache Solution/Workaround: 1. The permanent solution Upgrade to a version that contains a BIOS upgrade, as indicated by the Integrated-In Field of this defect 2. A suitable workaround is to apply the following COP file, which does NOT contain a BIOS upgrade, but instead contains a utility that enables Write cache outside of the BIOS settings. -ciscocm.CSCsm83603EnableCache7816-1.1.cop.sgn -A readme for the above COP file can provide additional information
Further Problem Description: -The BIOS Settings are set correctly, but they do not function properly

CUCM Admin password reset gives false negative after unsuccessful reset , Open CSCtd17258
Symptom: Receive error "Error: the password has NOT been reset." when providing an valid password to the admin password recovery tool
Conditions: Using the OS admin password recovery procedure where at some point previously on this system an invalid password has been previously entered during the password recovery procedure (either because the provided password was too short, was based on a dictionary word, or because the two passwords typed didn't match)
Workaround: Ignore the "Error: the password has NOT been reset." status message and press Ctrl-C to end the password recovery session. Even though that error was given, the new password just entered will work to successfully login to the OS admin.

set web-security CLI command not allowing spaces for O, OU, L, ST, and C , Fixed CSCsq53869
Symptom: set web-security command does not allow spaces or special characters for Org, Org. Unit, Location, State and Country.
Conditions: set web-security does not allow spaces or sepcial characters in the command arguments.
Workaround: None.
Further Problem Description:

Certificate management doesn't allow apostrophe (') in OU , Fixed CSCtd03662
--> Symptom: When on CUCM CLI try to run "set web-security" command, apostrophe in organization name doesn't get accepted. Certificate management blocks this character.
Conditions: CUCM 7.0.2.21900-10
Workaround: None

7.1(2b)su1 upgrade rules to prevent install over ineligible ES's , Fixed CSCtb50376
Symptom: Upgrades from CUCM 7.1(2b)su1, also known as 7.1.2.31900-x, don't display any es's less than or equal to 7.1.2.310xx-x as valid upgrades.
Conditions: Only applicable for customers running CUCM 7.1(2b)su1, also known as 7.1.2.31900-x. The upgrade rules included with this fix prevent customers from installing an Engineering Special that does not contain all of the fixes in 7.1(2b)su1.
Workaround: Upgrade to an es in the 7.1.2.32xxx-x range or higher