Error activating Subscriber UCCX services

After installing a second UCCX node, the CCX services (CCX Config/Agent/Historical Datastore) fail start.

nested exception is: com.cisco.database.util.DBException: Replication is not setup

Today, our UCCX subscriber (part of a 2-node cluster running 9.0(2)) went down and we needed to replace it by fresh installing another UCCX subscriber in its place.

I removed the UCCX-SUB virtual machine and launched a reinstall , but I faced an error in the “Network Connectivity Validation” step stating a problem with UCCX-PUB (it throws an error about a firewall issue, where both machines are on the same VLAN lol, anyway).

So I planned a maintenance window the same night in order to restart the UCCX Publisher node. (in fact, after removing a node in the cluster in order to readd it, the publisher needs to get restarted)

So I deleted first the UCCX-SUB virtual machine, then removed the instance of UCCX-SUB on the Publisher from the menu System > Server.

When UCCX-PUB is back online, I readded UCCX-SUB in the same menu System > Server.

I launched the reinstall and everything went smoothly.

uccx-install

To re-establish the replication between the Publisher and the Subscriber, we need to login to UCCX-SUB AppAdmin in order to give some information which are : UCCX-PUB IP address, administrator username and password and the type of deployment (LAN or WAN).

Next, the wizard will start the UCCX services. However the service activation step failed stating :

Activation failed – Activated
Component Controller CRS Agent Datastore on node 2 : Enable; nested exception is: com.cisco.database.util.DBException: Replication is not setup. Please setup replication and try activating the component.Number of nodes defined is 1

uccx-service-activation-failed.png

In fact, the problem is due to the UCCX replication not being setup. (I insist! UCCX replication, not the usual IDS database replication as in CUCM). The UCCX replication can be verified with the command “utils uccx dbreplication status” on the UCCX-SUB which will throw an error, whereas the usual command “utils dbreplication runtimestate” will show “2” good). We need to focus on UCCX Replication.

Also, the Config Datastore and Historical Datastore services activation throw the same errors.

The problem is because we need to do the following on UCCX Publisher side :

  1. reset the replication and
  2. Enable CDS and HDS

So go the UCCX Serviceability on UCCX-PUB, menu Tools > Datastore Control Center > Replication Servers.

reset-replication.png

Click “Reset Replication”, the step will take a couple of minutes (depending on the size of deployment). Following thing, click the “Enable CDS and HDS”. Wait for it to complete (nevermind if it will return back with the message “Timeout”, we will be running again the service activation step on UCCX-SUB and everything should work fine).

So, we get back to the UCCX-SUB AppAdmin web page, we login and input again the UCCX-PUB IP address, administrator username and password and the type of deployment (LAN or WAN), then click Next.

The services will be -hopefully- brought up and the you can go ahead to Next step which is a summary then your cluster is all good!

Of course, all this need to be done in offhours!

Hide participants in a WebEx Meeting

As a meeting host and in the WebEx meeting window, go to the menu Participants > Assign Privileges and untick View Participant List in the “Participants” tab.

I received a feature request from a customer who has a WebEx deployment.

He has a meeting with partners from other companies. The participants are competitors.
So the customer doesn’t want the participants to see each one others.

I wasn’t sure that this feature is available in Cisco WebEx, I google it and did not find relevant information, but while tweaking the WebEx Meeting menus I found that it’s possible to hide participants.

As the Meeting Host, go to the menu Participant then Assign Privileges.cwms-host

Go to the “Participants” tab. By default, the privilege View Participant List is assigned to all participants.

hide-participant-untick

Untick that checkbox and validate by clicking “Assign”.

This way the Participants List panel is removed and unavailable for meeting attendees.

Of course, do this before the meeting begins in order that the participants don’t see the other ones when joining the meeting.

7821/7841/7861 locale installation on CME

You need the file “sp-sip.jar”.
Don’t forget to put it in the correct folder such as “SIP_French_France” !

I tried to install French language to an IP Telephony deployment based on CME and the latest 7800 IP phone Series. I have searched on Google for the .jar file needed by these IP phones in order to be localized but have not find any clear information.

So I have turned on the “hard way” mode in order to find that information.

The first thing I’ve done is to enable the TFTP debug through “debug tftp events” on the CME. Generally speaking this command is a good troubleshooting tool, for example to double check that the IP phone is actually reaching the TFTP server and correctly downloading its files.

(I assume here that you already configured the locale commands on voice register global)

Here is the output :

locale-7821-cme

The interesting line in the debug is :

TFTP: Looking for SIP_French_France/sp-sip.jar

The IP phone looks for the file sp-sip.jar and not less importantly under the folder SIP_French_France.

So I uploaded the file to flash: and then published it on the tftp server through these commands :

Router#conf t
Router(config)# tftp-server flash:sp-sip.jar alias SIP_French_France/sp-sip.jar

It’s important to publish the file on the correct folder which is similar to the syntax “SIP_language_country“.

Afterwards the 7821/7841/7861 IP phones correctly downloads the locale file and hopefully reset and start using the target language.

The locale files are available through Cisco under this link (CCO account needed)

Yes the file is very heavy. Because it contains files for all languages, and just need to pick up the file for the language you need. All this for a 34kb file!

Enable-CsTopology : Error accessing folder \\sfbshare

I have made some changes to my Skype for Business topology and I finished that as usual with the cmdlet “Enable-CsTopology”.

However, I got like 51 errors all stating :

Enable-CsTopology : Error accessing folder
\\sfb-fe.contoso.com\sfbshare\1-ApplicationServer-1\AppServerFiles\CPS.
At line:1 char:1
+ Enable-CsTopology
+~~~~~~~~~~~~~~~~~~
+ CategoryInfo : Invalid Data: (:SourceCollection) [Enable-CsTopology], DeploymentException
+ FullyQualifiedErrorId : InvalidFolder,Microsoft.Rtc.Management.Deployment.ActivateTopologyCmdlet
Enable-CsTopology : Error accessing folder \\sfb-fe.contoso.com\sfbshare\1-ApplicationServer-1\AppServerFiles\Rgs.
At line:1 char:1
+ Enable-CsTopology
+~~~~~~~~~~~~~~~~~~
+ CategoryInfo : Invalid Data: (:SourceCollection) [Enable-CsTopology], DeploymentException
+ FullyQualifiedErrorId : InvalidFolder,Microsoft.Rtc.Management.Deployment.ActivateTopologyCmdlet
Enable-CsTopology : Error accessing folder \\sfb-fe.contoso.com\sfbshare\1-ApplicationServer-1\AppServerFiles\WinFabTraceFiles.
At line:1 char:1

lync

The errors clearly mention an access to lyncshare folder error. I tried to actually enter the share folder through its FQDN, and I browsed in Windows Explorer \\sfb-fe.contoso.com\sfbshare and SURPRISE!! I was denied.

I browsed again the Lyncshare but this time through its IP address \\@IP-ADDRESS\sfbshare and it works! So, I understood that it was a DNS related issue.

I opened a cmd shell and did a simple ping sfb-fe.contoso.com and I got an IPv6 response!

I disabled IPv6 on the network card like that :ipv6

and did an ipconfig /flushdns on cmd.

After these two actions, I was able to browse the SFB Share through its FQDN and finally the share folder is accessible and the Enable-CsTopology worked successfully.

It’s really surprising because the SFB was up and running in production. I don’t want caused the Enable-CsTopology where no recent changes happened.

Integrating Cisco Meeting Server with Lync/Skype for Business

I have implemented this week multiparty conferencing through the new Cisco’s CMS conferencing product. CMS enables Lync and Cisco users to participate in the same conference with a seamless behavior between Lync and Cisco experience.

The CMS is excellent for organizations where both Lync and Cisco UC deployments are used. (yes, Cisco is best in Telephony, yes Lync is best in Chat & IM, and yes opinions are shared for video/webconferencing)

The configuration is pretty straightforward. I’m really surprised with the ease of the configuration in CMS.

I will not detail how to integrate CMS with CUCM. A prerequisite of the following is that Cisco endpoints eg. Jabber and/or DX endpoints are able to connect to the same Space.

Note that the word “Space” in Cisco meeting Server is actually the equivalent of the Collaboration Meeting Room in Cisco Conductor.

The configuration steps on Lync/SfB Front-End are detailed in the CMS deployment guide Appendix C.

So basically, you need a choose domain for CMS  which needs to be different than the Lync domain. (for example, if your Lync deployment is contoso.com , you may choose for example the domain cms.contoso.com to route calls from Lync to CMS). This is because Lync needs to differentiate “Lync internal” calls (user@contoso.com) from “Lync external” calls (room@cms.contoso.com). You may even choose a completely different domain like “contoso2.com”, not only a different subdomain.

Here are the steps :

1- Download the Root CA certificate that signed the CMS certificate and import it to each Lync Front-End server. Put it in the “Trusted Root Certificate Authorities” folder. The Root CA certificate should be a .cer or .crt or whatever certificate file extension.

Importing the Root CA certificate that signed Lync Front-Ends to CMS is recommended however not required since CMS certificate verification is disabled by default.

2- Open Lync Server Management Shell to create the “Trunk” between Lync and CMS. The “Trunk” with CMS will be of type “Trusted Application Pool”; as far as Lync guys are concerned. Type the following cmdlets :

New-CsTrustedApplicationPool -Identity cms.contoso.com -ComputerFqdn cms.contoso.com-Registrar sfb-pool.contoso.com -site 1 -RequiresReplication $false -ThrottleAsServer $true -TreatAsAuthenticated $true

New-CsTrustedApplication -ApplicationId meetingserver-application -TrustedApplicationPoolFqdn cms.contoso.com -Port 5061

$x=New-CsStaticRoute -TLSRoute -Destination “cms.contoso.com” -MatchUri “cms.contoso.com” -Port 5061 – UseDefaultCertificate $true

Set-CsStaticRoutingConfiguration -Identity global -Route @{Add=$x}

and Finally : Enable-CsTopology

3- That’s it ! From a Lync client call “room@cms.contoso.com” and you should be in the virtual room. Don’t forget to open your webcam. Seems that Lync don’t negociate inbound Video until you actually open your webcam and trigger the outbound Video negociation.

4- You may face a problem where the Lync call to “room@cms.contoso.com” window appears and then disappears suddenly and the call gets disconnected.
I troubleshooted the issue in the CMS

2017-01-25 16:17:47.805 Info call 25: recognised as Lync
2017-01-25 16:17:47.805 Info call 25: incoming encrypted SIP call from “sip:lyncuser@contoso.com” to local URI “sip:room@cms.contoso.com” (Lync)
2017-01-25 16:17:47.817 Info conference “Test”: unencrypted call legs now present
2017-01-25 16:17:47.922 Info participant “lyncuser@contoso.com” joined space 32c459e9-fb89-4b5f-96d9-73aab7e5c7f4 (Test)
2017-01-25 16:17:47.962 Info call 25: ending; remote SIP teardown; ICE negotiation in process – connected for 0:00
2017-01-25 16:17:47.962 Info participant “lyncuser@contoso.com” left space 32c459e9-fb89-4b5f-96d9-73aab7e5c7f4 (Test)

A detailed call trace on CMS will show a SIP BYE message with the cause :
ms-client-diagnostics: 52001;reason=”Client side general processing error.”;UserType=”Callee”;MediaType=”audio”

2017-01-25 16:18:50.797 Info SIP trace: connection 31: incoming SIP TLS data from 10.210.30.9:59210 to 10.210.30.5:5061, size 699:
2017-01-25 16:18:50.797 Info SIP trace: BYE sip:room@cms.contoso.com;transport=tls SIP/2.0
2017-01-25 16:18:50.797 Info SIP trace: Via: SIP/2.0/TLS 10.210.30.9:59210;branch=z9hG4bKDEEFB2EF.FF22BB7957D3B46E;branched=FALSE
2017-01-25 16:18:50.797 Info SIP trace: Max-Forwards: 69
2017-01-25 16:18:50.797 Info SIP trace: Via: SIP/2.0/TLS 172.30.22.241:6174;ms-received-port=6174;ms-received-cid=E5400
2017-01-25 16:18:50.797 Info SIP trace: From: <sip:lyncuser@contoso.com>;tag=d8804dbf7f;epid=eedfaef46a
2017-01-25 16:18:50.797 Info SIP trace: To: “Test” <sip:7000@cms.contoso.com>;tag=a1ed2fe26aae6b2b
2017-01-25 16:18:50.798 Info SIP trace: Call-ID: 3238de7a9d0547978d0a00d24623c4cb
2017-01-25 16:18:50.798 Info SIP trace: CSeq: 2 BYE
2017-01-25 16:18:50.798 Info SIP trace: User-Agent: UCCAPI/15.0.4893.1000 OC/15.0.4893.1000 (Skype for Business)
2017-01-25 16:18:50.798 Info SIP trace: ms-client-diagnostics: 52001;reason=”Client side general processing error.”;UserType=”Callee”;MediaType=”audio”
2017-01-25 16:18:50.798 Info SIP trace: Content-Length: 0
2017-01-25 16:18:50.798 Info SIP trace: ms-routing-phase: from-uri-routing-done

The problem was that, by default, the “SIP Media Encryption” parameter on CMS (menu Configuration > Call Settings) was set to “disabled”. Since Lync requires encryption you may turn it to “allowed” and everything will work as a charm!

One last thing, the Lync client can also give you the presence status of the room. It will show Ready if nobody is in the Room and “In a call” if a meeting is in progress.

Collaboration Edge with Let’s Encrypt certificate and 7800/8800 IP phones

I have finished few weeks ago a Collaboration Edge deployment through Expressway for a customer.

We used in this deployment the new “free” Certificate Authority aka Let’s Encrypt.

In fact, this authority is supported by 7800/8800 series IP phones as stated by the Certificate Authority Trust List document.

Many websites provides simplified management and generation of Let’s Encrypt certificate (native one requires API integration etc. a hussel!), we used sslforfree.com for instance.

We have generated the certificate for ONLY the FQDN of Expressway as Subject Name. We did not put anything as SAN, no top-level domain especially. (Expressway version is 8.7.3 but it should work for later versions)

And it worked like a charm. One last thing is that the certificate has to be regenerated each 3 months so that it keeps current.

He’s my conclusion in two points:

  • Let’s Encrypt is trusted by Cisco IP phones and can be used as the Certificate Authority for Expressway-E
  • Putting ONLY the FQDN of Expressway in the certificate is pretty enough to make phones work outside the corporate LAN

 

Corporate directory for 7800/8800 IP phones on CME “Host not found”

Last week, I was deploying a new CME with some 7821 IP phones with some 8861 IP phones for the executivess.

However, I faced a problem regarding the corporate directory on those new SIP IP phones.

Phones were stating “Host not found”.

I searched around forums and blogs but didn’t find anything helpful.

As soon as I added the following three commands, the corporate directory worked.

CME(config)#ip http server

CME(config)#telephony-service

CME(config-telephony-service)#url directories http://CME-IP-Address/localdirectory

Following this, the Corporate directory worked like a charm! Yes it doesn’t mean anything to use “telephony-service” which is supposed to be used in case we have Skinny phones. Well not.

Don’t forget to save your config! “wr” 😉