Disabling the CUCM Corporate Directory
Situation: You want to disable the Corporate Directory on your Cisco IP Phones - maybe you're like me, and have ~300 phones in places where people shouldn't be able to look up the Managing Director's direct line.
Solution: You come across this awesome little post from the Chesapeake Netcraftsmen, which says (amongst other things) to assign these phones a new Common Phone Profile, configure Services Provisioning to use a External URL, and then disable Enterprise Subscription on the Corporate Directory. Right?
... not quite. In a situation where the Corporate Directory is required on more phones than not, having to subscribe each phone to it is a pain in the proverbial. However, if you plug a dummy URL into the 'Directory' Data Location on your phone configuration, it also wipes out your Missed/Placed/Received calls directories. Cisco's website is unsurprisingly sparse on where this information is pulled from, and how to get it back if you're in my situation.
Dumping the Console Log on a phone shows that the phone requests a phoneservices.xml file upon boot, which lists the Services that the Phone is subscribed to. This file looks very similar to the format of the http://cucmpublisher:8080/ccmcip/xmldirectory.jsp file (which provides the Personal/Corporate Directory URLs), other than it doesn't actually have any URLs in it.
On a whim, I cooked up an XML file with similar contents to the phoneservices.xml file, hosted it on a test server, and pointed a test phone to this file via their 'Directory' URL. The phone reloaded, and the Missed/Received/Placed Calls directories were back in business. You can get a copy of the file here, or view the actual code after the break.
Managing Volume License Keys
I spotted this little gem of a post while trawling Twitter for something totally unrelated, and realised that I've been looking for a tool like this for a number of years.
The tool in question is the Volume Activation Management Tool (VAMT) 2.0, a product which (in a nutshell) makes light work of managing all the keys you get as part of, say, your Technet Plus subscription. You import your keys, and your machines, and the app tracks how many activations you have left on each key as well as showing which machine is running what software.
Great stuff - and don't forget to check out the stack of relevant literature available about VAMT on the Microsoft website.
(Sidebar: Aaron Parker's Stealthpuppy Blog is a great resource, and will probably be the blog that gets me using an RSS Reader one of these days.)
When the Citrix IMA Service fails to start…
... you run, very quickly, as your phone is about to start ringing.
In all seriousness, this one has bitten me on a few separate occasions. Always seems to occur immediately after a reboot, and there's generally no way to replicate it or 'cause' the issue. I've had servers run for weeks without any sign of it, and servers that have it crop up on every restart. Extensive reading of the Citrix and Microsoft KBs has revealed zilch in the way of a permanent fix, with the 'band-aid' being the solution outlined below.
Here's hoping they figure it out soon!
Scenario:
HP BL460c G1, Windows Server 2003 Standard 32-bit w/ Service Pack 2, Citrix Presentation Server 4.5 w/ Rollup Pack 6.
Symptoms:
- Clients are unable to establish an ICA Connection
- Server reports it is unable to connect to a License Server
- Citrix Independent Management Architecture (IMA) Service won't start
- Citrix MFCOM Service (mfcom.exe) is hung in a 'Starting' state (when viewed in the Services MMC)
- Event Log has Error 26005: Could not start IMA Service in CTX_MF_IMA_StartIMAService State = 1
Cause:
The folder "Documents and Settings\NetworkService\Application Data\Microsoft\Crypto" does not exist.
Solution:
- Check the if the above folder exists. If not, create it on the affected server.
(Bootnote: You'll need to ensure you can view Hidden Files and Folders in order to see the NetworkService folder.) - Kill the process 'mfcom.exe' via the Task Manager
- Start the Citrix Independent Management Architecture Service
- Start the Citrix MFCOM Service
- Start the Citrix SMA Service (though, I've noticed this often start up by itself, so you may not have to do this)
Credits to my esteemed colleague Grant for finding the fix.
Exchange 2007 Services Shutdown Order
Following on from my earlier post regarding the fun and games I've had with Exchange 2007, here's a brief running sheet I use when I want to shut down Exchange services, but keep the server running.
This is especially handy when you're performing network adapter driver updates, and your Exchange Information Store is hosted on an ISCSI LUN. Driver updates while the Store is still running == weird, weird issues and potential Store corruption!
net stop msexchangeadtopology /y
net stop msftesql-exchange /y
net stop msexchangeis /y
net stop msexchangesa /y
net stop iisadmin /y
Once these services have shutdown, you're free to proceed with any driver updates or cable pulling, or to continue shutting down other services on the same server.
Troubleshooting Fun with Exchange 2007 Queues
I recently resolved an issue, involving two Exchange 2007 servers in two different AD Sites. The issue was simply slow email delivery when emailing from Site 'A' to Site 'B', and a quick check showed that both servers had backlogged mail queues with no obvious cause.
Both sites are part of the same domain, both servers are identical in hardware (HP DL380 G5) and patch levels (Windows Server 2003 Standard x64 R2, and Exchange 2007 SP2). Connectivity between both sites tested perfectly, and talking to other servers in each site also revealed no issues. It was only when both the Exchange servers attempted to communicate, that the issue occured.
Mail in both queues reported errors of "451 4.4.0 Primary target IP address responded with: "421 4.4.2 Connection dropped." Attempted failover to alternate host, but that did not succeed. Either there are no alternate hosts or delivery failed to all alternate hosts." or "421 4.4.2 Connection dropped.", which seemed to point to network issues. Packet captures from both servers also showed a large amount of retransmits on both SMTP and SMB communication:
SMTP:
338 XXXMAIL02 192.168.15.63 SMTP SMTP:Cmd EHLO XXXMAIL02.testdomain.com, 31 bytes
1197 192.168.15.63 XXXMAIL02 SMTP SMTP:Rsp 250 -YYYMAIL02.testdomain.com Hello [192.168.24.34], 255 bytes
1198 XXXMAIL02 192.168.15.63 SMTP SMTP:Data Payload, 16 bytes
4159 XXXMAIL02 192.168.15.63 TCP TCP:[ReTransmit #1198] [Bad CheckSum]Flags=...AP..., SrcPort=44217, DstPort=SMTP(25), PayloadLen=16, Seq=3183382952 - 3183382968, Ack=1774779495, Win=65181
8142 XXXMAIL02 192.168.15.63 TCP TCP:[ReTransmit #1198] [Bad CheckSum]Flags=...AP..., SrcPort=44217, DstPort=SMTP(25), PayloadLen=16, Seq=3183382952 - 3183382968, Ack=1774779495, Win=65181
11786 XXXMAIL02 192.168.15.63 TCP TCP:[ReTransmit #1198] [Bad CheckSum]Flags=...AP..., SrcPort=44217, DstPort=SMTP(25), PayloadLen=16, Seq=3183382952 - 3183382968, Ack=1774779495, Win=65181
15476 XXXMAIL02 192.168.15.63 TCP TCP:[ReTransmit #1198] [Bad CheckSum]Flags=...AP..., SrcPort=44217, DstPort=SMTP(25), PayloadLen=16, Seq=3183382952 - 3183382968, Ack=1774779495, Win=65181
17902 XXXMAIL02 192.168.15.63 TCP TCP:[ReTransmit #1198] [Bad CheckSum]Flags=...AP..., SrcPort=44217, DstPort=SMTP(25), PayloadLen=16, Seq=3183382952 - 3183382968, Ack=1774779495, Win=65181
20735 XXXMAIL02 192.168.15.63 TCP TCP:[ReTransmit #1198] [Bad CheckSum]Flags=...AP..., SrcPort=44217, DstPort=SMTP(25), PayloadLen=16, Seq=3183382952 - 3183382968, Ack=1774779495, Win=65181
23227 XXXMAIL02 192.168.15.63 TCP TCP:[ReTransmit #1198] [Bad CheckSum]Flags=...AP..., SrcPort=44217, DstPort=SMTP(25), PayloadLen=16, Seq=3183382952 - 3183382968, Ack=1774779495, Win=65181SMB:
1/5/2010 15:22 14560 {TCP:358, IPv4:16} XXXMAIL02 192.168.15.63 SMB SMB:R; Negotiate, Dialect is NT LM 0.12 (#5), SpnegoNegTokenInit
1/5/2010 15:22 14650 {TCP:358, IPv4:16} XXXXMAIL02 192.168.15.63 TCP TCP:[ReTransmit #14560] [Bad CheckSum]Flags=...AP..., SrcPort=Microsoft-DS(445), DstPort=44946, PayloadLen=186, Seq=3446414444 - 3446414630, Ack=2070264315, Win=65398 (scale factor 0x0) = 65398
1/5/2010 15:22 14943 {TCP:358, IPv4:16} XXXXMAIL02 192.168.15.63 TCP TCP:[ReTransmit #14560] [Bad CheckSum]Flags=...AP..., SrcPort=Microsoft-DS(445), DstPort=44946, PayloadLen=186, Seq=3446414444 - 3446414630, Ack=2070264315, Win=65398 (scale factor 0x0) = 65398
1/5/2010 15:23 15334 {TCP:358, IPv4:16} XXXXMAIL02 192.168.15.63 TCP TCP:[ReTransmit #14560] [Bad CheckSum]Flags=...AP..., SrcPort=Microsoft-DS(445), DstPort=44946, PayloadLen=186, Seq=3446414444 - 3446414630, Ack=2070264315, Win=65398 (scale factor 0x0) = 65398
1/5/2010 15:23 15862 {TCP:358, IPv4:16} XXXXMAIL02 192.168.15.63 TCP TCP:[ReTransmit #14560] [Bad CheckSum]Flags=...AP..., SrcPort=Microsoft-DS(445), DstPort=44946, PayloadLen=186, Seq=3446414444 - 3446414630, Ack=2070264315, Win=65398 (scale factor 0x0) = 65398
1/5/2010 15:23 16383 {TCP:358, IPv4:16} XXXXMAIL02 192.168.15.63 TCP TCP:[ReTransmit #14560] [Bad CheckSum]Flags=...AP..., SrcPort=Microsoft-DS(445), DstPort=44946, PayloadLen=186, Seq=3446414444 - 3446414630, Ack=2070264315, Win=65398 (scale factor 0x0) = 65398
1/5/2010 15:23 17225 {TCP:358, IPv4:16} XXXXMAIL02 192.168.15.63 TCP TCP:[ReTransmit #14560] [Bad CheckSum]Flags=...AP..., SrcPort=Microsoft-DS(445), DstPort=44946, PayloadLen=186, Seq=3446414444 - 3446414630, Ack=2070264315, Win=65398 (scale factor 0x0) = 65398
1/5/2010 15:24 18568 {TCP:358, IPv4:16} XXXXMAIL02 192.168.15.63 TCP TCP:[ReTransmit #14560] [Bad CheckSum]Flags=...AP..., SrcPort=Microsoft-DS(445), DstPort=44946, PayloadLen=186, Seq=3446414444 - 3446414630, Ack=2070264315, Win=65398 (scale factor 0x0) = 65398
Revisiting the issue, it was noticed that XXXMAIL02 had two network adapters in a Team, while YYYMAIL02 was running off a single network adapter. Both servers also had old network card drivers (the cards are HP NC373i Multifunction Gigabit Adapters, which are rebadged Broadcom cards, and were using driver v2.8.13.0 made on 30/06/2006), and as part of the troubleshooting we upgraded these drivers to the latest available versions (v5.0.13.0, 23/06/2009) at the next maintenance window. As part of the upgrade, XXXMAIL02 was changed from a Network Team to a single adapter, to match YYYMAIL02.
(Bootnote: We did the upgrade by installing the latest Proliant Support Pack, and ran into a small issue of note while doing so. You can't upgrade the network drivers straight to v5.0.13.0, otherwise the installation will fail with an error "HP Virtual Bus Device installation requires a newer version. Version 4.6.16.0 is required". The easy way around this is to download v4.6.16.0 from HP (64-bit here, 32-bit here), and install this prior to the running the PSP.)
Within minutes of the upgrade being completed, mail and other traffic was flowing freely between both servers. A speedtest was run using iperf, which showed speeds of ~60Mb/s (previously we were seeing ~557bytes/s), and new emails were being delivered to the server within seconds.
This was a tricky one to diagnose - but it proves how often simple things are overlooked, in search of a bigger problem!
The Fortigate and the 3G Modem
Say what you will about it, but I think that the Fortinet Fortigate 60B is a nice piece of gear.
For the purchase price, you get a grunty firewall with 2 WAN Ports, a dedicated DMZ Port, 6 Fast Ethernet Ports and a PCMCIA Slot. You also score two USB Ports, which can be used to power a USB 3G Modem. With the right type of 3G Service, this setup provides you with the perfect temporary office network - a situation where you only need basic services for few users who require access to the corporate LAN.
I'm going to quickly run through setting up the connection with some common Australian 3G Providers, and how to debug any issues that may arise.
For the purposes of this demo, I'm using:
- Fortinet Fortigate 60B (running FortiOS v4.1 Patch 1)
- Huawei E169G 3G USB Modem
- 3G Services provided by Three
Owing to some inconsistencies in the Administration Guide, this config will be entered using the CLI. Don't be scared - it won't bite!
Firstly, ensure that your modem is firmly plugged into a USB Port on the back of the Fortigate, and that you've power-cycled the unit to detect the modem. You'll need to enable the modem with the following command:
config system modem
set status enable
end
Next, try and detect the custom vendor and product IDs with the following command. Be sure to note it down, as you'll need it later!
FortiGate # diagnose sys modem wireless-id
vendor: 0x12d1, product: 0x1003, registered: yes
Next, we'll configure the modem settings in our FortiGate to activate the modem connection:
config system modem
set status enable
set status enable
set dial-on-demand enable
set connect-timeout 30
set wireless-custom-vendor-id 0x12d1
set wireless-custom-product-id 0x1003
set modem-dev1 pcmcia-wireless
set phone1 "*99#"
set username1 "3services" # Set this to your provider's APN
set altmode disable
end
Special Note: If you're a Virgin Broadband user, ensure you also configure set authtype1 pap. While every other provider has moved with the times, and utilise the more secure and robust CHAP Authentication (which is the default option on the FortiGate), Virgin still use PAP which needs to be manually configured to ensure a successful connection.
We're almost there! The last thing to do is to turn on debugging (to watch the progress of the dial), and to actually execute the dial:
diagnose debug enable
diagnose debug application ppp 255
diagnose debug app modemd 255
execute modem dial
With a little luck, and a little hope, you'll see the logs go rushing by, and the modem will establish a connection to your provider. You should now conduct tests to verify your connectivity (after establishing the appropriate firewall rules), or (if unsucessful) review the ppp and modemd logs to see if you can determine what fouled up. Common causes are the modem not being detected, the wrong APN being provided to the FortiGate, or (funnily enough) the SIM Card not being activated for the APN you're dialling!
Once you're done, don't forget to turn off your logging:
diagnose debug application modemd 0
diagnose debug application ppp 0
diagnose debug disable
And that's it!