Synchronization Service restarts every few hours ...
Posted 08-16-2009 at 04:00 AM by hdawg
Tags sync service restart
Posted by Otto ... Synchronization Service restarts every few hours ... this is an interesting one; take a look at your servers and see if you're experiencing something similar.
... and the content:
________________________
I figured I'd share this, as it has now happened recently on two different servers in our environment (both running BES 4.1 SP6 MR3).
Every few hours (based on a synchronization event), you will notice the Synchronization Service restarts with no explanation of a crash, even with logging turned up to 15.
Quote:
<table border="0" cellpadding="6" cellspacing="0" width="100%"> <tbody><tr> <td class="alt2" style="border: 1px inset ;"> [30000] (08/14 20:12:06.886):{0x20C4} Current Date: 2009/08/14
[30000] (08/14 20:12:06.886):{0x20C4} [DIAG] EVENT=Thread_report, THREADID=0x20C4, THREADNAME="DebugLogger"
[30000] (08/14 20:12:06.886):{0x20C4} [ENV] Current Process: C:\Program Files\Research In Motion\BlackBerry Enterprise Server\SyncServer\BlackBerrySyncServer.exe
[30000] (08/14 20:12:06.886):{0x20C4} [ENV] Current Process logged on as: DOMAIN\besadmin, Start Time: Aug 14 2009 20:12:06, Uptime (seconds): 0
[30000] (08/14 20:12:06.886):{0x20C4} [ENV] Current Process id: 9364 </td> </tr> </tbody></table>
Just prior to this alert is a seemingly harmless entry for a wireless backup synchronization event for a user's saved messages:
Quote:
<table border="0" cellpadding="6" cellspacing="0" width="100%"> <tbody><tr> <td class="alt2" style="border: 1px inset ;"> [46031] (08/14 20:12:00.620):{0xA6C} [SYNC-DSession] Submit XML request to Backup connector. [Doe, John:9600]
[46042] (08/14 20:12:00.620):{0xB64} [SYNC-DSession] Received 20786 bytes from device. [Doe, John:9600, SID=459392082, SEQ=25/28]
[46052] (08/14 20:12:00.620):{0xB64} [SYNC-DSession] Contain 3 record(s) and 1 command(s) from Backup.Saved Email Messages [Doe, John:9600]
[46031] (08/14 20:12:00.667):{0xB64} [SYNC-DSession] Submit XML request to Backup connector. [Doe, John:9600]
[46042] (08/14 20:12:00.667):{0xBB4} [SYNC-DSession] Received 12710 bytes from device. [Doe, John:9600, SID=459392082, SEQ=26/28]
[46052] (08/14 20:12:00.667):{0xBB4} [SYNC-DSession] Contain 3 record(s) and 1 command(s) from Backup.Saved Email Messages [Doe, John:9600]
[46031] (08/14 20:12:00.682):{0xBB4} [SYNC-DSession] Submit XML request to Backup connector. [Doe, John:9600]
[46042] (08/14 20:12:00.682):{0x520} [SYNC-DSession] Received 18843 bytes from device. [Doe, John:9600, SID=459392082, SEQ=27/28]
[46052] (08/14 20:12:00.682):{0x520} [SYNC-DSession] Contain 2 record(s) and 1 command(s) from Backup.Saved Email Messages [Doe, John:9600]
[46031] (08/14 20:12:00.729):{0x520} [SYNC-DSession] Submit XML request to Backup connector. [Doe, John:9600]
[46042] (08/14 20:12:00.729):{0xB40} [SYNC-DSession] Received 65 bytes from device. [Doe, John:9600, SID=459392082, SEQ=28/28]
[46052] (08/14 20:12:00.729):{0xB40} [SYNC-DSession] Contain 1 record(s) and 1 command(s) from DevMgmt.Handheld Agent [Doe, John:9600]
[46031] (08/14 20:12:00.729):{0xB40} [SYNC-DSession] Submit XML request to DevMgmt connector. [Doe, John:9600]
[46037] (08/14 20:12:00.761):{0xCC4} [SYNC-DSession] UID Logging: U67108864. [Doe, John:9600, DS=DevMgmt, DB=Handheld Agent] </td> </tr> </tbody></table>
Unfortunately, within these few synchronization requests, there is a corrupted and/or oversized message that is causing the Synchronization Service to become exhausted and simply recycle (roughly 6 times per day, or every 4-4.5 hours, which is the default cycle for wireless backup synchronization).
The current workaround is to delete the saved message(s). Unfortunately, there's no real good way of identifying which one is the culprit, so obviously the best choice would be to remove all of them.
If this is not an option, make a backup of the saved messages and delete them one by one starting with the largest first. This is a once-per-4-hours trial-and-error exercise. Once you have identified the corrupt message, restore the saved messages from the backup (you'll likely need to disable wireless redirection using the ALT+CFNG from the Enterprise Activation screen to restore this database) and delete the culprit from the saved messages.
The Synchronization Service restarting every few hours, believe it or not, is not really that big of an impact on your environment, so don't feel the need to fix the issue immediately and know that you have time to do this from the trial-and-error deletion approach.
A more permanent workaround will apparently be included in BES 4.1 SP7. I am unsure if this is also an issue with BES 5.0 (and if there is a fix slated for SP1, should it be an issue). I also do not see a RIM KB on this, although it is a known issue (based on the fix being in SP7), nor do I see any forum posts anywhere about such a symptom and resolution.
... and the content:
________________________
I figured I'd share this, as it has now happened recently on two different servers in our environment (both running BES 4.1 SP6 MR3).
Every few hours (based on a synchronization event), you will notice the Synchronization Service restarts with no explanation of a crash, even with logging turned up to 15.
Quote:
<table border="0" cellpadding="6" cellspacing="0" width="100%"> <tbody><tr> <td class="alt2" style="border: 1px inset ;"> [30000] (08/14 20:12:06.886):{0x20C4} Current Date: 2009/08/14
[30000] (08/14 20:12:06.886):{0x20C4} [DIAG] EVENT=Thread_report, THREADID=0x20C4, THREADNAME="DebugLogger"
[30000] (08/14 20:12:06.886):{0x20C4} [ENV] Current Process: C:\Program Files\Research In Motion\BlackBerry Enterprise Server\SyncServer\BlackBerrySyncServer.exe
[30000] (08/14 20:12:06.886):{0x20C4} [ENV] Current Process logged on as: DOMAIN\besadmin, Start Time: Aug 14 2009 20:12:06, Uptime (seconds): 0
[30000] (08/14 20:12:06.886):{0x20C4} [ENV] Current Process id: 9364 </td> </tr> </tbody></table>
Just prior to this alert is a seemingly harmless entry for a wireless backup synchronization event for a user's saved messages:
Quote:
<table border="0" cellpadding="6" cellspacing="0" width="100%"> <tbody><tr> <td class="alt2" style="border: 1px inset ;"> [46031] (08/14 20:12:00.620):{0xA6C} [SYNC-DSession] Submit XML request to Backup connector. [Doe, John:9600]
[46042] (08/14 20:12:00.620):{0xB64} [SYNC-DSession] Received 20786 bytes from device. [Doe, John:9600, SID=459392082, SEQ=25/28]
[46052] (08/14 20:12:00.620):{0xB64} [SYNC-DSession] Contain 3 record(s) and 1 command(s) from Backup.Saved Email Messages [Doe, John:9600]
[46031] (08/14 20:12:00.667):{0xB64} [SYNC-DSession] Submit XML request to Backup connector. [Doe, John:9600]
[46042] (08/14 20:12:00.667):{0xBB4} [SYNC-DSession] Received 12710 bytes from device. [Doe, John:9600, SID=459392082, SEQ=26/28]
[46052] (08/14 20:12:00.667):{0xBB4} [SYNC-DSession] Contain 3 record(s) and 1 command(s) from Backup.Saved Email Messages [Doe, John:9600]
[46031] (08/14 20:12:00.682):{0xBB4} [SYNC-DSession] Submit XML request to Backup connector. [Doe, John:9600]
[46042] (08/14 20:12:00.682):{0x520} [SYNC-DSession] Received 18843 bytes from device. [Doe, John:9600, SID=459392082, SEQ=27/28]
[46052] (08/14 20:12:00.682):{0x520} [SYNC-DSession] Contain 2 record(s) and 1 command(s) from Backup.Saved Email Messages [Doe, John:9600]
[46031] (08/14 20:12:00.729):{0x520} [SYNC-DSession] Submit XML request to Backup connector. [Doe, John:9600]
[46042] (08/14 20:12:00.729):{0xB40} [SYNC-DSession] Received 65 bytes from device. [Doe, John:9600, SID=459392082, SEQ=28/28]
[46052] (08/14 20:12:00.729):{0xB40} [SYNC-DSession] Contain 1 record(s) and 1 command(s) from DevMgmt.Handheld Agent [Doe, John:9600]
[46031] (08/14 20:12:00.729):{0xB40} [SYNC-DSession] Submit XML request to DevMgmt connector. [Doe, John:9600]
[46037] (08/14 20:12:00.761):{0xCC4} [SYNC-DSession] UID Logging: U67108864. [Doe, John:9600, DS=DevMgmt, DB=Handheld Agent] </td> </tr> </tbody></table>
Unfortunately, within these few synchronization requests, there is a corrupted and/or oversized message that is causing the Synchronization Service to become exhausted and simply recycle (roughly 6 times per day, or every 4-4.5 hours, which is the default cycle for wireless backup synchronization).
The current workaround is to delete the saved message(s). Unfortunately, there's no real good way of identifying which one is the culprit, so obviously the best choice would be to remove all of them.
If this is not an option, make a backup of the saved messages and delete them one by one starting with the largest first. This is a once-per-4-hours trial-and-error exercise. Once you have identified the corrupt message, restore the saved messages from the backup (you'll likely need to disable wireless redirection using the ALT+CFNG from the Enterprise Activation screen to restore this database) and delete the culprit from the saved messages.
The Synchronization Service restarting every few hours, believe it or not, is not really that big of an impact on your environment, so don't feel the need to fix the issue immediately and know that you have time to do this from the trial-and-error deletion approach.
A more permanent workaround will apparently be included in BES 4.1 SP7. I am unsure if this is also an issue with BES 5.0 (and if there is a fix slated for SP1, should it be an issue). I also do not see a RIM KB on this, although it is a known issue (based on the fix being in SP7), nor do I see any forum posts anywhere about such a symptom and resolution.
Total Comments 0
