Every listener needs to be operational 24/7 as its main function is to help other Member Associations (possibly being located in different time zones) avoid data duplication.


How to fix a non-responding listener


Overview


Step 1: Before you start


Do you have our logger implemented? We highly recommend that you do.


You can read how to implement the logger is SDK documentation in chapter 2.3


As a first step, you should check the logs if you see any errors or exceptions related to the listener there. It is possible that a solution will be available in the error message.


Step 2: Check this first


Make sure that the process that you use to host the listener is up and running.


What are the symptoms?

  • listener works correctly for some time, a then it doesn't -> check "Disposing of the listener Thread"
  • listener does not respond just for the first request -> check "Cold start problem"
  • listener does not respond to a request for a particular player -> check "Problems with particular Player"
  • listener does not respond to a request for a particular MA -> check "Wrong public key cached"
  • listener does not respond to any request -> check "Credentials problem"
  • listener works from time to time -> check "Timeout problem" and "Competing listeners"


Step 3: Common problems and solutions


  1. Disposing of the listener thread

    The root of the problem is the thread that is used to run the listener is disposed of. It may occur in some scenarios of hosting listeners. 

    Solution:
    You must understand why your thread was disposed of. Sometimes you might want to use libraries dedicated to run long-running tasks like Hangfire in .NET applications.


  2. Cold start problem

    Your application is working correctly, but it is put in idle mode after some time of not being used. This means that when the request comes, the application has to be woken up and it takes time. Because of this, sometimes it takes too long to serve the first request, and the operation times out.

    Solution:
    It depends on how you host your application. You might want to check if your way of hosting does not provide a way to avoid putting the application in idle mode. For example "Always On" flag in Azure web apps.
    As a last resort, you can periodically ping your application not allowing for putting it into idle mode.


  3. Wrong public key cached

    It can happen from time to time that particular MA uploads the new public key for encrypting communication. Your listener picks up the message and tries to compose the response to the requester. As part of the message composition it gets encrypted. Here is where this problem sometimes emerges. Your system is trying to use a cached public key, but it uses the wrong one.

    Solution:
    Clean cache. For .NET and Java simple restart should be enough. For PHP you might want to delete files under the directory: \vendor\fifaconnectid\php-sdk\lib\FifaConnectServiceBusSdk\.cache.


  4. Problems with a particular player

    When the problem is with the particular Player only, it is most likely not a problem with the listener. The problem sometimes emerges because your listener cannot compose a valid message with the data it has received.

    Solution:
    Investigate your logs. You should look for logs related to the validation of personal data.
    Investigate your implementation of PersonDataProvider (or IPersonDataProvider). 


  5. Credentials problem

    Each environment uses different credentials. It is not uncommon that the credentials for a lower environment are populated to higher environments. For example, beta credentials to the pre-production environment. When the listener is trying to read an incoming message, it fails due to the lack of valid credentials.

    Solution:
    Apply valid credentials. Make sure they are not hard-coded in the code. Put them in the environment configuration.


  6. Timeout problem

    The listener needs too much time to compose the response. The request is processed correctly on the recipient side, but it times out on the requester side.

    Solution:
    Investigate performance bottlenecks of your application. Most commonly, the problem is with an access to the database. You can investigate the logs to check how long did it take to process the request.


  7. Competing listeners

    There has to be only one listener running at the same time for a given environment.
    Otherwise, both listeners compete for incoming messages. It is quite common scenario to have 2 listeners, while only one of them is configured properly. The misconfigured one takes incoming messages, but does not reply back with details.

    Solution:
    Make sure that you do not have a forgotten listener process running somewhere.


  8. Getting additional SB logs

    You can try to obtain additional logs from the Service Bus in case of an Exception.

    Solution:
    Handle BackgroundListener's (EventHandler in PHP) ServiceBusException event and log exception details.



Still not working? Contact us by sending an email to our support at support.id@fifa.org

Don't forget to attach logs for our logger, information about how you host your listener and as much other information as possible.