TABLE OF CONTENTS

Introduction

This article describes the purpose, best practices and expected outcome of the exercise to bulk upload players' data in Preproduction environment.

 

One of the last mandatory checks before MA's integration with the Connect ID Service can be considered complete, is the bulk upload of players. A given MA is asked to register, in Preproduction environment, a significant number of their players - ideally all of them. 

 

There are a couple of reasons to do the bulk upload:

  • we want an MA to have an opportunity to test before they bulk upload data in Production. It is only natural that when you start dealing with high volume data, some of your code may fail or slow down unexpectedly. It is important to fix such errors before the Production stage.
  • since almost all MAs register their production data in Preproduction, a given MA can get an early understanding of how many international duplicates they may encounter in Production. For that benefit to work, real production data must be used, of course.
  • we want to verify that MA's system is able to handle and trace errors correctly. In particular the goal is to make sure that no data is lost as a result of connectivity or performance issues.


The process needs both some preparation and also some checks afterwards to prove that everything went as expected.


Before the process

  • make sure that you implemented the concept of a person local identifier. It is described in the SDK documentation, chapter: 3.1.5.2 Idempotency. In short, MA is expected, during players' registration, to send along a unique identifier of a player's record. These identifiers are stored in a hashed form and are used to prevent duplication of data in case of network or performance issues. Note that personLocalId is a parameter of the methods RegisterPersonAndWaitForDetailsInCaseOfDuplicates(...), RegisterPerson[Async](...)  and ForceRegisterPerson[Async](...), and has to be included whenever these methods are called, no matter if done from Registration Facade or Connect ID Client. 
  • make sure that your NRS is synchronised with Connect ID Service. For example, if you deleted some existing FIFA_IDs in your NRS, you should also request the clean-up in the Connect ID Service. In the bulk upload exercise we expect that both your NRS and Connect ID Service are in complete sync afterwards.
  • prepare your data set. You should know exactly how many records are in the registration scope. This will later simplify the checks and troubleshooting.
  • ensure that you implemented the Logger(...) interface. Again, this is something that is going to help troubleshoot any issues.
  • if time is an important factor, consider running the process on a smaller sample of data first and look for optimisations (mind the tips in the "Performance considerations" section)

Performance considerations

  • Make sure that your code is prepared for handling high volume of data. This usually includes parallel processing of up to 20 parallel concurrent registrations at the same time. More details can be found in the SDK documentation here: 3.2.2.2 Person bulk registration.
  • In case a duplicate is found, additional time is needed to fetch person details through the Service Bus if you use Facade.RegisterPersonAndWaitForDetailsInCaseOfDuplicates(...) method. Since you do not need the duplicate person details at that point in time, but just the information whether there is a potential duplicate, it is recommended to use the method client.RegisterPersonAsync(...) instead for the initial batch load process.
  • the process will run slower at first, because the environment needs to scale out. It will take around 90 minutes before the processing reaches its full capacity on the server side.
  • assuming that you are the only MA doing the upload, you may expect the performance of between 200 and 350 registrations / minute.

During the process

  • the process can run 24/7 - there are no limitations here.
  • when the processing finishes, we expect three groups of records to exist:
    • group 1 - most of the records will have received the FIFA_ID successfully
    • group 2 - for some records you will have got 409 errors. This is expected and means that there are potential duplicates detected. Don't do anything about those records, but in Production they will of course need to be resolved - we therefore highly recommend to develop a user interface that lists all persons that you have tried to register with FIFA Connect ID but that did not receive a FIFA ID, so that the registration officer can resolve those cases one by one. We also recommend that this list of duplicates can be filtered/ sorted by
      • National vs. International duplicates
      • Amateur vs. Professional player duplicates
      • Time of registration
    • group 3 - for a small number of records, you might have gotten neither a FIFA_ID, nor a 409. For example, you may have received a time-out error if there was a connectivity or performance issue. With some likelihood, FIFA_IDs for those records already exist on the Connect ID Service side but you've never received them.
  • run the process again for all records from group 3, e.g. through an automated retry of failed operations (obviously excluding 409 errors). If you have implemented person local identifier concept correctly, you should get the corresponding FIFA_IDs already existing on Connect ID Service side. The goal is that group 3 becomes empty.

After the process

  • export and send to Connect ID Team (support.id@fifa.org) the list of all FIFA_IDs of Persons that exist in your NRS. Note that we only need a flat list of FIFA_IDs for Persons. No other [personal] data is required. Organisations or Facilities are not required either.
  • Connect ID Team will perform a check to compare the list of FIFA_IDs sent with the list generated on the server. You will receive the results of this check.
  • if there is a discrepancy, e.g. a record exists centrally but not in the exported list, it needs to be investigated. In the ideal case, if all the steps have been carefully followed, there should be no discrepancy.