The core of Edlink's data pipeline is it's ability to retrieve roster data from an LMS or SIS. This data is used to power all other aspects of our platform. How we actually going about syncing this data varies by platform and the type of data we are syncing. This is a high-level overview of the different strategies that Edlink uses and why.
Sync Types
- Full Sync - A full sync is when we retrieve all of the data from the source system. This is necessary in most systems because there is not a reliable way to check for changed or deleted records.
- Partial Sync - Sometimes schools are too big for us to reliably conduct a full sync. When this is the case, we will internally adjust the sync strategy to use "partial syncs" instead. What this means is that a large job is split up into smaller jobs that are run over a longer period of time. This is a more complex strategy, but it allows us to sync data from very large schools. This will appear in the Edlink dashboard (to admins) as a "child sync". There may be many hundreds of child syncs attached to a single parent "full sync".
- LTI Sync - An LTI sync is specific to "pure" LTI 1.3 integrations (i.e. where Edlink has no API level access). Because the LTI specification does not allow Edlink to access class rosters before a person has attempted to sign into that class, we must perform a small and fast sync when a new class becomes visible to us.
- Attendance Sync - In certain cases, in order to support Near Realtime Attendance, attendance data is synced separately from roster data. This is because attendance data is typically much larger than roster data and is not needed as frequently. Attendance data is synced once every 24 hours.
Although some platforms are capable of providing deltas (i.e. changes) to Edlink, we generally choose not to sync this way. This is discussed in detail in our developer guide on syncing with events.
Sync Frequency
Roster data is synced from the LMS or SIS to Edlink on a regular basis. The frequency of the sync depends on the type of data that is being synced and the needs of the customer.
- Most data is synced once every 24 hours.
- There is no specific time of day that the sync happens. It can happen at any time during the day as our syncs are part of a large shared queue.
- Schools cannot currently control how frequently their data syncs, but this may be available in the future.
- It is unlikely that we will allow companies to initiate syncs as it might unintentionally interfere with school operations.
- School admins can manually trigger a sync from the Edlink dashboard.
Notes on Sync Frequency
- The Edlink team is able to adjust the sync frequency based on the needs of your product or the district.
- This type of adjustment should be specifically requested from your Edlink client success manager.
- We do not like to arbitrarily bump up sync frequency as this puts unnecessary strain on our servers and our partners' servers and typically results in minimal UX improvements.
- The larger the district, the harder it will be for us to adjust the frequency upward.
- Many of the systems that we integrate with do not provide good methods to retrieve data in bulk.
- In many cases, sync frequency is limited by the APIs that are provided by the source system.