The core functions of our product rely on our ability to sync roster data from the LMS or SIS into our database. In this article we'll discuss a bit about why we do this and provide some more details about how it works.
Why do we sync roster data?
- Data Consistency Writing an abstraction layer to handle the differences between LMS and SIS APIs is a complex and error-prone task. By syncing the data into our database, we can ensure that the data is consistent and reliable.
- API Consistency Every system that we connect to is a bit different and it would be a major headache if developers had to interact with the quirks of each one. By syncing the data into our database, we can provide a consistent API for our clients to interact with.
- Performance Most of the systems that we connect with are not designed for data extraction or permission management. Syncing the data into our database allows us to optimize the data for the specific needs of our clients. This can lead to significant performance improvements (e.g. larger page sizes, faster load times).
- Sharing & Privacy Most of the time, schools do not want to share all data with companies and companies don't want to receive more data than they need. By syncing the data into our database, we can provide a way for schools to control what data is shared and for companies to control what data they receive via Sharing Rules and the Data Privacy Mask. Loading this data as-needed would be virtually impossible at scale.
- Enichement, Transformation, and Overrides Sometimes, companies need data that is split between multiple systems, or they need it modified in a way that the source system doesn't support. By syncing the data into our database, we can provide a way for companies to enrich, transform, and override the data as needed.
How does it work?
Every day, at least once per day, Edlink initiates a new sync job for each of the schools that we are connected to. This job is responsible for syncing the roster data from the source system into our database. The job then waits in a queue of other jobs (if there are no free servers available) until it is processed by one of our servers.
When the job runs, Edlink will retrieve roster data from the source system and compare it to the data that we have stored in our database. If there are any differences, Edlink will update the data in our database to match the source system. If too much data is being deleted, Edlink will place the sync into the pending state where it can be reviewed by the school administrator or by our support team.
After the sync is complete, Edlink will begin the data enrichment process, followed by the materialization process. This process can take a few minutes, depending on the number of jobs that we are currently processing. Once the process is complete, the data will be available for use in our dashboard and via our API. These steps are detailed in our Data Pipeline document.
Frequently Asked Questions
Does the syncing process happen at a certain time of day?
The syncing process is initiated at least once per day, but the exact time that it happens is determined by our system. Syncs are happening constantly (not just at night) to distribute the load on our own servers as well as the LMS / SIS servers.
How long does the syncing process take?
The syncing process can take anywhere from a few minutes to a few hours, depending on the number of students and courses that need to be synced. The process is designed to be as fast as possible, but there are many factors that can affect the speed of the process.
What happens if the syncing process fails?
If the syncing process fails, Edlink will automatically retry the process.
Can I trigger the syncing process manually?
The syncing process is automated and can only be triggered manually by a school IT administrator or an Edlink support team member. Unfortunately, we cannot allow this to be triggered directly by companies. This is because there may be other companies connected to a data source and there may be valid reasons why a school admin does not want certain roster updates to be synced to Edlink.
Can I see the status of the syncing process?
Currently, Edlink reports the time of the last roster sync, but does not provide a detailed status of the syncing process. School IT administrators can monitor the sync status for their school. If you need more information about the syncing process, please contact our support team who may be able to provide more information.
What do I do if I want the process to happen more frequently?
It is possible for us to adjust the syncing process to happen more frequently for certain districts. If this is something that you believe you will need, please contact our support team.