How it works
In order to keep your datasources in sync and enable a single source of truth for all your data items rudol has to connect to your datasources with a certain frequency and read their metadata.
Ingestion
When you add your datasource you choose a certain connector to match the connection technology such as MySQL, PostgreSQL, Google BigQuery and more. We are constantly adding new connectors and you can ask for them too.
Each connector has a different set of login parameters and you'll have to create a connection user granting the required permissions so rudol can make a successful connection. Keep in mind that many features required different permissions like read only for schema meta information if you only want to sync structural information or read only for your schemas/tables if you intend to use Data Quality functions.
Learn how to secure your datasource connection following our Security Guidelines
You can add as many datasources as you want and organize them using tags or domains. Also, when you add a new datasources you have to choose the Scanning Frequency in which rudol will connect to your datasource to fetch the metadata. Choose a frequency that matches your team's deployment workflow.
Metadata sync
When a successful connection is made with your datasource and all metadata is extracted, rudol processes it to update your Data Catalog to exactly match your source structure. In this process different items can be created, updated or removed.
rudol will never modify your datasource's structure or content, removing a datasource from the platform will only delete its associated documentation and alerts. Keep in mind that removing a datasource from your Organization would cause all your teammates to lose access to it's items documentation.
rudol uses a Soft Delete approach when an existing item is missing from your datasource so you can still access to its documentation but it won't appear in Search results unless you explicitly add the Include deleted items filter
You can see a catalog item's evolution in the Activity section.
Subscriptions and alerts
After your datasource is synced you can Subscribe to any item to receive notifications when something changes. For example you can subscribe to a database table so that when a column is added (or removed) you are notified, that way you can prevent issues caused by changes that weren't communicated or planned.