Products - Velocity
Velocity is a SAP Certified solution for Data Ingestion/Syndication process that will transfer data from your SAP and non-SAP systems into your Azure data lake. Velocity can ingest huge data volumes in near real-time data from the full SAP suite including APO, HANA, S/4HANA.....
The process is:
- Fully automated
- Deployable in hours
- Does not require any additional infrastructure (ie. servers)
- Is fully end-user configurable and can be managed by administrators without any development to add new source SAP systems, new tables to the lake, etc.
Since the Velocity integration only has two endpoints (SAP and Azure), data is never 'at-rest' outside of the security of the SAP or Azure system(s), reducing security vulnerabilities such as Man-In-The-Middle attacks. During transfer, data is encrypted and only ever at-rest when saved within the security of the lake, no intermediate databases are required. Users also have the ability to encrypt the at-rest data in the lake if desired for additional security.
Velocity was created with data security in mind and leverages the SAP authorisation model to control data access from SAP to Azure.
Realtime SAP Reporting
Velocity keeps the data lake in sync with your SAP data and removes the dependency of overnight batch processing. There is no requirement for SAP-BW, nor any ETL tool/jobs nor any other batch file generation processes to complicate and elongate the time taken to process deltas/CDC into the target lake.
Velocity enables a near real-time data feed into your Data Lake providing visibility to issues instantly enabling action as they unfold as opposed to finding out about them 30+hrs later! This empowers you to take better control of your business by reacting to key metrics immediately as they occur, limiting impacts through immediate actionable insights.
As part of the Velocity deployment, we provide a suite of PowerBI dashboards that provide the ‘Top Ten of Everything’ for SAP systems, showing the best and worst key business metrics across your business, with real-time SAP data, enabling immediate action.
Deployment can be performed in hours using a straightforward process, and consists of only SAP transports and Azure code deployments. Once these are deployed, SAP datasets can be syndicated to Azure within minutes of installation.
Absolutely zero development is required to move any SAP data to the Azure data lake – including full CDC/deltas, ensuring that it is kept inline with source SAP system.
Users can add new SAP systems, new tables etc to the Azure Data lake in minutes without any user involvement - this is performed through a user configuration screen accessed via Azure.
The entire integration between SAP and Azure is configurable through Azure by admins which include:
- System connections
- Tables to ingest
- CDC method
- Data compression
- Max batch size (either in #records or KB)
- Destination file format (csv or Parquet)
- Target data encryption
Velocity also syndicates the SAP source data schema definition. This is extremely useful for processes that consume the data from the lake, since they can leverage this content to identify the structure definition when required. Schema definitions are held in the lake within a separate repository but available to any services that access the lake.
Logical Entity Maintenance in Azure
Velocity’s Logical Entity maintenance model minimises the need for external effort in understanding the complex SAP data model once your data is in the lake. Logical entities, such as Vendors, Customers, Materials and Purchasing data will come in logicalised structures that simplifies the consumption of your SAP data from the data lake.
While other solutions may simply syndicate the physical SAP tables to the data lake based on a user requirement definition – eg. MARA (client-level material masters).
This method serves the purpose of transferring the SAP physical tables to Azure, however this alone would require often costly knowledge and skills to make sense of the SAP physical data model in order to consume this information. Simply transferring physical SAP tables to the Azure data lake means that all consumers of the data need to understand the SAP physical data model, which is extremely inefficient. Velocity optimizes this for the business. With Velocity, users have the option of defining logical data objects (entities) to be maintained in the Lake in addition to the physical SAP tables. The benefits of this are significant.
Consider the following:
Imagine a PowerBI report that shows customer billing data, if this report was created against the SAP physical tables in the data lake, then this would require the report creator to understand and join around 50 tables. This means the creator requires an exhaustive understanding of the underlying SAP physical tables including cardinality, join types, foreign keys, etc.
Using Velocity, only 5 logical objects are created in the data lake. The report creator only needs to reference these objects for their report creation, significantly accelerating the development time.
Not only can report creators benefit from the logical entities maintained by Velocity, but any other process that consumes information from the data lake benefits too, with rapid delivery cycles only available as a result of a simplified data model to work with.
We have calculated reports can be produced 20x faster using Velocity produced data verses alternatives as a result of the efficiency gains provided.
Change Data Capture is the process of identifying changes (deltas) in the source system and replicating them across to the target, ensuring the target always reflects the data in the source. Velocity uses a sophisticated CDC solution that works at either the application or database log level to identify changes.
Velocity uses true CDC to ensure that the data lake filesystem contains a true representative image of the source data. Velocity identifies changes in the source SAP system and also maintains the same changes in the target data lake dataset. The data Velocity will work with all known SAP table types including Pool and Cluster tables.
No-one wants 1M records or 1Gb of data to be transferred from SAP to Azure in a single transfer. Velocity will always create packets/batches of data at user defined sizes (max 1k records or max 50KB) which are there streamed in parallel from SAP to Azure. This ensures infrastructure resources are protected from any large data movements.
Velocity leverages the inherent SAP resource management capabilities such as Load Balancing and Login Groups, that allow existing Basis team(s) to control the resource utilisation and monitoring of global ERP systems.
A user configurable option is available allowing the compression of in-transit data. Observations of high-volume CDC/Deltas of 100GB+ datasets have seen compression rates exceeding 95% which will keep your network team very happy indeed!