Azure Data Engineer Associate Boot Camp

Azure Data Engineer Associate Boot Camp

DreamsPlus provides a comprehensive Azure Data Engineer Associate Boot Camp in Singapore and online, tailored to offer hands-on training and prepare participants for the Microsoft Data Engineering certification. This program is ideal for professionals in Singapore seeking to enhance their skills in Azure data solutions, data management, and data engineering technologies.

Design and implement data storage (15–20%)

Implement a partition strategy

•       Create a file partition strategy;
•       create a partition strategy for analytical workloads;
•       create a streaming workload strategy;
•       create a partition strategy for Azure Synapse Analytics;
•       Determine which instances of Azure Data Lake Storage Gen2 require partitioning.

Design and implement the data exploration layer

•       Utilising a computational solution that makes use of Spark cluster and SQL serverless, create and run queries.
•       It is recommended to utilise Azure Synapse Analytics database templates and to put them into practice.
•       Upload updated or new data lineage to Microsoft Purview.
•       Use the Microsoft Purview Data Catalogue to browse and search metadata.

Develop data processing (40–45%)

Ingest and transform data

•       Create and apply incremental loads
•       The following can be used to transform data
•       clean data,
•       handle duplicate data,
•       handle missing data,
•       handle late-arriving data,
•       split data,
•       shred JSON,
•       encode and decode data,
•       configure error handling for a transformation,
•       transform data using Apache Spark,
•       transform data using Transact-SQL (T-SQL), ingest and transform data using
•       Azure Synapse Pipelines transform data using Azure Stream Analytics.

Develop a batch processing solution

•       Utilise Azure Data Lake Storage,
•       Azure Databricks,
•       Azure Synapse Analytics,
•       Azure Data Factory to create batch processing solutions.
•       Data pipelines can be created,
•       Resources can be scaled,
•       Batch sizes can be specified,
•       Jupyter or Python notebooks can be integrated into a data pipeline,
•       data can be upserted,
•       data can be reverted to a previous state,
•       exception handling can be configured,
•       batch retention can be configured,
•       data can be read from and written to a delta lake.

Develop a stream processing solution

•       Use Azure Event Hubs and Stream Analytics to create a stream processing solution.
•       Using Spark structured streaming
•       process data
•       create windowed aggregates
•       handle schema drift
•       process time series data
•       process data across partitions
•       process within a single partition
•       scale resources
•       create tests for data pipelines
•       optimise pipelines for analytical or transactional purposes
•       manage interruptions; configure exception handling
•       upsert data
•       replay archived stream data

Manage batches and pipelines

• Manage data pipelines in Azure Data Factory or Azure Synapse Pipelines
• Handle failed batch loads
• Validate batch loads
• Arrange data pipelines using Azure Synapse Pipelines or Data Factory.
•  Manage Spark jobs in a pipeline
• Apply version control for pipeline artefacts

Secure, monitor, and optimize data storage and data processing (30–35%)

Implement data security

•       Put data masking into practice.
•       Implement row-level and column-level security
•       enable Azure role-based access control (RBAC)
•       create POSIX-like access control lists (ACLs) for Data Lake Storage Gen2
•       encrypt data both in transit and at rest
•       Establish a strategy for data preservation
•       create secure endpoints, both public and private
•       add resource tokens to Azure Databricks
•       load sensitive data into a Data Frame write encrypted
•       data to tables or Parquet files
•       handle sensitive data.

Monitor data storage and data processing

•       Install the logging that Azure Monitor uses.
•       Monitoring service configuration,
•       stream processing monitoring,
•       data movement performance measurement,
•       system-wide data statistics monitoring and updating,
•       data pipeline performance monitoring,
•       query performance measurement,
•       pipeline test scheduling and monitoring,
•       Azure interpretation Track logs and metrics Put in place a pipeline alerting plan.

Optimize and troubleshoot data storage and data processing

•       Compact tiny files
•       Optimise resource management
•       manage data skew
•       manage data spill
•       tune queries using indexers
•       tune queries using cache
•       Investigate a failed Spark task.
•       Investigate a failed pipeline run, encompassing actions carried out in external service