Overview

GitLab is a complete DevOps platform delivered as a single application. GitLab provides a single UI for development and operational teams within organizations to work concurrently throughout the application development to delivery life cycle. As a result of applications becoming more and more complex, organizations turn to automation, like CI/CD, in hopes to streamline processes. However, to increase reliability and consistency in automation beyond software development, more and more teams are resorting to declarative automation and their respective workflows in code and storing them in code repositories. Often and dynamically triggering automation as code to quickly iterate and deliver value…


Background

https://www.confluent.io/blog/building-a-microservices-ecosystem-with-kafka-streams-and-ksql/

Objectives

  • Persist events into Kafka by producing records that represent customer orders
  • Write a service that validates customer orders
  • Write a service that joins streaming order information with streaming payment information and data from a customer database
  • Define one set of criteria to filter records in a stream based on some criteria
  • Create a session window to define five-minute windows for processing
  • Create a state store for the Inventory Service
  • Create one persistent query that enriches the orders stream with customer information using a stream-table join

Persist events

An event is simply a thing that happened or occurred. An event in a…


Crawl Input Sales Data on S3

You received a CSV file with 100k sales records from the Globomantics Corporation. You need to achieve two objectives. First, transform the file into a Parquet file compressed with Snappy, so that it takes less storage and can be used by other Globomantics departments. Second, analyze the data and return the total sales amount.

This diagram shows the high-level pictures of how the challenges fit together. The S3 input data has 100k lines with sales details. These challenges will help you analyze the input sales data using a high-performance and low-cost approach.

The first challenge is about creating a Glue…


  • Create a Teradata source system running on a Compute Engine instance
  • Prepare the Teradata source system for the schema and data transfer
  • Configure the schema and data transfer service
  • Migrate the schema and data from Teradata to BigQuery
  • Translate Teradata SQL queries into compliant BigQuery Standard SQL

Creating the Teradata VM and downloading Teradata Express

Enable the required Cloud APIs:

The following APIs are required: Compute Engine, Cloud Storage, Cloud Storage JSON, Pub/Sub, BigQuery, BigQuery Data Transfer Service, and Cloud Build.

Check which APIs are enabled for the project

gcloud services list --enabled

Enable the following APIs:

Required for the data transfer service

gcloud services enable bigquery-json.googleapis.com

gcloud services enable storage-api.googleapis.com

gcloud services enable storage-component.googleapis.com

gcloud services enable pubsub.googleapis.com

gcloud services enable bigquerydatatransfer.googleapis.com

Required to run Teradata on a GCE instance

gcloud services enable…


Implement a Data Ingestion Solution Using Amazon Kinesis Video Streams

1. Create a Kinesis Video Stream for Media Playback

Globomantics is an analytics firm which handles computer vision projects for object detection, image classification, and image segmentation. Your role as a Data Architect is to stream real time video feeds from a source to AWS for further analytics. You’ll create a Kinesis Video Stream where live video feeds will later be ingested

  1. Log in to the AWS Console.
  2. Under Find Services, type in and then click Kinesis Video Streams.
  3. Click on Create.
  4. For Video stream name enter VP52M8OQZ10HOQRB, and click on Create video stream.

You will be at the Video streams page for VP52M8OQZ10HOQRB, and in the Video stream…

IamVigneshC

Business Analyst, Product Consultant, Techie

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store