Data Engineer Data Pipeline Design Interview Questions

Tracy M. - "ingestion, processing & storage layer to handle document processing client ->API gateway/entry point->object storage-> queue-> worker-> database data flow: client initiates document upload + status processing API gateway (upload endpoint: authenticates & authorizes request, creates pre-assigned url to upload document); status endpoint object storage - stores uploaded document unstructured data (images, pdfs, docx etc) via preassigned url Message queue to decouple ingestion from proc"See full answer

Karthik R. - "This is yet another classic case of evolution of data landscape to account for diversities in the data formats sacrificing restrictive but key components at first and added later to make the solution more effective. Data warehouse -> Data Lake -> Data Lakehouse (Data Lake + Data Warehouse) Data warehouse - A solution to store data in central place (analytics (read) heavy) with stringent schema (structured). Very useful for historical queries and analytics. Schema on write check. Only used for"See full answer

Joshua R. - "Hadoop is better than PySpark when you are dealing with extremely large scale, batch oriented, non-iterative workloads where in-memory computing isn't feasible/ necessary, like log storage or ETL workflows that don't require high response times. It's also better in situations where the Hadoop ecosystem is already deeply embedded and where there is a need for resource conscious, fault tolerant computation without the overhead of Spark's memory constraints. In these such scenarios, Hadoop's disk-b"See full answer

Anzhe M. - "There are 2 questions popping into my mind: Should the 2nd job have to kick off at 12:30AM? Are there others depending on the 2nd job? If both answers are no, we may simply postpone the second job to allow sufficient time for the first one to complete. If they are yeses, we could let the 2nd job retry to a certain amount of times. Make sure that even reaching the maximum of retries won't delay or fail the following jobs."See full answer

Ramagiri P. - "Medallion architecture is a layered data architecture used in lakehouse systems. Data flows through Bronze, Silver, and Gold layers where each layer improves data quality. Bronze stores raw data, Silver contains cleaned and validated datasets, and Gold provides aggregated business-ready data for analytics and reporting bronzedf = spark.read.json("/landing/apidata") bronze_df.write.format("delta").save("/bronze/users")"See full answer

Nitish C. - "Delta lake is a metadata layer on top of cloud storage which helps giving datalake transactional capabilities. It helps implement upsert/merge as it conforms a schema to the data assets stored in cloud. It also offers various other capabilities like liquid clustering,time travel, schema evolution,deletes."See full answer

Nitish C. - "All purpose cluster remains up and running for longer duration irrespective of the job hence preferred for notebooks, adhoc work whereas job cluster spins up as per the submitted job and shuts down post the completion hence preferred for production scheduled workloads as it also offers compute isolation"See full answer