Workflow management platforms are what data engineers use to schedule and coordinate the steps in a data pipeline – an activity sometimes referred to as…
Loading data that’s been stored in an S3 bucket into a Snowflake data warehouse is an incredibly common task for a data engineer. In an…
Though the title of this post may sound obvious to some, it’s not how most organizations function. Many excellent software engineers are led to believe…
Data Engineers are a hot commodity in 2020, but it’s surprising how misunderstood they are. Are they a software engineer with a hyped up job…
A while back, I wrote a post on why ELT is preferable to ETL with Amazon Redshift and other modern data warehouses such as Snowflake…
In the last few years, there’s been a noticeable shift at cutting edge organizations in how data teams are structured. No longer is data engineering…
When I read about the $30 Million dollar series C funding raised by Crunchbase this morning, it reminded me of a common struggle I see…
Whenever you visit a website, fill out a signup form, or request data from an API, you’re most likely making an HTTP POST or GET…
Cron is perhaps the most universally used scheduling tool in the data engineering community. It has been battle tested over decades on Linux and Unix…
Last week I had the pleasure of attending and speaking at the MinneBOS 2019 data science and analytics conference in Boston. The folks at MinneAnalytics…