Spark Review

Spark is a popular and versatile tool for data movement and processing. It even does a pretty good job acting as a data warehouse using a pattern called a data lakehouse.

Spark itself is open-source and many organizations self-manage the open-source version. However, the people who originally authored the software started a company called Databricks which has become almost synonymous with the project.

For me, the primary use case for Spark is extracting semi-structured or unstructured data, then processing and dumping it into object storage (such as S3) to create a data lakehouse. It should be noted that this is a key task in creating an AI training dataset.

If your data is already in a relational format (for example, if it came from a database), I'd recommend checking out traditional data warehouses.

Spark, and by extension Databricks, is frequently compared to Snowflake. However, I think their products complement each other, and aren't really direct competitors.

Pros

• Open-source and actively developed.
• Is an excellent solution for marshalling semi-structured and unstructured data into a relational format - a key task for building AI datasets.
• When set up correctly (such as when rented through Databricks), Spark is unbelievably fast at processing data in a parallel fashion.
• Can do a respectable job approximating a traditional data warehouse.
• Has a huge and active community.

Cons

• No built in user access control. This is somewhat mitigated if you use Databricks' managed service.
• Other systems can change data in the object storage buckets that Spark relies on. This means it is up to you and your organization to be disciplined about controlling access to those storage mediums.
• Less consistency / ACID guarantees than traditional data warehouses.
• It is frustrating when you come across a feature that is available in Databricks, but isn't available in open-source Spark.