Steampipe is a new data pipeline tool and abstraction layer that offers a comprehensive suite of tools to help you bring your data to life. In a nutshell, steampipe abstracts data sources, if you can think of these data sources as APIs, for example, any cloud API; Azure, GCP or AWS APIs or literal data files; a local CSV file, in a repository or even inside an S3 bucket - Steampipe maps these data sources to database schemas, tables and fields for you to query at your hearts desire using plain old SQL!

 

This makes for some fun and interesting relationships, for example, you can painlessly join live cloud configuration data with internal or external data sets to create new insights, or map users from your Slack to a Cloud provider - see above!
Clearly, this allows you to combine the simplicity and power of ETL with the scalability of cloud-based data, giving users the ability to quickly and easily build powerful pipelines for their data. And with features like Dashboards and Mods, users can transform raw data into meaningful insights, quickly and efficiently.

…and in the most secure mode - read-only!

Plain old SQL!

Despite our personal opinions, SQL is still a powerful database language used around the world. This is mostly due to its versatility and scalability. It gives users the flexibility to query and analyze data in any manner they choose. Additionally, SQL is easy to learn, making it a great choice for users of any skill level. Finally, gaining the skill of knowing SQL transcends into many areas of engineering, including data and performance improvements.

Steampipe accepts valid Postgres SQL, this allows you to easily filter, aggregate and group your data sources and JOIN them together in all the various ways you can express using SQL.
Steampipe does not “currently” support updates or deletes downstream to the relative data sources for the moment, however from what we understand an upcoming feature “data pipelines” will allow users to push updates back to various data sources.

That's not all, folks!

Besides exposing APIs as a queriable database, Steampipe has built a number of great features on top of this concept and the “meat” of the solution shines through when it comes to something called “Mods”!

• Plugins

  • Data sources or akin to Client APIs.

  • Examples: AWS API, Azure, GCP, Terraform file interrogation, etc

• Dashboards

  • Define and build custom dashboards using config files containing dashboard layouts, components and queries.

• Mods

  • Modules are really great, as this brings the ease of queries and dashboards together and tacking on controls and benchmarks, these allow you to verify the result of datasets.

  • Example mods include: Compliance, Security and Well-Architected for all of the cloud providers.

• Service

  • Run Steampipe as a service and expose your data sources as a Postgres Database, connect using your favourite language or IDE and explore the data available.

• Steampipe Cloud (SaaS)

  • A new service offering from the folks at Steampipe is building out, which allows you to use Steampipe in the cloud and includes Git integration. 

Steampipe at Daemon

Steampipe's powerful capabilities to query multiple cloud providers enable us at Daemon to seamlessly run custom checks, controls, and benchmarks for our customer's cloud workloads, making it easier for us to identify and resolve potential issues or areas that require further optimization.

Furthermore, as a consultancy we are continuously exploring and developing new tooling solutions for our customers therefore Steampipe assists us in significantly speeding up time to delivery and performing comprehensive AWS Well-Architected Reviews based on data-driven decisions!

Overall, Steampipe has been an invaluable tool for us, enabling us to streamline our cloud workload management processes, optimize our infrastructure, and maintain a more secure and efficient cloud environment.