Function pipelines

URL: llms-txt#function-pipelines

Contents:

• Anatomy of a function pipeline
  • Timevectors
  • Custom operator
  • Pipeline elements
• Transform elements
  • Vectorized math functions
  • Unary mathematical functions
  • Binary mathematical functions
  • Compound transforms
  • Lambda elements

Function pipelines are an experimental feature, designed to radically improve how you write queries to analyze data in Postgres and SQL. They work by applying principles from functional programming and popular tools like Python Pandas, and PromQL.

Experimental features could have bugs. They might not be backwards compatible, and could be removed in future releases. Use these features at your own risk, and do not use any experimental features in production.

The timevector() function materializes all its data points in memory. This means that if you use it on a very large dataset, it runs out of memory. Do not use the timevector function on a large dataset, or in production.

SQL is the best language for data analysis, but it is not perfect, and at times it can be difficult to construct the query you want. For example, this query gets data from the last day from the measurements table, sorts the data by the time column, calculates the delta between the values, takes the absolute value of the delta, and then takes the sum of the result of the previous steps:

You can express the same query with a function pipeline like this:

Function pipelines are completely SQL compliant, meaning that any tool that speaks SQL is able to support data analysis using function pipelines.