LogoLogo
Start Trial
  • Overview
    • What is DeltaStream?
    • Core Concepts
      • Access Control
      • Compute Pools
      • Data Store
      • Database
      • Function
      • Query
      • SQL
      • Visualizing Data Lineage
  • Getting Started
    • Free Trial Quick Start
    • Starting with the Web App
    • Starting with the CLI
  • How do I...?
    • Create and Manage Data Stores
      • Create Data Stores for Streaming Data
      • Explore Data Store and Topic Details
      • Use Multiple Data Stores in Queries
    • Manage Users and User Roles
      • Inviting Users to an Organization
      • Administering Users in your Organization
      • Using the CLI to Manage User Roles
      • Example: Setting Up Custom Roles for Production and Stage
    • Create DeltaStream Objects to Structure Raw Data
    • Use Namespacing for Organizing Data
    • Create and Query Materialized Views
    • Create a Compute Pool to Work with Iceberg
    • Create a Function
    • Secure my Connection to a Data Store
      • Introducing DeltaStream Private Links
      • Creating an AWS Private Link from DeltaStream to your Confluent Kafka Dedicated Cluster
      • Enabling Private Link Connectivity to Confluent Enterprise Cluster and Schema Registry
      • Creating a Private Link from DeltaStream to Amazon MSK
      • Creating a Private Link for RDS Databases
      • Deleting a Private Link
    • Serialize my Data
      • Working with ProtoBuf Serialized Data and DeltaStream Descriptors
      • Working with Avro Serialized Data and Schema Registries
      • Configuring Deserialization Error Handling
  • Integrations
    • Setting up Data Store Integrations
      • AWS S3
      • ClickHouse
      • Confluent Cloud
      • Databricks
      • Iceberg REST Catalog
      • PostgreSQL
      • Snowflake
      • WarpStream
  • Setting up Enterprise Security Integrations
    • Okta SAML Integration
    • Okta SCIM Integration
  • use cases
    • Using an AWS S3 Store as a Source to Feed an MSK Topic
  • Reference
    • Metrics
      • Prometheus Integration
      • Built-In Metrics
      • Custom Metrics in Functions
    • SQL Syntax
      • Data Formats (Serialization)
        • Serializing with JSON
        • Serializing with Primitive Data Types
        • Serializing with Protobuf
      • Data Types
      • Identifiers and Keywords
      • Command
        • ACCEPT INVITATION
        • CAN I
        • COPY DESCRIPTOR_SOURCE
        • COPY FUNCTION_SOURCE
        • DESCRIBE ENTITY
        • DESCRIBE QUERY
        • DESCRIBE QUERY METRICS
        • DESCRIBE QUERY EVENTS
        • DESCRIBE QUERY STATE
        • DESCRIBE RELATION
        • DESCRIBE RELATION COLUMNS
        • DESCRIBE ROLE
        • DESCRIBE SECURITY INTEGRATION
        • DESCRIBE <statement>
        • DESCRIBE STORE
        • DESCRIBE USER
        • GENERATE COLUMNS
        • GENERATE TEMPLATE
        • GRANT OWNERSHIP
        • GRANT PRIVILEGES
        • GRANT ROLE
        • INVITE USER
        • LIST API_TOKENS
        • LIST COMPUTE_POOLS
        • LIST DATABASES
        • LIST DESCRIPTORS
        • LIST DESCRIPTOR_SOURCES
        • LIST ENTITIES
        • LIST FUNCTIONS
        • LIST FUNCTION_SOURCES
        • LIST INVITATIONS
        • LIST METRICS INTEGRATIONS
        • LIST ORGANIZATIONS
        • LIST QUERIES
        • LIST RELATIONS
        • LIST ROLES
        • LIST SCHEMAS
        • LIST SCHEMA_REGISTRIES
        • LIST SECRETS
        • LIST SECURITY INTEGRATIONS
        • LIST STORES
        • LIST USERS
        • PRINT ENTITY
        • REJECT INVITATION
        • REVOKE INVITATION
        • REVOKE PRIVILEGES
        • REVOKE ROLE
        • SET DEFAULT
        • USE
        • START COMPUTE_POOL
        • STOP COMPUTE_POOL
      • DDL
        • ALTER API_TOKEN
        • ALTER SECURITY INTEGRATION
        • CREATE API_TOKEN
        • CREATE CHANGELOG
        • CREATE COMPUTE_POOL
        • CREATE DATABASE
        • CREATE DESCRIPTOR_SOURCE
        • CREATE ENTITY
        • CREATE FUNCTION_SOURCE
        • CREATE FUNCTION
        • CREATE INDEX
        • CREATE METRICS INTEGRATION
        • CREATE ORGANIZATION
        • CREATE ROLE
        • CREATE SCHEMA_REGISTRY
        • CREATE SCHEMA
        • CREATE SECRET
        • CREATE SECURITY INTEGRATION
        • CREATE STORE
        • CREATE STREAM
        • CREATE TABLE
        • DROP API_TOKEN
        • DROP CHANGELOG
        • DROP COMPUTE_POOL
        • DROP DATABASE
        • DROP DESCRIPTOR_SOURCE
        • DROP ENTITY
        • DROP FUNCTION_SOURCE
        • DROP FUNCTION
        • DROP METRICS INTEGRATION
        • DROP RELATION
        • DROP ROLE
        • DROP SCHEMA
        • DROP SCHEMA_REGISTRY
        • DROP SECRET
        • DROP SECURITY INTEGRATION
        • DROP STORE
        • DROP STREAM
        • DROP USER
        • START/STOP COMPUTE_POOL
        • UPDATE COMPUTE_POOL
        • UPDATE ENTITY
        • UPDATE SCHEMA_REGISTRY
        • UPDATE SECRET
        • UPDATE STORE
      • Query
        • APPLICATION
        • Change Data Capture (CDC)
        • CREATE CHANGELOG AS SELECT
        • CREATE STREAM AS SELECT
        • CREATE TABLE AS SELECT
        • Function
          • Built-in Functions
          • Row Metadata Functions
        • INSERT INTO
        • Materialized View
          • CREATE MATERIALIZED VIEW AS
          • SELECT (FROM MATERIALIZED VIEW)
        • Query Name and Version
        • Resume Query
        • RESTART QUERY
        • SELECT
          • FROM
          • JOIN
          • MATCH_RECOGNIZE
          • WITH (Common Table Expression)
        • TERMINATE QUERY
      • Sandbox
        • START SANDBOX
        • DESCRIBE SANDBOX
        • STOP SANDBOX
      • Row Key Definition
    • DeltaStream OpenAPI
      • Deltastream
      • Models
Powered by GitBook
On this page
  • Syntax
  • Description
  • Arguments
  • Examples
  1. Reference
  2. SQL Syntax
  3. Query
  4. SELECT

WITH (Common Table Expression)

PreviousMATCH_RECOGNIZENextTERMINATE QUERY

Last updated 2 months ago

Syntax

WITH
    with_name AS (select_statement)
    [, ...]
select_statement;

Description

The WITH clause is used for defining named subqueries that can be used as a common expression in other SELECT statements. These common expressions are referred to as Common Table Expressions (CTEs), and present a temporary view of the data that is projected from its select_statement. As a result, CTEs modularize queries, making them more maintainable and versatile than subqueries.

The result of a CTE is effectively a that can be used just like any other relation defined with a DDL or subquery, and by definition takes precedence over other relations defined using a DDL.

CTEs may not be used within a .

Arguments

with_name

A name for the select_statement that defines the CTE.

select_statement

See SELECT.

Examples

Using CTEs from other CTEs

CTEs are just like any other relations, and they can be used within the SELECT statement of other CTEs. In this example, c1 projects the viewtime of each pageid from pageviews, and c2 adds a processing time to the result of that before eventually projecting proc_time, pageid and viewtime in the main SELECT statement:

WITH
    c1 AS (SELECT pageid, viewtime FROM pageviews),
    c2 AS (SELECT NOW() AS proc_time, * FROM c1)
SELECT * FROM c2;

Joining a Stream CTE with a Changelog CTE

Each CTE represts a local Relation meaning that their grouping/aggregation or project reflects how they present the underlying data to the query that they're part of.

WITH
    c1 AS (SELECT * FROM pageviews WHERE pageid = 'Page_6'),
    c2 AS (
        SELECT userid, count(interests) AS interest_count
        FROM users
        GROUP BY userid)
SELECT
    p.userid AS pvid,
    u.userid AS uid,
    p.pageid,
    u.interest_count AS interest_count
FROM
    c1 p
JOIN
    c2 u
ON u.userid = p.userid;

Self-joining CTEs

In this example, a single CTE is written to reshape the pageviews stream, but used twice in the JOIN operation to self-join for the resulting expanded data. The result of the joined data can be used as projected by the CTE's SELECT statement — htat is, user ID as an integer:

WITH
    c1 AS (
        SELECT
            viewtime,
            CAST(SUBSTRING(pageid FROM 6) AS INTEGER) AS pid,
            CAST(SUBSTRING(userid FROM 6) AS INTEGER) AS uid
        FROM pageviews)
SELECT
    pl.uid AS lid,
    pr.uid AS rid,
    pl.pid,
    pr.viewtime AS viewtime
FROM
    c1 pl
JOIN
    c1 pr
WITHIN 1 MINUTE
ON pr.uid = pl.uid
WHERE pl.uid != 5;

Create a new stream from MATCH_RECOGNIZEd CTE

CREATE STREAM trips_delay_increasing
WITH
    trip_updates AS (
        SELECT
            trip,
            "stopTimeUpdate",
            vehicle,
            CAST(FROM_UNIXTIME("timestamp") AS TIMESTAMP) AS ts,
            "timestamp" AS epoch_secs,
            delay
        FROM
            nyc_bus_trip_updates)
AS SELECT
    trip,
    vehicle,
    CAST(
      FROM_UNIXTIME((start_epoch_secs + end_epoch_secs) / 2)
      AS TIMESTAMP
    ) AS avg_ts
FROM trip_updates
  MATCH_RECOGNIZE(
    PARTITION BY trip
    ORDER BY "ts"
    MEASURES
      C.row_timestamp AS row_timestamp,
      C.row_key AS row_key,
      C.row_metadata AS row_metadata,
      C.vehicle AS vehicle,
      A.epoch_secs AS start_epoch_secs,
      C.epoch_secs AS end_epoch_secs
    ONE ROW PER MATCH
    AFTER MATCH SKIP TO LAST C
    PATTERN (A B C)
    DEFINE
      A AS delay > 0,
      B AS delay > A.delay + 30,
      C AS delay > B.delay + 30
  ) AS MR WITH ('timestamp'='ts');

In the following example, c1 represents a over pageviews where Page_6 has been visited by a user, and c2 represents a over users where its grouping the changes by the userid column. When joining these two CTEs, the JOIN operation treats this as a Stream-Changelog join, and doesn't require a WITHIN window for the join criteria:

This example shows a real-world query pattern matching over bus trip updates (redefined with CTEs from our ). A local bus trip updates relation as defined in the trip_updates CTE, which is then used in the MATCH_RECOGNIZE update to find each vehicles average time at each stop:

Analyzing NYC Bus Data blog
Stream
Changelog
DeltaStream Object
Window Function