API reference

Everything in this section is generated from the package’s docstrings. The narrative below orients you before the autodoc entries.

The public surface is small and stable. Everything importable from the top-level sql2sqlx package (listed in its __all__) is what semantic versioning covers.

Choosing an entry point

Three functions convert; two helpers support them.

Function	Input	Writes to disk?
convert_string(sql, options=None, name="input.sql")	a SQL string (one or many statements)	No - returns a result you can inspect or write_result.
convert_file(path, options=None)	one .sql file	No - returns a result.
convert_directory(input_dir, output_dir=None, options=None)	a directory tree (scanned by include_glob)	Only if output_dir is given.

parse_source(...) exposes just Phase 1 (lex + split + classify) for one source, returning a ParsedFile; write_result(result, output_dir) writes a ConversionResult to disk (and refuses paths that would escape the destination). All three converters run the same three-phase pipeline described in architecture; convert_directory additionally parallelizes parsing across a process pool.

sql2sqlx.convert_string(sql, options=None, name='input.sql')

Convert a SQL string to Dataform SQLX files.

Parameters:

• sql (str) – BigQuery SQL text (one or many statements).

• options (ConversionOptions | None) – Conversion options (defaults used when None).

• name (str) – Virtual file name used for layout, ordering and reports.

Returns:

The ConversionResult.

Return type:

ConversionResult

Example

>>> result = convert_string(
...     "CREATE TABLE ds.t AS SELECT 1 AS x;")
>>> result.files[0].action_type.value
'table'

sql2sqlx.convert_file(path, options=None)

Convert a single .sql file.

Parameters:

• path (str) – Path to the SQL file.

• options (ConversionOptions | None) – Conversion options (defaults used when None).

Returns:

The ConversionResult.

Return type:

ConversionResult

sql2sqlx.convert_directory(input_dir, output_dir=None, options=None)

Convert every matching SQL file under a directory tree.

Phase 1 (read + lex + split + classify) runs across a process pool sized by options.jobs (0 = one worker per CPU); linking and emission are a fast single-process metadata pass.

Parameters:

• input_dir (str) – Root directory scanned recursively with options.include_glob (default *.sql).

• output_dir (str | None) – When given, generated files are written beneath it (directories created as needed).

• options (ConversionOptions | None) – Conversion options (defaults used when None).

Returns:

The ConversionResult; per-file failures are reported in result.report.failures without aborting.

Raises:

NotADirectoryError – If input_dir is not a directory.

Return type:

ConversionResult

sql2sqlx.write_result(result, output_dir)

Write every generated file beneath output_dir.

Parameters:

• result (ConversionResult) – A conversion result.

• output_dir (str) – Destination root (created if missing).

Raises:

ConversionError – If a generated relative path would escape the destination (including through an existing symlink).

Return type:

None

sql2sqlx.parse_source(path, relpath, text, opts)

Parse one SQL source into action drafts.

Never raises for bad input: lexer failures are captured on the returned ParsedFile as error so a single broken file cannot abort a corpus conversion.

Parameters:

• path (str) – Original path (reporting only).

• relpath (str) – Path relative to the conversion root (layout + ordering).

• text (str) – Decoded file contents.

• opts (ConversionOptions) – Conversion options.

Returns:

The parsed file.

Return type:

ParsedFile

Options and results

ConversionOptions holds every knob; its defaults preserve semantics (see the CLI reference for the flag-to-field mapping). Its __post_init__ validates and normalizes on construction: the string enum values ("incremental", "mirror", …) are accepted as well as the enum members, so options are easy to build from JSON or config files, and invalid values raise ValueError immediately rather than silently mis-selecting a strategy.

A conversion returns a ConversionResult with two attributes: files (the generated SqlxFile objects, in deterministic sorted-path order) and report (a ConversionReport). All of these are plain, picklable dataclasses.

class sql2sqlx.ConversionOptions(default_project=None, default_dataset=None, insert_strategy=InsertStrategy.OPERATIONS, merge_strategy=MergeStrategy.OPERATIONS, plain_create_strategy=PlainCreateStrategy.OPERATIONS, if_not_exists_strategy=IfNotExistsStrategy.OPERATIONS, declare_external=False, protect_incrementals=True, annotate=True, tags=<factory>, layout=Layout.MIRROR, encoding='utf-8', include_glob='*.sql', jobs=0, default_location='US')

All knobs controlling a conversion. Safe defaults preserve semantics.

Parameters:

• default_project (str | None)

• default_dataset (str | None)

• insert_strategy (InsertStrategy)

• merge_strategy (MergeStrategy)

• plain_create_strategy (PlainCreateStrategy)

• if_not_exists_strategy (IfNotExistsStrategy)

• declare_external (bool)

• protect_incrementals (bool)

• annotate (bool)

• tags (List[str])

• layout (Layout)

• encoding (str)

• include_glob (str)

• jobs (int)

• default_location (str)

default_project

Project assumed for unqualified table paths, used for cross-file reference resolution and emitted as database only when the source SQL qualified it explicitly.

Type:

str | None

default_dataset

Dataset assumed for unqualified paths (also used to resolve single-part references).

Type:

str | None

default_location

Dataform project location used only when scaffolding workflow_settings.yaml.

Type:

str

insert_strategy

See InsertStrategy. The semantics-preserving default is operations; typed incremental conversion is opt-in.

Type:

sql2sqlx.model.InsertStrategy

merge_strategy

See MergeStrategy.

Type:

sql2sqlx.model.MergeStrategy

plain_create_strategy

See PlainCreateStrategy.

Type:

sql2sqlx.model.PlainCreateStrategy

if_not_exists_strategy

See IfNotExistsStrategy. The default keeps guarded table and view definitions verbatim as operations.

Type:

sql2sqlx.model.IfNotExistsStrategy

declare_external

If True, generate type: "declaration" actions for tables that are referenced but never produced by the corpus, and rewrite those references to ref() too. (INFORMATION_SCHEMA and region-* paths are excluded.)

Type:

bool

protect_incrementals

Emit protected: true on incrementals converted from INSERT/MERGE so an accidental --full-refresh cannot drop pre-existing rows.

Type:

bool

annotate

Prepend a provenance comment (source file/line and the original statement kind) to each generated body.

Type:

bool

tags

Extra Dataform tags added to every generated action.

Type:

List[str]

layout

See Layout.

Type:

sql2sqlx.model.Layout

encoding

Text encoding used to read input files.

Type:

str

include_glob

Filename glob for directory scans.

Type:

str

jobs

Worker processes for directory conversion; 0 = auto (os.cpu_count()), 1 = no multiprocessing.

Type:

int

class sql2sqlx.InsertStrategy(*values)

How INSERT INTO ... SELECT statements are converted.

INCREMENTAL = 'incremental'

Convert to a Dataform type: "incremental" action (opt-in).

OPERATIONS = 'operations'

Keep the statement verbatim as an operations action (default).

class sql2sqlx.MergeStrategy(*values)

How MERGE statements are converted.

OPERATIONS = 'operations'

Keep the statement verbatim as an operations action (default; exactly preserves semantics).

INCREMENTAL_WHEN_SAFE = 'incremental-when-safe'

Convert a shape-proven MERGE to type: "incremental" with uniqueKey and report the required target-schema precondition; otherwise fall back to operations.

class sql2sqlx.PlainCreateStrategy(*values)

How CREATE TABLE without AS SELECT is converted.

OPERATIONS = 'operations'

Keep the DDL verbatim as an operations action (default).

DECLARATION = 'declaration'

Emit a Dataform type: "declaration" instead (drops the DDL; use when such tables are externally managed sources).

class sql2sqlx.IfNotExistsStrategy(*values)

How guarded CREATE TABLE/VIEW ... AS statements are converted.

TABLE = 'table'

Convert to type: "table" and record a semantics warning (Dataform always creates-or-replaces). Opt-in.

OPERATIONS = 'operations'

Keep verbatim as operations (exact semantics; default).

class sql2sqlx.Layout(*values)

Output file layout for directory conversions.

MIRROR = 'mirror'

Mirror the input directory structure under the output root.

FLAT = 'flat'

Write all .sqlx files directly into the output root.

class sql2sqlx.ConversionResult(files, report)

Everything a conversion produces.

Parameters:

• files (List[SqlxFile])

• report (ConversionReport)

files

Generated .sqlx files (deterministic order: sorted by output path).

Type:

List[sql2sqlx.model.SqlxFile]

report

The run’s ConversionReport.

Type:

sql2sqlx.model.ConversionReport

class sql2sqlx.ConversionReport(files_read=0, statements=0, actions_by_type=<factory>, refs_rewritten=0, refs_unresolved=<factory>, warnings=<factory>, failures=<factory>, elapsed_seconds=0.0, input_bytes=0)

Aggregate result of a conversion run.

Parameters:

• files_read (int)

• statements (int)

• actions_by_type (Dict[str, int])

• refs_rewritten (int)

• refs_unresolved (List[str])

• warnings (List[ReportWarning])

• failures (Dict[str, str])

• elapsed_seconds (float)

• input_bytes (int)

files_read

Number of input files parsed (including failed ones).

Type:

int

statements

Number of top-level statements encountered.

Type:

int

actions_by_type

Count of generated actions per Dataform type.

Type:

Dict[str, int]

refs_rewritten

Number of table references rewritten to ref().

Type:

int

refs_unresolved

Distinct referenced-but-not-produced table paths that can be dataset-resolved, left as literals.

Type:

List[str]

warnings

All non-fatal findings.

Type:

List[sql2sqlx.model.ReportWarning]

failures

path -> error for files that could not be converted.

Type:

Dict[str, str]

elapsed_seconds

Wall-clock duration of the run.

Type:

float

input_bytes

Total size of the SQL read, in bytes.

Type:

int

to_dict()

Return a JSON-serializable representation of the report.

Return type:

Dict[str, Any]

class sql2sqlx.ReportWarning(code, message, path='', line=0)

A single non-fatal finding.

Parameters:

• code (str)

• message (str)

• path (str)

• line (int)

code

Stable machine-readable code (e.g. FALLBACK_OPERATIONS).

Type:

str

message

Human-readable explanation.

Type:

str

path

Source file the finding relates to.

Type:

str

line

1-based line number (0 when not applicable).

Type:

int

class sql2sqlx.SqlxFile(relpath, content, action_type, action_name, source_path='', source_line=0)

One generated .sqlx file.

Parameters:

• relpath (str)

• content (str)

• action_type (ActionType)

• action_name (str)

• source_path (str)

• source_line (int)

relpath

Output path relative to the output root (POSIX separators).

Type:

str

content

Full file contents.

Type:

str

action_type

The Dataform action type of the file.

Type:

sql2sqlx.model.ActionType

action_name

The Dataform action name (config.name).

Type:

str

source_path

Source file this action came from ("" for synthesized declarations).

Type:

str

source_line

1-based source line of the originating statement.

Type:

int

class sql2sqlx.ActionType(*values)

Dataform action types emitted by sql2sqlx.

class sql2sqlx.TableName(project, dataset, table)

A (possibly partial) BigQuery table path.

Parameters:

• project (str | None)

• dataset (str | None)

• table (str)

project

GCP project id, or None if the reference/target did not qualify one.

Type:

str | None

dataset

Dataset id, or None if unqualified.

Type:

str | None

table

Table id (always present).

Type:

str

static from_parts(parts)

Build a TableName from 1-3 decoded path parts.

Parameters:

parts (List[str]) – [table], [dataset, table] or [project, dataset, table].

Returns:

The corresponding TableName.

Raises:

ValueError – If parts is empty or longer than 3.

Return type:

TableName

resolve(default_project, default_dataset)

Fill missing qualifiers from defaults.

Parameters:

• default_project (str | None) – Project to assume when unqualified.

• default_dataset (str | None) – Dataset to assume when unqualified.

Returns:

A new TableName with defaults applied (fields that have no default remain None).

Return type:

TableName

key()

Return a hashable identity key (project, dataset, table).

Return type:

Tuple[str | None, str | None, str]

display()

Human-readable dotted form, e.g. proj.ds.table or ds.table.

Return type:

str

Errors

Every exception raised deliberately by the library derives from Sql2SqlxError, so a single except Sql2SqlxError catches them all. Location-aware errors (LexError, SplitError) carry 1-based line and column attributes. Note that unsupported-but-valid SQL never raises - it falls back to an operations action and a warning; the exceptions are for genuinely broken input (unreadable files, lexer failures). During directory conversion, a per-file failure is captured in report.failures rather than aborting the run.

Exception hierarchy for sql2sqlx.

All exceptions raised deliberately by this library derive from Sql2SqlxError, so callers can catch a single base class. Location-aware errors (lexing/splitting problems) carry a 1-based line and column pointing at the offending character in the original SQL text.

exception sql2sqlx.errors.Sql2SqlxError

Base class for all errors raised by sql2sqlx.

exception sql2sqlx.errors.LexError(message, line, column)

Raised when the SQL text cannot be tokenized.

Typical causes are unterminated string literals, unterminated backtick-quoted identifiers, or unterminated /* ... */ block comments.

Parameters:

• message (str)

• line (int)

• column (int)

Return type:

None

message

Human-readable description of the problem.

line

1-based line number of the offending character.

column

1-based column number of the offending character.

exception sql2sqlx.errors.SplitError(message, line=0, column=0)

Raised when statement splitting fails irrecoverably.

In practice the splitter is lenient (mismatched END markers are tolerated), so this error is reserved for structural impossibilities such as unbalanced parentheses at end of input.

Parameters:

• message (str)

• line (int)

• column (int)

Return type:

None

exception sql2sqlx.errors.ConversionError(message, path=None)

Raised when a statement or file cannot be converted at all.

Note that unsupported-but-valid SQL never raises: it falls back to a Dataform operations action and a warning is recorded in the ConversionReport. This exception is for genuinely broken input (e.g. unreadable files, lexer failures).

Parameters:

• message (str)

• path (Optional[str])

Return type:

None

Internals

The internal modules are stable enough to build on, but the public surface above is what semantic versioning covers.

A precise, linear-time lexer for GoogleSQL (BigQuery Standard SQL).

This module is the correctness foundation of sql2sqlx: every later stage (statement splitting, classification, reference rewriting) operates on the token stream produced here, so semicolons, keywords and table names that appear inside strings or comments can never be misinterpreted.

The lexer recognizes, per the GoogleSQL lexical specification:

• line comments (-- ... and # ...);

• block comments (/* ... */, non-nested, exactly as GoogleSQL defines them - the comment ends at the first */);

• single- and double-quoted string literals with backslash escape sequences ('it\'s');

• triple-quoted string literals (three single or double quotes) which may span lines and contain quotes;

• raw and bytes literal prefixes in any valid combination and case (r'...', B"...", rb'''...''', bR"...", …). In raw literals escape sequences are not interpreted (the backslash stays in the value), but a backslash still consumes the character after it, so r'a\'b' is one literal and a raw literal can never end in an odd number of backslashes - exactly GoogleSQL’s rules;

• backtick-quoted identifiers, including escape sequences and embedded dots (project.dataset.table wrapped in backticks);

• numeric literals (integers, decimals, exponents);

• named and positional query parameters (@param, ?) and system variables (@@var);

• identifiers/keywords and all GoogleSQL operators and punctuation.

Design notes

The scanner is a single compiled alternation executed by CPython’s C regex engine, which gives 20-60 MB/s throughput while remaining exactly character-accurate. Every alternative in the pattern is written so that matching is strictly linear (no ambiguous nested quantifiers, hence no catastrophic backtracking). Anything the master pattern cannot match is diagnosed by a small fallback that raises LexError with an exact line/column.

Tokens store (kind, text, start, end) where start/end are character offsets into the original text; downstream stages rewrite SQL by span edits on the original string, guaranteeing that untouched SQL is preserved character-for-character after decoding.

class sql2sqlx.lexer.Token(kind, text, start, end)

A single lexical token.

Parameters:

• kind (str)

• text (str)

• start (int)

• end (int)

kind

One of IDENT, BACKTICK, STRING, NUMBER, PARAM, OP, COMMENT, EOF.

text

The exact source text of the token (including quotes, prefixes and backticks where applicable).

start

Character offset of the first character in the source.

end

Character offset one past the last character in the source.

property upper: str

Uppercased token text, used for case-insensitive keyword tests.

sql2sqlx.lexer.tokenize(text, keep_comments=False, comment_spans_out=None)

Tokenize GoogleSQL text into a list of Token objects.

Whitespace is always dropped. Comments are dropped unless keep_comments is true (statement splitting only needs significant tokens); their spans can be captured in the same pass via comment_spans_out, which the conversion pipeline uses so files are lexed exactly once.

Parameters:

• text (str) – SQL source text.

• keep_comments (bool) – If True, COMMENT tokens are included in the returned stream (in source order).

• comment_spans_out (List[Tuple[int, int]] | None) – Optional list that receives every comment’s (start, end) span, regardless of keep_comments.

Returns:

List of significant tokens in source order, terminated by a synthetic EOF token whose span is (len(text), len(text)).

Raises:

LexError – If the text contains an unterminated string, unterminated backtick identifier, unterminated block comment, or a character that is not valid in GoogleSQL.

Return type:

List[Token]

Example

>>> [t.text for t in tokenize("SELECT 'a;b' -- c\n;")][:-1]
["SELECT", "'a;b'", ';']

sql2sqlx.lexer.comment_spans(text)

Return the (start, end) spans of every comment in text.

Used by the emitter to carry a statement’s leading comments into the generated .sqlx file. Runs the same master pattern, so a -- inside a string literal is never mistaken for a comment.

Parameters:

text (str) – SQL source text (must be lexable).

Returns:

List of half-open character spans, in source order.

Raises:

LexError – Propagated from tokenize() failure conditions.

Return type:

List[Tuple[int, int]]

class sql2sqlx.lexer.LineIndex(text)

Fast character-offset to (line, column) mapping for one text.

Builds the newline offset table once (O(n)) and answers each query in O(log n) via binary search. Used to attach accurate source locations to report warnings without rescanning the file per warning.

Parameters:

text (str)

locate(pos)

Return the 1-based (line, column) for character offset pos.

Parameters:

pos (int) – Character offset into the indexed text.

Returns:

Tuple of 1-based line and column numbers.

Return type:

Tuple[int, int]

sql2sqlx.lexer.unquote_identifier(text)

Return the logical name of an identifier token’s text.

For backtick-quoted identifiers the surrounding backticks are removed and GoogleSQL escape sequences are decoded (\` -> `` ` , ``\\ -> \, plus the standard C-style escapes). Unquoted identifiers are returned unchanged.

Parameters:

text (str) – Raw token text, e.g. "`my table`" or "orders".

Returns:

The decoded identifier, e.g. "my table" or "orders".

Return type:

str

Split a token stream into top-level GoogleSQL statements.

Splitting SQL on semicolons is only correct if three things are tracked:

1. Strings and comments - handled upstream: the splitter operates on the significant token stream from sql2sqlx.lexer, so a ; inside 'a;b' or /* ; */ simply never appears here.

2. Parenthesis depth - a ; can only end a statement at paren depth 0 (defensive; valid GoogleSQL has no parenthesized semicolons).

3. Procedural blocks - BigQuery scripting statements contain inner semicolons that must not split:

   BEGIN ... END;   IF c THEN ...; END IF;   LOOP ...; END LOOP;
   WHILE c DO ...; END WHILE;   REPEAT ...; UNTIL c END REPEAT;
   FOR x IN (q) DO ...; END FOR;   CASE WHEN c THEN ...; END CASE;
   

The block tracker maintains a frame stack. Openers are recognized in statement position (start of input, after ;, after BEGIN, LOOP, REPEAT, or after the THEN/ELSE/DO of a scripting frame), after a block label, or at the body of a CREATE PROCEDURE. Statement position distinguishes the scripting IF c THEN statement from the IF(a, b, c) expression function and the LOOP/WHILE/FOR statements from the same words used as (quoted-elsewhere) identifiers. CASE opens a frame in any position because both the expression form (CASE ... END) and the scripting form (CASE ... END CASE) are symmetric push/pop pairs; the two are told apart by whether the CASE itself sits in statement position. Inside an expression CASE, THEN/ELSE are followed by expressions, so they must not grant statement position - otherwise a column named loop or begin right after them would open a bogus frame, desynchronize END matching and glue unrelated statements together. Likewise ELSEIF is followed by a condition; only its own THEN starts the branch statements.

For the one genuinely ambiguous opener - IF in statement position can still be the expression function when it follows THEN/ELSE of a CASE expression - a bounded, nesting-aware lookahead searches for the scripting condition’s top-level THEN without confusing a nested CASE.

BEGIN immediately followed by ; or TRANSACTION is the transaction-control statement, not a block.

END closers are matched leniently (an END X pops the innermost frame even on a type mismatch) because the splitter’s only job is to find top-level semicolons; leniency degrades gracefully into “statements kept together”, which downstream classification handles safely as an operations action, whereas strictness would reject convertible files.

class sql2sqlx.splitter.RawStatement(tokens, terminated, end=None, terminator_end=None)

One top-level statement as a slice of the token stream.

Parameters:

• tokens (List[Token])

• terminated (bool)

• end (Optional[int])

• terminator_end (Optional[int])

tokens

The significant tokens of the statement, excluding the terminating semicolon.

start

Character offset of the first token.

end

Character offset one past the last token (semicolon excluded).

terminator_end

Offset immediately after the terminating semicolon, or end when the final statement was unterminated.

terminated

Whether the statement was followed by ; in source.

sql2sqlx.splitter.split_statements(tokens, statement_starts_out=None)

Split a significant-token stream into top-level statements.

Parameters:

• tokens (List[Token]) – Token stream from sql2sqlx.lexer.tokenize() (must end with the synthetic EOF).

• statement_starts_out (Set[int] | None) – Optional set that receives the source offset of every top-level or nested statement head. The offsets are derived from the same block-aware state machine used for splitting, so a keyword after a scripting THEN is included while the same keyword after a CASE expression is not.

Returns:

List of RawStatement, in source order. Empty statements (e.g. from ;; or a trailing ;) are omitted.

Return type:

List[RawStatement]

Example

>>> from sql2sqlx.lexer import tokenize
>>> stmts = split_statements(tokenize(
...     "CREATE TABLE d.a AS SELECT 1; BEGIN DELETE d.a WHERE true; END;"))
>>> len(stmts)
2

Statement classification and Dataform metadata extraction.

This module decides, for every top-level statement, which Dataform action type it becomes and extracts everything needed for the config { ... } block. The governing principle is:

Convert when provably safe; otherwise fall back to a verbatim ``operations`` action and record a warning.

An operations action executes the original SQL unchanged, so the fallback path can never alter behavior - it only forgoes idiomatic Dataform structure. Every fallback carries a stable warning code and a human-readable reason in the conversion report.

Mapping summary

Statement	Result
CREATE [OR REPLACE] TABLE .. AS	type: "table" (+ partition/cluster/ OPTIONS mapping)
CREATE [MATERIALIZED] VIEW .. AS	type: "view" (+ materialized)
INSERT INTO .. SELECT	type: "incremental" (configurable)
MERGE	operations by default; provably equivalent MERGEs can opt into incremental + uniqueKey
CREATE TABLE (no AS)	operations (or declaration)
UPDATE/DELETE/TRUNCATE/DROP/ALTER/	operations with write-target
LOAD	tracking for dependency chaining
CREATE/DROP/ALTER SEARCH|VECTOR	operations tracked as a writer of
INDEX, ROW ACCESS POLICY	the ON table (ordered after it,
DDL, table-scoped GRANT/	never elected as its owner)
REVOKE
everything else	operations

Select-list aliasing

INSERT INTO t (a, b) SELECT x, y ... and CREATE VIEW v (a, b) AS SELECT ... change output column names. To convert these faithfully the select list is rewritten item-by-item (x AS a, y AS b) - but only when every item’s existing output name/alias can be determined exactly at the token level. Constructs that make that ambiguous (* expansion, INTERVAL 1 DAY implicit-alias traps, naked STRUCT<...>/ ARRAY<...> constructors, SELECT AS STRUCT) trigger the operations fallback instead of a guess.

sql2sqlx.parser.classify_statement(stmt, text, opts)

Classify one top-level statement into a Dataform action draft.

Parameters:

• stmt (RawStatement) – The raw statement from the splitter.

• text (str) – Full source text of the file (for raw span capture).

• opts (ConversionOptions) – Conversion options.

Returns:

An ActionDraft. This function never raises for valid-but-unsupported SQL - such statements become operations drafts with explanatory warnings.

Return type:

ActionDraft

Table-path parsing and reference-site discovery.

The reference scanner finds every position in a statement where a table is being read - the places that can safely be rewritten to Dataform ${ref(...)} calls. Doing this on raw text would be hopeless; doing it on the token stream with a small amount of context tracking is exact for the constructs that matter and conservative everywhere else (when in doubt, a reference is left untouched, which always yields valid SQLX).

What is captured

A candidate reference site is a table path appearing:

• after FROM (except a FROM inside EXTRACT( ... ));

• after any JOIN;

• after a , inside an active FROM table list (comma joins);

• after USING of a top-level MERGE statement.

What is excluded

• UNNEST(...) and table-valued function calls (name( directly after the table position);

• subqueries and parenthesized join trees (recursed into instead);

• references to a CTE in the CTE’s actual visibility range;

• paths whose first segment is a visible range variable (FROM aliases are tracked per query block, including correlated scalar subqueries, without leaking aliases out of nested or set-operation query blocks);

• paths inside caller-supplied skip spans (e.g. the target of DELETE FROM target);

• paths longer than three segments (correlated array references such as FROM t, t.array_col at four+ parts).

Path syntax handled

dataset.table, project.dataset.table, any mix of backticked segments (`` p.d.t , `` `p`.`d`.`t` ``, `` `p.d`.t ``), whitespace around dots, and BigQuery *dashed* project names (``my-project-123.d.t) - dash joining requires character adjacency, exactly like BigQuery’s own lexer, so a - b in an expression is never mistaken for a path.

class sql2sqlx.refs.PathMatch(parts, start, end, next_index)

A parsed table path within the token stream.

Parameters:

• parts (List[str])

• start (int)

• end (int)

• next_index (int)

parts

Decoded path segments (backticks removed, escapes decoded, dotted backtick contents split).

start

Character offset of the first path character.

end

Character offset one past the last path character.

next_index

Token index immediately after the path.

sql2sqlx.refs.parse_table_path(tokens, i)

Parse a (possibly qualified) table path at token index i.

Parameters:

• tokens (Sequence[Token]) – The token stream (must be EOF-terminated).

• i (int) – Index of the first token of the candidate path.

Returns:

A PathMatch, or None when tokens[i] cannot begin a path (wrong kind, an unquoted reserved keyword, or an index at or past the end of the token list - script statement slices are not EOF-terminated, so a truncated DELETE/RENAME TO/… can legitimately ask for the position after the last token).

Return type:

PathMatch | None

Example

>>> from sql2sqlx.lexer import tokenize
>>> pm = parse_table_path(tokenize("`my-proj`.sales . orders x"), 0)
>>> pm.parts
['my-proj', 'sales', 'orders']

sql2sqlx.refs.scan_ref_sites(tokens, stmt_kind, skip_spans=(), statement_start_offsets=None)

Scan one statement’s tokens for rewritable table references.

Parameters:

• tokens (Sequence[Token]) – Significant tokens of the statement (EOF terminator is tolerated but not required).

• stmt_kind (str) – Uppercase leading keyword of the statement ("MERGE" enables USING as a table introducer at depth 0).

• skip_spans (Sequence[Tuple[int, int]]) – Character spans (absolute offsets) whose contained paths must not be reported - typically the statement’s own write target.

• statement_start_offsets (Set[int] | None) – Optional block-aware source offsets for nested statement heads. Supplying these lets a whole-script scan recognize nested MERGE ... USING statements without treating an arbitrary MERGE token as a statement.

Returns:

Filtered list of RefSite in source order. CTE references and paths rooted at a visible range variable are removed.

Return type:

List[RefSite]

Render Dataform .sqlx files.

The emitter’s contract is character fidelity for untouched SQL: generated bodies are produced by applying span edits (reference rewrites, select-list aliases, ${self()} substitutions) to slices of the original source text - tokens are never re-serialized, so the user’s formatting, casing and inline comments survive after input decoding.

SQLX-specific safety rules live here:

• Any literal ${ in a SQL string or quoted identifier is emitted through a constant JavaScript placeholder. Dataform does not provide a backslash escape for SQL placeholders, so \${ is insufficient. Replacement text inserted by sql2sqlx (${ref(...)}, ${self()}) stays active.

• GoogleSQL comments with SQLX-specific lexical hazards (# comments and exact --- separator-looking comments) are normalized without changing BigQuery semantics.

• Operations bodies have exactly one trailing semicolon stripped: Dataform executes the body as a BigQuery script, and a trailing empty statement is at best noise.

sql2sqlx.emitter.sqlx_constant(value)

Return a SQLX placeholder that evaluates to the constant value.

Parameters:

value (str)

Return type:

str

sql2sqlx.emitter.normalize_sqlx_comment(comment)

Return a SQLX-safe spelling of a single GoogleSQL comment token.

Dataform SQLX treats --- at the start of a line as a statement separator and does not recognize GoogleSQL # line comments. The rewrites here preserve BigQuery semantics while preventing comment text from being parsed as SQLX syntax.

Parameters:

comment (str)

Return type:

str

sql2sqlx.emitter.sqlx_escape_edits(tokens)

Build edits that keep literal SQL safe when parsed as SQLX.

Literal ${ in SQL strings, quoted identifiers, and quoted parameters is emitted through a constant JavaScript placeholder because Dataform has no SQL-level escape for placeholders. String literals are replaced as whole tokens so safety does not depend on Dataform’s per-quote lexer states. A string that carries a control character (for example a multi-line triple-quoted literal) cannot be embedded whole - JSON encoding would introduce a \n/\t escape that Dataform’s placeholder-string lexer rejects - so each ${ in such a string is escaped in place, where the surrounding SQL string state carries the raw characters unchanged. GoogleSQL comments that are unsafe in SQLX are normalized as token edits.

Parameters:

tokens (Sequence[Token])

Return type:

List[Tuple[int, int, str]]

sql2sqlx.emitter.render_config(config)

Render a Dataform config { ... } block.

Keys are emitted in canonical order (_KEY_ORDER, then any remaining keys in insertion order); the nested bigquery object is ordered by _BQ_ORDER. Ordering is purely cosmetic but makes output deterministic and diff-friendly.

Parameters:

config (Dict[str, Any]) – The config mapping.

Returns:

The complete config { ... } block, without a trailing newline.

Return type:

str

sql2sqlx.emitter.apply_edits_escaped(text, start, end, edits)

Slice text[start:end] and apply non-overlapping span edits.

Edits use absolute offsets into text. Edits outside the slice or overlapping an already-applied edit are skipped defensively. Literal SQLX interpolation is handled by sqlx_escape_edits(); replacements such as ref() and self() are inserted verbatim and remain active.

Parameters:

• text (str) – Full original source text.

• start (int) – Slice start offset.

• end (int) – Slice end offset.

• edits (Iterable[Tuple[int, int, str]]) – (start, end, replacement) triples, any order.

Returns:

The edited, escaped slice.

Return type:

str

sql2sqlx.emitter.ref_expr(name)

Build the ${ref(...)} expression for a produced table.

The reference carries exactly the qualification the producer was declared with: ${ref("name")}, ${ref("schema", "name")}, or the object form ${ref({database: ..., schema: ..., name: ...})} when a project was explicit.

Parameters:

name (TableName) – The producer’s original (pre-resolution) table name.

Returns:

The interpolation expression text.

Return type:

str

sql2sqlx.emitter.build_sqlx(config, body=None, annotation=None, leading_comments=None, trailing_comments=None)

Assemble a complete .sqlx file.

Layout: config block, blank line, then (in order) the provenance annotation comment, the statement’s original leading comments, and the SQL body. One trailing semicolon is stripped from the body. When the body is None (declarations) only the config block is emitted.

Parameters:

• config (Dict[str, Any]) – The config mapping (see render_config()).

• body (str | None) – Edited SQL body, or None.

• annotation (str | None) – Provenance comment line, or None.

• leading_comments (str | None) – Comments that preceded the statement, or None.

• trailing_comments (str | None) – Final file comments after the statement’s terminating semicolon, or None.

Returns:

The full file contents, newline-terminated.

Return type:

str

Command-line interface: sql2sqlx INPUT [-o OUTPUT] [options].

The CLI is a thin veneer over sql2sqlx.convert_file() and sql2sqlx.convert_directory():

• a file input with no --output prints the generated .sqlx to stdout (handy for piping and quick inspection);

• a directory input requires --output (unless --dry-run) and writes the converted tree beneath it;

• --report FILE writes the full machine-readable JSON report;

• --init-project additionally scaffolds a workflow_settings.yaml next to the output definitions/ directory.

Exit codes: 0 success, 1 when any input file failed to convert, 2 for usage errors.

sql2sqlx.cli.build_arg_parser()

Build the CLI argument parser.

Returns:

A fully configured argparse.ArgumentParser.

Return type:

ArgumentParser

sql2sqlx.cli.main(argv=None)

CLI entry point.

Parameters:

argv (List[str] | None) – Argument list (defaults to sys.argv[1:]).

Returns:

Process exit code (0 ok, 1 conversion failures, 2 usage error).

Return type:

int