BigQuery for .NET Developers: What You Need to Know Before It Gets Expensive

You write a query. It runs in four seconds and returns exactly what you need. You ship it. Three weeks later someone forwards you a billing alert. The query runs every five minutes in production. It scans 800 GB each time. At $6.25 per terabyte, that dashboard widget costs $11,700 a month.

This is not a hypothetical. It is what happens when a team adopts BigQuery without first understanding how it charges. BigQuery does not charge per query, per row returned, or per second of runtime. It charges for data read from storage. Every decision you make, from table structure to query shape to how you write data, either increases or reduces the amount of data read. Understanding that one fact is worth more than any optimization tip.

The Cost Model

On-demand pricing is $6.25 per terabyte of data processed. The first terabyte each month is free. The charge is calculated on bytes scanned from storage. Not rows returned. Not CPU time. Not query complexity.

BigQuery stores data in a columnar format. Each column in a table is stored in separate blocks on disk. When you run a query, BigQuery reads only the columns you reference. Columns not in your SELECT, WHERE, GROUP BY, or ORDER BY are never touched.

A 1 TB table with 100 columns stores roughly 10 GB per column on average. A query that selects 2 columns scans approximately 20 GB. The same query written as SELECT * scans the full terabyte and costs 50 times more.

One more charge to know: a minimum of 10 MB per table referenced in a query, regardless of how few rows you return. A query that reads a single row from a table still incurs the 10 MB minimum. This matters for high-frequency small lookup queries, not large analytical ones.

Dry Run: Estimate Before You Execute

BigQuery supports a dry run mode: a flag on the job configuration that validates the query and returns the estimated bytes processed without executing anything. It costs nothing and completes in under a second. Use it during development to understand what a query costs before it ships to production.

using Google.Apis.Bigquery.v2.Data;

// Dry run via the underlying service (BigQueryClient.Service)
var queryRequest = new QueryRequest
{
    Query       = sql,
    DryRun      = true,
    UseLegacySql = false,
};

var response = client.Service.Jobs
    .Query(queryRequest, projectId)
    .Execute();

long   bytes            = response.TotalBytesProcessed ?? 0;
double estimatedCostUsd = (bytes / 1_099_511_627_776.0) * 6.25;

Console.WriteLine($"Estimated scan: {bytes / 1_073_741_824.0:F2} GB");
Console.WriteLine($"Estimated cost: ${estimatedCostUsd:F4}");

In multi-tenant applications or internal tools where users write their own queries, dry run becomes a cost gate. Run the estimate first. If the result exceeds your threshold, reject the query before it executes. This pattern prevents a single ad hoc query from generating an unexpected charge.

On-Demand vs Slots

With on-demand pricing you pay per byte scanned and share a pool of up to 2,000 concurrent slots across your project. With reserved capacity (Standard, Enterprise, Enterprise Plus editions) you pay per slot-hour and get dedicated compute not shared with other tenants. On-demand is the right starting point for most teams. Reserved capacity becomes worth evaluating when your query workload is large, consistent, and predictable enough that slot-hour cost is lower than on-demand. Slots are covered in more detail in a later section.

Your Two Levers: Partitioning and Clustering

You pay for bytes scanned. Partitioning and clustering are the two mechanisms that reduce bytes scanned at the table design level. Neither requires changes to your application queries beyond filtering on the right columns. The investment is in table design, and the payoff is on every query that runs against that table.

Partitioning

A partitioned table divides its data into separate storage segments based on a column value or ingestion time. When a query includes a filter on the partition column, BigQuery identifies which partitions match before execution begins and reads only those. The excluded partitions are never touched and never charged.

Three partitioning types are available. Time-unit column partitioning is based on a DATE, TIMESTAMP, or DATETIME column in your table, with daily granularity being the most common choice for event and transaction data. Ingestion-time partitioning automatically assigns rows based on when they arrive and adds pseudo-columns _PARTITIONTIME and _PARTITIONDATE to the table. Integer range partitioning is based on an INTEGER column with a defined start, end, and interval, useful when time-based partitioning does not apply to your data.

To get the cost benefit, your queries must filter directly on the partition column. A query against a table partitioned by order_date that includes WHERE order_date = ‘2026-01-15’ reads only that partition. The same query without the filter reads the entire table. The filter is what triggers partition pruning.

You can enforce partition filtering at the table level with require_partition_filter. When this option is enabled, BigQuery rejects any query that omits a filter on the partition column instead of silently scanning the entire table. It turns a missing WHERE clause into an immediate, visible error rather than an unexpected line item on your bill.

-- On an existing table
ALTER TABLE `my-project.sales.orders`
SET OPTIONS (require_partition_filter = true);

-- At creation time
CREATE TABLE `my-project.sales.orders`
(
    order_id     INT64,
    order_date   DATE,
    customer_id  INT64,
    total_amount NUMERIC
)
PARTITION BY order_date
OPTIONS (require_partition_filter = true);

Clustering

Clustering organizes data within a partition by sorting storage blocks according to the values of up to four columns. When a query filters on a leading clustering column, BigQuery reads block metadata and skips entire blocks whose value ranges cannot contain matching rows. Skipped blocks are never read from storage and never charged.

Column order is critical. It works exactly like a composite index: filtering on the first column gives full benefit, filtering on the second column without the first gives no benefit, filtering on both gives maximum benefit. Choose the columns your queries filter most frequently and put the highest-cardinality column first.

BigQuery re-clusters the table automatically in the background as new data arrives. This is free and requires no manual maintenance on your part.

Both Together

Partition pruning fires first, eliminating entire partitions. Clustering pruning fires next within the surviving partitions. Combined, they can reduce bytes scanned by orders of magnitude. An events table partitioned by event_date and clustered by user_id, event_type: a query filtering on both WHERE event_date = ‘2026-01-15’ AND user_id = 12345 eliminates every partition except January 15, then within that partition eliminates every block that cannot contain rows for that user. The query may scan less than 1% of the total table data.

Calling BigQuery from .NET

Setup and Authentication

The .NET client library is Google.Cloud.BigQuery.V2. The primary class is BigQueryClient.

dotnet add package Google.Cloud.BigQuery.V2

Authentication uses Application Default Credentials. The client library resolves credentials in this order: the GOOGLE_APPLICATION_CREDENTIALS environment variable pointing to a service account key file, gcloud CLI credentials on the local machine, and the attached service account when running on GCP. In most cases you do not pass credentials explicitly.

// ADC is resolved automatically
var client = BigQueryClient.Create("your-project-id");

// Or async
var client = await BigQueryClient.CreateAsync("your-project-id");

Executing Queries

client.ExecuteQuery() submits the query, waits for completion, and returns results as a BigQueryResults object you iterate directly. Use this for queries expected to complete in seconds.

var results = client.ExecuteQuery(
    @"SELECT order_id, customer_id, total_amount
      FROM `my-project.sales.orders`
      WHERE order_date = '2026-01-15'
      ORDER BY total_amount DESC
      LIMIT 100",
    parameters: null
);

foreach (var row in results)
{
    Console.WriteLine($"Order {row["order_id"]}: ${row["total_amount"]}");
}

For queries that may run for minutes, use the async job pattern. CreateQueryJob() returns immediately with a job reference. Call PollUntilCompleted() to wait for the result without blocking a thread for the full duration.

var job = client.CreateQueryJob(sql, parameters: null);
var completed = job.PollUntilCompleted();

if (completed.Status.ErrorResult != null)
    throw new Exception($"Query failed: {completed.Status.ErrorResult.Message}");

var results = completed.GetQueryResults();

Parameterized Queries

Never concatenate user input into a query string. Parameterized queries serve two purposes: they prevent SQL injection, and they keep the query text identical across executions, which matters for caching.

var sql = @"
    SELECT order_id, total_amount, status
    FROM `my-project.sales.orders`
    WHERE customer_id  = @customer_id
      AND order_date  >= @start_date
      AND order_date   < @end_date
    ORDER BY order_date DESC";

var parameters = new[]
{
    new BigQueryParameter("customer_id", BigQueryDbType.Int64,  customerId),
    new BigQueryParameter("start_date",  BigQueryDbType.Date,   startDate),
    new BigQueryParameter("end_date",    BigQueryDbType.Date,   endDate),
};

var results = client.ExecuteQuery(sql, parameters);

Query Result Caching

BigQuery automatically caches query results for approximately 24 hours. When an identical query runs again, results are returned from cache with zero bytes processed and zero charge. For repeated dashboard queries or scheduled reporting, caching can eliminate the cost of those queries entirely.

The cache is keyed on exact query text plus parameter values. A whitespace change or an added comment in the query string breaks the match and re-executes the full query. Consistent query formatting matters here.

Several conditions disable caching. Know this list because violating any one of them silently bypasses the cache:

• The query uses non-deterministic functions: CURRENT_TIMESTAMP(), CURRENT_DATE(), NOW(), RAND(), GENERATE_UUID(). Any of these in the query means no cache hit.
• The referenced table has been modified since the cache entry was created.
• The table has an active streaming buffer. Even if no new rows arrived since the last query, the presence of the buffer disables caching for that table.
• A destination table is specified in the job configuration.
• The query uses wildcard table syntax such as my_table_*.
• The query targets tables protected by row-level or column-level security.

Writing Data: Three Options, Different Trade-offs

How you write data into BigQuery affects cost, latency, delivery guarantees, and whether caching works on the read side. There are three options.

Legacy Streaming Inserts

The tabledata.insertAll REST endpoint accepts rows in real time and makes them immediately queryable. It costs $0.05 per GB with a minimum charge of 1 KB per row. A 50-byte row costs the same as a 1,024-byte row. At high volume with small rows, the effective cost per actual byte is significantly higher than the headline rate suggests.

Beyond the per-GB charge: the streaming buffer disables query result caching on the entire table, and deduplication via insertId is only best-effort within roughly one minute. Google’s current documentation explicitly recommends the Storage Write API for all new implementations.

Storage Write API

Google’s current recommendation for streaming ingestion. Uses gRPC instead of REST, costs $0.025 per GB (half of legacy), and the first 2 TB per month are free. Three stream types give you control over the delivery and visibility trade-off: the default stream is at-least-once with immediate visibility, the committed stream is exactly-once via stream offsets, and the pending stream buffers data for atomic commit, similar to a batch load job with programmatic commit control.

Batch Load Jobs

Load jobs read files from Cloud Storage (Avro, Parquet, CSV, newline-delimited JSON) and import them into BigQuery. They are free. You pay only for storage. The trade-off is latency: load jobs complete in 30 seconds to a few minutes and data is not queryable until the job completes. Load jobs are fully transactional, do not touch the streaming buffer, and have no impact on query result caching.

Slots: The Compute Layer

A slot is BigQuery’s unit of compute capacity. It maps to a worker in the query execution engine: each slot reads data from storage, applies filters, and performs partial aggregations. A single complex query can consume thousands of slots simultaneously as BigQuery parallelizes the work across the table.

On-Demand Slot Pool

With on-demand pricing, your project shares a pool of up to 2,000 concurrent slots. When multiple queries run at the same time they compete for the available slots. When the pool is exhausted, BigQuery does not fail the query. It queues the work and waits for slots to free up. The query takes longer but completes. You will not see an error from slot contention, just increased latency. This makes it easy to miss until you investigate why a query that typically takes 10 seconds is suddenly taking 90.

Monitoring Slot Usage

SELECT
    SUM(total_slot_ms) / (1000 * 60 * 60 * 24) AS avg_slots_used,
    MAX(total_slot_ms) / 1000                   AS max_slot_seconds,
    COUNT(*)                                    AS query_count
FROM `region-us`.INFORMATION_SCHEMA.JOBS_BY_PROJECT
WHERE DATE(creation_time) = CURRENT_DATE()
  AND job_type            = 'QUERY'
  AND state               = 'DONE'

Reserved Capacity

Reserved slots give you dedicated compute not shared with other tenants. You pay per slot-hour regardless of utilization. The break-even point depends on your query volume: if you consistently use a large fraction of on-demand slots throughout the day, reserved capacity costs less than on-demand pricing. The BigQuery console includes a Slot Estimator showing 30-day historical usage to help with this calculation. Start with on-demand. Move to reserved when the usage pattern is consistent and large enough to justify the commitment.

Production Patterns

Async Jobs for Long-Running Queries

client.ExecuteQuery() is synchronous and appropriate for queries completing in seconds. For analytical queries that may run for minutes, use the async pattern: submit the job, store the job reference, poll for completion separately. BigQuery jobs can run for up to six hours. Do not block a request thread waiting for a long-running query to complete.

Handling Slot Contention and Query Timeouts

Slot contention makes query duration unpredictable. A query that normally completes in 30 seconds can take 10 or 20 minutes when the slot pool is saturated by concurrent workloads. If your application calls PollUntilCompleted without a timeout, it will block indefinitely waiting for slots to free up. In production, every long-running query needs both a client-side timeout and a server-side limit.

The client-side timeout uses a CancellationToken. When the token is cancelled, an OperationCanceledException is thrown. There is one critical detail: the exception stops your polling loop, but the job keeps running on BigQuery’s servers. You must cancel it explicitly or it will continue consuming slots and potentially incur costs.

using var cts = new CancellationTokenSource(TimeSpan.FromMinutes(2));

// Job creation completes in milliseconds — outside the try so it's in scope for catch
var job = await client.CreateQueryJobAsync(sql, parameters);

try
{
    var completed = await job.PollUntilCompletedAsync(cancellationToken: cts.Token);
    return completed.GetQueryResults();
}
catch (OperationCanceledException)
{
    await job.CancelAsync();
    throw new Exception("Query exceeded the 2-minute limit and was cancelled.");
}

For a server-side limit, set JobTimeoutMs on the job configuration. BigQuery will cancel the job automatically when this duration is exceeded, regardless of whether your client is still connected. This is the safety net: even if your application process restarts or loses the job reference, the job will not run indefinitely.

var options = new QueryOptions
{
    // BigQuery cancels the job server-side after this duration
    ConfigurationModifier = config =>
    {
        config.JobTimeoutMs = 600_000L; // 10 minutes in milliseconds
    }
};

var job = client.CreateQueryJob(sql, parameters, options);

Batch Priority for Non-Urgent Queries

Interactive queries (the default) are scheduled immediately and compete for on-demand slots with all other concurrent queries. Batch queries are queued separately and only execute when idle slots are available. They do not compete with interactive queries for the same slot pool.

Use batch priority for queries that do not need to complete immediately: scheduled reports, nightly aggregations, data exports, and any analytical workload that can tolerate variable latency. This keeps your interactive slot pool available for user-facing queries.

var options = new QueryOptions
{
    Priority = QueryPriority.Batch
};

var job = client.CreateQueryJob(sql, parameters, options);
// Batch jobs may start immediately or wait hours depending on slot availability
// They are not subject to the 2,000 concurrent slot limit

Error Handling and Retries

BigQuery returns rate limit errors as HTTP 403, not 429. Applications that only check for HTTP 429 when deciding whether to retry will miss these errors entirely. Inspect the error reason field in the response body.

try
{
    var results = client.ExecuteQuery(sql, parameters);
    // process results
}
catch (GoogleApiException ex)
{
    var reason = ex.Error?.Errors?.FirstOrDefault()?.Reason;

    switch (reason)
    {
        case "rateLimitExceeded":
            // Transient spike: retry with exponential backoff
            break;

        case "quotaExceeded":
            // Project quota exhausted: wait 10+ minutes, do not retry immediately
            break;

        default when (int)ex.HttpStatusCode == 500:
            // Transient backend error: retry with backoff
            break;

        default:
            // Bad request, permission denied: do not retry
            throw;
    }
}

• rateLimitExceeded (403): transient spike. Retry with exponential backoff starting at a few seconds.
• quotaExceeded (403): daily or project quota exhausted. Wait 10 or more minutes before retrying.
• backendError (500): transient infrastructure error. Retry with backoff.
• notFound, accessDenied, invalid (4xx): logic or permission errors. Do not retry.

API Rate Limits and Wrapper Projects

The BigQuery Connection API enforces a limit of 1,000 read requests per minute per project. This quota covers API calls that read connection data, and it is tracked per GCP project. For most applications this is fine. But at scale, a high-traffic service making one Connection API call per request can exceed this limit quickly. When it does, every call starts returning rateLimitExceeded (403) until the minute resets. Retrying within the same project just delays the next failure.

The solution is wrapper projects. Create additional GCP projects specifically to distribute Connection API traffic. Each project has its own independent 1,000/min quota. Five wrapper projects give you 5,000 requests per minute. All wrapper projects query tables in the main project using fully qualified names, and slot reservations on the main project cover compute for all of them via BigQuery Reservations assignments.

In .NET, register a singleton client pool that pairs each wrapper project client with a SemaphoreSlim(100). The semaphore limits concurrent calls per project so no single project gets flooded. The pool round-robins across all five entries and manages semaphore acquisition internally:

public sealed class BigQueryClientPool
{
    private readonly (BigQueryClient Client, SemaphoreSlim Semaphore)[] _pool;
    private int _index = -1;

    public BigQueryClientPool(IConfiguration config)
    {
        var projects = config
            .GetSection("BigQuery:WrapperProjects")
            .Get<string[]>()!;

        _pool = projects
            .Select(p => (BigQueryClient.Create(p), new SemaphoreSlim(100)))
            .ToArray();
    }

    public async Task<T> ExecuteAsync<T>(
        Func<BigQueryClient, Task<T>> operation,
        CancellationToken cancellationToken = default)
    {
        int i = Interlocked.Increment(ref _index) % _pool.Length;
        var (client, semaphore) = _pool[i];

        await semaphore.WaitAsync(cancellationToken);
        try
        {
            return await operation(client);
        }
        finally
        {
            semaphore.Release();
        }
    }
}

// appsettings.json
{
  "BigQuery": {
    "WrapperProjects": [
      "wrapper-project-1",
      "wrapper-project-2",
      "wrapper-project-3",
      "wrapper-project-4",
      "wrapper-project-5"
    ]
  }
}

Register as a singleton and inject wherever you need a client. The semaphore is acquired and released inside ExecuteAsync, so call sites stay clean:

var job = await _pool.ExecuteAsync(client =>
    client.CreateQueryJobAsync(
        "SELECT ... FROM `main-project.dataset.orders` WHERE ...",
        parameters));

Logging Query Statistics

Every completed BigQueryJob exposes execution statistics through its Resource.Statistics property. Log these values per query in production: elapsed time, slot-milliseconds consumed, bytes scanned, bytes billed, and whether the result came from cache. Over time this gives you a queryable record to detect slow queries, unexpected scan growth, and cache effectiveness before they show up on a bill.

var stats = completed.Resource.Statistics;
var query = stats.Query;

long elapsedMs    = (stats.EndTime    ?? 0) - (stats.StartTime ?? 0);
long slotMs       = stats.TotalSlotMs         ?? 0;
long bytesScanned = query.TotalBytesProcessed ?? 0;
long bytesBilled  = query.TotalBytesBilled    ?? 0;
bool fromCache    = query.CacheHit            ?? false;

_logger.LogInformation(
    "BigQuery {JobId}: {ElapsedMs}ms, {SlotMs} slot-ms, {BytesScanned} scanned, {BytesBilled} billed, cache={CacheHit}",
    completed.Reference.JobId,
    elapsedMs,
    slotMs,
    bytesScanned,
    bytesBilled,
    fromCache);

The 10 GB Response Limit

If a query returns more than 10 GB of data inline, BigQuery aborts with a response too large error. The fix is to specify a destination table in the job configuration. BigQuery writes results to the destination table instead of returning them inline. You then query or export from the destination table. Destination table results have no size limit.

Conclusion

BigQuery rewards the developer who designs tables correctly and writes queries that scan only what they need. It punishes SELECT *, unpartitioned tables, and queries with no filter on the partition column. These are not advanced optimizations. They are the baseline habits that separate a $50 monthly bill from a $5,000 one.

Partition your tables by time. Cluster by the columns you filter most. Never SELECT *. Use parameterized queries. Understand your write path before you commit to it. Run a dry run before shipping an expensive query to production. These decisions compound: a well-designed table that reduces scan cost by 90% does so on every query, every day, for the lifetime of that table.