102.2. Rubin Schedule Viewer: advanced queries#

102_2_Schedule_viewer_advanced_queries

102.2. Rubin Schedule Viewer: advanced queries¶

For the Rubin Science Platform at data.lsst.cloud.
Data Release: Prompt Products
Container Size: Large
LSST Science Pipelines version: r30.0.11
Last verified to run: 2026-10-02
Repository: github.com/lsst/tutorial-notebooks
DOI: 10.11578/rubin/dc.20250909.20

Learning objective: How to use the Rubin Schedule Viewer's advanced query options (predicate filtering, column selection, and alternate response formats), and how to analyze a retrieved schedule.

Rubin service: The Rubin Schedule Viewer service runs at the USDF and is publicly available at Rubin Schedule Viewer.

Packages: requests, pandas, numpy, astropy, holoviews, bokeh

Credit: Originally developed by the Rubin Community Science team. Please consider acknowledging them if this notebook is used for the preparation of journal articles, software releases, or other notebooks.

Get Support: Everyone is encouraged to ask questions or raise issues in the Support Category of the Rubin Community Forum. Rubin staff will respond to all questions posted there.

1. Introduction¶

Tutorial 102.1 covers the basics of querying the Rubin Schedule Viewer's /schedule endpoint. This tutorial covers its more advanced options -- predicate, columns, and RESPONSEFORMAT -- then uses them to retrieve and analyze a past night's schedule.

The Rubin scheduler team also publishes nightly reports at the schedview reports table of contents: prenight for the nominal simulation of an upcoming night, multiprenight to compare it against variant simulations (e.g. bad seeing, a late start), nightsum for a night that has already happened, which links a Parquet download of its observed visits at ls.st/nightsum, and compareprenight to compare that night's actual visits against its prenight simulation.

1.1. Import packages¶

Import the requests package to query the service, pandas and numpy to work with the results, astropy to read VOTable responses and handle times, and bokeh + holoviews for interactive visualization.

In [1]:

import io

import requests
import pandas as pd
import numpy as np
from astropy.table import Table
from astropy.time import Time
from zoneinfo import ZoneInfo
from datetime import datetime, timezone

import holoviews as hv
from holoviews import opts
from bokeh.models import HoverTool
from bokeh.palettes import Turbo

1.2. Define parameters and functions¶

Define the Rubin Schedule Viewer's base and schedule URLs.

In [2]:

obsloctap_url = "https://usdf-rsp.slac.stanford.edu/obsloctap"
schedule_url = obsloctap_url + "/schedule"

Enable the bokeh backend for holoviews.

In [3]:

hv.extension("bokeh")

ⓘ

2. Column selection¶

By default, /schedule returns all 28 columns of the ivoa.obsplan table (Tutorial 102.1, Section 1.2). Passing a comma-separated columns parameter returns only those columns, as a list of dictionaries rather than the full record.

Compute the UTC equivalent of local midday at Cerro Pachon for the night of June 08 2026 (as in Tutorial 102.1, Section 4). The examples in this section and the next reuse this night, so the row counts below stay reproducible.

In [4]:

local_noon = datetime(2026, 6, 8, 12, 0, 0, tzinfo=ZoneInfo("America/Santiago"))
night_utc = local_noon.astimezone(timezone.utc)

Retrieve only the position and target name columns for that night, and compare the size of the response to a full-column query.

In [5]:

night_params = {"time": "24", "start": night_utc.strftime("%Y-%m-%d %H:%M:%S")}
full_response = requests.get(schedule_url, params=night_params)
subset_response = requests.get(
    schedule_url, params={**night_params, "columns": "s_ra,s_dec,target_name"}
)

print(f"Full response: {len(full_response.content) / 1000: .0f} kB")
print(f"Subset response: {len(subset_response.content) / 1000: .0f} kB")

subset_schedule = pd.DataFrame(subset_response.json())
subset_schedule.head(5)

Full response:  721 kB
Subset response:  89 kB

Out[5]:

	s_ra	s_dec	target_name
0	46.691518	-25.856146	lowdust
1	48.637723	-23.598704	lowdust
2	47.456368	-20.608415	lowdust
3	50.507052	-21.302340	lowdust
4	51.837071	-24.251713	lowdust

3. Advanced predicate filtering¶

A predicate parameter adds a server-side filter, so only matching rows are returned (rather than retrieving everything and filtering in pandas). It supports =, !=, <, >, <=, >=, LIKE, AND, and OR, combined over any of the ivoa.obsplan columns. It also supports band as a shorthand column name, which expands to the equivalent em_min/em_max condition for that filter.

Retrieve only the visits planned for the r band on the night of June 08 2026.

In [6]:

band_params = {
    **night_params,
    "columns": "s_ra,s_dec,target_name,execution_status",
    "predicate": "band = 'r'",
}
band_response = requests.get(schedule_url, params=band_params)
band_schedule = pd.DataFrame(band_response.json())
print(f"{len(band_schedule)} r-band visits were planned for the night of {night_utc.date()}.")

819 r-band visits were planned for the night of 2026-06-08.

Combine two conditions with AND, to select only the r-band visits above declination -20 deg.

In [7]:

compound_params = {
    **night_params,
    "columns": "s_ra,s_dec,target_name",
    "predicate": "band = 'r' AND s_dec > -20",
}
compound_response = requests.get(schedule_url, params=compound_params)
compound_schedule = pd.DataFrame(compound_response.json())
print(
    f"{len(compound_schedule)} of the {len(band_schedule)} r-band visits "
    f"are at declination > -20 deg."
)

441 of the 819 r-band visits are at declination > -20 deg.

4. Alternate response formats¶

The RESPONSEFORMAT parameter selects the format of the response body: json (the default), csv, votable, or parquet. Retrieve the same query in each of the three alternate formats.

Retrieve the CSV response and read it with pandas.

In [8]:

format_params = {
    **night_params,
    "columns": "s_ra,s_dec,target_name,execution_status",
    "predicate": "band = 'r'",
}

csv_response = requests.get(
    schedule_url, params={**format_params, "RESPONSEFORMAT": "csv"}
)
csv_schedule = pd.read_csv(io.StringIO(csv_response.text))
print(f"CSV: {len(csv_schedule)} rows, content-type {csv_response.headers['content-type']}")

CSV: 819 rows, content-type text/csv; charset=utf-8

Retrieve the VOTable response and read it with astropy.table.Table.

In [9]:

votable_response = requests.get(
    schedule_url, params={**format_params, "RESPONSEFORMAT": "votable"}
)
votable_schedule = Table.read(io.BytesIO(votable_response.content), format="votable")
print(
    f"VOTable: {len(votable_schedule)} rows, "
    f"content-type {votable_response.headers['content-type']}"
)

VOTable: 819 rows, content-type application/x-votable+xml

Retrieve the Parquet response and read it directly into a pandas dataframe.

In [10]:

parquet_response = requests.get(
    schedule_url, params={**format_params, "RESPONSEFORMAT": "parquet"}
)
parquet_schedule = pd.read_parquet(io.BytesIO(parquet_response.content))
print(
    f"Parquet: {len(parquet_schedule)} rows, "
    f"content-type {parquet_response.headers['content-type']}"
)

Parquet: 819 rows, content-type application/x-parquet

5. Analyze a predicted schedule¶

Use the 24 hours of schedule prediction for June 08 2026, retrieved as in Tutorial 102.1, Section 4.

In [11]:

past_time = Time(night_utc).mjd
params = {"time": "24", "start": past_time}
response = requests.get(schedule_url, params=params)
next_visits_from_date_iso = pd.DataFrame(response.json())
print(
    f"{len(next_visits_from_date_iso)} visits were scheduled for the night of "
    f"{night_utc.date()}."
)

1041 visits were scheduled for the night of 2026-06-08.

Retrieve the Performed subset of the same night directly with the predicate option introduced in Section 3, rather than filtering in pandas.

In [12]:

performed_params = {**params, "predicate": "execution_status = 'Performed'"}
performed_response = requests.get(schedule_url, params=performed_params)
next_visits_performed = pd.DataFrame(performed_response.json())
print(f"{len(next_visits_performed)} visits were Performed that night.")

678 visits were Performed that night.

5.1. Inter-visit time gaps¶

Extract the planned observation times and calculate the differences between successive values, in seconds.

In [13]:

t_planning = next_visits_from_date_iso["t_planning"].values
obs_visit_gap = []
for i in range(len(t_planning) - 1):
    diff = abs(t_planning[i] - t_planning[i + 1]) * 60 * 60 * 24
    if np.isnan(diff):
        continue
    obs_visit_gap.append(diff)
    del diff
print(f"Number of differences, all planned: {len(obs_visit_gap)}")

t_planning_performed = next_visits_performed["t_planning"].values

obs_visit_gap_performed = []
for i in range(len(t_planning_performed) - 1):
    diff = abs(t_planning_performed[i] - t_planning_performed[i + 1]) * 60 * 60 * 24
    if np.isnan(diff):
        continue
    obs_visit_gap_performed.append(diff)
    del diff
print(f"Number of differences, only performed observations: {len(obs_visit_gap_performed)}")

Number of differences, all planned: 1040
Number of differences, only performed observations: 677

Print the minimum, maximum, and median visit gap time.

In [14]:

print("           All     Performed")
print(f"Min gap: {np.min(obs_visit_gap): .2f}   {np.min(obs_visit_gap_performed): .2f}")
print(f"Max gap: {np.max(obs_visit_gap): .2f} {np.max(obs_visit_gap_performed): .2f}")
print(f"Mean gap: {np.mean(obs_visit_gap): .2f}   {np.mean(obs_visit_gap_performed): .2f}")

           All     Performed
Min gap:  35.01    36.34
Max gap:  211.94  1921.81
Mean gap:  41.11    63.15

Create the hover tool.

In [15]:

hover = HoverTool(
    tooltips=[("Gap Range", "@left{0.00} - @right{0.00}"), ("Count", "@top")]
)

Create the histogram.

In [16]:

histogram1 = hv.Histogram(
    np.histogram(obs_visit_gap, bins=50), kdims="Time Gap", vdims="Counts"
).opts(
    title="Successive planned visit gaps for June 08 2026",
    xlabel="Gap (seconds)",
    ylabel="Count",
    width=600,
    height=400,
    logy=True,
    line_color="steelblue",
    fill_color="lightblue",
    line_width=1,
    show_grid=True,
    tools=["hover", "pan", "wheel_zoom", "box_zoom", "reset"],
)
histogram2 = hv.Histogram(
    np.histogram(obs_visit_gap_performed, bins=50), kdims="Time Gap", vdims="Counts"
).opts(
    title="Successive PERFORMED planned visit gaps for June 08 2026",
    xlabel="Gap (seconds)",
    ylabel="Count",
    width=600,
    height=400,
    logy=True,
    line_color="steelblue",
    fill_color="lightblue",
    line_width=1,
    show_grid=True,
    tools=["hover", "pan", "wheel_zoom", "box_zoom", "reset"],
)

histogram = histogram1 + histogram2
histogram

Out[16]:

Figure 1: The distribution of the number of planned observations with a given time gap between it and the next observation, for all planned observations (left) and only those that were Performed instead of Aborted (right).

5.2. Planned time by target field¶

Plot curves of the scheduled observation times for each target field that was planned to be observed the night of June 08 2026.

Get unique target names.

In [17]:

unique_targets = next_visits_from_date_iso["target_name"].unique()
target_to_y = {target: i for i, target in enumerate(unique_targets)}

Make copies of the retrieved tables of planned visits, for all and for the Performed visits only, and add the y-axis name to them.

In [18]:

df_plot = next_visits_from_date_iso.copy()
df_plot["y"] = df_plot["target_name"].map(target_to_y)

df_plot_performed = next_visits_performed.copy()
df_plot_performed["y"] = df_plot_performed["target_name"].map(target_to_y)

Create a color map from a palette with one color per target field.

In [19]:

np.random.seed(42)
palette = list(np.random.choice(Turbo[256], len(unique_targets), replace=False))
colors = {target: palette[i % len(palette)] for i, target in enumerate(unique_targets)}

Create the scatter plots for each target name, and overlay them, for both all visits and the Performed visits.

In [20]:

scatter_plots1 = []
for target in unique_targets:
    target_data = df_plot[df_plot["target_name"] == target]
    scatter = hv.Scatter(
        target_data, kdims=["t_planning"], vdims=["y", "target_name"], label=target
    )
    scatter.opts(opts.Scatter(color=colors[target]))
    scatter_plots1.append(scatter)
    del scatter, target_data

plot1 = hv.Overlay(scatter_plots1)

scatter_plots2 = []
for target in unique_targets:
    target_data = df_plot_performed[df_plot_performed["target_name"] == target]
    if len(target_data) > 0:
        scatter = hv.Scatter(
            target_data, kdims=["t_planning"], vdims=["y", "target_name"], label=target
        )
        scatter.opts(opts.Scatter(color=colors[target]))
        scatter_plots2.append(scatter)
        del scatter, target_data

plot2 = hv.Overlay(scatter_plots2)

Configure the hover tool.

In [21]:

hover = HoverTool(
    tooltips=[
        ("target_name", "@{target_name}"),
        ("t_planning", "@{t_planning}{0.000000}"),
    ]
)

Create the y-axis tick labels, one for each target name.

In [22]:

y_ticks = [(i, target) for target, i in target_to_y.items()]

Set all the other plot options and show the plot.

In [23]:

plot1 = plot1.opts(
    opts.Scatter(
        width=600,
        height=400,
        tools=[hover, "pan", "wheel_zoom", "box_zoom", "reset", "save"],
        xlabel="Scheduled Observation Time (MJD)",
        ylabel="Target Name",
        title="Planned observations by target for June 08 2026",
        size=8,
        alpha=0.7,
        legend_position="right",
        legend_offset=(10, 0),
        yticks=y_ticks,
        show_legend=False,
    )
)
plot2 = plot2.opts(
    opts.Scatter(
        width=600,
        height=400,
        tools=[hover, "pan", "wheel_zoom", "box_zoom", "reset", "save"],
        xlabel="Scheduled Observation Time (MJD)",
        ylabel="Target Name",
        title="Planned PERFORMED observations by target for June 08 2026",
        size=8,
        alpha=0.7,
        legend_position="right",
        legend_offset=(10, 0),
        yticks=y_ticks,
        show_legend=False,
    )
)

plot = plot1 + plot2
plot

Out[23]:

Figure 2: The time of the planned observation (x-axis) for each of the planned target fields (y-axis), for the night of June 08 2026. For all planned visits (left) and for only the Performed visits (right).