Download notebook Run in Google Colab

Batch processing¶

This tutorial covers the .many() method for efficient bulk feature extraction:

• Plain Python lists of (t, m, sigma) tuples
• nested-pandas with real ZTF survey data
• PyArrow List<Struct> arrays
• Polars Series

All Arrow-compatible inputs avoid Python-level iteration and pass data to Rust with zero copies.

In [1]:

# %pip install light-curve

# %pip install light-curve

Plain list of tuples¶

.many() accepts a list of (t, m, sigma) tuples and returns a 2-D NumPy array of shape (N, n_features). Multi-threading is enabled by default via the n_jobs parameter:

In [2]:

import light_curve as licu
import numpy as np

rng = np.random.default_rng(0)
light_curves = [
    (np.sort(rng.random(50)), rng.random(50), rng.random(50) * 0.1)
    for _ in range(1000)
]

results = licu.Amplitude().many(light_curves)
print(f'Extracted from {len(light_curves)} light curves: shape = {results.shape}')
print(f'Mean amplitude = {results.mean():.4f} mag')

import light_curve as licu import numpy as np rng = np.random.default_rng(0) light_curves = [ (np.sort(rng.random(50)), rng.random(50), rng.random(50) * 0.1) for _ in range(1000) ] results = licu.Amplitude().many(light_curves) print(f'Extracted from {len(light_curves)} light curves: shape = {results.shape}') print(f'Mean amplitude = {results.mean():.4f} mag')

Extracted from 1000 light curves: shape = (1000, 1)
Mean amplitude = 0.4806 mag

nested-pandas with ZTF survey data¶

nested-pandas extends pandas with nested Arrow column support, useful for catalog data such as ZTF or Rubin LSST.

In [3]:

# %pip install light-curve nested-pandas s3fs universal-pathlib

# %pip install light-curve nested-pandas s3fs universal-pathlib

In [4]:

import light_curve as licu
import nested_pandas as npd
import numpy as np
import pyarrow as pa
from upath import UPath

s3_path = UPath(
    "s3://ipac-irsa-ztf/contributed/dr23/lc/hats/ztf_dr23_lc-hats/dataset/Norder=6/Dir=30000/Npix=34623/part0.snappy.parquet",
    anon=True,
)
ndf = npd.read_parquet(
    s3_path,
    columns=["objectid", "lightcurve.hmjd", "lightcurve.mag", "lightcurve.magerr"],
)

ndf = ndf.loc[ndf["lightcurve"].len() > 10]

ndf["lightcurve.t"] = np.asarray(ndf["lightcurve.hmjd"] - 58000, dtype=np.float32)

feature = licu.Extractor(licu.Chi2Pvar(), licu.InterPercentileRange(quantile=0.25), licu.LinearFit())
result = feature.many(pa.array(ndf["lightcurve"]), n_jobs=-1,
                      arrow_fields={"t": "t", "m": "mag", "sigma": "magerr"})

ndf = ndf.assign(**dict(zip(feature.names, result.T)))
ndf.head()

import light_curve as licu import nested_pandas as npd import numpy as np import pyarrow as pa from upath import UPath s3_path = UPath( "s3://ipac-irsa-ztf/contributed/dr23/lc/hats/ztf_dr23_lc-hats/dataset/Norder=6/Dir=30000/Npix=34623/part0.snappy.parquet", anon=True, ) ndf = npd.read_parquet( s3_path, columns=["objectid", "lightcurve.hmjd", "lightcurve.mag", "lightcurve.magerr"], ) ndf = ndf.loc[ndf["lightcurve"].len() > 10] ndf["lightcurve.t"] = np.asarray(ndf["lightcurve.hmjd"] - 58000, dtype=np.float32) feature = licu.Extractor(licu.Chi2Pvar(), licu.InterPercentileRange(quantile=0.25), licu.LinearFit()) result = feature.many(pa.array(ndf["lightcurve"]), n_jobs=-1, arrow_fields={"t": "t", "m": "mag", "sigma": "magerr"}) ndf = ndf.assign(**dict(zip(feature.names, result.T))) ndf.head()

Out[4]:

	objectid	lightcurve	chi2_pvar	inter_percentile_range_25	linear_fit_slope	linear_fit_slope_sigma	linear_fit_reduced_chi2
0	1248202100002710	hmjd mag magerr t 59176.3144 19.841591 0.140125 1176.314453 +10 rows ... ... ...	0.000000	0.305351	0.001163	0.000185	5.509358
1	1248202100002733	hmjd mag magerr t 59176.31488 19.836166 0.139713 1176.314819 +15 rows ... ... ...	0.204448	0.216278	0.000245	0.000145	1.167804
2	1248202100002739	hmjd mag magerr t 59124.43823 19.081635 0.086184 1124.438232 +22 rows ... ... ...	0.734284	0.092379	0.000048	0.000083	0.818141
5	1248202100002819	hmjd mag magerr t 59124.43823 17.424206 0.029763 1124.438232 +25 rows ... ... ...	0.000120	0.042589	0.000031	0.000028	2.430087
8	1248202100002918	hmjd mag magerr t 59124.43823 18.022591 0.041814 1124.438232 +25 rows ... ... ...	0.008479	0.045643	0.000043	0.000040	1.825333

5 rows x 7 columns

PyArrow¶

PyArrow is the reference Python implementation of Apache Arrow. Pass a List<Struct<t, m, band>> array directly to .many() for multiband extraction without sigma.

In [5]:

# %pip install light-curve pyarrow

# %pip install light-curve pyarrow

In [6]:

import light_curve as licu
import numpy as np
import pyarrow as pa

BANDS = ["g", "r"]
rng = np.random.default_rng(42)
n_lc, n_per_band = 200, 40

struct_type = pa.struct([
    ("t", pa.float64()),
    ("m", pa.float64()),
    ("band", pa.string()),
])


def make_lc():
    rows = []
    for b in BANDS:
        t = rng.uniform(0, 100, n_per_band)
        m = rng.normal(15.0 if b == "g" else 15.3, 0.3, n_per_band)
        rows.extend({"t": float(ti), "m": float(mi), "band": b} for ti, mi in zip(t, m))
    rows.sort(key=lambda r: r["t"])
    return rows


lcs_arrow = pa.array([make_lc() for _ in range(n_lc)], type=pa.list_(struct_type))

feature = licu.Extractor(
    licu.InterPercentileRange(quantile=0.1, bands=BANDS),  # robust amplitude per band
    licu.AndersonDarlingNormal(bands=BANDS),  # normality test per band
    licu.ColorOfMaximum(BANDS),  # colour at brightness peak
    licu.ColorOfMinimum(BANDS),  # colour at brightness trough
)
result = feature.many(
    lcs_arrow,
    sorted=True,
    arrow_fields={"t": "t", "m": "m", "band": "band"},
)
print(f"shape: {result.shape}")  # (200, 6)
print("names:", feature.names)

import light_curve as licu import numpy as np import pyarrow as pa BANDS = ["g", "r"] rng = np.random.default_rng(42) n_lc, n_per_band = 200, 40 struct_type = pa.struct([ ("t", pa.float64()), ("m", pa.float64()), ("band", pa.string()), ]) def make_lc(): rows = [] for b in BANDS: t = rng.uniform(0, 100, n_per_band) m = rng.normal(15.0 if b == "g" else 15.3, 0.3, n_per_band) rows.extend({"t": float(ti), "m": float(mi), "band": b} for ti, mi in zip(t, m)) rows.sort(key=lambda r: r["t"]) return rows lcs_arrow = pa.array([make_lc() for _ in range(n_lc)], type=pa.list_(struct_type)) feature = licu.Extractor( licu.InterPercentileRange(quantile=0.1, bands=BANDS), # robust amplitude per band licu.AndersonDarlingNormal(bands=BANDS), # normality test per band licu.ColorOfMaximum(BANDS), # colour at brightness peak licu.ColorOfMinimum(BANDS), # colour at brightness trough ) result = feature.many( lcs_arrow, sorted=True, arrow_fields={"t": "t", "m": "m", "band": "band"}, ) print(f"shape: {result.shape}") # (200, 6) print("names:", feature.names)

shape: (200, 6)
names: ['inter_percentile_range_10_g', 'inter_percentile_range_10_r', 'anderson_darling_normal_g', 'anderson_darling_normal_r', 'color_max_g_r', 'color_min_g_r']

Polars¶

Polars is a fast DataFrame library built on Arrow. Group a flat multiband DataFrame by object and pass the nested Series to .many().

In [7]:

# %pip install light-curve polars

# %pip install light-curve polars

In [8]:

import light_curve as licu
import numpy as np
import polars as pl

BANDS = ["g", "r"]
rng = np.random.default_rng(42)
n_obj, n_per_band = 200, 40

object_id = np.repeat(np.arange(n_obj), n_per_band * len(BANDS))
band_col = np.tile(np.repeat(BANDS, n_per_band), n_obj)
t = np.sort(rng.uniform(0, 100, n_obj * n_per_band * len(BANDS)))
m = rng.normal(15.0, 0.3, len(object_id))
sigma = rng.uniform(0.01, 0.1, len(object_id))

df = pl.DataFrame({"object_id": object_id, "band": band_col, "t": t, "m": m, "sigma": sigma})
nested = df.group_by("object_id").agg(pl.struct("t", "m", "sigma", "band").alias("lc"))

feature = licu.Extractor(
    licu.ExcessVariance(bands=BANDS),  # variability excess over noise per band
    licu.StetsonK(bands=BANDS),  # variability index per band
    licu.BeyondNStd(nstd=1.5, bands=BANDS),  # outlier fraction per band
    licu.ColorOfMedian(BANDS),  # colour at median brightness
    licu.ColorSpread(BANDS),  # std dev of per-band means
)
result = feature.many(
    nested["lc"],
    arrow_fields={"t": "t", "m": "m", "sigma": "sigma", "band": "band"},
)
nested = nested.with_columns(
    [pl.Series(name, result[:, i]) for i, name in enumerate(feature.names)]
)
nested.select(["object_id"] + feature.names)

import light_curve as licu import numpy as np import polars as pl BANDS = ["g", "r"] rng = np.random.default_rng(42) n_obj, n_per_band = 200, 40 object_id = np.repeat(np.arange(n_obj), n_per_band * len(BANDS)) band_col = np.tile(np.repeat(BANDS, n_per_band), n_obj) t = np.sort(rng.uniform(0, 100, n_obj * n_per_band * len(BANDS))) m = rng.normal(15.0, 0.3, len(object_id)) sigma = rng.uniform(0.01, 0.1, len(object_id)) df = pl.DataFrame({"object_id": object_id, "band": band_col, "t": t, "m": m, "sigma": sigma}) nested = df.group_by("object_id").agg(pl.struct("t", "m", "sigma", "band").alias("lc")) feature = licu.Extractor( licu.ExcessVariance(bands=BANDS), # variability excess over noise per band licu.StetsonK(bands=BANDS), # variability index per band licu.BeyondNStd(nstd=1.5, bands=BANDS), # outlier fraction per band licu.ColorOfMedian(BANDS), # colour at median brightness licu.ColorSpread(BANDS), # std dev of per-band means ) result = feature.many( nested["lc"], arrow_fields={"t": "t", "m": "m", "sigma": "sigma", "band": "band"}, ) nested = nested.with_columns( [pl.Series(name, result[:, i]) for i, name in enumerate(feature.names)] ) nested.select(["object_id"] + feature.names)

Out[8]:

shape: (200, 9)

object_id	excess_variance_g	excess_variance_r	stetson_K_g	stetson_K_r	beyond_2_std_g	beyond_2_std_r	color_median_g_r	color_spread
i64	f64	f64	f64	f64	f64	f64	f64	f64
93	0.000307	0.000316	0.708758	0.645206	0.1	0.15	-0.014721	0.093878
167	0.000443	0.000308	0.726951	0.695289	0.075	0.15	-0.002568	0.049795
48	0.000417	0.000271	0.739802	0.714027	0.075	0.1	-0.031613	0.056682
146	0.000434	0.000427	0.732108	0.709927	0.125	0.15	0.025346	0.006484
137	0.000275	0.000446	0.702491	0.718973	0.175	0.125	-0.023324	0.022564
…	…	…	…	…	…	…	…	…
187	0.0004	0.00039	0.633116	0.697914	0.125	0.125	-0.083745	0.049358
119	0.00037	0.000321	0.675758	0.662108	0.15	0.125	-0.045653	0.020228
50	0.000377	0.000463	0.536434	0.670722	0.1	0.15	0.044519	0.073465
68	0.000384	0.00032	0.72798	0.789902	0.1	0.15	0.050836	0.053354
77	0.000463	0.000298	0.640967	0.728134	0.175	0.125	-0.125955	0.037047

Next steps¶

• Feature basics tutorial — single features, Extractor, multiband intro
• Multiband tutorial — per-band and cross-band features
• Periodogram tutorial — Lomb–Scargle and period search
• API reference — full signatures and equations