Pipeline¶
This is a tutorial showcasing VASCA’s pipeline flow on a simple example. We will go
through all the steps equivalent to what is done in vasca_pipe.run_from_file.
This is the same function that is called when starting the pipeline from the CLI using vasca-pipe.
The goal is to create a VASCA Region from multiple GALEXField for which we
download the raw data online from MAST.
We apply quality cuts and do source clustering followed by variability analysis.
For this tutorial we are interested in the near-ultraviolet (NUV) observations by GALEX. We are going to look at neighboring/overlapping fields all of which contain the location of famous Tidal Disruption Event PS1-10jh discovered by Pan-STARRS and observed by GALEX in 2010.
GALEX sky map with field footprints of observations around the location of PS1-10jh ( purple crosshair). Screenshot from MAST Portal¶
General Configuration¶
The standard pipeline processing starts by reading a yaml file. To keep this tutorial simple, we are going to introduce parts of the configuration step by step at the point where they are required in the pipeline flow.
Note
An important premise of the configuration is that each parameter needs to be configured explicitly. This means even default values are specified all the time. This is a design decision purposefully made in order to ensure transparent and complete configurations. As a result, all possible parameters are always included when looking at configuration file.
Let’s begin with the general section. Here, basic information and functionality is
configured. The name of the pipeline run specifies also the name of directory in
which all results will be stored. The location of output directory is at out_dir_base
relative to the root directory of the package.
VASCA uses the powerful logging system provided by loguru.
The configuration specifies the log_level,
which we are going to set to debugging mode here. By default VASCA is going to save
all logging messages in a file stored in the output directory. log_file specifies
the name of that file, while default tells the pipeline to use a default name.
Parallel processing of the field-level analysis can be enabled when setting the number
of CPU threads nr_cpus > 1.
VASCA can include field-averaged reference data, if such data is available additional
to the visit-level data from the instruments mission pipeline. To save memory/storage
and computation time it is configurable wether to include reference sources in the
final Region-file (save_ref_srcs) and to repeat already processed fields that
are included in the region (run_fields).
# Dictionary holding the configuration
config = {}
# General section of the configuration
config["general"] = {
"name": "simple_pipe",
"out_dir_base": "docs/tutorial_resources/vasca_pipeline",
"log_level": "DEBUG",
"log_file": "default",
"nr_cpus": 3,
"save_ref_srcs": True,
"run_fields": True,
}
Tip
In case the output location is on a remote server and multiple users should be
able to edit the data, i.e., reprocess data using an updated configruation, then
one needs to manage user access priviliges. For convenience this can be done
using umask:
import os
os.umask("0o003", 0)
This will grand user and group full permissions. The setting is only persistant throughout the Python session.
The pipeline begins with some prerequisites including enabling logging and creating the output directory
import sys
from loguru import logger
from pathlib import Path
from importlib.resources import files
# Setup output directory
out_dir_base = Path(files("vasca").parent / config["general"]["out_dir_base"])
out_dir_base.mkdir(parents=True, exist_ok=True)
pipe_dir = out_dir_base / config["general"]["name"]
pipe_dir.mkdir(parents=True, exist_ok=True)
# Path to log file
log_file = (
pipe_dir / f'{config["general"]["name"]}.log'
) # Pipeline name is used by default
# Logger configuration, both to stdout and .log file
log_cfg = {
"handlers": [
{
"sink": sys.stdout,
"format": "<green>{time:YYYY-MM-DD HH:mm:ss.SSSS}</green> "
"<cyan>{name}</cyan>:<cyan>{line}</cyan> |"
"<level>{level}:</level> {message}",
"level": config["general"]["log_level"],
"colorize": True,
"backtrace": True,
"diagnose": True,
},
],
}
logger.configure(**log_cfg) # Set config
logger.add(log_file) # Add file logger
logger.enable("vasca") # Enable logging
# Some initial logging messages
logger.info(f"Runing '{config['general']['name']}'")
logger.debug(f"Config. file: {log_file}")
2024-11-11 08:55:49.3891 __main__:38 |INFO: Runing 'simple_pipe'
2024-11-11 08:55:49.3901 __main__:39 |DEBUG: Config. file: /opt/hostedtoolcache/Python/3.11.10/x64/lib/python3.11/site-packages/docs/tutorial_resources/vasca_pipeline/simple_pipe/simple_pipe.log
Resources¶
Next is the section about resource handling. This specifies the method used to load
(load_method) field data and which data products should be included (tables,
tables plus visit-level images, or only metadata load_products). Additionally
the coadd_exists flag tells the pipeline wether it can expect co-added (field-
averaged) data. Finally, field_kwargs allows to pass parameters directly to
the init function of a field class.
Here we are going to initialize fields from local raw data, if present. Else the
required data is downloaded from MAST. All data products will be included including
co-add data. Using the ResourceManager we can tell the field class where to
locate the data.
from vasca.resource_manager import ResourceManager
# Get the data locations using ResourceManager
rm = ResourceManager()
data_path = rm.get_path("docs_resources", "vasca")
visits_data_path = rm.get_path("gal_visits_list", "vasca")
# Resource section of the configuration
config["resources"] = {
"load_method": "MAST_LOCAL",
"load_products": "ALL",
"coadd_exists": True,
"field_kwargs": {
"data_path": data_path,
"visits_data_path": visits_data_path,
},
}
Observations¶
The observations section of the configuration is responsible for configuring
which combination of instrument (observatory) and filter (obs_filter)to
load data. Also it specifies the exact list of fields to load (obs_field_ids).
In a later step we will also add here the selection parameters (selection) used
for the quality cuts on the data and the field-level clustering settings
(cluster_det).
config["observations"] = [
{
"observatory": "GALEX",
"obs_filter": "NUV",
"obs_field_ids": [
3880803393752530944, # MISGCSN2_10493_0117
2529969795944677376, # ELAISN1_09
2597664528044916736, # PS_ELAISN1_MOS15
],
# "cluster_det": {},
# "selection": {},
},
# More instruments/filters...
]
Find below the visit metadata about the fields under investigation.
Show code cell source
from astropy.table import Table
df_gal_visits = (
Table.read(visits_data_path)
.to_pandas()
.apply(lambda x: x.str.decode("utf-8") if x.dtype == "O" else x)
)
query = f"ParentImgRunID in {list(config['observations'][0]['obs_field_ids'])}"
df_select = df_gal_visits.query(query)
show(
df_select,
classes="display nowrap compact",
scrollY="300px",
scrollCollapse=True,
paging=False,
columnDefs=[{"className": "dt-body-left", "targets": "_all"}],
)
| RATileCenter | DECTileCenter | survey | nexptime | fexptime | imgRunID | ParentImgRunID | joinID | tileNum | specTileNum | source | nPhotoObjects | nPhotoVisits | PhotoObsDate | spectra | nSpectra | nSpectraVisits | SpecObsDate | visitNum | subvis | minPhotoObsDate | maxPhotoObsDate | minSpecObsDate | maxSpecObsDate | PhotoObsDate_MJD | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
Loading ITables v2.2.3 from the init_notebook_mode cell...
(need help?) |
In the next step we will initialize a VASCA Region with all fields sequentially.
load_from_config is a convenience function that acts as an interface between the
region object and field-specific loading functions. This will downloads the data from
MAST, it will detect if the data is already present on disc and loads the cashed
files. To safe compute time later, a VASCA-field file is written to the download
location so that one can use this file instead of creating a new field from raw data.
This will be used during the field-level processing.
from vasca.region import Region
rg = Region.load_from_config(config)
Show code cell output
2024-11-11 08:55:50.8793 vasca.tables:156 |DEBUG: Adding table 'region:tt_fields'
2024-11-11 08:55:50.8819 vasca.region:71 |DEBUG: Loading fields from config file
2024-11-11 08:55:50.8826 vasca.field:747 |INFO: Loading field '3880803393752530944' with method 'MAST_LOCAL' for filter 'NUV' and load_products 'ALL'.
2024-11-11 08:55:50.8839 vasca.field:533 |DEBUG: Field data path set to: '/home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/3880803393752530944'
2024-11-11 08:55:50.8844 vasca.field:534 |DEBUG: Visits data path set to: '/home/runner/work/vasca-mirror/vasca-mirror/vasca/test/resources/GALEX_visits_list.fits'
2024-11-11 08:55:50.8855 vasca.field:888 |DEBUG: Downloading archive field info and saving to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/3880803393752530944/MAST_3880803393752530944_NUV_coadd.fits.
2024-11-11 08:55:51.1738 vasca.field:908 |DEBUG: Constructing 'tt_fields'.
2024-11-11 08:55:51.1763 vasca.tables:156 |DEBUG: Adding table 'base_field:tt_fields'
2024-11-11 08:55:51.1782 vasca.field:957 |DEBUG: Loading GALEX visit info :3880803393752530944 NUV None
2024-11-11 08:55:51.1787 vasca.field:988 |DEBUG: Reading archive visit info from cashed file '/home/runner/work/vasca-mirror/vasca-mirror/vasca/test/resources/GALEX_visits_list.fits'
2024-11-11 08:55:51.2051 vasca.tables:156 |DEBUG: Adding table 'galex_field:tt_visits'
2024-11-11 08:55:51.2066 vasca.field:403 |DEBUG: Getting parameter 'field_id': b'GNU3880803393752530944', , S
2024-11-11 08:55:51.2072 vasca.field:403 |DEBUG: Getting parameter 'field_name': b'MISGCSN2_10493_0117', None, S
2024-11-11 08:55:51.2079 vasca.field:403 |DEBUG: Getting parameter 'ra': 242.64270077057597, deg, f
2024-11-11 08:55:51.2086 vasca.field:403 |DEBUG: Getting parameter 'dec': 53.1235646075049, deg, f
2024-11-11 08:55:51.2095 vasca.field:403 |DEBUG: Getting parameter 'fov_diam': 1.2000000476837158, deg, f
2024-11-11 08:55:51.2100 vasca.field:403 |DEBUG: Getting parameter 'observatory': b'GALEX', None, S
2024-11-11 08:55:51.2106 vasca.field:403 |DEBUG: Getting parameter 'obs_filter': b'NUV', None, S
2024-11-11 08:55:51.2112 vasca.field:403 |DEBUG: Getting parameter 'ra': 242.64270077057597, deg, f
2024-11-11 08:55:51.2118 vasca.field:403 |DEBUG: Getting parameter 'dec': 53.1235646075049, deg, f
2024-11-11 08:55:51.2134 vasca.field:324 |DEBUG: Set attributes: {'tt_fields': ['field_id', 'field_name', 'ra', 'dec', 'fov_diam', 'observatory', 'obs_filter', 'center'], 'tt_visits': ['nr_vis', 'time_bin_size_sum', 'time_start', 'time_stop']}
2024-11-11 08:55:51.2146 vasca.field:1076 |DEBUG: Reading archive field info from cashed file '/home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/3880803393752530944/MAST_3880803393752530944_NUV_coadd.fits'
2024-11-11 08:55:51.2292 vasca.field:1081 |DEBUG: Downloading archive data products list and saving to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/3880803393752530944/MAST_3880803393752530944_NUV_data.fits.
2024-11-11 08:55:51.4854 vasca.field:1164 |DEBUG: Downloading archive data products. Manifest saved to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/3880803393752530944/MAST_3880803393752530944_NUV_down.ecsv.
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR7/pipe/01-vsn/44763-MISGCSN2_10493_0117/d/00-visits/0001-img/07-try/MISGCSN2_10493_0117_0001-nd-int.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/3880803393752530944/mastDownload/GALEX/3880803256112250880/MISGCSN2_10493_0117_0001-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR7/pipe/01-vsn/44763-MISGCSN2_10493_0117/d/00-visits/0001-img/07-try/MISGCSN2_10493_0117_0001-xd-mcat.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/3880803393752530944/mastDownload/GALEX/3880803256112250880/MISGCSN2_10493_0117_0001-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR7/pipe/01-vsn/44763-MISGCSN2_10493_0117/d/01-main/0007-img/07-try/MISGCSN2_10493_0117-nd-int.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/3880803393752530944/mastDownload/GALEX/3880803393752530944/MISGCSN2_10493_0117-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR7/pipe/01-vsn/44763-MISGCSN2_10493_0117/d/01-main/0007-img/07-try/MISGCSN2_10493_0117-xd-mcat.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/3880803393752530944/mastDownload/GALEX/3880803393752530944/MISGCSN2_10493_0117-xd-mcat.fits.gz ...
[Done]
2024-11-11 08:55:54.3697 vasca.tables:156 |DEBUG: Adding table 'galex_field:tt_detections'
2024-11-11 08:55:54.3742 vasca.field:1391 |DEBUG: Constructed 'tt_detections'.
2024-11-11 08:55:54.3906 vasca.tables:156 |DEBUG: Adding table 'galex_field:tt_coadd_detections'
2024-11-11 08:55:54.3947 vasca.field:1396 |DEBUG: Constructed 'tt_coadd_detections'.
2024-11-11 08:55:54.3954 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/3880803393752530944/mastDownload/GALEX/3880803393752530944/MISGCSN2_10493_0117-nd-int.fits.gz'
2024-11-11 08:55:54.9054 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/3880803393752530944/mastDownload/GALEX/3880803256112250880/MISGCSN2_10493_0117_0001-nd-int.fits.gz'
2024-11-11 08:55:55.4835 vasca.tables:211 |INFO: Writing file with name '/home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/3880803393752530944/VASCA_GALEX_3880803393752530944_NUV_field_data.fits'
2024-11-11 08:55:55.4848 vasca.tables:222 |DEBUG: Storing image data of shape (3840, 3840)
2024-11-11 08:55:55.4988 vasca.tables:236 |DEBUG: Storing image data of shape (3840, 3840)
2024-11-11 08:55:57.3454 vasca.tables:251 |DEBUG: Writing table 'tt_fields'
2024-11-11 08:55:57.3592 vasca.tables:251 |DEBUG: Writing table 'tt_visits'
2024-11-11 08:55:57.3708 vasca.tables:251 |DEBUG: Writing table 'tt_detections'
2024-11-11 08:55:57.4031 vasca.tables:251 |DEBUG: Writing table 'tt_coadd_detections'
2024-11-11 08:55:57.4632 vasca.field:683 |INFO: Loaded new GALEX field '3880803393752530944' with obs_filter 'NUV'from MAST data .
2024-11-11 08:55:57.4650 vasca.field:747 |INFO: Loading field '2529969795944677376' with method 'MAST_LOCAL' for filter 'NUV' and load_products 'ALL'.
2024-11-11 08:55:57.4658 vasca.field:533 |DEBUG: Field data path set to: '/home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2529969795944677376'
2024-11-11 08:55:57.4666 vasca.field:534 |DEBUG: Visits data path set to: '/home/runner/work/vasca-mirror/vasca-mirror/vasca/test/resources/GALEX_visits_list.fits'
2024-11-11 08:55:57.4672 vasca.field:888 |DEBUG: Downloading archive field info and saving to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2529969795944677376/MAST_2529969795944677376_NUV_coadd.fits.
2024-11-11 08:55:57.5818 vasca.field:908 |DEBUG: Constructing 'tt_fields'.
2024-11-11 08:55:57.5846 vasca.tables:156 |DEBUG: Adding table 'base_field:tt_fields'
2024-11-11 08:55:57.5861 vasca.field:957 |DEBUG: Loading GALEX visit info :2529969795944677376 NUV None
2024-11-11 08:55:57.5866 vasca.field:988 |DEBUG: Reading archive visit info from cashed file '/home/runner/work/vasca-mirror/vasca-mirror/vasca/test/resources/GALEX_visits_list.fits'
2024-11-11 08:55:57.6120 vasca.tables:156 |DEBUG: Adding table 'galex_field:tt_visits'
2024-11-11 08:55:57.6136 vasca.field:403 |DEBUG: Getting parameter 'field_id': b'GNU2529969795944677376', , S
2024-11-11 08:55:57.6142 vasca.field:403 |DEBUG: Getting parameter 'field_name': b'ELAISN1_09', None, S
2024-11-11 08:55:57.6148 vasca.field:403 |DEBUG: Getting parameter 'ra': 241.60667709058498, deg, f
2024-11-11 08:55:57.6154 vasca.field:403 |DEBUG: Getting parameter 'dec': 53.531821052495, deg, f
2024-11-11 08:55:57.6160 vasca.field:403 |DEBUG: Getting parameter 'fov_diam': 1.2000000476837158, deg, f
2024-11-11 08:55:57.6166 vasca.field:403 |DEBUG: Getting parameter 'observatory': b'GALEX', None, S
2024-11-11 08:55:57.6180 vasca.field:403 |DEBUG: Getting parameter 'obs_filter': b'NUV', None, S
2024-11-11 08:55:57.6186 vasca.field:403 |DEBUG: Getting parameter 'ra': 241.60667709058498, deg, f
2024-11-11 08:55:57.6192 vasca.field:403 |DEBUG: Getting parameter 'dec': 53.531821052495, deg, f
2024-11-11 08:55:57.6204 vasca.field:324 |DEBUG: Set attributes: {'tt_fields': ['field_id', 'field_name', 'ra', 'dec', 'fov_diam', 'observatory', 'obs_filter', 'center'], 'tt_visits': ['nr_vis', 'time_bin_size_sum', 'time_start', 'time_stop']}
2024-11-11 08:55:57.6210 vasca.field:1076 |DEBUG: Reading archive field info from cashed file '/home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2529969795944677376/MAST_2529969795944677376_NUV_coadd.fits'
2024-11-11 08:55:57.6357 vasca.field:1081 |DEBUG: Downloading archive data products list and saving to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2529969795944677376/MAST_2529969795944677376_NUV_data.fits.
2024-11-11 08:55:58.1629 vasca.field:1164 |DEBUG: Downloading archive data products. Manifest saved to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2529969795944677376/MAST_2529969795944677376_NUV_down.ecsv.
INFO: 173 of 1824 products were duplicates. Only downloading 1651 unique product(s). [astroquery.mast.observations]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/06370-ELAISN1_09/d/00-visits/0001-img/07-try/ELAISN1_09_0001-nd-int.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658505723904/ELAISN1_09_0001-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/06370-ELAISN1_09/d/00-visits/0001-img/07-try/ELAISN1_09_0001-xd-mcat.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658505723904/ELAISN1_09_0001-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/06370-ELAISN1_09/d/00-visits/0003-img/07-try/ELAISN1_09_0003-nd-int.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658572832768/ELAISN1_09_0003-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/06370-ELAISN1_09/d/00-visits/0003-img/07-try/ELAISN1_09_0003-xd-mcat.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658572832768/ELAISN1_09_0003-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/06370-ELAISN1_09/d/00-visits/0004-img/07-try/ELAISN1_09_0004-nd-int.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658606387200/ELAISN1_09_0004-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/06370-ELAISN1_09/d/00-visits/0004-img/07-try/ELAISN1_09_0004-xd-mcat.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658606387200/ELAISN1_09_0004-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/06370-ELAISN1_09/d/00-visits/0005-img/07-try/ELAISN1_09_0005-nd-int.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658639941632/ELAISN1_09_0005-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/06370-ELAISN1_09/d/00-visits/0005-img/07-try/ELAISN1_09_0005-xd-mcat.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658639941632/ELAISN1_09_0005-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/06370-ELAISN1_09/d/00-visits/0006-img/07-try/ELAISN1_09_0006-nd-int.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658673496064/ELAISN1_09_0006-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/06370-ELAISN1_09/d/00-visits/0006-img/07-try/ELAISN1_09_0006-xd-mcat.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658673496064/ELAISN1_09_0006-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/06370-ELAISN1_09/d/00-visits/0007-img/07-try/ELAISN1_09_0007-nd-int.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658707050496/ELAISN1_09_0007-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/06370-ELAISN1_09/d/00-visits/0007-img/07-try/ELAISN1_09_0007-xd-mcat.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658707050496/ELAISN1_09_0007-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/06370-ELAISN1_09/d/00-visits/0008-img/07-try/ELAISN1_09_0008-nd-int.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658740604928/ELAISN1_09_0008-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/06370-ELAISN1_09/d/00-visits/0008-img/07-try/ELAISN1_09_0008-xd-mcat.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658740604928/ELAISN1_09_0008-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/06370-ELAISN1_09/d/00-visits/0009-img/07-try/ELAISN1_09_0009-nd-int.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658774159360/ELAISN1_09_0009-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/06370-ELAISN1_09/d/00-visits/0009-img/07-try/ELAISN1_09_0009-xd-mcat.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658774159360/ELAISN1_09_0009-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/06370-ELAISN1_09/d/00-visits/0010-img/07-try/ELAISN1_09_0010-nd-int.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658807713792/ELAISN1_09_0010-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/06370-ELAISN1_09/d/00-visits/0010-img/07-try/ELAISN1_09_0010-xd-mcat.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658807713792/ELAISN1_09_0010-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/06370-ELAISN1_09/d/00-visits/0011-img/07-try/ELAISN1_09_0011-nd-int.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658841268224/ELAISN1_09_0011-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/06370-ELAISN1_09/d/00-visits/0011-img/07-try/ELAISN1_09_0011-xd-mcat.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658841268224/ELAISN1_09_0011-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/06370-ELAISN1_09/d/00-visits/0012-img/07-try/ELAISN1_09_0012-nd-int.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658874822656/ELAISN1_09_0012-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/06370-ELAISN1_09/d/00-visits/0012-img/07-try/ELAISN1_09_0012-xd-mcat.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658874822656/ELAISN1_09_0012-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/06370-ELAISN1_09/d/00-visits/0013-img/07-try/ELAISN1_09_0013-nd-int.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658908377088/ELAISN1_09_0013-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/06370-ELAISN1_09/d/00-visits/0013-img/07-try/ELAISN1_09_0013-xd-mcat.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658908377088/ELAISN1_09_0013-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/06370-ELAISN1_09/d/00-visits/0014-img/07-try/ELAISN1_09_0014-nd-int.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658941931520/ELAISN1_09_0014-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/06370-ELAISN1_09/d/00-visits/0014-img/07-try/ELAISN1_09_0014-xd-mcat.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658941931520/ELAISN1_09_0014-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/06370-ELAISN1_09/d/00-visits/0015-img/07-try/ELAISN1_09_0015-nd-int.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658975485952/ELAISN1_09_0015-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/06370-ELAISN1_09/d/00-visits/0015-img/07-try/ELAISN1_09_0015-xd-mcat.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658975485952/ELAISN1_09_0015-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/06370-ELAISN1_09/d/00-visits/0017-img/07-try/ELAISN1_09_0017-nd-int.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969659042594816/ELAISN1_09_0017-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/06370-ELAISN1_09/d/00-visits/0017-img/07-try/ELAISN1_09_0017-xd-mcat.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969659042594816/ELAISN1_09_0017-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/06370-ELAISN1_09/d/00-visits/0018-img/07-try/ELAISN1_09_0018-nd-int.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969659076149248/ELAISN1_09_0018-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/06370-ELAISN1_09/d/00-visits/0018-img/07-try/ELAISN1_09_0018-xd-mcat.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969659076149248/ELAISN1_09_0018-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/06370-ELAISN1_09/d/01-main/0001-img/07-try/ELAISN1_09-nd-int.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969795944677376/ELAISN1_09-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/06370-ELAISN1_09/d/01-main/0001-img/07-try/ELAISN1_09-xd-mcat.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969795944677376/ELAISN1_09-xd-mcat.fits.gz ...
[Done]
2024-11-11 08:56:32.0261 vasca.tables:156 |DEBUG: Adding table 'galex_field:tt_detections'
2024-11-11 08:56:32.0593 vasca.field:1391 |DEBUG: Constructed 'tt_detections'.
2024-11-11 08:56:32.0926 vasca.tables:156 |DEBUG: Adding table 'galex_field:tt_coadd_detections'
2024-11-11 08:56:32.0992 vasca.field:1396 |DEBUG: Constructed 'tt_coadd_detections'.
2024-11-11 08:56:32.1000 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969795944677376/ELAISN1_09-nd-int.fits.gz'
2024-11-11 08:56:32.6923 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658505723904/ELAISN1_09_0001-nd-int.fits.gz'
2024-11-11 08:56:33.3187 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658572832768/ELAISN1_09_0003-nd-int.fits.gz'
2024-11-11 08:56:33.9426 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658606387200/ELAISN1_09_0004-nd-int.fits.gz'
2024-11-11 08:56:34.5944 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658639941632/ELAISN1_09_0005-nd-int.fits.gz'
2024-11-11 08:56:35.2463 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658673496064/ELAISN1_09_0006-nd-int.fits.gz'
2024-11-11 08:56:35.8989 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658707050496/ELAISN1_09_0007-nd-int.fits.gz'
2024-11-11 08:56:36.5564 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658740604928/ELAISN1_09_0008-nd-int.fits.gz'
2024-11-11 08:56:37.2145 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658774159360/ELAISN1_09_0009-nd-int.fits.gz'
2024-11-11 08:56:37.8796 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658807713792/ELAISN1_09_0010-nd-int.fits.gz'
2024-11-11 08:56:38.5528 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658841268224/ELAISN1_09_0011-nd-int.fits.gz'
2024-11-11 08:56:39.2362 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658874822656/ELAISN1_09_0012-nd-int.fits.gz'
2024-11-11 08:56:39.9087 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658908377088/ELAISN1_09_0013-nd-int.fits.gz'
2024-11-11 08:56:40.5939 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658941931520/ELAISN1_09_0014-nd-int.fits.gz'
2024-11-11 08:56:41.2779 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658975485952/ELAISN1_09_0015-nd-int.fits.gz'
2024-11-11 08:56:41.9610 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969659042594816/ELAISN1_09_0017-nd-int.fits.gz'
2024-11-11 08:56:42.6491 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969659076149248/ELAISN1_09_0018-nd-int.fits.gz'
2024-11-11 08:56:43.3461 vasca.tables:211 |INFO: Writing file with name '/home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2529969795944677376/VASCA_GALEX_2529969795944677376_NUV_field_data.fits'
2024-11-11 08:56:43.3472 vasca.tables:222 |DEBUG: Storing image data of shape (3840, 3840)
2024-11-11 08:56:43.3552 vasca.tables:236 |DEBUG: Storing image data of shape (16, 3840, 3840)
2024-11-11 08:56:59.1873 vasca.tables:251 |DEBUG: Writing table 'tt_fields'
2024-11-11 08:56:59.2008 vasca.tables:251 |DEBUG: Writing table 'tt_visits'
2024-11-11 08:56:59.2121 vasca.tables:251 |DEBUG: Writing table 'tt_detections'
2024-11-11 08:56:59.3051 vasca.tables:251 |DEBUG: Writing table 'tt_coadd_detections'
2024-11-11 08:56:59.3842 vasca.field:683 |INFO: Loaded new GALEX field '2529969795944677376' with obs_filter 'NUV'from MAST data .
2024-11-11 08:56:59.3875 vasca.field:747 |INFO: Loading field '2597664528044916736' with method 'MAST_LOCAL' for filter 'NUV' and load_products 'ALL'.
2024-11-11 08:56:59.3883 vasca.field:533 |DEBUG: Field data path set to: '/home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736'
2024-11-11 08:56:59.3888 vasca.field:534 |DEBUG: Visits data path set to: '/home/runner/work/vasca-mirror/vasca-mirror/vasca/test/resources/GALEX_visits_list.fits'
2024-11-11 08:56:59.3896 vasca.field:888 |DEBUG: Downloading archive field info and saving to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/MAST_2597664528044916736_NUV_coadd.fits.
2024-11-11 08:56:59.4963 vasca.field:908 |DEBUG: Constructing 'tt_fields'.
2024-11-11 08:56:59.4987 vasca.tables:156 |DEBUG: Adding table 'base_field:tt_fields'
2024-11-11 08:56:59.5005 vasca.field:957 |DEBUG: Loading GALEX visit info :2597664528044916736 NUV None
2024-11-11 08:56:59.5010 vasca.field:988 |DEBUG: Reading archive visit info from cashed file '/home/runner/work/vasca-mirror/vasca-mirror/vasca/test/resources/GALEX_visits_list.fits'
2024-11-11 08:56:59.5270 vasca.tables:156 |DEBUG: Adding table 'galex_field:tt_visits'
2024-11-11 08:56:59.5287 vasca.field:403 |DEBUG: Getting parameter 'field_id': b'GNU2597664528044916736', , S
2024-11-11 08:56:59.5293 vasca.field:403 |DEBUG: Getting parameter 'field_name': b'PS_ELAISN1_MOS15', None, S
2024-11-11 08:56:59.5299 vasca.field:403 |DEBUG: Getting parameter 'ra': 242.396514585217, deg, f
2024-11-11 08:56:59.5306 vasca.field:403 |DEBUG: Getting parameter 'dec': 53.9999872930018, deg, f
2024-11-11 08:56:59.5312 vasca.field:403 |DEBUG: Getting parameter 'fov_diam': 1.2000000476837158, deg, f
2024-11-11 08:56:59.5318 vasca.field:403 |DEBUG: Getting parameter 'observatory': b'GALEX', None, S
2024-11-11 08:56:59.5323 vasca.field:403 |DEBUG: Getting parameter 'obs_filter': b'NUV', None, S
2024-11-11 08:56:59.5329 vasca.field:403 |DEBUG: Getting parameter 'ra': 242.396514585217, deg, f
2024-11-11 08:56:59.5336 vasca.field:403 |DEBUG: Getting parameter 'dec': 53.9999872930018, deg, f
2024-11-11 08:56:59.5353 vasca.field:324 |DEBUG: Set attributes: {'tt_fields': ['field_id', 'field_name', 'ra', 'dec', 'fov_diam', 'observatory', 'obs_filter', 'center'], 'tt_visits': ['nr_vis', 'time_bin_size_sum', 'time_start', 'time_stop']}
2024-11-11 08:56:59.5359 vasca.field:1076 |DEBUG: Reading archive field info from cashed file '/home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/MAST_2597664528044916736_NUV_coadd.fits'
2024-11-11 08:56:59.5503 vasca.field:1081 |DEBUG: Downloading archive data products list and saving to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/MAST_2597664528044916736_NUV_data.fits.
2024-11-11 08:57:00.1411 vasca.field:1164 |DEBUG: Downloading archive data products. Manifest saved to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/MAST_2597664528044916736_NUV_down.ecsv.
INFO: 173 of 2328 products were duplicates. Only downloading 2155 unique product(s). [astroquery.mast.observations]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0001-img/07-try/PS_ELAISN1_MOS15_0001-nd-int.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390404636672/PS_ELAISN1_MOS15_0001-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0001-img/07-try/PS_ELAISN1_MOS15_0001-xd-mcat.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390404636672/PS_ELAISN1_MOS15_0001-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0002-img/07-try/PS_ELAISN1_MOS15_0002-nd-int.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390438191104/PS_ELAISN1_MOS15_0002-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0002-img/07-try/PS_ELAISN1_MOS15_0002-xd-mcat.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390438191104/PS_ELAISN1_MOS15_0002-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0003-img/07-try/PS_ELAISN1_MOS15_0003-nd-int.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390471745536/PS_ELAISN1_MOS15_0003-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0003-img/07-try/PS_ELAISN1_MOS15_0003-xd-mcat.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390471745536/PS_ELAISN1_MOS15_0003-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0004-img/07-try/PS_ELAISN1_MOS15_0004-nd-int.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390505299968/PS_ELAISN1_MOS15_0004-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0004-img/07-try/PS_ELAISN1_MOS15_0004-xd-mcat.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390505299968/PS_ELAISN1_MOS15_0004-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0005-img/07-try/PS_ELAISN1_MOS15_0005-nd-int.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390538854400/PS_ELAISN1_MOS15_0005-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0005-img/07-try/PS_ELAISN1_MOS15_0005-xd-mcat.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390538854400/PS_ELAISN1_MOS15_0005-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0006-img/07-try/PS_ELAISN1_MOS15_0006-nd-int.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390572408832/PS_ELAISN1_MOS15_0006-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0006-img/07-try/PS_ELAISN1_MOS15_0006-xd-mcat.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390572408832/PS_ELAISN1_MOS15_0006-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0007-img/07-try/PS_ELAISN1_MOS15_0007-nd-int.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390605963264/PS_ELAISN1_MOS15_0007-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0007-img/07-try/PS_ELAISN1_MOS15_0007-xd-mcat.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390605963264/PS_ELAISN1_MOS15_0007-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0008-img/07-try/PS_ELAISN1_MOS15_0008-nd-int.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390639517696/PS_ELAISN1_MOS15_0008-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0008-img/07-try/PS_ELAISN1_MOS15_0008-xd-mcat.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390639517696/PS_ELAISN1_MOS15_0008-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0009-img/07-try/PS_ELAISN1_MOS15_0009-nd-int.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390673072128/PS_ELAISN1_MOS15_0009-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0009-img/07-try/PS_ELAISN1_MOS15_0009-xd-mcat.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390673072128/PS_ELAISN1_MOS15_0009-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0010-img/07-try/PS_ELAISN1_MOS15_0010-nd-int.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390706626560/PS_ELAISN1_MOS15_0010-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0010-img/07-try/PS_ELAISN1_MOS15_0010-xd-mcat.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390706626560/PS_ELAISN1_MOS15_0010-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0011-img/07-try/PS_ELAISN1_MOS15_0011-nd-int.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390740180992/PS_ELAISN1_MOS15_0011-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0011-img/07-try/PS_ELAISN1_MOS15_0011-xd-mcat.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390740180992/PS_ELAISN1_MOS15_0011-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0012-img/07-try/PS_ELAISN1_MOS15_0012-nd-int.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390773735424/PS_ELAISN1_MOS15_0012-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0012-img/07-try/PS_ELAISN1_MOS15_0012-xd-mcat.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390773735424/PS_ELAISN1_MOS15_0012-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0018-img/07-try/PS_ELAISN1_MOS15_0018-nd-int.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390975062016/PS_ELAISN1_MOS15_0018-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0018-img/07-try/PS_ELAISN1_MOS15_0018-xd-mcat.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390975062016/PS_ELAISN1_MOS15_0018-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0021-img/07-try/PS_ELAISN1_MOS15_0021-nd-int.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664391075725312/PS_ELAISN1_MOS15_0021-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0021-img/07-try/PS_ELAISN1_MOS15_0021-xd-mcat.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664391075725312/PS_ELAISN1_MOS15_0021-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0022-img/07-try/PS_ELAISN1_MOS15_0022-nd-int.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664391109279744/PS_ELAISN1_MOS15_0022-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0022-img/07-try/PS_ELAISN1_MOS15_0022-xd-mcat.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664391109279744/PS_ELAISN1_MOS15_0022-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0023-img/07-try/PS_ELAISN1_MOS15_0023-nd-int.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664391142834176/PS_ELAISN1_MOS15_0023-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0023-img/07-try/PS_ELAISN1_MOS15_0023-xd-mcat.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664391142834176/PS_ELAISN1_MOS15_0023-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0024-img/07-try/PS_ELAISN1_MOS15_0024-nd-int.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664391176388608/PS_ELAISN1_MOS15_0024-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0024-img/07-try/PS_ELAISN1_MOS15_0024-xd-mcat.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664391176388608/PS_ELAISN1_MOS15_0024-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0025-img/07-try/PS_ELAISN1_MOS15_0025-nd-int.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664391209943040/PS_ELAISN1_MOS15_0025-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0025-img/07-try/PS_ELAISN1_MOS15_0025-xd-mcat.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664391209943040/PS_ELAISN1_MOS15_0025-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0026-img/07-try/PS_ELAISN1_MOS15_0026-nd-int.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664391243497472/PS_ELAISN1_MOS15_0026-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0026-img/07-try/PS_ELAISN1_MOS15_0026-xd-mcat.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664391243497472/PS_ELAISN1_MOS15_0026-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0027-img/07-try/PS_ELAISN1_MOS15_0027-nd-int.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664391277051904/PS_ELAISN1_MOS15_0027-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0027-img/07-try/PS_ELAISN1_MOS15_0027-xd-mcat.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664391277051904/PS_ELAISN1_MOS15_0027-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0028-img/07-try/PS_ELAISN1_MOS15_0028-nd-int.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664391310606336/PS_ELAISN1_MOS15_0028-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0028-img/07-try/PS_ELAISN1_MOS15_0028-xd-mcat.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664391310606336/PS_ELAISN1_MOS15_0028-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0029-img/07-try/PS_ELAISN1_MOS15_0029-nd-int.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664391344160768/PS_ELAISN1_MOS15_0029-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0029-img/07-try/PS_ELAISN1_MOS15_0029-xd-mcat.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664391344160768/PS_ELAISN1_MOS15_0029-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/01-main/0007-img/07-try/PS_ELAISN1_MOS15-nd-int.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664528044916736/PS_ELAISN1_MOS15-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/01-main/0007-img/07-try/PS_ELAISN1_MOS15-xd-mcat.fits.gz to /home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664528044916736/PS_ELAISN1_MOS15-xd-mcat.fits.gz ...
[Done]
2024-11-11 08:57:40.4795 vasca.tables:156 |DEBUG: Adding table 'galex_field:tt_detections'
2024-11-11 08:57:40.5232 vasca.field:1391 |DEBUG: Constructed 'tt_detections'.
2024-11-11 08:57:40.5606 vasca.tables:156 |DEBUG: Adding table 'galex_field:tt_coadd_detections'
2024-11-11 08:57:40.5678 vasca.field:1396 |DEBUG: Constructed 'tt_coadd_detections'.
2024-11-11 08:57:40.5685 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664528044916736/PS_ELAISN1_MOS15-nd-int.fits.gz'
2024-11-11 08:57:41.1578 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390404636672/PS_ELAISN1_MOS15_0001-nd-int.fits.gz'
2024-11-11 08:57:41.7856 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390438191104/PS_ELAISN1_MOS15_0002-nd-int.fits.gz'
2024-11-11 08:57:42.4404 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390471745536/PS_ELAISN1_MOS15_0003-nd-int.fits.gz'
2024-11-11 08:57:43.0806 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390505299968/PS_ELAISN1_MOS15_0004-nd-int.fits.gz'
2024-11-11 08:57:43.7224 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390538854400/PS_ELAISN1_MOS15_0005-nd-int.fits.gz'
2024-11-11 08:57:44.3711 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390572408832/PS_ELAISN1_MOS15_0006-nd-int.fits.gz'
2024-11-11 08:57:45.0172 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390605963264/PS_ELAISN1_MOS15_0007-nd-int.fits.gz'
2024-11-11 08:57:45.6733 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390639517696/PS_ELAISN1_MOS15_0008-nd-int.fits.gz'
2024-11-11 08:57:46.3387 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390673072128/PS_ELAISN1_MOS15_0009-nd-int.fits.gz'
2024-11-11 08:57:47.0081 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390706626560/PS_ELAISN1_MOS15_0010-nd-int.fits.gz'
2024-11-11 08:57:47.6843 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390740180992/PS_ELAISN1_MOS15_0011-nd-int.fits.gz'
2024-11-11 08:57:48.3604 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390773735424/PS_ELAISN1_MOS15_0012-nd-int.fits.gz'
2024-11-11 08:57:49.0341 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390975062016/PS_ELAISN1_MOS15_0018-nd-int.fits.gz'
2024-11-11 08:57:49.7177 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664391075725312/PS_ELAISN1_MOS15_0021-nd-int.fits.gz'
2024-11-11 08:57:50.4075 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664391109279744/PS_ELAISN1_MOS15_0022-nd-int.fits.gz'
2024-11-11 08:57:51.0979 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664391142834176/PS_ELAISN1_MOS15_0023-nd-int.fits.gz'
2024-11-11 08:57:51.7893 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664391176388608/PS_ELAISN1_MOS15_0024-nd-int.fits.gz'
2024-11-11 08:57:52.4841 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664391209943040/PS_ELAISN1_MOS15_0025-nd-int.fits.gz'
2024-11-11 08:57:53.1797 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664391243497472/PS_ELAISN1_MOS15_0026-nd-int.fits.gz'
2024-11-11 08:57:53.8820 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664391277051904/PS_ELAISN1_MOS15_0027-nd-int.fits.gz'
2024-11-11 08:57:54.5899 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664391310606336/PS_ELAISN1_MOS15_0028-nd-int.fits.gz'
2024-11-11 08:57:55.2959 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664391344160768/PS_ELAISN1_MOS15_0029-nd-int.fits.gz'
2024-11-11 08:57:56.0378 vasca.tables:211 |INFO: Writing file with name '/home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/VASCA_GALEX_2597664528044916736_NUV_field_data.fits'
2024-11-11 08:57:56.0391 vasca.tables:222 |DEBUG: Storing image data of shape (3840, 3840)
2024-11-11 08:57:56.0511 vasca.tables:236 |DEBUG: Storing image data of shape (22, 3840, 3840)
2024-11-11 08:58:16.8106 vasca.tables:251 |DEBUG: Writing table 'tt_fields'
2024-11-11 08:58:16.8242 vasca.tables:251 |DEBUG: Writing table 'tt_visits'
2024-11-11 08:58:16.8355 vasca.tables:251 |DEBUG: Writing table 'tt_detections'
2024-11-11 08:58:16.9546 vasca.tables:251 |DEBUG: Writing table 'tt_coadd_detections'
2024-11-11 08:58:17.0392 vasca.field:683 |INFO: Loaded new GALEX field '2597664528044916736' with obs_filter 'NUV'from MAST data .
2024-11-11 08:58:17.0454 vasca.tables:156 |DEBUG: Adding table 'region:tt_filters'
This populates the region object with all specified fields, the relevant metadata is stored in tt_fields.
rg.info()
# rg.tt_fields
tt_fields:
<Table length=3>
name dtype unit description
----------------- ------- ---- -------------------------------------------------------
field_id bytes32 Field source ID number
field_name bytes32 Field name
project bytes32 Field project, typically survey name
ra float64 deg Sky coordinate Right Ascension (J2000)
dec float64 deg Sky coordinate Declination (J2000)
observatory bytes22 Telescope of the observation (e.g. GALEX)
obs_filter bytes8 Filter of the observation (e.g. NUV)
fov_diam float32 deg Field radius or box size (depending on the observatory)
sel bool Selection of rows for VASCA analysis
nr_vis int32 Total number of visits of the field
time_bin_size_sum float32 s Total exposure time
time_start float64 d Start date and time in MJD
time_stop float64 d End time of last exposure
rg_fd_id int64 Region field ID number
{'DATAPATH': 'None', 'INFO': 'Field information table'}
tt_filters:
<Table length=2>
name dtype description
-------------- ------ ---------------------------------------
obs_filter_id int32 Observation filter ID number
obs_filter bytes8 Filter of the observation (e.g. NUV)
obs_filter_idx int32 Filter index in filter dependent arrays
{'INFO': 'Filters, their IDs and index, the last is specific for this region.'}
| field_id | field_name | project | ra | dec | observatory | obs_filter | fov_diam | sel | nr_vis | time_bin_size_sum | time_start | time_stop | rg_fd_id |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
Loading ITables v2.2.3 from the init_notebook_mode cell...
(need help?) |
Field-level analysis¶
The field-level analysis incorporates, first, the data reduction and parameter mapping from raw data to VASCA field objects, second, the data quality selection and finally source clustering on the remaining high-quality detections.
The first step is implicitly taken care of by the GALEXField class, where the raw
data is loaded and only the column parameters are kept that are specified in the tables_dict
module. A shortcut is provided through the Region.get_field method which is an
interface to the load method of
any field class.
The configuration for the next two step requires the selection and cluster_det
entries under the observations section.
Data selection¶
Note
A crucial part of VASCA’s flexibility to adapt to raw data of virtually any instrument
comes from the fact that the parameter list used for data quality selection is not
fixed and is allowed to vary for different instruments and filters. The only
requirement is an existent entry in the tables_dict module for any parameter and
a corresponding field class that includes these parameters in the tt_detections
table.
The API for the data selection is provided by the TableCollection.select_rows
method. Each entry under selection maps to this interface. The table parameters
specifies which table to select on. Any selection operation modifies the sel
column of a given table. It contains boolean values so 0 means unselected and
1 means selected.
By specifying the presel_typeparameter, one controls the logic by which an
existing selection is combined with a new one. The sel_type parameter specifies
the logic by which the selection on a set of multiple column parameters is combined.
Parameters range and bitmask provide the column parameter and artifact
bitflag values that are used to make the selection. Using set_range on can choose
to clip values of a certain column to minimum and maximum values.
In combination with sel_type = "is_in" and var parameters, it is possible to
select the rows of given column varin the target table if a value is also present
in the same column of a reference table (ref_table).
import numpy as np
# Updating the observations for GALEX-NUV observations
config["observations"][0].update(
{
"selection": {
# Quality cuts on visit-level detections
"det_quality": {
"table": "tt_detections",
"presel_type": "and",
"sel_type": "and",
"range": {
"s2n": [3.0, np.inf],
"r_fov": [0.0, 0.5],
"ellip_world": [0.0, 0.5],
"size_world": [0.0, 6.0],
"class_star": [0.15, 1.0],
"chkobj_type": [-0.5, 0.5],
"flux_app_ratio": [0.3, 1.05],
},
"bitmask": {
"artifacts": [2, 4, 8, 128, 256],
},
"set_range": {"pos_err": [0.5, 5]},
},
# Quality cuts on field-averaged detections
"coadd_det_quality": {
"table": "tt_detections",
"presel_type": "and",
"sel_type": "and",
"range": {
"s2n": [5.0, np.inf],
"r_fov": [0.0, 0.5],
"ellip_world": [0.0, 0.5],
"size_world": [0.0, 6.0],
"class_star": [0.15, 1.0],
"chkobj_type": [-0.5, 0.5],
"flux_app_ratio": [0.3, 1.05],
},
"bitmask": {
"artifacts": [2, 4, 8, 128, 256],
},
},
# Selection on only those detections wich are part of clusters
"det_association": {
"table": "tt_detections",
"presel_type": "and",
"sel_type": "is_in",
"ref_table": "tt_sources",
"var": "fd_src_id",
},
},
}
)
Clustering¶
Also the field-level clustering configuration showcases VASCA’s modular
approach. In the cluster_det section on specifies the clustering algorithm
which, in principle, can be different for each instrument and filter. Although,
at the moment only mean-shift clustering
is supported by VASCA.
Again, the responsible API is provided by TableCollection.cluster_meanshift.
This method wraps a method provided by the scikit-learn package. The end result is that each field optains
a new tt_visits table that lists all identified sources as defined
by their clustered detections. Sources have at the minimum one and as manny as n_vis
detections.
Mean-shift is well suited for this use case due to several reasons. Most importantly it is that the algorithm doesn’t require the total number of clusters as a parameter. In fact it is determining that number which would be otherwise very difficult to predict from the visit-level detections before having done the clustering.
Another reason is its relatively simple algorithm where only one parameters is
required. It is called the bandwidth which means, translated to the astronomy use
case, the radial size of a typical source on the sky. It should be roughly chosen
to match the instrument’s PSF, which, for GALEX, is about 5 arcseconds. We set
it slightly smaller to limit false associations also considering that the
source center is usually much better constrained than the PSF might suggest.
# Updating the observations for GALEX-NUV observations continued...
config["observations"][0].update(
{
"cluster_det": {
"meanshift": {
"bandwidth": 4,
"seeds": None,
"bin_seeding": False,
"min_bin_freq": 1,
"cluster_all": True,
"n_jobs": None,
"max_iter": 300,
"table_name": "tt_detections",
},
},
},
)
Pipeline flow¶
According to the configuration above, we can finally run the analysis. VASCA
implements parallel processing (multiprocessing.pool.Pool.starmap) for this part of the pipeline
by applying the run_field method in parallel for each field.
First, the parameters for run_field are collected.
import vasca.utils as vutils
# Collect parameters from config
fd_pars: list[list[int, str, Region, dict]] = []
vobs: list[dict] = config["observations"]
obs_nr: int
field_nr: str
# Loop over observation list index
for obs_nr, _ in enumerate(vobs):
# Loop over fields
for field_nr in vobs[obs_nr]["obs_field_ids"]:
# Construct VASCA field ID (prepending instrument/filter identifier)
iprefix: str = vutils.dd_obs_id_add[
vobs[obs_nr]["observatory"] + vobs[obs_nr]["obs_filter"]
]
field_id: str = f"{iprefix}{field_nr}"
# Save parameters outside loop
fd_pars.append([obs_nr, field_id, rg, config])
Second, all fields are processed in parallel.
from multiprocessing.pool import Pool
import vasca.vasca_pipe as vpipe
# Run each field in a separate process in parallel
nr_cpus = config["general"]["nr_cpus"]
logger.info(f"Analyzing {len(fd_pars)} fields on {nr_cpus} parallel threads.")
with Pool(processes=nr_cpus) as pool:
pool_return = pool.starmap(vpipe.run_field_docs, fd_pars)
pool.join()
logger.info("Done analyzing individual fields.")
Show code cell output
2024-11-11 08:58:17.1006 __main__:6 |INFO: Analyzing 3 fields on 3 parallel threads.
2024-11-11 08:58:17.1564 vasca.vasca_pipe:224 |INFO: Analysing field: GNU2529969795944677376
2024-11-11 08:58:17.1572 vasca.vasca_pipe:224 |INFO: Analysing field: GNU2597664528044916736
2024-11-11 08:58:17.1571 vasca.vasca_pipe:224 |INFO: Analysing field: GNU3880803393752530944
2024-11-11 08:58:17.1613 vasca.field:747 |INFO: Loading field '2529969795944677376' with method 'VASCA' for filter 'NUV' and load_products 'ALL'.
2024-11-11 08:58:17.1627 vasca.field:747 |INFO: Loading field '3880803393752530944' with method 'VASCA' for filter 'NUV' and load_products 'ALL'.
2024-11-11 08:58:17.1629 vasca.field:747 |INFO: Loading field '2597664528044916736' with method 'VASCA' for filter 'NUV' and load_products 'ALL'.
2024-11-11 08:58:17.1633 vasca.field:533 |DEBUG: Field data path set to: '/home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2529969795944677376'
2024-11-11 08:58:17.1647 vasca.field:534 |DEBUG: Visits data path set to: '/home/runner/work/vasca-mirror/vasca-mirror/vasca/test/resources/GALEX_visits_list.fits'
2024-11-11 08:58:17.1645 vasca.field:533 |DEBUG: Field data path set to: '/home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736'
2024-11-11 08:58:17.1649 vasca.field:533 |DEBUG: Field data path set to: '/home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/3880803393752530944'
2024-11-11 08:58:17.1662 vasca.tables:310 |DEBUG: Loading file with name '/home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2529969795944677376/VASCA_GALEX_2529969795944677376_NUV_field_data.fits'
2024-11-11 08:58:17.1668 vasca.field:534 |DEBUG: Visits data path set to: '/home/runner/work/vasca-mirror/vasca-mirror/vasca/test/resources/GALEX_visits_list.fits'
2024-11-11 08:58:17.1678 vasca.field:534 |DEBUG: Visits data path set to: '/home/runner/work/vasca-mirror/vasca-mirror/vasca/test/resources/GALEX_visits_list.fits'
2024-11-11 08:58:17.1685 vasca.tables:310 |DEBUG: Loading file with name '/home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/3880803393752530944/VASCA_GALEX_3880803393752530944_NUV_field_data.fits'
2024-11-11 08:58:17.1696 vasca.tables:310 |DEBUG: Loading file with name '/home/runner/work/vasca-mirror/vasca-mirror/docs/tutorial_resources/2597664528044916736/VASCA_GALEX_2597664528044916736_NUV_field_data.fits'
2024-11-11 08:58:17.1805 vasca.tables:321 |DEBUG: Loading table 'tt_fields'
2024-11-11 08:58:17.1834 vasca.tables:321 |DEBUG: Loading table 'tt_fields'
2024-11-11 08:58:17.1853 vasca.tables:321 |DEBUG: Loading table 'tt_fields'
2024-11-11 08:58:17.2028 vasca.tables:321 |DEBUG: Loading table 'tt_visits'
2024-11-11 08:58:17.2075 vasca.tables:321 |DEBUG: Loading table 'tt_visits'
2024-11-11 08:58:17.2135 vasca.tables:321 |DEBUG: Loading table 'tt_visits'
2024-11-11 08:58:17.2197 vasca.tables:321 |DEBUG: Loading table 'tt_detections'
2024-11-11 08:58:17.2313 vasca.tables:321 |DEBUG: Loading table 'tt_detections'
2024-11-11 08:58:17.2378 vasca.tables:321 |DEBUG: Loading table 'tt_detections'
2024-11-11 08:58:17.2682 vasca.tables:321 |DEBUG: Loading table 'tt_coadd_detections'
2024-11-11 08:58:17.2761 vasca.tables:321 |DEBUG: Loading table 'tt_coadd_detections'
2024-11-11 08:58:17.3208 vasca.tables:321 |DEBUG: Loading table 'tt_coadd_detections'
2024-11-11 08:58:18.2183 vasca.field:403 |DEBUG: Getting parameter 'field_id': b'GNU3880803393752530944', None, S
2024-11-11 08:58:18.2215 vasca.field:403 |DEBUG: Getting parameter 'field_name': b'MISGCSN2_10493_0117', None, S
2024-11-11 08:58:18.2245 vasca.field:403 |DEBUG: Getting parameter 'ra': 242.64270077057597, deg, f
2024-11-11 08:58:18.2277 vasca.field:403 |DEBUG: Getting parameter 'dec': 53.1235646075049, deg, f
2024-11-11 08:58:18.2307 vasca.field:403 |DEBUG: Getting parameter 'fov_diam': 1.2000000476837158, deg, f
2024-11-11 08:58:18.2335 vasca.field:403 |DEBUG: Getting parameter 'observatory': b'GALEX', None, S
2024-11-11 08:58:18.2363 vasca.field:403 |DEBUG: Getting parameter 'obs_filter': b'NUV', None, S
2024-11-11 08:58:18.2390 vasca.field:403 |DEBUG: Getting parameter 'ra': 242.64270077057597, deg, f
2024-11-11 08:58:18.2417 vasca.field:403 |DEBUG: Getting parameter 'dec': 53.1235646075049, deg, f
2024-11-11 08:58:18.2471 vasca.field:324 |DEBUG: Set attributes: {'tt_fields': ['field_id', 'field_name', 'ra', 'dec', 'fov_diam', 'observatory', 'obs_filter', 'center'], 'tt_visits': ['nr_vis', 'time_bin_size_sum', 'time_start', 'time_stop']}
2024-11-11 08:58:18.2493 vasca.field:403 |DEBUG: Getting parameter 'field_id': b'GNU3880803393752530944', None, S
2024-11-11 08:58:18.2520 vasca.field:403 |DEBUG: Getting parameter 'field_name': b'MISGCSN2_10493_0117', None, S
2024-11-11 08:58:18.2548 vasca.field:403 |DEBUG: Getting parameter 'ra': 242.64270077057597, deg, f
2024-11-11 08:58:18.2576 vasca.field:403 |DEBUG: Getting parameter 'dec': 53.1235646075049, deg, f
2024-11-11 08:58:18.2603 vasca.field:403 |DEBUG: Getting parameter 'fov_diam': 1.2000000476837158, deg, f
2024-11-11 08:58:18.2631 vasca.field:403 |DEBUG: Getting parameter 'observatory': b'GALEX', None, S
2024-11-11 08:58:18.2646 vasca.field:403 |DEBUG: Getting parameter 'obs_filter': b'NUV', None, S
2024-11-11 08:58:18.2674 vasca.field:403 |DEBUG: Getting parameter 'ra': 242.64270077057597, deg, f
2024-11-11 08:58:18.2697 vasca.field:403 |DEBUG: Getting parameter 'dec': 53.1235646075049, deg, f
2024-11-11 08:58:18.2729 vasca.field:324 |DEBUG: Set attributes: {'tt_fields': ['field_id', 'field_name', 'ra', 'dec', 'fov_diam', 'observatory', 'obs_filter', 'center'], 'tt_visits': ['nr_vis', 'time_bin_size_sum', 'time_start', 'time_stop']}
2024-11-11 08:58:18.2753 vasca.tables:417 |INFO: Applying selection on table 'tt_detections'
2024-11-11 08:58:18.2777 vasca.tables:451 |DEBUG: AND selecting 's2n' [3.0, inf], kept: 67.5798%
2024-11-11 08:58:18.2799 vasca.tables:451 |DEBUG: AND selecting 'r_fov' [0.0, 0.5], kept: 45.6911%
2024-11-11 08:58:18.2825 vasca.tables:451 |DEBUG: AND selecting 'ellip_world' [0.0, 0.5], kept: 37.9818%
2024-11-11 08:58:18.2842 vasca.tables:451 |DEBUG: AND selecting 'size_world' [0.0, 6.0], kept: 37.8166%
2024-11-11 08:58:18.2869 vasca.tables:451 |DEBUG: AND selecting 'class_star' [0.15, 1.0], kept: 36.5501%
2024-11-11 08:58:18.2897 vasca.tables:451 |DEBUG: AND selecting 'chkobj_type' [-0.5, 0.5], kept: 35.4626%
2024-11-11 08:58:18.2917 vasca.tables:451 |DEBUG: AND selecting 'flux_app_ratio' [0.3, 1.05], kept: 33.4389%
2024-11-11 08:58:18.2940 vasca.tables:468 |DEBUG: AND selecting bitmask 'artifacts' removing [2, 4, 8, 128, 256], kept: 31.1123%
2024-11-11 08:58:18.2964 vasca.tables:500 |INFO: Total table entries 7264, pre-selected rows 7264, rows after new selection 2260
2024-11-11 08:58:18.2987 vasca.tables:512 |INFO: Setting range for 'pos_err' to '[0.5, 5]'
2024-11-11 08:58:18.3020 vasca.tables:417 |INFO: Applying selection on table 'tt_detections'
2024-11-11 08:58:18.3044 vasca.tables:451 |DEBUG: AND selecting 's2n' [5.0, inf], kept: 23.5683%
2024-11-11 08:58:18.3066 vasca.tables:451 |DEBUG: AND selecting 'r_fov' [0.0, 0.5], kept: 15.4460%
2024-11-11 08:58:18.3088 vasca.tables:451 |DEBUG: AND selecting 'ellip_world' [0.0, 0.5], kept: 14.3172%
2024-11-11 08:58:18.3112 vasca.tables:451 |DEBUG: AND selecting 'size_world' [0.0, 6.0], kept: 14.1520%
2024-11-11 08:58:18.3132 vasca.tables:451 |DEBUG: AND selecting 'class_star' [0.15, 1.0], kept: 12.9130%
2024-11-11 08:58:18.3157 vasca.tables:451 |DEBUG: AND selecting 'chkobj_type' [-0.5, 0.5], kept: 11.8254%
2024-11-11 08:58:18.3180 vasca.tables:451 |DEBUG: AND selecting 'flux_app_ratio' [0.3, 1.05], kept: 11.8254%
2024-11-11 08:58:18.3202 vasca.tables:468 |DEBUG: AND selecting bitmask 'artifacts' removing [2, 4, 8, 128, 256], kept: 11.0683%
2024-11-11 08:58:18.3224 vasca.tables:500 |INFO: Total table entries 7264, pre-selected rows 2260, rows after new selection 804
2024-11-11 08:58:18.3238 vasca.vasca_pipe:257 |INFO: Clustering field detections (GNU3880803393752530944)
2024-11-11 08:58:18.3269 vasca.tables:650 |INFO: Clustering with MeanShift with table: tt_detections
2024-11-11 08:58:18.3310 vasca.tables:667 |DEBUG: Clustering 804 detections/sources
2024-11-11 08:58:18.3324 vasca.tables:668 |DEBUG: MeanShift with parameters (bandwith in degrees): '{'bandwidth': 0.0011111111111111111, 'seeds': None, 'bin_seeding': False, 'min_bin_freq': 1, 'cluster_all': True, 'n_jobs': None, 'max_iter': 300}'
2024-11-11 08:58:19.0562 vasca.tables:675 |DEBUG: Done with clustering, found 804 clusters
2024-11-11 08:58:19.0592 vasca.tables:156 |DEBUG: Adding table 'base_field:tt_sources'
2024-11-11 08:58:19.0672 vasca.field:250 |DEBUG: Scanning for doubled visit detections
2024-11-11 08:58:19.4787 vasca.tables:417 |INFO: Applying selection on table 'tt_detections'
2024-11-11 08:58:19.4812 vasca.tables:500 |INFO: Total table entries 7264, pre-selected rows 804, rows after new selection 804
2024-11-11 08:58:19.4837 vasca.tables:211 |INFO: Writing file with name '/opt/hostedtoolcache/Python/3.11.10/x64/lib/python3.11/site-packages/docs/tutorial_resources/vasca_pipeline/simple_pipe/fields/field_GNU3880803393752530944.fits'
2024-11-11 08:58:19.4865 vasca.tables:222 |DEBUG: Storing image data of shape (3840, 3840)
2024-11-11 08:58:19.5002 vasca.tables:236 |DEBUG: Storing image data of shape (3840, 3840)
2024-11-11 08:58:22.2509 vasca.tables:251 |DEBUG: Writing table 'tt_fields'
2024-11-11 08:58:22.2700 vasca.tables:251 |DEBUG: Writing table 'tt_visits'
2024-11-11 08:58:22.2907 vasca.tables:251 |DEBUG: Writing table 'tt_detections'
2024-11-11 08:58:22.3347 vasca.tables:251 |DEBUG: Writing table 'tt_coadd_detections'
2024-11-11 08:58:22.3863 vasca.tables:251 |DEBUG: Writing table 'tt_sources'
2024-11-11 08:58:22.4955 vasca.tables:183 |DEBUG: Removing unselected rows in table 'tt_detections'
2024-11-11 08:58:22.5004 vasca.tables:183 |DEBUG: Removing unselected rows in table 'tt_coadd_detections'
2024-11-11 08:58:22.5091 vasca.vasca_pipe:274 |INFO: Done with field: GNU3880803393752530944
2024-11-11 08:58:22.5458 vasca.field:403 |DEBUG: Getting parameter 'field_id': b'GNU2529969795944677376', None, S
2024-11-11 08:58:22.5472 vasca.field:403 |DEBUG: Getting parameter 'field_name': b'ELAISN1_09', None, S
2024-11-11 08:58:22.5486 vasca.field:403 |DEBUG: Getting parameter 'ra': 241.60667709058498, deg, f
2024-11-11 08:58:22.5500 vasca.field:403 |DEBUG: Getting parameter 'dec': 53.531821052495, deg, f
2024-11-11 08:58:22.5513 vasca.field:403 |DEBUG: Getting parameter 'fov_diam': 1.2000000476837158, deg, f
2024-11-11 08:58:22.5528 vasca.field:403 |DEBUG: Getting parameter 'observatory': b'GALEX', None, S
2024-11-11 08:58:22.5538 vasca.field:403 |DEBUG: Getting parameter 'obs_filter': b'NUV', None, S
2024-11-11 08:58:22.5552 vasca.field:403 |DEBUG: Getting parameter 'ra': 241.60667709058498, deg, f
2024-11-11 08:58:22.5569 vasca.field:403 |DEBUG: Getting parameter 'dec': 53.531821052495, deg, f
2024-11-11 08:58:22.5607 vasca.field:324 |DEBUG: Set attributes: {'tt_fields': ['field_id', 'field_name', 'ra', 'dec', 'fov_diam', 'observatory', 'obs_filter', 'center'], 'tt_visits': ['nr_vis', 'time_bin_size_sum', 'time_start', 'time_stop']}
2024-11-11 08:58:22.5621 vasca.field:403 |DEBUG: Getting parameter 'field_id': b'GNU2529969795944677376', None, S
2024-11-11 08:58:22.5633 vasca.field:403 |DEBUG: Getting parameter 'field_name': b'ELAISN1_09', None, S
2024-11-11 08:58:22.5648 vasca.field:403 |DEBUG: Getting parameter 'ra': 241.60667709058498, deg, f
2024-11-11 08:58:22.5661 vasca.field:403 |DEBUG: Getting parameter 'dec': 53.531821052495, deg, f
2024-11-11 08:58:22.5673 vasca.field:403 |DEBUG: Getting parameter 'fov_diam': 1.2000000476837158, deg, f
2024-11-11 08:58:22.5687 vasca.field:403 |DEBUG: Getting parameter 'observatory': b'GALEX', None, S
2024-11-11 08:58:22.5700 vasca.field:403 |DEBUG: Getting parameter 'obs_filter': b'NUV', None, S
2024-11-11 08:58:22.5713 vasca.field:403 |DEBUG: Getting parameter 'ra': 241.60667709058498, deg, f
2024-11-11 08:58:22.5727 vasca.field:403 |DEBUG: Getting parameter 'dec': 53.531821052495, deg, f
2024-11-11 08:58:22.5750 vasca.field:324 |DEBUG: Set attributes: {'tt_fields': ['field_id', 'field_name', 'ra', 'dec', 'fov_diam', 'observatory', 'obs_filter', 'center'], 'tt_visits': ['nr_vis', 'time_bin_size_sum', 'time_start', 'time_stop']}
2024-11-11 08:58:22.5763 vasca.tables:417 |INFO: Applying selection on table 'tt_detections'
2024-11-11 08:58:22.5802 vasca.tables:451 |DEBUG: AND selecting 's2n' [3.0, inf], kept: 49.4597%
2024-11-11 08:58:22.5829 vasca.tables:451 |DEBUG: AND selecting 'r_fov' [0.0, 0.5], kept: 33.9603%
2024-11-11 08:58:22.5854 vasca.tables:451 |DEBUG: AND selecting 'ellip_world' [0.0, 0.5], kept: 29.8416%
2024-11-11 08:58:22.5880 vasca.tables:451 |DEBUG: AND selecting 'size_world' [0.0, 6.0], kept: 29.6152%
2024-11-11 08:58:22.5901 vasca.tables:451 |DEBUG: AND selecting 'class_star' [0.15, 1.0], kept: 26.4344%
2024-11-11 08:58:22.5925 vasca.tables:451 |DEBUG: AND selecting 'chkobj_type' [-0.5, 0.5], kept: 25.8593%
2024-11-11 08:58:22.5947 vasca.tables:451 |DEBUG: AND selecting 'flux_app_ratio' [0.3, 1.05], kept: 25.6896%
2024-11-11 08:58:22.5967 vasca.tables:468 |DEBUG: AND selecting bitmask 'artifacts' removing [2, 4, 8, 128, 256], kept: 24.3663%
2024-11-11 08:58:22.5980 vasca.tables:500 |INFO: Total table entries 170929, pre-selected rows 170929, rows after new selection 41649
2024-11-11 08:58:22.5993 vasca.tables:512 |INFO: Setting range for 'pos_err' to '[0.5, 5]'
2024-11-11 08:58:22.6070 vasca.tables:417 |INFO: Applying selection on table 'tt_detections'
2024-11-11 08:58:22.6098 vasca.tables:451 |DEBUG: AND selecting 's2n' [5.0, inf], kept: 22.2244%
2024-11-11 08:58:22.6121 vasca.tables:451 |DEBUG: AND selecting 'r_fov' [0.0, 0.5], kept: 15.2549%
2024-11-11 08:58:22.6145 vasca.tables:451 |DEBUG: AND selecting 'ellip_world' [0.0, 0.5], kept: 14.1556%
2024-11-11 08:58:22.6168 vasca.tables:451 |DEBUG: AND selecting 'size_world' [0.0, 6.0], kept: 13.9356%
2024-11-11 08:58:22.6193 vasca.tables:451 |DEBUG: AND selecting 'class_star' [0.15, 1.0], kept: 11.5100%
2024-11-11 08:58:22.6218 vasca.tables:451 |DEBUG: AND selecting 'chkobj_type' [-0.5, 0.5], kept: 10.9349%
2024-11-11 08:58:22.6243 vasca.tables:451 |DEBUG: AND selecting 'flux_app_ratio' [0.3, 1.05], kept: 10.9338%
2024-11-11 08:58:22.6267 vasca.tables:468 |DEBUG: AND selecting bitmask 'artifacts' removing [2, 4, 8, 128, 256], kept: 10.3803%
2024-11-11 08:58:22.6280 vasca.tables:500 |INFO: Total table entries 170929, pre-selected rows 41649, rows after new selection 17743
2024-11-11 08:58:22.6294 vasca.vasca_pipe:257 |INFO: Clustering field detections (GNU2529969795944677376)
2024-11-11 08:58:22.6307 vasca.tables:650 |INFO: Clustering with MeanShift with table: tt_detections
2024-11-11 08:58:22.6464 vasca.tables:667 |DEBUG: Clustering 17743 detections/sources
2024-11-11 08:58:22.6478 vasca.tables:668 |DEBUG: MeanShift with parameters (bandwith in degrees): '{'bandwidth': 0.0011111111111111111, 'seeds': None, 'bin_seeding': False, 'min_bin_freq': 1, 'cluster_all': True, 'n_jobs': None, 'max_iter': 300}'
2024-11-11 08:58:25.1350 vasca.field:403 |DEBUG: Getting parameter 'field_id': b'GNU2597664528044916736', None, S
2024-11-11 08:58:25.1365 vasca.field:403 |DEBUG: Getting parameter 'field_name': b'PS_ELAISN1_MOS15', None, S
2024-11-11 08:58:25.1381 vasca.field:403 |DEBUG: Getting parameter 'ra': 242.396514585217, deg, f
2024-11-11 08:58:25.1396 vasca.field:403 |DEBUG: Getting parameter 'dec': 53.9999872930018, deg, f
2024-11-11 08:58:25.1409 vasca.field:403 |DEBUG: Getting parameter 'fov_diam': 1.2000000476837158, deg, f
2024-11-11 08:58:25.1420 vasca.field:403 |DEBUG: Getting parameter 'observatory': b'GALEX', None, S
2024-11-11 08:58:25.1430 vasca.field:403 |DEBUG: Getting parameter 'obs_filter': b'NUV', None, S
2024-11-11 08:58:25.1442 vasca.field:403 |DEBUG: Getting parameter 'ra': 242.396514585217, deg, f
2024-11-11 08:58:25.1458 vasca.field:403 |DEBUG: Getting parameter 'dec': 53.9999872930018, deg, f
2024-11-11 08:58:25.1492 vasca.field:324 |DEBUG: Set attributes: {'tt_fields': ['field_id', 'field_name', 'ra', 'dec', 'fov_diam', 'observatory', 'obs_filter', 'center'], 'tt_visits': ['nr_vis', 'time_bin_size_sum', 'time_start', 'time_stop']}
2024-11-11 08:58:25.1502 vasca.field:403 |DEBUG: Getting parameter 'field_id': b'GNU2597664528044916736', None, S
2024-11-11 08:58:25.1516 vasca.field:403 |DEBUG: Getting parameter 'field_name': b'PS_ELAISN1_MOS15', None, S
2024-11-11 08:58:25.1526 vasca.field:403 |DEBUG: Getting parameter 'ra': 242.396514585217, deg, f
2024-11-11 08:58:25.1539 vasca.field:403 |DEBUG: Getting parameter 'dec': 53.9999872930018, deg, f
2024-11-11 08:58:25.1550 vasca.field:403 |DEBUG: Getting parameter 'fov_diam': 1.2000000476837158, deg, f
2024-11-11 08:58:25.1560 vasca.field:403 |DEBUG: Getting parameter 'observatory': b'GALEX', None, S
2024-11-11 08:58:25.1573 vasca.field:403 |DEBUG: Getting parameter 'obs_filter': b'NUV', None, S
2024-11-11 08:58:25.1587 vasca.field:403 |DEBUG: Getting parameter 'ra': 242.396514585217, deg, f
2024-11-11 08:58:25.1600 vasca.field:403 |DEBUG: Getting parameter 'dec': 53.9999872930018, deg, f
2024-11-11 08:58:25.1620 vasca.field:324 |DEBUG: Set attributes: {'tt_fields': ['field_id', 'field_name', 'ra', 'dec', 'fov_diam', 'observatory', 'obs_filter', 'center'], 'tt_visits': ['nr_vis', 'time_bin_size_sum', 'time_start', 'time_stop']}
2024-11-11 08:58:25.1631 vasca.tables:417 |INFO: Applying selection on table 'tt_detections'
2024-11-11 08:58:25.1672 vasca.tables:451 |DEBUG: AND selecting 's2n' [3.0, inf], kept: 51.0152%
2024-11-11 08:58:25.1702 vasca.tables:451 |DEBUG: AND selecting 'r_fov' [0.0, 0.5], kept: 36.1159%
2024-11-11 08:58:25.1729 vasca.tables:451 |DEBUG: AND selecting 'ellip_world' [0.0, 0.5], kept: 31.5473%
2024-11-11 08:58:25.1758 vasca.tables:451 |DEBUG: AND selecting 'size_world' [0.0, 6.0], kept: 31.1979%
2024-11-11 08:58:25.1785 vasca.tables:451 |DEBUG: AND selecting 'class_star' [0.15, 1.0], kept: 27.4451%
2024-11-11 08:58:25.1815 vasca.tables:451 |DEBUG: AND selecting 'chkobj_type' [-0.5, 0.5], kept: 26.8908%
2024-11-11 08:58:25.1842 vasca.tables:451 |DEBUG: AND selecting 'flux_app_ratio' [0.3, 1.05], kept: 26.7283%
2024-11-11 08:58:25.1868 vasca.tables:468 |DEBUG: AND selecting bitmask 'artifacts' removing [2, 4, 8, 128, 256], kept: 25.5369%
2024-11-11 08:58:25.1881 vasca.tables:500 |INFO: Total table entries 228387, pre-selected rows 228387, rows after new selection 58323
2024-11-11 08:58:25.1902 vasca.tables:512 |INFO: Setting range for 'pos_err' to '[0.5, 5]'
2024-11-11 08:58:25.1988 vasca.tables:417 |INFO: Applying selection on table 'tt_detections'
2024-11-11 08:58:25.2021 vasca.tables:451 |DEBUG: AND selecting 's2n' [5.0, inf], kept: 22.6051%
2024-11-11 08:58:25.2049 vasca.tables:451 |DEBUG: AND selecting 'r_fov' [0.0, 0.5], kept: 16.0994%
2024-11-11 08:58:25.2076 vasca.tables:451 |DEBUG: AND selecting 'ellip_world' [0.0, 0.5], kept: 14.8756%
2024-11-11 08:58:25.2103 vasca.tables:451 |DEBUG: AND selecting 'size_world' [0.0, 6.0], kept: 14.5376%
2024-11-11 08:58:25.2130 vasca.tables:451 |DEBUG: AND selecting 'class_star' [0.15, 1.0], kept: 11.6535%
2024-11-11 08:58:25.2158 vasca.tables:451 |DEBUG: AND selecting 'chkobj_type' [-0.5, 0.5], kept: 11.0991%
2024-11-11 08:58:25.2185 vasca.tables:451 |DEBUG: AND selecting 'flux_app_ratio' [0.3, 1.05], kept: 11.0974%
2024-11-11 08:58:25.2210 vasca.tables:468 |DEBUG: AND selecting bitmask 'artifacts' removing [2, 4, 8, 128, 256], kept: 10.6136%
2024-11-11 08:58:25.2223 vasca.tables:500 |INFO: Total table entries 228387, pre-selected rows 58323, rows after new selection 24240
2024-11-11 08:58:25.2238 vasca.vasca_pipe:257 |INFO: Clustering field detections (GNU2597664528044916736)
2024-11-11 08:58:25.2254 vasca.tables:650 |INFO: Clustering with MeanShift with table: tt_detections
2024-11-11 08:58:25.2475 vasca.tables:667 |DEBUG: Clustering 24240 detections/sources
2024-11-11 08:58:25.2491 vasca.tables:668 |DEBUG: MeanShift with parameters (bandwith in degrees): '{'bandwidth': 0.0011111111111111111, 'seeds': None, 'bin_seeding': False, 'min_bin_freq': 1, 'cluster_all': True, 'n_jobs': None, 'max_iter': 300}'
2024-11-11 08:58:34.0920 vasca.tables:675 |DEBUG: Done with clustering, found 2318 clusters
2024-11-11 08:58:34.0937 vasca.tables:156 |DEBUG: Adding table 'base_field:tt_sources'
2024-11-11 08:58:34.1032 vasca.field:250 |DEBUG: Scanning for doubled visit detections
2024-11-11 08:58:38.4655 vasca.tables:417 |INFO: Applying selection on table 'tt_detections'
2024-11-11 08:58:38.4712 vasca.tables:500 |INFO: Total table entries 170929, pre-selected rows 17743, rows after new selection 17743
2024-11-11 08:58:38.4733 vasca.tables:211 |INFO: Writing file with name '/opt/hostedtoolcache/Python/3.11.10/x64/lib/python3.11/site-packages/docs/tutorial_resources/vasca_pipeline/simple_pipe/fields/field_GNU2529969795944677376.fits'
2024-11-11 08:58:38.4748 vasca.tables:222 |DEBUG: Storing image data of shape (3840, 3840)
2024-11-11 08:58:38.4874 vasca.tables:236 |DEBUG: Storing image data of shape (16, 3840, 3840)
2024-11-11 08:58:41.1826 vasca.tables:675 |DEBUG: Done with clustering, found 2475 clusters
2024-11-11 08:58:41.1840 vasca.tables:156 |DEBUG: Adding table 'base_field:tt_sources'
2024-11-11 08:58:41.1928 vasca.field:250 |DEBUG: Scanning for doubled visit detections
2024-11-11 08:58:47.0405 vasca.tables:417 |INFO: Applying selection on table 'tt_detections'
2024-11-11 08:58:47.0454 vasca.tables:500 |INFO: Total table entries 228387, pre-selected rows 24240, rows after new selection 24240
2024-11-11 08:58:47.0470 vasca.tables:211 |INFO: Writing file with name '/opt/hostedtoolcache/Python/3.11.10/x64/lib/python3.11/site-packages/docs/tutorial_resources/vasca_pipeline/simple_pipe/fields/field_GNU2597664528044916736.fits'
2024-11-11 08:58:47.0484 vasca.tables:222 |DEBUG: Storing image data of shape (3840, 3840)
2024-11-11 08:58:47.0567 vasca.tables:236 |DEBUG: Storing image data of shape (22, 3840, 3840)
2024-11-11 08:58:54.2381 vasca.tables:251 |DEBUG: Writing table 'tt_fields'
2024-11-11 08:58:54.2524 vasca.tables:251 |DEBUG: Writing table 'tt_visits'
2024-11-11 08:58:54.2637 vasca.tables:251 |DEBUG: Writing table 'tt_detections'
2024-11-11 08:58:54.3776 vasca.tables:251 |DEBUG: Writing table 'tt_coadd_detections'
2024-11-11 08:58:54.4118 vasca.tables:251 |DEBUG: Writing table 'tt_sources'
2024-11-11 08:58:54.4992 vasca.tables:183 |DEBUG: Removing unselected rows in table 'tt_detections'
2024-11-11 08:58:54.5153 vasca.tables:183 |DEBUG: Removing unselected rows in table 'tt_coadd_detections'
2024-11-11 08:58:54.5283 vasca.vasca_pipe:274 |INFO: Done with field: GNU2529969795944677376
2024-11-11 08:59:08.1619 vasca.tables:251 |DEBUG: Writing table 'tt_fields'
2024-11-11 08:59:08.1761 vasca.tables:251 |DEBUG: Writing table 'tt_visits'
2024-11-11 08:59:08.1872 vasca.tables:251 |DEBUG: Writing table 'tt_detections'
2024-11-11 08:59:08.3575 vasca.tables:251 |DEBUG: Writing table 'tt_coadd_detections'
2024-11-11 08:59:08.3907 vasca.tables:251 |DEBUG: Writing table 'tt_sources'
2024-11-11 08:59:08.4828 vasca.tables:183 |DEBUG: Removing unselected rows in table 'tt_detections'
2024-11-11 08:59:08.4967 vasca.tables:183 |DEBUG: Removing unselected rows in table 'tt_coadd_detections'
2024-11-11 08:59:08.5069 vasca.vasca_pipe:274 |INFO: Done with field: GNU2597664528044916736
2024-11-11 08:59:08.5677 __main__:12 |INFO: Done analyzing individual fields.
Finally, the pool results are unpacked and the region object is updated with processed field information.
A memory-saving procedure is used where first all fields are brought to the
scope of the region object by filling the Region.field dictionary from
which the field-level data is taken and stacked in respective region-owned
tables using the Region.add_table_from_fields method. After this step
all fields are discarded from the scope, to be deleted from the garbage collector.
Hint
In VASCA a Region object keeps track of its fields in the Region.tt_fields
table. At any time one can load field data of a specific field using Region.get_field.
from astropy.table import unique
# Loop over processed fields
for field in pool_return:
# Populate region field dictionary with field data
rg.fields[field.field_id] = field
logger.info(f"Added field {field.field_id} from pool to region")
# Add field tables to region
# Visits metadata
rg.add_table_from_fields("tt_visits")
rg.tt_visits = unique(rg.tt_visits, keys="vis_id")
# Clustered sources
rg.add_table_from_fields("tt_sources")
# Visit-level detections
rg.add_table_from_fields("tt_detections", only_selected=False)
# Field-averaged detectsion
rg.add_table_from_fields("tt_coadd_detections")
# Discard fields. All that was needed has been transferred to region tables
del rg.fields
Show code cell output
2024-11-11 08:59:08.5749 __main__:7 |INFO: Added field GNU3880803393752530944 from pool to region
2024-11-11 08:59:08.5761 __main__:7 |INFO: Added field GNU2529969795944677376 from pool to region
2024-11-11 08:59:08.5767 __main__:7 |INFO: Added field GNU2597664528044916736 from pool to region
2024-11-11 08:59:08.5778 vasca.region:152 |DEBUG: Adding table from fields: tt_visits
2024-11-11 08:59:08.5800 vasca.tables:156 |DEBUG: Adding table 'region:tt_visits'
2024-11-11 08:59:08.5832 vasca.region:152 |DEBUG: Adding table from fields: tt_sources
2024-11-11 08:59:08.5957 vasca.tables:156 |DEBUG: Adding table 'region:tt_sources'
2024-11-11 08:59:08.6042 vasca.region:152 |DEBUG: Adding table from fields: tt_detections
2024-11-11 08:59:08.6288 vasca.tables:156 |DEBUG: Adding table 'region:tt_detections'
2024-11-11 08:59:08.6634 vasca.region:152 |DEBUG: Adding table from fields: tt_coadd_detections
2024-11-11 08:59:08.6783 vasca.tables:156 |DEBUG: Adding table 'region:tt_coadd_detections'
Region-level analysis¶
In the final stage of the pipeline, all region-level analysis steps are performed. This stage encompasses three key tasks: first, managing sources located in overlapping sky regions; second, evaluating statistics for use in variability detection; and finally, preparing the pipeline results for writing to disk and generating the VASCA variable source catalog.
Overlapping fields¶
VASCA merges field-level sources in overlapping sky regions in a second clustering step, where the same mean-shift algorithm is used but with a dedicated configuration.
In case field-averaged (co-added) data exists, the field-level detections are merged in the same way, again, with a separate configuration.
config["cluster_src"] = {
"meanshift": {
"bandwidth": 4,
"seeds": None,
"bin_seeding": False,
"min_bin_freq": 1,
"cluster_all": True,
"n_jobs": 1,
"max_iter": 300,
"table_name": "tt_sources",
}
}
config["cluster_coadd_dets"] = {
"meanshift": {
"bandwidth": 4,
"seeds": None,
"bin_seeding": False,
"min_bin_freq": 1,
"cluster_all": True,
"n_jobs": 1,
"max_iter": 300,
"table_name": "tt_coadd_detections",
}
}
# Cluster field sources and co-adds in parallel
ll_cluster = [
[
config["cluster_src"]["meanshift"],
rg.tt_sources,
rg.tt_detections,
False,
],
[
config["cluster_coadd_dets"]["meanshift"],
rg.tt_coadd_detections,
False,
True,
],
]
with Pool(processes=2) as pool:
pool_return = pool.starmap(vpipe.run_cluster_fields, ll_cluster)
pool.join()
# Copy parallelized results into region
for pool_rg in pool_return:
if "tt_sources" in pool_rg._table_names:
rg.tt_sources = pool_rg.tt_sources
rg.tt_detections = pool_rg.tt_detections
else:
rg.add_table(pool_rg.tt_coadd_sources, "region:tt_coadd_sources")
rg.tt_coadd_detections = pool_rg.tt_coadd_detections
Show code cell output
2024-11-11 08:59:08.7855 vasca.tables:156 |DEBUG: Adding table 'tt_sources'
2024-11-11 08:59:08.7903 vasca.tables:156 |DEBUG: Adding table 'tt_detections'
2024-11-11 08:59:08.7928 vasca.tables:650 |INFO: Clustering with MeanShift with table: tt_sources
2024-11-11 08:59:08.7964 vasca.tables:667 |DEBUG: Clustering 5597 detections/sources
2024-11-11 08:59:08.7975 vasca.tables:668 |DEBUG: MeanShift with parameters (bandwith in degrees): '{'bandwidth': 0.0011111111111111111, 'seeds': None, 'bin_seeding': False, 'min_bin_freq': 1, 'cluster_all': True, 'n_jobs': 1, 'max_iter': 300}'
2024-11-11 08:59:08.8046 vasca.tables:156 |DEBUG: Adding table 'tt_coadd_detections'
2024-11-11 08:59:08.8085 vasca.tables:650 |INFO: Clustering with MeanShift with table: tt_coadd_detections
2024-11-11 08:59:08.8159 vasca.tables:667 |DEBUG: Clustering 54330 detections/sources
2024-11-11 08:59:08.8174 vasca.tables:668 |DEBUG: MeanShift with parameters (bandwith in degrees): '{'bandwidth': 0.0011111111111111111, 'seeds': None, 'bin_seeding': False, 'min_bin_freq': 1, 'cluster_all': True, 'n_jobs': 1, 'max_iter': 300}'
2024-11-11 08:59:12.1820 vasca.tables:675 |DEBUG: Done with clustering, found 4928 clusters
2024-11-11 08:59:13.6075 vasca.tables:156 |DEBUG: Adding table 'region:tt_sources'
2024-11-11 08:59:13.7756 vasca.tables:719 |DEBUG: Merged sources: 669 (11.9528%)
2024-11-11 08:59:43.6652 vasca.tables:675 |DEBUG: Done with clustering, found 45315 clusters
2024-11-11 08:59:46.9240 vasca.tables:156 |DEBUG: Adding table 'region:tt_coadd_sources'
2024-11-11 08:59:47.0112 vasca.tables:156 |DEBUG: Adding table 'region:tt_coadd_sources'
Source statistics¶
The primary statistic used by VASCA to detect variability is the probability of
obtaining a flux with the observed fluctuations under the assumption that the null
hypothesis is true, that is, constant flux (flux_cpval). The default selection is
such that the chance for the observed variability being purely random can be ruled out
at 5-sigma significance.
Additionally the normalized excess variance (flux_nxv) and absolute flux limits
is used to limit a contamination due to potential hidden systematic flux variations.
Similarly a selection on variation of the spatial coordinates (pos_cpval. pos_xv)
is used to reduce the contamination du to false association of visit-levl detections
in the clustering step.
To ensure the statistical correctness only sources with more than three detections are
considers (n_det>3).
Note
The source selection is configured independently for each observational filter. This allows to adapt to potential systematics in a very flexible way.
For a more concrete example on these calculations in VASCA, have a look at the tutorial on Variability Statistics.
config["selection_src"] = {
"src_variability_nuv": {
"table": "tt_sources",
"presel_type": "or",
"sel_type": "and",
"obs_filter": "NUV",
"range": {
"nr_det": [3, np.inf],
"pos_cpval": [0.0001, np.inf],
"pos_xv": [-np.inf, 2],
"flux_cpval": [-0.5, 0.000000573303],
"flux_nxv": [0.001, np.inf],
"flux": [0.144543, 575.43],
},
},
"src_coadd_diff_nuv": {
"table": "tt_sources",
"presel_type": "or",
"sel_type": "and",
"obs_filter": "NUV",
"range": {
"nr_det": [2, np.inf],
"pos_cpval": [0.0001, np.inf],
"pos_xv": [-np.inf, 2],
"coadd_ffactor": [2.0, np.inf],
"coadd_fdiff_s2n": [7, np.inf],
},
},
}
An additional selection may be possible if co-added data is available. In this case the association between VASCA sources and the field-averaged input data can be made. This serves as cross-check since most static sources should be recovered in the clustering step and match the field average.
config["assoc_src_coadd"] = {
"dist_max": 1, # Associate nearest source below this distance in arc_sec OR
"dist_s2n_max": 3, # Associate nearest source with this distance in units of "squared summed position error"
}
import astropy.units as uu
# Calculate source statistics
rg.set_src_stats(src_id_name="rg_src_id")
rg.set_src_stats(src_id_name="coadd_src_id")
# Match sources to coadd sources
rg.cross_match(
dist_max=config["assoc_src_coadd"]["dist_max"] * uu.arcsec,
dist_s2n_max=config["assoc_src_coadd"]["dist_s2n_max"],
)
# Select variable sources, deselect all sources before and
# make sure all tables containting the region source ID
# are syncronized to this selection
rg.tt_sources["sel"] = False
rg.select_from_config(
config["selection_src"]
) # Loops over TableCollection.select_rows()
rg.synch_src_sel(remove_unselected=False)
# Set source ID mapping table
rg.set_src_id_info()
Show code cell output
2024-11-11 08:59:47.0331 vasca.tables:789 |DEBUG: Calculating source statistics.
2024-11-11 08:59:49.3153 vasca.tables:789 |DEBUG: Calculating source statistics.
2024-11-11 09:00:00.4331 vasca.tables:1099 |INFO: Cross matching table tt_sources
2024-11-11 09:00:00.4588 vasca.tables:402 |INFO: Applying source selection 'src_variability_nuv'
2024-11-11 09:00:00.4595 vasca.tables:417 |INFO: Applying selection on table 'tt_sources'
2024-11-11 09:00:00.4608 vasca.tables:451 |DEBUG: AND selecting 'nr_det' [3, inf], kept: 64.1031%
2024-11-11 09:00:00.4616 vasca.tables:451 |DEBUG: AND selecting 'pos_cpval' [0.0001, inf], kept: 59.5170%
2024-11-11 09:00:00.4624 vasca.tables:451 |DEBUG: AND selecting 'pos_xv' [-inf, 2], kept: 59.3750%
2024-11-11 09:00:00.4632 vasca.tables:451 |DEBUG: AND selecting 'flux_cpval' [-0.5, 5.73303e-07], kept: 1.2378%
2024-11-11 09:00:00.4640 vasca.tables:451 |DEBUG: AND selecting 'flux_nxv' [0.001, inf], kept: 1.1364%
2024-11-11 09:00:00.4648 vasca.tables:451 |DEBUG: AND selecting 'flux' [0.144543, 575.43], kept: 1.1364%
2024-11-11 09:00:00.4653 vasca.tables:500 |INFO: Total table entries 4928, pre-selected rows 0, rows after new selection 56
2024-11-11 09:00:00.4659 vasca.tables:402 |INFO: Applying source selection 'src_coadd_diff_nuv'
2024-11-11 09:00:00.4665 vasca.tables:417 |INFO: Applying selection on table 'tt_sources'
2024-11-11 09:00:00.4673 vasca.tables:451 |DEBUG: AND selecting 'nr_det' [2, inf], kept: 71.4692%
2024-11-11 09:00:00.4683 vasca.tables:451 |DEBUG: AND selecting 'pos_cpval' [0.0001, inf], kept: 66.8831%
2024-11-11 09:00:00.4692 vasca.tables:451 |DEBUG: AND selecting 'pos_xv' [-inf, 2], kept: 66.4976%
2024-11-11 09:00:00.4704 vasca.tables:451 |DEBUG: AND selecting 'coadd_ffactor' [2.0, inf], kept: 0.0812%
2024-11-11 09:00:00.4711 vasca.tables:451 |DEBUG: AND selecting 'coadd_fdiff_s2n' [7, inf], kept: 0.0609%
2024-11-11 09:00:00.4716 vasca.tables:500 |INFO: Total table entries 4928, pre-selected rows 56, rows after new selection 59
2024-11-11 09:00:00.4723 vasca.region:805 |DEBUG: Synchronizing selection in table tt_sources
2024-11-11 09:00:00.4738 vasca.region:805 |DEBUG: Synchronizing selection in table tt_detections
2024-11-11 09:00:00.4750 vasca.region:805 |DEBUG: Synchronizing selection in table tt_coadd_sources
2024-11-11 09:00:00.4763 vasca.region:412 |DEBUG: Setting table tt_src_id_map.
2024-11-11 09:00:01.3471 vasca.tables:156 |DEBUG: Adding table 'region:tt_src_id_map'
# View all table attributes that have been added to the region object
rg.info()
Show code cell output
tt_fields:
<Table length=3>
name dtype unit description
----------------- ------- ---- -------------------------------------------------------
field_id bytes32 Field source ID number
field_name bytes32 Field name
project bytes32 Field project, typically survey name
ra float64 deg Sky coordinate Right Ascension (J2000)
dec float64 deg Sky coordinate Declination (J2000)
observatory bytes22 Telescope of the observation (e.g. GALEX)
obs_filter bytes8 Filter of the observation (e.g. NUV)
fov_diam float32 deg Field radius or box size (depending on the observatory)
sel bool Selection of rows for VASCA analysis
nr_vis int32 Total number of visits of the field
time_bin_size_sum float32 s Total exposure time
time_start float64 d Start date and time in MJD
time_stop float64 d End time of last exposure
rg_fd_id int64 Region field ID number
{'DATAPATH': 'None', 'INFO': 'Field information table'}
tt_filters:
<Table length=2>
name dtype description
-------------- ------ ---------------------------------------
obs_filter_id int32 Observation filter ID number
obs_filter bytes8 Filter of the observation (e.g. NUV)
obs_filter_idx int32 Filter index in filter dependent arrays
{'INFO': 'Filters, their IDs and index, the last is specific for this region.'}
tt_visits:
<Table length=39>
name dtype unit description
-------------- ------- ---- -----------------------------------------
vis_id uint64 Visit ID number
time_bin_start float64 d Visit exposure start date and time in MJD
time_bin_size float32 s Visit exposure time in s
sel bool Selection of rows for VASCA analysis
obs_filter_id int32 Observation filter ID number
{'INFO': 'Visit information table'}
tt_sources:
<Table length=4928>
name dtype shape unit description
--------------- ------- ----- --------- -------------------------------------------------------------
fd_src_id int32 Source ID associated to the visit detection
nr_det int32 (1,) Number of detections
ra float64 deg Sky coordinate Right Ascension (J2000)
dec float64 deg Sky coordinate Declination (J2000)
pos_err float32 arcsec Sky coordinate position error
pos_xv float32 arcsec2 Sky position excess variance
pos_var float32 arcsec2 Sky position variance
pos_cpval float32 Sky position quality
pos_rchiq float32 Sky position reduced chisquared of the constant mean
coadd_src_id int64 Co-add source ID number
coadd_dist float32 arcsec Angular distance to associated source
obs_filter_id int32 (1,) Observation filter ID number
sel bool
flux float32 (1,) 1e-06 Jy Flux density
flux_err float32 (1,) 1e-06 Jy Flux density error
flux_nxv float32 (1,) Flux normalised excess variance
flux_var float32 (1,) 1e-12 Jy2 Flux variance
flux_cpval float32 (1,) Probability value for a constant flux from the chisquare test
flux_rchiq float32 (1,) Flux reduced chisquared of the constant mean
coadd_ffactor float32 Source flux divided by flux of the associated co-add source
coadd_fdiff_s2n float32 Signal to noise of the flux difference
rg_fd_id int64 Region field ID number
rg_src_id int32 Region source ID number
nr_fd_srcs int32 Number of field sources
{'INFO': 'Source infomation table', 'CLUSTALG': 'None'}
tt_detections:
<Table length=42787>
name dtype unit description
-------------- ------- -------- ------------------------------------------------------------------------------------------------
vis_id uint64 Visit ID number
fd_src_id int32 Source ID associated to the visit detection
ra float64 deg Sky coordinate Right Ascension (J2000)
dec float64 deg Sky coordinate Declination (J2000)
pos_err float32 arcsec Sky coordinate position error
flux float32 1e-06 Jy Flux density
flux_err float32 1e-06 Jy Flux density error
flux_app_ratio float32 Flux ratio measured at a different apertures (3.8 arcsec / 6 arcsec radius for GALEX)
s2n float32 Flux signal to noise
obs_filter_id int32 Observation filter ID number
sel bool
r_fov float32 deg Distance from center of FOV in degrees
artifacts int64 Logical OR of artifact flags
class_star float32 Point-source probability: 0.0 (resolved), 1.0 (unresolved, mcat file filter_CLASS_STAR variable)
chkobj_type int32 Detection matched to a known star (bright_match=1, mcat file chkobj_type variable)
size_world float32 arcsec Mean RMS of the ellipse size: (major axis + minor axis) / 2
ellip_world float32 Ellipticity of the detection or source
flux_auto float32 Jy Flux from SExtractor auto option
flux_auto_err float32 Jy Flux error from a fixed circular aperture (3.8 arcsec radius for GALEX)
rg_fd_id int64 Region field ID number
rg_src_id int32 Region source ID number
{'INFO': 'Visit detections table', 'Name': 'tt_detections'}
tt_coadd_detections:
<Table length=54330>
name dtype unit description
-------------- ------- -------- -------------------------------------------------------------------------------------
det_id int64 Reference source ID number
ra float64 deg Sky coordinate Right Ascension (J2000)
dec float64 deg Sky coordinate Declination (J2000)
pos_err float32 arcsec Sky coordinate position error
flux float32 1e-06 Jy Flux density
flux_err float32 1e-06 Jy Flux density error
flux_app_ratio float32 Flux ratio measured at a different apertures (3.8 arcsec / 6 arcsec radius for GALEX)
s2n float32 Flux signal to noise
obs_filter_id int32 Observation filter ID number
sel bool Selection of rows for VASCA analysis
rg_fd_id int64 Region field ID number
coadd_src_id int64 Co-add source ID number
{'INFO': 'Reference detections table', 'Name': 'tt_coadd_detections'}
tt_coadd_sources:
<Table length=45315>
name dtype shape unit description
--------------- ------- ----- --------- -------------------------------------------------------------
fd_src_id int32 Source ID associated to the visit detection
nr_det int32 (1,) Number of detections
ra float64 deg Sky coordinate Right Ascension (J2000)
dec float64 deg Sky coordinate Declination (J2000)
pos_err float32 arcsec Sky coordinate position error
pos_xv float32 arcsec2 Sky position excess variance
pos_var float32 arcsec2 Sky position variance
pos_cpval float32 Sky position quality
pos_rchiq float32 Sky position reduced chisquared of the constant mean
coadd_src_id int64 Co-add source ID number
coadd_dist float32 arcsec Angular distance to associated source
obs_filter_id int32 (1,) Observation filter ID number
sel bool
flux float32 (1,) 1e-06 Jy Flux density
flux_err float32 (1,) 1e-06 Jy Flux density error
flux_nxv float32 (1,) Flux normalised excess variance
flux_var float32 (1,) 1e-12 Jy2 Flux variance
flux_cpval float32 (1,) Probability value for a constant flux from the chisquare test
flux_rchiq float32 (1,) Flux reduced chisquared of the constant mean
coadd_ffactor float32 Source flux divided by flux of the associated co-add source
coadd_fdiff_s2n float32 Signal to noise of the flux difference
rg_fd_id int64 Region field ID number
nr_fd_dets int32 Number of field detections
rg_src_id int32 Region source ID number
{'INFO': 'Source infomation table', 'CLUSTALG': 'None'}
tt_src_id_map:
<Table length=5597>
name dtype description
--------- ----- -------------------------------------------
rg_src_id int32 Region source ID number
rg_fd_id int64 Region field ID number
fd_src_id int32 Source ID associated to the visit detection
sel bool Selection of rows for VASCA analysis
{'INFO': 'Map between region and field source IDs'}
Show code cell source
# View all sources that passed the selection
df_sources = (
vutils.select_obs_filter(rg.tt_sources, obs_filter_id=1)
.to_pandas()
.apply(lambda x: x.str.decode("utf-8") if x.dtype == "O" else x)
)
df_select = df_sources.query("sel")
show(
df_select,
classes="display nowrap compact",
scrollY="300px",
scrollCollapse=True,
paging=False,
columnDefs=[{"className": "dt-body-left", "targets": "_all"}],
)
| fd_src_id | nr_det | ra | dec | pos_err | pos_xv | pos_var | pos_cpval | pos_rchiq | coadd_src_id | coadd_dist | obs_filter_id | sel | flux | flux_err | flux_nxv | flux_var | flux_cpval | flux_rchiq | coadd_ffactor | coadd_fdiff_s2n | rg_fd_id | rg_src_id | nr_fd_srcs | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
Loading ITables v2.2.3 from the init_notebook_mode cell...
(need help?) |
Pipeline Output¶
With a few simple export functions the full region file, the variable source catalog and its pipeline configuration are saved to the pipeline output directory.
This concludes the tutorial. Readers are invited to look into the post-processing notebooks as listed here.
import yaml
# Write region file
rg.write_to_fits(file_name=pipe_dir / f'region_{config["general"]["name"]}.fits')
# Export variable source catalog (only selected sources are included)
rc = rg.get_region_catalog()
rc.write_to_fits(file_name=pipe_dir / f'region_{config["general"]["name"]}_cat.fits')
# Write used config file
yaml_out_name = pipe_dir / f'cfg_ran_{config["general"]["name"]}.yaml'
with open(yaml_out_name, "w") as yaml_file:
yaml.dump(config, yaml_file)
logger.info("Done running VASCA pipeline.")
Show code cell output
2024-11-11 09:00:01.4086 vasca.tables:211 |INFO: Writing file with name '/opt/hostedtoolcache/Python/3.11.10/x64/lib/python3.11/site-packages/docs/tutorial_resources/vasca_pipeline/simple_pipe/region_simple_pipe.fits'
2024-11-11 09:00:01.4113 vasca.tables:251 |DEBUG: Writing table 'tt_fields'
2024-11-11 09:00:01.4325 vasca.tables:251 |DEBUG: Writing table 'tt_filters'
2024-11-11 09:00:01.4393 vasca.tables:251 |DEBUG: Writing table 'tt_visits'
2024-11-11 09:00:01.4487 vasca.tables:251 |DEBUG: Writing table 'tt_sources'
2024-11-11 09:00:01.4872 vasca.tables:251 |DEBUG: Writing table 'tt_detections'
2024-11-11 09:00:01.5341 vasca.tables:251 |DEBUG: Writing table 'tt_coadd_detections'
2024-11-11 09:00:01.5636 vasca.tables:251 |DEBUG: Writing table 'tt_coadd_sources'
2024-11-11 09:00:01.6206 vasca.tables:251 |DEBUG: Writing table 'tt_src_id_map'
2024-11-11 09:00:01.6971 vasca.region:805 |DEBUG: Synchronizing selection in table tt_sources
2024-11-11 09:00:01.6983 vasca.region:805 |DEBUG: Synchronizing selection in table tt_detections
2024-11-11 09:00:01.6995 vasca.region:805 |DEBUG: Synchronizing selection in table tt_coadd_sources
2024-11-11 09:00:01.7008 vasca.region:805 |DEBUG: Synchronizing selection in table tt_src_id_map
2024-11-11 09:00:01.7029 vasca.tables:156 |DEBUG: Adding table 'tt_fields'
2024-11-11 09:00:01.7044 vasca.tables:156 |DEBUG: Adding table 'tt_filters'
2024-11-11 09:00:01.7056 vasca.tables:156 |DEBUG: Adding table 'tt_visits'
2024-11-11 09:00:01.7076 vasca.tables:156 |DEBUG: Adding table 'tt_sources'
2024-11-11 09:00:01.7108 vasca.tables:156 |DEBUG: Adding table 'tt_detections'
2024-11-11 09:00:01.7666 vasca.tables:156 |DEBUG: Adding table 'tt_coadd_detections'
2024-11-11 09:00:01.7696 vasca.tables:156 |DEBUG: Adding table 'tt_coadd_sources'
2024-11-11 09:00:01.7713 vasca.tables:156 |DEBUG: Adding table 'tt_src_id_map'
2024-11-11 09:00:01.7726 vasca.tables:211 |INFO: Writing file with name '/opt/hostedtoolcache/Python/3.11.10/x64/lib/python3.11/site-packages/docs/tutorial_resources/vasca_pipeline/simple_pipe/region_simple_pipe_cat.fits'
2024-11-11 09:00:01.7743 vasca.tables:251 |DEBUG: Writing table 'tt_fields'
2024-11-11 09:00:01.7931 vasca.tables:251 |DEBUG: Writing table 'tt_filters'
2024-11-11 09:00:01.7997 vasca.tables:251 |DEBUG: Writing table 'tt_visits'
2024-11-11 09:00:01.8087 vasca.tables:251 |DEBUG: Writing table 'tt_sources'
2024-11-11 09:00:01.8444 vasca.tables:251 |DEBUG: Writing table 'tt_detections'
2024-11-11 09:00:01.8740 vasca.tables:251 |DEBUG: Writing table 'tt_coadd_detections'
2024-11-11 09:00:01.9021 vasca.tables:251 |DEBUG: Writing table 'tt_coadd_sources'
2024-11-11 09:00:01.9378 vasca.tables:251 |DEBUG: Writing table 'tt_src_id_map'
2024-11-11 09:00:02.0384 __main__:15 |INFO: Done running VASCA pipeline.
# To be continued ...