Explorer STAC loading limit issue

STAC_loading_explorer.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "b57757fc-58e8-4f44-a44e-ce36b90dfff2",
   "metadata": {},
   "source": [
    "# Explorer STAC API limit issue\n",
    "\n",
    "**Expectation:** Users get back all relevant data for their query without having to manually configure limits (exception being perhaps extremely large queries)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "20008b2f-e080-4b08-872e-103640525987",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import pystac_client, odc.stac\n",
    "\n",
    "datetime = \"2023-12-01/2024-02-28\"\n",
    "bbox = [146.04, -34.30, 146.05, -34.28]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0d596499-3bbc-4fb6-92f1-b11f396200a7",
   "metadata": {},
   "source": [
    "## Element84 STAC API\n",
    "\n",
    "All data returned without `limit` being specified:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "c60603b4-7ae2-42da-9ec5-5cecf0816254",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "36"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "client = pystac_client.Client.open(\"https://earth-search.aws.element84.com/v1\")\n",
    "\n",
    "# Search for items in the collection\n",
    "collections = [\"sentinel-2-l2a\"]\n",
    "query = client.search(\n",
    "    collections=collections,\n",
    "    bbox=bbox,\n",
    "    datetime=datetime,\n",
    ")\n",
    "\n",
    "# Search the STAC catalog for all items matching the query\n",
    "len([i.properties[\"datetime\"] for i in query.items()])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2660ced5-f03a-4903-894c-7c1eb28f6f8a",
   "metadata": {},
   "source": [
    "## Microsoft Planetary Computer STAC API\n",
    "\n",
    "All data returned without `limit` being specified:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "05873759-fee9-45dd-9971-fcdef0e91a08",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "36"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "client = pystac_client.Client.open(\"https://planetarycomputer.microsoft.com/api/stac/v1\")\n",
    "\n",
    "# Search for items in the collection\n",
    "collections = [\"sentinel-2-l2a\"]\n",
    "query = client.search(\n",
    "    collections=collections,\n",
    "    bbox=bbox,\n",
    "    datetime=datetime,\n",
    ")\n",
    "\n",
    "# Search the STAC catalog for all items matching the query\n",
    "len([i.properties[\"datetime\"] for i in query.items()])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "096633d5-e07f-4b18-bca6-242972e56850",
   "metadata": {},
   "source": [
    "## Explorer \n",
    "### Default: only 20 items returned"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "ef371092-0a29-45ff-86cf-ba841fa56124",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "20"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "client = pystac_client.Client.open(\"https://explorer.sandbox.dea.ga.gov.au/stac\")\n",
    "\n",
    "# Search for items in the collection\n",
    "collections = [\"ga_s2am_ard_3\", \"ga_s2bm_ard_3\"]\n",
    "query = client.search(\n",
    "    collections=collections,\n",
    "    bbox=bbox,\n",
    "    datetime=datetime,\n",
    ")\n",
    "\n",
    "# Search the STAC catalog for all items matching the query\n",
    "len([i.properties[\"datetime\"] for i in query.items()])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b7166b3-efa4-4047-a75f-9aae9d75f044",
   "metadata": {},
   "source": [
    "### Using custom limit returns all"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "09119f31-488c-4516-ad95-8bed4c238f57",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "37"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "query = client.search(\n",
    "    collections=collections,\n",
    "    bbox=bbox,\n",
    "    datetime=datetime,\n",
    "    limit=1000,\n",
    ")\n",
    "\n",
    "# Search the STAC catalog for all items matching the query\n",
    "len([i.properties[\"datetime\"] for i in query.items()])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d86682a-cf82-47e5-9ab9-8fc18dfda374",
   "metadata": {},
   "source": [
    "### More confusing: expanding the time window returns 60 items?\n",
    "\n",
    "60 = 3 * 20 default page size?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "1e260dec-d265-46ca-a60b-a3ab99b1ba05",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "60"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "client = pystac_client.Client.open(\"https://explorer.sandbox.dea.ga.gov.au/stac\")\n",
    "\n",
    "# Search for items in the collection\n",
    "collections = [\"ga_s2am_ard_3\", \"ga_s2bm_ard_3\"]\n",
    "query = client.search(\n",
    "    collections=collections,\n",
    "    bbox=bbox,\n",
    "    datetime=\"2023-05-01/2024-02-28\",\n",
    ")\n",
    "\n",
    "# Search the STAC catalog for all items matching the query\n",
    "len([i.properties[\"datetime\"] for i in query.items()])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "939de483-05c8-4c34-bd86-f94437da64a0",
   "metadata": {},
   "source": [
    "With a high limit we get back 120!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "b07c8fe3-e1d3-48ed-aa2c-12086bd8a594",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "122"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "client = pystac_client.Client.open(\"https://explorer.sandbox.dea.ga.gov.au/stac\")\n",
    "\n",
    "# Search for items in the collection\n",
    "collections = [\"ga_s2am_ard_3\", \"ga_s2bm_ard_3\"]\n",
    "query = client.search(\n",
    "    collections=collections,\n",
    "    bbox=bbox,\n",
    "    datetime=\"2023-05-01/2024-02-28\",\n",
    "    limit=1000,\n",
    ")\n",
    "\n",
    "# Search the STAC catalog for all items matching the query\n",
    "len([i.properties[\"datetime\"] for i in query.items()])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {},
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}