Update DataSpace inquire to ensure that results are the same when repeated.

Changed insert of new columns to dataframe after efficiency warning.

  py/cso_superobs.py

v2.9.7

~~~~~~

Increased maximum number of records and introduced sort order to avoid that

search request return a different result when repeated.

  py/cso_dataspace.py

# 2024-11, Arjo Segers

#   Inquire Dataspace per month after change in allowed maximum number of records.

#   Sort listing files by processing and processor version.

#   Increased maximum number of records and introduced sort order to avoid that

#   search request return a different result when repeated.

#

########################################################################

        # fill product type:

        params["productType"] = producttype

        # search query could only return a maximum number of records:

        nrow = 100

        # search query could only return a maximum number of records;

        # use large value to have single page with full request,

        # since the catalogue might return different pages due to inconsistent sorting ...

        nrow = 2000

        # fill paging info:

        params["maxRecords"] = nrow

        # sort on publication date to ensure that pages are the same when requested again:

        params["sortParam"] = "published"

        params["sortOrder"] = "ascending"

        # start of first month:

        tm = datetime.datetime(t1.year,t1.month,1)

                #  }

                #

                # save result?

                if True:

                if False:

                    # targefile:

                    qfile = "query.json"

                    # save:

                    if len(output_df) > 0:

                        # same href already stored?

                        if href in output_df["href"].values:

                            ## testing ...

                            # logging.warning(f"ignore double product_id: {product_id}")

                            # testing ...

                            logging.warning(f"{indent}      ignore double product_id: {product_id}")

                            # ignore record:

                            continue

                        # endif