Climate Indices for Pacific Sectoral Applications Portal

Overview of the Upload Steps

The data upload process is divided into four main steps:

1. Select Files: Choose the CSV files you wish to import.
2. Confirm File Details: Verify or manually input the Country Code, Station Number, Time Series Period, and Data Source Type for each file.
3. Validate Data: The system performs client-side checks on your files for schema compliance and data integrity.
4. Review & Submit: Review the validation summary and submit your valid files to the server.

Step 1: Select Files

This initial step involves selecting the CSV files you intend to upload.

File Selection Input

Click on the "Choose files" or "Browse" button (the appearance may vary by browser) associated with the "CSV File(s)" label. This will open your system's file dialog, allowing you to select one or multiple CSV files.

Accepted File Type

Only files with a .csv extension are accepted. If you select files of other types, they will be ignored by the system and will not appear in the "Selected files" list.

Upload Limits

Please be aware of the following limits for each upload batch: a maximum of 50 files can be selected, and each individual file must not exceed 5MB in size. The system will prevent the selection of files that exceed these limits.

Selected Files Table

Once you've selected files, a table will appear below the input field, listing the names of all valid (.csv) files you've chosen. Each file in this list will have a "Remove" button next to it. Clicking this button will deselect that specific file and remove it from the list.

Proceeding to the Next Step

The "Next" button to proceed to Step 2 will become active only after you have selected at least one valid CSV file.

Detailed CSV File Requirements

For successful data import, your CSV files must strictly adhere to the following schema and fixed-column format:

• Filename Convention (Recommended):

  Naming your files using one of the following conventions is highly recommended, as it allows the system to automatically attempt to parse key details:

  • CountryCode_StationNumber_StationName_Period_DataSource.csv (e.g., AUS_001_WillisIsland_dly_hom.csv)
  • CountryCode_StationNumber_Period_DataSource.csv (e.g., FJI_001_dly_qc.csv, TUV_002_mly_hom.csv)

  The StationName component in the first format is optional and will be ignored by the parsing logic if present; the system relies on CountryCode and StationNumber for station identification. Country codes are ISO 3166-1 alpha-3 (e.g., FJI, TUV, AUS). The period must be either 'daily' or 'monthly' (or 'dly', 'mly'). The DataSource part of the filename should correspond to one of the available data source types (e.g., qc, hom), all case-insensitive in the filename.

• File Structure (No Header Row, Fixed Columns):

  Your CSV files must not contain a header row. The data should start directly from the first line. The system expects data in a fixed 6-column format, corresponding to:

  1. Year
  2. Month
  3. Day
  4. Rainfall (mm)
  5. Maximum Temperature (°C)
  6. Minimum Temperature (°C)

  The system automatically assigns these meanings to the columns based on their order. The "Data Source Type" selected in Step 2 will determine how this data is further processed and stored.

  Important: The order of these 6 columns is critical. If your data deviates from this specific order, incorrect data may be ingested into the climate tables.
• Date Columns (Columns 1-3):

  The first three columns of your CSV file represent the date:

  • Column 1 (Year): Must be a numeric, 4-digit year (e.g., 2023). Values should be within a reasonable range (e.g., 1700-2100).
  • Column 2 (Month): Must be a numeric month, where 1 represents January and 12 represents December.
  • Column 3 (Day): Must be a numeric day of the month.
    • For daily data, this should be the day of the month (e.g., 1 to 31).
    • For monthly data (as indicated by the "Time Series Period" selection in Step 2), this column must always contain the value 0.

  The combination of year, month, and day (using day 1 for validation purposes if the input day is 0 for monthly data) must form a valid calendar date (e.g., a day value of 29 for month 2 in a non-leap year like 2023 is invalid). Data must also be in chronological order, without duplicate dates or gaps in the time series.

• Climate Data Columns (Columns 4-6):

  The next three columns of your CSV file represent the climate data values:

  • Column 4 (Rainfall): Rainfall total, in millimetres (mm).
  • Column 5 (Maximum Temperature): Maximum temperature, in degrees Celsius (°C).
  • Column 6 (Minimum Temperature): Minimum temperature, in degrees Celsius (°C).

  The "Data Source Type" (e.g., Pre-Hom, Homogenised) selected in Step 2 determines how this data is categorised and stored by the system.

  Additionally, if valid Maximum Temperature (Column 5) and Minimum Temperature (Column 6) values are provided, the system will automatically calculate and store the Mean Temperature (Tmean) and Diurnal Temperature Range (DTR) during data processing.
• Data Values (Columns 4-6):
  • All climate data values must be numeric.
  • Do not include units (e.g., "mm", "°C") or any other non-numeric text (e.g., "N/A", "missing") within the data cells.
  • Missing data must be represented by -99.9 or -99.90. Empty cells for climate data values are not permitted and will cause validation errors.
  • Precision Requirements:
    • Rainfall values (Column 4) must have no more than four digits before the decimal point.
    • Temperature values (Columns 5 and 6) must have no more than two digits before the decimal point.

Example CSV Structures (No Header Row):

Daily Data Example:

2023,1,1,5.2,30.1,20.5
2023,1,2,-99.9,29.8,20.0
2023,1,3,10.0,-99.9,21.0

In this daily example (Columns: Year, Month, Day, Rain, Tmax, Tmin):

• The data for 2023-01-02 has missing rainfall (-99.9).
• The data for 2023-01-03 has missing maximum temperature (-99.9).

Monthly Data Example:

2023,1,0,150.5,22.5,15.0
2023,2,0,120.0,23.0,15.5
2023,3,0,200.1,-99.9,16.0

In this monthly example (Columns: Year, Month, Day, Rain, Tmax, Tmin):

• Note that the Day column (Column 3) is always 0.
• The data for March 2023 has missing maximum temperature (-99.9).

Step 2: Confirm File Details

This step allows you to associate each uploaded CSV file with a specific Country, Station, Time Series Period, and Data Source Type within the portal's database. Accurate association is crucial for correct data storage and retrieval.

Confirmation Table

A table will display each file you selected in Step 1. For each file, you will see:

• Original Filename: The name of the file you uploaded.
• Country Code: A dropdown menu to select the country.
• Station Number: A dropdown menu to select the station.
• Time Series Period: A dropdown menu to select the period (Daily/Monthly).
• Data Source Type: A dropdown menu to select the type of data (e.g., Pre-Hom, Homogenised).

Automatic Parsing

If your filename followed one of the recommended conventions (e.g., CountryCode_StationNumber_StationName_Period_DataSource.csv or CountryCode_StationNumber_Period_DataSource.csv), the system will attempt to automatically select the correct Country, Station, Time Series Period, and Data Source Type in the dropdowns. Please verify these automatic selections. If the Data Source Type cannot be parsed from the filename or is ambiguous, it will require manual selection or confirmation.

Manual Selection

If the filename did not match the convention, or if the automatic parsing was incorrect or incomplete, you must manually select or confirm all details for each file:

1. Select Country: Click the "Country Code" dropdown and choose the appropriate country from the list. This list is populated from the countries known to the portal system.
2. Select Station: Once a country is selected, the "Station Number" dropdown for that file will become active. Click it and choose the correct station. The list will show station names and their unique station numbers/IDs for the selected country.
3. Select Time Series Period: Click the "Time Series Period" dropdown and choose either "Daily" or "Monthly".
4. Select Data Source Type: Click the "Data Source Type" dropdown and choose the appropriate type (e.g., Pre-Hom for pre-homogenised, Hom for homogenised). This selection is critical as it determines how the data is processed and stored.

Note: The station dropdown will remain disabled until a country is selected. If the dropdowns do not populate, it might be because the necessary station metadata is still loading. In such cases, try waiting a moment or reopening the upload modal.

Requirement for Proceeding

You must select a valid Country Code, a valid Station Number, a Time Series Period, and a Data Source Type for every file in the list. If any file is missing any of these selections, the respective dropdown will be highlighted (e.g., with a red border), and you will not be able to proceed to the next step. The "Next" button remains disabled until all files have these details confirmed. Additionally, the combination of Country Code, Station Number, Time Series Period, and Data Source Type must be unique across all files being uploaded in the current batch.

Navigation

You can use the "Back" button to return to Step 1 (Select Files) if you need to change your file selection. Click "Next" to proceed to Step 3 (Validate Data) once all file details are confirmed.

Step 3: Validate Data

After confirming the details for each file, this step performs automated client-side validation. This comprehensive check helps catch common formatting issues and data inconsistencies across the entire file before submission to the server.

Validation Process & Progress Display

When you click "Next" from Step 2, the system initiates the validation process for all selected files. A table will appear, listing each file with a dedicated progress bar and status indicator:

• File Name: The name of the file being validated.
• Validation Progress: A progress bar showing the validation percentage for that specific file (0% to 100%).
• Status: Indicates the current state, such as "Pending...", "Validating...", "Complete", or "Error".

The system validates files, and the progress bars update in real-time. This is particularly useful for larger files, allowing you to monitor the process.

Checks Performed (Full File Validation)

The validator performs several checks on the entire content of each file, including but not limited to:

• Country Code and Station ID: Verifies that the selected Country Code and Station Number combination exists in the portal's database.
• File Emptiness: Ensures the file is not empty.
• Fixed Column Structure: Verifies the file contains data for the expected 6 columns (Year, Month, Day, Rain, Tmax, Tmin).
• Data Integrity (for all data rows):
  • Checks that values in the date columns (Columns 1-3) are numeric and form a valid calendar date. For monthly data, the Day (Column 3) must be 0.
  • Ensures that values in all climate data columns (Columns 4-6) are numeric or the specific missing data flags (-99.9 or -99.90). Empty cells for climate data are not allowed.
  • Validates data precision: Rainfall (Column 4) must have no more than four digits before the decimal; Temperature (Columns 5, 6) must have no more than two digits before the decimal.
  • Checks for chronological order of dates.
  • Identifies duplicate dates within the file.
  • Detects gaps in the time series.
• Automatic Data Generation: The system automatically calculates Mean Temperature (Tmean) and Diurnal Temperature Range (DTR) from the provided Tmax and Tmin values. These derived values are included in the data sent to the server.

Note on Performance: While the system is designed to handle files efficiently, validating very large files can still take some time. The progress bars provide feedback during this period.

Displaying Validation Results

Once validation is complete for all files, a summary of results is displayed below the progress table. For a comprehensive breakdown of all checks and errors for each file, consult the 'Detailed Log' section available in this step.

• Errors: Critical issues that prevent a file from being uploaded are listed. The file's row in the progress table will also indicate an "Error" status. Examples include incorrect column count, invalid dates, non-numeric data where numbers are expected, data precision errors, or empty files.

A general message will also indicate if any files have errors or if all files validated successfully without any issues.

Proceeding to the Next Step

The "Next" button to proceed to Step 4 (Review & Submit) will only be enabled if no files have critical errors.

Navigation

If there are errors, you should use the "Back" button to return to Step 2 (Confirm File Details). From there, you can go back to Step 1 to remove problematic files or re-select corrected ones. You cannot directly edit file content within this upload tool; corrections must be made to your local CSV files before re-selecting them for upload.

Step 4: Review & Submit

This is the final step before your data is sent to the server. It provides a summary of all files and their validation status.

Review Area

The "Review Area" displays a table listing all files processed through the validation step. For each file, it will show:

• The original filename.
• The confirmed Country Code, Station Number, Time Series Period, and Data Source Type.
• A summary of its client-side validation status (e.g., "Ready" or "Error").
• An "Upload Status" column that will update after submission to reflect server processing results (e.g., "Complete", "Error", "Overwritten").

An "Overall Status" message above this list will summarise, for example, "X file(s) ready for upload. Y file(s) with errors." A detailed log is also available in this step, providing comprehensive information about client-side validation and data ingestion results for each file, including notifications if existing data was replaced. This log is crucial for understanding the outcome of each file's submission.

Submit Button

The "Upload Data" button will be enabled if there is at least one file ready for upload (i.e., passed client-side validation without errors). Files with client-side warnings can still be submitted.

Submission Process

When you click the "Upload Data" button:

1. The button's text will change to indicate submission is in progress.
2. Only files that passed client-side validation (warnings are acceptable) will be sent to the server.
3. The files, along with their associated metadata, are sent to the server for ingestion into the database.

Asynchronous Processing

Upon submission, the server accepts the files and queues them for background processing. This means you can continue using the portal or even close the browser while the data is being ingested and processed.

Monitoring Job Progress

You can monitor the status of your upload and subsequent Climpact processing by navigating to the Climpact Jobs tool, accessible from the main administration panel. This tool provides real-time updates on the status of each batch and individual jobs within it.

Server Response

After you submit, the "Upload Status" in the review table will immediately change to "Submitted" for all accepted files. The server performs the following steps in the background:

• Data Uploading: The raw data is ingested into the database.
• Updating Availability: The system updates its records to reflect the newly available data.
• Pending: The job is waiting to be picked up by a processing worker.
• Processing: The Climpact indices are being calculated.
• Completed: The job has finished successfully.
• Failed: The job failed at some stage. The Climpact Jobs tool will provide error details.

A toast notification will confirm that your files have been accepted for background processing. For detailed outcomes, always refer to the Climpact Jobs tool.

Navigation

You can use the "Back" button to return to Step 3 (Validate Data) if you need to re-check client-side validation messages before submitting.

Troubleshooting / Common Issues

• File not appearing in "Selected Files" list (Step 1): Ensure the file has a .csv extension. Other file types are ignored.
• Cannot select Country/Station, or dropdowns are empty (Step 2): The portal needs to load station metadata first. Try waiting a few moments. If the problem persists, try closing and reopening the Data Upload tool, or reloading the portal page.
• "Data Source Type" not selected (Step 2): Ensure you have selected a Data Source Type (Pre-Hom, Homogenised) for every file, or that it was correctly parsed from the filename.
• "Invalid Country Code or Station ID" error (Step 3): Double-check your selections in Step 2. Ensure the station belongs to the selected country and both are valid entries in the portal system.
• Common CSV validation errors (Step 3):
  • Incorrect number of columns: Ensure your CSV file contains exactly 6 columns of data per row, corresponding to Year, Month, Day, Rainfall, Max Temp, and Min Temp. Do not include a header row.
  • Invalid date values: Ensure Year (Col 1), Month (Col 2), and Day (Col 3) values are numeric and form a valid calendar date. For monthly data, Day (Col 3) must be 0.
  • Non-numeric value found: Climate data columns (4-6) must contain only numbers or -99.9 / -99.90 for missing data. Remove any text, symbols, or units.
  • Empty cells for climate data: Climate data values in columns 4-6 cannot be empty. Use -99.9 or -99.90 to represent missing data.
  • Data precision errors: Rainfall (Col 4) must have no more than four digits before the decimal. Temperature (Cols 5, 6) must have no more than two digits before the decimal.
  • Chronological, Duplicate, or Gap errors: Ensure data is sorted chronologically by date, contains no duplicate dates, and has no gaps in the time series.
  If you encounter these errors, you must correct your local CSV file(s) and then restart the upload process from Step 1 with the corrected file(s). Refer to the detailed log in Step 3 for specific error messages.
• Server submission errors (Step 4): If the server reports an error after submission, review the "Upload Status" for the affected file and check the detailed log for messages from the server. This may indicate server-side problems.