Welcome to IMDAA Studio

Your guide to converting IMDAA reanalysis archives into research-ready CSV datasets.

What Can You Do?

Upload & Convert

Upload raw IMDAA ZIP archives and convert them into structured CSV files with live progress tracking.

Apply Corrections

Run scientific correction engines on your CSV data — temperature, radiation, humidity, wind speed, and more.

Convert Units

Transform units across your datasets — Kelvin to Celsius, W/m² to MJ/m², and other scientific conversions.

Go Offline

Install the PWA app and process unlimited files locally without any upload size limits.

Online vs Offline Mode

Feature	Online (Cloud)	Offline (Local)
Upload Limit	3 GB per file	Unlimited
Processing	Cloud server	Your machine
Requires Login	Yes	Optional
Requires Install	No	Yes (PWA + Node.js)
Internet Required	Yes	No

Uploading Archives

How to upload IMDAA reanalysis ZIP files for conversion.

Step-by-Step

1

Go to the Dashboard

Sign in to your account and navigate to the Home page. You'll see the upload area at the center of the screen.

2

Select Your Archive

Drag and drop your IMDAA .zip archive onto the upload zone, or click to browse. The file must contain NetCDF (.nc) files from IMDAA reanalysis datasets.

3

Configure & Submit

After the archive is inspected, you can choose your output format and variable selections. Click Submit to begin processing.

4

Download Results

Track progress in the My Requests page. Once complete, download your CSV files directly from the dashboard.

💡

You can re-use previously uploaded archives. Go to the upload step and select from your existing archive list instead of uploading again.

Tracking Requests

Monitor your conversion jobs in real time.

Request Statuses

Status	Meaning
Queued	Your request is waiting to be processed.
Processing	The server is actively converting your archive. You'll see a live progress bar.
Completed	All CSV files are ready for download.
Failed	Something went wrong. Check the error message and try again.

What You Can Do

View Progress — Live percentage and ETA updates as your job runs.
Download Files — Once complete, download individual CSVs or all at once.
Delete Requests — Remove completed or failed requests from your history.
Re-run — Use the same archive to create new CSV variants with different settings.

Offline Mode

Process unlimited files locally on your own computer — no internet required.

📱

Offline mode is only available when you install IMDAA Studio as a PWA app. Visit the site in Chrome or Edge, then click "Install" in the browser's address bar.

How It Works

When you switch to Offline mode in the dashboard, IMDAA Studio connects to a local server running on your machine instead of the cloud. This removes the 3GB upload limit and processes everything locally.

Setting Up Offline Mode

1

Install the PWA App

Open IMDAA Studio in Chrome or Edge. Click the install icon in the address bar (or the "Install App" prompt). This adds the app to your desktop.

2

Install Node.js

Download and install Node.js (v18 or later) from nodejs.org.

3

Install IMDAA Studio Package

Open a terminal and run:

$ npm i -g imdaa-studio

4

Start the Local Server

$ imdaa-studio start

This starts the local server. Open the IMDAA Studio PWA app and switch to Offline mode using the toggle in the header.

⚠️

Keep the terminal window open while using offline mode. Closing it will stop the local server. To restart, run imdaa-studio start again.

Correction Engines

Apply scientific correction pipelines to your processed CSV data.

What Are Correction Engines?

IMDAA provides raw 3-hourly reanalysis data. Correction engines transform this raw data into daily, scientifically standardized products that are ready for research — converting units, aggregating timesteps, and applying domain-specific formulas.

How to Apply Corrections

1

Navigate to Corrections

Go to the Corrections page from the sidebar menu.

2

Select a Source File

Choose a processed CSV from your completed requests. The system will show which correction modules are compatible with your data.

3

Choose a Module

Select the correction engine to apply. Each module shows what it does and what outputs you'll receive.

4

Review & Download

Once processing completes, preview the results and download the corrected CSV files.

Available Modules

Module	What It Does	Output
🌡️ Temperature	Converts 3-hourly Kelvin data to daily °C products	Tmean, Tmax, Tmin
☀️ Radiation	Integrates W/m² flux into daily energy totals	SW, LW, Net Radiation (MJ/m²/day)
💧 Relative Humidity	Converts fractional RH to daily percentage stats	RH mean, max, min (%)
☁️ Vapor Pressure	Derives actual vapor pressure from dewpoint	ea (kPa)
🌬️ Wind Speed	Computes magnitude from U/V components at 2m height	Daily mean wind speed (m/s)
🌧️ Rainfall Erosivity	Estimates R-factor from sub-daily rainfall	Erosivity index (MJ·mm/ha/hr)
🌱 Soil Moisture	Processes 4-layer Noah LSM soil moisture data	Layer-wise daily mean, SWS (mm), Root Zone, Surface

Unit Conversion

Transform units across your CSV datasets with built-in scientific formulas.

How to Use

1

Open the Unit Converter

Navigate to the Unit Conversion page from the sidebar. You'll see the converter interface.

2

Upload Your CSV

Upload a processed CSV file. The system reads the columns and lets you select which ones to convert.

3

Choose Conversions

Select the source unit and target unit for each column. The formula used is shown transparently so you can verify the transformation.

4

Download Converted CSV

Click Download to get your unit-converted CSV with standardized column names and clear unit labels.

Common Conversions

From	To	Formula
Kelvin (K)	Celsius (°C)	T(°C) = T(K) − 273.15
W/m²	MJ/m²/day	E = flux × 0.0864
Pa	kPa	P(kPa) = P(Pa) / 1000
m/s	km/h	v(km/h) = v(m/s) × 3.6
kg/m²/s	mm/day	P(mm) = P(kg/m²/s) × 86400

File Management

Organize, download, and manage your processed archives and CSV outputs.

Your Workspace

The Files page shows all your archives and their associated outputs organized by upload. Each archive can have multiple CSV variants from different processing runs.

What You Can Do

Download — Download individual CSV files or all outputs from an archive at once.
Preview — Quick-view the first rows of any CSV to verify the data before downloading.
Re-process — Select an existing archive and run it again with different settings.
Delete — Remove files you no longer need to free up storage.

💡

Your storage quota depends on your membership plan. Check your Account page to see how much storage you have remaining.

Scientific Modules Reference

How each correction module transforms your IMDAA data.

🌡️ Temperature

Converts 3-hourly temperatures from Kelvin to Celsius, then produces daily aggregates.

UNIT CONVERSION

T(°C) = T(K) − 273.15

Products: Tmean (daily mean), Tmax (daily maximum), Tmin (daily minimum).

☀️ Radiation

Integrates instantaneous flux values (W/m²) into daily energy totals (MJ/m²/day).

FLUX INTEGRATION

R_daily = Σ(i=1 to 8) [ R_i × 0.03888 ]
where 0.03888 = 10800 seconds / 10⁶ (J → MJ)

Products: Net Shortwave, Net Longwave, Net Radiation.

💧 Relative Humidity

Converts fractional RH (0–1) to percentage and computes daily statistics.

CONVERSION

RH(%) = RH(fraction) × 100

Products: RH mean, RH max, RH min.

☁️ Vapor Pressure

Derives actual vapor pressure from dewpoint temperature using the Magnus formula.

MAGNUS FORMULA

ea = 0.6108 × exp[ (17.27 × Td) / (Td + 237.3) ]

Products: Daily mean actual vapor pressure (kPa).

🌬️ Wind Speed

Computes wind speed from U and V vector components with FAO-56 height adjustment.

VECTOR MAGNITUDE + HEIGHT ADJ.

WS = √(U² + V²)
WS_2m = WS_z × [ 4.87 / ln(67.8z − 5.42) ]

Products: Daily mean wind speed at 2m (m/s).

🌧️ Rainfall Erosivity

Estimates the R-factor (erosive potential) from sub-daily rainfall intensity.

R-FACTOR

R = Σ(EI₃₀) where EI₃₀ = kinetic energy × max 30-min intensity

Products: Daily precipitation (mm), Erosivity index.

🌱 Soil Moisture

Processes 4-layer Noah LSM soil moisture data (m³/m³) to compute layer-wise daily means, total soil water storage, and root zone averages.

SOIL WATER STORAGE

SWS = Σ(θᵢ × Zᵢ) → mm of water
Layers: L1 (0–10cm, 100mm) · L2 (10–35cm, 250mm) · L3 (35–100cm, 650mm) · L4 (100–300cm, 2000mm)

ROOT ZONE MOISTURE

θ_root = (θ₁×Z₁ + θ₂×Z₂ + θ₃×Z₃) / (Z₁ + Z₂ + Z₃) → m³/m³

Products: Layer-wise daily mean, Total SWS (mm), Root Zone moisture (0–1 m), Surface layer (0–10 cm), Monthly summaries.

Build Custom Correction Engines

Create your own scientific correction modules and share them with the research community.

🔧

Custom engines let you define your own scientific pipeline — input variables, formulas, output schema, and documentation — all uploadable from the Corrections page in the dashboard.

What You Need

Every custom correction module requires two files:

File	Format	Purpose
logic.py	Python (.py)	The processing script with your scientific logic
theory.md	Markdown (.md)	Scientific documentation shown to users in the wizard

Input File Requirements

Your module will receive a processed IMDAA CSV file. These CSVs always contain:

Column	Type	Description
`lat`	Float	Latitude coordinate
`lon`	Float	Longitude coordinate
`time`	Datetime	Timestamp of the observation
`station`	String	Station name or ID
`height`	Float	Station elevation (m)
`grid`	String	Grid cell identifier
`loc_id`	String	Location identifier (lat_lon)
Variable columns	Float	The actual data values (e.g., temperature, wind speed)

Variable Mapping & Unit Conversion Flow

Before your apply_correction() runs, the engine performs two preprocessing steps automatically:

1

Column Mapping

The user maps each of your REQUIRED_VARIABLES to a CSV column. The engine sends the column indices (0-based) to the Python worker. Your module receives only the mapped columns plus the standard metadata columns (lat, lon, time, station, height, grid, loc_id, date).

2

Unit Conversion

If the user specifies source → target unit conversions (e.g., Kelvin → Celsius), the engine applies the conversion formula before calling your apply_correction(). Your function receives already-converted data. You do not need to handle unit conversion in your logic.

The complete data flow from user input to your function:

/* Data preprocessing pipeline */ User selects CSV file → User maps REQUIRED_VARIABLES to CSV columns (by index) → User sets source unit → target unit for each variable → Engine selects mapped columns from DataFrame → Engine applies unit formulas (e.g., "x - 273.15" for K→°C) → Engine generates loc_id column from lat/lon if missing → Engine calls apply_correction(df, frequency, aggregations, output_prefix, transpose) → Your function receives the converted DataFrame

💡

The "original" baseline option gives the user access to the pre-conversion data, while "converted" gives the post-conversion data. Both are handled by the engine — your module only sees the converted data.

Python Script Structure

Your logic.py must define two things:

1. Metadata Constants

These tell the UI what your module needs and what it produces:

# Variables the user must map from their CSV REQUIRED_VARIABLES = ["Temperature", "Dewpoint"] # Output schema — what the module produces AGGREGATION_SCHEMA = { "baseline": { "options": [ {"id": "original", "label": "Original Values", "default": True} ] }, "correctionOutputs": { "note": "Optional description shown in the wizard", "aggregations": [ { "temporal": "sub-daily", "label": "Sub-Daily", "defaultEnabled": False, "types": [ {"id": "instant", "label": "Instantaneous Value", "default": True} ] }, { "temporal": "daily", "label": "Daily", "defaultEnabled": True, "types": [ {"id": "mean", "label": "Daily Mean", "default": True}, {"id": "max", "label": "Daily Max", "default": False}, {"id": "min", "label": "Daily Min", "default": False} ] }, { "temporal": "monthly", "label": "Monthly", "defaultEnabled": False, "types": [ {"id": "mean", "label": "Monthly Mean", "default": True} ] } ] } }

2. How the Output Selection Plan Works

When a user selects your module in the correction wizard, the Output Selection Plan section is dynamically generated from your AGGREGATION_SCHEMA. This means the checkboxes, group labels, and default states are entirely driven by what you define — no UI code changes are needed.

Data Flow

/* How the schema reaches the UI */ Python AGGREGATION_SCHEMA constant → inspect_module.py extracts it at server boot → Cached in server settings (sysSettings.corrections[].aggregationSchema) → Served via GET /api/settings/public → Frontend renderAggregations(schema) generates DOM checkboxes

Schema Structure Breakdown

Field	Level	Purpose
`baseline.options`	Root	Defines baseline checkboxes (e.g., "Original Values", "Unit Converted"). Each option has `id`, `label`, and `default` (checked state).
`correctionOutputs.note`	Root	A descriptive string shown above the aggregation checkboxes (e.g., "Temperature Product Suite: Generates Tmax, Tmin, Tmean").
`aggregations[]`	Output group	Each entry defines a temporal group (row of checkboxes).
`temporal`	Group	The temporal key — `"sub-daily"`, `"daily"`, `"monthly"`, or `"yearly"`. This value is passed to `apply_correction()` in the `aggregations` payload.
`label`	Group	Display label for the group (e.g., "DAILY PRODUCTS", "Monthly").
`defaultEnabled`	Group	If `true`, the group's checkboxes are checked by default. If `false` or omitted, they start unchecked.
`types[]`	Group	Individual output checkboxes within the group.
`types[].id`	Type	Unique identifier for the output type (e.g., `"max"`, `"sws"`). This `id` is what your `apply_correction()` receives in the `aggregations` payload.
`types[].label`	Type	User-visible label (e.g., "Tmax (Max)", "Total Soil Water Storage (mm)").
`types[].default`	Type	If `true`, this checkbox starts checked.

Supported Temporal Values

Value	Use Case	Example
`"sub-daily"`	Instantaneous or same-resolution outputs	10m wind speed from hourly U/V
`"daily"`	Daily aggregations (mean, max, min, sum)	Daily mean temperature
`"monthly"`	Monthly summaries	Monthly mean soil water storage
`"yearly"`	Annual summaries	Mean annual erosivity
`"none"`	Baseline only (no temporal grouping)	Original or unit-converted data

What the User's Selection Becomes

When the user clicks Execute Pipeline, the checked checkboxes are collected into an aggregations array and sent to apply_correction():

// If user checks: Daily → Tmax, Tmin | Monthly → Tmean aggregations = [ { "temporal": "daily", "types": ["max", "min"] }, { "temporal": "monthly", "types": ["avg"] } ]

Your apply_correction() function iterates over this array to produce the requested outputs. The temporal and types[].id values are what you defined in your schema — they are the contract between your UI declaration and your processing logic.

💡

If no AGGREGATION_SCHEMA is defined, the system falls back to a default schema with a single "none" temporal group producing "original" and "converted" baseline outputs.

3. The apply_correction Function

This is the entry point the engine calls with the user's mapped data:

# This function is called by the engine def apply_correction(df, frequency="hourly", aggregations=None, output_prefix="Output", transpose=True, **kwargs): """ df: Pandas DataFrame with metadata + mapped columns frequency: Data frequency ("hourly", "3hourly", "daily") aggregations: List of user-selected outputs from AGGREGATION_SCHEMA output_prefix: Prefix for output filenames (default: "Output") transpose: If True, pivots output to station-per-column format """ # 1. Get variable columns (everything except metadata) _META = {"lat", "lon", "time", "station", "height", "grid", "loc_id", "date"} var_cols = [c for c in df.columns if c.lower() not in _META] # 2. Validate required variables exist if len(var_cols) < 1: return {"error": "No variable columns found"} # 3. Your scientific computation here result = df[var_cols].mean() # Example # 4. Return a dict of filename → DataFrame return { f"{output_prefix}_Daily_Mean.csv": result_df }

Output Schema

Your function must return a dictionary where:

Keys = output filenames (e.g., "Daily_Mean.csv")
Values = Pandas DataFrames with the computed results

Each DataFrame should include lat, lon, date columns plus your computed values. The engine handles storage and download links automatically.

Output File Naming Convention

Every correction run generates one or more CSV files. Understanding the naming convention helps you identify outputs programmatically and build downstream automation.

How Filenames Are Resolved

The engine uses a two-stage resolution process to produce final filenames:

1

Your apply_correction() returns a dict

Each dict key is either a complete filename (ending in .csv) or a suffix (without .csv). The engine treats these differently.

2

write_results() resolves the final name

If the key ends with .csv → used as-is (e.g., "Output_Tmax_Daily.csv")
If the key does not end with .csv → the engine prepends {output_prefix}_ and appends .csv (e.g., "Daily_Mean" becomes "Output_Daily_Mean.csv")

Baseline Output Names

Baseline outputs are generated automatically by the engine (not by your module):

Baseline Option	Filename
`"original"`	`{output_prefix}_Original.csv`
`"converted"`	`{output_prefix}_UnitConverted.csv`

Built-in Module Naming Patterns

The table below shows the naming convention used by each built-in module, so you know what to expect when running them:

Module	Dict Key Pattern	Final Filename Example	Has .csv in key?
Temperature	`{prefix}_{Label}_{Temporal}.csv`	`Output_Tmax_Daily.csv`	Yes
Relative Humidity	`{prefix}_{Label}_{Temporal}.csv`	`Output_RHmean_Daily.csv`	Yes
Radiation	`{prefix}_{Product}_Daily_MJ.csv`	`Output_NetSW_Daily_MJ.csv`	Yes
Rainfall Erosivity	`{prefix}_{Product}_{Temporal}.csv`	`Output_Rainfall_Daily.csv`	Yes
Soil Moisture	`{prefix}_{Product}_{Temporal}`	`Output_SoilWaterStorage_Daily.csv`	No — engine adds .csv
Wind Speed	`{Product}`	`Output_Daily_Wind_Speed_10m.csv`	No — engine adds prefix + .csv
Vapor Pressure	`{Product}`	`Output_Daily_Saturation_Vapor_Pressure_ES.csv`	No — engine adds prefix + .csv

Recommended Pattern for Custom Modules

For consistency with the built-in modules, we recommend including both output_prefix and .csv in your dict keys. This makes filenames predictable and self-documenting:

# Recommended: include prefix + .csv in the key return { f"{output_prefix}_Daily_Mean.csv": daily_df, f"{output_prefix}_Monthly_Sum.csv": monthly_df, }

ℹ️

The output_prefix comes from the user's "Output Prefix" field in the wizard (defaults to "Output"). If multiple files are generated, each gets its own name — no index suffixes are added by the engine.

Transpose Output Format

The transpose parameter controls the output format:

transpose=True (default): Output is pivoted — rows are timestamps, columns are station/location IDs. Best for time-series analysis.
transpose=False: Output is in long format — each row is a single observation with lat, lon, time, and value columns.

ℹ️

The engine automatically handles the loc_id column creation from lat/lon coordinates when needed for grouping.

Error Handling

To signal errors, return a dictionary with an "error" key:

if len(var_cols) < 2: return {"error": "Wind Speed module requires both U and V components."}

The engine will catch this and display the error message to the user instead of generating output files.

Writing Theory Documentation

The theory.md file is shown to users in the correction wizard. It should include:

1

Overview

What the module does, which variables it processes, and what scientific products it generates.

2

Calculation Logic

The formulas used. Use LaTeX notation: $$ T(°C) = T(K) - 273.15 $$

3

Expected Outputs

List every output file the module can generate, with column names and units.

Getting Started Template

Copy this starter template and modify it for your use case:

# my_custom_module/logic.py import pandas as pd import numpy as np REQUIRED_VARIABLES = ["MyVariable"] AGGREGATION_SCHEMA = { "baseline": { "options": [ {"id": "original", "label": "Original Data", "default": True} ] }, "correctionOutputs": { "aggregations": [ { "temporal": "daily", "label": "Daily Products", "defaultEnabled": True, "types": [ {"id": "daily_mean", "label": "Daily Mean", "default": True} ] } ] } } def apply_correction(df, frequency="hourly", aggregations=None, output_prefix="Output", transpose=True, **kwargs): # Extended metadata columns _META = {"lat", "lon", "time", "station", "height", "grid", "loc_id", "date"} var_cols = [c for c in df.columns if c.lower() not in _META] # Validate required variables if not var_cols: return {"error": "No variable columns found"} # Parse time and create location ID df = df.copy() df["time"] = pd.to_datetime(df["time"]) df["loc_id"] = df["lat"].round(4).astype(str) + "_" + df["lon"].round(4).astype(str) # Pivot for time-series operations ts_pivot = df.pivot_table(index="time", columns="loc_id", values=var_cols[0]) # Compute daily mean daily = ts_pivot.resample("D").mean() # Format output based on transpose flag if not transpose: daily = daily.reset_index().melt(id_vars="time", var_name="loc_id", value_name="MyVariable_daily_mean") daily[["lat", "lon"]] = daily["loc_id"].str.split("_", expand=True).astype(float) daily = daily.drop(columns=["loc_id"]) return {f"{output_prefix}_Daily_Mean.csv": daily}

How to Upload

1

Go to Corrections Page

Navigate to Corrections in the dashboard sidebar.

2

Fill in Module Details

Enter a label (e.g., "Bias Correction v2"), upload your logic.py and theory.md files.

3

Share (Optional)

Check "Share with Community" to make your module available in the global research library for other users.

4

Upload

Click UPLOAD_SCIENTIFIC_MODULE. Your module will appear in the correction engine dropdown immediately.

⚠️

Your Python script runs in a sandboxed environment. Only pandas, numpy, and scipy are available. If you need additional packages, contact the admin.

Frequently Asked Questions

Common questions about using IMDAA Studio.

What file formats can I upload?

IMDAA Studio accepts .zip archives containing NetCDF (.nc) files from the IMDAA reanalysis dataset.

What is the maximum upload size?

In Online mode, the limit is 3 GB per file. In Offline mode, there is no limit — processing happens locally on your machine.

Do I need Python installed?

Only if you use Offline mode. The local server uses Python for scientific data processing. Python 3.9 or later is required. Online mode handles everything on the cloud.

Can I re-use uploaded archives?

Yes. Previously uploaded archives are saved in your workspace. You can select them again during the upload step to create new CSV variants without re-uploading.

What happens to my data?

In Online mode, your data is stored securely on our servers during processing and is only accessible from your authenticated account. In Offline mode, all data stays on your local machine.

How do I switch between Online and Offline mode?

Use the mode toggle in the dashboard header. Offline mode only appears if you've installed the IMDAA Studio PWA app.

What if the offline server won't start?

Run imdaa-studio doctor in your terminal to diagnose common issues. Make sure Node.js v18+ and Python 3.9+ are installed.