Prerequisites

Ornitela

You will need:

• An account with them, with Admin privileges.   

Gundi

• Gundi SFTP account details.

EarthRanger

To send movement data to EarthRanger, you will need:

• A special EarthRanger user (e.g., “Gundi Service Account”) for this integration. Please refer to EarthRanger's documentation or contact Support.
• A long-lived token for authentication assigned to the user created in the previous step. Please refer to EarthRanger's documentation or contact Support.

Configuration

GUIDES

1. Gundi SFTP Server Settings

Log in to the Gundi SFTP Admin Panel.

Go to Users and find an existing integration to use as a template (e.g. another Ornitela user).

Click Actions > Use as Template.

Under Users, set the username and password for the external provider to use when connecting to the SFTP server. Leave the public key field blank.

Note: The username and password set in step 5 are the credentials you will hand off to the external provider (OrniTrack) for their SFTP upload configuration (see next section).

Under File System, update the following fields:

1. Storage: leave as GCS (default)
2. Root directory: set to /srv/sftpgo/data/{sitename} (e.g. /srv/sftpgo/data/skyrangers)
3. Bucket: leave unchanged from the template (sintegrate-sftpgo-storage-prod)
4. Automatic credentials: leave toggled on
5. Key Prefix: set to {provider}/{sitename}/ (e.g. ornitela/skyrangers/)
6. Click Save.

2. OrniTrack Settings

2.1 Go to OrniTrack.

2.2 Go to Settings.

2.3 Enable data upload to FTP/SFTP server.

2.3 Select CSV 2 as the data file format.

2.4 Use sftp.gundiservice.org as the Host name.

2.5 Use 2022 as the Port number.

2.6 Use / as the Remote Directory.

2.7 Configure the username and password used by the Gundi SFTP account.

3. Gundi Connection Settings

3.1 Log in to Gundi.

3.2 Click Create Route.

3.3 Select Ornitela from the list of supported technologies

3.4 Configure these fields:

Workspace name

Route name (e.g., “Ornitela (Account)”)

Bucket Path: ornitela/{username}

Batch Size: 300

Chunk Size: 3000

Max Files per run: 5

Historical Limit Days: 1

3.10 Click Next.

3.11 Click Add destination on the flow map and select a destination. Please consult our guides.

4. EarthRanger Settings

The following configurations are optional:

4.1. Select the data you want to see in the map.

4.1.1 Log in to EarthRanger as an admin (site.pamdas.org/admin).
4.1.2 Go to Source Providers.
4.1.3 Select the Source Provider created for this integration (e.g., "gundi_ornitela_UUID")

4.1.4 Click on Subject Details Configuration.
4.1.5 Configure the additional information you would like to see on the map.

4.2 Assign your Subjects to a Subject Group.

Please refer to EarthRanger's documentation or contact Support.

Troubleshooting

Q: How does file processing work end-to-end?
Files move through three bucket folders in sequence:

1. Root (ornitela/) — Ornitela drops raw CSV files here. The connector checks this every 5 minutes.
2. in_progress/ — When processing starts, the connector carves a chunk from the root file, compresses
     it, and places it here. The root file is updated with remaining rows (or deleted if empty).
3. archive/ — On success, the chunk is moved here. If the root file still has rows, the next chunk is
     triggered automatically (chaining).
4. dead_letter/ — On failure or timeout, the chunk is moved here instead.

Files named like bird001_a3f9c1b2_2.csv.gz encode: <original_stem>_<chain_id>_<chunk_index>.csv.gz.

Q: How do I trace a single file's full processing history in logs?
Every log line for a chunk is tagged with the filename in square brackets:

   [bird001_a3f9c1b2_1.csv.gz] Starting processing for integration 97eed06a-...
   [bird001_a3f9c1b2_1.csv.gz] Processing in_progress/bird001_a3f9c1b2_1.csv.gz: 3000 rows
   [bird001_a3f9c1b2_1.csv.gz] Archived successfully
   [bird001_a3f9c1b2_1.csv.gz] Processed: extracted 3000 records, sent 3000 observations

To trace an entire chain (all chunks from one source file), filter logs by the chain_id (the 8-character
hex segment, e.g. a3f9c1b2). All chunks carved from the same source file share the same chain_id.

In Cloud Logging / GCP, use:

   textPayload=~"a3f9c1b2"

Q: A file ended up in dead_letter/. What happened?
Check the logs for the filename. Common causes:

• “Timed out — moving to dead_letter/”. Processing exceeded MAX_ACTION_EXECUTION_TIME (default: 8 min). File may be too large, Gundi API was slow, or Cloud Run killed the container.
• "Error: <exception>" + “Successfully moved to dead_letter/”. Unhandled exception during CSV parsing, download, or observation sending. See exception on preceding line.
• “Could not move <file> to dead_letter/ — file remains in    in_progress/”. The dead-letter move itself failed (GCS permission or network issue). File is still in in_progress/.

Retrying dead-letter files:  Move the file back to the root folder (ornitela/). The cron job will pick it up on the next 5-minute tick.

Q: Files are piling up in in_progress/ and not moving anywhere.
This usually means the processing action was interrupted before it could archive or dead-letter the file.

Causes:

• Cloud Run SIGTERM at 5 minutes: if PROCESS_PUBSUB_MESSAGES_IN_BACKGROUND is not "true", Cloud Run kills the container at the request timeout (300s) before the action completes. The file stays in in_progress/ indefinitely.
• Container restart / OOM: same result — no cleanup path ran.

To diagnose:
Check if the file has been in in_progress/ longer than 10 minutes with no log activity. If so, move it to root to retry.

Permanent fix:
Set PROCESS_PUBSUB_MESSAGES_IN_BACKGROUND=true in Cloud Run.

Q: process_new_files reports new_files_found: 0 but files exist in the bucket.
The action only counts root-level files — files not under in_progress/, archive/, or dead_letter/. Files already being processed or archived are intentionally excluded.

If the root appears empty but you expect files:

1. Check that files were uploaded to the correct bucket and bucket_path prefix (default: ornitela/).
2. Verify the INFILE_STORAGE_BUCKET env var is set to the correct bucket.
3. Look for log lines: "Could not get metadata for file <name>" — this indicates a permissions or network issue during listing.

Q: process_new_files returns success immediately but observations haven't arrived yet.
This is expected behavior. process_new_files returns as soon as the chunk sub-actions are triggered — it does not wait for them to finish.

The actual observations are sent by action_process_ornitela_file, which runs asynchronously.

Check logs for:
[<chunk_name>] Processed: extracted N records, sent N observations

Q: Observations are arriving but they're all filtered out (nothing appears in EarthRanger).
Check generate_gundi_observations filtering. Rows are dropped if:

• recorded_at is older than historical_limit_days (default: 5 days). If the file contains older historical data, increase this value in the action config.
• The UTC_datetime field is missing or malformed. The row will raise a ValueError and be skipped, with a log like:
  Error parsing CSV row in bird001_a3f9c1b2_1.csv.gz: time data '' does not match format '%Y-%m-%d %H:%M:%S'

Q: The chain stopped — only chunk 1 was processed, chunks 2+ never appeared.
The self-chaining mechanism triggers the next chunk from _trigger_next_chunk after archiving chunk N.

If the chain breaks:

1. Lock contention: "Source file is locked — cron will handle the next chunk"
   This is normal. The 5-minute cron will pick up the root file on the next run.
2. Source file 404: "Source file no longer exists — chain complete"
   The root file was already consumed or deleted. The chain finished normally.
3. Trigger error: "Error triggering next chunk: <error>"
   The trigger_action call failed. The root file still exists with remaining rows, so the cron will retry.
4. Cron picked it up independently:
   Both the self-chain and cron attempted processing. The lock prevented duplication, and the successful chain ID appears in logs.

Q: How do I check how many observations a specific run sent?
The activity log in the Gundi portal shows per-chunk results. Each process_ornitela_file action logs:

[<chunk_name>] Processed: extracted N records, sent N observations

The activity log entry includes chain_id as structured metadata, allowing filtering of all chunks from one source file.

Q: The cleanup action deleted files it shouldn't have, or isn't deleting anything.
action_cleanup_archive runs at midnight and deletes files from archive/ older than delete_after_archive_days (default: 3 days), based on GCS timeCreated.

• Deleting too aggressively: increase delete_after_archive_days in the action config.
• Not deleting anything: ensure the CleanupArchive action config exists and the value is set correctly. Look for:
  Deleted old archived file: archive/<name>
• Errors during cleanup:
  "Could not process archived file archive/<name>: <error>"
  Individual file errors are logged as warnings and skipped. The rest continue.

Q: What do the log levels mean for this connector?
Level: INFO
Meaning: Normal progress such as chunk creation, row parsing, archiving, and triggering the next chunk

Level: WARNING
Meaning: Recoverable issues such as metadata fetch failures, transient GCS interruptions, or file locks

Level: ERROR
Meaning: File sent to dead letter, failure moving to dead letter, or unhandled exceptions

Level: DEBUG
Meaning: Encoding detection per file (visible only if LOGGING_LEVEL=DEBUG)

Other Animal Tracking