File Copy Jobs from SFTP to the LMS

This article explains the process of copying files from SFTP to the LMS, describes how to resolve issues relating to ‘files received late or not imported’, and explains how to set up the process for a new customer during onboarding.

Three audiences are addressed: Customers (what you deliver and what to check first), Support agents (how to diagnose and resolve an incident ticket), and the Project / Onboarding team (how to configure the process for a new customer).

1. How the File Process Works

There are three stages, run by at least two scheduled jobs. A file must be delivered before it can be copied, and copied before it can be executed.

The intended timing relationship is:

The gaps ('buffers') between stages exist on purpose: they absorb a late delivery or a slow transfer so the next stage never acts on a missing or half-written file.

2. Time Zones and Daylight Saving Time (the Key Concept)

The problem appears when different stages follow different time references. If the copy job runs in UTC (fixed all year) but the delivery and/or execution follow local time (which shifts by one hour at each DST changeover), then the gap between the stages changes by one hour twice a year – even though no one changed the schedule.

Worked example:
Viewed in local time, the intended schedule is Delivery 02:00, Copy 04:00, Execution 06:00 (2h + 2h buffers). If the copy job is pinned to UTC while the others follow local time, a DST changeover makes the copy job’s apparent local time move by an hour while the others stay put, so the same schedule looks like one of these:

The two rows correspond to the two DST changeovers of the year (clocks forward vs. back). The exact direction depends on the server’s base time zone and on when the schedule was first set, so confirm it against your own logs rather than assuming which row you get.

Why this causes “late” files: In the second row the copy finishes only one hour before execution, so any extra delay in delivery or in the copy itself can push the file past the execution time – whereas the other half of the year the same delay was harmlessly absorbed by the 2-hour buffer.

3. For the Customer

3.1 Customer’s Responsibilities

• Deliver the file to the SFTP server by the agreed deadline (time X), in the agreed directory, with the agreed file name pattern and format.

• Ensure the upload is complete before the copy job runs – see the safe-upload tip below.

• Use the agreed time reference (we recommend UTC end-to-end). If you schedule your upload in local time, remember the buffer to the copy step appears to change by one hour at each DST changeover, so upload as early as you safely can.

3.2 The Schedule (Fill in and Share with the Customer)

Our Copy step runs on <UTC> and does not change for Daylight Saving Time.

3.3 File Examples

CSV Example

File name : <import_YYYYMMDD.csv>

Delivered : by <02:00 UTC>

Format    : <UTF-8, comma-delimited, header row required>

Sample    :

XML Example

<!-- File name: <mapping_YYYYMMDD.xml> -->
<!-- Delivered: by <02:00 UTC> -->
<!-- Format: <UTF-8, well-formed, valid against schema name/version> -->
<attributeMapping isReference="false">
       <mapping sourceField="ID" clixField="EXT_ID_CSV"/>
       <mapping sourceField="Login" clixField="LOGIN"/>
       <mapping sourceField="Firstname" clixField="FIRSTNAME"/>
       <mapping sourceField="Lastname" clixField="LASTNAME"/>
</attributeMapping> 

3.4 If your Import Looks Late or Wrong

1. Confirm your file is on the SFTP server with the correct name, directory and format.

2. Confirm the upload completed (not zero bytes, not still transferring).

3. Confirm you delivered before the deadline, in the agreed time zone.

4. If all of the above are correct, raise a ticket with: the file name, the exact upload time and time zone, and a screenshot or listing showing the file on the SFTP server.

4. Support Agent Responsibilities (Incident Handling)

4.1 First Response – Collect These Facts for one Failing Run

• File name and the timestamp on the SFTP server (when the upload actually finished)

• Timestamp the copy job started and finished (copy-job log)

• Timestamp the execution job ran (execution-job log)

• Whether the file was present, missing, or incomplete when the execution job ran

These four facts alone usually identify the failing stage.

4.2 Diagnostic Flow

 Was the file present and complete when the execution job (Z) ran?
 
  MISSING       -> copy job didn't finish in time     -> 4.3 Copy stage
                   (first confirm it was delivered     -> 4.3 Delivery stage)
  INCOMPLETE    -> copy started before upload finished,
                   or copy was interrupted             -> 4.3 Copy stage (partial file)
  PRESENT but
  PROCESSED
  WRONGLY       -> format / encoding / schema issue    -> 4.3 Execution stage
  PRESENT and
  JUST LATE     -> a buffer was eaten: check DST (Sec 2)
                   and copy duration                   -> 4.3 Copy stage

4.3 Stage-by-stage Checks

Delivery (Stage 1 – Customer side)

• Compare the file’s timestamp on the SFTP server with the agreed delivery time X (state clearly whether X is UTC or local – see Section 2).

• Confirm the file is in the expected directory with the expected name pattern. A case difference, wrong sub-folder, extra prefix/suffix or wrong extension makes the copy job ‘see nothing.’

• Look for zero-byte or truncated uploads (a failed customer transfer).

• If deliveries are chronically late, this is an SLA / customer-communication issue, not a technical fault.

Copy (Stage 2 – the job you control)

Did it run at all?

• Ensure the cron entry exists, is uncommented and correctly spelled in the crontab of <the service account> (<crontab -l -u service-account>).

• Check that the cron daemon / service is running (<systemctl status cron/crond> or platform equivalent).

• Verify the cron / system log shows the job was invoked at time Y (<journalctl / /var/log/syslog> – path varies by platform).

• Confirm the schedule fields really mean what you think (minute, hour, day-of-month, month, day-of-week). A frequent trap is a job that only runs on certain weekdays, so weekend deliveries sit untouched.

Did it run but fail or do nothing?

• Check exit status / errors in the job’s own log (<path/to/copy-job.log> – capture stdout and stderr there).

• SFTP connectivity / authentication: Can the service account reach the SFTP host and log in right now? Test manually with the same key/credentials the job uses (<sftp -i key user@sftp-host>). Watch for expired passwords, rotated SSH keys, changed host keys / known_hosts, which silently break automated logins.

• File not found at copy time – the job ran but the file wasn’t there yet → delivery-timing problem (Stage 1).

• Destination problems on the LMS side – target directory missing, wrong permissions, disk full / quota exceeded, or a read-only mount (a full disk produces partial or zero-byte files).

• Wrong source / target path after an environment change (e.g. a folder renamed during migration).

Did it run on time but take too long?

• Compare copy start and finish timestamps. If the copy itself now takes much longer, look at file size growth, number of files per run, SFTP server load, network throughput / latency. Strong candidate for the ‘worse over time’ pattern.

• Check whether runs overlap (a slow run still going when the next starts). Consider a lock file / single-instance guard.

Partial / incomplete file (race with the upload)

• If the copy starts while the customer is still uploading, it copies a truncated file and the execution job then processes a broken file.

• Implement mitigations (match to what was agreed at onboarding): customer uploads to a temp name then renames; customer writes a marker / .done file the copy job waits for; the copy job checks the file size is stable across two reads a few seconds apart; copy to a temp name on the LMS side and rename into place only after a complete transfer.

Execution (Stage 3 – inside the LMS)

• Confirm the execution job’s schedule (Z) and its time-zone reference, and re-check the buffer against Y in light of Section 2.

• If the file was present and complete but processed incorrectly, suspect content / format rather than timing:

  • Check CSV vs XML – the file type matches what the execution job expects, and both formats are actually supported where both are used.

  • Verify encoding (UTF-8 vs UTF-8-with-BOM vs a regional code page), line endings, delimiter, header row, column order.

  • Validate schema / structure for XML (Well-formed? Valid against the expected schema / DTD?).

• Confirm the execution job is pointed at the same directory the copy job writes to.

4.4 The DST / Time-zone Check

1. Identify what time zone is each machine set to (SFTP host, copy-job host, LMS host)?

2. Identify what time zone is each job configured in? A crontab can override the system zone with a TZ / CRON_TZ entry – check the copy job and the execution job.

3. Ensure all three stages use the same reference. The cleanest design is UTC end-to-end so the buffers never change.

4. Verify whether the customer’s delivery times are stated in UTC or local time in the schedule / SLA you share with them. A mismatch here is a frequent root cause.

4.5 Verification Checklist

• File present on SFTP at X (server-side timestamp captured)

• File complete on SFTP (not zero-byte / not still uploading)

• Copy job invoked at Y (cron / system log)

• Copy job exited successfully (job log, exit status)

• SFTP login still valid (key / password / host key)

• Destination writable, with free space

• Copy finished comfortably before Z (start / finish timestamps)

• File present and complete on the LMS when execution ran at Z

• All three stages on the same time-zone reference (UTC end-to-end recommended)

• File format / encoding correct for the execution job (CSV and XML cases)

4.6 Escalation

• If the cause is late or malformed delivery → customer / account contact

• If the cause is SFTP credentials / connectivity → Infrastructure / Hosting team

• If the cause is the LMS execution job or import logic → Incident Management / Hosting team

• If the cause is a time-zone / DST mismatch in the schedule → Incident Management team

5. Project Phase – Onboarding a new Customer

5.1 Setup Checklist

1. Provision the SFTP account / credentials (key or password) for the customer and confirm they can connect.

2. Agree the directory structure and file naming convention.

3. Agree the file format(s) – CSV, XML, or both – and the encoding and schema.

4. Agree the safe-upload method (temp-name-then-rename, or a marker / .done file).

5. Agree the delivery deadline (X) and confirm the time-zone reference. Recommend UTC end-to-end so DST never changes the buffers.

6. Configure the copy job (Y) with an adequate buffer after X.

7. Confirm the execution job (Z) schedule and that its buffer after Y survives the worst-case one-hour DST shift.

8. Set up logging (delivered / copy-start / copy-finish / executed timestamps) and alerting on a shrinking buffer, not only on outright failure.

5.2 Agree the Schedule and Time Zone (Sizing)

Pick X, Y, Z so that even the reduced buffer (after a one-hour DST shift, per Section 2) is larger than the largest realistic delivery delay + copy duration. Keeping the two buffers equal and generous is the simplest way to stay safe across both DST changeovers.

5.3 Validate the First Files

1. Run a full test cycle with a representative CSV file: deliver → copy → execute, checking timestamps and the file at each stage.

2. Repeat with a representative XML file (well-formed and schema-valid).

3. Verify the safe-upload method actually prevents a partial copy (e.g. test an interrupted upload).

4. Capture the actual buffers observed and compare them to the agreed schedule.

5.4 Go-live Checklist

• SFTP access confirmed by the customer.

• Naming convention, directory, format and encoding documented and agreed.

• Safe-upload method implemented and tested.

• Schedule (X, Y, Z) and time-zone reference documented in section 3.2 and shared with the customer.

• Buffers verified to survive a one-hour DST shift.

• Logging and alerting in place.

• Test CSV and XML files processed successfully end-to-end.

• Per-customer values recorded in Section 6.2.

6. Reference

6.1 Glossary

6.2 Per-customer / Per-environment Values (Fill in)