SFTP pull

Many ERPs and suppliers still exchange data as files on an SFTP server. This page shows the two pulls the templates are built on: invoices into Nuntiq (get_invoice, templates inbound_sftp_*) and master data from ERP export files (get_supplier and friends, templates erp_sftp_*). The same code works on the customer's S3 bucket (S3Store), on a folder of Nuntiq's own SFTP server (context.files(), see Nuntiq SFTP below) and, for testing, on a folder on your PC (LocalStore).

There are two places the files can be:

• An SFTP server outside Nuntiq (the customer's, the ERP vendor's, a supplier's): the connector signs in with its own credentials through SftpStore. Most of this page.
• Nuntiq SFTP: the customer's ERP or a supplier uploads to Nuntiq's hosted SFTP server as a Data Integration SFTP user, and the connector reads the files in that user's home folder, without credentials of its own.

Logging in

from lib.helpers import Settings, SftpStore

s = Settings(context)
store = SftpStore.from_settings(s)

from_settings reads the standard keys:

Name	Where	Example
sftp_host	setting	sftp.example.com
sftp_port	setting	22
sftp_username	setting	nuntiq_acme
sftp_host_key	setting	SHA256:3q2+7w... or ssh-ed25519 AAAAC3...
sftp_password	secret	for password login, or together with a key
sftp_private_key	secret	RSA, ECDSA or Ed25519 private key (PEM or OpenSSH format)
sftp_private_key_passphrase	secret	when the key is encrypted
sftp_certificate	secret or setting	an OpenSSH user certificate (...-cert.pub, one line) signed by the customer's CA

Which secrets you fill decides the login: a password; a key; a key and a password (servers that ask for both); or a key with a certificate. That last one is the "passwordless with certificates" setup: the customer's CA signs our public key, and the server trusts every key its CA signed.

The host key is always checked. Leave sftp_host_key empty on the first run: it fails and prints the fingerprint the server offered. Confirm it with the server's owner (they read it with ssh-keygen -lf on the server's host key), then paste it. A changed host key fails every run until someone looks at it.

Making a key pair for a customer:

ssh-keygen -t ed25519 -f nuntiq_acme -C nuntiq_acme

Send nuntiq_acme.pub to the server's owner, paste nuntiq_acme (the private key) into the sftp_private_key secret. One key pair per customer.

Taking files one at a time

from lib.helpers import Inbox, InvoiceIntake

def run(context):
    s = Settings(context)
    store = SftpStore.from_settings(s)
    intake = InvoiceIntake(context)                    # target capture, einvoice or json
    inbox = Inbox(store, s.str('import_folder', 'outgoing'),
                  archive=s.str('archive_folder', 'outgoing/archive'),
                  error=s.str('error_folder', 'outgoing/error'),
                  pattern=s.str('file_pattern', '*.pdf'),
                  min_age_seconds=s.int('min_age_seconds', 60),
                  max_files=s.int('max_files', 100), context=context)
    counts = inbox.process(lambda f: intake.submit_file(f.name, f.read_bytes()))
    store.close()
    return intake.summary.result(**counts)

For each file, oldest first:

1. A file younger than min_age_seconds is skipped: it may still be uploading.
2. With a processing folder set, the file is moved there first, so two runs never take the same file.
3. Your function gets the file (f.name, f.read_bytes(), f.text()).
4. On success the file moves to the archive (a time suffix is added when the name is already there). On an error it moves to the error folder with <name>.error.txt next to it, saying why. A RetryableError (Nuntiq or the target is down) puts the file back and stops the run; the next run tries again.

One file at a time keeps the run inside its 4 GB of disk and 512 MB of memory. max_files caps a run; a backlog is taken over several runs.

A source you may not change

When the connector may only read the folder (no archive, no delete), use ReadOnlyInbox(context, store, folder, stream='invoices'). It remembers in a delta cursor which files it took (by modification time and name), so the next run takes only new ones.

Master data from export files

from lib.helpers import read_csv
from lib.objects.supplier import SupplierLoad

def handle(f):
    load = SupplierLoad(context, source_system=s.source_system)
    for row in read_csv(f.read_bytes()):            # encoding, BOM and delimiter detected
        map_supplier(load, row)
    load.save_all(full_load=True)                    # a full extract: removes suppliers no longer in it

Inbox(store, 'export', archive='export/archive', error='export/error',
      pattern='suppliers*.csv', context=context).process(handle)

full_load=True fits a file that always holds every record (the usual ERP export). For a file with only the changes, use save_all() and leave removals to explicit deletes. read_xlsx and read_xml read the other formats; files encrypted with PGP are decrypted with Pgp(context) first.

Nuntiq SFTP

When the customer's ERP or a supplier has no SFTP server of its own, or the customer would rather not open one to Nuntiq, they upload to Nuntiq's hosted SFTP server instead. The customer creates a Data Integration SFTP user (Integrations, tab SFTP Users) and gives its sign-in to whoever uploads. The connector run that is given that user reaches its home folder: there is no host, password, key, host key or IP allowlist in the connector's settings.

"""get_supplier: the ERP's supplier export, uploaded to Nuntiq SFTP."""
from lib.helpers import Inbox, RunSummary, Settings, read_csv
from lib.objects.supplier import SupplierLoad


def run(context):
    s = Settings(context)
    summary = RunSummary(context)
    files = context.files()                            # the SFTP user's home folder, instead of SftpStore.from_settings(s)

    def handle(f):
        load = SupplierLoad(context, source_system=s.source_system)
        for row in read_csv(f.read_bytes()):
            map_supplier(load, row)
        load.save_all(full_load=True)

    counts = Inbox(files, s.str('import_folder', 'export'),
                   archive=s.str('archive_folder', 'export/archive'),
                   error=s.str('error_folder', 'export/error'),
                   pattern=s.str('file_pattern', 'suppliers*.csv'),
                   min_age_seconds=s.int('min_age_seconds', 60),
                   context=context, summary=summary).process(handle)
    return summary.result(**counts)

Then, on the connector_dispatch scheduler task, under Connector Settings, choose the SFTP user. The run reads and writes anywhere in that user's home folder, so the inbox can move files to the archive. If the user's folder holds more than this exchange, point the folder settings (import_folder and the others) at the part the connector should use. To leave the files where they are, use ReadOnlyInbox(context, files, 'export', stream='supplier_files').

Invoices do not need a connector on Nuntiq SFTP: an Invoice Upload SFTP user sends what is uploaded straight into invoice processing. A connector with InvoiceIntake on Nuntiq SFTP files fits when the files need work first (an archive to unpack, a format to convert, PGP to decrypt).

What differs from an outside server:

• The malware scan. A file is readable only after the scan cleared it; until then a read raises RetryableError and the file waits for the next run. min_age_seconds of 60 or more leaves fresh files alone anyway. A file the scan does not clear is taken out of the folder before the connector sees it.
• File types. The SFTP user's allowed file extensions apply to the uploader and to the connector. Allow .txt as well if you want the inbox's <name>.error.txt notes.
• Size. Uploads are at most 50 MB per file.
• Backups. files.backup(path) moves a finished file to the SFTP user's backup area, kept for the user's backup period and restorable by the customer, as an alternative to an archive folder.

All methods and errors: Nuntiq SFTP files.

Testing locally

Set local_folder (a setting of every file template) to a folder on your PC and put sample files in its import folder. The template then uses LocalStore on that folder instead of the server; folders, archive and error moves behave the same.

For Nuntiq SFTP, create the folder fixtures/sftp/ in the toolkit (here fixtures/sftp/export/) and put the sample files in it: fixtures/sftp/ is the SFTP home folder of a local run. See SFTP files locally.

Deployment notes

• An outside server must accept connections from Nuntiq's outbound IP address on port 22. Another port can be opened on request. Nuntiq SFTP needs nothing of this kind; the uploader connects to Nuntiq's server.
• Schedule the entrypoint with a scheduler task (connector_dispatch), for example every 15 minutes.
• Duplicates: Nuntiq answers a file the connector sent in the last 24 hours with the earlier result, but archiving the file is what keeps it from being sent again a week later.