Skip to main content

SFTP push

Many ERPs import invoices from files: the connector writes one file per invoice (plus the PDF) into a folder the ERP reads, and the ERP writes an ack or error file back once it has booked or refused it. The erp_sftp_* and erp_s3 templates are this pattern, complete with master data in and the ack files back.

Because the ERP decides later, this is an async push: push_invoice reports the invoice as delivered, get_invoice_status reports the outcome. See Async push and status for how the invoice waits in Nuntiq.

The folder the ERP reads can be on an SFTP server outside Nuntiq (the connector signs in with SftpStore, as below) or on Nuntiq's own hosted SFTP server, where the ERP signs in as a Data Integration SFTP user and the connector writes into that user's home folder (see Nuntiq SFTP).

push_invoice: write the files​

"""push_invoice: one CSV + the PDF per invoice into the ERP's import folder."""
from lib.helpers import Settings, SftpStore, Outbox, Pgp, PushResult, run_push, write_csv


def run(context):
    s = Settings(context)
    store = SftpStore.from_settings(s)
    outbox = Outbox(store, s.str('export_folder', 'import'))
    pgp = Pgp(context) if s.bool('pgp_encrypt', False) else None

    def push_one(invoice):
        base = f"INV_{invoice.get_field('invoiceNumber')}_{invoice.invoice_token[:8]}"
        data = write_csv(invoice_rows(invoice), delimiter=';')      # bytes
        pdf = invoice_pdf(invoice)                              # the INVOICE_IMAGE attachment
        files = ([(base + '.pdf', pdf.read_bytes())] if pdf else []) + [(base + '.csv', data)]
        for name, body in files:                                # data file last: the PDF is already there
            if pgp:
                body, name = pgp.encrypt(body, s.secret('pgp_public_key')), name + '.pgp'
            outbox.put(name, body)
        return PushResult.delivered(external_id_1=base, external_message_1='DELIVERED')

    try:
        return run_push(context, push_one)
    finally:
        store.close()

What matters here:

  • Atomic files: Outbox.put writes under a temporary name and renames it when complete, so the ERP never imports half a file.
  • The data file last: an ERP that starts importing when the data file appears finds the PDF already in place.
  • The file name is the reference: it goes into external id 1, and the ERP's ack file names it. That is how the outcome finds the invoice again, also when the ERP never sees the Nuntiq token.
  • A retry writes the same name: the name comes from the invoice, so a retry replaces a file the ERP has not taken yet. Once the ERP has imported and moved the file, a retry writes it again: let the ERP refuse a reference it already booked (most imports do), or use Outbox(store, folder, overwrite=False), which refuses a name that is still there.
  • Batch or one by one: run_push serves the process engine (one invoice, immediately) and a scheduler task (claims the ready invoices in batches), so the ERP can get invoices as they are approved or in a nightly run.

get_invoice_status: read the ack files​

"""get_invoice_status: ERP ack files -> processed or failed."""
from lib.helpers import Settings, SftpStore, Inbox, ResultReporter, read_csv


def run(context):
    s = Settings(context)
    store = SftpStore.from_settings(s)
    reporter = ResultReporter(context)

    def handle(f):
        for row in read_csv(f.read_bytes()):                 # reference;status;document_number;message
            ref = {'external_id_1': row['reference']}
            if row['status'] == 'BOOKED':
                reporter.posted(ref, external_id_2=row['document_number'], external_message_2='BOOKED')
            elif row['status'] == 'REJECTED':
                reporter.failed(ref, 'ERP_REJECTED', row['message'])

    Inbox(store, 'ack', archive='ack/archive', error='ack/error', context=context).process(handle)
    store.close()
    reporter.stale(s.int('stale_after_hours', 72))            # warn about invoices still waiting
    return reporter.result()

ResultReporter finds the invoice by the external id stored at delivery (only invoices this connector delivered), posts the integration result, and the invoice moves on in its workflow: processed, or the failure path with the ERP's reason. A status the template does not know leaves the invoice waiting.

Variants​

  • XML or UBL instead of CSV: invoice_format in the templates (csv, xml); a UBL layout goes in the mapping's ADAPT section.
  • PGP: pgp_encrypt with the ERP's public key in pgp_public_key; pgp_decrypt for encrypted ack files with our key in pgp_private_key.
  • S3: S3Store.from_settings(s) instead of SftpStore; nothing else changes.
  • No ack files: when the ERP books on import and never answers, report PushResult.posted(...) instead of delivered; the invoice is processed as soon as the file is written.

Nuntiq SFTP​

When the ERP cannot host an SFTP server, or the customer would rather not open one to Nuntiq, the ERP signs in to Nuntiq's hosted SFTP server as a Data Integration SFTP user (the customer creates it under Integrations, tab SFTP Users), picks up the invoice files there and leaves its ack files next to them. A connector run that is given that user reaches its home folder, so it needs no host, credentials or host key. Only the store changes:

def run(context):
    s = Settings(context)
    files = context.files()                          # instead of SftpStore.from_settings(s)
    outbox = Outbox(files, s.str('export_folder', 'import'))
    ...                                              # push_one as above
    return run_push(context, push_one)

and in get_invoice_status:

    files = context.files()
    Inbox(files, 'ack', archive='ack/archive', error='ack/error', context=context).process(handle)

No close() is needed. The run reads and writes anywhere in the SFTP user's home folder. Choose the SFTP user where each run starts:

  • push_invoice from the process engine: on the integration rule.
  • push_invoice in batches and get_invoice_status: on each connector_dispatch scheduler task, under Connector Settings.

Choose the same SFTP user everywhere and use the same folder settings, so the acks land where get_invoice_status looks.

What differs from an outside server:

  • File types. The SFTP user's allowed file extensions apply to what the connector writes: allow the data format (.csv, .xml), .pdf for the invoice image and .pgp when you encrypt. A name outside the list raises PermanentError('FILE_TYPE_NOT_ALLOWED') and the invoice fails with that code.
  • Atomic writes are built in: a file appears in the folder only once it is complete, under its final name.
  • The malware scan. The ERP's ack files are readable once the scan has cleared them, usually within seconds; until then a read raises RetryableError and the inbox leaves the file for the next run.
  • Size. At most 50 MB per file.

All methods and errors: Nuntiq SFTP files.

Testing locally​

Set local_folder to a folder on your PC: push_invoice writes the files there; put an ack file in its ack folder and run get_invoice_status. The toolkit's fake Nuntiq has invoices waiting for integration in its fixtures.

For Nuntiq SFTP, create fixtures/sftp/ in the toolkit: it is the SFTP home folder of a local run, so push_invoice writes into fixtures/sftp/import/ and an ack file you put in fixtures/sftp/ack/ is read by get_invoice_status. See SFTP files locally.

Deployment notes​

  • An outside server: port 22 from Nuntiq's outbound IP address; the host key pinned. Nuntiq SFTP: nothing to open; the ERP connects to Nuntiq's server.
  • Schedule get_invoice_status (every 15 minutes is common) next to the push. An engine-pushed invoice that gets no answer before its integration rule's timeout is failed by Nuntiq, so set that timeout to what the ERP needs (a nightly import: more than 24 hours).