[Bug]: [E-Document] Receive Single Document procedure

[fmoric]

Describe the issue

I have been looking at ReceiveSingleDocument implementation, and as i can see that all of standard implementations (Pagero, Avalara etc...) use something like:
EDocumentErrorHelper.LogSimpleErrorMessage

to log error, and attach it to that E-Document, and that is ok, I mean, like, you want to try to download document, but something happens and you want to log error. The problem is that if you log error, when downloader exits, it measures success by counts of errors:

        ReceiveContext.Status().SetStatus("E-Document Service Status"::Imported);
        ErrorCount := EDocumentErrorHelper.ErrorMessageCount(EDocument);
        RunDownloadDocument(EDocument, EDocumentService, DocumentMetadata, IDocumentReceiver, ReceiveContext);
        Success := EDocumentErrorHelper.ErrorMessageCount(EDocument) = ErrorCount;

And if it is not success, the document is deleted:

 if ReceiveSingleDocument(EDocument, EDocumentService, DocumentMetadata, IDocumentReceiver) then begin
                // Insert shared data for all imported documents        
                EDocumentLog.SetBlob(EDocument."File Name", "E-Doc. File Format"::Unspecified, DocumentMetadata);
                EDocumentLog.InsertLog(Enum::"E-Document Service Status"::"Batch Imported");
                EDocumentLog.InsertIntegrationLog(EDocument, EDocumentService, ReceiveContext.Http().GetHttpRequestMessage(), ReceiveContext.Http().GetHttpResponseMessage());
            end else
                EDocument.Delete();

Am i missing something here?

Expected behavior

For E-Document to remain so that logged error, that is attached to it to be seen.

Steps to reproduce

Try to receive documents, and download it with error, or as i see LogSimpleErrorMessage.

Additional context

No response

I will provide a fix for a bug

• I will provide a fix for a bug

[georgepomazkov]

I'm pretty sure that the idea is that if there is a severe error, then the end user should not be able to see the invalid E-Document.
In these cases you should store the data in an intermediate table (e.g. "E-Documents to Download") and try to fetch them in the background before putting them in the E-Documents queue for the user to see.

Also, if you still want to keep the E-Document entry, you can call "E-Document Error Helper".LogWarningMessage() instead of LogSimpleErrorMessage(), which should not cause ReceiveSingleDocument() to return false.

[fmoric]

1. There is no intermediate table.
2. Why is LogSimpleErrorMessage then implemented in Microsoft E-Document solutions if the message cannot be seen? I see that it is in DownloadDocument procedure in Microsoft's Avalara, Pagero, Outlook...

[georgepomazkov]

Just to clarify: this is just my interpretation of what was intented by Microsoft.

There is no intermediate table.

Correct, I meant that you can implement such table yourself if needed.

Why is LogSimpleErrorMessage then implemented in Microsoft E-Document solutions if the message cannot be seen?

Well, this procedure not only saves the error in the Error Message table but also logs it to telemetry. Both of those could be useful even if there in no E-Document record, so I guess this is more of a good practice approach.

---

Generally, the e-document publishing web service is expected to keep the document available until you send some sort of a confirmation to the web service:

1. The web service publishes the documents for you to download.
2. You get a list of documents to be downloaded via IDocumentReceiver.ReceiveDocuments()
3. Then for each document:
   3.1. You download it via IDocumentReceiver.DownloadDocument()
   3.2. You confirm the receipt of the document via IReceivedDocumentMarker.MarkFetched()
   3.3. The web service removes the document from the download queue - you will not see it the next time you call IDocumentReceiver.ReceiveDocuments().

In this scenario if you fail to download the document at step 3.1., the web service should still keep it available so you can obtain it again at step 2. And so there is no reason for you to generate an E-Document record for a failed download, since it is expected to be downloaded the next time.

If the web service you integrate with does not support this behavior, or you want to display the download errors to the user, then you might want to implement some logic to handle the failed downloads, as I mentioned earlier.

[fmoric]

Yea, I considered telemetry, and that it makes sense, i see it now, but it still doesn’t feel quite right, especially when comparing it to the older version V1, which only includes AfterInsertImportedEdocument. It seems like there could at least be a choice for deletion. The code between V1 and V2 is similar, and removing document without any choice feels like a big step between V1 and V2 for what is essentially the same step in the process.

But ok, if there will be no other input, i will close it.

[fmoric]

I understand the implementation and telemetry aspects, but the issue is this, we had a client who downloaded 9 out of 10 documents. Because of this deletion behavior, they weren’t aware that there was a problem with the provider for the 10th document until late. As I mentioned, this behavior was unexpected, especially since in V1 the error would have been visible.

[Groenbech96]

Hey @fmoric.
Thanks for the discussion.

What type of issue was with the document that did not download?
Could it be handled in the connector implementation?

[fmoric]

It was a technical problem on providers side, but problem here is that user did not see it because deletion is "automated", I think there should be more freedom, especially when deleting documents. Log in telemetry is something you do not look at on daily basis. No, this cannot be handled differently, maybe if as @georgepomazkov said use new table to save it, but then I do not know how would I use Download code again, lots of hoops to go through, for something simple to handle, and not to delete.

On topic of autodeleting, when searching for duplicated, there is no way to skip that, or put my own duplicate filter, in my country you can change and send same document with a thing called copy indicator, and I had to do a runaround, witch I did not like, to skip deleting.

Something to think about.