@Mathias Dahl 
of course for test I selected a doc class with approx. 250 documents behind and it went through within approx. 5 mins, however seeing the counts of the most heavy doc classes I don’t think we can run it for ¾ million of documents in 1 go, I guess we will have to split the transfer into some batches (probably by date) which is unf. not possible in the screen so most probably we will have to do it from backend. More over some of the documents have been archived in backup table in different DB schema, so not all the docs are really having file behind.

@ORFJAN 

Okay, thanks for explaining and testing. So a little less than 1 document per second. Not very fast, perhaps IFS Connect can be tweaked to allow parallel executions from several queues/processes (whatever they might have there)?

100,000 documents should take ~ 100,000 / 50 minutes = 100,000 / 50 / 60 hours = 333 hours ...

I think you should consider using other means to migrate those documents. There is probably not much we can do with that feature itself. We have no other option than to use IFS Connect there, and the per-document operation means there will be some overhead per document, which adds up.

In theory, the built-in solution (Transfer Documents) could be enhanced and we could transfer documents in batches, or send information via IFS Connect in batches. That might speed things up, but it's something that needs to be prioritized and planned, and it's not an option that's available to you right now.

The FS mig tool is also per-document, but it's more efficient since it does not use IFS Connect. It reads the file from the database and stream that into the FS directly.

Again, another option is to build something more "custom" that can take shortcuts that work for you. A "problem" with generic solutions like Transfer Documents and even the FS mig tool is that it needs to think about all scenarios, all customers. General/generic is often more complex and slower than specific/custom...
 

Hi @Mathias Dahl, 
just quick update from our end on documents transition to Azure FSS.
We have analyzed the counts and volumes we are supposed to transfer.
 

We also did some further testing on DB transfers, due to the high volumes we are not able to transfer by doc class, so we created following query to do the transfer by year, month and doc class.

Then we adjusted script being used by the screen “Transfer Documents” and created own PL/SQL script to transfer docs by year, month and doc class and it seems to be working fine. So far we have migrated approx. 60K documents.
Since it’s being executed via connect we have been focusing on performance too, 1 batch with 20K docs takes approx. 1 hour.

I suggest you enhancing the screen “Transfer Documents” to support year, month, doc class and batch size range too, I guess it might be appreciated by other customers who has high volumes of documents.

Hi @Mathias Dahl,
I’m also sharing custom field expression to populate Azure Base64 file name on IFS screen below.

Any thoughts/comments on previous post?
BR Jan
 

utl_raw.cast_to_varchar2(utl_encode.base64_encode(utl_raw.cast_to_raw(substr(file_name,1,instr(file_name,'.')-1))))||'.'||Edm_Application_Api.Get_File_Extention(file_type)
 

@ORFJAN 

Hi,

> Since it’s being executed via connect we have been focusing on performance too, 1 batch with 20K docs takes approx. 1 hour.

That's not bad, actually

> I suggest you enhancing the screen “Transfer Documents” to support year, month, doc class and batch size range too, I guess it might be appreciated by other customers who has high volumes of documents.

Agreed! I'm looking forward to see an idea added in the Ideas section of IFS Community then The challenge is to find a way to control what to migrate in more detail and which is useful to many customers. Would batch sizes (being able to limit the number of files) only be enough?

> I’m also sharing custom field expression to populate Azure Base64 file name on IFS screen below.

Clever Thanks for sharing!