Document offloading from FTP to Azure Blob storage

Hi,

The support for FTP and Shared repos is mentioned directly after the section you linked to:

To be clear: it IS possible to migrate from FTP and Shared repositories, but it's not as convenient as when using Database storage since you need to populate that text file plus make sure the tool has access to read the source files.

If it's an option to pre-migrate from FTP/Shared into the Database storage in the source environment, and then use the tool, things might go smoother, but that's an extra step that takes time of course, and it will make the database bigger.

I encourage you to consult the documentation, perhaps read it several times, perhaps even try a small test migration to get a feeling for the tool works. The tool does a lot for you, compared to you doing the migration using some custom method, but it still means work and using it requires planning.

If you have any questions not covered in the documentation, or if anything in there is not clear, just come back here and ask.

Good luck!

PS. Try to use the name File Storage, Cloud File Storage or IFS Cloud File Storage. While Azure Blob Storage is used in the background (in Cloud deployment), it is in a way an implementation detail. IFS Cloud might in theory switch to a different storage option (no plans as far as I know).

Hi @Mathias Dahl 

Please help me specify the FTP location and file details in the FsMigTool.cmd script (for files in FTP: ftp://FTP_USER@IP_ADDRESS/CLASS_FOLDER/​​​​​​​ or ftp://FTP_USER@IP_ADDRESS/​​​​​​​ ) as it is not stated in the link below.

https://docs.ifs.com/techdocs/24r1/030_administration/210_cloud_file_storage/fsmigtool/#format_of_the_document_information_text_file

I tried with the following command line arguments and it doesn't work.
Try 1:
  fsmigtool --docsfromfile TEST.txt --docsfromfilesource  FOLDER --dbconn "jdbc:oracle:thin:@//XXX.ifscloud.net/XXXMGR" --dbuser IFSAPP --dbpass XXX--cloudurl https://XXX-XXX.ifs.cloud --cloudclientid IFS_fsmigtool --cloudclientsecret XXX --cloudrealm XXXXXX1 --logmigtable

----------File TEST.txt-----------

3682697,1,A1,ORIGINAL,1,00XXX004-3682697-1-A1-1.PDF, ABCINV2932090.pdf, ftp://FTP_USER@IP_ADDRESS/CLASS_FOLDER/

Regards,

Oshada

@Oshada Samarasinghe 

You don't - the FS mig tool doesn't have support for accessing files from the FTP server (check the documentation for the tool). You need to make sure the files are copied to some place where the tool can read them, and then list each file path and name in the text file (also documented).

The tool has "somewhat" support for FTP and Shared repositories, you can say. Again, as documented. Have a read, especially the introduction and what to expect from the tool, then come back and ask more questions.

In short, you should provide path and file names to a local drive and folder or a network share, where the file can be accessed by the tool.

Hi @Mathias Dahl ,

Thanks for the support so far, I noted that you have updated below community post:

UPDATE, November 2024

Although the support for Shared and FTP repositories are still deprecated (it will be removed in a coming release), it's still available. However, since 24R1 it's disabled by default. Refer to our release notes for information on how to enable it again. Before you do that though, please consider migrating to one of the other options we have: Database or File Storage.
 

According to the updated section you added, if a system administrator can change the value of the object property FTP_SHARED_OVERRIDE in the object LU EdmLocation from DISABLED to FORCE_ENABLE_DEPRECATED_FEATURE, FTP repository support is enabled.

All of the data in APP9 is now transferred to FTP and kept there because Cloud Storage is not supported for IFSAPP9. Can we use this FTP by forcing it to be enabled for our go live, and then, after the go live, can we choose File Storage and use a cloud tool to migrate the documents to File Storage from FTP?

Best Regards,

Oshada

@Oshada Samarasinghe What you suggest might work but it's nothing we designed the File Storage migration tool for and nothing we tested, so be prepared for some trial and error.

Just to verify, this is Remote deployment, right?

@Mathias Dahl Thank you for your input thus far. We are hosted on the IFS managed cloud. I'm trying now setting up a network share with the files from FTP so that I may begin moving files from FTP to Cloud File Storage.

/Oshada

FTP is NOT supported in Docman in IFS' managed cloud!

You have only two options: use Database storage, or File Storage, or a combination.

The FS mig tool can help migrating into File Storage from the source database, and the source database/installation can use Database, FTP or Shared repositories, but works best with Database. There are many discussions around this here on IFS Community. 

​@Mathias Dahl :

I've read this section of the document(Cloud File Storage Migration Tool - Technical Documentation For IFS Cloud), but when I try to use the file path and name in the text file, I keep getting errors. 

Since I've configured FTP as the VM's nextwork share folder (scenario 2), I'm testing scenario 1 by trying to send some files from a folder that is located in the VM (I've added some images). Do you know what causes this error and how to fix it? I'm getting a similar error in all scenarios below.

Below are the steps followed in Scenario 1:
1.Selected valid documents (Cloud File Storage Migration Tool - Technical Documentation For IFS Cloud)

2. Copied those two files from FTP location to below 00FIN001 folder

3. Created a file with comma separated values OOFIN001NOV.txt

4. command used
   fsmigtool --docsfromfile OOFIN001NOV.txt --docsfromfilesource 00FIN001 --dbconn "jdbc:oracle:thin:@//XXXtest01.ifscloud.net/XXXXXX" --dbuser IFSAPP --dbpass XXXXXXXXXXXXXXX --cloudurl https://XXXX-cfg.ifs.cloud --cloudclientid IFS_fsmigtool --cloudclientsecret XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX --cloudrealm XXXcfg1 --logmigtable
5. Error in Unsupported file source type 00FIN001. and error in logfile [2024-11-20 16:03:44] [WARNING] Error: Unsupported file source type 00FIN001. 
   
   Below are the steps followed in Scenario 2
6. Selected valid documents (Cloud File Storage Migration Tool - Technical Documentation For IFS Cloud)

   

7. Connected ftp201 to the VM (As per the internal infra team) and added as a network shortcut 

   

   8. Created a file with comma separated values OOFIN002NOVFTP.txt

   

   9. command used
   fsmigtool --docsfromfile OOFIN002NOVFTP.txt --docsfromfilesource FTP201 --dbconn "jdbc:oracle:thin:@//XXXtest01.ifscloud.net/XXXXXX" --dbuser IFSAPP --dbpass XXXXXXXXXXXXXXX --cloudurl https://XXXX-cfg.ifs.cloud --cloudclientid IFS_fsmigtool --cloudclientsecret XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX --cloudrealm XXXcfg1 --logmigtable
   
   10. Error in Unsupported file source type 00FIN001. and error in logfile [2024-11-20 16:03:44] [WARNING] Error: Unsupported file source type 00FIN001. 
   
   Thanks,
   /Oshada

Hi ​@charith95 ,

Thank you for pointing this out. I was able to fix Unsupported file source type error with this. I'm still  getting different errors when I try to migrate files from the FTP folder that I've configured as a network share, like "File 00FIN002-1669524-1-A1-1.MSG Failed to upload. "File path or file does not exist." 

Hi ​@Oshada Samarasinghe ,

Make sure you give the details in the text file correctly, you need to provide the file details that’s inside your network share correctly. This error is related to a wrong file name.