Deprecation of the FTP and Shared repository types in Docman in 24R1 - a heads-up

  • 5 October 2023
  • 57 replies
  • 2004 views
Mathias Dahl
Rank
Userlevel 7
Badge +30

Deprecation of the FTP and Shared repository types in Docman in 24R1 - a heads-up

 

This is a heads-up on IFS deprecating support for the FTP and Shared repository types in Docman in IFS Cloud 23R2, for complete removal in 24R1.

If you have customers using these two repository types today, you should plan for migrating to Database or File Storage.

If you are planning to implement this in Remote Deployment in IFS Cloud right now, you should reconsider, and use File Storage or Database.

In Remote deployment, File Storage will still have support for local/external storage (outside the database) by supporting SMB/network shares.

 

BACKGROUND

 

Back in the day, Docman started out with support for "external" file storage via our support for FTP and Shared (folders) as the only repository types. Later, Database storage was added upon request from customers. And, much later (IFS Cloud 21R2), IFS Cloud File Storage was added as yet another repository option where, initially, File Storage was limited to Cloud Deployment where Azure Storage is used as the backend.

Recently, IFS Cloud File Storage got support for Remote Deployment, where a shared network folder can be selected for storing files.

 

TODAY

 

We are now in a position where we can remove the support in Docman for FTP and Shared repositories since every customer, from IFS Cloud 23R2 and later, will have two options for document file storage: "internal" (inside the database) and "external" (outside the database).

Read about the capabilities of IFS Cloud File Storage here, before you continue reading:

https://docs.ifs.com/techdocs/23r1/030_administration/210_cloud_file_storage/#introduction

Some customers in Remote Deployment might want to use their FTP servers or multiple existing shared network folders. Although there is of course a benefit to this, we think that one "external" file storage option should really be enough and customers should be able to change to use fewer options.

We are considering deprecating the Database repository option as well at some point, but that has not been decided yet. If you have use cases where you think this is strictly needed, let us know.

 

WHAT HAPPENS NOW?

 

This deprecation will be communicated officially via the release notes in IFS Cloud 23R2 GA, but we wanted to communicate this as early as possible, to avoid unnecessary migration work.

Documentation on migration paths will be available soon. We already have migration tools for migrating customers from Database, FTP, and Shared repositories to File Storage and we will work on making them even better.

Comments, questions?

 

/Mathias

57 replies

Mathias Dahl
Rank
Userlevel 7
Badge +30

A more technical question related to the subject. It is specific for Remote Deployment but also interested for Cloud Deployment.

The document files are temporarily stored in the Application Server when working with IFS Applications 10 and older (at least IEE version).

How does it work in IFS Cloud?
Will the documents be temporarily stored somewhere during the process or will they “only” pass by the Kubernetes Cluster on the way to the client? 

Hi Stefan, thanks for asking here.

The file content is streamed all the way, we are not creating any temporary files. All parts of the flow that's controlled by IFS is written in Java and there, instead of working with files, we are working with input and output streams. While it was possible in Apps 10, it's much harder to handle temporary files in Kubernetes, especially if they are large, since each container has a fixed amount of storage space.

 

/Mathias
M
Rank
Badge +1

Hi Mathias Dahl

We are a larger company that is currently in the process of porting to ifs cloud.
(remote managed solution, servers are on premise)

We are using DOCMAN heavily and today there are more than 4 million files (Size over 3 TB) increasing daily.
Our today Solution is relaying on Shared Folder (Shared repository). The main problems of changing to "IFS Cloud File Storage" would be:

a) no file server likes to have more than some 100k files in the same folder
b) we see high risk to be non-compliant with government orders and regulations, if we are no longer able to separate Files in different folders and locations,  which could cause in massive negative impact to our business
c) we have documents on some different (older) ifs systems where we can use the "old" servers without need to duplicate the physical files
d) we have a long-term archive solution that represents the files as UNC path\file

So basically, if the possibility to use shared folders (Shared repository) is removed with 24/1 we will run into a deep problem that will seriously impact our project in a most critical way.

what are the chances, that we still can use shared folders and what are the required measures on IFS side to ensure this?

Regards

Stefan

B
Rank
Badge +2

@Mathias Dahl

 

I've just come from a meeting where we discussed exactly this topic. 
Concentrating on repository type "file storage" may well be the right way from a technical point of view. But the limitations and restrictions that this decision entails are significant for our customers. I fully agree with the existing opinions that i have read.
But this change is only half of the truth. Our customer projects always have the same requirements.
It is about internal order documents created by IFS, e.g. order confirmation. This document (PDF) should be automatically attached to the object (customer order). IFS offers "Report Rules" to use for this purpose. This also works well as a standalone solution. But Report Rules do not work together with repository type "File Storage". It is currently not possible in IFS CLOUD to implement a continuous process flow with standard functionality. Creation and printing of an order document incl. attachment to an object and then a transfer to a repository (file storage). We practised for a long time in our last customer project and then we had to go back to "Shared Repository". You may can imagine that our customer wasen’t really happy.
My expectation is, that IFS will deliver a standard solution for this requirement in a short time.

Thanks and best regards,
Beat

 

Mathias Dahl
Rank
Userlevel 7
Badge +30

Thanks @BytBeatbW 

We are aware of the fact that not all functionality that worked with a Database repository in Docman works with File Storage. I think there is a plan to handle each gap there.

As for the problem you mention (archiving reports in Docman), depending on your usage, it should not be a problem to keep those types of documents in the database, even if the bulk of your documentation reside in File Storage. I say "depending on your usage" because it might be that you are archiving so many documents that it becomes a problem with the database size growing too much. Archiving reports doesn't strike me as a process that should fill the database too much though, but then again I don't know how many reports your customers are archiving…

This issue has little to do with the deprecation of FTP and Shared repository types though, and much more to do with the long term focus on File Storage.

Although it's related, it's not the same thing. We have not decided to remove the Database repository type yet.

 

/Mathias
Mathias Dahl
Rank
Userlevel 7
Badge +30

@maxonbs 

 

> a) no file server likes to have more than some 100k files in the same folder

I have been here a long time and I have seen that problem, at least in the old days.

These seems to be the limits of file systems in use today:

File System    Maximum Files per Folder

NTFS           4,294,967,295 (2^32 - 1) files

ext4           Variable (Practically unlimited)

XFS            Variable (Practically unlimited)

Btrfs          Variable (Practically unlimited)

ext3           Around 32,000 to 64,000 files (depends on block size)

Seems we should avoid ext3 nowadays... Apart from that, we should be fine.

It's also something you as a customer should not need to think about. If IFS Cloud File Storage supports "local" file storage in Remote, it's its job to make sure it works for a very large number of files. If there turns out to be problems with too many files, it's something IFS needs to figure out a solution for.

Agreed?

> b) we see high risk to be non-compliant with government orders and regulations, if we are no longer able to separate Files in different folders and locations,  which could cause in massive negative impact to our business

We would like to challenge this, and/or understand more. It seems to be similar to what I'm commenting on here, even if there might be other underlying reasons:

https://community.ifs.com/document-management-docman-248/reinstate-support-for-external-document-repositories-and-mulitple-repositories-42349?postid=158085#post158085

> c) we have documents on some different (older) ifs systems where we can use the "old" servers without need to duplicate the physical files

Are you talking about backup and recovery here? If so, it's an interesting point and it's something that should be directed against the IFS Cloud File Storage team (a way to put away files to slower/cheaper hardware/storage).

> d) we have a long-term archive solution that represents the files as UNC path\file

It should be possible to make that work with the SMB storage support in IFS Cloud File Storage, don't you think?

> what are the chances, that we still can use shared folders and what are the required measures on IFS side to ensure this?

The idea is to remove the current support for shared folders “soon” and replace it with support for multiple storage locations (also shares) in IFS Cloud File Storage. 
 

/Mathias
Tor
Rank
Userlevel 3
Badge +4

Very useful heads-up @Mathias Dahl . May I ask a somewhat related question? My customer has got classified data in their files that they don’t want to leave their network, but IFS is running at an IFS partner data center. If using File Storage would the files ever be stored (temporarily) on IFS servers using this option? 

Mathias Dahl
Rank
Userlevel 7
Badge +30

Very useful heads-up @Mathias Dahl . May I ask a somewhat related question? My customer has got classified data in their files that they don’t want to leave their network, but IFS is running at an IFS partner data center. If using File Storage would the files ever be stored (temporarily) on IFS servers using this option? 

The data is never stored on disk in the "middle tier" (a Kubernetes cluster with multiple containers), but the data will of course go through it, via the network calls that streams the files from and to the backend storage.

 

/Mathias

Reply


Terms & ConditionsCookie settings