Skip to main content

Deprecation of the FTP and Shared repository types in Docman in 24R1 - a heads-up

  • 5 October 2023
  • 67 replies
  • 2713 views

Deprecation of the FTP and Shared repository types in Docman in 24R1 - a heads-up

 

This is a heads-up on IFS deprecating support for the FTP and Shared repository types in Docman in IFS Cloud 23R2, for complete removal in 24R1.

If you have customers using these two repository types today, you should plan for migrating to Database or File Storage.

If you are planning to implement this in Remote Deployment in IFS Cloud right now, you should reconsider, and use File Storage or Database.

In Remote deployment, File Storage will still have support for local/external storage (outside the database) by supporting SMB/network shares.

 

BACKGROUND

 

Back in the day, Docman started out with support for "external" file storage via our support for FTP and Shared (folders) as the only repository types. Later, Database storage was added upon request from customers. And, much later (IFS Cloud 21R2), IFS Cloud File Storage was added as yet another repository option where, initially, File Storage was limited to Cloud Deployment where Azure Storage is used as the backend.

Recently, IFS Cloud File Storage got support for Remote Deployment, where a shared network folder can be selected for storing files.

 

TODAY

 

We are now in a position where we can remove the support in Docman for FTP and Shared repositories since every customer, from IFS Cloud 23R2 and later, will have two options for document file storage: "internal" (inside the database) and "external" (outside the database).

Read about the capabilities of IFS Cloud File Storage here, before you continue reading:

https://docs.ifs.com/techdocs/23r1/030_administration/210_cloud_file_storage/#introduction

Some customers in Remote Deployment might want to use their FTP servers or multiple existing shared network folders. Although there is of course a benefit to this, we think that one "external" file storage option should really be enough and customers should be able to change to use fewer options.

We are considering deprecating the Database repository option as well at some point, but that has not been decided yet. If you have use cases where you think this is strictly needed, let us know.

 

WHAT HAPPENS NOW?

 

This deprecation will be communicated officially via the release notes in IFS Cloud 23R2 GA, but we wanted to communicate this as early as possible, to avoid unnecessary migration work.

Documentation on migration paths will be available soon. We already have migration tools for migrating customers from Database, FTP, and Shared repositories to File Storage and we will work on making them even better.

Comments, questions?

 

67 replies

Mathias Dahl
Rank
Userlevel 7
Badge +31

A more technical question related to the subject. It is specific for Remote Deployment but also interested for Cloud Deployment.

The document files are temporarily stored in the Application Server when working with IFS Applications 10 and older (at least IEE version).

How does it work in IFS Cloud?
Will the documents be temporarily stored somewhere during the process or will they “only” pass by the Kubernetes Cluster on the way to the client? 

Hi Stefan, thanks for asking here.

The file content is streamed all the way, we are not creating any temporary files. All parts of the flow that's controlled by IFS is written in Java and there, instead of working with files, we are working with input and output streams. While it was possible in Apps 10, it's much harder to handle temporary files in Kubernetes, especially if they are large, since each container has a fixed amount of storage space.

 

/Mathias
M
Rank
Badge +2

Hi Mathias Dahl

We are a larger company that is currently in the process of porting to ifs cloud.
(remote managed solution, servers are on premise)

We are using DOCMAN heavily and today there are more than 4 million files (Size over 3 TB) increasing daily.
Our today Solution is relaying on Shared Folder (Shared repository). The main problems of changing to "IFS Cloud File Storage" would be:

a) no file server likes to have more than some 100k files in the same folder
b) we see high risk to be non-compliant with government orders and regulations, if we are no longer able to separate Files in different folders and locations,  which could cause in massive negative impact to our business
c) we have documents on some different (older) ifs systems where we can use the "old" servers without need to duplicate the physical files
d) we have a long-term archive solution that represents the files as UNC path\file

So basically, if the possibility to use shared folders (Shared repository) is removed with 24/1 we will run into a deep problem that will seriously impact our project in a most critical way.

what are the chances, that we still can use shared folders and what are the required measures on IFS side to ensure this?

Regards

Stefan

B
Rank
Badge +2

@Mathias Dahl

 

I've just come from a meeting where we discussed exactly this topic. 
Concentrating on repository type "file storage" may well be the right way from a technical point of view. But the limitations and restrictions that this decision entails are significant for our customers. I fully agree with the existing opinions that i have read.
But this change is only half of the truth. Our customer projects always have the same requirements.
It is about internal order documents created by IFS, e.g. order confirmation. This document (PDF) should be automatically attached to the object (customer order). IFS offers "Report Rules" to use for this purpose. This also works well as a standalone solution. But Report Rules do not work together with repository type "File Storage". It is currently not possible in IFS CLOUD to implement a continuous process flow with standard functionality. Creation and printing of an order document incl. attachment to an object and then a transfer to a repository (file storage). We practised for a long time in our last customer project and then we had to go back to "Shared Repository". You may can imagine that our customer wasen’t really happy.
My expectation is, that IFS will deliver a standard solution for this requirement in a short time.

Thanks and best regards,
Beat

 

Mathias Dahl
Rank
Userlevel 7
Badge +31

Thanks @BytBeatbW 

We are aware of the fact that not all functionality that worked with a Database repository in Docman works with File Storage. I think there is a plan to handle each gap there.

As for the problem you mention (archiving reports in Docman), depending on your usage, it should not be a problem to keep those types of documents in the database, even if the bulk of your documentation reside in File Storage. I say "depending on your usage" because it might be that you are archiving so many documents that it becomes a problem with the database size growing too much. Archiving reports doesn't strike me as a process that should fill the database too much though, but then again I don't know how many reports your customers are archiving…

This issue has little to do with the deprecation of FTP and Shared repository types though, and much more to do with the long term focus on File Storage.

Although it's related, it's not the same thing. We have not decided to remove the Database repository type yet.

 

/Mathias
Mathias Dahl
Rank
Userlevel 7
Badge +31

@maxonbs 

 

> a) no file server likes to have more than some 100k files in the same folder

I have been here a long time and I have seen that problem, at least in the old days.

These seems to be the limits of file systems in use today:

File System    Maximum Files per Folder

NTFS           4,294,967,295 (2^32 - 1) files

ext4           Variable (Practically unlimited)

XFS            Variable (Practically unlimited)

Btrfs          Variable (Practically unlimited)

ext3           Around 32,000 to 64,000 files (depends on block size)

Seems we should avoid ext3 nowadays... Apart from that, we should be fine.

It's also something you as a customer should not need to think about. If IFS Cloud File Storage supports "local" file storage in Remote, it's its job to make sure it works for a very large number of files. If there turns out to be problems with too many files, it's something IFS needs to figure out a solution for.

Agreed?

> b) we see high risk to be non-compliant with government orders and regulations, if we are no longer able to separate Files in different folders and locations,  which could cause in massive negative impact to our business

We would like to challenge this, and/or understand more. It seems to be similar to what I'm commenting on here, even if there might be other underlying reasons:

https://community.ifs.com/document-management-docman-248/reinstate-support-for-external-document-repositories-and-mulitple-repositories-42349?postid=158085#post158085

> c) we have documents on some different (older) ifs systems where we can use the "old" servers without need to duplicate the physical files

Are you talking about backup and recovery here? If so, it's an interesting point and it's something that should be directed against the IFS Cloud File Storage team (a way to put away files to slower/cheaper hardware/storage).

> d) we have a long-term archive solution that represents the files as UNC path\file

It should be possible to make that work with the SMB storage support in IFS Cloud File Storage, don't you think?

> what are the chances, that we still can use shared folders and what are the required measures on IFS side to ensure this?

The idea is to remove the current support for shared folders “soon” and replace it with support for multiple storage locations (also shares) in IFS Cloud File Storage. 
 

/Mathias
Tor
Rank
Userlevel 3
Badge +4

Very useful heads-up @Mathias Dahl . May I ask a somewhat related question? My customer has got classified data in their files that they don’t want to leave their network, but IFS is running at an IFS partner data center. If using File Storage would the files ever be stored (temporarily) on IFS servers using this option? 

Mathias Dahl
Rank
Userlevel 7
Badge +31

Very useful heads-up @Mathias Dahl . May I ask a somewhat related question? My customer has got classified data in their files that they don’t want to leave their network, but IFS is running at an IFS partner data center. If using File Storage would the files ever be stored (temporarily) on IFS servers using this option? 

The data is never stored on disk in the "middle tier" (a Kubernetes cluster with multiple containers), but the data will of course go through it, via the network calls that streams the files from and to the backend storage.

 

/Mathias
S
Rank
Userlevel 3
Badge +7

I have asked how does this relate to database copying in:
 

 

B
Rank
Userlevel 1
Badge +3

@Mathias Dahl 

 

I can’t see how this shall be working for us. We have replicated solution with more than 10TB of files, currently growing with several TB each year.

Current solution is to have “dynamic” (editable) files in DB and use IFS replication technology to handle these two way replicating files. Most of these are relatively small, less than 10 MB

Static files (received from project doc system) stored on application server, in separate folders for each unit they shall be replicated to (like e:\docs\Unit1\TechDocs), having 3rd party SW replicating them to same path on offshore unit, which enables the doc link to work from both places. Similar for Unit2, 3, and so on. Standard IFS replication is not working for our high volume of large files (up to 2 GB) over satellite links

Onshore only files (like invoices), kept in separate location on application server. 

Files purely kept for archiving, which are fastest growing category, are stored on low cost HW with separate backup routines.

How is your new solution intended to work for replicated solutions? How shall it work for large volume systems like we have? We already had to implement a separate doc system for our projects as IFS DOCMAN have kept up development with external user access and simultaneously editing of documents. These changes may even prevent us from full usage internally.  

Bjørn T. Hetland
Mathias Dahl
Rank
Userlevel 7
Badge +31

@BjornH 

Once IFS Cloud File Storage has support for multiple SMB shares (which File Storage uses in Remote ("on premise") deployment), you should be able to do the same thing, on a high level. You might need many shares though, unless we get the ability to point out a specific sub folder under the share. This is something that doesn't work in Cloud deployment today, where the files end up in Azure Blob Storage.

/Mathias
B
Rank
Userlevel 1
Badge +3

@Mathias Dahl

When will IFS Cloud application get support for “multiple SMB shares”?

How will you support replicated documents when not stored in the DB anymore?

Bjørn T. Hetland
Mathias Dahl
Rank
Userlevel 7
Badge +31

> When will IFS Cloud application get support for “multiple SMB shares”?

We cannot commit to a particular release right now, but we want to add support for it as soon as we can.

> How will you support replicated documents when not stored in the DB anymore?

We have hinted at database repository support being dropped at some point, but it will not happen before important features like replication supports IFS Cloud File Storage.

@BjornH 

/Mathias
B
Rank
Userlevel 1
Badge +3

@Mathias Dahl 

I do find it very strange that you remove critical functionality (share and FRP) before replacement (multiple SMB Shares) is in place.

Can be btw use the SMB shares same way that we use shares today, so that those we do replicate outside of IFS always point to local copy, like “e:\docs\Unit1\TechDocs” that will always one each application server itself?

Bjørn T. Hetland

Bjørn T. Hetland
Mathias Dahl
Rank
Userlevel 7
Badge +31

@BjornH 

The support for FTP and Shared is still deprecated (= "Stop using it if you can") but was not removed yet due to feedback from people like you. It was a heads up after all, right (and meant to create a discussion like this)? 🙂

As for your question, and from my quite shallow understanding of what you do, yes you should be able to do something similar. Perhaps you will need a unique share for each sub folder that you have today, since File Storage does not expose/uses a concept of sub folders. SMB is, for most practical purposes, "Windows network drives".

We are trying to get multiple SMB support prioritized as we speak.

/Mathias
B
Rank
Userlevel 1
Badge +3

@Mathias Dahl Thanks for lot for your clarification. I thought it would be removed before new solution came in place, very glad to hear that was a misunderstanding. As I understand you now, it will be kept until new solution is in place, right?

 

I’m OK creating multiple shares, if that is what is needed. Current solution is straight on server without going through shares, but we do survive a little extra administrative burden. Main reason for us not going with share was to make the path same for onshore and offshore servers, always pointing at local servers. This also removes possibility for Test environments starting to view or change documents stored on Prod, so for us it’s a win-win.

Trust you know how to reach me if you want to dig deeper into the details.

Bjørn T. Hetland
Mathias Dahl
Rank
Userlevel 7
Badge +31

@BjornH 

> As I understand you now, it will be kept until new solution is in place, right?

Correct.

> Main reason for us not going with share was to make the path same for onshore and offshore servers, always pointing at local servers.

I'm not sure if you mean pointing to a local drive (with a drive letter, as you showed) there, but that's not possible in the container era anyway, and was changed in the first release of IFS Cloud. Now we require a share (//server/share/) even when using Shared repositories. It similar for the SMB shares pointed to by IFS Cloud File Storage.

 

/Mathias
Mathias Dahl
Rank
Userlevel 7
Badge +31

Cross-linking to a recent question about this:

 

 

/Mathias

Reply


Terms & ConditionsCookie settings