Deprecation of the FTP and Shared repository types in Docman in 24R1 - a heads-up

Hi Stefan, thanks for asking here.

The file content is streamed all the way, we are not creating any temporary files. All parts of the flow that's controlled by IFS is written in Java and there, instead of working with files, we are working with input and output streams. While it was possible in Apps 10, it's much harder to handle temporary files in Kubernetes, especially if they are large, since each container has a fixed amount of storage space.

Hi Mathias Dahl

We are a larger company that is currently in the process of porting to ifs cloud.
(remote managed solution, servers are on premise)

We are using DOCMAN heavily and today there are more than 4 million files (Size over 3 TB) increasing daily.
Our today Solution is relaying on Shared Folder (Shared repository). The main problems of changing to "IFS Cloud File Storage" would be:

a) no file server likes to have more than some 100k files in the same folder
b) we see high risk to be non-compliant with government orders and regulations, if we are no longer able to separate Files in different folders and locations,  which could cause in massive negative impact to our business
c) we have documents on some different (older) ifs systems where we can use the "old" servers without need to duplicate the physical files
d) we have a long-term archive solution that represents the files as UNC path\file

So basically, if the possibility to use shared folders (Shared repository) is removed with 24/1 we will run into a deep problem that will seriously impact our project in a most critical way.

what are the chances, that we still can use shared folders and what are the required measures on IFS side to ensure this?

Regards

Stefan

@Mathias Dahl

I've just come from a meeting where we discussed exactly this topic. 
Concentrating on repository type "file storage" may well be the right way from a technical point of view. But the limitations and restrictions that this decision entails are significant for our customers. I fully agree with the existing opinions that i have read.
But this change is only half of the truth. Our customer projects always have the same requirements.
It is about internal order documents created by IFS, e.g. order confirmation. This document (PDF) should be automatically attached to the object (customer order). IFS offers "Report Rules" to use for this purpose. This also works well as a standalone solution. But Report Rules do not work together with repository type "File Storage". It is currently not possible in IFS CLOUD to implement a continuous process flow with standard functionality. Creation and printing of an order document incl. attachment to an object and then a transfer to a repository (file storage). We practised for a long time in our last customer project and then we had to go back to "Shared Repository". You may can imagine that our customer wasen’t really happy.
My expectation is, that IFS will deliver a standard solution for this requirement in a short time.

Thanks and best regards,
Beat

Thanks @BytBeatbW 

We are aware of the fact that not all functionality that worked with a Database repository in Docman works with File Storage. I think there is a plan to handle each gap there.

As for the problem you mention (archiving reports in Docman), depending on your usage, it should not be a problem to keep those types of documents in the database, even if the bulk of your documentation reside in File Storage. I say "depending on your usage" because it might be that you are archiving so many documents that it becomes a problem with the database size growing too much. Archiving reports doesn't strike me as a process that should fill the database too much though, but then again I don't know how many reports your customers are archiving…

This issue has little to do with the deprecation of FTP and Shared repository types though, and much more to do with the long term focus on File Storage.

Although it's related, it's not the same thing. We have not decided to remove the Database repository type yet.

@maxonbs 

> a) no file server likes to have more than some 100k files in the same folder

I have been here a long time and I have seen that problem, at least in the old days.

These seems to be the limits of file systems in use today:

File System    Maximum Files per Folder

NTFS           4,294,967,295 (2^32 - 1) files

ext4           Variable (Practically unlimited)

XFS            Variable (Practically unlimited)

Btrfs          Variable (Practically unlimited)

ext3           Around 32,000 to 64,000 files (depends on block size)

Seems we should avoid ext3 nowadays... Apart from that, we should be fine.

It's also something you as a customer should not need to think about. If IFS Cloud File Storage supports "local" file storage in Remote, it's its job to make sure it works for a very large number of files. If there turns out to be problems with too many files, it's something IFS needs to figure out a solution for.

Agreed?

> b) we see high risk to be non-compliant with government orders and regulations, if we are no longer able to separate Files in different folders and locations,  which could cause in massive negative impact to our business

We would like to challenge this, and/or understand more. It seems to be similar to what I'm commenting on here, even if there might be other underlying reasons:

https://community.ifs.com/document-management-docman-248/reinstate-support-for-external-document-repositories-and-mulitple-repositories-42349?postid=158085#post158085

> c) we have documents on some different (older) ifs systems where we can use the "old" servers without need to duplicate the physical files

Are you talking about backup and recovery here? If so, it's an interesting point and it's something that should be directed against the IFS Cloud File Storage team (a way to put away files to slower/cheaper hardware/storage).

> d) we have a long-term archive solution that represents the files as UNC path\file

It should be possible to make that work with the SMB storage support in IFS Cloud File Storage, don't you think?

> what are the chances, that we still can use shared folders and what are the required measures on IFS side to ensure this?

The idea is to remove the current support for shared folders “soon” and replace it with support for multiple storage locations (also shares) in IFS Cloud File Storage. 
 

Very useful heads-up @Mathias Dahl . May I ask a somewhat related question? My customer has got classified data in their files that they don’t want to leave their network, but IFS is running at an IFS partner data center. If using File Storage would the files ever be stored (temporarily) on IFS servers using this option? 

The data is never stored on disk in the "middle tier" (a Kubernetes cluster with multiple containers), but the data will of course go through it, via the network calls that streams the files from and to the backend storage.

I have asked how does this relate to database copying in:
 

@Mathias Dahl 

I can’t see how this shall be working for us. We have replicated solution with more than 10TB of files, currently growing with several TB each year.

Current solution is to have “dynamic” (editable) files in DB and use IFS replication technology to handle these two way replicating files. Most of these are relatively small, less than 10 MB

Static files (received from project doc system) stored on application server, in separate folders for each unit they shall be replicated to (like e:\docs\Unit1\TechDocs), having 3rd party SW replicating them to same path on offshore unit, which enables the doc link to work from both places. Similar for Unit2, 3, and so on. Standard IFS replication is not working for our high volume of large files (up to 2 GB) over satellite links

Onshore only files (like invoices), kept in separate location on application server. 

Files purely kept for archiving, which are fastest growing category, are stored on low cost HW with separate backup routines.

How is your new solution intended to work for replicated solutions? How shall it work for large volume systems like we have? We already had to implement a separate doc system for our projects as IFS DOCMAN have kept up development with external user access and simultaneously editing of documents. These changes may even prevent us from full usage internally.  

@BjornH 

Once IFS Cloud File Storage has support for multiple SMB shares (which File Storage uses in Remote ("on premise") deployment), you should be able to do the same thing, on a high level. You might need many shares though, unless we get the ability to point out a specific sub folder under the share. This is something that doesn't work in Cloud deployment today, where the files end up in Azure Blob Storage.

@Mathias Dahl

When will IFS Cloud application get support for “multiple SMB shares”?

How will you support replicated documents when not stored in the DB anymore?

> When will IFS Cloud application get support for “multiple SMB shares”?

We cannot commit to a particular release right now, but we want to add support for it as soon as we can.

> How will you support replicated documents when not stored in the DB anymore?

We have hinted at database repository support being dropped at some point, but it will not happen before important features like replication supports IFS Cloud File Storage.

@BjornH 

@Mathias Dahl 

I do find it very strange that you remove critical functionality (share and FRP) before replacement (multiple SMB Shares) is in place.

Can be btw use the SMB shares same way that we use shares today, so that those we do replicate outside of IFS always point to local copy, like “e:\docs\Unit1\TechDocs” that will always one each application server itself?

Bjørn T. Hetland

@BjornH 

The support for FTP and Shared is still deprecated (= "Stop using it if you can") but was not removed yet due to feedback from people like you. It was a heads up after all, right (and meant to create a discussion like this)? 🙂

As for your question, and from my quite shallow understanding of what you do, yes you should be able to do something similar. Perhaps you will need a unique share for each sub folder that you have today, since File Storage does not expose/uses a concept of sub folders. SMB is, for most practical purposes, "Windows network drives".

We are trying to get multiple SMB support prioritized as we speak.

@Mathias Dahl Thanks for lot for your clarification. I thought it would be removed before new solution came in place, very glad to hear that was a misunderstanding. As I understand you now, it will be kept until new solution is in place, right?

I’m OK creating multiple shares, if that is what is needed. Current solution is straight on server without going through shares, but we do survive a little extra administrative burden. Main reason for us not going with share was to make the path same for onshore and offshore servers, always pointing at local servers. This also removes possibility for Test environments starting to view or change documents stored on Prod, so for us it’s a win-win.

Trust you know how to reach me if you want to dig deeper into the details.

@BjornH 

> As I understand you now, it will be kept until new solution is in place, right?

Correct.

> Main reason for us not going with share was to make the path same for onshore and offshore servers, always pointing at local servers.

I'm not sure if you mean pointing to a local drive (with a drive letter, as you showed) there, but that's not possible in the container era anyway, and was changed in the first release of IFS Cloud. Now we require a share (//server/share/) even when using Shared repositories. It similar for the SMB shares pointed to by IFS Cloud File Storage.

Cross-linking to a recent question about this:

​@Mathias Dahl 
Have anything happend on this topic for 24R2 or plans for 25R1, or is it still the same as for IFS Cloud 24R1?

/David

FTP and Shared are still available in 24R2 and it will probably not be removed in 25R1 either, from what it seems. We need support for multiple SMB servers/shares in File Storage before we can remove it and it doesn't seem to be prioritized to be done soon...

​@Mathias Dahl  I'd like to confirm the following. Will FTP repositories be configurable in 24R2 and 25R1, or will they be completely removed?

Thanks
/Oshada

​@Oshada Samarasinghe 

See my earlier reply above.

hello,

For managing a large quantity of files, S3 object storage is a great feature as it eliminates the limitations imposed by traditional filesystems. This can be particularly beneficial for scalability and performance.

However, it seems that S3 object storage is not yet a feature in the current solution. Is this mentioned in the roadmap?

Storing files in a database can indeed lead to significant growth in database size over time, which can impact performance and manageability. Using S3 for file storage can help mitigate these issues by offloading file storage to a scalable and efficient object storage system.

Best Regards

​@MCIPSTEV 

There are no plans to support S3 as far as I know. What are the advantages over Azure Blob Storage as you see it?