Deprecation of the FTP and Shared repository types in Docman in 24R1 - a heads-up

  • 5 October 2023
  • 57 replies
  • 2074 views
Mathias Dahl
Rank
Userlevel 7
Badge +30

Deprecation of the FTP and Shared repository types in Docman in 24R1 - a heads-up

 

This is a heads-up on IFS deprecating support for the FTP and Shared repository types in Docman in IFS Cloud 23R2, for complete removal in 24R1.

If you have customers using these two repository types today, you should plan for migrating to Database or File Storage.

If you are planning to implement this in Remote Deployment in IFS Cloud right now, you should reconsider, and use File Storage or Database.

In Remote deployment, File Storage will still have support for local/external storage (outside the database) by supporting SMB/network shares.

 

BACKGROUND

 

Back in the day, Docman started out with support for "external" file storage via our support for FTP and Shared (folders) as the only repository types. Later, Database storage was added upon request from customers. And, much later (IFS Cloud 21R2), IFS Cloud File Storage was added as yet another repository option where, initially, File Storage was limited to Cloud Deployment where Azure Storage is used as the backend.

Recently, IFS Cloud File Storage got support for Remote Deployment, where a shared network folder can be selected for storing files.

 

TODAY

 

We are now in a position where we can remove the support in Docman for FTP and Shared repositories since every customer, from IFS Cloud 23R2 and later, will have two options for document file storage: "internal" (inside the database) and "external" (outside the database).

Read about the capabilities of IFS Cloud File Storage here, before you continue reading:

https://docs.ifs.com/techdocs/23r1/030_administration/210_cloud_file_storage/#introduction

Some customers in Remote Deployment might want to use their FTP servers or multiple existing shared network folders. Although there is of course a benefit to this, we think that one "external" file storage option should really be enough and customers should be able to change to use fewer options.

We are considering deprecating the Database repository option as well at some point, but that has not been decided yet. If you have use cases where you think this is strictly needed, let us know.

 

WHAT HAPPENS NOW?

 

This deprecation will be communicated officially via the release notes in IFS Cloud 23R2 GA, but we wanted to communicate this as early as possible, to avoid unnecessary migration work.

Documentation on migration paths will be available soon. We already have migration tools for migrating customers from Database, FTP, and Shared repositories to File Storage and we will work on making them even better.

Comments, questions?

 

/Mathias

57 replies

S
Rank
Userlevel 6
Badge +12

Hi Mathias,

 

One scenario comes to my mind is report rules only work for database option. Therefore  need to consider that before deprecating db option.

 

Regards,

Sahan  

william.klotz
Rank
Userlevel 7
Badge +21

@Mathias Dahl ,

 

We currently use a hybrid approach with document management between database and shared folders.  We typically keep our large volume of document attachments in shared folders to prevent database bloat and lower volume or more sensitive documents in the database.  With IFS decision to deprecate shared folder in 23R2 and eliminate shared folders completely in 24R1 we have decided to migrate all documents into our database but this is not ideal for us.  Moving into the database or to a Cloud File Storage adds cost for us we just hadn’t planned on.  Clustered NAS storage isn’t to expensive in our data center and it kept all our documents within our facilities.  Yes moving them into the database will keep them within our data center but database storage is more expensive for us as its on our SAN’s using SSD which we really don’t need for documents that are access irregularly.

 

We seem to be incurring more and more costs with IFS Cloud and functionality we’ve come to rely on is being removed.  While we like the IFS Aurena interface we are concerned with what we will have to redo just to get back to where we are with IFS Application 10 and IEE.

 

Regards,

William Klotz

DeanBurell
Rank
Userlevel 3
Badge +5

This is a huge mistake and will cause many customers to not opt for IFS in the cloud.  Many of our customers in the Utilities Industries and those that serve Aerospace and Defense industries are dependent on the capability to have external document storage for certain classes of documents. these documents need to be stored on prem or on secure non cloud vaults by regulation.   The ability to link these documents to metadata and processes is one of the key features of IFS.

This lack of capability will force many companies in regulated industries to opt for an on prem solution or potentially a competitor solution that is cloud based that allows this capability.    Not having these capabilities is taking away key functionality existing customers are using and will hurt us in the marketplace.

You mentioned checking in “Link” documents in a previous message however I have found that doing that is somewhat inconsistent in its behavior and doesnt work very well. 

Astra Canyon Group is a Channel Sales and Services Partner that helps customers maximize the value of their IFS Solutions since 1999 * We are growing! We are looking for experienced consultants to join our team *
william.klotz
Rank
Userlevel 7
Badge +21

Hi @Mathias Dahl ,

 

I must have read your initial email 4 or 5 times and I didn’t get it until your last response where I decided to reference the online documentation and go through the IFS Cloud File Storage information.  Then I read your initial email again and I understood what the second paragraph under TODAY was saying.  

I would recommend you include this reference to the documentation will visually really shows what you are explaining.  For me it really did the trick.

 

https://docs.ifs.com/techdocs/23r1/020_lifecycle/100_file_storage_service/#introduction

 

Regards,

William Klotz

J
Rank
Userlevel 4
Badge +9

Hi @Mathias Dahl ,
Currently, we are also using a hybrid approach with repository types Database and Shared.

Database is used for frequently used documents that require quick access. 

Repository type Shared is used for 2 different types of documents. These 2 different types of documents are assigned to several different document classes.
a) Documents with legal requirements to electronic archiving. This applies for all types of outgoing and incoming invoices and contracts. 
legal requirement: For (tax) audit security, only data carriers that can be written once (CD-R, DVD-R, Blue-ray) or special, hard-drive-based archive software come into consideration. This repository is special WORM, relatively expensive.

b) Documents that are generated by different IFS transactions (eg. purchase orders, sales quotations, customer orders, delivery notes, ...) without legal requirments. These repositories are on cheaper hardware.

Based on your description and the information in About Document file repositories , I understand it is not possible to use different physical storage devices for different document classes, because this would require the possibility to use multiple existing shared network folders. My understanding is, that IFS cloud file storage only supports the configuration of one single SMB share path in custom_values.yaml

I understand, that there are pros and cons of the different repository types. Anyways, with the move to IFS Cloud, migrating our document repositories will require a considerable effort, even with the migration tools IFS will provide for this.

In addition to the above, reality is, that documents are also created and stored in other source systems like MS-Teams/Sharepoint environments or PLM-systems (eg. drawings in Siemens Teamcenter) just to name some examples. As processes span across different systems and integration gets thighter, my question: Is there any intention for data virtualization of documents? Meaning, to have the possibility to connect documents from other source systems / source repositories to IFS objects, without the need of physical duplication of document file data and maybe even document meta data.

Regards

Johannes Wittwer

M
Rank
Badge +1

Hi Mathias Dahl

We are a larger company that is currently in the process of porting to ifs cloud.
(remote managed solution, servers are on premise)

We are using DOCMAN heavily and today there are more than 4 million files (Size over 3 TB) increasing daily.
Our today Solution is relaying on Shared Folder (Shared repository). The main problems of changing to "IFS Cloud File Storage" would be:

a) no file server likes to have more than some 100k files in the same folder
b) we see high risk to be non-compliant with government orders and regulations, if we are no longer able to separate Files in different folders and locations,  which could cause in massive negative impact to our business
c) we have documents on some different (older) ifs systems where we can use the "old" servers without need to duplicate the physical files
d) we have a long-term archive solution that represents the files as UNC path\file

So basically, if the possibility to use shared folders (Shared repository) is removed with 24/1 we will run into a deep problem that will seriously impact our project in a most critical way.

what are the chances, that we still can use shared folders and what are the required measures on IFS side to ensure this?

Regards

Stefan

Mathias Dahl
Rank
Userlevel 7
Badge +30

Hi @DeanBurell ,

About this:

Many of our customers in the Utilities Industries and those that serve Aerospace and Defense industries are dependent on the capability to have external document storage for certain classes of documents.

External storage (storage outside the database) is not going away. What's going away is Docman's explicit/native support for FTP and Shared repositories. It's replaced by Docman using IFS Cloud File Storage which now has the option for external storage in Remote Deployment. And, like before, you can control what documents classes are stored in the database and what documents are stored externally. Both in Cloud and Remote Deployment.

This is a point that others seem to have missed too and I can only apologize for not expressing this point clearly enough.

I hope this helps.

/Mathias
DeanBurell
Rank
Userlevel 3
Badge +5

I have created an Idea to keep the support of external and multiple repositories. Please promote this to everyone that will be affected…

 

There will be many people affected... here are just some use cases

Use case 1:  Customer manufactures dual use products and technologies and sells to both  commercial and defense markets. They have some ITAR /defense requirements for some items only, and do not need a full Gov Cloud solution. In this scenario with the capability to have external repositories, we can implement this customer in IFS Cloud, with IFS Cloud hosting and manage any sensitive documents in an external repository, but still have them linked to the appropriate functionality and control their access through security. Their storage in an ITAR controlled repository gives them the coverage they need.

 

Use case 2: utility provider implementing in IFS Cloud– by law / regulation power and utility companies in NA cannot store potentially sensitive documents on the cloud. Any documents that may contain any information that could expose any information of vulnerabilities cannot be stored in a cloud.  Having these sensitive documents in an on-prem location, but linked to metadata in IFS Cloud enables them to maintain an extra layer of security.

 

Use case 3: utility provider – utility provider is implemented on prem, but as many of our customers are, they are comprised of multiple legal and operating entities each with their own set of users and documentation.  Because IFS implementation of document management is global, i.e. is not company specific, we implemented multiple repositories and classes for each company  to ensure that we physically kept each companies documents separate. This prevented anyone from being able to gain access to all of the documents even in the event of a breach.

 

Use case 4: Multi-Company, Multi-National customers – in many cases we would want to implement document repositories in the region they are located and or segregated by country/company for performance and security reasons.   i.e.  companies with Israeli locations or entities, and also locations or entities in the middle east.   Or companies with legal entities in France or Canada which have regulations that require data reside in country.

 

@ksander @JohannesWittwer @Mike The FSM TechnoGeek  @william.klotz  @Sahan Udana 

Astra Canyon Group is a Channel Sales and Services Partner that helps customers maximize the value of their IFS Solutions since 1999 * We are growing! We are looking for experienced consultants to join our team *
Mathias Dahl
Rank
Userlevel 7
Badge +30

Thanks @Kelum-WIA 

Then I think they should migrate to FS now. We have a built-in assistant for that. As I said, I don't think they will be able to use another Azure Storage account, so they will not save money that way, but they should save money on not using the more expensive database storage, so it should be a win anyway.

/Mathias
Mike The FSM TechnoGeek
Rank
Userlevel 5
Badge +17

Who benefits from this and how do they benefit?  This seems rather capricious and difficult for customers who rely on currently available options.  Aren’t there more important things we could do to help our customers instead of making their lives more difficult?

There is only one truth.
Mathias Dahl
Rank
Userlevel 7
Badge +30

A more technical question related to the subject. It is specific for Remote Deployment but also interested for Cloud Deployment.

The document files are temporarily stored in the Application Server when working with IFS Applications 10 and older (at least IEE version).

How does it work in IFS Cloud?
Will the documents be temporarily stored somewhere during the process or will they “only” pass by the Kubernetes Cluster on the way to the client? 

Hi Stefan, thanks for asking here.

The file content is streamed all the way, we are not creating any temporary files. All parts of the flow that's controlled by IFS is written in Java and there, instead of working with files, we are working with input and output streams. While it was possible in Apps 10, it's much harder to handle temporary files in Kubernetes, especially if they are large, since each container has a fixed amount of storage space.

 

/Mathias
Mathias Dahl
Rank
Userlevel 7
Badge +30

Who benefits from this and how do they benefit?  This seems rather capricious and difficult for customers who rely on currently available options.  Aren’t there more important things we could do to help our customers instead of making their lives more difficult?

By having fewer repository options we will save time spent on maintaining them, and we can spend that time on "fun" things which will benefit customers. Also, FTP support is getting rather long in the tooth and does not provide much benefit over IFS Cloud File Storage, provided by the platform.

 

Seems these options are there for ages. Thus, I’m a bit skeptic about the workload spent in support. We’ve been smart enough to let our customers choose between remote and Cloud hosting, which a differenciator from the competition. Even if Cloud hosting on Azure is in our favor from a business perspective of course. It’s like we want our customers choose the most expensive option with file storage or premium database.

I think there is a misunderstanding here.

IFS don't support FTP and shared repositories in Cloud deployment and in Remote customers will have two options as well, File Storage and Database, both under their complete control. In Cloud the cheapest option is for sure File Storage.

 

/Mathias
Mathias Dahl
Rank
Userlevel 7
Badge +30

It's also not the case that, once implemented, something like FTP support is free of maintenance. Platform and other technological changes requires reimplantation. I think we have had three or four implementations of FTP support already now. There is a cost with that.

/Mathias
B
Rank
Badge +2

@Mathias Dahl

 

I've just come from a meeting where we discussed exactly this topic. 
Concentrating on repository type "file storage" may well be the right way from a technical point of view. But the limitations and restrictions that this decision entails are significant for our customers. I fully agree with the existing opinions that i have read.
But this change is only half of the truth. Our customer projects always have the same requirements.
It is about internal order documents created by IFS, e.g. order confirmation. This document (PDF) should be automatically attached to the object (customer order). IFS offers "Report Rules" to use for this purpose. This also works well as a standalone solution. But Report Rules do not work together with repository type "File Storage". It is currently not possible in IFS CLOUD to implement a continuous process flow with standard functionality. Creation and printing of an order document incl. attachment to an object and then a transfer to a repository (file storage). We practised for a long time in our last customer project and then we had to go back to "Shared Repository". You may can imagine that our customer wasen’t really happy.
My expectation is, that IFS will deliver a standard solution for this requirement in a short time.

Thanks and best regards,
Beat

 

Mathias Dahl
Rank
Userlevel 7
Badge +30

@maxonbs 

 

> a) no file server likes to have more than some 100k files in the same folder

I have been here a long time and I have seen that problem, at least in the old days.

These seems to be the limits of file systems in use today:

File System    Maximum Files per Folder

NTFS           4,294,967,295 (2^32 - 1) files

ext4           Variable (Practically unlimited)

XFS            Variable (Practically unlimited)

Btrfs          Variable (Practically unlimited)

ext3           Around 32,000 to 64,000 files (depends on block size)

Seems we should avoid ext3 nowadays... Apart from that, we should be fine.

It's also something you as a customer should not need to think about. If IFS Cloud File Storage supports "local" file storage in Remote, it's its job to make sure it works for a very large number of files. If there turns out to be problems with too many files, it's something IFS needs to figure out a solution for.

Agreed?

> b) we see high risk to be non-compliant with government orders and regulations, if we are no longer able to separate Files in different folders and locations,  which could cause in massive negative impact to our business

We would like to challenge this, and/or understand more. It seems to be similar to what I'm commenting on here, even if there might be other underlying reasons:

https://community.ifs.com/document-management-docman-248/reinstate-support-for-external-document-repositories-and-mulitple-repositories-42349?postid=158085#post158085

> c) we have documents on some different (older) ifs systems where we can use the "old" servers without need to duplicate the physical files

Are you talking about backup and recovery here? If so, it's an interesting point and it's something that should be directed against the IFS Cloud File Storage team (a way to put away files to slower/cheaper hardware/storage).

> d) we have a long-term archive solution that represents the files as UNC path\file

It should be possible to make that work with the SMB storage support in IFS Cloud File Storage, don't you think?

> what are the chances, that we still can use shared folders and what are the required measures on IFS side to ensure this?

The idea is to remove the current support for shared folders “soon” and replace it with support for multiple storage locations (also shares) in IFS Cloud File Storage. 
 

/Mathias
DeanBurell
Rank
Userlevel 3
Badge +5

@Mathias Dahl Seems ridiculous to have to create an idea for this as if it were new functionality rather than functionality that already exists that you are taking away.  Perhaps it would be better if you surveyed your customer base before removing functionality.

As I do not have access to all of the customer base, that is using this functionality and I know many people do not frequent the community., the only way I can effectively draw attention to this would be through public social media.  And I would rather not air IFS shortcomings to the world.

Astra Canyon Group is a Channel Sales and Services Partner that helps customers maximize the value of their IFS Solutions since 1999 * We are growing! We are looking for experienced consultants to join our team *
C
Rank
Badge +2

What are the implications to A&D customers under ITAR?

Mathias Dahl
Rank
Userlevel 7
Badge +30

Hi @Mathias Dahl ,

 

I must have read your initial email 4 or 5 times and I didn’t get it until your last response where I decided to reference the online documentation and go through the IFS Cloud File Storage information.  Then I read your initial email again and I understood what the second paragraph under TODAY was saying.  

I would recommend you include this reference to the documentation will visually really shows what you are explaining.  For me it really did the trick.

 

https://docs.ifs.com/techdocs/23r1/020_lifecycle/100_file_storage_service/#introduction

 

Regards,

William Klotz

Unfortunetaly the link does not seem to be available any more … 

Thanks for reporting that. Seems the document moved here:

https://docs.ifs.com/techdocs/23r1/030_administration/210_cloud_file_storage/#introduction

I updated the link in my original post.

/Mathias
A
Rank
Badge +2

What I have understood is that it is only possible to define one File Storage location in Cloud version.

This will cause problems for e.g. Defence industry when classified document shall be stored. The demand is that they must be stored in the country where the XC license is valid.

For an installation where there are sites in different countries it must be possible to define more than one location as each country have their own XC licenses.

How is this demand addressed in the change of the functionality?

Mathias Dahl
Rank
Userlevel 7
Badge +30

Hi Mikael,

What I have understood is that it is only possible to define one File Storage location in Cloud version.

Correct. In Cloud Deployment, all files are stored in Azure Storage and in Remote Deployment all files are stored under one SMB/Samba/Windows network share.

This will cause problems for e.g. Defence industry when classified document shall be stored. The demand is that they must be stored in the country where the XC license is valid.

For an installation where there are sites in different countries it must be possible to define more than one location as each country have their own XC licenses.

It's a problem but it's not a new problem and we haven't supported that since Apps 7 when our "fat Windows client" directly communicated with the document file repository. The change/deprecation we intended to communicate through this post does not change anything related to what you wrote there.

How is this demand addressed in the change of the functionality?

It's not addressed now and it was not addressed since Apps 7 (which had it's own problems, mostly from a security point of view).

To summarize, we argue that the change/deprecation neither makes the situation better or worse with regards to the requirement you describe (which is about ITAR, Export Control, whatever you want to call it).

Is it clear?

 

/Mathias
K
Rank
Badge +3

Hi @Mathias Dahl, 

 

As always, thanks for sharing and providing valuable clarifications. I may add just one more related question connected to one of our customers.

They are on 23R2 and planning to move to 24R1 before going LIVE later in the year. It’s a Cloud Deployment and currently uses the default Database as the File Storage option. They won’t move from Database to File Storage. As indicated in the documentation (Index - Technical Documentation For IFS Cloud), the File Storage uses an Azure Blob Storage Account to store files in the Cloud deployment model. Is it possible to have a completely different Azure subscription for this Azure Blob Storage?

Thanks

Kelum Ferdinandez

Kelum Ferdinandez
Mathias Dahl
Rank
Userlevel 7
Badge +30

Hi @Kelum-WIA 

 > currently uses the default Database ... They won’t move from Database to File Storage

Do you know why? File Storage is cheaper and backup and restore operations quicker.

> the File Storage uses an Azure Blob Storage Account to store files in the Cloud deployment model. Is it possible to have a completely different Azure subscription for this Azure Blob Storage?

This is really a question about the IFS Cloud File Storage feature and not Document Management, but no, not as far as I know.

Any comments, @chanaka-shanil ?

 

/Mathias
K
Rank
Badge +3

Thanks @Mathias Dahl, 

They are mainly looking for a cheaper alternative due to the rapid growth of the DB due to documents. As they already have a separate Azure subscription, the idea is to utilize that.  

Maybe there is no price difference from azure perspective. Still, they would like to explore all possible options before deciding.

Thanks for directing to @chanaka-shanil 

Cheers !

 

 

Kelum Ferdinandez
K
Rank
Badge +3

Agree. I’ll wait for @chanaka-shanil thoughts from the IFS Cloud File Storage side before returning to the customer

Thanks again, @Mathias Dahl.

Cheers

Kelum Ferdinandez
K
Rank
Badge +3

Hi @chanaka-shanil,

May I get your advice on the matter, please. 

Your support is much appreciated.

Thanks

 

Kelum Ferdinandez

Reply


Terms & ConditionsCookie settings