Deprecation of the FTP and Shared repository types in Docman in 24R1 - a heads-up

  • 5 October 2023
  • 58 replies
  • 2115 views


Show first post

58 replies

Badge +3

Hi @chanaka-shanil,

May I get your advice on the matter, please. 

Your support is much appreciated.

Thanks

 

Userlevel 2
Badge +6

Hi Mathias,

Regarding possible deprecation of the db option I am for the moment maybe too stuck in “old” pros we’ve been communication for a while now, to be able to see this is a good way to go. There are of course cons with db storage as well (like db size and recovery issues) but the pros I’m thinking of are;
- Easy copying of a prod environment to test retaining documents and connections,
- SOX (Sarbanes Oxley) support for tracking read and write activities on files,
- Strong security (Oracle as a container also for the documents).

Are there any means to mitigate or meet those arguments in a solution with only “external” storage?

Regards,
Erik

Userlevel 7
Badge +30

Regarding possible deprecation of the db option I am for the moment maybe too stuck in “old” pros we’ve been communication for a while now, to be able to see this is a good way to go. There are of course cons with db storage as well (like db size and recovery issues) but the pros I’m thinking of are;
- Easy copying of a prod environment to test retaining documents and connections,
- SOX (Sarbanes Oxley) support for tracking read and write activities on files,
- Strong security (Oracle as a container also for the documents).

Are there any means to mitigate or meet those arguments in a solution with only “external” storage?

My comments on your valid comments on database storage:

Easy copying

This is true. In our managed cloud offering, I think they are working on a way to clone the files in the File Storage repository (which use Azure Storage). As for Remote deployment, I think you are on your own, like with FTP and Shared repositories.

SOX support

With IEE out of the picture, there is no longer a need to be able to read data directly from the database view, which means we can control all access to document file data with code, which in turn mean we can log every file operations. So I would argue that the SOX support will be as strong as before.

Security

I think it's hard to beat the security we have in Oracle, and keeping the files in a separate place introduces another attack vector for sure, but I think with the right setup (only the middle tier/Kubernetes should be able to access the file server) you can get strong security even with external storage.

Userlevel 7
Badge +30

Who benefits from this and how do they benefit?  This seems rather capricious and difficult for customers who rely on currently available options.  Aren’t there more important things we could do to help our customers instead of making their lives more difficult?

By having fewer repository options we will save time spent on maintaining them, and we can spend that time on "fun" things which will benefit customers. Also, FTP support is getting rather long in the tooth and does not provide much benefit over IFS Cloud File Storage, provided by the platform.

 

Badge +1

Can a customer access documents stored on Sharepoint if set to External?

Badge +4

Hi!

 

A more technical question related to the subject. It is specific for Remote Deployment but also interested for Cloud Deployment.

The document files are temporarily stored in the Application Server when working with IFS Applications 10 and older (at least IEE version).

How does it work in IFS Cloud?
Will the documents be temporarily stored somewhere during the process or will they “only” pass by the Kubernetes Cluster on the way to the client? 

 

Best regards,
Stefan

Userlevel 7
Badge +30

Can a customer access documents stored on Sharepoint if set to External?

There is no direct support for using SharePoint as a repository in IFS Document Management.

Userlevel 5
Badge +22

Who benefits from this and how do they benefit?  This seems rather capricious and difficult for customers who rely on currently available options.  Aren’t there more important things we could do to help our customers instead of making their lives more difficult?

By having fewer repository options we will save time spent on maintaining them, and we can spend that time on "fun" things which will benefit customers. Also, FTP support is getting rather long in the tooth and does not provide much benefit over IFS Cloud File Storage, provided by the platform.

 

Seems these options are there for ages. Thus, I’m a bit skeptic about the workload spent in support. We’ve been smart enough to let our customers choose between remote and Cloud hosting, which a differenciator from the competition. Even if Cloud hosting on Azure is in our favor from a business perspective of course. It’s like we want our customers choose the most expensive option with file storage or premium database.

Userlevel 7
Badge +30

@william.klotz 

I think there is a misunderstanding here.

The ability for cheap external (outside of the database) storage in Remote deployment will not go away. Can you read my original post again?

 

Userlevel 7
Badge +30

Thanks, @william.klotz, I added a link to the technical documentation on IFS Cloud File Storage now. So, you are all good now, right? :-) 

Userlevel 7
Badge +30

@JohannesWittwer 

Hi, thanks for your comments and sharing the setup you use. I will comment on some of what you wrote, below.

Database is used for frequently used documents that require quick access.

Can you explain what "quick" means, here? Is it noticeable faster for your users to view a document that's stored in the database compared to being stored externally?

Based on your description and the information in About Document file repositories , I understand it is not possible to use different physical storage devices for different document classes, because this would require the possibility to use multiple existing shared network folders. My understanding is, that IFS cloud file storage only supports the configuration of one single SMB share path in custom_values.yaml

That's correct. Since we will rely on the capabilities of IFS Cloud File Storage, we will be limited with what it provides. I would recommend you file an idea here on IFS Community to have our platform team consider more flexibility in this area.

Is there any intention for data virtualization of documents? Meaning, to have the possibility to connect documents from other source systems / source repositories to IFS objects, without the need of physical duplication of document file data and maybe even document meta data.

The only thing I can think of is that we discussed how we could integrate with Microsoft Teams. But I think the main intent there is to, in some way, collaborate on documents within Teams and, at a later time, check in the final documents to IFS Document Management.

In general, a simple way to keep track of documents in other systems is to check in "link files" to them in Docman. It can be a .lnk or .url file as per what Windows supports, or you can check in a simple text file with a link in it and a macro could follow/open the link. Or the file could be an empty dummy file and the link can be stored in some field on the document revision and the macro could open that one. This solution is easy to setup but has limits when it comes to editing documents and when it comes to authentication (the user will have to login to that other system).

Thanks!

Userlevel 7
Badge +30

Thanks @BytBeatbW 

We are aware of the fact that not all functionality that worked with a Database repository in Docman works with File Storage. I think there is a plan to handle each gap there.

As for the problem you mention (archiving reports in Docman), depending on your usage, it should not be a problem to keep those types of documents in the database, even if the bulk of your documentation reside in File Storage. I say "depending on your usage" because it might be that you are archiving so many documents that it becomes a problem with the database size growing too much. Archiving reports doesn't strike me as a process that should fill the database too much though, but then again I don't know how many reports your customers are archiving…

This issue has little to do with the deprecation of FTP and Shared repository types though, and much more to do with the long term focus on File Storage.

Although it's related, it's not the same thing. We have not decided to remove the Database repository type yet.

 

Userlevel 3
Badge +4

Very useful heads-up @Mathias Dahl . May I ask a somewhat related question? My customer has got classified data in their files that they don’t want to leave their network, but IFS is running at an IFS partner data center. If using File Storage would the files ever be stored (temporarily) on IFS servers using this option? 

Userlevel 7
Badge +30

Very useful heads-up @Mathias Dahl . May I ask a somewhat related question? My customer has got classified data in their files that they don’t want to leave their network, but IFS is running at an IFS partner data center. If using File Storage would the files ever be stored (temporarily) on IFS servers using this option? 

The data is never stored on disk in the "middle tier" (a Kubernetes cluster with multiple containers), but the data will of course go through it, via the network calls that streams the files from and to the backend storage.

 

Userlevel 4
Badge +9

Hi @Mathias Dahl ,

As i understand the requirement mentioned by @DeanBurell, my question for clarification is:

Will the future solution support the capability to assign document classes to different external file storages. 

/Johannes

Userlevel 3
Badge +5

@Mathias Dahl  @JohannesWittwer Correct.  Cloud storage for documents is not allowed for certain types of documentation in the US utility industry (sensitive documents that expose cyber security risks).  Also for companies that serve both commercial and defense industries that do not have a full requirement to be in gov cloud but do have some classes of documents that are ITAR (drawings) or have similar requirements.

The ability to have an on prem/noncloud  document vault/repository is a critical requirement.

It sounds like from your description that you will not support a cloud deployment with a non cloud external repository.  This is a problem not only with new cloud prospects, but preventing existing customers from moving to the IFS Cloud.

 

Dean

Userlevel 7
Badge +30

@DeanBurell 

It sounds like from your description that you will not support a cloud deployment with a non cloud external repository. 

That's where we are today, yes.

If you need storage outside the database and cannot use Azure Storage, then Remote Deployment is your only option, and has been for a number of releases now. What I wrote in my original post here doesn't change this fact.

What changes is the number of storage options outside the database in Remote Deployment, from 3 to 1. And, at least initially, the setup will be less flexible since you can only configure one external location.

I recommend everyone who want more options and capabilities to create an idea in the Ideas section here on IFS Community.

Thanks!

 

 

 

Userlevel 3
Badge +5

@Mathias Dahl  by number of releases you mean IFS Cloud.  because  this was supported up through IFS 10 and had been since I started working with IFS in 1997.    What you have essentially done is remove functionality that customers have been using for 20+ years and have now prevented them from using IFS in the Cloud.   

You have also hampered any efforts to sell IFS in the Cloud to any new customers that have regulatory requirements or the desire to store documents externally for other reasons i.e. size, network distribution, etc.  We will now need to either get them to pay the 30% premium to move into Gov Cloud or recommend they go on prem, which will probably get us eliminated from opportunities that are looking for cloud solutions.

If your goal is to get all of the customers into the IFS hosted cloud, this just worked against that.

This isnt about people who want more options and capabilities, it is about you taking away options and capabilities they already have with out asking.  and now you are also only enabling 1 external location?   You do realize that IFS is a multi country, multi company application?   we utilize the multiple location capabilities to segregate documents across companies in IFS as well.   

I am getting the feeling that R&D is not thinking through these changes or asking any of their customers or partner about the impact of removing functionality prior to doing so.

Dean

 

Userlevel 7
Badge +30

@DeanBurell 

Thanks for the input. 

I recommend you file an idea here on IFS Community to get support for multiple storage locations in IFS Cloud File Storage in Remote deployment. If that can be implemented, Docman can also support it, and it will then be very similar to how it works today with our support for FTP and Shared repositories.

 

Userlevel 3
Badge +5

@Caleb Dahl   For companies that are commercial and sell to A&D, they may only have a few documents that are ITAR controlled, usually drawings or images of the part.   they cannot put those documents into a cloud repository whose infrastructure is not ITAR controlled or meets the stringent requirements that we see in the GOV cloud.  However, with an external repository that is secured and controlled,  they can still link those documents to objects in IFS.    That is the impact with respect to external repository.   The other issue is the limitation of the number of repositories, the limitation to one file storage repository is also problematic even if the customer is in the GOV cloud.  even with in the ITAR and defense requirements there are always requirements for encapsulation such that documents across programs are not co-mingled.  this is also a challenge for multi company, multi national companies that wish to keep physical documents segregated based on company or country.   Israel, France, China, Canada are all examples where you would want to segregate and regionalize the document repositories for security and performance reasons.

Userlevel 3
Badge +5

@Caleb Dahl   The only option if they want external or non-cloud repository is to implement on-prem or remote,  but even in this case my understanding is that they will still be limited to 1 database and 1 fileshare.

 

Badge +2

@Mathias Dahl the terminology and implications of this are incredibly confusing. what is the reason these options are going to be done from a technical standpoint? We have constant sales pressure to sell deals and storage options is discussed on almost every deal. Limiting options here could be lose deals in the multiple millions of dollars. The documentation implies to use anything other than Internal the customer needs to be using Remote (aka on-premise deployment) which IFS does NOT want to sell. 

Userlevel 7
Badge +30

@Caleb Dahl 

The documentation implies to use anything other than Internal the customer needs to be using Remote (aka on-premise deployment)

This post is about the deprecation of options in Remote deployment only.

Both Cloud and Remote deployment will continue to have an internal and external storage option, and both can be used in the same installation.

As for the reasons why we do this, I will have to refer to my original post as well as some of my answers for the first couple of comments. If things are still not clear, ask away and we will try to answer.

 

Userlevel 7
Badge +30

@DeanBurell 

It seems your idea ended up as a new post in this forum. You need to post it in the Ideas section.

Thanks!

 

Userlevel 7
Badge +30

@DeanBurell 

Also, your post is missing important details:

R&D has decided to drop support for external document repositories which has been core capabilities for at least 20 years.

Firstly, we remove two of three external storage options when using Remote deployment. Cloud deployment remains unchanged. Cloud deployment is limited compared to what you have been used to in IFS Applications, yes. It's been like that for three years.

Secondly, both Cloud and Remote deployment will have a internal (database) and an external (IFS Cloud File Storage) option.

What is missing to fulfill the requirements you mention is a way to configure IFS Cloud File Storage to support multiple network shares. That's what you should ask for when you create your idea.

Reply