Changing document repository

About this document

The purpose of this document is to give you the knowledge needed to "manually" migrate files between document repositories. It's important to first understand how Docman keeps track of the files that is stored in IFS.

Data model

These are the entities involved when document files are stored and retrieved.

Apart from just reading the below, it is a good idea to actually look at the contents of the database tables for the entities. Just by looking at the data in EdmFile and EdmLocation, together with the data in your repostitory, should give you a good idea on how things fit together.

EdmFile (edm_file_tab)

This is what we call the "file reference" of document / document revision. This entity is used to store the original file name as well as the file name in the document repository. Another important attribute is the storage location.

Here are the most important attributes:

• FileName: This is the file name used in the repository / backend. It's not relevant for Database repositories, but it's important for the other repository types. This is usually generated by the system and it MUST be unique in the folder/place where the files are stored.
• LocationName: This is the name of the document repository where the file is stored. It points to/references a "location" (repository) in EdmLocation
• Path: If the file is not stored in the "root" of the repository, then the path is kept here.

EdmFileStorage (edm_file_storage_tab)

Used in Database story only. This is where the file information is stored. One record per file stored.

Here are the most important attributes:

• FileData: This is a BLOB column (binary large object) that can keep large file content.

EdmLocation (edm_location_tab)

This is where the repositories are defined, which are pointed to/referenced by the EdmFile entity (file reference).

• LocationName: The repository name. This is the ID/name and its the primary key of the repository.
• LocationType: The repository type. This is a number and can have one of the following values: 1) Shared, 2) FTP, 3) Database, and 4) File Storage
• LocationAddress: Relevant for Shared and FTP repositories. The address as defined in EdmLocationUser.
• Path: if the repository type allows to point to a specific default value for the folder/directory inside the repository, it's set here. Relevant only for Shared and FTP repositories.

EdmLocationUser (edm_location_user_tab)

• LocationAddress: The IP addres or name of the server. Relevant only for Shared and FTP repositories.
• LocationUser: The username used to log in to the server.
• LocationPassword: The password used to log in to the server.

Scenarios

FTP or Shared server name needs to change

This scenario is very simple to handle: just update the repository address in the EdmLocation and EdmLocationUser entities.

Change repository type

This can be easy or hard depending on what the change is.

FTP <-> Shared

Changing between FTP and Shared repositories is quite simple:

• Move the files (if needed, they might already be stored in the correct physical location)
• Change the repository type in EdmLocation (this needs to be done from an SQL tool)
• Change the Repository Address to have a value that matches the repository type
• Make sure the Path in EdmFile, of not empty, exists in the new repo and that the file is stored there.

FTP/Shared -> Database

This is more complex because the files cannot just be moved using normal file copy tools.

• Change repository type in the EdmLocation entity.
• Read file content from the FTP or Shared folder and write file to EdmFileStorage.FileData. The record should be removed from EdmFileStorage once the data has been read from it and written to a file in the new repository.
• Add Path information in EdmFile if a file has been moved to a FTP or Shared repository and if the file is not stored in the root (top) folder.

FTP/Shared/Database -> File Storage

This is the most complex option because File Storage (the IFS Cloud File Storage service/capability of the platform) has its own file reference registry. For each file that Docman keeps, and apart from the EdmFile record in Docman, a record is created in the file reference of File Storage.

In order to understand how to move a file from FTP/Shared/Database in Docman, we need to understand how File Storage keeps information about the files it stores.

FssFile (fss_file_tab))

This is where the file references are stored in File Storage.

These are the important attributes:

• FileId: This is the unique ID of the file which is used as the ID/key in the back end to File Storage. In the case of Docman, we use the FileName from EdmFile, since that is unique.
• LuName: The owner of the data. Meant to be an LU name, but in the case of Docman, the value is "docman".
• FileName: This is the original, or "nice" filename. It's secondary and just extra metadata in the backend storage. In the case of Docman, the original file name is used (EdmFile.UserFileName)
• FileExtension: The file extension, in lowercase.
• FileType: Seems to be hard coded to "application/x-www-form-urlencoded".
• FileLength: This MUST be set to the length of the file content, in bytes.
• RowState: Will have the value "Active" once the file has been virus scanned. The vlaue "QueuedForScanning" means the file is in the queue to be scanned. If any other value, something went wrong. If you want to disable virus scanning during migration, set this value to "ScanDisabled".

Use the Fss_File_API.Create_File_Ref_Rec method to create records in this entity.

If records are created here by INSERT statements, remember to populate also the RowKey and RowVersion attributes with a GUID and SYSDATE, respectively.

Steps

These would be the theoretical steps to move a file to File Storage:

• Create the record in the FssFile entity. FileId is very important and is the FileName attribute from EdmFile (see the earlier section for information on the other attributes in FssFile).
• Change the repository type in EdmLocation OR update LocationName on all relevant records in EdmFile to the new repository that uses File Storage.
• Copy the file to the docman folder in the share controlled by File Storage. Do not create any extra sub folder since this is currently not supported in File Storage.

Special file attributes

It has been reported that File Storage sets some special file attributes in Remote. This does not happen from the File Storage service itself so it might be Kubernetes that does this. We currently don't know if this is important or not for the operation.

Related discussion: