Can you share some screenshots? If you create a new revision of a document that had a .doc file checked in, you will get a copy of that in the new revision if you select to copy the file (if not, no file is copied). If you then want to change the file type, we don't really have a convenient way of doing that. You then need to get a copy of the new revision (by using View or Copy File To...), then delete the file from that revision, then upload the new file (.docx).
If you tried to Edit the revision that has a .doc file, and check it in using a .docx file, it might fail, since it's a scenario I don't think we have code to handle.