First, copy the Mnemosyne program directory from C:\Program Files\Mnemosyne to your USB drive, and then copy your data directory to inside the Mnemosyne program directory on the USB drive.

Your data directory is usually C:\Users\<your user name>\Application Data\Mnemosyne, but depending on your Windows version and setup, it could also be C:\Users\<your user name>\AppData\Roaming\Mnemosyne, or a different variant of that. Note that this directory is typically hidden, so you'll have to configure Windows Explorer to also show hidden files.

Alternatively, you can start Mnemosyne with the -d option to specify where the data directory is located, e.g. F:\Mnemosyne\mnemosyne.exe -d F:\mnemosyne_data_dir. This method has the disadvantage however that it's not robust against drive letter changes.

If you have Mnemosyne both on your USB drive and on your hard drive, make sure that you start the correct version. Starting the program on your hard drive will use the data directory on your hard drive, and starting the program on your USB drive will use the data directory on your USB drive. If you mix these two up, you will get two diverging copies of your data.