One common cause is a virus scan program, they often do a scan when shutting down to check for a boot virus on any floppy that might be in the drive.  They can also be set to scan at regular intervals or when the screen saver activates.

Another cause is an entry in the recent document list, try right clicking on the task bar, select properties from the pop-up menu.  Then on the dialog box that appears, click on the start menu programs tab, then click on the clear button, then click OK or Cancel to get rid of the dialog box.

Some people have also reported this happening after running executable programs from the floppy drives, although I seem to recall this problem went away after putting a disk in the drive.

DOS programs run from the floppy may leave a .PIF file on the system that was created to tell the computer how to run the program (memory space to use, full screen or run in a window, etc.).  You can scan for these by placing a FORMATTED floppy in the drive, then clicking on start | find | files or folders, then on the dialog box type *.pif in the named area, then click on the advanced tab and in the containing text area type A: or whatever drive letter your floppy drive uses.  You can send any of these you don't want to your recycle bin.  You would probably want to keep programs in your start menu.  I believe that the ones in the PIF folder in the windows directory are ones that a created by windows and could be deleted, although if you are not sure about a file, you should probably keep it.

During startup it is normal.  You would normally want the computer to boot from the floppy drive if a disk is present, if not it boots from your hard drive.  This makes getting access to the computer if there is a system crash easier by letting you boot to DOS from a floppy.
NOTE: Booting from a floppy does expose your system to boot viruses.  It is possible to change the boot order in most BIOS's to start from the hard drive first.  If you do need to boot from the floppy you can go into the BIOS and change it back to floppy drive first as needed.

Another possibility is there is a reference to your floppy drive in your autoexec.bat file's path statement.

Another possibility is the office 97 fast find utility.

Another option would be that there is an entry in a program's recent file list.  It might be trying to locate files that had previously been opened by the program.