Yes, I've found countless issues with ArcGIS 10.1. Here's what I would do based on my own experience as there are tons of things that could affect performance:
1) Clear the results from the results tab. These stored 'results' can cause ArcGIS to take a long time in loading.
2) If you have a lot of toolboxes, only load the ones you need. I've had countless issues with Production Mapping and ArcHydro toolboxes in the past.
3) Consider using mosaic datasets if you need to have lots of imagery in your table of contents. Also, ArcGIS tends to handle GeoTiffs (even though they are larger) better than compressed imagery such as ecw's due to external pyramid creation.
4) If the rasters are located on a network, then try putting them on your local drive and then loading them into ArcMap. I try not to work on large datasets over the network if possible. I copy it over when I'm done.
5) If your toolbars look like a CAD workspace, then you should probably streamline the amount of tools you actively display.
6) Delete your "normal.mxt" as a last resort.
7) Turn off any antivirus software.
8) Ensure that all service packs are up-to-date and all patches have been applied. Often, these are bugs that are ironed out in service packs.
9) If you are trying to load online imagery into ArcGIS, ensure that caching is enabled in the properties tab of the imagery (if possible).
10. If you don't need to use ArcGIS Online then rename the ArcGISConnectionBin.exe to ArcGISConnectioBin.exe.BAK. You will not be able to use bing imagery or any other type of online service when you do this.
11.) If you have both internal and discrete graphics cards on your computer, try disabling one and using the other and see if that helps. I've had issues with my nvidia card before with ArcGIS.
12.) Ensure that windows 7 is completely up-to-date as well as all your hardware.
13.) Try to repair the install from the disc or image you've used.
14.) Wipe out the program, remove the registry entries and start fresh.
Curious, when you say "using many rasters" do you mean types other than ecw or actually using lots of rasters in a map (i.e. having 50 raster datasets in one map document)? If the latter is the case, then this is definitely going to slow you down and you should create a mosaic dataset or at the very least, merge them all into one single dataset.
ArcMap 10.1 is very very fickle. I've had to delete my normal template more times than I can count, whereas in ArcMap 10 I rarely had to do it.