I use my old Trek 2120 that has Shimano 105 components in the basement with my trainer. Recently, the right shifter has been acting up. Sometimes when trying to downshift to smaller cogs it won't shift at all, and other times it will actually start shifting the rear derailleur to higher cogs instead of smaller cogs!

Any know what might be causing this problem and how to repair it?

Spray some lube in the shifter and shift it for a while to work the lube into the shifter. You may end up using a bit of lube but it should clear out and shift again.

let me know how it works out for you.

Couple of points: First, going to smaller cogs is UPshifting, to a higher gear, not DOWNshifting. The "higher" and "smaller" cogs are the same thing. Getting the terminology right makes it easier to describe your problem.
Shifting 101: The derailleur is forced onto the lower gears (larger cogs) by hand pressure. It goes to the higher gears (smaller cogs) with the help of a spring. If the shifter, cable or derailleur are dirty or unlubricated, sometimes the spring isn't strong enough to overcome the friction and move the derailleur outward. So lube the shifter, as another post recommended, but also clean and lube the pivot points on the derailleur and either replace or remove, clean and lube the cable. One place to check is where the cable goes under the bottom bracket. Sometimes just a drop of oil right where it rubs on the BB shell is enough to fix this problem.
Lube the chain while you're down there, too. It won't help with this specific problem, but I imagine it hasn't been done in awhile.
If it's really shifting to a larger cog (lower gear) on its own, the bike is possessed by demons. The only thing I can think of offhand that would cause that is if the cable etc are REALLY dirty and the adjustment is a little bit off, so it takes a few revolutions for everything to realize what you've asked it to do. If you get everything cleaned up and that still happens, google "rear derailleur adjustment" or go to www.parktool.com for instructions.

In addition to Cory's good advice, a bent derailleur hanger could also account for mystery shifting.
If the cables and housing are than two years old they are likely the problem. Replacing the cables and especially the cable housings may greatly improve shifting performance. And the short housing just before the rear derailleur is the most critical.
Hosing out the shifters with WD 40 may also help.

Al